Mobile robot trajectory planning method based on improved DDPG algorithm

Abstract

A mobile robot trajectory planning method based on an improved DDPG algorithm selects a double-wheel differential drive mobile robot as a design and simulation object, and enhances the effect of visual navigation in an optical feature complex environment by optimizing a visual preprocessing process. According to the visual image characteristics, the method adjusts and optimizes the network structure of the basic DDPG algorithm. Meanwhile, for the problem of sparse rewards in the complex space, a fusion reward function combining sparse rewards and non-sparse rewards is designed, and the learning guiding capacity of the algorithm in the complex space is improved. In addition, a priority experience playback mechanism is introduced into the improved method, and the learning ability of important features in part of known environments is remarkably improved. According to the method, the training convergence speed is higher, the training stability is higher, and the trajectory obtained through planning is better.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile robot trajectory planning, and in particular to an efficient trajectory planning method based on an improved deep deterministic policy gradient (DDPG) algorithm. Background Art

[0002] With the growing demand for automation in various industries, the application scope of mobile robots continues to expand and technical requirements continue to increase. New industrial scenarios require robots to work in more complex environments. As one of the core technologies for mobile robot navigation planning, trajectory planning technology needs to guide robots to complete navigation planning tasks in complex environments such as those without global map information. Ideal and efficient navigation planning technology can quickly plan safe and feasible trajectories, providing guarantees for the robot's navigation performance in complex tasks.

[0003] In recent years, deep reinforcement learning methods have gradually become a research hotspot in the field of mobile robot trajectory planning because of their powerful feature capture capabilities and the requirements of navigation planning in complex environments. At present, deep reinforcement learning methods are mainly divided into two categories: value function-based methods and policy-based methods. Value function-based methods select agent strategies by iteratively updating value functions and are mainly applicable to discrete spaces. For example, the deep Q network (DQN) proposed by the Google DeepMind team in 2013 uses a multi-layer neural network instead of a Q value table to achieve state-to-action mapping based on traditional reinforcement learning methods. However, the DQN algorithm cannot be applied to continuous action spaces, and there are problems such as overestimation, high training costs, and insufficient memory of historical information. To solve these problems, Hausknecht M et al. added a recurrent neural network layer (LSTM) to the DQN structure and proposed a deep recurrent Q network (DPQN). Although DPQN solves some of the problems of DQN, it still has defects such as high training costs and difficulty in handling continuous action space tasks.

[0004] Policy-based deep reinforcement learning methods directly learn policies and are applicable to continuous action spaces and high-dimensional spaces. Silver D et al. proposed a deterministic policy gradient algorithm (DPG), which uses deterministic sampling and gradient descent to learn policies. In order to further expand the scope of application of the DQN algorithm, Lillicrap TP et al. merged the DQN structure with the DPG structure and proposed a deep deterministic policy gradient algorithm (DDPG). The DDPG algorithm extends the scope of application of the DQN algorithm to continuous action spaces by adding a policy network and using a deep neural network to approximate the deterministic strategy of the value function. Although the DDPG algorithm has excellent results in trajectory planning, it cannot effectively utilize the time series information in the environment space, has a complex structure and high training cost. In addition, the DDPG algorithm has limited ability to utilize important and effective features in complex unknown environments, and cannot optimize strategies in a direction for important features.

[0005] In addition, the current mobile robot trajectory planning method based on the DDPG algorithm is mainly based on the design of the algorithm structure based on the laser sensor, and visual sensors are rarely used or used as auxiliary sensors for multimodal fusion systems. Laser sensors have the advantages of high ranging accuracy and strong anti-interference ability, but their ability to identify the types of objects in the environment is insufficient and the cost is high. In contrast, visual sensors can better capture environmental features, but are sensitive to changes in environmental optical features and have poor performance in dark, shadowy environments. Summary of the invention

[0006] In order to overcome the shortcomings of the existing technology, the present invention provides a mobile robot trajectory planning method based on an improved DDPG algorithm. The method selects a two-wheel differentially driven mobile robot as the design and simulation object, and optimizes the visual preprocessing process to enhance the effect of visual navigation in an environment with complex optical features. According to the characteristics of the visual image, the method adjusts and optimizes the network structure of the basic DDPG algorithm. At the same time, in response to the problem of sparse rewards in complex spaces, a fusion reward function combining sparse rewards and non-sparse rewards is designed to improve the learning guidance ability of the algorithm in complex spaces. In addition, the improved method introduces a priority experience replay mechanism, which significantly improves the learning ability of important features in some knowable environments.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A mobile robot trajectory planning method based on an improved DDPG algorithm comprises the following steps:

[0009] Step 1: Mobile robot motion control modeling: Analyze the kinematic characteristics of the two-wheel differential motion robot, establish a motion control model, determine the input and output of the controller, and establish a closed-loop control system;

[0010] Step 2: Preprocessing of visual sensor images: register the image information collected by the binocular visual sensor, use Gaussian bilateral filtering to reduce noise and downsample the original image, use histogram equalization and Sobel operator sharpening methods to highlight the edge features of the image, and perform maximum-minimum normalization on the preprocessed image to obtain the input of the intelligent agent decision network;

[0011] Step 3: Build and initialize the agent network: The current Actor network and the target Actor network of the improved DDPG algorithm have the same structure and parameter settings, including the input layer, convolution layer, linear layer, GRU network layer, fully connected layer and output layer. The current Critic network and the target Critic network have the same structure and parameter settings. The Critic network adds an action input layer, a Concat connection layer and a third linear layer based on the Actor action network structure.

[0012] Step 4: Initialize the experience replay pool and build a priority experience replay mechanism: Initialize the experience replay pool, set up the sampling mechanism and initialize the segment tree root node;

[0013] Step 5. Design a fusion reward function: To address the sparse reward problem in deep reinforcement learning, optimize the "0-1" reward setting, combine the characteristics of trajectory planning tasks and obstacle avoidance requirements, design a reward function that integrates sparse rewards and non-sparse rewards, and adjust the distribution of rewards in the environment space;

[0014] Step 6: Improve the DDPG algorithm deterministic strategy to add noise;

[0015] Step 7: Agent interacts with the environment: The agent observes the environment and obtains the state s t , the state s t Input the current Actor network to get the optimal action a t , perform action a t The next state s in the post-observation environment t+1 , get reward r t , the state s t and action a t Input the current Critic network to get the action value Q, calculate the sampling probability P according to the sampling probability formula, and convert the five-tuple (s t , a t ,s t+1 , r t， P) Put it into the experience replay pool, calculate the gradient, and update the current Actor network and the current Critic network parameters;

[0016] Step 8. Update the target network parameters: Use soft update to regularly update the target Actor network and target Critic network parameters. After each training round, assign the target network parameters to the current network to improve the stability of training. When the experience stored in the experience replay pool reaches the threshold, the model extracts experience according to the sampling probability and adds it to the training.

[0017] Step 9: Iterative training: Repeat steps 7 to 8 until the model converges.

[0018] Furthermore, in step 1, the process of establishing the mobile robot motion control model is:

[0019] The linear velocity and angular velocity of the robot's center of mass coordinate system can be directly calculated from the rotation speed of the left and right wheels of the two-wheeled mobile robot. The calculation method is as follows:

[0020]

[0021] Among them, ω c and v c is the angular velocity and linear velocity of the robot; r c is its real-time turning radius; L is the wheelbase of the robot’s left and right wheels; the rotation speeds of the left and right wheels relative to the center of the robot are ω L ,ω R ; r is the radius of the left and right wheels;

[0022] Considering that the left and right wheel motors of the mobile robot are brushless DC motors, a mapping relationship between the left and right wheel speeds of the robot and the working voltages of the left and right motors is established; the corresponding left and right wheel speeds are obtained by controlling the left and right wheel motor voltages of the mobile robot, thereby achieving the desired angular velocity and linear velocity of the robot. The calculation method is as follows:

[0023]

[0024] Among them, ω L is the left wheel angular velocity; ω R is the right wheel angular velocity; v L is the linear velocity of the left wheel; v R is the linear speed of the right wheel; V L is the voltage applied to the left wheel motor; V R is the voltage applied to the right wheel motor; k e is the back electromotive force coefficient of the motor; the transmission ratio G is the ratio of the motor angular velocity to the wheel angular velocity, and the calculation method is as follows:

[0025]

[0026] Preferably, in step 1, in order to improve the stability of the mobile robot control, a voltage closed-loop control system is added, the speed detected by the odometer is fed back to the input end, converted into motor voltage and compared with the target voltage, and the output is fine-tuned by the controller to enhance the control effect.

[0027] Furthermore, in step 2, the image collected by the visual sensor is converted into a grayscale image after registration, and a Gaussian bilateral filter is used for downsampling filtering. The Gaussian bilateral filter calculation method is:

[0028]

[0029] Among them, x i is the current pixel coordinate; I(x i ) is the pixel x in the original image i The value of f r is the range filter; f s is the spatial filter; f r and f s All are functions based on Gaussian distribution.

[0030] Preferably, in step 2, the filtered image is processed by histogram equalization to stretch the contrast space, and the Sobel operator sharpening method is used to highlight the image edge. The Sobel edge detection operator uses the weighted sum of 8 pixels in the square area around the detection pixel to determine whether the intensity of the point belongs to the edge.

[0031] In the step 2, the sharpened image is normalized to the maximum and minimum values ​​to obtain the improved DDPG algorithm decision network input image.

[0032] Furthermore, in step 3, the current Actor network uses the preprocessed image information as state input, the input layer is connected to the first convolutional layer to preliminarily extract image features, the first convolutional layer is connected to the second convolutional layer after Max Pooling, and further extracts image information, the second convolutional layer is connected to the first linear layer after Max Pooling, the first linear layer is connected to the GRU memory network layer, and after the GRU network layer memorizes the historical information, it is connected to the output layer to output a two-dimensional action vector.

[0033] In step 3, the current Critic network is also identical to the target Critic network structure. The current Critic network has two input layers. The first input layer has the same structure as the Actor network input layer. The preprocessed image information is input. The first input layer is connected to the first convolutional layer. The first convolutional layer is connected to the second convolutional layer through Max Pooling. The second convolutional layer is connected to the first linear layer through MaxPooling to obtain the state vector to be spliced. The second input layer inputs the two-dimensional action vector output by the current Actor network. The second input layer is connected to the second linear layer to extract action features to obtain the action vector to be spliced. The state vector to be spliced ​​and the action vector to be spliced ​​are input to the Concat layer for splicing, and then pass through the GRU memory network layer and output the value Q after full connection. Each layer uses relu as the activation function.

[0034] In step 4, the experience replay pool is initialized and the priority experience replay mechanism is constructed. The weights corresponding to different experiences are measured by the time difference error (TD-error) parameter. During model training, the experience in the replay pool is extracted according to the weight corresponding to the experience to update the model parameters. The sampling probability of each experience is:

[0035]

[0036] Where P is the sampling probability; δ t is the time difference error; ∈ is a very small number; α is the playback strength parameter.

[0037] In step 5, the fusion reward function includes two parts: a sparse reward and a non-sparse reward. The sparse reward includes a reward for the robot to reach the target area, a penalty when the robot collides with an obstacle, and a penalty for the robot falling into a loop; the non-sparse reward includes a penalty for the distance between the robot and the target position, a reward for the robot to move forward, backward, and rotate, and a safety penalty for the distance between the robot and the nearest obstacle. The reward represents a positive value, and the penalty represents a negative value.

[0038] In step 6, the deterministic strategy noise adding method is as follows:

[0039] a t =μ θ (s t |θ μ )+N t

[0040] at is the action selection at time t; μθ is the deterministic strategy; Nt is the noise value at time t. Since the numerical calculation of the decision-making process is discrete, the OU noise is discretized and its update formula is obtained as follows:

[0041]

[0042] Where θ is the speed parameter that controls the noise to return to the mean μ; μ is the long-term mean of the noise; σ is the parameter that controls the noise amplitude; N(0,1) is a random variable of the standard normal distribution; Δ t is the time step.

[0043] In step 8, after each training round, the method of regularly updating the parameters of the target actor network and the target critic network by soft updating is as follows:

[0044] θ′←-τθ+(1-τ)θ′

[0045] w′←τw+(1-τ)ω′

[0046] Among them, θ′ is the target Actor network parameter; θ is the current Actor network parameter; ω′ is the target Critic network parameter; ω is the current Critic network parameter; τ is the soft update target smoothing factor.

[0047] The beneficial effects of the present invention are mainly manifested in: motion control modeling of a two-wheel differential motion robot is carried out, a preprocessing process based on Gaussian bilateral filtering and Sobel sharpening is added, a trajectory planning module based on the DDPG algorithm is designed, and the network structure and strategy exploration mechanism therein are improved to solve the problems of insufficient historical information memory and exploration capabilities of visual planning tasks in map-free environments. At the same time, a fusion reward function is designed, and the reward sparsity problem is solved by adjusting the distribution of non-sparse rewards and sparse rewards in space, thereby improving the safety and convergence speed of robot training in complex environments. In addition, the introduction of a priority playback mechanism effectively improves the robot's utilization of important environmental features and reduces the time cost of robot training. Compared with existing methods, this method has faster training convergence speed, stronger training stability, and better planned trajectories. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A structural block diagram of a mobile robot trajectory planning method based on an improved DDPG algorithm provided by the present invention;

[0049] Figure 2 A structural block diagram of a mobile robot motion voltage closed-loop control system provided by the present invention;

[0050] Figure 3 The current Actor network and target Actor network structure diagram of the improved DDPG algorithm provided by the present invention;

[0051] Figure 4 The current critic network and target critic network structure diagram of the improved DDPG algorithm provided by the present invention;

[0052] Figure 5 A structural block diagram of the fusion reward function provided by the present invention;

[0053] Figure 6 A simulation environment diagram provided for an example of the present invention;

[0054] Figure 7 This is a reward curve diagram for simulation training of a mobile robot trajectory planning algorithm based on the improved DDPG algorithm provided by an example of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings.

[0056] Reference Figure 1 to Figure 7 , a mobile robot trajectory planning method based on the improved DDPG algorithm, comprising the following steps:

[0057] Step 1: Mobile robot motion control modeling, the process is as follows:

[0058] In this embodiment, the mobile robot uses the motion control model established by the earth center OXY coordinate system and the robot axis center coordinate system oxy respectively. The linear velocity and angular velocity of the robot in the oxy coordinate system can be directly calculated by the left and right wheel speeds of the two-wheeled mobile robot in the OXY coordinate system. The calculation method is as follows:

[0059]

[0060]

[0061]

[0062]

[0063]

[0064] Among them, ω c and v c is the robot angular velocity and linear velocity; r c is its real-time turning radius; L is the wheelbase of the robot’s left and right wheels; the rotation speeds of the left and right wheels relative to the center of the robot are ω L ,ω R ; The linear velocities of the left and right wheels in the world coordinate system are v L and v R ; r is the radius of the left and right wheels.

[0065] The left and right wheel motors of the two-wheeled mobile robot use brushless DC motors. According to the mapping relationship between the left and right wheel speeds of the robot and the left and right wheel motor voltages, the left and right wheel motor voltages of the mobile robot are controlled to obtain the corresponding left and right wheel speeds. The method is as follows:

[0066]

[0067]

[0068]

[0069] Where ke is 0.1V·s / rad; ω motor is the motor rotation angular velocity; ω wheel is the wheel rotation angular velocity.

[0070] In this example, a voltage closed-loop control system for a mobile robot is established. Figure 2 , the system outputs a two-dimensional action vector [V L ,V R ] and the mapping relationship between the speed voltage to calculate the desired left and right wheel speeds; to ensure the effectiveness of the control, a measurement and transmission closed-loop feedback loop is established based on the robot's angular velocity and linear velocity measured in real time by the robot's motion odometer, and the actual angular velocity and linear velocity are inversely transformed into actual voltage values ​​and fed back to the input end. Compared with the target voltage, the output is fine-tuned through the PI controller to stably control the robot to complete the trajectory planning task. The actual voltage calculation method is as follows:

[0071]

[0072] Step 2: Preprocessing of visual sensor images. The process is as follows:

[0073] After the image information collected by the binocular vision sensor is registered and converted into a grayscale image, a Gaussian bilateral filter is used for downsampling filtering. The Gaussian bilateral filter calculation formula is:

[0074]

[0075] where x i is the current pixel coordinate; I(x i ) is the pixel x in the original image i The value of f r is the range filter; f s is the spatial filter; f r and f s They are all functions based on Gaussian distribution, and the calculation methods are as follows:

[0076]

[0077]

[0078] Gaussian bilateral filter range filter standard deviation σ r Take 50; spatial filter standard deviation σ s Take 11.

[0079] The filtered image is processed by histogram equalization to stretch the contrast space, and the Sobel edge detection operator is used to calculate the weighted sum of 8 pixels in the square area around each pixel to determine whether the intensity of the point meets the edge characteristics. The horizontal and vertical convolution kernels of the Sobel operator are as follows:

[0080]

[0081] The sharpened image is normalized to the maximum and minimum values ​​to obtain the robot's current environmental state observation, i.e., the input of the agent decision network. The maximum and minimum value normalization method is:

[0082]

[0083] where x max is the point with the maximum pixel value in the image; x min is the minimum point of the image pixel value; the pixel values ​​of the processed image are all within [0,1].

[0084] Step 3: Build and initialize the agent network. The process is as follows:

[0085] In the present invention, the current Actor network and the target Actor network have the same structure. Figure 3 , network parameters refer to Table 1. The current Actor network takes the preprocessed image information as state input, the input layer is connected to the first convolution layer to preliminarily extract image features, the first convolution layer is connected to the second convolution layer through Max Pooling to further extract image features, the second convolution layer is connected to the first linear layer through Max Pooling, the first linear layer is connected to the GRU memory network layer, and the GRU network layer is connected to the output layer of the output action after linear full connection.

[0086] The current critic network has the same structure as the target critic network. Figure 4 , network parameters refer to Table 1, Table 1 shows the parameters of Actor network and Critic network,

[0087]

[0088]

[0089] Table 1

[0090] Step 4: Initialize the experience replay pool and build a priority experience replay mechanism. The process is as follows:

[0091] Initialize the experience replay pool and segment tree root node, and set up the probability sampling mechanism;

[0092] Improve the DDPG algorithm experience replay pool to store the five-tuple (s t , a t ,s t+1 , r t , P) as an experience, where s t is the current state, a t For the current Actor network according to s t The optimal action obtained, s t+1 is the state at the next moment, that is, the agent performs action a t The state of the post-environment, r t is the reward value obtained by the agent during the interaction with the environment in this round, and P is the sampling probability.

[0093] During the model training process, the size of the temporal difference error (TD-error) is proportional to the importance of the experience sample features and the difficulty of feature learning. Different experiences are assigned corresponding weights according to their size. The algorithm loss function updates the goal to minimize the TD-error value. The TD-error calculation formula is as follows:

[0094]

[0095] When training the model, the experience in the replay pool is extracted according to the weight corresponding to each experience in the replay pool and added to the training to update the model parameters. The sampling probability of each experience is:

[0096]

[0097] Where P is the sampling probability; δ t is the time difference error; ∈ is taken as 0.001; α is taken as 0.65.

[0098] In order to solve the problem of sample distribution feature loss caused by using TD-error parameters to establish a priority experience playback mechanism, the present invention uses importance sampling and normalization methods to correct the experience sampling probability. The corrected experience sampling probability calculation method is:

[0099]

[0100] Where ω is the correction sampling probability; N is the number of sampling times; β is the playback repair strength parameter; the initial value of β is 0.4, and it is continuously increased during training until it is close to 1.

[0101] The example of the present invention adopts a segmented tree structure to implement a priority experience replay sampling mechanism. The experience replay pool stores subclasses, parent class information, feature quintuples and TD-error values ​​through nodes. During the sampling process, the sampling interval is first divided according to the sampling probability of the sample batch and the first-level leaf nodes of the segmented tree. Then, a random number is generated in each interval, and the area division and expansion are performed starting from the second-level leaf nodes. If the random number is less than the left node value, the search is expanded along the left branch, otherwise it is expanded to the right. When the random number is expanded along the branches, it needs to be updated: the random number is subtracted from the left node value that falls into the interval to obtain a new random number for the next round of iteration. This process is repeated until it is expanded to the last layer of leaf nodes, and the experience obtained is the sampling result of this round.

[0102] Step 5: Design the fusion reward function. The process is as follows:

[0103] In view of the sparse reward problem in deep reinforcement learning, the "0-1" reward setting is optimized. Combined with the characteristics of trajectory planning and obstacle avoidance tasks, a reward function that integrates sparse rewards and non-sparse rewards is designed, and the spatial distribution of rewards is adjusted. For the structure of the integrated reward function, please refer to Figure 5 , the calculation method is as follows:

[0104] r=r sparse +r dense

[0105] Preferably, the sparse reward r designed in the present invention is sparse Including negative rewards r when the robot collides with an obstacle collision and the positive reward r when the robot reaches a certain range of the target point goal To prevent the robot from falling into a loop, when the total number of actions taken by the robot exceeds the threshold, a negative reward r of path dissipation is added deplete , the sparse reward calculation method is as follows:

[0106]

[0107] where t s Indicates the maximum time of a single iteration to prevent the algorithm from falling into a local optimal loop; d sp Indicates the minimum radius of the target area; d csp Indicates the minimum threshold of the safe distance from obstacles; && indicates an AND operation.

[0108] The non-sparse reward r designed by the example of the present invention dense Including the positive reward r for the robot to perform forward and backward actions action , the segmented reward r for the robot’s rotation action to adjust its own angle orient , the negative reward r in the direction of the distance between the robot and the target position destination, and the safety negative reward r of the robot’s distance to the nearest obstacle safety , || represents an OR operation, and the non-sparse reward calculation method is as follows:

[0109]

[0110] In order to improve the robot's exploration ability and prevent the robot from falling into a divergent state of continuous rotation, the robot's rotation action reward r orient Set it as segmented reward, that is, when the rotation angle is within the threshold, the robot is given a positive reward according to the rotation angle, and when the rotation angle exceeds the threshold, a negative reward is set. The calculation method is as follows:

[0111]

[0112] The distance direction reward r between the robot and the target position destination The calculation method is as follows, d goal Represents the Euclidean distance from the current position to the target position, K d is the target distance proportional constant.

[0113] r destination =K d *d goal

[0114] Safety reward r for the distance between the robot and the obstacle safety Refer to the gravity formula in physics. The calculation method is as follows. Only the obstacle closest to the robot is considered, K s is the safety distance proportional constant.

[0115]

[0116] Preferably, the fusion reward function parameter settings of the example of the present invention are shown in Table 2. Table 2 is the fusion reward function parameter settings.

[0117]

[0118] Step 6: Improve the DDPG algorithm deterministic strategy to add noise. The process is as follows:

[0119] In order to overcome the exploration capability limitation of the improved DDPG algorithm policy network, OU noise is added on the basis of the deterministic strategy to obtain a more exploratory action strategy selection. At the same time, adding OU noise when selecting actions can also effectively improve the exploration capability and robustness of the improved DDPG algorithm in the temporal inertial environment. The action strategy calculation method after adding noise is as follows:

[0120] a t =μ θ (s t |θ μ )+Nt

[0121]

[0122] The regression speed coefficient θ is taken as 0.15; the long-term mean value μ of the noise is taken as 0; the fluctuation amplitude σ is taken as 0.2; Δ t Take 1.

[0123] Step 7: The agent interacts with the environment. The process is as follows:

[0124] The agent observes the environment and obtains the state s t , the state s t Input the current Actor network to get the optimal action a t , perform action a t The next state s in the post-observation environment t+1 and reward r t , the state s t and action a t Input the current Critic network to get the action value Q, calculate the sampling probability P according to the training TD-error, and convert the five-tuple (s t , a t ,s t+1 , r t , P) put it into the priority experience replay pool, calculate the gradient and update the current Actor network and Critic network parameters;

[0125] Step 8: Update the target network parameters. The process is as follows:

[0126] After each training round, the target network parameters are assigned to the current network. The target Actor network and target Critic network parameters are updated regularly using a soft update method. When the experience stored in the priority experience replay pool reaches the threshold, the model extracts experience according to the sampling probability and adds it to the training. The storage threshold of the experience replay pool is 100,000.

[0127] Step 9, iterative training: Repeat steps 7 to 8, and iterative training until the model converges.

[0128] The present invention uses ROS environment and Gazebo to establish a simulation platform, and uses Turtlebot3-Waffle two-wheel differential motion robot as the simulation object. Figure 6 The simulation environment shown. The simulation environment size is 5*5m 2The scene resolution is 0.1, including 8 rectangular static obstacles with a length of 1m and a width of 0.2m, and two circular dynamic obstacles with a diameter of 0.4m. The black robot in the middle of the simulation environment is the initial position of the Turtlebot3-Waffle simulation robot. The path planning target point is randomly placed at any position on the map except for obstacles in each training, and the dynamic obstacles move randomly. This scene is used to test the performance of the algorithm in path planning in a complex dynamic environment.

[0129] The training rounds set in the example of the present invention are 200 times, and the simulation rounds are 40 times. During the simulation process, Turtlebot3-Waffle obtains the relative position of its own coordinate system to obtain the distance to the target point, and uses visual image input for navigation.

[0130] The upper limit of the linear speed of Turtlebot3-Waffle is set to 1m / s; the speed of dynamic obstacles is 0.5m / s; the upper limit of the linear speed during rotation is set to 0.05m / s; and the upper limit of the rotation angular speed is 30° / s.

[0131] The comparison of the simulation results of the improved DDPG algorithm provided by the present invention and its basic algorithms DDPG algorithm, DQN algorithm and PPO algorithm is shown in Table 3. Table 3 is the algorithm simulation results.

[0132]

[0133] Table 3

[0134] The improved DDPG trajectory planning method in the example of the present invention has relatively optimal effects in terms of training time, navigation success rate and average navigation time.

[0135] The training reward curve provided by the example of the present invention is as follows Figure 7 As shown in the figure, compared with the DDPG trajectory planning method, the mobile robot trajectory planning method of the improved DDPG converges faster and obtains a higher reward value in the end. The mobile robot trajectory planning method of the improved DDPG adds GRU to process time series information, and introduces a priority experience replay structure to guide the robot to learn important samples in a targeted manner, with a higher planning success rate. At the same time, directional learning also increases the learning efficiency of the algorithm, and the algorithm converges the fastest. The GRU structure and priority experience replay mechanism enable the robot to make full use of historical environmental information and predict the movement of obstacles to a certain extent, so the effectiveness of the planned path is also significantly improved.

[0136] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A mobile robot trajectory planning method based on an improved DDPG algorithm, characterized in that: The method comprises the following steps: Step 1: Mobile robot motion control modeling: Analyze the kinematic characteristics of the two-wheel differential motion robot, establish a motion control model, determine the input and output of the controller, and establish a closed-loop control system; Step 2: Preprocessing of visual sensor images: register the image information collected by the binocular visual sensor, use Gaussian bilateral filtering to reduce noise and downsample the original image, use histogram equalization and Sobel operator sharpening methods to highlight the edge features of the image, and perform maximum-minimum normalization on the preprocessed image to obtain the input of the intelligent agent decision network; Step 3: Build and initialize the agent network: The current Actor network and the target Actor network of the improved DDPG algorithm have the same structure and parameter settings, including the input layer, convolution layer, linear layer, GRU network layer, fully connected layer and output layer. The current Critic network and the target Critic network have the same structure and parameter settings. The Critic network adds an action input layer, a Concat connection layer and a third linear layer based on the Actor action network structure. Step 4: Initialize the experience replay pool and build a priority experience replay mechanism: Initialize the experience replay pool, set up the sampling mechanism and initialize the segment tree root node; Step 5. Design a fusion reward function: To address the sparse reward problem in deep reinforcement learning, optimize the "0-1" reward setting, combine the characteristics of trajectory planning tasks and obstacle avoidance requirements, design a reward function that integrates sparse rewards and non-sparse rewards, and adjust the distribution of rewards in the environment space; Step 6: Improve the DDPG algorithm deterministic strategy to add noise; Step 7: Agent interacts with the environment: The agent observes the environment and obtains the state s t , the state s t Input the current Actor network to get the optimal action a t , perform action a t The next state s in the post-observation environment t+1 , get reward r t , the state s t and action a t Input the current Critic network to get the action value Q, calculate the sampling probability P according to the sampling probability formula, and convert the five-tuple (s t , a t ,s t+1 , r t， P) Put it into the experience replay pool, calculate the gradient, and update the current Actor network and the current Critic network parameters; Step 8. Update the target network parameters: Use soft update to regularly update the target Actor network and target Critic network parameters. After each training round, assign the target network parameters to the current network to improve the stability of training. When the experience stored in the experience replay pool reaches the threshold, the model extracts experience according to the sampling probability and adds it to the training. Step 9: Iterative training: Repeat steps 7 to 8 until the model converges.

2. The mobile robot trajectory planning method based on the improved DDPG algorithm as claimed in claim 1, characterized in that: In step 1, the process of establishing the mobile robot motion control model is: The linear velocity and angular velocity of the robot's center of mass coordinate system can be directly calculated from the rotation speed of the left and right wheels of the two-wheeled mobile robot. The calculation method is as follows: Among them, ω c and v c is the angular velocity and linear velocity of the robot; r c is its real-time turning radius; L is the wheelbase of the robot’s left and right wheels; the rotation speeds of the left and right wheels relative to the center of the robot are ω L ,ω R ; r is the radius of the left and right wheels; Considering that the left and right wheel motors of the mobile robot are brushless DC motors, a mapping relationship between the left and right wheel speeds of the robot and the working voltages of the left and right motors is established; the corresponding left and right wheel speeds are obtained by controlling the left and right wheel motor voltages of the mobile robot, thereby achieving the desired angular velocity and linear velocity of the robot. The calculation method is as follows: Among them, ω L is the left wheel angular velocity; ω R is the right wheel angular velocity; v L is the linear velocity of the left wheel; v R is the linear speed of the right wheel; V L is the voltage applied to the left wheel motor; V R is the voltage applied to the right wheel motor; k e is the back electromotive force coefficient of the motor; the transmission ratio G is the ratio of the motor angular velocity to the wheel angular velocity, and the calculation method is as follows:

3. The mobile robot trajectory planning method based on the improved DDPG algorithm as described in claim 2 is characterized in that: In step 1, in order to improve the stability of the mobile robot control, a voltage closed-loop control system is added, the speed detected by the odometer is fed back to the input end, converted into motor voltage and compared with the target voltage, and the output is fine-tuned by the controller to enhance the control effect.

4. The mobile robot trajectory planning method based on the improved DDPG algorithm according to any one of claims 1 to 3, characterized in that: In step 2, the image collected by the visual sensor is converted into a grayscale image after registration, and a Gaussian bilateral filter is used for downsampling filtering. The Gaussian bilateral filter calculation method is: Among them, x i is the current pixel coordinate; I(x i ) is the pixel x in the original image i The value of f r is the range filter; f s is the spatial filter; f r and f s All are functions based on Gaussian distribution; The filtered image is processed by histogram equalization to stretch the contrast space, and the Sobel operator sharpening method is used to highlight the image edge. The Sobel edge detection operator uses the weighted sum of 8 pixels in the square area around the detection pixel to determine whether the intensity of the point belongs to the edge; The sharpened image is normalized to the maximum and minimum values ​​to obtain the input image of the improved DDPG algorithm decision network.

5. The mobile robot trajectory planning method based on the improved DDPG algorithm according to any one of claims 1 to 3, characterized in that: In the step 3, the current Actor network uses the preprocessed image information as state input, the input layer is connected to the first convolutional layer to preliminarily extract image features, the first convolutional layer is connected to the second convolutional layer after Max Pooling, and further extracts image information, the second convolutional layer is connected to the first linear layer through Max Pooling, the first linear layer is connected to the GRU memory network layer, and after the GRU network layer memorizes the historical information, it is connected to the output layer to output a two-dimensional action vector.

6. The mobile robot trajectory planning method based on the improved DDPG algorithm according to any one of claims 1 to 3, characterized in that: In the step 3, the current Critic network is also exactly the same as the target Critic network structure. The current Critic network has two input layers. The first input layer has the same structure as the Actor network input layer. The preprocessed image information is input. The first input layer is connected to the first convolutional layer. The first convolutional layer is connected to the second convolutional layer through Max Pooling. The second convolutional layer is connected to the first linear layer through MaxPooling to obtain the state vector to be spliced; the second input layer inputs the two-dimensional action vector output by the current Actor network. The second input layer is connected to the second linear layer to extract action features to obtain the action vector to be spliced. The state vector to be spliced ​​and the action vector to be spliced ​​are input to the Concat layer for splicing, and then pass through the GRU memory network layer. After full connection, the value Q is output, and each layer uses relu as the activation function.

7. The mobile robot trajectory planning method based on the improved DDPG algorithm according to any one of claims 1 to 3, characterized in that: In step 4, the experience replay pool is initialized and the priority experience replay mechanism is constructed. The weights corresponding to different experiences are measured by the time difference error (TD-error) parameter. During model training, the experience in the replay pool is extracted according to the weight corresponding to the experience to update the model parameters. The sampling probability of each experience is: Where P is the sampling probability; δ t is the time difference error; ∈ is a very small number; α is the playback strength parameter.

8. The mobile robot trajectory planning method based on the improved DDPG algorithm according to any one of claims 1 to 3, characterized in that: In step 5, the fusion reward function includes two parts: a sparse reward and a non-sparse reward. The sparse reward includes a reward for the robot to reach the target area, a penalty when the robot collides with an obstacle, and a penalty for the robot falling into a loop; the non-sparse reward includes a penalty for the distance between the robot and the target position, a reward for the robot to move forward, backward, and rotate, and a safety penalty for the distance between the robot and the nearest obstacle. The reward represents a positive value, and the penalty represents a negative value.

9. The mobile robot trajectory planning method based on the improved DDPG algorithm according to any one of claims 1 to 3, characterized in that: In step 6, the deterministic strategy noise adding method is as follows: a t =μ θ (s t |θ μ )+N t a t is the action selection at time t; μ θ is a deterministic strategy; N t is the noise value at time t. Since the numerical calculation of the decision process is discrete, the OU noise is discretized and its update formula is obtained as follows: Among them, θ is the speed parameter that controls the noise to return to the mean μ; μ is the long-term mean of the noise; σ is the parameter that controls the noise amplitude; N(0,1) is a random variable of the standard normal distribution; Δ t is the time step.

10. The mobile robot trajectory planning method based on improved DDPG according to any one of claims 1 to 3, characterized in that: In step 8, after each training round, the method of regularly updating the parameters of the target actor network and the target critic network by soft updating is as follows: θ′←τθ+(1-τ)θ′ w′←τw+(1-τ)w′ Among them, θ′ is the target Actor network parameter; θ is the current Actor network parameter; ω′ is the target Critic network parameter; ω is the current Critic network parameter; τ is the soft update target smoothing factor.

Images