TD3 map-free navigation method based on dynamic window method guidance

Abstract

The invention discloses a TD3 map-free navigation method based on dynamic window method guidance, and particularly relates to the technical field of path planning, and the method comprises the following steps: obtaining current state information and a target position of a robot; current state information and target position information are preprocessed and then input into an improved TD3 network to obtain a TD3 output action, an improved DWA output action is obtained in combination with an improved evaluation function and an escape mechanism, and the two obtained actions are input into an action selector to output an optimal action; and the robot executes the optimal action to complete the map-free navigation task. The TD3 network structure is improved, a DWA (dynamic window method) and an LSTM (long short-term memory network) are fused and applied to map-free navigation, the network structure can directly output robot actions, namely the linear velocity and the angular velocity, according to the input relative position of a target point and the state of the robot, and end-to-end map-free navigation is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and in particular to a robot map-free navigation method based on the combination of a DWA algorithm and a TD3 algorithm. Background Art

[0002] In recent years, with the rapid development of intelligent technology and autonomous driving technology, path planning methods based on deep reinforcement learning have received widespread attention from scholars. Navigation tasks are the core part of robot intelligent technology. Most path planning algorithms need to obtain prior maps in advance. However, it is extremely difficult to obtain prior maps in special scenarios such as underground exploration and post-disaster rescue. Therefore, map-free navigation algorithms have gradually become a key issue for the further development of mobile robots. At present, map-free navigation solutions can be divided into classic methods and deep reinforcement learning-based methods. In order to achieve a certain degree of autonomy, mobile robots need to find an optimal collision-free path from the starting position to the target end point based on environmental and task constraints. Although traditional algorithms can ensure that no collisions occur in most cases without maps, the robot's movement path is not optimal, resulting in poor navigation results. It is necessary to traverse the entire map to find the optimal path, which consumes a lot of time and computing resources.

[0003] Traditional navigation algorithms are not good enough in mapless scenarios. The environment in which robots are currently located is becoming increasingly complex and changeable. Previous methods cannot solve these problems well. Navigation methods based on deep reinforcement learning are gradually being widely used in robot path planning tasks. Deep reinforcement learning combines the unique advantages of deep learning and reinforcement learning, making it have both the decision-making learning ability of reinforcement learning and the feature extraction ability of deep learning, which can meet the needs of mapless navigation tasks. Compared with previous reinforcement learning methods, the TD3 (Dual Delayed Deterministic Policy Gradient) algorithm has been improved on the basis of predecessors. Its main improvement is based on the DDPG (Deep Deterministic Policy Gradient) algorithm. The TD3 algorithm improvement method draws on the idea of ​​DDQN (Dual Deep Q Network). In order to solve the overestimation problem, the Actor-Critic framework is adopted in the TD3 network architecture, which contains four Critic networks and two Actor networks. In addition, the update frequency of the Critic network is higher than that of the Actor, which makes the training process more stable and has a greater advantage in the performance of navigation tasks. However, the navigation method based on TD3 relies on training with a large amount of data. This process requires enough high-quality data to train a satisfactory model. In the early stage of training, the network model is not fully learned, resulting in poor navigation effect. The algorithm cannot collect high-quality training data and the learning efficiency is low. Summary of the invention

[0004] In order to solve the shortcomings of the background technology, the present invention improves the TD3 network structure, integrates DWA (dynamic window method) and LSTM (long short-term memory network) for map-free navigation, and the network structure can directly output the robot action, namely the linear velocity and angular velocity, according to the input target point relative position and robot state, to realize end-to-end map-free navigation.

[0005] In order to achieve the above-mentioned purpose, the present invention provides the following technical solutions: a TD3 mapless navigation method guided by a dynamic window method, obtaining the current state information and target position of the robot; after preprocessing the current state information and target position information, the preprocessed current state information and target position information are input into an improved TD3 network model to obtain a TD3 output action, and an improved DWA output action is obtained by combining an improved evaluation function with an escape mechanism, and the two obtained actions are input into an action selector to output an optimal action; the robot executes the optimal action to complete the mapless navigation task;

[0006] Among them, the improved TD3 network model introduces LSTM modules in the Actor and Critic networks. The training steps of the improved TD3 network model include:

[0007] (a) Start the Gazebo simulation environment, start the ROS node, load the robot model, obtain the current state of the robot and the target point position, and perform relevant preprocessing;

[0008] (b) Input the preprocessed action into the Actor network to obtain action a net , and obtain a through the output of the DWA planner dwa , input the two actions into the action selector, and output the final optimal action a max(Q) ;

[0009] (c) The robot performs the optimal action a in the Gazebo environment max(Q) , and returns the new state S t+1 , reward value r and whether the current round is completed, and the current state value, action value, reward value and state value at time t+1 (s, a, r, s t+1 ) is stored in the experience revisit pool;

[0010] (d) Until the amount of data in the experience replay pool is sufficient, randomly sample a sample batch of size batch_size from the experience replay pool; the (s, a, r, s t+1 ) sample batches to update the Actor network. During the TD3 algorithm training process, the network obtains (s, a, r, s) from the experience pool. t+1), the action at time t+1 is inferred through the target Actor network, and then the Q value is calculated by the two Critic networks, the target value y is calculated, the target Q value y is used to calculate the loss function of the Critic network, and the two Q networks are updated respectively;

[0011] (e) Evaluate the performance of the current strategy through the Critic network, calculate the average negative value of the Critic network output Q value to perform the gradient ascent operation, back-propagate to calculate the parameter gradient of the Actor network and update the Actor network parameters; the update of the Actor network is delayed, reducing the frequency of strategy updates and preventing the strategy from falling into a suboptimal solution too early. After each Q network and strategy network update, use soft update to update the parameters of the target network;

[0012] (f) Repeat steps (a)-(e) until the strategy converges or the preset number of training times is reached.

[0013] Preferably, the evaluation function of the improved DWA consists of three items: heading, dist, and vel, which respectively represent the azimuth angle, obstacle distance, and speed. When the robot falls into a local optimum, the heading item in the speed evaluation function will be ignored, and the dist item will be used as the main speed evaluation indicator, so as to prompt the robot to escape from the local optimum first.

[0014] Preferably, the escape mechanism is designed as follows:

[0015] Design a method to determine the robot's steering angular velocity range based on the robot's left and right laser radar detection data, and guide the robot to turn to a place where obstacles are not dense; the probability of turning left and right is:

[0016]

[0017] Where: dist left Represents the sum of the distances to the left obstacle detected by the LiDAR; dist right Represents the sum of the distances to the obstacles on the right detected by the LiDAR; L prob is the probability of turning left; R prob is the probability of turning right; C is a constant that can be determined according to the laser radar ranging range; the final escape linear velocity v and escape angular velocity ω are:

[0018]

[0019] Where: random(-R prob ,L prob ) represents a random number in the range from x to y.

[0020] Preferably, the improved TD3 network model introduces LSTM modules in the Actor and Critic networks to enhance the network's ability to process continuous navigation information, so that the Q value of the action can be obtained by inputting the lidar data, the target point position and the action to be evaluated into the Critic network, and the linear velocity and angular velocity can be obtained by inputting the lidar data and the target point position into the Actor network.

[0021] Preferably, the Actor network introducing the LSTM module concatenates the lidar data and the target point position together with a size of 1×22, inputs it into a fully connected layer of size 22×600, inputs the output features into a 600×600 LSTM layer, and then inputs the output features into a 600×2 fully connected layer, and the output through the Tanh activation function is the final action.

[0022] Preferably, the Critic network with the LSTM module concatenates the lidar data and the target point position and inputs them into a 22×600 fully connected layer, inputs the output features into a 600×600 LSTM layer, and then outputs the intermediate feature S1. The action to be evaluated is input into a 2×600 fully connected layer, which outputs the intermediate feature S2; the features S1 and S2 are combined, and the intermediate feature S is generated through the ReLU activation function, and S is input into a 600×1 fully connected layer, which outputs the Q value.

[0023] Preferably, an action selector based on the ε-greedy strategy is designed, which uses a large number of dwa As the agent performs actions, it helps the agent quickly learn to avoid obstacles and move towards the target point; at the end of training, the Action network is more used to obtain a net , which is conducive to the robot to fully explore the optimal action and improve the upper limit of the intelligent body's navigation performance. The mathematical expression of the action selector based on the ε-greedy strategy is as follows:

[0024]

[0025] In the formula, a max(Q) Represents the output action of the action selector; a net represents the action directly output by the TD3 network; a represents the action performed by the final robot, where the probability of 1-ε is a max(Q) , the probability of ε is a net ; In the early stage of training, the initial value of ε is small, and gradually increases to 1 as the training progresses; ensure that the agent learns more action selector output actions in the initial stage to improve training efficiency. In the later stage of training, the agent is allowed to explore more to further improve the model performance.

[0026] The present invention has the following advantages:

[0027] 1. The present invention makes improvements on the evaluation function. When trapped in the local optimum, an escape mechanism is adopted to preferentially escape from the local optimum. A method is designed to determine the robot's steering angular velocity range according to the left and right laser radar detection data of the robot, and guide the robot to rotate to places where obstacles are not dense, so as to solve the problem that in a map-free environment, the original DWA algorithm navigation often falls into the local optimum. At this time, the action with the highest evaluation score of the robot may be standing still. Although this can ensure that the robot will not collide, the robot cannot reach the target position.

[0028] 2. The present invention obtains the current state information and target position information of the robot; obtains the output linear velocity and angular velocity through the DWA planner; adds an LSTM module to the network structure of the TD3 algorithm to improve the network's ability to understand continuous navigation information, and obtains the angular velocity and linear velocity output by the improved TD3 algorithm; and uses an action selector to select the actions output by the DWA planner and the TD3 algorithm to achieve the purpose of improving the action quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the kinematic model of the dual-wheel differential chassis robot provided by the present invention;

[0030] Figure 2 A schematic diagram of the Actor-Critic network structure provided by the present invention;

[0031] Figure 3 Schematic diagram of the overall architecture of the DWA-guided TD3 algorithm provided by the present invention. DETAILED DESCRIPTION

[0032] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] In a first aspect, the present invention provides an improved DWA map-free navigation method;

[0034] Improved DWA mapless navigation method, including:

[0035] Get the robot's current state information and target position information;

[0036] Calculate the dynamic window and calculate the speed range in the future based on the current state and maximum acceleration of the robot;

[0037] Evaluate the optimal speed, the speed within the sampling window, and evaluate the sampled speed using the original evaluation function to select the optimal speed;

[0038] To escape from the local optimum, the present invention makes improvements to the evaluation function. When trapped in the local optimum, an escape mechanism is used to preferentially escape from the local optimal position. At this time, the evaluation function does not consider the heading item, and uses the dist item as the main basis for evaluating the speed, so as to prompt the robot to preferentially escape from the local optimum. Due to the limitation of the mechanical structure, the back of the robot is usually blocked, resulting in a blind spot in the observation area. Sometimes it is necessary to turn the direction to find a better solution. If the speed obtained by improving the evaluation function still cannot escape the local optimum, the robot is guided to obtain more environmental information through the in-situ rotation scheme. Therefore, the present invention designs a method to determine the robot's steering angular velocity range according to the laser radar detection data on the left and right sides of the robot, and guide the robot to rotate to a place where obstacles are not dense.

[0039] In a second aspect, the present invention provides an improved TD3 method guided by DWA;

[0040] DWA-guided improved TD3 methods, including:

[0041] Get the robot state, including current state information and target position information;

[0042] Get the output linear velocity and angular velocity through the DWA planner;

[0043] Adding LSTM modules to the network structure of the TD3 algorithm improves the network's ability to understand continuous navigation information and obtains the angular velocity and linear velocity output by the improved TD3 algorithm;

[0044] The action selector will be used to select the actions output by the DWA planner and the TD3 algorithm in order to improve the quality of the actions.

[0045] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0046] As an embodiment of the present invention, the overall architecture is as follows Figure 3 As shown, a TD3 map-free navigation method based on dynamic window method guidance includes the following steps:

[0047] Step 1: The data preprocessing phase includes the following steps:

[0048] Step 1.1: Set the trained model path, batch size, and other basic parameters;

[0049] Step 1.2: Processing LiDAR data;

[0050] Step 1.3: Get the current state information and target position information of the robot;

[0051] Step 2: Get DWA output action:

[0052] 2.1: Establish the kinematic model of the two-wheel differential robot Figure 1 , taking full account of its kinematic characteristics. Its kinematic model can be expressed as:

[0053]

[0054] Where: x, y represent the coordinate position of the robot; ω represents the angular velocity of the robot; v1, v2 represent the linear velocity of the left and right wheels of the robot respectively; v is the centerline velocity of the robot, which is (v1+v2) / 2; θ represents the angle between the center line of the two wheels and the x-axis; t represents the current time, and Δt represents the time change.

[0055] 2.2: Establish an environmental perception model to obtain surrounding obstacle information in real time. Divide the laser radar point cloud data into 18 intervals according to angle, take the laser beam with the closest distance in each interval as the state value of the interval, and form a state vector with a length of 18;

[0056] 2.3: Calculate the speed sampling interval of the DWA algorithm. Before speed sampling, the sampling interval needs to be calculated. Due to the inherent characteristics of the robot and environmental constraints, the speed space will be limited to a certain range, so there is a maximum and minimum value. The speed space selection needs to consider its limit speed constraint V m :

[0057] V m ={(v,ω)v∈[v min ,v max ],ω∈[ω min ,ω max ]};

[0058] Where: v min ,v max Represents the minimum and maximum linear velocity of the robot; ω min ,ω max Indicates the minimum and maximum angular velocity of the robot;

[0059] In actual engineering, the acceleration and deceleration of the robot are limited by the robot's power performance. Therefore, the robot's speed space is also constrained by the power strength, and the acceleration constraint V d The mathematical expression is as follows:

[0060]

[0061] Where: a v Indicates the maximum linear acceleration; a ω represents the maximum angular acceleration; v c ,ω c Respectively represent the current linear velocity and current angular velocity;

[0062] The robot can obtain the distance between itself and the obstacle through the laser radar, and a certain distance should be left to ensure that the robot can decelerate before hitting the obstacle. If the distance is not enough, the robot speed cannot be reduced and a collision is likely to occur. Therefore, the present invention constrains the speed of the robot before the collision, and this speed should be proportional to the distance to the obstacle. The obstacle constraint V a The mathematical expression is as follows:

[0063]

[0064] Where: dist(v,ω) represents the minimum distance between the predicted trajectory and the obstacle at the current speed. Combining the above three speed constraints, the final sampling speed space V in the DWA algorithm can be obtained as:

[0065] V=V m ∩V d ∩V a ;

[0066] 2.4 Speed ​​evaluation, use the evaluation function to evaluate the sampled speed to determine the optimal speed under this state. Since DWA is a local path planning algorithm, it is very easy to fall into the local optimum without a priori map. To address this problem, the present invention designs an escape mechanism based on the original rating function to improve the local optimum problem. The principle of the evaluation function is to sample multiple groups of speeds in the speed space that meets various constraints, simulate trajectories within a certain period of time, evaluate these trajectories through the evaluation function, and select a relatively optimal trajectory and speed. The mathematical expression of the evaluation function is as follows:

[0067] G(v,ω)=σ(α·heading(v,ω)+β·dist(v,ω)+γ·vel(v,ω));

[0068] Where: heading(v,ω) represents the angle error between the trajectory end direction and the target point at the current sampling speed; vel(v,ω) usually refers to the linear velocity; σ, α, β, γ are constants;

[0069] In a mapless environment, the original DWA algorithm navigation often falls into a local optimum. At this time, the action with the highest evaluation score of the robot may be to stay still. Although this can ensure that the robot will not collide, the robot cannot reach the target position. To solve this problem, the present invention makes improvements to the evaluation function. When falling into a local optimum, an escape mechanism is used to preferentially escape from the local optimum. At this time, the evaluation function does not consider the heading item, and uses the dist item as the main basis for evaluating the speed, so as to encourage the robot to preferentially escape from the local optimum. Therefore, when falling into a local optimum, the improved dynamic window method evaluation function becomes:

[0070] G(v,ω)=σ′(β′·dist(v,ω)+γ′·vel(v,ω));

[0071] Where: σ′, β′, γ′ are all constants;

[0072] Due to the limitation of mechanical structure, the back of the robot is usually blocked, resulting in blind spots in the observation area. Sometimes it is necessary to turn the direction to find a better solution. Therefore, the present invention designs a method to determine the robot's steering angular velocity range based on the robot's left and right laser radar detection data, and guide the robot to turn to a place where obstacles are not dense. The size of the left turn and right turn probability is:

[0073]

[0074] Where: dist left Represents the sum of the distances to the left obstacle detected by the LiDAR; dist right Represents the sum of the distances to the obstacles on the right detected by the LiDAR; L prob is the probability of turning left; R prob is the probability of turning right. C is a constant that can be determined according to the laser radar ranging range; the final escape linear velocity v and escape angular velocity ω are:

[0075]

[0076] Where: random(-R prob ,L prob ) indicates from -R prob To L prob A random number in a range;

[0077] Step 3: Get the TD3 algorithm output action:

[0078] Step 3.1: Build the TD3 network. Use Pytorch to create two Actor networks and four Critic networks. The structures of the two Actor networks are exactly the same, and the structures of the four Critic networks are also exactly the same. Introduce LSTM modules in the Actor and Critic networks to improve the feature extraction capability of continuous navigation information, fully refer to the previous data, and make the output trajectory smoother, which is more in line with the robot's safe movement requirements. The Actor and Critic network structure is as follows: Figure 2 As shown;

[0079] Step 3.2: The robot's current observed state s t and the current action a t Input into the built TD3 model, the output of the network is a TD3 ;

[0080] Step 4: Action selection based on ε-greedy strategy:

[0081] A ε-greedy action selector is proposed to guide the agent to perform more effective actions. The TD3 algorithm contains a Critic network, which is mainly used to judge the value of actions during training. During training, the action calculated by the decision network is a net , a net Input into the Critic network and get Q net1 , Q net2 , and the smaller value of the two Min(Q net1 , Q net2 ) is input to the action selector; the action obtained by the DWA controller is a dwa , the same dwa Input into the Critic network and get Q dwa1 , Q dwa2 , and the smaller value of the two Min(Q dwa1 , Q dwa2 ) is input to the action selector. The agent will select actions according to a certain strategy based on the Q value and execute the action with a larger Q value. max(Q) . The action selector based on the ε-greedy strategy will use a large number of a in the early stage of training. dwa As the agent performs actions, it helps the agent quickly learn to avoid obstacles and move towards the target point. At the end of training, more Action networks are used to obtain a net , which is conducive to the robot to fully explore the optimal action and improve the upper limit of the intelligent body's navigation performance. The mathematical expression of the final execution action a based on the ε-greedy strategy action selector is as follows:

[0082]

[0083] Where: a max(Q) Represents the output action of the action selector; a net represents the action directly output by the TD3 network; a represents the action performed by the final robot; in the early stage of training, the initial value of ε is small, and gradually increases to 1 as the training progresses. Ensure that the agent learns more actions output by the action selector in the initial stage to improve training efficiency. In the later stage of training, let the agent explore more to further improve model performance.

[0084] The specific steps for training the TD3 algorithm guided by the dynamic window method are:

[0085] Step A1: Initialize the training environment, obstacles, robot starting and ending points, and start Rviz;

[0086] Step A2: Get the current state of the robot, the LiDAR data and the relative coordinates of the target point. Preprocess the LiDAR through step 2.2, and finally get a LiDAR data with a length of 18. Input this information into the Actor network. The network will output the linear velocity and angular velocity, and get the DWA output action at the same time. The final robot execution action is output through the action selector, and the action reward value and the next state are saved.

[0087] Step A3: When the amount of data in the experience replay pool reaches a preset threshold, the Actor network and the Critic network are trained by randomly sampling the stored experience data;

[0088] Repeat the above process to complete the model training.

[0089] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A TD3 map-free navigation method based on dynamic window method to obtain the current state information and target position of the robot; After the current state information and the target position information are preprocessed, the characteristics are: The preprocessed current state information and target position information are input into the improved TD3 network model to obtain the TD3 output action, and the improved DWA output action is obtained by combining the improved evaluation function and the escape mechanism. The two actions obtained are input into the action selector to output the optimal action; the robot executes the optimal action and completes the map-free navigation task; Among them, the improved TD3 network model introduces LSTM modules in the Actor and Critic networks. The training steps of the improved TD3 network model include: (a) Start the Gazebo simulation environment, start the ROS node, load the robot model, obtain the current state of the robot and the target point position, and perform relevant preprocessing; (b) Input the preprocessed action into the Actor network to obtain action a net , and obtain a through the output of the DWA planner dwa , input the two actions into the action selector, and output the final optimal action a max(Q) ; (c) The robot performs the optimal action a in the Gazebo environment max(Q) , and returns the new state S t+1 , reward value r and whether the current round is completed, and the current state value, action value, reward value and state value at time t+1 (s, a, r, s t+1 ) is stored in the experience revisit pool; (d) Until the amount of data in the experience replay pool is sufficient, a sample batch of size batch_size is randomly sampled from the experience replay pool; the (s, a, r, s t+1 ) sample batches to update the Actor network. During the TD3 algorithm training process, the network obtains (s, a, r, s) from the experience pool. t+1 ), the action at time t+1 is inferred through the target Actor network, and then the Q value is calculated by the two Critic networks, the target value y is calculated, the target Q value y is used to calculate the loss function of the Critic network, and the two Q networks are updated respectively; (e) Evaluate the performance of the current strategy through the Critic network, calculate the average negative value of the Critic network output Q value to perform the gradient ascent operation, back-propagate to calculate the parameter gradient of the Actor network and update the Actor network parameters; after each Q network and policy network update, use soft update to update the parameters of the target network; (f) Repeat steps (a)-(e) until the strategy converges or the preset number of training times is reached.

2. The TD3 map-free navigation method based on dynamic window method guidance according to claim 1, characterized in that: The evaluation function of the improved DWA consists of heading, dist, and vel, which represent the azimuth, obstacle distance, and speed, respectively. When the robot falls into a local optimum, the heading term in the speed evaluation function will be ignored, and the dist term will be used as the main speed evaluation indicator, so as to prompt the robot to escape from the local optimum first.

3. The TD3 map-free navigation method based on dynamic window method guidance according to claim 1, characterized in that: The escape mechanism is designed as follows: Design a method to determine the robot's steering angular velocity range based on the robot's left and right laser radar detection data, and guide the robot to turn to a place where obstacles are not dense; the probability of turning left and right is: Where: dist left Represents the sum of the distances of the obstacles on the left detected by the LiDAR; dist right Represents the sum of the distances to the obstacles on the right detected by the LiDAR; L prob is the probability of turning left; R prob is the probability of turning right; C is a constant that can be determined according to the laser radar ranging range; the final escape linear velocity v and escape angular velocity ω are: Where: random(-R prob ,L prob ) indicates from -R prob To L prob A random number in the range.

4. The TD3 map-free navigation method based on dynamic window guidance according to claim 1, characterized in that: The improved TD3 network model inputs the lidar data, target point position and the action to be evaluated into the Critic network to obtain the Q value of the action, and inputs the lidar data and target point position into the Actor network to obtain the linear velocity and angular velocity.

5. The TD3 map-free navigation method based on dynamic window guidance according to claim 4 is characterized in that: The Actor network with the LSTM module concatenates the lidar data and the target point position together with a size of 1×22, inputs it into a fully connected layer of size 22×600, inputs the output features into a 600×600 LSTM layer, and then inputs the output features into a 600×2 fully connected layer. The final action is output through the Tanh activation function.

6. The TD3 map-free navigation method based on dynamic window guidance according to claim 4, characterized in that: The Critic network with the LSTM module concatenates the lidar data and the target point position together and inputs them into a 22×600 fully connected layer, inputs the output features into a 600×600 LSTM layer, and then outputs the intermediate feature S1; the action to be evaluated is input into a 2×600 fully connected layer, which outputs the intermediate feature S2; features S1 and S2 are combined, and the intermediate feature S is generated through the ReLU activation function, and S is input into a 600×1 fully connected layer, which outputs the Q value.

7. The TD3 map-free navigation method based on dynamic window guidance according to claim 1, characterized in that: Design an action selector based on the ε-greedy strategy, using a large number of dwa As an agent, it performs actions to help the agent quickly learn to avoid obstacles and move toward the target point. At the end of the training, the Action network is used to obtain a net , the mathematical expression of the action selector based on the ε-greedy strategy is as follows: In the formula, a max(Q) Represents the output action of the action selector; a net represents the action directly output by the TD3 network; a represents the action performed by the final robot; where the probability of 1-ε is a max(Q) , the probability of ε is a net ; In the early stage of training, the initial value of ε is small, and it gradually increases to 1 as the training progresses.

Images