A td3 map-free navigation method based on dynamic window method guidance

Abstract

The application discloses a TD3 map-free navigation method based on a dynamic window method, and particularly relates to the technical field of path planning, and acquires current state information and a target position of a robot; after the current state information and the target position information are preprocessed, the information is input into an improved TD3 network to obtain TD3 output actions, and improved DWA output actions are obtained by combining an improved evaluation function and an escape mechanism; the two obtained actions are input into an action selector to output an optimal action; the robot executes the optimal action to complete a map-free navigation task. The application improves the TD3 network structure, and applies DWA (dynamic window method) and LSTM (long short-term memory network) to map-free navigation. The network structure can directly output robot actions, namely linear velocity and angular velocity, according to the input target point relative position and the robot state, and realizes end-to-end map-free navigation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of path planning, in particular to a robot map-free navigation method based on the combination of DWA algorithm and TD3 algorithm. BACKGROUND

[0002] In recent years, with the rapid development of intelligent technology and autonomous driving technology, path planning methods based on deep reinforcement learning have attracted widespread attention from scholars. As a core part of robot intelligence technology, most path planning algorithms require prior maps to be obtained in advance. However, it is extremely difficult to obtain prior maps in special scenarios such as underground exploration and post-disaster rescue. Therefore, map-free navigation algorithms have gradually become a key problem for the further development of mobile robots. Current map-free navigation solutions can be divided into classical methods and deep reinforcement learning-based methods. In order to achieve a certain degree of autonomy, mobile robots need to find an optimal collision-free path from the starting position to the target endpoint according to the environment and task constraints. Although traditional algorithms can ensure that no collision occurs in most cases, the path of the robot is not optimal, resulting in poor navigation effect. It is necessary to traverse the entire map to find the optimal path, which consumes a lot of time and computational resources.

[0003] Traditional navigation algorithms have some shortcomings in map-free scenarios. The environment in which current robots operate is becoming increasingly complex and variable. Previous methods cannot well solve these problems. Deep reinforcement learning-based navigation methods have been widely applied in robot path planning tasks. Deep reinforcement learning combines the unique advantages of deep learning and reinforcement learning, enabling it to have both the decision-making learning ability of reinforcement learning and the feature extraction ability of deep learning, and can meet the needs of map-free navigation tasks. Compared with previous reinforcement learning methods, the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm improves on the basis of previous methods. Its main improvement is based on the DDPG (Deep Deterministic Policy Gradient) algorithm. The TD3 algorithm improves the idea of DDQN (Double Deep Q-Network). To solve the overestimation problem, the TD3 network architecture uses an Actor-Critic framework, which contains four Critic networks and two Actor networks. The Critic network updates more frequently than the Actor, which makes the training process more stable and has a great advantage in navigation tasks. However, the navigation method based on TD3 relies on a large amount of data for training, which requires sufficient high-quality data to train a satisfactory model. In the early stages of training, the network model has not learned enough, resulting in poor navigation effect. The algorithm cannot collect high-quality training data, and the learning efficiency is low. SUMMARY

[0004] The application is to solve the problems in the background art, improve the TD3 network structure, and fuse DWA (dynamic window method) and LSTM (long short-term memory network) for map-free navigation. The network structure can directly output the robot action, i.e. linear velocity and angular velocity, according to the input target point relative position and robot state, and realize end-to-end map-free navigation.

[0005] In order to achieve the above purpose, the application provides the following technical scheme: a TD3 map-free navigation method based on dynamic window method guidance, obtaining the current state information and target position of the robot; after preprocessing the current state information and target position information, inputting the preprocessed current state information and target position information into the improved TD3 network model to obtain the TD3 output action, and combining the improved evaluation function and the escape mechanism to obtain the improved DWA output action, inputting the two obtained actions into the action selector to output the optimal action; the robot executes the optimal action to complete the map-free navigation task.

[0006] Among them, the improved TD3 network model introduces an LSTM module in the Actor and Critic network, and the training steps of the improved TD3 network model include:

[0007] (a) Gazebo simulation environment starts, ROS node starts, robot model loads, obtains the current state and target point position of the robot, and performs related preprocessing;

[0008] (b) input the preprocessed action into the Actor network to obtain the action a net , and obtain a dwa through the output of the DWA planner, input the two actions into the action selector, and output the final selected optimal action a max(Q) ;

[0009] (c) the robot executes the optimal action a max(Q) in the Gazebo environment, and returns the new state S t+1 , the reward value r and the flag whether the current round is completed, stores the state value, action value, reward value and state value at time t+1 (s, a, r, s t+1 ) at the current time into the experience replay pool;

[0010] (d) until the amount of data in the experience replay pool is sufficient, randomly sample a sample batch with a size of batch_size from the experience replay pool; update the Actor network through the (s, a, r, s t+1 ) sample batch obtained by sampling, and in the TD3 algorithm training process, the network obtains (s, a, r, s t+1), the action at time t+1 is obtained by the target Actor network inference, and the Q value is calculated by the two Critic networks, the target value y is calculated, the loss function of the Critic network is calculated using the target Q value y, and the two Q networks are updated respectively;

[0011] (e) The performance of the current policy is evaluated by the Critic network, the average negative value of the Q value output by the Critic network is calculated to perform a gradient ascent operation, the parameter gradient of the Actor network is calculated by back propagation, and the Actor network parameters are updated; The update of the Actor network is delayed, which reduces the frequency of policy update and prevents the policy from falling into a suboptimal solution too early. After each Q network and policy network update, the parameters of the target network are updated using soft update;

[0012] (f) Steps (a)-(e) are repeatedly performed until the policy converges or a preset number of training times is reached.

[0013] Preferably, the improved evaluation function is composed of heading, dist, and vel, which represent the azimuth angle, obstacle distance, and speed size, respectively; When the robot falls into a local optimum, the heading term in the speed evaluation function will be ignored, and the dist term will be used as the main speed evaluation index to encourage the robot to escape from the local optimum.

[0014] Preferably, the escape mechanism is designed as follows:

[0015] A robot turning angular velocity range is designed according to the left and right laser radar detection data of the robot, which guides the robot to turn to the place where the obstacles are not dense; The size of the left and right turning probability is:

[0016]

[0017] In the formula: dist left represents the sum of the distances of the left obstacles detected by the laser radar; dist right represents the sum of the distances of the right obstacles detected by the laser radar; L prob is the left turning probability; R prob is the right turning probability; C is a constant which can be determined according to the ranging range of the laser radar; The final escape linear velocity v and the escape angular velocity ω are:

[0018]

[0019] In the formula: random(-R prob ,L prob ) represents a random number in the range from x to y.

[0020] The preferred and improved TD3 network model introduces LSTM modules into the Actor and Critic networks to enhance the network's ability to process continuous navigation information. This allows the Q-value of the action to be obtained by inputting LiDAR data, target point position, and the action to be evaluated into the Critic network, and the linear velocity and angular velocity to be obtained by inputting LiDAR data and target point position into the Actor network.

[0021] Preferably, the Actor network with LSTM module concatenates the LiDAR data and target point location together, with a size of 1×22. It is then input into a fully connected layer of size 22×600, and the output features are input into a 600×600 LSTM layer. The output features are then input into a 600×2 fully connected layer, and the final action is output through the Tanh activation function.

[0022] Preferably, the Critic network incorporating an LSTM module concatenates the LiDAR data and target point location together and inputs them into a 22×600 fully connected layer. The output features are then input into a 600×600 LSTM layer, which outputs intermediate features S1. The action to be evaluated is input into a 2×600 fully connected layer, which outputs intermediate features S2. Features S1 and S2 are combined and a ReLU activation function is used to generate intermediate features S. S is then input into a 600×1 fully connected layer, which outputs the Q-value.

[0023] Preferably, an ε-greedy action selector is designed, which extensively uses a algorithm in the early stages of training. dwa As an intelligent agent, it executes actions to help the agent quickly learn to avoid obstacles and move towards the target point; in the later stages of training, the Action network is used more often to obtain a net This allows the robot to fully explore optimal actions, thus improving the upper limit of the agent's navigation performance. The mathematical expression for the action selector based on the ε-greedy policy is as follows:

[0024]

[0025] In the formula, a max(Q) Indicates the output action of the action selector; a net Let represent the action directly output by the TD3 network; 'a' represents the final action performed by the robot, where the probability of 1-ε is 'a'. max(Q) The probability of ε is a net In the early stages of training, the initial value of ε is small, gradually increasing to 1 as training progresses; this ensures that the agent learns more of the action selector's output actions in the initial phase, improving training efficiency. In the later stages of training, the agent is encouraged to explore more, further improving model performance.

[0026] The present invention has the following advantages:

[0027] 1. This invention improves the evaluation function by employing an escape mechanism to prioritize escaping local optima when the robot gets trapped in a local optimum. It also designs a method to determine the robot's turning angular velocity range based on data from the robot's left and right lidar sensors, guiding the robot to turn towards areas with fewer obstacles. This addresses the problem that in map-free environments, the original DWA algorithm often gets trapped in local optima, where the robot's highest evaluation score action might be standing still. While this ensures the robot avoids collisions, it also prevents it from reaching the target location.

[0028] 2. This invention obtains the robot's current state information and target position information; obtains the output linear velocity and angular velocity through the DWA planner; adds an LSTM module to the network structure of the TD3 algorithm to improve the network's ability to understand continuous navigation information and obtain the improved angular velocity and linear velocity output by the TD3 algorithm; and uses an action selector to select the actions output by the DWA planner and the TD3 algorithm to improve the quality of the actions. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the kinematic model of the two-wheel differential chassis robot provided by the present invention;

[0030] Figure 2 This is a schematic diagram of the Actor-Critic network structure provided by the present invention;

[0031] Figure 3 A schematic diagram of the overall architecture of the DWA-guided TD3 algorithm provided by this invention. Detailed Implementation

[0032] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] In a first aspect, the present invention provides an improved DWA (Diagram-less Navigation) method;

[0034] Improvements to DWA's mapless navigation methods include:

[0035] Obtain the robot's current state and target location information;

[0036] Calculate the dynamic window, and based on the robot's current state and maximum acceleration, calculate the velocity range over a future period of time;

[0037] The optimal speed is evaluated by sampling the speed within the window and then using the original evaluation function to evaluate the sampled speeds and select the optimal speed.

[0038] To escape local optima, this invention improves the evaluation function. When trapped in a local optimum, an escape mechanism is employed to prioritize escaping the local optimum. In this case, the evaluation function disregards the heading term and uses the dist term as the primary basis for evaluating speed, prompting the robot to prioritize escaping local optima. Due to mechanical limitations, the robot's back is often obstructed, resulting in blind spots in the observation area. Sometimes, rotation is necessary to find a better solution. If the speed obtained from the improved evaluation function is still insufficient to escape the local optimum, a stationary rotation scheme is used to guide the robot to acquire more environmental information. Therefore, this invention designs a method to determine the robot's turning angular velocity range based on data from the robot's left and right lidar sensors, guiding the robot to rotate towards areas with fewer obstacles.

[0039] Secondly, this invention provides an improved TD3 method for DWA-guided programming;

[0040] Improved TD3 methods guided by DWA include:

[0041] Obtain the robot's state, including the current state information and the target position information;

[0042] Obtain the output linear velocity and angular velocity through the DWA planner;

[0043] Adding an LSTM module to the network structure of the TD3 algorithm improves the network's ability to understand continuous navigation information and obtains the angular velocity and linear velocity output by the improved TD3 algorithm.

[0044] An action selector will be used to select actions from the DWA planner and the TD3 algorithm to improve action quality.

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] As an embodiment of the present invention, the overall architecture is as follows: Figure 3 As shown, a mapless navigation method for TD3 based on dynamic windowing includes the following steps:

[0047] Step 1: The data preprocessing stage includes the following steps:

[0048] Step 1.1: Set the basic parameters such as the path of the trained model and the batch size;

[0049] Step 1.2: Process LiDAR data;

[0050] Step 1.3: Obtain the robot's current state information and target location information;

[0051] Step 2: Obtain the DWA output action:

[0052] 2.1: Establishing the kinematic model of the two-wheeled differential robot, as follows Figure 1 Its kinematic characteristics are fully considered. Its kinematic model can be expressed as:

[0053]

[0054] In the formula: x, y represent the robot's coordinate position; ω represents the robot's angular velocity; v1, v2 represent the linear velocities of the robot's left and right wheels, respectively; v is the robot's center linear velocity, with a magnitude of (v1+v2) / 2; θ represents the angle between the line connecting the centers of the two wheels and the x-axis; t represents the current time; and Δt represents the change in time.

[0055] 2.2: Establish an environmental perception model to acquire real-time information about surrounding obstacles. Divide the LiDAR point cloud data into 18 intervals according to angle, and take the nearest laser beam in each interval as the state value of that interval, forming a state vector of length 18;

[0056] 2.3: Calculating the velocity sampling interval for the DWA algorithm. Before velocity sampling, the sampling interval needs to be calculated. Due to the inherent characteristics of the robot and environmental constraints, the velocity space is limited to a certain range, thus having a maximum and a minimum value. The selection of the velocity space needs to consider its limiting velocity constraint V. m :

[0057] V m ={(v,ω)|v∈[v min ,v max ],ω∈[ω min ,ω max ]};

[0058] In the formula: v min ,v max Represents the minimum and maximum linear velocities of the robot; ω min ,ω max This represents the minimum and maximum angular velocities of the robot;

[0059] In practical engineering, the acceleration and deceleration of a robot are limited by its dynamic performance. Therefore, the robot's velocity space is also constrained by the strength of its dynamics, with acceleration constrained by V. d The mathematical expression is as follows:

[0060]

[0061] In the formula: a v Indicates the maximum linear acceleration; a ω Indicates the maximum angular acceleration; v c ,ω c These represent the current linear velocity and the current angular velocity, respectively.

[0062] The robot uses lidar to determine the distance to obstacles and must maintain a sufficient distance to decelerate before colliding with them. Insufficient distance will prevent the robot from slowing down, potentially leading to a collision. Therefore, this invention constrains the robot's speed before a collision; this speed should be proportional to the distance to the obstacle, with the obstacle constraint V... a The mathematical expression is as follows:

[0063]

[0064] In the formula: dist(v,ω) represents the minimum distance between the predicted trajectory and the obstacle at the current speed. Combining the above three speed constraints, the sampling speed space V in the final DWA algorithm can be obtained as:

[0065] V = V m ∩V d ∩V a ;

[0066] 2.4 Velocity Evaluation: An evaluation function is used to assess the sampled velocities to determine the optimal velocity under this condition. Since DWA, as a local path planning algorithm, is prone to getting trapped in local optima without a prior map, this invention designs an escape mechanism based on the original evaluation function to improve the local optima problem. The principle of the evaluation function is to sample multiple sets of velocities in a velocity space that meets various constraints, simulate trajectories over a certain time period, evaluate these trajectories using the evaluation function, and select a relatively optimal trajectory and velocity. The mathematical expression of the evaluation function is as follows:

[0067] G(v,ω)=σ(α·heading(v,ω)+β·dist(v,ω)+γ·vel(v,ω));

[0068] In the formula: heading(v,ω) represents the angle error between the trajectory endpoint direction and the line connecting the target point at the current sampling velocity; vel(v,ω) usually refers to the magnitude of the linear velocity; σ, α, β, γ are constants;

[0069] In map-less environments, navigation using the original DWA algorithm often gets stuck in local optima. In this situation, the robot's highest-scoring action might be remaining stationary. While this ensures the robot avoids collisions, it also prevents it from reaching the target location. To address this issue, this invention improves the evaluation function by employing an escape mechanism to prioritize escaping local optima when trapped. In this improved function, the heading term is disregarded, and the dist term is used as the primary criterion for evaluating speed, thus encouraging the robot to prioritize escaping local optima. Therefore, when trapped in a local optimum, the improved dynamic window method evaluation function becomes:

[0070] G(v,ω)=σ′(β′·dist(v,ω)+γ′·vel(v,ω));

[0071] In the formula: σ′, β′, γ′ are all constants;

[0072] Due to mechanical limitations, the robot's back is often obstructed, creating blind spots in the observation area. Sometimes, it's necessary to rotate the robot to find a better solution. Therefore, this invention designs a method to determine the robot's turning angular velocity range based on left and right-side lidar detection data, guiding the robot to turn towards areas with fewer obstacles. The probabilities of turning left and right are:

[0073]

[0074] In the formula: dist left This represents the sum of distances from which the lidar detected obstacles on the left; dist right L represents the sum of distances from which the lidar detected obstacles on the right; prob R represents the probability of turning left. prob The probability of turning right is given by [variable name]. C is a constant that can be determined based on the range of the lidar; the final escape linear velocity v and escape angular velocity ω are:

[0075]

[0076] In the formula: random(-R) prob ,L prob ) indicates from -R prob To L prob A random number within a given range;

[0077] Step 3: Obtain the output action of the TD3 algorithm:

[0078] Step 3.1: Construct the TD3 network. Use PyTorch to create two Actor networks and four Critic networks. The two Actor networks and four Critic networks have identical structures. Introduce LSTM modules into the Actor and Critic networks to improve the feature extraction capability of continuous navigation information. By fully referencing preceding data, the output trajectory is smoother and better meets the robot's safe movement requirements. The Actor and Critic network structures are as follows: Figure 2 As shown;

[0079] Step 3.2: Set the robot's current observation state s t and the current action a t When input into the constructed TD3 model, the network output is a. TD3 ;

[0080] Step 4: Action selection based on the ε-greedy strategy:

[0081] A greedy action selector based on an ε-policy is proposed to guide the agent to execute more effective actions. The TD3 algorithm includes a Critic network, which is mainly used to evaluate the value of actions during training. During training, the decision network calculates the action 'a'. net , will a net The inputs are fed into the Critic network to obtain Q. net1 Q net2 And the smaller of the two values ​​is Min(Q) net1 Q net2 The input is given to the action selector; the action received by the DWA controller is a. dwa Similarly, a dwa The inputs are fed into the Critic network to obtain Q. dwa1 Q dwa2 And the smaller of the two values ​​is Min(Q) dwa1 Q dwa2 The input is fed into the action selector. The agent will select actions based on the Q-value according to a certain strategy, executing the action with the larger Q-value. max(Q) However, action selectors based on an ε-greedy strategy will heavily utilize a-type algorithms in the early stages of training. dwa As an intelligent agent, it executes actions to help the agent quickly learn to avoid obstacles and move towards the target point. In the later stages of training, the Action network is used more often to obtain the desired result. net This allows the robot to fully explore the optimal action, thus improving the upper limit of the agent's navigation performance. The mathematical expression for the final action 'a' obtained based on the ε-greedy policy action selector is as follows:

[0082]

[0083] In the formula: a max(Q) Indicates the output action of the action selector; a net ε represents the action directly output by the TD3 network; 'a' represents the final action performed by the robot; in the early stages of training, ε has a small initial value and gradually increases to 1 as training progresses. This ensures that the agent learns more actions from the action selector in the initial stage, improving training efficiency. In the later stages of training, the agent is encouraged to explore more, further improving model performance.

[0084] The specific steps for training the TD3 algorithm guided by the dynamic window method are as follows:

[0085] Step A1: Initialize the training environment, obstacles, robot start and end points, and start Rviz;

[0086] Step A2: Obtain the robot's current state, LiDAR data, and relative coordinates of the target point. Preprocess the LiDAR data through Step 2.2 to obtain a LiDAR data with a length of 18. Input this information into the Actor network. The network will output the linear velocity and angular velocity, and simultaneously obtain the DWA output action. After passing through the action selector, the final robot action will be output, and the action reward value and the next state will be saved.

[0087] Step A3: When the amount of data in the experience replay pool reaches a preset threshold, train the Actor network and Critic network by randomly sampling the stored experience data;

[0088] Repeat the above process to complete the training of the model.

[0089] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A mapless navigation method for TD3 based on dynamic window method, which obtains the robot's current state information and target position; After preprocessing the current state information and target location information, the feature is: The preprocessed current state information and target position information are input into an improved TD3 network model that incorporates a Long Short-Term Memory (LSTM) module to obtain the TD3 output action. Based on the dynamic window method DWA planner, and combined with an improved evaluation function and escape mechanism, the improved DWA output action is obtained. The obtained TD3 output action and the improved DWA output action are then input into a system based on... - The greedy strategy action selector outputs the optimal action; the robot executes the optimal action to complete the mapless navigation task; The improved TD3 network model introduces an LSTM module into the Actor and Critic networks. The training steps for the improved TD3 network model include: (a) Start the Gazebo simulation environment, start the ROS node, load the robot model, obtain the current state of the robot and the position of the target point, and perform relevant preprocessing; (b) Input the preprocessed action into the Actor network to obtain the action. And obtain it through the output of the DWA planner. Two actions are input into the action selector, which then outputs the optimal action. ; (c) The robot performs the optimal action in the Gazebo environment. and return to a new state. Reward Value And a flag indicating whether the current round is complete, along with the current state value, action value, reward value, and State value at time Stored in the experience revisit pool; (d) Until the amount of data in the experience replay pool is sufficient, randomly sample a batch of samples of size batch_size from the experience replay pool; obtain the sampled data... The Actor network is updated using sample batches. During the TD3 algorithm training process, the network obtains data from the experience pool. Obtained through target Actor network reasoning The action is performed at each moment, and the Q-value is calculated by two Critic networks to calculate the target value. Use target Q value Calculate the loss function of the Critic network and update the two Q networks respectively; (e) Evaluate the performance of the current policy through the Critic network, calculate the average negative value of the Q-value of the Critic network output to perform gradient ascent operation, backpropagate to calculate the parameter gradient of the Actor network and update the Actor network parameters; after each Q-network and policy network update, use soft update to update the parameters of the target network. (f) Repeat steps (a)-(e) until the policy converges or the preset number of training iterations is reached; Wherein, the improved evaluation function is composed of The three components represent the azimuth angle, obstacle distance, and velocity magnitude, respectively; when the robot gets stuck in a local optimum, the velocity evaluation function will... The item will be ignored and... The term serves as the primary speed evaluation indicator, prompting the robot to prioritize escaping local optima. The escape mechanism is designed as follows: Design a method to determine the robot's turning angular velocity range based on left and right LiDAR detection data, guiding the robot to turn towards areas with fewer obstacles; where the probabilities of turning left and right are: ； In the formula: This represents the sum of the distances to the obstacles detected by the lidar on the left. This represents the sum of distances from which the lidar detected obstacles on the right. The probability of turning left; The probability of turning right; The constant value can be determined based on the range of the lidar; the final escape linear velocity and escape angular velocity for: ； In the formula: Indicates from arrive A random number within a given range; The improved TD3 network model inputs lidar data, target point position, and the action to be evaluated into the Critic network to obtain the Q value of the action, and inputs lidar data and target point position into the Actor network to obtain the linear velocity and angular velocity. The Actor network, which incorporates an LSTM module, stitches together LiDAR data and target point locations, with a size of 1. 22, input it into a size of 22 A 600-pin fully connected layer inputs the output features into a 600-pin circuit. A 600 LSTM layer is used, and then the output features are input into another 600 LSTM layer. A fully connected layer of size 2, and the output of the Tanh activation function is the final action; The Critic network, which incorporates an LSTM module, concatenates LiDAR data and target location information and inputs them into a 22-bit microcontroller. A 600-pin fully connected layer inputs the output features into a 600-pin circuit. A 600-layer LSTM is then used to output intermediate features. The action to be evaluated is input into a 2 A 600-pin fully connected layer that outputs intermediate features. ; Features and Combined, intermediate features are generated using the ReLU activation function. ,Will Enter into a 600 A fully connected layer with a value of 1, the output of which is the Q-value; Design a based - Greedy action selector strategy, widely used in the early stages of training. As an intelligent agent, it executes actions to help the agent quickly learn to avoid obstacles and move towards the target point; at the end of training, the Action Network is used to obtain... ,based on The mathematical expression for the greedy strategy action selector is as follows: ； In the formula, Indicates the output action of the action selector; This indicates the action directly output by the TD3 network; This indicates the final action performed by the robot; where The probability is , The probability is In the early stages of training, The initial value is small, and it gradually increases to 1 as training progresses.

Images