Mobile robot path planning method based on SAC algorithm

Abstract

A mobile robot path planning method based on an SAC algorithm belongs to the technical field of deep reinforcement learning and mobile robot navigation, integrates the thought of hierarchical reinforcement learning skill combination into the SAC algorithm, and uses a time sequence abstraction method to integrate low-level fine-grained actions, so as to improve the precision of the SAC algorithm. A higher-level coarse-grained action with definite semantics can be learned; in reinforcement learning training, a distributed soft strategy iteration method is used, maximum entropy reinforcement learning is expanded into a distributed learning version, and variance and overestimation of a Q value are reasonably controlled through a learning state-action return distribution function, so that the overall strategy quality of robot path planning is improved. According to the method, training and testing are carried out in a simulation environment Image-Env, and the performance of the algorithm is verified on Turtlebot2. The method can efficiently avoid obstacles, is short in path, and has a good obstacle avoidance effect in a complex dynamic environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep reinforcement learning and mobile robot navigation, and specifically relates to a mobile robot path planning method based on a SAC algorithm. Background Art

[0002] With the rapid development of science and technology and the deepening of research in the field of artificial intelligence, indoor robots, especially service robots and cleaning robots, play an increasingly important role in daily life. These robots can perform tasks such as guidance, cleaning, and logistics in indoor environments such as hospitals, schools, shopping malls, and factories, greatly improving human work efficiency and quality of life. However, in order for indoor mobile robots to complete these tasks efficiently and safely, efficient path planning algorithms are essential.

[0003] Robot path planning is a key area in robot navigation technology. During the navigation process, it is responsible for generating the specific route of the robot from its current position to the target position, while considering avoiding obstacles and satisfying certain specific constraints (such as the shortest path, minimum path finding cost, etc.).

[0004] In the field of robot path planning, path planning algorithms are roughly divided into path planning with maps and path planning without maps, depending on whether they rely on prior environmental maps. Traditional path planning algorithms require the robot to build a global map before path planning can be performed. Modeling is difficult in dynamic environments, and the algorithm converges slowly. Mapless path planning does not rely on prior environmental maps. It collects environmental information in real time through robot sensors, interacts with the environment and generates path planning strategies to achieve end-to-end path planning. Mapless path planning requires higher perception and processing capabilities of the robot. The deep reinforcement learning algorithm happens to have the dynamic perception capabilities of deep learning and the excellent decision-making performance of reinforcement learning. It can directly learn the robot's control strategy from high-dimensional sensor inputs, and has the advantages of simple deployment, end-to-end learning, and strong environmental adaptability.

[0005] Deep reinforcement learning combines deep learning's ability to perceive complex information with reinforcement learning's decision-making ability in unknown environments to complete the interaction process between the intelligent agent and the environment. Initially, DeepMind pioneered the publication of DQN, using convolutional neural networks to simulate the learning and updating process of Q-learning, and built an end-to-end framework from perception to decision-making. Improvements to the DQN model are often achieved by adding other functional modules to the network. For example, Hausknecht et al. proposed DRQN ​​by using recurrent neural networks to learn historical information about environmental states. Subsequently, the Actor-Critic structure was introduced into the policy gradient method, which can effectively evaluate the impact of each action on the final reward in sequential decision problems, and solve the problem that reinforcement learning can only update the network at the end of the interaction. Many subsequent excellent algorithms are based on the AC structure, such as the SAC algorithm.

[0006] The SAC algorithm is a deep reinforcement learning method. Before the SAC algorithm was proposed, there were algorithms such as DDPG, PPO, and A3C in the field of deep reinforcement learning. These algorithms often face problems such as complex input sample data, difficulty in network convergence in high-dimensional scenarios, and difficulty in balancing exploration and utilization in robot path planning, which makes model training encounter bottlenecks. The SAC algorithm proposes a deep reinforcement learning algorithm based on maximum entropy, heterogeneous strategies, and Actor-Critic framework. Maximum entropy promotes better exploration by maximizing rewards, making the strategy as random as possible while ensuring the completion of the task. Heterogeneous strategies mean that in the reinforcement learning process, the strategy for generating robot behavior can be different from the target strategy for evaluating robot behavior. Data in the historical experience pool can be used to improve sample efficiency and enhance the stability of the algorithm.

[0007] With the rapid development of mobile robots, the complexity of the environment faced by robots has gradually increased. It is inevitable that other mobile robots and pedestrians will appear in the operating environment. Robots need to interact with other mobile robots and pedestrians and reach the destination safely, which puts higher requirements on the path planning and collision avoidance of robots. Intelligent mobile robots need to have a certain ability to adapt to unknown environments and be able to navigate safely in dynamic environments. Summary of the invention

[0008] In order to overcome the shortcomings of the existing technology, to improve the path planning decision-making ability of the robot, and to accelerate the convergence speed of the reinforcement learning algorithm, the present invention provides a mobile robot path planning method based on the SAC algorithm, improves the SAC algorithm, uses a distributed training method, accelerates the convergence speed of the algorithm, and optimizes the iterative process of the SAC algorithm. The distributed SAC method helps to identify and utilize the potential uncertainty of the robot's actions, thereby improving the quality of the overall strategy and the robot's path planning decision-making level; this method improves the efficiency and safety of the mobile robot in path planning.

[0009] The technical solution adopted by the present invention to solve its technical problem is:

[0010] A mobile robot path planning method based on the SAC algorithm comprises the following steps:

[0011] S1: The robot's path planning problem is modeled as a partially observable Markov decision process. The seven-tuple describing the process includes observation space, state space, action space, state transition probability, reward function, observation probability and discount factor.

[0012] S2: Use the distributed SAC algorithm to solve the partially observable Markov decision process and make decisions on the robot's navigation strategy;

[0013] S3: The current state and observation of the robot obtained by the partially observable Markov tuple, and the state-action value pair of the robot is estimated using a deep learning network;

[0014] S4: In the simulation environment, the robot in each sub-process interacts with the surrounding environment for learning. After the sub-process network is updated, it interacts with the main process and writes the interactive experience into the temporary experience buffer pool.

[0015] S5: Use a distributed soft policy iterative training method to extend maximum entropy reinforcement learning to a distributed learning version, learning the distribution function of state-action rewards to identify and exploit the potential uncertainty of moving obstacles;

[0016] S6: The update of the state-action reward distribution value function is divided into two parts: soft policy evaluation and soft policy improvement. After evaluating the state-action reward distribution, identify and select a new strategy that makes the robot's path planning effect better than the old strategy. Alternately perform soft policy evaluation and soft policy improvement to finally obtain the optimal strategy.

[0017] Furthermore, in step S1, the seven-tuple of partially observable Markov decision process is expressed as:

[0018] M=(S,A,P,R,Ω,O,Γ)

[0019] Where S = {s1,s2,…,s n} is the state space, which represents the set of all states of the robot in the driving space; A={a1,a2,…,a n} is the action space; P(s t+1 |s t ,a t ) is the state transition probability; R is the reward function; Ω(o|s t ,a t ) is the observation probability; γ is the discount coefficient; the reward function is set as:

[0020]

[0021] Where k represents the current time step, N represents the number of multi-step reward steps, T represents the total number of steps in the trajectory, and the proxy reward function r′, r t represents the reward at time t, γ represents the decay rate; where r t It is expressed as follows:

[0022]

[0023] The reward function includes: the penalty for collision with dynamic obstacles, the constraint on the distance between the robot and the moving obstacle, the penalty for the distance to the target point, and the time cost; where d t is the distance between the robot and the moving obstacle, when d t <0, it means a collision occurs and the robot is penalized; 0 <d t When <0.15, a penalty is imposed based on the degree of proximity. The closer the robot is to the moving obstacle, the greater the penalty. When the robot reaches the target position within the specified time, it is rewarded, and the time taken to reach the destination is negatively correlated with the reward. This constraint encourages the robot to reach the target point in the shortest time.

[0024] Furthermore, in step S2, in the SAC algorithm, the input is composed of a lidar map and a dynamic obstacle information map obtained by the spencer framework, and the action to be performed in the next step is determined by the output of the policy network in the SAC algorithm; wherein, the SAC algorithm incorporates the idea of ​​hierarchical skills to improve the efficiency of the robot's path exploration in a feedback sparse environment; wherein, the upper-level skill is composed of multiple actions, and the upper-level strategy determines which skill should be selected under a given state; its decision-making process considers the goal and current state of the overall task to select the most appropriate skill to execute; the internal strategy targets a specific skill and determines what sequence of actions should be taken during the execution of the skill; each skill has its own internal strategy, which focuses on completing the sub-goals defined by the skill.

[0025] Furthermore, in step S5, a distributed training method is used to accelerate the collection of reward experience. Each robot interacts with the environment according to its current state, and the network model subprocess corresponding to each robot learns the interaction data. After the main process interacts with multiple subprocesses, it writes the data into the experience buffer pool and updates the network parameters. represents the observation of the surrounding environment by the ith robot at time t, represents the action taken by the ith robot at time t, represents the speed of the ith robot at time t, Represents the reward obtained by the i-th robot at time t.

[0026] Preferably, the distributed policy iterative training method learns the state-action return distribution of the robot, updates the distribution value network based on samples collected from the buffer, uses the output of the value network to guide the update of the policy network, and reasonably controls the variance and overestimation of the Q value by learning the distribution function of the state-action return.

[0027] In step S6, for soft policy evaluation, a policy is given Then the update of the soft state action reward distribution is described as:

[0028]

[0029] Among them, r~R(·|s,a), s~p, a~π. R(·|s,a) represents the distribution of the reward function, p represents the state space probability distribution associated with the strategy π, and the equal sign indicates that the variables on both sides conform to the same probability distribution;

[0030] The goal of soft policy improvement is to identify a new policy π new , making its effect exceed the current strategy π old , using the maximum entropy enhancement objective function variance to update the strategy, assuming Then the soft state action reward distribution is expressed as:

[0031]

[0032] KL divergence is used to calculate the distance between the two policy distributions, thereby deriving a better soft-state action reward distribution value function. The above two formulas are calculated alternately and iteratively to finally obtain the optimal policy.

[0033] The technical concept of the present invention is: to integrate the idea of ​​hierarchical reinforcement learning skill combination into the SAC algorithm, and use the method of temporal abstraction to integrate lower-level fine-grained actions to learn higher-level coarse-grained actions with clear semantics; in reinforcement learning training, the distributed soft policy iteration method is used to expand the maximum entropy reinforcement learning into a distributed learning version, and the variance and overestimation of the Q value are reasonably controlled by learning the distribution function of the state-action return, thereby improving the overall strategy quality of the robot path planning. The present invention is trained and tested in the simulation environment Image-Env, and the performance of the algorithm is verified on Turtlebot2. The present invention can efficiently avoid obstacles, and the path is short, and it has a good obstacle avoidance effect in complex dynamic environments.

[0034] The beneficial effects of the present invention are mainly manifested in:

[0035] (1) The present invention introduces the idea of ​​hierarchical reinforcement learning into the SAC algorithm, dividing the decision-making process of robot path planning into multiple levels. The upper-level strategy focuses on long-term goals and decisions, while the lower-level skills execute specific robot control commands in a refined manner, optimizing the robot's actions and skills in a dynamic environment.

[0036] (2) The present invention uses a distributed policy iteration method to optimize the iteration process of the SAC algorithm, helping the robot to learn obstacle avoidance strategies more efficiently, improving the learning speed of the deep reinforcement learning method, reasonably controlling the variance and the overestimation of the Q value, and helping to identify and utilize the potential uncertainty of the robot's actions, thereby improving the strategy quality of the robot's path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is the strategy update flowchart of the hierarchical SAC algorithm.

[0038] Figure 2 This is the policy iteration training framework diagram of the distributed SAC algorithm. (Designated as an abstract figure)

[0039] Figure 3 This is a line chart comparing the rewards of the method of the present invention and other four reinforcement learning algorithms.

[0040] Figure 4 It is the robot planning trajectory diagram in the simulation environment. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings.

[0043] Reference Figure 1 to Figure 4 , a mobile robot path planning method based on SAC algorithm, comprising the following steps:

[0044] S1: The robot's path planning problem is modeled as a partially observable Markov decision process. The seven-tuple describing the process includes observation space, state space, action space, state transition probability, reward function, observation probability and discount factor.

[0045] In step S1, the seven-tuple of partially observable Markov decision process is expressed as:

[0046] M=(S,A,P,R,Ω,O,Γ)

[0047] Where S = {s1,s2,…,s n} is the state space, which represents the set of all states of the robot in the driving space; A={a1,a2,…,a n} is the action space; P(s t+1 |s t ,a t ) is the state transition probability; R is the reward function; Ω(o|s t ,a t ) is the observation probability; γ is the discount coefficient; the reward function is set as:

[0048]

[0049] Where k represents the current time step, N represents the number of multi-step reward steps, T represents the total number of steps in the trajectory, and the proxy reward function r′, r t represents the reward at time t, γ represents the decay rate; where r t It is expressed as follows:

[0050]

[0051] The reward function includes: the penalty for collision with dynamic obstacles, the constraint on the distance between the robot and the moving obstacle, the penalty for the distance to the target point, and the time cost; where d t is the distance between the robot and the moving obstacle, when d t <0, it means a collision occurs and the robot is penalized; 0 <d t When <0.15, a penalty is imposed based on the degree of proximity. The closer the robot is to the moving obstacle, the greater the penalty. When the robot reaches the target position within the specified time, it is rewarded, and the time taken to reach the destination is negatively correlated with the reward. This constraint encourages the robot to reach the target point in the shortest time.

[0052] S2: Use the distributed SAC algorithm to solve the partially observable Markov decision process and make decisions on the robot's navigation strategy;

[0053] In step S2, in the SAC algorithm, the input is composed of a lidar map and a dynamic obstacle information map obtained by a spencer framework, and the action to be performed next is determined by the output of a policy network in the SAC algorithm; wherein, the SAC algorithm incorporates the idea of ​​hierarchical skills to improve the efficiency of the robot's path exploration in a feedback sparse environment; wherein, the upper-level skill is composed of multiple actions, and the upper-level strategy determines which skill should be selected under a given state; its decision-making process considers the goal and current state of the overall task to select the most appropriate skill to execute; the internal strategy targets a specific skill and determines what sequence of actions should be taken during the execution of the skill; each skill has its own internal strategy, which focuses on completing the sub-goals defined by the skill.

[0054] S3: The current state and observation of the robot obtained by the partially observable Markov tuple, and the state-action value pair of the robot is estimated using a deep learning network;

[0055] S4: In the simulation environment, the robot in each sub-process interacts with the surrounding environment for learning. After the sub-process network is updated, it interacts with the main process and writes the interactive experience into the temporary experience buffer pool.

[0056] S5: Use a distributed soft policy iterative training method to extend maximum entropy reinforcement learning to a distributed learning version, learning the distribution function of state-action rewards to identify and exploit the potential uncertainty of moving obstacles;

[0057] In step S5, a distributed training method is used to accelerate the collection of reward experience. Each robot interacts with the environment according to its current state, and the network model subprocess corresponding to each robot learns the interaction data. After the main process interacts with multiple subprocesses, it writes the data into the experience buffer pool and updates the network parameters. represents the observation of the surrounding environment by the ith robot at time t, represents the action taken by the ith robot at time t, represents the speed of the ith robot at time t, Represents the reward obtained by the i-th robot at time t.

[0058] Preferably, the distributed policy iterative training method learns the state-action return distribution of the robot, updates the distribution value network based on samples collected from the buffer, uses the output of the value network to guide the update of the policy network, and reasonably controls the variance and overestimation of the Q value by learning the distribution function of the state-action return.

[0059] S6: The update of the state-action reward distribution value function is divided into two parts: soft policy evaluation and soft policy improvement. After evaluating the state-action reward distribution, identify and select a new policy that makes the robot's path planning effect better than the old policy. Soft policy evaluation and soft policy improvement are performed alternately to finally obtain the optimal policy.

[0060] In step S6, for soft policy evaluation, a policy is given Then the update of the soft state action reward distribution is described as:

[0061]

[0062] Among them, r~R(·|s,a), s~p, a~π. R(·|s,a) represents the distribution of the reward function, p represents the state space probability distribution associated with the strategy π, and the equal sign indicates that the variables on both sides conform to the same probability distribution;

[0063] The goal of soft policy improvement is to identify a new policy π new , making its effect exceed the current strategy π old , using the maximum entropy enhancement objective function variance to update the strategy, assuming Then the soft state action reward distribution is expressed as:

[0064]

[0065] KL divergence is used to calculate the distance between the two policy distributions, thereby deriving a better soft-state action reward distribution value function. The above two formulas are calculated alternately and iteratively to finally obtain the optimal policy.

[0066] After training the improved SAC algorithm in a simulation environment, it was deployed on the Turtlebot2 robot and verified in a real machine experiment. The results showed that the robot can safely plan paths in complex dynamic environments, and its arrival rate and collision rate are better than the current optimal deep reinforcement learning algorithm.

[0067] The mobile robot path planning method based on the SAC algorithm of this embodiment is trained in the Imageenv simulation environment proposed by Chen et al., which is customized by the OpenCV computer vision library. Compared with the commonly used simulation environments Gazebo and Stage, it has faster operation speed and lower latency.

[0068] In the Image env simulation environment, the robot is driven by differential drive (the same as in the real environment), and its motion update depends on the output of the policy network. The kinematic model update is shown in the following formula:

[0069]

[0070] Among them, v, ω, Δt are the robot's linear velocity, angular velocity and the interval time of each movement, and the interval time is 6Hz, the purpose is to narrow the gap between simulation and real experimental environment. (x, y, θ) is the current position of the robot, and (x′, y′, θ′) is the position of the robot at the next moment. The hierarchical SAC algorithm process is as follows Figure 1 As shown in Figure 1, the Actor and Critic modules are the same as those of the SAC algorithm. On this basis, the skill learning and updating of the upper-level strategy and internal strategy are added. The iterative training framework of the distributed SAC algorithm is shown in Figure 1. Figure 2 As shown, the algorithm models used in each sub-process and the overall process are Figure 1 The hierarchical SAC algorithm in . Multiple subprocesses are trained simultaneously, each robot interacts with the environment according to its current state, and the network model subprocess corresponding to each robot learns the interaction data After the main process interacts with multiple subprocesses, it writes the data into the experience buffer pool and updates the network parameters. Based on the samples collected from the buffer, the distribution value network is updated, and the output of the value network is used to guide the update of the policy network. The robot performs an action based on the observation and strategy in the current environment and plans a safe path.

[0071] like Figure 3 As shown, the comparison algorithms of the method of the present invention in the simulation environment are SFW-SAC, PPO-PSC, DDPG-DP and the ablation algorithm of the present invention. Among them, the reward return value obtained by the method of the present invention is the largest.

[0072] like Figure 4 As shown, it is the path planning trajectory of the method of the present invention in the simulation environment, where the room size is 8*8m, ​​multiple obstacles are arranged in the middle, and there are other mobile robots and pedestrians in the environment (in the simulation environment, the robot is set to a circle and the pedestrian is set to a square). The path planning trajectory of the method of the present invention in the real environment. Room size: width is 3m and length is 8m. It includes 2 mobile robots, one of which is an experimental robot Turtlebot2, using Kobuki chassis, equipped with Rplidar A3 laser radar and KinectV2 depth camera, and the host computer is a laptop equipped with i5-7300HQ CPU and GeForce1050 Ti GPU; the other Rikibot robot uses Raspberry Pi 4B CPU, equipped with Silan A1 laser radar and LeTMC-520 depth camera, Rikibot length is 17.8cm, width is 23.5cm. In the experiment, the experimental comparison algorithm is deployed on the Turtlebot2 robot as its path planning algorithm. The role of the Rikibot robot is a mobile obstacle, and it uses the ROS Navigation framework for path planning.

[0073] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A mobile robot path planning method based on SAC algorithm, characterized in that: The method comprises the following steps: S1: The robot's path planning problem is modeled as a partially observable Markov decision process. The seven-tuple describing the process includes observation space, state space, action space, state transition probability, reward function, observation probability and discount factor. S2: Use the distributed SAC algorithm to solve the partially observable Markov decision process and make decisions on the robot's navigation strategy; S3: The current state and observation of the robot obtained by the partially observable Markov tuple, and the state-action value pair of the robot is estimated using a deep learning network; S4: In the simulation environment, the robot in each sub-process interacts with the surrounding environment for learning. After the sub-process network is updated, it interacts with the main process and writes the interactive experience into the temporary experience buffer pool. S5: Use a distributed soft policy iterative training method to extend maximum entropy reinforcement learning to a distributed learning version, learning the distribution function of state-action rewards to identify and exploit the potential uncertainty of moving obstacles; S6: The update of the state-action reward distribution value function is divided into two parts: soft policy evaluation and soft policy improvement. After evaluating the state-action reward distribution, identify and select a new strategy that makes the robot's path planning effect better than the old strategy. Alternately perform soft policy evaluation and soft policy improvement to finally obtain the optimal strategy.

2. A mobile robot path planning method based on the SAC algorithm as claimed in claim 1, characterized in that: In step S1, the seven-tuple of partially observable Markov decision process is expressed as: M=(S,A,P,R,Ω,O,Γ) Where S = {s1,s2,…,s n } is the state space, which represents the set of all states of the robot in the driving space; A={a1,a2,…,a n } is the action space; P(s t+1 |s t ,a t ) is the state transition probability; R is the reward function; Ω(o|s t ,a t ) is the observation probability; γ is the discount coefficient; the reward function is set as: Where k represents the current time step, N represents the number of multi-step reward steps, T represents the total number of steps in the trajectory, and the proxy reward function r′, r t represents the reward at time t, γ represents the decay rate; where r t It is expressed as follows: The reward function includes: the penalty for collision with dynamic obstacles, the constraint on the distance between the robot and the moving obstacle, the penalty for the distance to the target point, and the time cost; where d t is the distance between the robot and the moving obstacle, when d t <0, it means a collision occurs and the robot is penalized; 0 <d t When <0.15, a penalty is imposed based on the degree of proximity. The closer the robot is to the moving obstacle, the greater the penalty. When the robot reaches the target position within the specified time, it is rewarded, and the time taken to reach the destination is negatively correlated with the reward. This constraint encourages the robot to reach the target point in the shortest time.

3. A mobile robot path planning method based on the SAC algorithm as described in claim 1 or 2, characterized in that: In step S2, in the SAC algorithm, the input is composed of a lidar map and a dynamic obstacle information map obtained by a spencer framework, and the action to be performed next is determined by the output of a policy network in the SAC algorithm; wherein, the SAC algorithm incorporates the idea of ​​hierarchical skills to improve the efficiency of the robot's path exploration in a feedback sparse environment; wherein, the upper-level skill is composed of multiple actions, and the upper-level strategy determines which skill should be selected under a given state; its decision-making process considers the goal and current state of the overall task to select the most appropriate skill to execute; the internal strategy targets a specific skill and determines what sequence of actions should be taken during the execution of the skill; each skill has its own internal strategy, which focuses on completing the sub-goals defined by the skill.

4. A mobile robot path planning method based on the SAC algorithm as described in claim 1 or 2, characterized in that: In step S5, a distributed training method is used to accelerate the collection of reward experience. Each robot interacts with the environment according to its current state, and the network model subprocess corresponding to each robot learns the interaction data. After the main process interacts with multiple subprocesses, it writes the data into the experience buffer pool and updates the network parameters. represents the observation of the surrounding environment by the ith robot at time t, represents the action taken by the ith robot at time t, represents the speed of the ith robot at time t, Represents the reward obtained by the i-th robot at time t.

5. A mobile robot path planning method based on the SAC algorithm as claimed in claim 4, characterized in that: The distributed policy iterative training method learns the state-action return distribution of the robot, updates the distribution value network according to the samples collected from the buffer, uses the output of the value network to guide the update of the policy network, and reasonably controls the variance and overestimation of the Q value by learning the distribution function of the state-action return.

6. A mobile robot path planning method based on the SAC algorithm as described in claim 1 or 2, characterized in that: In step S6, for soft policy evaluation, a policy is given Then the update of the soft state action reward distribution is described as: Among them, r~R(·|s,a), s~p, a~π, R(·|s,a) represents the distribution of the reward function, p represents the state space probability distribution associated with the strategy π, and the equal sign indicates that the variables on both sides conform to the same probability distribution; The goal of soft policy improvement is to identify a new policy π new , making its effect exceed the current strategy π old , using the maximum entropy enhancement objective function variance to update the strategy, assuming Then the soft state action reward distribution is expressed as: KL divergence is used to calculate the distance between the two policy distributions, thereby deriving a better soft-state action reward distribution value function. The above two formulas are calculated alternately and iteratively to finally obtain the optimal policy.

Images