A multi-objective path planning method based on improved SAC algorithm

Abstract

The present application belongs to the field of reinforcement learning, and particularly relates to a multi-objective path planning method based on improved SAC algorithm. The present application intends to collect sufficient path experience of robots reaching each shelf position, read offline expert experience before actual distribution to perform supervised learning assistance, and improve distribution efficiency; based on SumTree priority experience playback, improve the adoption rate of effective path sample experience; based on multi-step TD-error calculation reward mechanism, comprehensively consider subsequent multi-step rewards. The present application combines robot navigation and SAC algorithm in reinforcement learning, gets rid of the limitation of traditional path planning algorithm on model, improves the learning speed of robot and the utilization efficiency of experience sample, and solves the problems that the optimal path selection effect of robot in goods distribution link for one-time distribution of multiple goods is poor, it is easy to fall into local optimum, cannot reasonably plan path and accurately avoid environmental obstacles.

Description

Technical Field

[0001] This invention belongs to the field of reinforcement learning, specifically relating to a multi-objective path planning method based on an improved SAC algorithm. Background Technology

[0002] In recent years, due to manpower shortages, the demand for mobile robots in performing tedious and repetitive tasks has been rapidly increasing. Many mobile service robots are being used in warehouses to reduce operating costs and expenses. Autonomous navigation of mobile robots for picking and placing items on shelves within warehouses saves time and money for warehousing operations. To enable autonomous navigation within warehouses, mobile robots require path planning algorithms. To help robots adapt more quickly to large indoor environments like warehouses, we combine robot navigation with the SAC algorithm from reinforcement learning. This breaks free from the limitations of traditional path planning algorithms on the model, improving the robot's learning speed and efficiency in utilizing experience samples.

[0003] Although the traditional SAC algorithm has achieved many results and progress, there are still problems that need further research and solutions: (1) When sampling in the experience pool, the traditional SAC algorithm uses random uniform sampling, which often leads to the neglect of good experience data, resulting in slow network parameter updates and thus reduced algorithm convergence speed. (2) The priority of experience is defined by TD error. In terms of the priority of updating samples, the traditional SAC algorithm uses single-step TD error to calculate the priority. Such an experience replay algorithm ignores the importance of future gaps and is not conducive to improving stability under different training number limitations. (3) From a safety perspective, allowing an agent that has not yet learned to interact with the environment may lead to danger or cause huge losses. In path planning tasks, robots need to find a safe and efficient path in unknown or dynamic environments. Summary of the Invention

[0004] To address the problems existing in the aforementioned background technology, this invention provides a multi-objective path planning method based on an improved SAC algorithm. This method aims to collect sufficient path experience of robots reaching each shelf location, and improve delivery efficiency by reading offline expert experience before actual delivery through supervised learning assistance; it also improves the adoption rate of effective path sample experience based on SumTree's priority experience replay; and it comprehensively considers subsequent multi-step rewards based on a multi-step TD-error calculation reward mechanism.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0006] A multi-objective path planning method based on an improved SAC algorithm includes the following steps:

[0007] Step 1: Build a simulation environment and design the robot's initial position and the positions of multiple target points;

[0008] Step 2, initialize network input parameters;

[0009] Step 3: Design the path planning reward and penalty function;

[0010] Step 4: Pre-train using offline expert experience based on the current simulation environment;

[0011] Step 5: Improve the traditional SAC algorithm based on the priority experience replay mechanism of SumTree;

[0012] Step 6, optimize single-step TD-error;

[0013] Step 7, update the policy network Actor;

[0014] Step 8, update Q network Q Critic;

[0015] Step 9, update the value network V Critic;

[0016] Step 10: Update the weights, update the coefficient alpha of the entropy regularization term, update the target network, and plan the optimal path.

[0017] Furthermore, the specific process of step 1 is as follows:

[0018] Step 1.1: Build a simulation environment in Gazebo that is similar to a real warehouse environment for training and testing;

[0019] Step 1.2: Set the robot's initial position and target position;

[0020] Step 1.3: Use the robot's lidar sensor to obtain its distance information in the surrounding environment and interact with the surrounding environment. Design topics to receive radar information and use the radar information as algorithm input.

[0021] Furthermore, the specific process of step 2 is as follows:

[0022] Step 2.1: Set the sampling dimension of the robot's LiDAR sensor and the robot's overall state dimension;

[0023] Step 2.2, goal refers to the coordinates of the target position, goal_distance represents the distance between the robot's current position and the target position, heading represents the robot's orientation relative to the target point, and scan_ranges represents the distance data between the robot and surrounding obstacles returned by the LiDAR sensor;

[0024] Step 2.3: Set the robot's minimum linear velocity, maximum linear velocity, minimum angular velocity, and maximum angular velocity.

[0025] Furthermore, the specific process of step 3 is as follows:

[0026] Step 3.1, set function parameters: using R goal Represents the positive reward for reaching the specified target location; denoted by R. collision D represents a negative reward for encountering environmental obstacles; t D represents the distance between the robot and the center point of the target location at the current moment; t-1 L represents the distance between the robot and the center point of the target position at the previous moment; d This represents the threshold for reaching the center point of the target location, when D t A value less than this indicates that the robot has reached the target position; Min represents the minimum distance of the lidar emitted by the robot; S c Indicates the safe distance from environmental obstacles; R1 is a set ratio of R goal A slightly smaller positive reward parameter; R2 is set compared to R. collision A slightly larger negative reward parameter;

[0027] Step 3.2, set the reward and punishment function: (1) If the distance D between the robot and the target position is... t Not greater than the threshold L d This indicates that the target point has been reached, and a positive reward R is given. goal (2) If the minimum distance Min of the laser radar emitted by the robot is not greater than the safe distance S from environmental obstacles. c If it hits the wall, it indicates a negative reward; (3) when D t-1 -D t When <= 0, it indicates that the robot's current distance from the target position is greater than the previous moment, so a negative reward R2 is given to encourage it to avoid environmental obstacles; (4) when D t-1 -D t When the distance is greater than 0, it indicates that the robot is closer to the target position than at the previous time step, meaning it is closer to the target position but has not yet reached it. In this case, a positive reward of R1 and the product of the distance difference between the two time steps is given to encourage it to get closer to the target position.

[0028] Step 3.3, conditions for ending training: (1) In step 3.2 (1), reaching the target position ends the current training round; (2) In step 3.2 (2), hitting the wall ends the current training round; (3) Set the maximum number of steps for each training round, and end the current training round when the maximum number of steps is reached.

[0029] Furthermore, the specific process of step 4 is as follows:

[0030] Step 4.1: Run a path planning robot or other expert system to generate a batch of path planning schemes;

[0031] Step 4.2: For each path planning scheme, input it into the simulation environment for simulation to obtain the robot's real performance in the simulation environment;

[0032] Step 4.3: Record the actual performance of the robot corresponding to each path planning scheme, and add it to the experience pool as expert experience data for training the path planning robot.

[0033] Step 4.4: Repeat steps 4.1 to 4.3 until the experience pool has accumulated enough expert experience data.

[0034] Furthermore, the specific process of step 5 is as follows:

[0035] Step 5.1: Initialize the SumTree by creating a SumTree of size n, with each node's value initialized to 0.

[0036] Step 5.2, the sample priority is defined by TD-error, as shown in equation (1):

[0037] δ t =r t +γQ(s t+1 ,a t+1 )-Q(s t ,a t (1)

[0038] Where, δ t Represents the TD error, r t Let γ represent the reward value at time t, γ be the discount factor, Q be the state value of the Critic, and a be the value of the Critic. t and a t+1 Let s represent the actions at time t and time t+1, respectively. t and s t+1 These represent the states at time t and time t+1, respectively.

[0039] Step 5.3: Sample from SumTree. The priority value sampling range is [0, SumTree's total weight]. When extracting data, assuming the extracted data is s, start traversing its child nodes from the topmost parent node of the tree structure. If the left child node is greater than s, use that node as the parent node for the next round of traversal and traverse its child nodes. If the left child node is less than s, subtract the value of the left child node from s, and then select the right child node as the parent node for the next round of traversal and traverse its child nodes until the bottommost leaf node is reached. The value of this leaf node is the priority. Based on this priority value, find the corresponding experience element from the experience pool.

[0040] Step 5.4, train the policy network by using the experience elements obtained from the experience pool to train the policy network;

[0041] Step 5.5: Update SumTree by adjusting the priority of each trained experience element using a formula and updating it in SumTree to ensure that the value of each node is still the sum of all priorities.

[0042] Step 5.6: Repeat steps 5.3 to 5.5 until training is complete.

[0043] Furthermore, the specific process of step 6 is as follows:

[0044] Using a one-step TD to estimate the action state value function V(s) t When ), its update target is a one-step return, as shown in equation (2):

[0045]

[0046] in, Indicates a one-step return, r t+1 V(s) represents the reward value at time t+1. t+1 ) represents the action state value function at time t+1.

[0047] Therefore, the update objective of the n-step TD is the n-step return, and the update formula for estimating the value function using the n-step TD method is shown in equation (3):

[0048]

[0049] Where α is the reward coefficient of entropy, and the updated target For cumulative returns, the formula is shown in equation (4):

[0050]

[0051] Wherein, V(s) t+n ) is the action state value function at time t+n.

[0052] Furthermore, the specific process in step 7 is as follows:

[0053] Step 7.1 involves sampling data (s) from the experience pool. t a t s t+ 1, r t+ 1) Input into the policy network, the policy network outputs all possible actions predicted by the policy class for the next step, and calculates the entropy of these actions;

[0054] Step 7.2, in the policy network, the loss function for gradient descent training is defined as shown in equation (5):

[0055]

[0056] Where B represents a batch of data taken from the experience pool, (s t ,a t ,r t+1 ,s t+1 Let ρ be an experience, E be the expected reward of the current state, and ρ be the expected reward of the current state. π Let a′ be the set of all states and actions. t For the Actor network, predict all possible next actions for policy π, q(s) t ,a′ t ) represents the predictive value estimate, lnπ(a′) t |s t ,θ) is the entropy.

[0057] Furthermore, the specific process in step 8 is as follows:

[0058] Step 8.1: Sample a set of data (s) from the experience pool. t a t s t+1 r t+1 ), will state s t+1 Input the target value network to obtain the V value in this state. According to the optimal Bellman equation, use equation (6) as the state s. t True value estimate:

[0059]

[0060] in, For state s t The true value estimate, R t+1 For the data collected at time t+1 of the strategy, E π This represents the expected cumulative return in the current state;

[0061] Step 8.2, set state s t The input is fed into two Q-networks, Q0 and Q1, to obtain the corresponding action 'a' output from each network. t Action value function q(s) t ,a t This value serves as the state s. t Predictive value estimation;

[0062] Step 8.3: Use the mean squared loss function as the loss function to train and update the neural network Q. The loss function is defined by equation (7):

[0063]

[0064] Where ω represents the network parameters.

[0065] Furthermore, the specific process of step 9 is as follows:

[0066] Step 9.1 involves sampling data (s) from the experience pool. t a t s t+1 r t+1 The input is fed into the value network, and the state value is estimated using an entropy-containing formula. The true value of the value network output is obtained through equation (8).

[0067]

[0068] Where, a′ t For the policy faction of the Actor network, predict all possible next actions, lnπ(a′) t |s t Let θ be the entropy, and α be the reward coefficient of the entropy;

[0069] Step 9.2: Using the output of the value network as the predicted value and MSELoss as the loss function, train the neural network V and calculate the loss of the value network as shown in equation (9):

[0070]

[0071] Furthermore, the specific process of step 10 is as follows:

[0072] Step 10.1: Update the target policy network, target Q network, and target value function network at a certain frequency to increase the stability of the algorithm;

[0073] Step 10.2: Determine whether the algorithm has reached the preset termination condition;

[0074] Step 10.3: Output the final result of the algorithm.

[0075] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0076] This invention combines robot navigation with the SAC algorithm in reinforcement learning, overcoming the limitations of traditional path planning algorithms on the model. It improves the robot's learning speed and the efficiency of utilizing experience samples, solving the problems of poor optimal path selection for delivering multiple items at once, the tendency to get trapped in local optima, and the inability to rationally plan paths and accurately avoid environmental obstacles in the goods delivery process. Specifically, this is reflected in the following:

[0077] 1) From a safety perspective, allowing an agent that has not yet learned to interact with the environment may lead to danger or cause huge losses. This invention introduces offline reinforcement learning, which allows the agent to obtain a better strategy from a pre-collected and defined dataset without interacting with the environment.

[0078] 2) Compared with the traditional experience replay mechanism of uniform sampling with equal probability, the present invention uses a priority experience replay mechanism based on SumTree to break the correlation between samples and effectively improve the utilization rate of important samples by considering the importance of the samples themselves.

[0079] 3) Regarding the priority method for updating samples, compared with calculating single-step TD error, this invention takes future gaps into account by calculating multi-step TD error. By increasing the step size, it enhances the importance of future rewards, reduces bias, avoids overfitting, and maintains the diversity of data samples. Attached Figure Description

[0080] Figure 1 This is a flowchart of a multi-objective path planning method based on the improved SAC algorithm;

[0081] Figure 2 This is a flowchart of the steps based on SumTree's priority experience replay mechanism;

[0082] Figure 3 This is a diagram of an offline reinforcement learning method. Detailed Implementation

[0083] To facilitate understanding of the present invention, a more comprehensive description will be given below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the present invention.

[0084] A multi-objective path planning method based on an improved SAC algorithm includes the following steps:

[0085] Step 1, Environment Setup: Design a simulated indoor environment in the Gazebo simulator within ROS for training purposes. This includes designing the robot's initial position and the positions of multiple target points. The specific steps are as follows:

[0086] Step 1.1: Build a simulation environment in Gazebo that is similar to a real warehouse environment for training and testing;

[0087] Step 1.2: The initial position of the mobile robot is (8,0). Three target points are to be set, with positions of (5,-5.5), (-3,6), and (-7,-5.5).

[0088] Step 1.3: Use the robot's lidar sensor to obtain its distance information in the surrounding environment and interact with the surrounding environment. Design topics to receive radar information and use the radar information as algorithm input.

[0089] Step 2, initialize network input parameters: including readings from the mobile robot's radar in each direction, the robot's angular velocity and linear velocity. The specific steps are as follows:

[0090] Step 2.1: The robot has a 360-degree LiDAR to perceive the surrounding environment and divides its perception data into 24 discrete dimensions, that is, its sampling dimension is set to 24 dimensions; at the same time, in the simulation environment, the robot's total state is represented by a 28-dimensional vector, including position, velocity and other information.

[0091] Step 2.2, goal refers to the coordinates of the target position, goal_distance represents the distance between the robot's current position and the target position, heading represents the robot's orientation relative to the target point, and scan_ranges represents the 24-dimensional data of the distance to surrounding obstacles returned by the LiDAR sensor;

[0092] Step 2.3: Set the robot's minimum linear velocity to 0.0 m / s, maximum linear velocity to 0.3 m / s, minimum angular velocity to -1.0 rad / s, and maximum angular velocity to 2.0 rad / s.

[0093] Step 3, design the path planning reward and penalty function. The specific steps are as follows:

[0094] Step 3.1, set function parameters: using R goal Represents the positive reward for reaching the specified target location; denoted by R. collision D represents a negative reward for encountering environmental obstacles; t D represents the distance between the robot and the center point of the target location at the current moment; t-1 L represents the distance between the robot and the center point of the target position at the previous moment; d This represents the threshold for reaching the center point of the target location, when D t A value less than this indicates that the robot has reached the target position; Min represents the minimum distance of the lidar emitted by the robot; S c Indicates the safe distance from environmental obstacles; R1 is a set ratio of R goal A slightly smaller positive reward parameter; R2 is set compared to R. collision A slightly larger negative reward parameter;

[0095] Step 3.2, set the reward and punishment function: (1) If the distance D between the robot and the target position is... t Not greater than the threshold L dThis indicates that the target point has been reached, and a positive reward R is given. goal (2) If the minimum distance Min of the laser radar emitted by the robot is not greater than the safe distance S from environmental obstacles. c If it hits the wall, it indicates a negative reward; (3) when D t-1 -D t When <= 0, it indicates that the robot's current distance from the target position is greater than the previous moment, so a negative reward R2 is given to encourage it to avoid environmental obstacles; (4) when D t-1 -D t When the distance is greater than 0, it indicates that the robot is closer to the target position than at the previous time step, meaning it is closer to the target position but has not yet reached it. In this case, a positive reward of R1 and the product of the distance difference between the two time steps is given to encourage it to get closer to the target position.

[0096] Step 3.3, conditions for ending training: (1) In step 3.2 (1), reaching the target position ends the training round; (2) In step 3.2 (2), hitting the wall ends the training round; (3) Set the maximum number of steps per training round to 5000, and end the training round when the number of steps reaches 5000.

[0097] Step 4: Pre-train using offline expert experience based on the current simulation environment. The specific steps are as follows:

[0098] Step 4.1: Run a path planning robot or other expert system to generate a batch of path planning schemes;

[0099] Step 4.2: For each path planning scheme, input it into the simulation environment for simulation to obtain the robot's real performance in the simulation environment, including information such as the robot's path, speed, and posture.

[0100] Step 4.3: Record the actual performance of the mobile robot corresponding to each path planning scheme and add it to the experience pool as expert experience data. This experience data will be used to train the path planning robot to improve its path planning ability in the real environment.

[0101] Step 4.4: Repeat steps 4.1 to 4.3 until the experience pool has accumulated enough expert experience data.

[0102] Step 5: Improve the traditional SAC algorithm by adding a SumTree-based priority experience replay mechanism. Sort the experiences from the experience pool according to their importance; the more important the experience, the higher the probability of it being sampled and used. The specific steps are as follows:

[0103] Step 5.1: Initialize the SumTree. Create a SumTree of size n, with each node's value initially set to 0. The SumTree tree structure is used to store sample priorities, with priority information stored only in the leaf nodes. Store all generated experience elements into the experience pool, retrieving them sequentially from left to right and updating the corresponding parent node. Assign a priority to each element. During sampling, prioritize samples with higher priorities, i.e., samples that are helpful for updating the value function, thereby improving training speed and efficiency.

[0104] Step 5.2, the sample priority is defined by TD-error, as shown in equation (1):

[0105] δ t =r t +γQ(s t+1 ,a t+1 )-Q(s t ,a t (1)

[0106] Where, δ t Represents the TD error, r t Let γ represent the reward value at time t, γ be the discount factor, Q be the state value of the Critic, and a be the value of the Critic. t and a t+1 Let s represent the actions at time t and time t+1, respectively. t and s t+1 These represent the states at time t and time t+1, respectively.

[0107] The larger the TD-error, the higher the sample priority.

[0108] Step 5.3: Sample from SumTree. The priority value sampling range is [0, SumTree's total weight]. When extracting data, assuming the extracted data is s, start traversing its child nodes from the topmost parent node of the tree structure. If the left child node is greater than s, use that node as the parent node for the next round of traversal and traverse its child nodes. If the left child node is less than s, subtract the value of the left child node from s, and then select the right child node as the parent node for the next round of traversal and traverse its child nodes until the bottommost leaf node is reached. The value of this leaf node is the priority. Based on this priority value, find the corresponding experience element from the experience pool.

[0109] Step 5.4, train the policy network by using the experience elements obtained from the experience pool to train the policy network;

[0110] Step 5.5: Update SumTree by adjusting the priority of each trained experience element using a formula and updating it in SumTree to ensure that the value of each node is still the sum of all priorities.

[0111] Step 5.6: Repeat steps 5.3 to 5.5 until training is complete.

[0112] Step 6: Optimize single-step TD-error by calculating the n-step TD-error to more accurately update the priority values ​​in the SumTree. The specific steps are as follows:

[0113] Using a one-step TD to estimate the action state value function V(s) t When ), its update target is a one-step return, as shown in equation (2):

[0114]

[0115] in, Indicates a one-step return, r t+1 V(s) represents the reward value at time t+1. t+1 ) represents the action state value function at time t+1.

[0116] Therefore, the update objective of the n-step TD is the n-step return, and the update formula for estimating the value function using the n-step TD method is shown in equation (3):

[0117]

[0118] Where α is the reward coefficient of entropy, and the updated target For cumulative returns, the formula is shown in equation (4):

[0119]

[0120] Wherein, V(s) t+n ) is the action state value function at time t+n.

[0121] Step 7: Update the policy network (Actor). Sample actions using the reparameterization technique, and then update the current Actor network using the loss function. The specific steps are as follows:

[0122] Step 7.1 involves sampling data (s) from the experience pool. t a t s t+1 r t+1 The input is fed into the policy network, which outputs all possible actions predicted by the policymakers for the next step, and the entropy of these actions is calculated.

[0123] Step 7.2, in the policy network, the loss function for gradient descent training is defined as shown in equation (5):

[0124]

[0125] Where B represents a batch of data taken from the experience pool, (s t ,a t ,r t+1 ,s t+1 Let ρ be an experience, E be the expected reward of the current state, and ρ be the expected reward of the current state. π Let a′ be the set of all states and actions. t For the Actor network, predict all possible next actions for policy π, q(s) t ,a′ t ) represents the predictive value estimate, lnπ(a′) t |s t ,θ) is the entropy.

[0126] Step 8, update the Q network Q Critic by adjusting the state S t The true value estimation and predicted value estimation are performed by averaging a batch of data taken from the experience pool, and then training neural networks Q0 and Q1. The specific steps are as follows:

[0127] Step 8.1: Sample a set of data (s) from the experience pool. t a t s t+1 r t+1 ), will state s t+1 Input the target value network to obtain the V value in this state. According to the optimal Bellman equation, use equation (6) as the state s. t True value estimate:

[0128]

[0129] in, For state s t The true value estimate, R t+1 For the data collected at time t+1 of the strategy, E π This represents the expected cumulative return in the current state;

[0130] Step 8.2, set state s t The input is fed into two Q-networks, Q0 and Q1, to obtain the corresponding action 'a' output from each network. t Action value function q(s) t ,a t This value serves as the state s. t The predictive value estimate.

[0131] Step 8.3: Use the mean squared loss function MSELoss as the loss function to train and update the neural network Q. The loss function is defined by equation (7):

[0132]

[0133] Where ω represents the network parameters.

[0134] Taking the MSELoss value means averaging the data in this batch. Two Q-networks are used to prevent overestimation of the V-network.

[0135] Step 9: Update the value network V Critic. Estimate the state value using an entropy-containing formula to obtain the true value of the V Critic network output. Use the output of the V Critic network as the predicted value and MSEloss as the loss function to train the neural network V. The specific steps are as follows:

[0136] Step 9.1 involves sampling data (s) from the experience pool. t a t s t+1 r t+1 The input is fed into the value network, and the state value is estimated using an entropy-containing formula. The true value of the value network output is obtained through equation (8).

[0137]

[0138] Where, a′ t For the policy faction of the Actor network, predict all possible next actions, lnπ(a′) t |s t Let θ be the entropy, and α be the reward coefficient of entropy (which determines the importance of entropy; the larger the reward coefficient, the more important the entropy).

[0139] Step 9.2: Using the output of the value network as the predicted value and MSELoss as the loss function, train the neural network V and calculate the loss of the value network as shown in equation (9):

[0140]

[0141] Step 10: Update the weights, update the coefficient alpha of the entropy regularization term, update the target network, and plan the optimal path.

[0142] Step 10.1: Update the target policy network, target Q network, and target value function network at a certain frequency to increase the stability of the algorithm;

[0143] Step 10.2: Determine whether the algorithm has reached the preset termination condition, such as reaching the maximum training period or reaching the target performance.

[0144] Step 10.3: Output the final result of the algorithm, such as the robot's optimal path planning scheme.

[0145] To demonstrate that this invention can significantly improve the utilization rate of effective path sample experience by the intelligent agent, enable it to adapt to large navigation environments more quickly, and efficiently reach the specified target point within a limited number of steps, tests were conducted in a simulation environment. Experiments were compared with the currently mainstream DDPG algorithm and the DDPG algorithm based on priority experience replay (DDPG_PER). The results are shown in Table 1. The results indicate that this invention can effectively solve the problem of low utilization rate of effective path samples in robot reinforcement learning, thus accelerating the robot's learning speed.

[0146] Table 1. Path planning test results of different algorithms

[0147]

[0148] The embodiments described above merely illustrate specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A multi-objective path planning method based on an improved SAC algorithm, characterized in that, Includes the following steps: Step 1: Build a simulation environment and design the robot's initial position and the positions of multiple target points; Step 2, initialize network input parameters; Step 3: Design the path planning reward and penalty function; Step 4: Pre-train using offline expert experience based on the current simulation environment; Step 5: Improve the traditional SAC algorithm based on the priority experience replay mechanism of SumTree; Step 6, optimize single-step TD-error; Step 7, update the policy network Actor; Step 8, update Q network Q Critic; Step 9, update the value network V Critic; Step 10: Update the weights, update the coefficient alpha of the entropy regularization term, update the target network, and plan the optimal path; The specific process of step 5 is as follows: Step 5.1: Initialize the SumTree by creating a SumTree of size n, with each node's value initialized to 0. Step 5.2, the sample priority is defined by TD-error, as shown in equation (1): （1） in, Indicates TD error, Let represent the reward value at time t, γ be the discount factor, and Q be the state value of the Critic. and These represent the actions at time t and time t+1, respectively. and These represent the states at time t and time t+1, respectively. Step 5.3: Sample from SumTree. The priority value sampling range is [0, SumTree's total weight]. When extracting data, assuming the extracted data is s, start traversing its child nodes from the topmost parent node of the tree structure. If the left child node is greater than s, use that node as the parent node for the next round of traversal and traverse its child nodes. If the left child node is less than s, subtract the value of the left child node from s, and then select the right child node as the parent node for the next round of traversal and traverse its child nodes until the bottommost leaf node is reached. The value of this leaf node is the priority. Based on this priority value, find the corresponding experience element from the experience pool. Step 5.4, train the policy network by using the experience elements obtained from the experience pool to train the policy network; Step 5.5: Update SumTree by adjusting the priority of each trained experience element using a formula and updating it in SumTree to ensure that the value of each node is still the sum of all priorities. Step 5.6, repeat steps 5.3 to 5.5 until training is complete; The specific process of step 6 is as follows: Using a one-step TD estimation motion state value function At that time, its update target is a one-step return, as shown in equation (2): （2） in, This indicates a one-step return. This represents the reward value at time t+1. The function representing the action state value at time t+1; Therefore, the update objective of the n-step TD is the n-step return, and the update formula for estimating the value function using the n-step TD method is shown in equation (3): （3） Where α is the reward coefficient of entropy, and the updated target For cumulative returns, the formula is shown in equation (4): （4） in, This is the action state value function at time t+n.

2. The multi-objective path planning method based on the improved SAC algorithm according to claim 1, characterized in that, The specific process of step 1 is as follows: Step 1.1: Build a simulation environment in Gazebo that is similar to a real warehouse environment for training and testing; Step 1.2: Set the robot's initial position and target position; Step 1.3: Use the robot's lidar sensor to obtain its distance information in the surrounding environment and interact with the surrounding environment. Design topics to receive radar information and use the radar information as algorithm input.

3. A multi-objective path planning method based on an improved SAC algorithm according to claim 2, characterized in that, The specific process of step 2 is as follows: Step 2.1: Set the sampling dimension of the robot's LiDAR sensor and the robot's overall state dimension; Step 2.2, goal refers to the coordinates of the target position, goal_distance represents the distance between the robot's current position and the target position, heading represents the robot's orientation relative to the target point, and scan_ranges represents the distance data between the robot and surrounding obstacles returned by the LiDAR sensor; Step 2.3: Set the robot's minimum linear velocity, maximum linear velocity, minimum angular velocity, and maximum angular velocity.

4. A multi-objective path planning method based on an improved SAC algorithm according to claim 3, characterized in that, The specific process of step 3 is as follows: Step 3.1, set function parameters: using R goal Represents the positive reward for reaching the specified target location; denoted by R. collision D represents a negative reward for encountering environmental obstacles; t D represents the distance between the robot and the center point of the target location at the current moment; t-1 L represents the distance between the robot and the center point of the target position at the previous moment; d This represents the threshold for reaching the center point of the target location, when D t A value less than this indicates that the robot has reached the target position; Min represents the minimum distance of the lidar emitted by the robot; S c Indicates the safe distance from environmental obstacles; R1 is a set ratio of R goal A slightly smaller positive reward parameter; R2 is set compared to R. collision A slightly larger negative reward parameter; Step 3.2, set the reward and punishment function: (1) If the distance D between the robot and the target position is... t Not greater than threshold L d This indicates that the target point has been reached, and a positive reward R is given. goal (2) If the minimum distance Min of the laser radar emitted by the robot is not greater than the safe distance S from the environmental obstacle. c If it hits the wall, it indicates a negative reward; (3) when D t-1 -D t When <= 0, it indicates that the robot's current distance from the target position is greater than the previous moment, so a negative reward R2 is given to encourage it to avoid environmental obstacles; (4) when D t-1 -D t When the distance is greater than 0, it indicates that the robot is closer to the target position than at the previous time step, meaning it is closer to the target position but has not yet reached it. In this case, a positive reward of R1 and the product of the distance difference between the two time steps is given to encourage it to get closer to the target position. Step 3.3, conditions for ending training: (1) In step 3.2 (1), reaching the target position ends the current training round; (2) In step 3.2 (2), hitting the wall ends the current training round; (3) Set the maximum number of steps for each training round, and end the current training round when the maximum number of steps is reached.

5. A multi-objective path planning method based on an improved SAC algorithm according to claim 4, characterized in that, The specific process of step 4 is as follows: Step 4.1: Run a path planning robot or other expert system to generate a batch of path planning schemes; Step 4.2: For each path planning scheme, input it into the simulation environment for simulation to obtain the robot's real performance in the simulation environment; Step 4.3: Record the actual performance of the robot corresponding to each path planning scheme, and add it to the experience pool as expert experience data for training the path planning robot. Step 4.4: Repeat steps 4.1 to 4.3 until the experience pool has accumulated enough expert experience data.

6. A multi-objective path planning method based on an improved SAC algorithm according to claim 5, characterized in that, The specific process in step 7 is as follows: Step 7.1 involves sampling data from the experience pool. The input is fed into the policy network, which outputs all possible actions predicted by the policymakers and calculates the entropy of these actions. Step 7.2, in the policy network, the loss function for gradient descent training is defined as shown in equation (5): （5） Where B represents a batch of data taken from the experience pool. Let E be an experience, and E be the expected reward for the current state. For the set of all states and actions, For the Actor network, predict all possible next actions for policy π. For predictive value estimation, Entropy.

7. A multi-objective path planning method based on an improved SAC algorithm according to claim 6, characterized in that, The specific process in step 8 is as follows: Step 8.1: Sample a set of data from the experience pool. , will state Input the target value network to obtain the V value in this state. According to the optimal Bellman equation, use equation (6) as the state. True value estimate: (6) in, For state The true value estimate The data collected for the strategy at time t+1 This represents the expected cumulative return in the current state; Step 8.2, change the status The input is fed into two Q-networks, Q0 and Q1, to obtain the corresponding network output actions. Action value function This value serves as the state. Predictive value estimation; Step 8.3: Use the mean squared loss function as the loss function to train and update the neural network Q. The loss function is defined by equation (7): （7） in, These are network parameters.

8. A multi-objective path planning method based on an improved SAC algorithm according to claim 7, characterized in that, The specific process of step 9 is as follows: Step 9.1 involves sampling data from the experience pool. The input is fed into the value network, and the state value is estimated using an entropy-containing formula. The true value of the value network output is obtained through equation (8): （8） in, For the policy faction of the Actor network, predict all possible next actions. Let α be the entropy, and α be the reward coefficient of entropy; Step 9.2: Using the output of the value network as the predicted value and MSELoss as the loss function, train the neural network V and calculate the loss of the value network as shown in equation (9): （9） The specific process of step 10 is as follows: Step 10.1: Update the target policy network, target Q network, and target value function network at a certain frequency to increase the stability of the algorithm; Step 10.2: Determine whether the algorithm has reached the preset termination condition; Step 10.3: Output the final result of the algorithm.

Images