RRT path planning method and device based on police-thief mechanism and discrete SAC and medium

Abstract

The invention discloses an RRT path planning method and device based on a police-thief mechanism and discrete SAC, and a medium. The method comprises the following steps: initializing a fast random tree structure and node attributes; establishing a police-thief node mechanism, realizing node role division and dynamic pruning, and performing bridging optimization on nodes; a reinforcement learning decision network based on discrete Soft Actor-Critic is constructed, and joint decision making is carried out on the node expansion direction, the step length and the execution strategy of a police-thief mechanism; and updating reinforcement learning network parameters according to the reward shaping function until the optimal path is converged. The planning efficiency and the path quality are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the technical field of intelligent robot path planning, and particularly relates to a path planning method, device and medium for RRT based on the "police-thief" mechanism and discrete SAC. Background Technology

[0002] The Rapidly-exploring Random Tree (RRT) algorithm is a commonly used sampling-based path planning method with advantages such as high computational efficiency and wide applicability. However, in complex obstacle environments, traditional RRT algorithms often generate a large number of redundant nodes due to the strong randomness of the expansion process, leading to an bloated search tree structure and slow convergence speed. Although improved algorithms such as RRT* can optimize the path, they still require a large number of iterations to converge, and redundancy pruning is mostly performed after planning is completed, affecting real-time performance. In addition, traditional RRT algorithms lack an adaptive adjustment mechanism for environmental complexity, and the step size and sampling strategy are usually fixed parameters, making it difficult to balance convergence speed and path smoothness.

[0003] In recent years, reinforcement learning (RL) technology has been gradually introduced into the field of path planning. The discrete Soft Actor-Critic (SAC) algorithm has the advantages of good stability and fast convergence, but existing RL-RRT combined methods are mostly focused on "expansion direction optimization", and have not yet achieved self-learning and online optimization in the "node pruning" stage. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide an RRT path planning method, device and medium based on the "police-thief" mechanism and discrete SAC, which significantly improves planning efficiency and path quality.

[0005] Technical solution: To achieve the above objectives, the RRT path planning method of the present invention includes the following steps: S1. Initialize the fast random tree structure and node attributes; S2. Establish a police-thief node mechanism to realize node role division and dynamic pruning, and optimize node bridging; S3. Construct a reinforcement learning decision network based on discrete Soft Actor-Critic to jointly decide on the node expansion direction, step size, and execution strategy of the police-thief mechanism; S4. Update the reinforcement learning network parameters according to the reward shaping function until convergence to the optimal path.

[0006] Optionally, step S1 initializes the fast random tree structure and node attributes, specifically including: Given the starting and ending coordinates of a wheeled mobile robot, define a node structure. ,in Represents the physical location coordinates of the node in the two-dimensional working environment; This indicates the index of the parent node of the node in the fast random tree, which is used to describe the topological connection relationship of the search tree; This represents the cumulative geometric path cost from the starting point along the search tree to this node, and is the path length. This represents the heuristic estimate of the Euclidean distance from the node to the target location, which is the straight-line distance from the node to the target. Initialize the random tree with the starting node as the root node, and create a list of nodes. This is used to store all nodes in the search tree; Maximum expansion step size set With sampling quantity Initialize the node role set, including the police node set. thief node set The set of deleted nodes ; Establish a grid hash table based on the environmental boundary dimensions .

[0007] Optionally, step S2 establishes a police-thief node mechanism, specifically including: Based on node cost Compared with the current global optimal cost The ratio determines the node role, satisfying: , , in Represents a node in a fast random tree. For the set of police nodes; For the set of thief nodes; , These are the node role promotion threshold and demotion threshold, respectively; Represents a node The cumulative geometric path cost, This represents the cost of the globally optimal path obtained during the current search process; For a node in the thief state, calculate its distance to the police node: , when When this happens, it is considered a captured node and removed from the tree structure, where For the capture radius, For the police node coordinates, The coordinates of the thief's node; information about the captured or deleted node is recorded in the deleted node set. .

[0008] Optionally, step S2 may also include a dynamic balancing step: Statistics on the current ratio of police to thieves If it deviates from the target ratio range The target parameter for the thief ratio is then dynamically adjusted to maintain a balance between expansion and pruning; among which... This represents the ratio of police nodes to thief nodes. These represent the lower limit and upper limit threshold of the quantity ratio, respectively.

[0009] Optionally, the specific steps for bridging optimization of nodes in step S2 include: If node The connection between the parent node and its child nodes satisfies the no-collision constraint, and the nodes The number of child nodes does not exceed the threshold Then the node The parent pointer of each child node is determined by the node. Change it to its parent node and delete the node. If node If a parent node cannot establish a collision-free connection with its child nodes, then backtrack upwards along the parent pointer of the search tree to search for the nearest undeleted ancestor node. When the ancestor node It satisfies the no-collision constraint with its child nodes, and the number of child nodes does not exceed a threshold. When necessary, reattach the child node to the ancestor node. And delete the node ;in, This is the threshold for the number of child nodes. Represents the distance from a node in a tree topology. The most recent ancestor node that has not been deleted.

[0010] Optionally, step S3, which involves constructing the reinforcement learning decision network, includes the following steps: Establish a discrete Soft Actor-Critic (SAC) network structure, including a policy network Actor and two state-action value networks Q1 and Q2, and their corresponding target networks Q1_target and Q2_target, wherein: , , in, Represents the state feature vector; For discrete actions; This represents the normalized exponential function; It is a strategy distribution; For the parameterized functions of the policy network Actor, For the policy network Actor; For the first indivual Network-based estimation of action value For the first indivual Network parameters; The policy network Actor and the dual-Q networks Q1 and Q2 are updated independently using the Adam optimizer, and the soft update coefficient of the target network is set. Update the parameters according to the formula: , in, The parameters of the current policy network (Actor network) are as follows: The parameters of the target policy network (Actor network); Based on sampling experience Perform Q-value target calculation: , in, For instant rewards; Temperature coefficient; This is the end marker; This is a discount factor with a value range of (0,1), used to represent the weight of future rewards in the current decision; This is the state feature vector for the next state. Discrete actions for the next state; Update the dual-Q network and the Actor network respectively by minimizing the following loss functions: , , in Let be the loss function for the Q-network loss; Let Actor be the optimization objective function for the policy network. This represents the mathematical expectation of the distribution of state-action samples obtained from the experience replay pool.

[0011] Optionally, in step S3, the reinforcement learning decision network introduces an expansion head and a pruning head. The expansion head is used to adjust the decision network based on the given state. The output node extends the action, which includes at least the selection of the extension direction and the distance of the step, guiding the wheeled mobile robot to extend the direction and distance of the next geometric path in the two-dimensional plane; the trimming head is used to adjust the extension direction and distance of the next geometric path given a state. The output is a pruning execution strategy related to the police-thief mechanism, used to determine whether to perform capture deletion or bridging operations on nodes in a thief state or with redundant structures, in order to control the size of the search tree and the quality of the path; the dual-Q network Q1 and Q2 perform long-term reward evaluation on the joint decision generated by the expansion head and the pruning head.

[0012] Optionally, step S4, which updates the reinforcement learning network parameters based on the reward shaping function, specifically includes the following steps: Define a reward shaping function based on path improvement and node pruning. It consists of path shortening reward, redundancy penalty, step penalty, and success reward, and the calculation formula is as follows: , in , , and All are adjustable weight parameters. This indicates the improvement in path length compared to the previous plan. The penalty represents the number of nodes that are deleted or captured. A slight penalty for each step of expansion. This represents the global reward value when the plan is successful; Set aggressive reward shaping parameters during the training initialization phase and apply them during the failure round. Negative penalties are applied by dynamically adjusting the sampling quantity and evaluation interval through reinforcement learning. After each node expansion and police-thief interaction, the state transition sample is... Store the data in the experience replay pool and update the network parameters by batch sampling according to the set parameters. If the number of samples in the experience pool reaches the threshold If the optimal path does not improve in multiple iterations, the path planning process is considered to have converged.

[0013] In one embodiment, an electronic device includes a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method described above.

[0014] In one embodiment, a computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described above.

[0015] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: This invention achieves dynamic role assignment, capture and deletion, and structural bridging and reattaching during the node expansion phase. It also significantly improves planning efficiency and path quality by using a reinforcement learning agent to jointly decide on the expansion direction, step size, and pruning strategy. This invention implements dynamic pruning of redundant nodes, and the police-thief mechanism enables real-time role assignment and online capture and deletion of nodes, significantly reducing redundant expansion. This invention achieves self-repair of the tree structure; the bridging and reattaching mechanism automatically restores parent-child relationships after node deletion, ensuring path connectivity. This invention achieves adaptive expansion and step size control; the discrete SAC enables joint decision-making on direction and step size, allowing for flexible adjustment of the expansion strategy based on environmental complexity. The reward shaping function of this invention comprehensively considers path length, number of nodes, and success rate, guiding the learning strategy to stable convergence. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the "police-thief" node mechanism of the present invention; Figure 3 This is a diagram of the discrete SAC reinforcement learning decision network structure of the present invention; Figure 4 This is a schematic diagram of the path planning simulation results in this invention. Detailed Implementation

[0017] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0018] like Figure 1 and Figure 2 As shown, the RRT path planning method based on the "police-thief" mechanism and discrete SAC in this invention includes the following steps: (1) Initialize the fast random tree structure and node attributes; Step (1) involves initializing the fast random tree structure and node attributes, including: (1.1) Given the coordinates of the starting and ending points of the wheeled mobile robot, define the node structure. ,in Represents the physical location coordinates of the node in the two-dimensional working environment; This indicates the index of the parent node of the node in the fast random tree, which is used to describe the topological connection relationship of the search tree; This represents the cumulative geometric path cost from the starting point along the search tree to this node, and is the path length. This represents the heuristic estimate of the Euclidean distance from the node to the target location, which is the straight-line distance from the node to the target. (1.2) Initialize the random tree with the starting node as the root node and create a list of nodes. This is used to store all nodes in the search tree; (1.3) Based on the set maximum expansion step size With sampling quantity This is used to control the tree's expansion speed and search density; among which, This represents the maximum geometric step size allowed during a single tree expansion, which is the maximum geometric distance that a wheeled mobile robot can travel in a single planning expansion within a two-dimensional plane. This represents the number of samples used to generate candidate expansion directions during the planning process, which is the number of samples per unit planning period; (1.4) Initialize the node role set, including the police node set. thief node set The set of deleted nodes This is used for subsequent police-thief mechanism management and path cost recording; (1.5) Establish a grid hash table based on the environmental boundary dimensions This enables fast nearest neighbor lookup for nodes, ensuring that sampled nodes do not cross boundaries and satisfy obstacle constraints.

[0019] (2) Establish a police-thief node mechanism to realize node role division and dynamic pruning; Step (2) describes the establishment of a police-thief node mechanism, used to implement node role division and dynamic pruning, including: (2.1) During the rapid exploration of random tree expansion, based on node cost Compared with the current global optimal cost The ratio determines the node role, satisfying: , , in Represents a node in a fast random tree. This is a set of police nodes, used to represent nodes with better current path quality that should be prioritized for retention; This is a set of thief nodes, used to represent nodes with relatively high path costs and potential redundancy; , These are the node role promotion threshold and demotion threshold, used to adjust the tightness of node role classification; Represents a node The cumulative geometric path cost, This represents the cost of the globally optimal path obtained during the current search process; (2.2) For a node in the thief state, calculate its distance to the police node: , when When this happens, it is considered a captured node and removed from the tree structure, where For the capture radius, ( ) represents the coordinates of the police node. The coordinates of the thief's node; (2.3) After each expansion, count the ratio of the number of police nodes to the number of thief nodes. If it deviates from the target ratio range The target parameter for the thief ratio is then dynamically adjusted to maintain a balance between expansion and pruning; among which... This represents the ratio of police nodes to thief nodes. These represent the lower and upper limits of the quantity ratio, respectively, used to constrain the pruning intensity of the search tree; (2.4) Record the information of the captured or deleted nodes to the set of deleted nodes. This is used to calculate the redundancy penalty value in the subsequent reward function; (2.5) If node The connection between the parent node and its child nodes satisfies the no-collision constraint, and the nodes The number of child nodes does not exceed the threshold Then the node The parent pointer of each child node is determined by Change it to its parent node and delete the node. This reduces redundant intermediate nodes in the search tree; if a node If a parent node cannot establish a collision-free connection with its child nodes, then backtrack upwards along the parent pointer of the search tree to search for the nearest undeleted ancestor node. ;when It satisfies the no-collision constraint with its child nodes, and the number of child nodes does not exceed a threshold. When that happens, reattach the child node to And delete the node ;in, This is a threshold for the number of child nodes, used to limit the maximum size of the branching nodes after bridging; Represents the distance from a node in a tree topology. The most recent ancestor node that has not been deleted.

[0020] (3) Construct a reinforcement learning decision network to jointly decide on the node expansion direction, step size, and execution strategy of the police-thief mechanism; like Figure 3 As shown, step (3) involves constructing a reinforcement learning decision network to jointly decide on the node expansion direction, step size, and execution strategy of the police-thief mechanism, including: (3.1) Establish a discrete Soft Actor-Critic (SAC) network structure, including a policy network Actor and two state-action value networks Q1 and Q2, and their corresponding target networks Q1_target and Q2_target, wherein: , , in, The state feature vector represents a feature vector that describes the local geometry / obstacle information, tree structure information, path progress information, and historical information related to the current extended decision. For discrete actions, dimensionless indexes, the physical meaning is the joint selection of "extended direction category + discrete step length category"; This represents the normalization exponential function, used to map the real-valued vectors output by the policy network Actor to the probability distributions corresponding to each discrete action, so that the sum of the probabilities of all actions is 1; It is a strategy distribution; For the parameterized functions of the policy network Actor, For the policy network Actor; For the first indivual Network-based estimation of action value For the first indivual Network parameters of the network.

[0021] (3.2) The policy network Actor and the dual Q networks Q1 and Q2 are updated independently using the Adam optimizer, and the soft update coefficient of the target network is set. Update the parameters according to the formula: , in, The parameters of the current policy network (Actor network) are as follows: The parameters of the target policy network (Actor network); (3.3) Based on sampling experience Perform Q-value target calculation: , in, For instant rewards; The temperature coefficient is used to adjust the weight of the policy entropy term in the objective function to balance the policy's exploratory nature with the maximization of returns. This is a termination marker; This is a discount factor with a value range of (0,1), used to represent the weight of future rewards in the current decision; This is the state feature vector for the next state. Discrete actions for the next state; (3.4) Update the double Q network and the Actor network respectively by minimizing the following loss functions: , , in This is the loss function for the Q-network, used to measure the error between the action value output by the Q-network and the target Q-value; This is the optimization objective function for the policy network (Actor network), used to guide the policy network in updating while considering the constraints of action value and policy entropy. This represents the mathematical expectation of the distribution of state-action samples obtained from the experience replay pool. (3.5) Introducing an extend head and a prune head to achieve joint decision-making on node expansion and pruning. The extend head is used to determine the node expansion and pruning process in a given state. The output node extends the action, which includes at least the selection of the extension direction and the distance of the step, guiding the wheeled mobile robot to extend the direction and distance of the next geometric path in the two-dimensional plane; the trimming head is used to adjust the extension direction and distance of the next geometric path given a state. The output is a pruning execution strategy related to the police-thief mechanism. Its physical meaning is to determine whether to perform capture, deletion, or bridging operations on nodes in the thief state or redundant structures, so as to control the size of the search tree and the quality of the path. The dual Q network Q1 and Q2 evaluate the long-term reward of the joint decision generated by the expansion head and the pruning head, and determine the retention, deletion, and bridging strategies of nodes in the "expansion-pruning" loop, thereby improving the success rate and search efficiency of the wheeled mobile robot path planning.

[0022] (4) Update the reinforcement learning network parameters according to the reward shaping function until convergence to the optimal path.

[0023] Step (4), which involves updating the reinforcement learning network parameters according to the reward shaping function until the optimal path is obtained, includes: (4.1) Define a reward shaping function based on path improvement and node pruning. It consists of path shortening reward, redundancy penalty, step penalty, and success reward, and the calculation formula is as follows: , in , , and All are adjustable weight parameters. This indicates the improvement in path length compared to the previous plan. The penalty represents the number of nodes that are deleted or captured. A slight penalty for each step of expansion. This represents the global reward value when the plan is successful; (4.2) Set aggressive reward shaping parameters during the training initialization phase: , And apply in the failed round Negative penalties are applied by dynamically adjusting the sampling quantity and evaluation interval through reinforcement learning.

[0024] (4.3) After each node expansion and police-thief interaction, the state transition sample is... Store the data in the experience replay pool and update the network parameters by batch sampling according to the set parameters. If the number of samples in the experience pool reaches the threshold If the optimal path does not improve in multiple iterations, the parameter update process is triggered; if the optimal path does not improve in consecutive iterations, the path planning process is considered to have converged. This design avoids the limitation of a single shortest path objective, enabling the algorithm to achieve a balance between exploration breadth and search accuracy.

[0025] This invention achieves adaptive role partitioning, online redundancy suppression, and dynamic expansion optimization of search nodes, enabling efficient path planning and self-learning decision-making for mobile robots in complex obstacle environments. Addressing the issues of excessive node redundancy and slow convergence in traditional RRT algorithms in complex environments, this invention introduces a "police-thief" node mechanism, dividing nodes into police and thief nodes. Through capture and bridging rules, online redundancy pruning is implemented during the expansion phase, preventing disordered tree expansion. To address the fixed step size and random sampling issues in traditional RRT algorithms, this invention introduces a discrete SAC reinforcement learning agent, dynamically deciding the expansion direction and step size during the expansion phase to form an adaptive joint strategy, improving the targeting and search efficiency of tree expansion. Finally, addressing the problem that traditional RL-RRT methods only optimize the expansion direction and ignore node pruning, this invention deeply couples the reinforcement learning output with the police-thief mechanism, achieving an integrated intelligent optimization process of "expansion-capture-pruning-bridging".

[0026] Example: In The simulation was performed on a maze map containing multiple obstacle courses and bottleneck areas. The starting coordinates were set to (1,1), and the ending coordinates were set to (19,19). The main parameters are shown in Table 1. Table 1 Parameter Table

[0027] Initialization phase: Load the map and establish a random tree structure and node set; Expansion Phase: The SAC agent selects the expansion direction and step size, and new nodes are written. ; Pruning phase: Based on the relationship between node cost and distance, execute the police-thief mechanism to delete redundant nodes and bridge them; Network update phase: Calculating rewards Perform Actor network and Q network parameter updates; Convergence criterion: The path planning process is considered converged when the optimal path does not improve in multiple consecutive iterations.

[0028] like Figure 4 As shown, after 2000 training iterations, the algorithm performs as follows on the maze map: the average path length is shortened by about 24% compared to the traditional RRT; the average number of expanded nodes is reduced by 32%; the average computation time is shortened by about 40%; and the success rate is increased from 86% to 96%.

[0029] In one embodiment, an electronic device includes a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method described above.

[0030] In one embodiment, a computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described above.

Claims

1. A path planning method for RRT based on the "police-thief" mechanism and discrete SAC, characterized in that, Includes the following steps: S1. Initialize the fast random tree structure and node attributes; S2. Establish a police-thief node mechanism to realize node role division and dynamic pruning, and optimize node bridging; S3. Construct a reinforcement learning decision network based on discrete Soft Actor-Critic to jointly decide on the node expansion direction, step size, and execution strategy of the police-thief mechanism; S4. Update the reinforcement learning network parameters according to the reward shaping function until convergence to the optimal path.

2. The RRT path planning method according to claim 1, characterized in that, Step S1 initializes the fast random tree structure and node attributes, specifically including: Given the starting and ending coordinates of a wheeled mobile robot, define a node structure. ,in Represents the physical location coordinates of the node in the two-dimensional working environment; This indicates the index of the parent node of the node in the fast random tree, which is used to describe the topological connection relationship of the search tree; This represents the cumulative geometric path cost from the starting point along the search tree to this node, and is the path length. This represents the heuristic estimate of the Euclidean distance from the node to the target location, which is the straight-line distance from the node to the target. Initialize the random tree with the starting node as the root node, and create a list of nodes. This is used to store all nodes in the search tree; Maximum expansion step size set With sampling quantity Initialize the node role set, including the police node set. thief node set The set of deleted nodes ; Establish a grid hash table based on the environmental boundary dimensions .

3. The RRT path planning method according to claim 2, characterized in that, Step S2 establishes a police-thief node mechanism, specifically including: Based on node cost Compared with the current global optimal cost The ratio determines the node role, satisfying: ， ， in Represents a node in a fast random tree. For the set of police nodes; For the set of thief nodes; , These are the node role promotion threshold and demotion threshold, respectively; Represents a node The cumulative geometric path cost, This represents the cost of the globally optimal path obtained during the current search process; For a node in the thief state, calculate its distance to the police node: ， when When this happens, it is considered a captured node and removed from the tree structure, where For the capture radius, For the police node coordinates, The coordinates of the thief's node; information about the captured or deleted node is recorded in the deleted node set. .

4. The RRT path planning method according to claim 3, characterized in that, Step S2 also includes a dynamic balancing step: Calculate the current ratio of police to thief nodes: ， If it deviates from the target ratio range Then, the target parameter for the thief ratio is dynamically adjusted to maintain a balance between expansion and pruning; in This represents the ratio of police nodes to thief nodes. , These represent the lower limit and upper limit threshold of the quantity ratio, respectively.

5. The RRT path planning method according to claim 4, characterized in that, The specific steps for bridging optimization of nodes in step S2 include: If node The connection between the parent node and its child nodes satisfies the no-collision constraint, and the nodes The number of child nodes does not exceed the threshold Then the node The parent pointer of each child node is determined by the node. Change it to its parent node and delete the node. If node If a parent node cannot establish a collision-free connection with its child nodes, then backtrack upwards along the parent pointer of the search tree to search for the nearest undeleted ancestor node. When the ancestor node It satisfies the no-collision constraint with its child nodes, and the number of child nodes does not exceed a threshold. When necessary, reattach the child node to the ancestor node. And delete the node ;in, This is the threshold for the number of child nodes. Represents the distance from a node in a tree topology. The most recent ancestor node that has not been deleted.

6. The RRT path planning method according to claim 5, characterized in that, Step S3, which involves constructing the reinforcement learning decision network, specifically includes the following steps: Establish a discrete Soft Actor-Critic network structure, including a policy network Actor and two state-action value networks Q1 and Q2, and their corresponding target networks Q1_target and Q2_target, wherein: ， ， in, Represents the state feature vector; For discrete actions; This represents the normalized exponential function; It is a strategy distribution; For the parameterized functions of the policy network Actor, For the policy network Actor; For the first The Q-network estimates the value of actions. For the first Network parameters of a Q-network; The policy network Actor and the dual-Q networks Q1 and Q2 are updated independently using the Adam optimizer, and the soft update coefficient of the target network is set. Update the parameters according to the formula: ， in, The parameters of the current policy network (Actor network) are as follows: The parameters of the target policy network (Actor network); Based on sampling experience Perform Q-value target calculation: ， in, For instant rewards; Temperature coefficient; This is a termination marker; Discount factor; This is the state feature vector for the next state. Discrete actions for the next state; Update the dual-Q network and the Actor network respectively by minimizing the following loss functions: ， ， in The loss function is the loss function for Q-networks; Let Actor be the optimization objective function for the policy network. This represents the mathematical expectation of the distribution of state-action samples obtained from the experience replay pool.

7. The RRT path planning method according to claim 6, characterized in that, In step S3, the reinforcement learning decision network introduces an expansion head and a pruning head. The expansion head is used to adjust the input flow in a given state. The output node extends the action, which includes at least the selection of the extension direction and the distance of the step, guiding the wheeled mobile robot to extend the direction and distance of the next geometric path in the two-dimensional plane; the trimming head is used to adjust the extension direction and distance of the next geometric path given a state. The output is a pruning execution strategy related to the police-thief mechanism, used to determine whether to perform capture deletion or bridging operations on nodes in a thief state or with redundant structures, in order to control the size of the search tree and the quality of the path; the dual-Q network Q1 and Q2 perform long-term reward evaluation on the joint decision generated by the expansion head and the pruning head.

8. The RRT path planning method according to claim 7, characterized in that, Step S4, which updates the reinforcement learning network parameters based on the reward shaping function, specifically includes the following steps: Define a reward shaping function based on path improvement and node pruning. It consists of path shortening reward, redundancy penalty, step penalty, and success reward, and the calculation formula is as follows: ， in , , and All are adjustable weight parameters. This indicates the improvement in path length compared to the previous plan. The penalty represents the number of nodes that are deleted or captured. A slight penalty for each step of expansion. This represents the global reward value when the plan is successful; Set aggressive reward shaping parameters during the training initialization phase and apply them during the failure round. Negative penalties are applied by dynamically adjusting the sampling quantity and evaluation interval through reinforcement learning. After each node expansion and police-thief interaction, the state transition sample is... Store the data in the experience replay pool and update the network parameters by batch sampling according to the set parameters. If the number of samples in the experience pool reaches the threshold If the optimal path does not improve in multiple iterations, the path planning process is considered to have converged.

9. An electronic device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 8.

Images