Virtual navigator path planning method based on SAC-RRT*

Abstract

The invention provides a virtual navigator path planning method based on SAC-RRT *, which innovatively introduces formation geometric envelope parameters to map multi-agent entity constraints into virtual navigator geometric constraints, replaces traditional RRT uniform sampling with non-deterministic Gaussian sampling of an SAC algorithm, and realizes adaptive step length extension and formation-level collision detection in combination with a Critic network. Meanwhile, the path cost optimization quantity of RRT rewiring is creatively converted into rewards to be fed back to the SAC network to complete backward fusion, network training is enhanced in cooperation with a real and virtual dual-experience playback strategy, formation trafficability factors are fused in a state space, a multi-dimensional reward function optimization decision is designed, and finally, the formation trafficability is improved through three times of B spline curve smoothing processing. The multi-agent formation path planning considering safety, optimality and convergence efficiency is realized, and the problems that a traditional method ignores formation geometric constraints, sampling blindness is large, and reinforcement learning and sampling algorithm fusion is not deep are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a virtual navigator path planning method based on SAC-RRT*. Background Technology

[0002] Multi-agent formation control is widely used in scenarios such as drone swarm performances, joint search and rescue operations by unmanned surface vessels (USVs) at sea, and warehousing and logistics by ground mobile robots. Currently, multi-agent formation control architectures mainly include various forms such as leader-follower, behavior-based, and virtual structure / virtual leader.

[0003] In this model, the virtual navigator guides the macroscopic movement of the entire formation by planning a computational reference point (i.e., the virtual navigator) that does not have a physical entity. Follower agents maintain their formation by maintaining their relative geometric position to this reference point. Compared to the physical navigator model, the virtual navigator architecture effectively decouples "global path planning" from "local formation maintenance" and will not cause the entire formation to be paralyzed due to the failure of a single physical navigator. Therefore, it has unique advantages in navigation in complex environments.

[0004] However, existing path planning methods for virtual navigators or formation reference points have the following main drawbacks: 1. Ignoring formation geometry constraints: Traditional algorithms such as RRT and A* typically treat the planning object as a point mass. However, while the virtual leader is virtual, it represents a physical formation with a specific geometric envelope. Narrow gaps that a point mass can pass through may not be passable by a physical formation. Directly applying single-unit planning algorithms can easily lead to collisions between followers at the formation's edge and the environment, or force the formation to disintegrate when passing through narrow areas.

[0005] 2. Blindness of RRT* algorithm: The traditional RRT* algorithm performs uniform random sampling in the entire state space, which leads to an extremely high invalid sampling rate in complex mazes or narrow passage environments, resulting in huge time costs for converging to the optimal path.

[0006] 3. Insufficient integration of reinforcement learning and sampling algorithms: Existing integration methods (such as TD3-RRT) mostly use the deterministic actions output by the reinforcement learning network to replace random sampling. Although this improves directionality, it loses the probabilistic completeness of the RRT algorithm and fails to make full use of the unique rewire mechanism of RRT* to back-optimize the reinforcement learning network, making it difficult for the network to learn global topological features. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a virtual navigator path planning method based on SAC-RRT*, which solves the problems of poor trafficability caused by the virtual navigator path planning ignoring the formation geometry, slow convergence speed and high blindness of traditional RRT algorithms in complex environments, and failure of existing deep reinforcement learning combined methods to make full use of rewiring mechanisms for feedback optimization.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a virtual navigator path planning method based on SAC-RRT*, comprising the following steps: Step 1: Construct a multi-agent formation virtual navigator-follower system architecture, calculate the formation geometric envelope parameters based on the formation, and model the path planning problem as a Markov decision process; Step 2: Construct an action policy network based on the soft actor-critic SAC algorithm. The action policy network includes an Actor policy network and a Critic evaluation network, which are used to output nondeterministic Gaussian sampling distribution parameters. Step 3: Set the starting point and target point of the virtual navigator, and initialize the Rapid Expanding Random Tree (RRT); Step 4: Perform the RRT node sampling fusion step guided by the SAC strategy, and use the distribution output by the Actor strategy network to replace the uniform random sampling of the traditional RRT to generate exploratory guide points. Step 5: Execute the adaptive step-size expansion step based on Critic value evaluation. Use the Critic evaluation network to dynamically adjust the expansion step size based on the value score of the current action, and perform collision detection based on the formation geometric envelope. Step 6: Execute the policy reinforcement step based on rerouting feedback, calculate the optimization amount of path cost in the RRT* rerouting operation, and convert it into rerouting reward feedback to the SAC network to complete the backward fusion of the algorithm. Step 7: Use a dual experience replay strategy that includes both real and virtual experience pools to perform offline training and parameter updates for the action policy network; Step 8: Determine whether the target point has been reached. If so, backtrack to generate a set of path nodes and smooth the path using a cubic B-spline curve. Output the path as a navigation reference trajectory for the multi-agent formation.

[0009] Preferably, the method for calculating the formation geometric envelope parameters in Step 1 is as follows: ; in, The maximum outer tangent radius of the formation. For the number of agents, For the first The ideal position coordinates of each agent relative to the navigator The virtual navigator's location coordinates, For the physical radius of a single agent, For safety redundancy distance; this parameter is used to map the entity constraints of a multi-agent formation to the geometric constraints of a virtual navigator.

[0010] Preferably, the state space of the Markov decision process in Step 1 is... Defined as: ; in, This represents the normalized Euclidean distance between the current position and the target point; This indicates the azimuth of the target point relative to the current heading; This represents the current velocity vector of the virtual navigator. This indicates the data returned by the multi-beam forward-looking detection sensor. Obstacle distance vectors in each direction; The formation passability factor is used to characterize whether the local space can accommodate a formation, and its calculation formula is as follows: in The width of continuous free space detected within the forward-looking detection sector; when At that time, the state space indicates that the current direction is impassable.

[0011] Preferably, the construction of the action policy network and the action sampling process in Step 2 include: 1) Define the action space of the SAC network ,in This is the sampling angle offset. This is the step size adjustment factor; 2) Gaussian distribution parameters of the Actor network output actions: mean and logarithmic standard deviation ; 3) Motion sampling is performed using the reparameterization trick, calculated using the following formula: , ; in For noise variables sampled from a standard normal distribution, These are the parameters of the Actor network; 4) Objective function of SAC algorithm It includes a term that maximizes entropy, defined as: ; in For temperature coefficient, This is the strategy entropy, used to enhance the algorithm's exploration capabilities in narrow formation channels.

[0012] Preferably, the specific calculation formula for the RRT node sampling and fusion step based on the SAC strategy in Step 4 is as follows: ; in For the generated guide sampling points, The node in the tree that is closest to the target. Based on the search radius, For nodes The heading angle, This step generates the action components output by the SAC network; it replaces the uniform sampling function in the RRT* algorithm with the probability distribution of the reinforcement learning output. .

[0013] Preferably, the adaptive step-size expansion step based on Critic value assessment in Step 5 includes: 1) Input the current state s and the SAC output action a into the Critic network, and obtain the minimum of the two Q values. The calculation formula is as follows: ; 2) Calculate the adaptive expansion step size The calculation formula is: ; in This is the base step size for RRT. This is the step size gain coefficient; 3) Based on step size Generate new nodes And perform collision detection based on the formation geometry envelope: if If so, the node is determined to be invalid. , which is the set of obstacle coordinates in the environment. This detection ensures that the generated path satisfies the physical passage constraints of the formation.

[0014] Preferably, the policy reinforcement step based on rewiring feedback in Step 6 specifically includes: 1) In After adding to the tree, search the neighborhood radius. The set of nodes inside ; 2) Traversal ,like Then a rewiring operation will be performed. The parent node is updated to ; 3) Calculate the cumulative path optimization amount for a single expansion. ; 4) Calculate rewiring rewards And it feeds back to the SAC network, the calculation formula is: ; in, This is the reward scaling factor.

[0015] Preferably, the total reward function of the Markov decision process in Step 1 is... Defined as: ; in, Rewards are derived from goals; As a collision penalty, when a collision is detected Triggered when invalid; To smooth out rewards, penalties involve sharp turns. The rewiring reward as described in claim 7; The formula for calculating the formation safety constraint penalty is as follows: When the detected obstacle approaches the formation radius At that time, an exponential penalty is imposed, forcing the SAC network to learn and output a safe path located in the center of an open area.

[0016] Preferably, the dual experience replay strategy in step 7 includes: constructing a pool of real experience. Storing actual interaction data ; Construct a virtual experience pool The hindsight experience replay (HER) strategy is adopted; for formation size limitations, The path that caused the expansion failure is selected as the virtual target by choosing the node state before the blockage. Recalculate the reward and virtual experience deposit , used to train the network to recognize formation geometric traps.

[0017] Preferably, the B-spline smoothing process in step tep8 uses a cubic B-spline curve model: in For path control points, For the basis functions, their expressions of each order are as follows: .

[0018] This invention provides a virtual navigator path planning method based on SAC-RRT*, which has the following beneficial effects: 1. Ensure safe passage of the formation. By calculating the geometric envelope parameters of the formation, the constraints of the multi-agent entities are mapped to the geometric constraints of the virtual navigator. Combined with the formation passability factor, the feasibility of passage in the local space is clearly indicated. With the formation safety constraint penalty mechanism, it is ensured that the planned path always reserves sufficient safety space for the formation, and completely solves the problem of edge collision or formation disintegration caused by the traditional algorithm ignoring the geometric size of the formation.

[0019] 2. Improve path planning efficiency by replacing the uniform random sampling of traditional RRT with nondeterministic Gaussian sampling output by SAC algorithm. Combined with entropy regularization objective, enhance the ability to explore narrow channels and significantly reduce the invalid sampling rate in complex environments. At the same time, dynamically adjust the expansion step size through Critic network value evaluation. In open environments, increase the step size to accelerate convergence and significantly improve the algorithm's convergence speed.

[0020] 3. Optimize the global optimality of the path by making full use of the RRT* rewiring mechanism to transform the path cost optimization into a reward feedback to the SAC network. Construct a two-way fusion mechanism of planning and decision-making, so that the SAC network can not only learn the strategy to reach the goal, but also master the global topological features of the optimized path structure, thus solving the problem of insufficient integration between existing reinforcement learning and sampling algorithms.

[0021] 4. Enhance environmental adaptability by designing a dual experience replay strategy with real and virtual experience pools. The real experience pool stores effective interaction data, while the virtual experience pool transforms failed paths into virtual experiences through hindsight experience replay, helping the network identify geometric traps in formation and significantly improving its adaptability and generalization performance in complex obstacle environments.

[0022] 5. To ensure path smoothness and kinematic adaptation, a multi-dimensional reward function is used to penalize large-angle turns, and a cubic B-spline curve is used to smooth the backtracking path, generating a continuous and smooth trajectory. This satisfies the kinematic continuity requirements of multi-agent formation and avoids the operational risks caused by sharp turns.

[0023] 6. It has a wide range of applications and can be effectively used in various multi-agent formation operation scenarios such as drone swarm performances, joint search and rescue of unmanned surface vessels at sea, and warehousing and logistics of ground mobile robots, providing safe, efficient and optimal navigation reference trajectories for formation navigation in different fields. Attached Figure Description

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the geometric envelope calculation of the present invention; Figure 3This is a schematic diagram of the state space and multi-beam forward-looking detection principle of the present invention, which includes formation passability factors; Figure 4 This is a schematic diagram illustrating the principle of SAC policy network-guided RRT* node expansion and policy reinforcement based on rewiring feedback in this invention. Figure 5 This is a simulation result of the smooth reference trajectory generated by the present invention in a complex obstacle environment. Detailed Implementation

[0025] This invention proposes a virtual navigator path planning method based on SAC-RRT*. The overall idea is as follows: For multi-agent formations, the geometric constraints of the entities are abstracted into the "formation geometric envelope" of the virtual navigator; the exploration capability of maximum entropy deep reinforcement learning (SAC) is used to guide the growth direction of the RRT tree in the limited feasible space; the rewire benefit of RRT* is innovatively fed back to the SAC network to realize the reverse optimization of the decision layer by the planning layer, and finally generate a reference trajectory that satisfies both global optimality and ensures the overall safety of the formation.

[0026] like Figure 1 As shown, the present invention specifically includes the following steps: 1. Construct the system architecture and geometric envelope model 1) Constructing a Virtual Leader-Follower System Architecture. This embodiment adopts a Virtual Leader architecture. The system includes... There is a physical follower agent (such as a drone or unmanned boat) and a logical virtual navigator. The virtual navigator is responsible for perceiving the global environment and planning a reference path, while the follower is responsible for maintaining its relative geometric position to the virtual navigator.

[0027] 2) such as Figure 2 As shown, the formation geometric envelope parameters are calculated to map the entity constraints of the multi-agent system to the planning space of the virtual navigator. The maximum external tangent radius of the formation is defined. Let the virtual navigator be located at the origin of the local coordinate system, and let the first... The ideal relative position coordinates of each follower are: ( The virtual navigator's location is... The formation geometric envelope parameters are calculated using the following formula: ; in: The physical radius of a single agent (considering the safety boundary during obstacle avoidance); The safe redundancy distance required for formation maneuvers. Physical meaning: This parameter... It defines the minimum clearance radius that the virtual navigator must maintain when planning a path to ensure that the formation does not collide with the environment when rotating or translating.

[0028] 3) Construct a Markov Decision Process (MDP) to model the path planning problem as MDP tuples. .

[0029] 2. Define the state space The expression is: ; In the formula, This is the normalized Euclidean distance between the current node position and the target point position, with a value range of [0,1]. The azimuth angle of the target point relative to the current virtual navigator's heading, with a value range of [−π,π]. This represents the current normalized velocity vector of the virtual navigator. This is the obstacle distance vector (e.g., k=15) returned by the multibeam forward-looking detection sensor in k directions. For formation passability factors. For example... Figure 4 As shown, this feature is used to explicitly describe whether the local space in front can accommodate the formation to pass through, and the calculation formula is: ; in, This refers to the width of the continuous free space detected within the forward-looking detection sector. The decision logic is as follows: when... When this occurs, it indicates that the width of the passage ahead is less than the formation diameter, and the state space indicates that the current direction is impassable, forcing the SAC network to adjust its sampling strategy.

[0030] 3. Construct the SAC action policy network Construct an action policy network based on the soft actor-critic (SAC) algorithm, including an actor policy network and a critic evaluation network.

[0031] 1) Define the action space The expression is: ; in, This is the angle offset for guiding sampling. This is the adjustment factor for expanding the step size.

[0032] 2) Network output and sampling Actor network input state Gaussian distribution parameters of the output action: mean and logarithmic standard deviation Motion sampling is performed using reparameterization techniques: ; in This is standard normally distributed noise.

[0033] 3) Entropy Regularization Objective: The objective function of the SAC algorithm Including a term that maximizes entropy to enhance exploration capabilities in narrow passages, the expression is: ; in, For temperature coefficient, Let be the policy entropy.

[0034] 4. Initialize the RRT* tree Set the starting point for the virtual navigator With the target point Initialize a fast-expanding random tree ,Will Add it to the tree as the root node.

[0035] 5. Perform RRT* node sampling fusion guided by the SAC strategy, such as... Figure 3 As shown, Repeat the following steps until a path is found or the maximum number of iterations is reached. Guided sampling using a SAC network: 1) Find the node closest to the target point in the current RRT* tree. .

[0036] 2) Extraction state vector Input the Actor network and sample the actions. .

[0037] 3) Calculate the guiding sampling points The expression is: ; in, Based on the search radius, For nodes The heading angle. Technical effect: This step replaces the uniform random sampling of traditional RRT* with the probability distribution of reinforcement learning output, giving the sampling clear target guidance and obstacle avoidance intelligence.

[0038] 6. Perform adaptive step-size expansion and collision detection based on Critic value evaluation to expand nodes: 1) Adaptive step size calculation: Input the state s and action a into the Critic network to obtain the minimum Q value. The adaptive expansion step size is obtained by mapping using the Sigmoid function. The expression is: ; in Based on step size, This is the gain coefficient. When the Q value is high (the environment is open and the direction is correct), the step size is increased to accelerate convergence.

[0039] 2) New node generation and formation collision detection based on step size Generate new nodes Perform collision detection based on formation geometry envelope: calculate To the nearest obstacle in the environment The distance. If the following conditions are met: ; If the node is deemed invalid (even though the navigator did not collide, the formation edge will), it is abandoned and not added to the tree; otherwise, the node is deemed valid and added to the tree.

[0040] 7. Implement policy reinforcement based on rerouting feedback, perform RRT*-specific rerouting operations, and calculate feedback rewards: 1) Rewire: In After adding to the tree, search the set of nodes within the neighborhood radius. Traversal If through arrive Path cost Then update The parent node is .

[0041] 2) Calculate the path optimization amount, expressed as: ; 3) Feedback Rewards: Rebate for Computational Rewiring This information is then fed back to the SAC network. Technical effect: This achieves backward fusion of the algorithm, enabling the SAC network to not only learn "reaching the goal" but also "optimizing the tree structure." Reward function definition: Total reward function. Defined as: ; in, Formation safety constraints and penalties:

[0042] When the detected obstacle approaches the formation radius At that time, an exponential penalty is imposed, forcing the SAC network to learn and output a safe path located in the center of an open area.

[0043] 8. Dual experience replay and network update: Training is performed using a dual experience replay strategy: Real experience pool : Stores the actual generated valid path data. Virtual experience pool For formation size limitations (i.e.) The path that caused the expansion to fail is selected as the virtual target before the blockage. Recalculate the reward and deposit it. This allows the network to learn local navigation strategies from "dead end" experiences.

[0044] 9. Path backtracking and smoothing: After a node reaches the target area, backtracking generates a set of path nodes. The path is smoothed using a cubic B-spline curve model, expressed as: ; Wherein, basis functions The expression is: ; ; ; The final output is a navigation reference trajectory that satisfies the kinematic continuity constraints of multi-agent formation.

[0045] To verify the effectiveness of the method of the present invention, a specific simulation example is constructed for illustration.

[0046] 1. Simulation Environment and Parameter Settings 1) Map scene: A 1000m×1000m sea environment, containing several circular obstacles and a narrow passage with a width of 120m.

[0047] 2) Task settings: Start=(50,50), Goal=(900,900).

[0048] 3) Formation configuration: A triangular formation is adopted, which includes 1 virtual navigator and 2 followers.

[0049] 4) Relative position of follower 1: (-30m, 30m) 5) Relative position of follower 2: (-30m, -30m) 6) Individual radius

[0050] 7) Safety margin

[0051] 8) Algorithm parameters: search radius Basic step size Gain coefficient (Maximum step length 100m).

[0052] 2. Calculation process of key steps (taking the first step of expansion as an example) 1) Calculate the geometric envelope of the formation, expressed as:

[0053] Among them, the minimum safe passage width for convoy passage must be greater than In the scene The passage meets the passage requirements.

[0054] 2. State Awareness and SAC Decision Assumption: The current virtual navigator is located in... , directly ahead Unobstructed, but on the left side There are obstacles there.

[0055] 3. Calculate the passability factor: The status indicator is "passable".

[0056] 4. SAC network input state, output action distribution, sampled to obtain .

[0057] 5. Guide point generation assuming current heading ( ).

[0058]

[0059] 6. Adaptive step size and collision detection, Critic network scoring The Sigmoid mapping value is approximately 0.95. Calculation step size: Generate a new node The distance to the nearest obstacle (on the left) is [distance missing]. . judge: The node is valid.

[0060] 7. Although the calculation of the formation constraint penalty is valid for the node, it is too close to the formation boundary (only the remaining nodes are valid). Calculate the penalty: This negative reward will be fed back to the SAC network, causing it to tend to guide sampling to the right (away from obstacles) in subsequent iterations, thereby achieving proactive defensive planning.

[0061] 8. Rewiring Feedback Assumption The addition of a node in the tree makes a certain node The path cost was reduced Calculate rewiring rewards: Through iterative steps, the route is eventually generated that maintains a minimum distance from obstacles throughout. The reference trajectory, which is distanced and smoothed by B-splines, ensures the safe and efficient passage of multi-agent formations in complex environments.

[0062] like Figure 5 As shown, the figure compares the path planned by the present invention with that of the traditional algorithm in a complex obstacle environment. The starting point, the ending point, and the distributed obstacles are clearly marked in the figure. The path planned by the present invention is safe and centered, maintains a sufficient safe distance from obstacles throughout the entire process, and has a smooth and continuous trajectory. In contrast, the path of the traditional algorithm is close to the obstacles, which poses a danger to passage. This intuitively demonstrates the significant advantages of the present invention in satisfying the geometric constraints of multi-agent formation and ensuring the safety and smoothness of the path, and verifies its high efficiency and adaptability in complex environments.

[0063] This invention provides a virtual navigator path planning method based on SAC-RRT*. It innovatively introduces formation geometric envelope parameters to map multi-agent entity constraints to virtual navigator geometric constraints. It replaces traditional RRT uniform sampling with nondeterministic Gaussian sampling of the SAC algorithm, and combines a Critic network to achieve adaptive step size expansion and formation-level collision detection. Furthermore, it innovatively transforms the path cost optimization of RRT rewiring into reward feedback to the SAC network for post-fusion. A dual-experience replay strategy (real and virtual) is used to strengthen network training. Formation passability factors are incorporated into the state space, and a multi-dimensional reward function is designed to optimize decision-making. Finally, through 3rd-order B-spline curve smoothing, it achieves multi-agent formation path planning that balances safety, optimality, and convergence efficiency. This effectively solves the problems of traditional methods ignoring formation geometric constraints, high sampling blindness, and insufficient integration of reinforcement learning and sampling algorithms.

[0064] The above embodiments are merely preferred technical solutions of the present invention and should not be considered as limitations on the present invention. The scope of protection of the present invention should be defined as the technical solutions described in the claims, including equivalent substitutions of the technical features described in the claims. That is, equivalent substitutions and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. A virtual navigator path planning method based on SAC-RRT*, characterized in that, Includes the following steps: Step 1: Construct a multi-agent formation virtual navigator-follower system architecture, calculate the formation geometric envelope parameters based on the formation, and model the path planning problem as a Markov decision process; Step 2: Construct an action policy network based on the soft actor-critic SAC algorithm. The action policy network includes an Actor policy network and a Critic evaluation network, which are used to output nondeterministic Gaussian sampling distribution parameters. Step 3: Set the starting point and target point of the virtual navigator, and initialize the Rapid Expanding Random Tree (RRT); Step 4: Perform the RRT node sampling fusion step guided by the SAC strategy, and use the distribution output by the Actor strategy network to replace the uniform random sampling of the traditional RRT to generate exploratory guide points. Step 5: Execute the adaptive step-size expansion step based on Critic value evaluation. Use the Critic evaluation network to dynamically adjust the expansion step size based on the value score of the current action, and perform collision detection based on the formation geometric envelope. Step 6: Execute the policy reinforcement step based on rerouting feedback, calculate the optimization amount of path cost in the RRT* rerouting operation, and convert it into rerouting reward feedback to the SAC network to complete the backward fusion of the algorithm. Step 7: Use a dual experience replay strategy that includes both real and virtual experience pools to perform offline training and parameter updates for the action policy network; Step 8: Determine whether the target point has been reached. If so, backtrack to generate a set of path nodes and smooth the path using a cubic B-spline curve. Output the path as a navigation reference trajectory for the multi-agent formation.

2. The virtual navigator path planning method based on SAC-RRT* according to claim 1, characterized in that, The method for calculating the formation geometric envelope parameters in Step 1 is as follows: ； in, The maximum outer tangent radius of the formation. For the number of agents, For the first The ideal position coordinates of each agent relative to the navigator The virtual navigator's location coordinates, For the physical radius of a single agent, For safety redundancy distance; this parameter is used to map the entity constraints of a multi-agent formation to the geometric constraints of a virtual navigator.

3. The virtual navigator path planning method based on SAC-RRT* according to claim 1, characterized in that, The state space of the Markov decision process in Step 1 Defined as: ； in, This represents the normalized Euclidean distance between the current position and the target point; This indicates the azimuth of the target point relative to the current heading; This represents the current velocity vector of the virtual navigator. This indicates the data returned by the multi-beam forward-looking detection sensor. Obstacle distance vectors in each direction; The formation passability factor is used to characterize whether the local space can accommodate a formation, and its calculation formula is as follows: in The width of continuous free space detected within the forward-looking detection sector; when At that time, the state space indicates that the current direction is impassable.

4. The virtual navigator path planning method based on SAC-RRT* according to claim 1, characterized in that, The construction of the action policy network and the action sampling process in Step 2 include: 1) Define the action space of the SAC network ,in This is the sampling angle offset. This is the step size adjustment factor; 2) Gaussian distribution parameters of the Actor network output actions: mean and logarithmic standard deviation ; 3) Motion sampling is performed using the reparameterization trick, calculated using the following formula: , ； in For noise variables sampled from a standard normal distribution, These are the parameters of the Actor network; 4) Objective function of SAC algorithm It includes a term that maximizes entropy, defined as: ； in For temperature coefficient, This is the strategy entropy, used to enhance the algorithm's exploration capabilities in narrow formation channels.

5. The virtual navigator path planning method based on SAC-RRT* according to claim 1, characterized in that, The specific calculation formula for the RRT node sampling and fusion step based on the SAC strategy in Step 4 is as follows: ； in For the generated guide sampling points, The node in the tree that is closest to the target. Based on the search radius, For nodes The heading angle, This step generates the action components output by the SAC network; it replaces the uniform sampling function in the RRT* algorithm with the probability distribution of the reinforcement learning output. .

6. The virtual navigator path planning method based on SAC-RRT* as described in claim 1, characterized in that, The adaptive step-size expansion step based on Critic value assessment in Step 5 includes: 1) Input the current state s and the SAC output action a into the Critic network, and obtain the minimum of the two Q values. The calculation formula is as follows: ； 2) Calculate the adaptive expansion step size The calculation formula is: ； in This is the base step size for RRT. This is the step size gain coefficient; 3) Based on step size Generate new nodes And perform collision detection based on the formation geometry envelope: if If so, the node is determined to be invalid. , which is the set of obstacle coordinates in the environment. This detection ensures that the generated path satisfies the physical passage constraints of the formation.

7. The virtual navigator path planning method based on SAC-RRT* according to claim 1, characterized in that, The policy reinforcement step based on rewiring feedback in Step 6 specifically includes: 1) In After adding to the tree, search the neighborhood radius. The set of nodes inside ; 2) Traversal ,like Then a rewiring operation will be performed. The parent node is updated to ; 3) Calculate the cumulative path optimization amount for a single expansion. ; 4) Calculate rewiring rewards And it feeds back to the SAC network, the calculation formula is: ； in, This is the reward scaling factor.

8. The virtual navigator path planning method based on SAC-RRT* according to claim 3, characterized in that, The total reward function of the Markov decision process in Step 1 Defined as: ； in, Rewards are derived from goals; As a collision penalty, when a collision is detected Triggered when invalid; To smooth out rewards, penalties involve sharp turns. The rewiring reward as described in claim 7; The formula for calculating the formation safety constraint penalty is as follows: When the detected obstacle approaches the formation radius At that time, an exponential penalty is imposed, forcing the SAC network to learn and output a safe path located in the center of an open area.

9. The virtual navigator path planning method based on SAC-RRT* according to claim 1, characterized in that, The dual-experience replay strategy in Step 7 includes: constructing a pool of real experiences. Storing actual interaction data ; Construct a virtual experience pool A hindsight experience replay strategy is adopted; for formation size limitations, The path that caused the expansion failure is selected as the virtual target by choosing the node state before the blockage. Recalculate the reward and virtual experience deposit , used to train the network to recognize formation geometric traps.

10. The virtual navigator path planning method based on SAC-RRT* according to claim 1, characterized in that, The B-spline smoothing process described in step tep8 uses a cubic B-spline curve model. in For path control points, For the basis functions, their expressions of each order are as follows: .

Images