An AUV path planning and obstacle avoidance method based on improved SAC algorithm

Abstract

The application belongs to the technical field of underwater robot control, and relates to an AUV path planning and obstacle avoidance method based on an improved SAC algorithm, comprising the following steps: constructing an environment interaction model of an autonomous underwater vehicle; constructing an MSR-SAC deep reinforcement learning network; updating a value network target by using a multi-step return mechanism MSR according to the environment interaction model, calculating a comprehensive reward, updating parameters of the MSR-SAC deep reinforcement learning network until convergence is achieved, and obtaining a trained strategy network; inputting current sensor data into the trained strategy network, outputting a thrust and rudder angle control instruction, and generating a collision-free trajectory, wherein a tangent sub-target planner is introduced in the training and execution process, a tangent direction is calculated to generate a tangent sub-target to guide the AUV to pass around an obstacle. Through the determinacy guidance provided by the TSP and the long-term value evaluation of the MSR mechanism, the obstacle avoidance success rate, the convergence speed and the trajectory smoothness of the AUV in an unknown environment are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of underwater robot control technology, and in particular to an AUV path planning and obstacle avoidance method based on an improved SAC algorithm. Background Technology

[0002] Autonomous underwater vehicles (AUVs) are crucial equipment in marine scientific exploration, oil and gas development, and maritime rescue. Their autonomy primarily relies on efficient and reliable trajectory planning and obstacle avoidance algorithms. Traditional path planning methods based on environmental modeling often decouple perception, planning, and control, making it difficult to cope with sensor noise, ocean current interference, and unknown obstacles in complex marine environments. Furthermore, many reinforcement learning-based algorithms only utilize single-step rewards for policy updates, lacking consideration for long-term decision-making and prone to short-sighted behavior, resulting in lengthy obstacle avoidance paths or unstable training.

[0003] Due to the dynamic nature of unknown underwater environments and the need for long-term decision-making, existing methods struggle to balance training efficiency, long-term planning capabilities, and obstacle avoidance robustness. Therefore, it is necessary to propose an AUV path planning and obstacle avoidance method based on an improved SAC algorithm to enhance the algorithm's generalization performance in unknown environments. Summary of the Invention

[0004] The purpose of this invention is to provide an AUV path planning and obstacle avoidance method based on an improved SAC algorithm. It introduces a multi-step reward mechanism, a dynamic step size strategy, and a sub-target planning based on sonar perception within a reinforcement learning framework, taking into account energy consumption, smoothness, and target alignment. It improves the convergence speed through dense rewards, thereby enabling efficient autonomous navigation of AUVs in unknown underwater environments.

[0005] To achieve the above objectives, the present invention provides the following solution: An AUV path planning and obstacle avoidance method based on an improved SAC algorithm includes: Construct an environmental interaction model for an autonomous underwater vehicle (AUV); Construct an MSR-SAC deep reinforcement learning network, wherein the MSR-SAC deep reinforcement learning network is obtained by introducing a multi-step reward mechanism MSR into the SAC deep reinforcement learning network; The value network target is updated using the multi-step reward mechanism (MSR) according to the environmental interaction model, and the comprehensive reward is calculated. Data is stored in the experience replay pool, and the parameters of the MSR-SAC deep reinforcement learning network are updated until convergence, thereby obtaining the trained policy network. The comprehensive reward is calculated based on the motion state of the AUV, its relative relationship with the target, and energy consumption constraints. The current sensor data is input into the trained policy network, which outputs thrust and rudder angle control commands to generate a collision-free trajectory. During training and execution, a tangential sub-target planner is introduced to calculate the tangential direction and generate tangential sub-targets to guide the AUV around obstacles.

[0006] Optionally, constructing an environmental interaction model for an autonomous underwater vehicle (AUV) includes: The state space is constructed based on the normalized distance measurements of the obstacle avoidance sonar in several directions, the relative distance between the AUV and the target, the angle between the target direction and the current heading, and the longitudinal and yaw angular velocities. The action space is constructed based on the forward speed command and the yaw angular velocity command to obtain the environmental interaction model.

[0007] Optionally, calculating the tangential direction to generate tangential sub-targets to guide the AUV around obstacles includes: The system uses sonar to detect obstacle obstruction in real time. When obstruction is detected, the system calculates the tangent sub-target point based on the obstacle edge information and uses it as a temporary navigation target. The AUV then arrives at the temporary navigation target before proceeding to the target point.

[0008] Optionally, when occlusion is detected, calculating the tangent sub-target point based on obstacle edge information includes: When an obstruction is detected, edge detection is performed. When the distance difference between adjacent sonar beams that are obstructed exceeds a threshold, it is determined to be an obstacle boundary, i.e., a sudden change is detected. For each pair of adjacent sonar beams that detect a mutation, the obstacle edge direction is estimated based on azimuth and distance data interpolation, and a set of candidate edge directions is generated. The minimum deviation angle is selected from the candidate edge direction set as the obstacle bypass direction, and the obstacle edge abrupt change point corresponding to the obstacle bypass direction is obtained; Draw a circle with a preset radius centered at the abrupt change point on the edge of the obstacle, and calculate the two external tangent lines and the point of tangency from the current position of the AUV to the circle; Calculate the angle between the tangent direction and the obstacle direction based on the tangent point to obtain the tangent sub-target.

[0009] Optionally, after obtaining the tangent sub-target, if the tangent sub-target is located inside an obstacle, then jump to the next candidate edge and recalculate.

[0010] Optionally, updating the value network objective according to the multi-step reward mechanism MSR includes: The multi-step reward mechanism MSR adaptively adjusts the step size parameter of the multi-step reward based on the distance between the AUV and the nearest obstacle, updates the value network target using multi-step Bellman reward based on the step size parameter, and accumulates the reward only up to the final step through a truncation mechanism.

[0011] Optional, the step size parameter n for adaptively adjusting multi-step rewards includes: ; in, The distance to the nearest obstacle. The ranging value of the i-th azimuth angle of the obstacle avoidance sonar. Distance is the threshold for risk assessment.

[0012] Optionally, the comprehensive reward r is calculated as: ; in, To terminate the event reward, For distance-based rewards, Penalties for posture alignment mechanism Forward speed reward incentive, Due to multi-dimensional physical limitations and energy consumption penalties, This is a near-field obstacle avoidance penalty item. This is an ocean current synergy term.

[0013] The beneficial effects of this invention are as follows: The MSR-SAC deep reinforcement learning network proposed in this invention is used to solve the trajectory planning problem of autonomous underwater vehicles in unknown underwater environments. This invention enhances long-term decision-making capabilities in complex obstacle avoidance scenarios through an adaptive multi-step reward mechanism and trajectory truncation strategy. The introduction of a TSP (Tangent Subgoal Planner) module provides guidance for local paths, effectively reducing redundant actions in obstacle avoidance paths. The designed composite reward function integrates multi-dimensional objectives such as task completion, path optimality, motion smoothness, energy efficiency optimization, and safe obstacle avoidance, ensuring stable and efficient training while guaranteeing that the final strategy is both physically feasible and interpretable. Simulation experiments show that MSR-SAC outperforms mainstream algorithms such as DDPG, TD3, and SAC in terms of convergence speed, final reward, policy stability, and anti-interference ability. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of an AUV path planning and obstacle avoidance method based on an improved SAC algorithm according to an embodiment of the present invention. Figure 2This is a schematic diagram illustrating the principle of TSP using obstacle avoidance sonar data to select sub-target points according to an embodiment of the present invention; Figure 3 This is a comparison chart of the convergence curves of the MSR-SAC algorithm and the benchmark algorithm in this embodiment of the invention; Figure 4 The trajectory planning test results of the MSR-SAC and the benchmark algorithm in this embodiment of the invention are shown in the figure. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] like Figure 1 As shown, this embodiment proposes an AUV path planning and obstacle avoidance method based on an improved SAC algorithm, including: Construct an environmental interaction model for an autonomous underwater vehicle (AUV); Construct an MSR-SAC deep reinforcement learning network, wherein the MSR-SAC deep reinforcement learning network is obtained by introducing a multi-step reward mechanism MSR into the SAC deep reinforcement learning network; The multi-step reward mechanism (MSR) is used to update the target of the Critic network according to the environmental interaction model, and the comprehensive reward is calculated. Data is stored in the experience replay pool, and the parameters of the MSR-SAC deep reinforcement learning network are updated until convergence, and the trained policy (actor) network is obtained. The comprehensive reward is calculated based on the motion state of the AUV, its relative relationship with the target, and energy consumption constraints. The current sensor data is input into the trained policy network, which outputs thrust and rudder angle control commands to generate a collision-free trajectory. The current sensor data includes the normalized ranging vector of the obstacle avoidance sonar, the relative displacement vector between the AUV and the target point, the angle between the target direction and the current heading, and the longitudinal velocity and yaw rate obtained from the inertial navigation system. During training and execution, a tangential sub-target planner is introduced to calculate the tangential direction and generate tangential sub-targets to guide the AUV around obstacles.

[0019] Furthermore, the construction of an environmental interaction model for autonomous underwater vehicles (AUVs) includes: The state space is constructed based on the normalized distance measurements of the obstacle avoidance sonar in several directions, the relative distance between the AUV and the target, the angle between the target direction and the current heading, and the longitudinal and yaw angular velocities. The action space is constructed based on the forward speed command and the yaw angular velocity command to obtain the environmental interaction model.

[0020] Specifically, the state of a three-degree-of-freedom system on a horizontal plane is determined by the velocity vector. With position vector Common description, among which These represent the lateral, longitudinal, and yaw angular velocities, respectively. () represents the position of the AUV in the inertial coordinate system. For heading angle, superscript This is a transpose.

[0021] Kinematic equations: ; In the formula, This is the time derivative of position and heading.

[0022] Where the rotation matrix Defined as: ; Dynamic model: ; in, For the composite inertia matrix, For acceleration, The Coriolis-centrifugal force matrix, Here is the fluid damping matrix. The torque term is the force generated by gravity and buoyancy. To control the input, This refers to environmental disturbances.

[0023] state space : ; in, This indicates that obstacle avoidance sonar is in Normalized distance measurements in each direction The relative distance from the AUV to the target, ( , () represents the coordinates of the target point. The angle between the target direction and the current heading. This is the current heading angle of the AUV, to avoid exist The network input is discontinuous and uses [a specific method / approach]. .

[0024] Action space : ; in, These correspond to forward speed commands and yaw rate commands, respectively.

[0025] Furthermore, calculating the tangential direction to generate tangential sub-targets to guide the AUV around obstacles includes: The system uses sonar to detect obstacle obstruction in real time. When obstruction is detected, a tangent sub-target point is calculated based on the obstacle edge information and used as a temporary navigation target. The AUV then arrives at the temporary navigation target before proceeding to the target point.

[0026] Furthermore, when occlusion is detected, the tangent sub-target point is calculated based on obstacle edge information, including: When an obstruction is detected, edge detection is performed. When the distance difference between adjacent sonar beams that are obstructed exceeds a threshold, it is determined to be an obstacle boundary, i.e., a sudden change is detected. For each pair of adjacent sonar beams that detect a mutation, the obstacle edge direction is estimated based on azimuth and distance data interpolation, and a set of candidate edge directions is generated. Select the minimum deviation angle from the candidate edge direction set as the obstacle bypass direction, and obtain the obstacle edge abrupt change point corresponding to the obstacle bypass direction; Draw a circle with a preset radius centered at the abrupt change point on the obstacle edge, and calculate the two external tangent lines and the point of tangency from the current position of the AUV to the circle; Calculate the angle between the tangent direction and the obstacle direction based on the tangent point to obtain the tangent sub-target.

[0027] Specifically, such as Figure 2 The diagram illustrates the principle of TSP (Tracking Point Service) using obstacle avoidance sonar data to select sub-target points. The specific steps for tangential sub-target planning are as follows: Obstruction detection: Determine the ranging values ​​of adjacent sonar beams in the direction of the target. Is it less than the straight-line distance from the AUV to the target? If the value is less than 0, then occlusion is determined. ; Mutation block extraction: If occlusion is detected, the algorithm performs edge detection. When the adjacent sonar beams of the occlusion are detected... The distance difference exceeds the threshold D2 When the boundary is determined to be an obstacle, this abrupt change represents the transition from unobstructed to near-obstruction. Multiple edges may exist that satisfy the condition.

[0028] ; in, N This refers to the total number of obstacle avoidance sonar rays.

[0029] Edge angle calculation: For each pair of adjacent sonar beams that detect abrupt changes, based on azimuth angle... and With distance data and Interpolation to estimate the direction of the e-th obstacle edge Beam directions with smaller distances from the data are closer to the obstacle edge. Candidate edge directions are sorted in ascending order of their deviation angle from the AUV-target line, ultimately generating a set of candidate edge directions. .

[0030] ; Edge mutation point calculation: from the set of candidate edge directions The minimum deviation angle is selected as the obstacle avoidance direction, and the corresponding abrupt change point at the obstacle edge is... for: ; in, This is the threshold for edge mutation.

[0031] Select the tangential sub-target: After determining the abrupt change point at the obstacle edge, use it as the center of the circle. Drawing radius is Given a circle, calculate the two external tangent lines and the point of tangency from the current position of the AUV to that circle. : ; ; Among them, the vector pointing from the abrupt change point at the obstacle edge to the current position of the AUV. , center coordinate vector The perpendicular vector of vector d d is a scalar value pointing from the abrupt change point at the edge of the obstacle to the current position of the AUV.

[0032] To select the optimal bypass side, calculate the angle between the tangent direction and the obstacle direction. ,in, Let j be the coordinates of the tangent point. Select the current position coordinates of the AUV. The larger side is used as a temporary sub-target. If the included angles on both sides are similar, a random selection should be made to avoid oscillation. If a temporary sub-target... If the target is located inside an obstacle, jump to the next candidate edge and recalculate; otherwise, determine it as the final sub-target.

[0033] Select tangential subtarget: Select a subtarget as the instantaneous navigation target. If the final target is still obscured after arrival, iteratively calculate the next subtarget based on the new sonar data; otherwise, continue moving towards the final target until the mission is completed.

[0034] Furthermore, updating the Critic network objectives according to the multi-step reward mechanism MSR includes: The multi-step reward mechanism MSR adaptively adjusts the step size parameter of the multi-step reward based on the distance between the AUV and the nearest obstacle. Based on the step size parameter, it updates the Critic network target using multi-step Bellman rewards and accumulates the rewards only up to the final step through a truncation mechanism.

[0035] Specifically, the multi-step reward mechanism uses multi-step Bellman rewards to update the Critic network objective. : ; Where γ∈(0,1) is the discount factor. This represents the immediate reward at step t+b. End state Value estimation.

[0036] The truncation mechanism is as follows: ; If the trajectory terminates prematurely within n steps from time t (e.g., due to a collision or reaching the target), the reward is only accumulated up to the termination step m, and no further end-point value estimation is added. The Critic target is simplified to a finite m-step form.

[0037] Furthermore, the step size parameter n for adaptively adjusting the multi-step reward includes: ; in, The distance to the nearest obstacle. The ranging value of the i-th azimuth angle of the obstacle avoidance sonar. Distance is the threshold for risk assessment.

[0038] Furthermore, the overall reward r is calculated as follows: ; in, To terminate the event reward, For distance-based rewards, Penalties for posture alignment mechanism Forward speed reward incentive, Due to multi-dimensional physical limitations and energy consumption penalties, This is a near-field obstacle avoidance penalty item. This is an ocean current synergy term.

[0039] Termination Event Rewards : ; in, A positive reward for reaching the target point. The penalty imposed for colliding with an obstacle.

[0040] The reward function introduces a distance reward item. : ; in, These represent the distances from the AUV to the target at the termination time and the current time, respectively. and For the corresponding weights.

[0041] Posture alignment mechanism penalty : ; in, This represents the absolute deviation between the heading angle and the target direction. For the corresponding weights.

[0042] Forward speed reward incentive : ; in, For the corresponding weights.

[0043] Multi-dimensional physical limitations and energy consumption penalties : ; ; ; ; Among them, energy consumption penalty Suppress the magnitude of velocity and angular velocity, and smooth out the penalty. Limiting abrupt changes in control commands, angular velocity penalty This directly curbs inefficient behaviors such as spinning in place. For the corresponding weights, , These are the forward velocity and yaw rate at the previous moment.

[0044] Near-field obstacle avoidance penalty : ; This is achieved by calculating the minimum distance of the laser scanning data: when Less than the threshold D3 At the same time, apply a linearly increasing penalty to ensure that the AUV maintains a safe margin with respect to obstacles.

[0045] Introducing ocean current co-terms into the reward function .

[0046] ; Specifically, by calculating the cosine similarity between the AUV's heading and the ocean current direction, a positive reward is given when the angle is less than 90°, guiding it to sail with the current to reduce energy consumption. For the corresponding weights, Let the ocean current velocity vector be... Its amplitude; This is the unit vector of the AUV's current heading; The angle between the heading and the direction of the ocean current.

[0047] Figure 3 The convergence curves of MSR-SAC and the benchmark algorithm were compared. It is evident that MSR-SAC exhibits rapid reward growth in the early stages of training and converges within approximately 10 rounds, significantly outperforming the comparison algorithms. Its final average reward is approximately 230, a significant improvement over DDPG (Deep Deterministic Policy Gradient Algorithm), TD3 (Dual Delay Deep Deterministic Policy Gradient Algorithm), and SAC (Flexible Action-Evaluation Algorithm). Furthermore, the enhanced SAC exhibits smaller reward fluctuations and a smoother convergence curve in the later stages, reflecting the suppressive effect of the multi-step reward combined with dynamic step size mechanism on estimation variance, resulting in more stable policy updates. In summary, MSR-SAC, incorporating multi-step rewards and TSP, demonstrates significant advantages in both sample efficiency and performance.

[0048] Table 1 summarizes the performance indicators of 30 simulation experiments. As shown in Table 1, the MSR-SAC algorithm performed best across all evaluation indicators, achieving the highest success rate of 97.62% and the shortest average arrival time. Compared to the average level of other algorithms, MSR-SAC improved the success rate by approximately 13% and reduced the mission completion time by 30.67%, demonstrating significant advantages. The DDPG algorithm performed the worst, with the lowest success rate, reflecting its insufficient planning stability under ocean current interference. The SAC and TD3 algorithms had moderate performance, with success rates slightly lower than MSR-SAC but better than DDPG, although the TD3's travel time was significantly longer.

[0049] Table 1 From the perspective of trajectory morphology, although the path length of MSR-SAC is slightly longer than that of SAC, its total time is the shortest, demonstrating superior overall performance. This is mainly due to its long-term decision-making capabilities: such as... Figure 4As shown, MSR-SAC proactively employs forward-looking course-avoidance in the initial phase, adjusting its course in advance to align with the target point. This allows the AUV to accelerate ahead and maintain high-speed cruising, thus compensating for the slight difference in path length with a higher average speed. In contrast, the trajectories of algorithms like SAC and TD3 are more convoluted, leading to frequent turns and speed adjustments by the AUV, resulting in speed loss and increased energy consumption. Therefore, by generating a smoother trajectory, MSR-SAC globally optimizes both speed and energy efficiency, achieving a better balance between path length and navigation efficiency. This demonstrates its superior practicality and robustness in complex marine environments.

[0050] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for AUV path planning and obstacle avoidance based on an improved SAC algorithm, characterized in that, include: Construct an environmental interaction model for an autonomous underwater vehicle (AUV); Construct an MSR-SAC deep reinforcement learning network, wherein the MSR-SAC deep reinforcement learning network is obtained by introducing a multi-step reward mechanism MSR into the SAC deep reinforcement learning network; The value network target is updated using the multi-step reward mechanism (MSR) according to the environmental interaction model, and the comprehensive reward is calculated. Data is stored in the experience replay pool, and the parameters of the MSR-SAC deep reinforcement learning network are updated until convergence, thereby obtaining the trained policy network. The comprehensive reward is calculated based on the motion state of the AUV, its relative relationship with the target, and energy consumption constraints. The current sensor data is input into the trained policy network, which outputs thrust and rudder angle control commands to generate a collision-free trajectory. During training and execution, a tangential sub-target planner is introduced to calculate the tangential direction and generate tangential sub-targets to guide the AUV around obstacles.

2. The AUV path planning and obstacle avoidance method based on the improved SAC algorithm according to claim 1, characterized in that, The construction of an environmental interaction model for an autonomous underwater vehicle (AUV) includes: The state space is constructed based on the normalized distance measurements of the obstacle avoidance sonar in several directions, the relative distance between the AUV and the target, the angle between the target direction and the current heading, and the longitudinal and yaw angular velocities. The action space is constructed based on the forward speed command and the yaw angular velocity command to obtain the environmental interaction model.

3. The AUV path planning and obstacle avoidance method based on the improved SAC algorithm according to claim 1, characterized in that, Calculating the tangential direction to generate tangential sub-targets to guide the AUV around obstacles includes: The system uses sonar to detect obstacle obstruction in real time. When obstruction is detected, the system calculates the tangent sub-target point based on the obstacle edge information and uses it as a temporary navigation target. The AUV then arrives at the temporary navigation target before proceeding to the target point.

4. The AUV path planning and obstacle avoidance method based on the improved SAC algorithm according to claim 2, characterized in that, When occlusion is detected, the calculation of the tangent sub-target point based on obstacle edge information includes: When an obstruction is detected, edge detection is performed. When the distance difference between adjacent sonar beams that are obstructed exceeds a threshold, it is determined to be an obstacle boundary, i.e., a sudden change is detected. For each pair of adjacent sonar beams that detect a mutation, the obstacle edge direction is estimated based on azimuth and distance data interpolation, and a set of candidate edge directions is generated. The minimum deviation angle is selected from the candidate edge direction set as the obstacle bypass direction, and the obstacle edge abrupt change point corresponding to the obstacle bypass direction is obtained; Draw a circle with a preset radius centered at the abrupt change point on the edge of the obstacle, and calculate the two external tangent lines and the point of tangency from the current position of the AUV to the circle; Calculate the angle between the tangent direction and the obstacle direction based on the tangent point to obtain the tangent sub-target.

5. The AUV path planning and obstacle avoidance method based on the improved SAC algorithm according to claim 4, characterized in that, After obtaining the tangent sub-target, if the tangent sub-target is located inside an obstacle, then jump to the next candidate edge and recalculate.

6. The AUV path planning and obstacle avoidance method based on the improved SAC algorithm according to claim 1, characterized in that, According to the multi-step reward mechanism, the MSR update value network objectives include: The multi-step reward mechanism MSR adaptively adjusts the step size parameter of the multi-step reward based on the distance between the AUV and the nearest obstacle, updates the value network target using multi-step Bellman reward based on the step size parameter, and accumulates the reward only up to the final step through a truncation mechanism.

7. The AUV path planning and obstacle avoidance method based on the improved SAC algorithm according to claim 6, characterized in that, The step size parameter n for adaptively adjusting multi-step rewards includes: ； in, The distance to the nearest obstacle. The ranging value of the i-th azimuth angle of the obstacle avoidance sonar. Distance is the threshold for risk assessment.

8. The AUV path planning and obstacle avoidance method based on the improved SAC algorithm according to claim 1, characterized in that, The comprehensive reward r is calculated as follows: ； in, To terminate the event reward, For distance-based rewards, Penalties for posture alignment mechanism Forward speed reward incentive, Due to multi-dimensional physical limitations and energy consumption penalties, This is a near-field obstacle avoidance penalty item. This is an ocean current synergy term.

Images