Complex sea area AUV three-dimensional path planning and control method

Abstract

The invention discloses an AUV (Autonomous Underwater Vehicle) path planning and control integrated method for a complex marine environment, and provides an intelligent path control scheme integrating reinforcement learning and local obstacle avoidance mainly aiming at underwater navigation problems such as submarine topography fluctuation, frequent ocean current disturbance and uncertain dynamic obstacle distribution. According to the method, a realistic three-dimensional ocean grid environment is constructed, an improved Q-learning algorithm is utilized to complete global path planning, a group of optimal path point sequences are generated, and it is ensured that the AUV has the macroscopic navigation capability. Local obstacle avoidance is performed in combination with a three-dimensional dynamic window method (3D DWA), and sudden obstacles are dynamically avoided through speed sampling and trajectory prediction. A DDPG control strategy based on deep reinforcement learning is further introduced, end-to-end training of path planning and control execution is realized, and the system can directly output a control instruction according to a real-time sensing state. A cosine attenuation learning rate, a self-adaptive exploration mechanism, an energy consumption penalty function and a dynamic obstacle prediction and control action filtering technology are fused in the method, so that the intelligence of path planning, the stability of control output and the robustness of obstacle avoidance response are effectively improved. According to the invention, the autonomous navigation capability, the environmental adaptability and the safety task completion efficiency of the AUV in a complex marine environment can be obviously enhanced, and reliable intelligent control technical support is provided for underwater operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underwater robot navigation and artificial intelligence control, and particularly relates to a three-dimensional path planning and control method for AUV in complex sea areas. The method comprehensively uses key technologies such as environment modeling, reinforcement learning, path optimization and intelligent control, and realizes intelligent navigation and accurate control of AUV in complex and dynamic marine environment by integrating improved Q-learning algorithm, three-dimensional dynamic window obstacle avoidance method and deep reinforcement learning control strategy. The present application is suitable for the requirements of high precision, robustness and self-adaptive ability of underwater autonomous navigation system in the scenes of marine resource exploration, deep sea inspection, underwater target search, aquaculture monitoring and the like. BACKGROUND

[0002] As an important representative of deep-sea intelligent equipment, autonomous underwater vehicle (AUV) has been widely used in the fields of marine scientific research, resource development, military reconnaissance and the like in recent years. The core task of AUV is to complete long-time and high-precision seabed task operation through autonomous path planning and intelligent control, and the path planning capability is the core embodiment of the intelligence level of AUV. Especially in complex marine environment, due to the existence of ocean current disturbance, seabed undulating topography and a large number of static and dynamic obstacles and the like, the traditional path planning and control algorithm exposes many deficiencies in adaptability, real-time performance and safety.

[0003] At present, the mainstream AUV path planning method can be divided into graph search (such as A*), optimization algorithm (such as genetic algorithm, particle swarm algorithm) and reinforcement learning and the like. Among them, reinforcement learning is considered as the core technology direction of the next generation of underwater path planning because of its characteristics of not needing to explicitly define the environment model and being able to adaptively learn the strategy. The classical Q-learning algorithm is widely used in path optimization problems, and through constructing state-action value function, the optimal strategy is constantly approximated, which is suitable for most two-dimensional path planning tasks. However, this method has problems of slow convergence and being easy to fall into local optimum when dealing with high-dimensional continuous state space and complex constraints.

[0004] In order to cope with the problem of dynamic adjustment of path in complex environment, researchers introduce a local obstacle avoidance module, typically the dynamic window approach (DWA), which realizes the avoidance operation of sudden obstacles in the movement process by real-time velocity sampling and combining cost function to evaluate and select each group of motion trajectories. Although the DWA method has good real-time performance and local response ability, it still has some deficiencies in the overall optimality of path, prediction ability and environmental adaptability, especially in three-dimensional marine space, the two-dimensional structure of which limits the performance of trajectory optimization.

[0005] In addition, in the actual AUV execution process, the path planning and control usually adopt a "staged" structure design: that is, path planning is carried out first, and then control tracking is executed. This "separated" architecture has defects such as strategy fragmentation, discontinuous control, and serious path deviation when facing dynamic environment, complex disturbance and nonlinear behavior. In order to solve the above problems, in recent years, a research trend of integrating path planning and control has emerged, that is, through deep reinforcement learning to realize end-to-end strategy training, so that the AUV can directly output control instructions based on the environment state, thereby improving the response ability and control accuracy of the overall system.

[0006] However, the existing reinforcement learning control method generally relies on idealized simulation environment, and its training efficiency, convergence speed and deployment stability still have many challenges in actual complex water area. The main problems include: the environment modeling is not real enough, the reward function design is not reasonable, the state space expression is not complete, and the control signal is not continuous, etc. These problems directly affect the training effect of the control strategy and the feasibility of underwater application.

[0007] Therefore, an integrated path control method is needed, which integrates global path guidance, local obstacle avoidance response and deep reinforcement control, and can realize stable, efficient and safe control of AUV in real marine environment, and improve its complex task execution ability. SUMMARY

[0008] The present application provides a three-dimensional path planning and control method for AUV in complex sea area, which is suitable for intelligent navigation task of autonomous underwater vehicle (AUV) in complex dynamic marine environment. In view of the problems of separation of path planning and control strategy, slow response, poor obstacle avoidance robustness and discontinuous control in the prior art, the present application proposes a technical scheme integrating improved Q-learning global path planning, three-dimensional dynamic window method local obstacle avoidance and deep reinforcement learning (DDPG) integrated control, which realizes high-precision path guidance, stable control and dynamic obstacle avoidance function of AUV in complex scene.

[0009] The present application proposes an integrated path planning and control method for autonomous underwater vehicle (AUV) in complex marine environment, which introduces innovative mechanisms in multiple key technical links, and significantly improves the autonomy, stability and environmental adaptability of the system in dynamic three-dimensional environment. The core innovations include: improved Q-learning path search algorithm based on high-precision three-dimensional marine modeling, dynamic reward function integrating energy consumption and collision penalty, three-dimensional DWA local obstacle avoidance mechanism with speed prediction capability, and end-to-end DDPG deep reinforcement learning control framework. The overall method can be divided into the following five steps: Step one: environment modeling 1.1 Collect the DEM data of the seabed, and refine the original 10m x 10m grid to 1m x 1m through bilinear interpolation; 1.2 Based on historical hydrological data, construct a three-dimensional vector ocean current field using the Lamb vortex analytical solution, and generate time-varying disturbances at 30-minute intervals for the ocean current vectors (Ux, Uy, Uz) through Cauchy interpolation; 1.3 Generate a convex hull grid for static obstacles, and use a uniform speed or trajectory prediction model to obtain the bounding box parameters <center coordinates, size, heading, speed> for dynamic obstacles; 1.4 Construct a 1m x 1m x 1m voxel grid in the geocentric fixed coordinate system, and integrate the data from 1.1-1.3 to form a three-dimensional training space. Step two global path planning 2.1 Set the state space S = {x, y, z} and the action space A = {Δx, Δy, Δz}, where Δx, Δy, Δz ∈ {-1, 0, 1}; initialize the |S| x |A| Q table to zero; 2.2 Construct the reward function R = -α·d - β·e + γ·f where d is the Euclidean distance increment (the straight-line distance change from the current position to the target point, unit: meters), e is the unit propulsion energy consumption (kilojoules / meter), f = exp(-λ·Dist obs ), λ > 0 is the decay coefficient, Dist obs is the minimum distance between the current AUV and the nearest obstacle (meters); 2.3 At each time step t, use the ε-greedy strategy with ε(t) = ε0·cos(πt / T), η(t) = η0·cos(πt / T) as the exploration probability and learning rate respectively (ε0, η0 are initial values, T is the maximum number of training steps), update the Q table; 2.4 End the round when ‖p-p goal ‖ < δ goal or the number of steps N max is reached; output the global reference path P episode = {p0,...,p global} after accumulating N M rounds. Step three local obstacle avoidance 3.1 Generate an error vector e p based on the current coordinates p cur and the reference nodes p ref , and in the six-dimensional space of velocity (v x , v y , vz ,ω x ,ω y ,ω z Generate candidate velocity set V cand ; 3.2 For each candidate velocity v i The trajectory ξ is obtained by integrating the dynamic equation within the time window Δt. i (t), and calculate: cost trk =||ξ i (Δt)-p ref ||(path deviation), 3.3 Combine costs and select v using the weight vector W = [w1, w2, w3, w4]. best As a local control speed; when the minimum safe distance Dist min ≤Dist thr Emergency deceleration v is triggered at time safe And replan. Step 4: End-to-end control 4.1 Construct an Actor-Critic network based on DDPG, where the Actor input is [x,y,z,φ,θ,ψ,v]. best Output control command u = [T, τ] ψ ,τ θ ]; 4.2 Critic input <state, action> outputs Q-value; training uses 10 5 Capacity experience playback and soft update coefficient τ′=0.005; 4.3 After being saturated and mapped by the scaling factor κ and filtered by a first-order low-pass filter for 0.2s, u is sent to the thruster PWM and servo motor for execution. Step Five: Training and Deployment 5.1 In the MATLAB / Simulink-Hydrolib co-simulation, steps two to four are executed cyclically with a simulation step size of 0.02s; 5.2 If ||p cur -p goal ||<δ goal A collision occurs or the number of steps is greater than N. max Then the round ends; 5.3 Export the Actor weights θ after training. A Optimal Q-table * and parameters {W,Δt,Dist} thr The data is then sent to the ARM-Linux controller via ROS2DDS. 5.4 During the sea trial phase, θ was remotely adjusted using an encrypted OTA interface. A With {W,Δt}, keep the weights synchronized with the hardware clock. Attached Figure Description

[0010] Figure 1 A flowchart of a three-dimensional path planning and control method for AUVs in complex sea areas;

[0011] Figure 2 Data flow diagram for the global path planning module;

[0012] Figure 3 This is a data flow diagram for the local obstacle avoidance module;

[0013] Figure 4 A data flow diagram for the integrated planning and control module;

[0014] Figure 5 To improve the flowchart of the Q-learning algorithm;

[0015] Figure 6 This is a flowchart illustrating the integrated control process of global path planning and local obstacle avoidance.

[0016] Figure 7 This is a training framework diagram for a path control method based on deep reinforcement learning.

[0017] Figure 8 This is a flowchart of the training process for a control method based on deep reinforcement learning. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0019] like Figure 1 The diagram shown is an overall flowchart of a three-dimensional path planning and control method for AUVs in complex sea areas proposed in this invention. This method comprises three core modules: a global path planning module, a local obstacle avoidance module, and an integrated planning and control module. These modules work in sequence based on the input environmental modeling data to form a closed-loop autonomous underwater navigation and control system.

[0020] First, the system acquires environmental modeling data, including seabed topography, obstacle information, and ocean current parameters, as the input for path planning. The global path planning module uses an improved Q-learning algorithm to generate a global path point sequence from the starting point to the target.

[0021] Then, the local obstacle avoidance module, based on the three-dimensional dynamic window method (3D DWA), combines global path points and real-time perception data to make avoidance judgments on possible static or dynamic obstacles and outputs a corrected feasible trajectory.

[0022] Next, the integrated planning and control module uses the DDPG deep reinforcement learning algorithm to map the current state and the desired trajectory input into propulsion force and attitude control torque, thereby achieving continuous control output from perception to execution.

[0023] Ultimately, the AUV executes its motion based on the received control commands, enabling autonomous navigation, obstacle avoidance, and path tracking in complex marine environments.

[0024] like Figure 2 The diagram shown is a simplified data flow logic diagram of the global path planning module proposed in this invention. Based on 3D marine environment modeling, this module combines the Q-learning algorithm from reinforcement learning with multiple optimization strategies to achieve path search capability for AUVs in complex marine environments. The following is a detailed explanation of the steps in the diagram:

[0025] 1. Environmental Data Input and Preprocessing. First, the system collects multi-source environmental data, including seabed elevation data, obstacle information, and ocean current modeling parameters. These data sources typically originate from various sensor platforms or prior marine mapping data. Seabed elevation information reflects the topographical variations and is the foundation for avoiding terrain obstacles and selecting low-resistance paths in path planning. Obstacle data can cover obstacle areas such as fixed seabed structures, shipwrecks, pipelines, and reef groups. Ocean current modeling parameters are used to express the velocity vector distribution of ocean water in space and are key factors affecting AUV energy consumption, speed, and control strategies. All raw data undergoes preprocessing such as filtering, interpolation, and registration after input to ensure modeling accuracy and consistency.

[0026] 2. 3D Raster Environment Modeling. After data preprocessing, the system performs 3D spatial discretization on the entire ocean area, generating a unified 3D rasterized ocean environment model. This model divides the continuous space into several voxel grid cells, each assigned environmental attributes corresponding to its location, such as whether it is a navigable area, seabed depth, ocean current velocity, and direction vector. This environment modeling not only preserves spatial detail but also facilitates state mapping and path reasoning for reinforcement learning algorithms, providing a realistic and effective training space for subsequent Q-learning policy learning.

[0027] 3. Improved Q-learning path search initialization and execution. After the three-dimensional environment model is constructed, the system initializes the Q-learning path search module. First, define the state space and action space: the state space is composed of the position of the AUV in the three-dimensional grid environment and its local environment state, and the action space includes the movement selection of the AUV in six basic directions (forward, backward, left, right, up, down) or their combined directions. The system sets the initial learning rate (a), exploration rate (e) and discount factor (g) and other basic parameters. Then, the system takes the starting point as the initial state, uses the e-greedy strategy to select actions, and in each interaction, obtains the immediate reward according to the state transition, and uses the Q-learning update rule: Q(s,a)←Q(s,a)+a×[R+g×max(Q(s′,a′))-Q(s,a)] Q value iteration update. This process is repeated continuously, and the optimal path strategy from the starting point to the end point is gradually learned.

[0028] 4. Application of reinforcement learning optimization mechanism (reward function and parameter adaptive strategy). In order to improve the path learning efficiency and path quality, the invention introduces two key optimization mechanisms in the Q-learning path search process: (1) Reward function optimization: the system designs a multi-objective dynamic reward structure, which considers path length (expecting the shortest path), obstacle avoidance distance (maintaining a safe distance from obstacles) and energy consumption indicators (considering the influence of ocean currents on propulsion). For example, avoiding strong adverse ocean currents will receive positive rewards, while approaching obstacles or taking too long paths will be punished, guiding the learning strategy to develop in a safer, energy-saving and efficient direction. (2) Learning rate and exploration rate adaptive adjustment mechanism: the system dynamically adjusts the learning rate and exploration rate to adapt to the training progress and state space complexity. A higher exploration rate is maintained in the early stage to facilitate path diversity discovery; the exploration rate is gradually reduced in the later training stage to improve the stability of the strategy and the accuracy of the path. At the same time, the learning rate is adjusted with the training round, which helps to prevent premature convergence or falling into local optimum.

[0029] 5. Path output and task transfer. When the Q-learning training process meets the pre-set termination condition (such as reaching the target position, the number of training rounds reaching the upper limit or the strategy being stable and no longer updating), the system maps the optimal strategy learned in the Q-table to a specific three-dimensional track point sequence. The output path includes consecutive position points from the starting point to the end point, each point being a navigable, energy-controllable navigation node and meeting the obstacle avoidance constraint. This global path is passed to the subsequent local obstacle avoidance module and execution control module as prior trajectory information, serving as a basis for further navigation decision and motion control, ensuring that the AUV has a clear direction guide and path reference in actual operation.

[0030] In summary, Figure 2 The data flow structure shown clearly shows the technical points and key processes of the global path planning module proposed by the application. The module integrates environment modeling, reinforcement learning, reward mechanism design and strategy adaptive parameter adjustment to form a path learning system with high robustness and adaptability, significantly enhancing the navigation intelligence and task execution capability of the AUV in complex three-dimensional marine environments.

[0031] As Figure 3 As shown in the flow chart of the local obstacle avoidance module proposed by the application. Based on the three-dimensional dynamic window method (3D Dynamic Window Approach, 3D DWA), the module constructs a real-time obstacle avoidance mechanism suitable for complex marine environments based on the combination of global path points and real-time perception data. The system comprehensively considers the kinematic constraints of AUV, prediction mechanism, weight design and trajectory correction to generate a safe and feasible local path, and the specific steps are as follows:

[0032] 1. Input global path and real-time perception information. The system first obtains the global path point sequence from the output of the path planning module as the target guide for local path generation. At the same time, real-time environmental perception data is accessed, including the current detected obstacle position, AUV navigation speed, attitude angle and other key dynamic states. These data are provided by AUV body sensors (such as sonar, IMU, vision system, etc.), ensuring real-time and environmental adaptability of path adjustment, providing necessary environmental and state inputs for the dynamic window method.

[0033] 2. Three-dimensional dynamic window method (3D DWA) modeling. Based on the data input, the system calls the three-dimensional dynamic window method model (3DDWA) to generate a set of candidate motion trajectories. This method maps the current state of the AUV to the velocity space and combines physical feasibility and control constraints to dynamically calculate a series of possible velocity pairs (linear velocity + angular velocity) and corresponding trajectories. Compared with the traditional DWA, 3DDWA adds modeling of the depth dimension to adapt to the motion characteristics of the three-dimensional underwater space, thereby expanding the trajectory prediction range and improving the obstacle avoidance flexibility.

[0034] 3. Constructing dynamic prediction time model and AUV kinematic constraints. In the trajectory generation process, the system constructs a dynamic prediction time window for each candidate trajectory. The window length is dynamically set according to the current velocity, obstacle distribution density and response time requirement. At the same time, the system has an AUV kinematic model built in to clearly define acceleration limits, turning radius, maximum attitude change rate and other control boundaries to ensure that the predicted trajectory is physically achievable and dynamically controllable, avoiding the generation of unachievable trajectory paths.

[0035] 4. Design adaptive weight factors and introduce energy consumption / deviation penalty mechanism. To optimize the trajectory evaluation function, the system introduces multiple weight factors and implements an adaptive adjustment strategy. The weight factors mainly include: obstacle avoidance distance weight, target approach weight, speed stability weight, etc., which are dynamically adjusted according to the task stage and environmental characteristics. In addition, to improve the energy efficiency and execution accuracy of the trajectory, the system introduces two types of penalty terms in the evaluation function: (1) Energy consumption penalty, used to constrain the high energy consumption path of AUV in counterflow or frequent acceleration and deceleration; (2) Path deviation penalty, which gives negative evaluation to the trajectory deviating from the main line of the global path, guiding the trajectory to run close to the main path.

[0036] 5. Obstacle motion trajectory prediction and avoidance judgment. The system performs obstacle motion trajectory prediction and collision detection on each candidate trajectory generated by the dynamic window method. Obstacle prediction uses linear extrapolation, Bayesian filtering and other methods to estimate its future position, and the system performs spatial and temporal overlap judgment on the prediction results and the trajectory. If there is a potential collision risk, the corresponding trajectory is excluded. This mechanism improves the avoidance ability and response timeliness of AUV when facing dynamic obstacles such as mobile underwater equipment and swimming organisms.

[0037] 6. Output the modified local feasible trajectory. After trajectory evaluation and collision avoidance screening, the system selects the current optimal trajectory as the local path instruction executable by AUV. The trajectory has good target pointing, obstacle avoidance ability, reasonable energy consumption and physical feasibility. The final output local trajectory will be used for control integration module to perform control, and the trajectory adjustment result will be fed back to the path management module to realize task closed loop.

[0038] In summary, Figure 3 The local obstacle avoidance module shown in the figure integrates three-dimensional dynamic window method, prediction time modeling, adaptive parameter design and obstacle avoidance mechanism to build a high-robustness and high-precision local path correction scheme. This module ensures the safe navigation of AUV while improving the path tracking accuracy and task completion efficiency, and is an important part of the overall intelligent navigation architecture of the system. The integrated execution logic of global path point and local obstacle avoidance strategy is shown in Figure 6 .

[0039] As Figure 4 shown, it is the data flow diagram of the control integration module proposed by the present application. Based on the DDPG algorithm in deep reinforcement learning, the module builds an end-to-end path planning and execution control integration method from perception input to action control. This method effectively improves the adaptive control ability and navigation accuracy of AUV in complex environments. The following is a detailed description of each step:

[0040] 1. Input state perception information. This module first collects multi-source state information from the AUV body and the environment perception system, including: (1) AUV current state parameters such as position, attitude, speed, acceleration, etc.; (2) desired state information, i.e. the reference attitude and desired speed provided by the path planning module; (3) obstacle distance information, used to reflect the spatial position distribution of obstacles in the surrounding environment; (4) local trajectory correction term, used to correct disturbance errors or obstacle avoidance offsets in the path planning process. These information together constitute the observation input of the deep reinforcement learning model, ensuring that the control strategy has sufficient state perception basis.

[0041] 2. Construct a deep reinforcement learning model (DDPG). Based on the above state information, the system constructs a deep deterministic policy gradient (DDPG, Deep Deterministic Policy Gradient) model. The model is composed of a policy network (Actor) and a value network (Critic), which can handle complex control problems in continuous action space. By mapping the input state to continuous control actions, the DDPG model can directly output the physical quantities required for propulsion control, such as thrust vector, pitch moment, yaw moment, etc., realizing an end-to-end closed-loop control process of perception-decision-control.

[0042] 3. Policy network training (based on state-action mapping). In the training phase, the policy network learns the mapping relationship between state and action. The system generates control action output according to each frame of state input, and gets immediate reward through interaction with the environment. The policy network continuously optimizes its parameters through backpropagation to improve the effectiveness and robustness of the control action. The value network evaluates the merits of the current policy, providing a reference signal for the update of the policy network. The stability and convergence of the model are enhanced through experience replay and target network mechanisms during training.

[0043] 4. Control strategy optimization mechanism design. In order to improve the smoothness and execution stability of the control output, the invention designs three control strategy optimization mechanisms based on DDPG: (1) First-order action filter, used to suppress high-frequency fluctuations in control output, achieve smooth processing of control signals, and prevent the system from experiencing drastic command changes; (2) Action chattering penalty term, a penalty for the rate of change of control quantity is added to the reward function, guiding the model to generate stable and gradual action sequences, reducing attitude control fluctuations; (3) Arc tangent type reward function design, the change range of the reward function is nonlinearly compressed, enhancing the sensitivity to small action errors, which is beneficial to improve the micro-control precision and accelerate the training convergence.

[0044] 5. Output control force and torque commands. Through the trained converged DDPG policy network, the system can map the perception state to precise control output in real-time during actual operation. The output includes linear thrust commands for the propulsion system and control torques for pitch, roll, and yaw directions for attitude control. All control signals are continuous variables, suitable for driving interfaces of AUV thrusters and rudder systems, providing directly usable physical control quantities for the underlying execution system.

[0045] 6. AUV propulsion and steering execution control. Finally, the system issues control commands to the AUV execution module to drive the propulsion device and attitude control mechanism to realize actual action. The controller adjusts in a closed loop according to the current feedback signal to ensure that the AUV moves accurately according to the preset path. This process realizes the whole process of automatic response from perception to control, supports stable navigation and dynamic adjustment in complex tasks, and improves the overall response speed and operation efficiency of the system.

[0046] In summary, Figure 4 The control integration module shown realizes a high-precision integrated control scheme for AUV from state perception to control execution by introducing a DDPG-based deep reinforcement learning architecture and integrating multiple control optimization strategies. This module can maintain high responsiveness, high stability, and task adaptability in dynamic marine environments, and is a core component of the AUV autonomous control system of the present application.

[0047] Figure 5 The overall process of the improved Q-learning algorithm for the AUV path planning and control integration method in complex marine environments is shown in detail. This algorithm is mainly used to realize global path search and strategy learning in a three-dimensional grid environment. By introducing various optimization mechanisms, the intelligence, convergence speed, and energy efficiency of path planning are improved. The following is an explanation of each step in the figure:

[0048] 1. This step marks the beginning of the entire path planning algorithm process. In this stage, the system initializes the three-dimensional training environment, including loading the sea elevation information, obstacle distribution information, and Lamb vortex model parameters, to construct a realistic 20x20x20 three-dimensional grid environment, providing a basic spatial structure for subsequent path search.

[0049] 2. In the initial parameter setting stage, the system sets the relevant parameters of the Q-learning algorithm. These include the maximum number of training episodes (max_episode), the maximum number of steps per episode (max_step), the learning rate (learning_rate), the exploration rate (exploring_rate), the reward discount factor (γ), the action space set (A), the energy consumption weight factor (σ), and the establishment of the initial Q-table. In addition, the starting point and target point of the AUV need to be set for the definition of the path search target.

[0050] 3. This step is the beginning of each training episode. First, the episode counter is incremented by 1 (episode = episode + 1), and then the internal step count of the current episode is incremented by 1 (step = step + 1) to prepare for the episode state update.

[0051] 4. In the dynamic parameter adjustment stage, the system dynamically adjusts the real-time learning rate and exploration rate based on the current training episode number and state complexity. The learning rate is gradually reduced using a cosine decay strategy to improve stability, while the exploration rate is adjusted according to the environmental state to increase exploration in the early stages of training and strengthen convergence in the later stages.

[0052] 5. After the AUV performs an action in a certain state, the system calculates the immediate reward value based on the behavior feedback. The reward function considers multiple factors such as path length (shorter is better), distance from obstacles (further is better), and path energy consumption (lower is better) to strengthen the agent's perception and preference for safety and energy efficiency.

[0053] 6. After the reward calculation is completed, the system iterates the state-action values in the Q-table according to the standard Q-learning update formula. This process allows the agent to continuously learn and approach the optimal strategy through trial and error using the core mechanism of reinforcement learning.

[0054] 7. This step determines whether the current episode meets the termination conditions, including whether the target point has been reached, whether the AUV has collided with obstacles or fallen into a dead zone, whether the maximum number of steps has been reached, etc. If any of these conditions are met, the current episode ends; if not, it returns to step

[0060] to continue the next action selection and learning.

[0055] 8. If the current episode ends but the training has not reached the maximum number of episodes, the system resets the AUV state to the starting point while preserving the Q-table learning results, preparing for the next training episode to continue optimizing the strategy.

[0056] 9. This stage determines whether the algorithm has reached the pre-set maximum number of training episodes (max_episode). If the condition has not been met, the system performs the episode + 1 operation again and enters a new round of training; if the termination condition is met, it enters the next stage.

[0057] 10. Once the training is completed, the system outputs the optimal path point sequence represented by the Q-table, which is the basis for the AUV to navigate in the current three-dimensional marine environment, and the algorithm process ends.

[0058] Through the improved Q-learning process described above, the present application not only improves the adaptability of three-dimensional path planning in complex environments, but also significantly improves the path convergence speed, safety and global optimality by introducing a dynamic adjustment mechanism, an energy penalty term and a real environment modeling, and has good engineering realizability and application prospect.

[0059] Figure 6 The overall process of the global path planning based on Q-learning and the local obstacle avoidance fusion control of the three-dimensional dynamic window method (3D DWA) is shown in detail. This method combines global path and real-time obstacle avoidance strategy to realize accurate path tracking and safety avoidance mechanism of AUV in dynamic marine environment. The following is a specific explanation of each step in the figure:

[0060] 1. This process is marked by the starting point of the path tracking control task, and the system initializes to enter the autonomous navigation mode, preparing to perform the path planning and obstacle avoidance control task.

[0061] 2. In the global path planning stage, the system calls the improved Q-learning algorithm described above to search and train the three-dimensional grid marine environment, and outputs a set of discrete path points to form a global reference path. This path can avoid large-scale static obstacles and optimize the energy consumption, providing macro guidance for subsequent navigation.

[0062] 3. After the global path points are completed, they are used as input for the local control module, and the system sequentially takes the path points as target reference points for local obstacle avoidance and tracking. This path point sequence is sequentially called into the control process.

[0063] 4. In the velocity sampling stage, combined with the current state of the AUV (position, velocity, attitude, etc.), a set of candidate control actions is formed in the feasible solution space of three-dimensional linear velocity and angular velocity for trajectory prediction.

[0064] 5. Three-dimensional space sampling covers the trajectory set that the AUV may move within the forward time window, while considering the physical constraints such as turning angle, speed change, attitude dynamics, etc., forming a high-dimensional motion prediction space.

[0065] 6. In the trajectory prediction stage, the system predicts the future trajectory of the AUV based on each set of sampled velocity combinations, and simulates the proximity of obstacles in the environment, path deviation and other influencing factors to form a set of candidate trajectories.

[0066] 7. The trajectory evaluation stage introduces an improved cost function. This function integrates multiple evaluation factors: path deviation, minimum distance to obstacles, energy consumption estimation, target guidance rate, etc. At the same time, a parameter adaptive mechanism is introduced, which allows the evaluation index weight to be dynamically adjusted, enhancing the flexibility and generalization ability of the system.

[0067] 8. The system scores all candidate trajectories and selects the one with the highest score as the optimal trajectory, and the corresponding control action as the optimal motion strategy for the current time.

[0068] 9. Execution stage: The system immediately executes the optimal action to drive the AUV thruster or rudder system, completes the corresponding action, and updates the state information (position, velocity, attitude angle, etc.) of the system in real time.

[0069] 10. Judgment stage 1: The system determines whether the AUV has reached the current target path point. If not (e.g., the distance is greater than the set threshold), it returns to the velocity sampling stage to continue planning the local trajectory; if it has reached, it continues to enter the path point update process.

[0070] 11. Judgment stage 2: The system further determines whether the current path point is the end point path point in the global path. If not, the target path point index is incremented by 1, and a new round of local obstacle avoidance control is started; if it has reached the end point, it indicates that the entire navigation task is completed, and the process enters the termination stage.

[0071] 12. Algorithm end stage: When the AUV successfully navigates to the end point of the global path, the system stops action output, records the path tracking and obstacle avoidance data, and marks the completion of the entire integrated control process.

[0072] Through the integration method shown in the figure, the Q-learning global guidance and three-dimensional dynamic window method (3DDWA) local real-time control are highly integrated, which not only ensures that the AUV obtains the global shortest path reference in the complex marine environment, but also realizes dynamic response and real-time avoidance of sudden obstacles, improving the autonomy, robustness and task completion efficiency of the entire system.

[0073] Figure 7 The figure shows the AUV path planning and control integrated training framework based on deep reinforcement learning proposed by the present application. The training environment mainly consists of two parts: AUV simulation environment and reinforcement learning training environment, which cooperate to form a closed-loop control system of state perception-control decision-action feedback-policy update.

[0074] Firstly, the AUV simulation environment module builds a three-dimensional underwater virtual space that conforms to the real operation situation, including the six-degree-of-freedom kinematics and dynamics model of the AUV, as well as the external water environment modeling, including the complex factors such as water flow interference and obstacle distribution. The system control input is the action signal, which theoretically corresponds to the propeller speed and rudder angle instruction. To simplify the training process, this paper abstracts it into three continuous control quantities: longitudinal thrust, turning moment and pitch moment.

[0075] After the action signal is input, the simulation environment calculates and outputs the next time state of the AUV based on the dynamics and kinematics equations, including position, velocity, attitude angle and acceleration, etc. At the same time, combined with the spatial position of the obstacle and the modeling results of the forward-looking sonar, the system updates the ranging information of the AUV in real time for environmental perception input.

[0076] To match it, the reinforcement learning training environment is built based on the DDPG (Deep Deterministic Policy Gradient) algorithm, which contains the state coding module, action policy network (Policy Network) and value evaluation network (Value Network) inside, together forming the Actor-Critic architecture.

[0077] The state coding module receives state information and sensor measurement values (such as sonar ranging data) from the simulation environment, normalizes and extracts features, generates a standardized observation vector, and inputs it to the two neural networks for processing. The action network outputs continuous control actions, and the evaluation network scores the current state-action pair, generating Q values to guide the update direction of network parameters.

[0078] To reduce the jitter of action output and improve the control stability, a first-order low-pass filter is added to the output end of the policy network to smooth the longitudinal thrust and moment signals, ensuring that the action signal is more consistent with the actual hardware system response characteristics.

[0079] Finally, the filtered control signal is transmitted to the simulation environment to drive the AUV movement and produce new state changes. The system calculates the immediate reward value based on the error between the current position and the target, the degree of path deviation, the control stability, and the obstacle avoidance situation, etc., and feeds it back to the training system to update the policy network and value network parameters.

[0080] The overall training process is realized by periodic calling of the Simulink simulation module by the m file: the m file is responsible for network building, state management and parameter optimization, and Simulink is used for simulation operation and environment feedback update. The two work together to build a deep reinforcement learning control system that can learn, evaluate and deploy.

[0081] The integrated training framework not only realizes the synchronous fusion of path control and strategy optimization, but also effectively improves the continuity and robustness of control actions, significantly enhancing the navigation ability and autonomy of AUVs in complex dynamic environments, providing highly feasible engineering support for the deployment and operation of underwater intelligent agents.

[0082] Figure 8 The training flowchart of the AUV path control method based on deep reinforcement learning is shown. This flowchart relies on the control system architecture constructed Figure 7 The DDPG (Deep Deterministic Policy Gradient) algorithm is used to realize the end-to-end integrated training of path control strategy and motion control behavior. The entire training process consists of multiple reinforcement learning episodes. The training steps are as follows:

[0083] 1. At the beginning of training, the system first imports the initial information, including the initial angle, position, velocity and other state information of the AUV, and initializes the hyperparameters required for training, such as learning rate, exploration rate, maximum training episodes (max_episode) and maximum step per episode (max_step) and other parameters, providing basic settings for the subsequent reinforcement learning process.

[0084] 2. Then, the system determines the input structure according to the type of the current control target, which will determine the input feature composition method of the subsequent DDPG training module. According to the different training tasks, the system mainly includes the following two setting methods:

[0085] 2.1 If it is a dynamics controller training, the system sets the control target as the desired attitude angle information, i.e. the desired yaw angle and pitch angle. This training mode focuses on attitude control, and the goal is to quickly and accurately adjust the AUV's heading and pitch state to meet the needs of path navigation or obstacle avoidance control. The system obtains the actual yaw and pitch angles of the AUV in real time at each training time step, and compares them with the set desired values to calculate the current attitude error. This error value is used as one of the observation inputs to guide the DDPG policy network to generate the corresponding adjustment action to correct the attitude deviation.

[0086] 2.2 If it is a path tracking controller training, the system sets the target as the spatial coordinates of the target position point, usually including three-dimensional coordinate information (x, y, z), to guide the AUV to complete the end-to-end path navigation task. In this process, the system calculates the Euclidean distance or position deviation between the current position of the AUV and the target point in real time, and uses it as one of the key observation variables to describe the distance relationship between the current position and the task target. In the above two types of task scenarios, the system will splice the calculated angle error or position deviation with other state information of the current AUV (such as speed, attitude angle, angular velocity, acceleration, sensor data, etc.) to generate a unified observation value vector. After normalization and preprocessing, the vector is input into the policy network and value network in the DDPG training module as input, used to output the current control action and perform policy optimization.

[0087] 3. In the DDPG training module, the observation value is first input into the action network (Policy Net) to generate the continuous control action at the current time; at the same time, the observation value and the action are input into the evaluation network (Value Net) to output the corresponding Q value, reflecting the value evaluation of the action in the current state. The system uses the time difference (TD) error and the reward signal to continuously update the network parameters through the back propagation mechanism to realize policy optimization.

[0088] 4. Then, the system inputs the control action (thrust, rudder moment, pitch moment) output by the action network into the AUV simulation module. First, it is input into the AUV dynamics model to update the AUV's angle, speed, angular acceleration, and acceleration information; then, these state information is input into the AUV kinematics model to update the AUV's position state. If it is an integrated controller training, the obstacle detection mechanism also needs to be activated, and the current ranging information is returned.

[0089] 5. The simulation module calculates the immediate reward value r_t at the current time step according to the current position of the AUV, the target state, the control behavior stability, and the obstacle detection feedback, etc. The reward value is returned to the DDPG training module to further optimize the neural network weights and improve the policy convergence and control robustness.

[0090] 6. After completing the simulation execution and state update within one time step, the system needs to determine whether the current round should be terminated to ensure the efficiency and safety of the training process. The system evaluates several termination conditions to determine whether to enter the next training round or continue the current training step, including the following three types of judgment criteria:

[0091] 6.1 If the AUV has successfully reached the target state, i.e., the error between the current actual state and the task target is below the preset tolerance threshold, the system will determine that the task has been completed. The state may include a position error less than the target distance threshold, or the difference between the attitude angle (heading angle, pitch angle) and the expected angle is less than the set error range. At this time, it means that the control strategy can effectively guide the AUV to reach the target point or achieve the target attitude, and there is no need to continue the current round of training.

[0092] 6.2 If the AUV encounters a collision event during the simulation, such as physical contact with obstacles, boundary walls, or seabed terrain in the simulated environment, the system will determine that there is a safety problem with the current strategy. Collision detection is usually based on abnormal range finding by forward-looking sonar, obstacle distance below a certain threshold, or illegal state transitions, etc. After detecting a collision, the system will immediately terminate the current round and use it as a negative training sample for experience feedback.

[0093] 6.3 If the number of simulation steps in the current round has reached the maximum step limit (max_step), the system will also terminate the training round. This situation usually indicates that the control strategy has failed to effectively complete the task within the specified time, and there may be problems such as insufficient strategy exploration, unreasonable reward function, or insufficient coverage of the state space. If any of the above conditions are met, the system determines that the current round of training is complete and enters the round summary and next round initialization process; if none of the conditions are met, the system performs the step = step + 1 operation, increments the current simulation step by one, and then returns to step 88 to continue the action selection, environment interaction, and policy update processes to advance the training process of the current round.

[0094] 7. After completing each training round, the system checks whether the current training episode number has reached the maximum number of training episodes max_episode. If not, the training environment is reset, the state is initialized, and the next round of training is entered; if the condition is met, the training is determined to be complete, and the system outputs the final trained controller strategy.

[0095] 8. Through the DDPG training mechanism constructed by this process, the system can achieve policy optimization of high-dimensional continuous control signals, while combining reward function constraints and simulation feedback to achieve multi-objective integrated learning of AUV path tracking, attitude control, and dynamic obstacle avoidance, with good generalization ability and potential for practical deployment.

Claims

1. A three-dimensional path planning and control method for AUV in complex sea area, characterized in that it comprises the following steps in sequence: Step one: environment modeling 1.1 Collect the DEM data of the seabed, and refine the original 10m x 10m grid to 1m x 1m through bilinear interpolation; 1.2 Based on historical hydrological data, construct a three-dimensional vector ocean current field using the Lamb vortex analytical solution, and generate time-varying disturbances by Cauchy interpolation of ocean current vectors (Ux, Uy, Uz) at 30-minute intervals; 1.3 Generate a convex hull grid for static obstacles, and use a uniform speed or trajectory prediction model to obtain the bounding box parameters (center coordinates, size, heading, speed) for dynamic obstacles; 1.4 Construct a 1m x 1m x 1m voxel grid in the fixed coordinate system of the Earth's center, and form a three-dimensional training space by integrating data from 1.1 to 1.

3. Step two: global path planning 2.1 Set the state space S = {x, y, z} and the action space A = {Δx, Δy, Δz}, where Δx, Δy, Δz ∈ {-1, 0, 1}; initialize the |S| x |A| Q table to zero; 2.2 Construct the reward function R = -α·d - β·e + γ·f where · d is the Euclidean distance increment (the straight-line distance change from the current position to the target point, in meters), · e is the unit propulsion energy consumption (kilojoules per meter), • f = exp(-λ-Dist obs ), λ > 0 is the decay coefficient, Dist obs is the minimum distance (meters) from the current AUV to the nearest obstacle; 2.3 At each time step t, use the ε-greedy strategy with ε(t) = ε0·cos(πt / T), η(t) = η0·cos(πt / T) as the exploration probability and learning rate (ε0, η0 are initial values, and T is the maximum number of training steps) to update the Q table; 2.4 End round when ||p-p goal || < δ goal or step count N max cumulative N episode Output global reference path after rounds P global = {p0,...,p M}. Step three: local obstacle avoidance 3.1 According to the current coordinate p cur With reference to the node p ref Generate error vector e p , in the six-dimensional space of speed (v x , v y , v z , ω x , ω y , ω z ) to produce a candidate speed set V cand ; 3.2 For each candidate velocity v i Integrating the dynamics equation over the time window At gives the trajectory ξ i (t), and we compute: • cost trk = ||ξ i (Δt)-p ref || (path deviation), • (propulsion energy consumption), • (safety distance count down), • (targeted degree of orientation); 3.3 Merge cost with weight vector W = [wl, w2, w3, w4] and select v best As local control speed; when minimum safety distance Dist min ≤ Dist thr Emergency deceleration v safe is triggered and replanned. Step four: end-to-end control 4.1 Build the Actor-Critic network based on DDPG, Actor input [x, y, z, φ, θ, ψ, v best ], output control instruction u = [T, τ ψ , τ θ ] ; 4.2 Critic input <state, action> output Q value; training uses 10 5 Capacity experience replay with soft update coefficient τ' = 0.005; 4.3 After saturating the mapping of u by a proportion factor κ and applying a 0.2s first-order low-pass filter, send it to the propeller PWM and rudder for execution. Step five: training and deployment 5.1 In MATLAB / Simulink-Hydrolib joint simulation, Steps 2 to 4 are executed in a loop with a simulation step size of 0.02s; 5.2 If ||p cur -p goal ‖ < δ goal , a collision occurs or the number of steps > N max then terminate the round; 5.3 Export Actor weights θ after training A , optimal Q table Q * and parameters {W, Δt, Dist thr} are issued to ARM-Linux controller through ROS2DDS. 5.4 Remote adjustment of θ using encrypted OTA interface in sea trial phase A With {W, Δt}, keep the weight synchronized with the hardware clock.

2. The method of claim 1, wherein, When the training space area is greater than 10km x 10km, the grid length is set to 5m, and the state and action discrete resolution is correspondingly reduced to reduce storage overhead.

3. The method of claim 1, wherein, The reward function weights satisfy α ∈ [0.3, 0.5], β ∈ [0.2, 0.4], γ ∈ [0.2, 0.4], α + β + γ = 1.

4. The method of claim 1, wherein, The time window Δt of Step three 1 is determined by the fuzzy logic controller DynamicFuzzy(ρ obs ,▽U): Δt = 0.5 when both the obstacle density ρ obs and the current velocity gradient ▽U are high; Δt = 3 when both are low, and linear interpolation otherwise.

5. The method of claim 1, wherein, The DDPG network is a three-layer fully connected structure, with hidden layer node numbers of 256, 256, and 128, respectively. The activation function uses tanh, and the output action is normalized to the driver executable interval by a proportion factor κ ∈ (0, 1].

6. The method of claim 1, wherein, The critic network training adopts Huber loss function and adds L2 regularization term, and the regularization coefficient λ reg = 1 x 10 -4 , the target network is soft updated at a frequency of 2 kHz / τ'.

7. The method of claim 1, wherein, The thrust-speed relationship is T = a0+ a1n+ a2n 2 , The rudder angle-lift coefficient relationship is where a0, a1, a2, b0, b1, and b2 are calibrated through 10kN level tank towing tests.

8. The method of claim 1, wherein, The method can be deployed in FPGA-DSP cooperative controller (20 kHz control frequency), ARM-Linux edge node (200 Hz control frequency) or ROS2 DDS-based distributed architecture, and upgraded through encrypted OTA A With the configuration file Config.xml.

Images