An underwater vehicle path following control method

Abstract

This invention discloses a path-following control method for underwater vehicles, comprising: acquiring discrete waypoints containing position and attitude adjustment targets; constructing an external tracking framework to decompose the global task into waypoint-to-point tracking subtasks; designing an improved line-of-sight guidance law; establishing a Markov decision process model containing attitude adjustment targets, with the reward function including position and attitude deviation terms; offline training of a control strategy network using an improved dual-delay depth deterministic policy gradient algorithm, the algorithm including a two-stage exploration strategy, a reward-based dynamic two-level priority experience replay, and a dynamic target Q-value estimation strategy; and real-time control using the trained strategy network, switching waypoints when conditions are met. This invention can guide an underactuated autonomous underwater vehicle to meet preset attitude adjustment targets while arriving at discrete waypoints, requiring only current waypoint information to achieve high-precision position and attitude synchronous following, adapting to different dynamics and environmental disturbances, and exhibiting strong robustness and generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of underwater vehicle path control technology, specifically an underwater vehicle path following control method. Background Technology

[0002] Autonomous underwater vehicles (AUVs) are crucial tools for exploring and developing marine resources, and their autonomous tracking and control capabilities directly determine the accuracy and efficiency of mission execution. While significant progress has been made in AUV tracking and control research, existing technologies still face three key limitations and technical bottlenecks in practical applications: First, the assumption of a continuous path deviates from actual planned output. Most existing tracking and control studies assume a continuous curve as the reference path, but in actual engineering, most planners generate paths consisting of a set of discrete waypoints. Second, model simplification leads to performance degradation because many studies use reduced-order dynamic models, which cannot capture the coupling characteristics of underactuated AUVs or fully represent six-degree-of-freedom motion. Finally, traditional control methods heavily rely on accurate models and hydrodynamic parameters. However, AUV hydrodynamic parameters are often difficult to obtain accurately and are perturbed by mission loads, posing a serious challenge to the robustness and environmental adaptability of traditional control methods. Summary of the Invention

[0003] To address the problems existing in the prior art, the present invention aims to provide a path-following control method for underwater vehicles. This invention can effectively guide underactuated autonomous underwater vehicles to meet preset attitude adjustment targets while reaching discrete waypoints. It can operate with only the current waypoint information, achieving high-precision position and attitude synchronization and following capability. It can quickly adapt to different dynamic characteristics and environmental disturbances, and has good robustness and generalization ability.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A path-following control method for an underwater vehicle includes the following steps: Step S1: Obtain a discrete waypoint sequence and construct an external tracking framework; obtain a discrete waypoint sequence containing position information and attitude adjustment targets generated by a path planner, and decompose the global discrete path-following task into a series of waypoint-to-point tracking sub-tasks using the constructed external tracking framework; Step S2: Design an improved line-of-sight guidance law to provide guidance constraints; Step S3: Establish a Markov decision process model containing attitude adjustment targets, defining the action space, state space, and reward function, wherein the reward function includes a position deviation term and an attitude adjustment term. Deviation term; Step S4: The control strategy network is trained offline using an improved dual-delay deep deterministic strategy gradient algorithm; The improved dual-delay deep deterministic strategy gradient algorithm includes a two-stage exploration strategy, a reward-based dynamic two-level priority experience replay mechanism, and a dynamic target Q-value estimation strategy; Step S5: Real-time control and waypoint switching are performed based on the trained strategy network; Using the trained control strategy network, control commands are generated according to the current state of the underwater vehicle to drive the actuators, and the vehicle switches to the next target waypoint after the preset waypoint switching conditions are met, until the full path following is completed.

[0006] As a further improvement to the above technical solution:

[0007] In step S2, the improved line-of-sight guidance law constructs guidance constraints through the following geometric relationships: Definition To connect the AUV quality center to the waypoint The line of position, Indicates the distance from the current position of the AUV to... The line of the central axis, angle Defined as waypoint central axis and The included angle, angle The current center axis of the AUV and The angle between the two points, and the waypoints for AUV tracking. Sometimes, Approaching .

[0008] In step S3, the reward function Defined as:

[0009]

[0010]

[0011]

[0012] in, Here are the weight coefficients for each optimization objective, and t is the time step. The penalty imposed for collision behavior. Given a safe distance for obstacle avoidance, This indicates that the AUV arrives at time t. Central axis The distance.

[0013] In step S4, the two-stage exploration strategy includes: in the early stage of training, randomly sampling actions from a uniform distribution to fill the experience replay pool and increase sample diversity; in the middle and late stages of training, transitioning to a policy-guided exploration mode that superimposes Gaussian noise on the output of the policy network.

[0014] In step S4, the reward-based dynamic two-level priority experience replay mechanism includes: defining a sampling priority for each sample based on the level of reward obtained by the experience sample; and extracting a preset number of samples in each training iteration, wherein the number of samples sampled based on priority is dynamically adjusted and gradually decreases with the number of training steps until it eventually degenerates into uniform sampling.

[0015] The reward-based dynamic two-level priority experience replay mechanism also includes: applying importance sampling weights to correct the gradient estimation bias introduced by priority replay.

[0016] In step S4, the dynamic target Q-value estimation strategy includes: introducing a weighting coefficient that adapts to the training time, and calculating the target Q-value as a weighted sum of the minimum and maximum values ​​of the two target Q-network outputs, so as to suppress the value overestimation bias in the early stage of training and overcome the value underestimation bias in the later stage of training.

[0017] In step S5, the waypoint switching condition is:

[0018]

[0019] in It is the position of the AUV at time t. The AUV begins tracking waypoints. The position at that time The current waypoint being followed The location.

[0020] In step S3, the AUV is equipped with a rudder cross and a rear thruster, and the state space is a multi-dimensional vector that contains at least... ,in, , , , , , , , , These are the longitudinal speed, lateral speed, heave speed, roll rate, pitch rate, yaw rate, roll angle, pitch angle, and yaw angle of an AUV. , These represent the AUVs at waypoints. The pitch and yaw angles are given, and x, y, and z are the position coordinates of the AUV in the northeast coordinate system. Indicates thrust. and These represent the angles of the rudder and elevator, respectively.

[0021] The AUV is an underactuated autonomous underwater vehicle, and its dynamic model adopts a six-degree-of-freedom nonlinear model. The control strategy network is an end-to-end black box network. During the training and inference phases, the control strategy network only uses the observation vector in the state space as input and outputs action commands, without using the hydrodynamic parameters or dynamic equations of the underwater vehicle as network input or as part of the network structure.

[0022] The beneficial effects of this invention are:

[0023] (1) By designing and improving the line-of-sight guidance law and incorporating both position and attitude deviations into the reward function framework of the Markov decision process, this invention can effectively guide an underactuated autonomous underwater vehicle to meet preset attitude adjustment targets while arriving at discrete waypoints, achieving high-precision position and attitude synchronization tracking capability. Simulation results show that after sufficient training, the average position deviation at waypoint switching moments can converge to 0.15 meters, and the average attitude deviation can converge to 0.17 radians, achieving accurate spatial pose synchronization tracking and solving the problem that existing technologies struggle to balance position accuracy and arrival attitude requirements.

[0024] (2) An end-to-end black-box control architecture is adopted. The developed intelligent controller does not rely on precise hydrodynamic coefficients and dynamic equations, but only uses input-output data for policy learning. The two-stage exploration strategy and the reward-based dynamic two-level priority experience replay mechanism introduced in the improved dual-delay deep deterministic policy gradient algorithm work together to significantly improve the sample utilization rate in the early stage of training and the convergence stability in the middle and late stages. This enables the controller to quickly adapt to different dynamic characteristics and environmental disturbances, and has good robustness and generalization ability.

[0025] (3) The proposed dynamic target Q-value estimation strategy effectively suppresses the excessive underestimation bias caused by the minimum value operation during reinforcement learning training by introducing adaptive weighting coefficients to dynamically adjust the calculation method of target Q-value, balances the bias and variance of value estimation, and significantly improves the training stability of policy network and the reliability of final control performance.

[0026] (4) The external tracking framework established in this invention decomposes the global discrete path into independent waypoint sub-tasks, and the controller only needs the current waypoint information to run. This allows the method to be directly connected to various mainstream path planners (including offline global planning and online replanning) without modification, and can adapt to the dynamic waypoint sequence output by the planner in real time, which has good engineering practical value and broad application prospects. Attached Figure Description

[0027] Figure 1 This invention relates to an embodiment of the six-degree-of-freedom motion state, forces, and torques of an AUV in a dual-coordinate system.

[0028] Figure 2 This is a schematic diagram of a discrete path for an AUV with an attitude target according to an embodiment of the present invention.

[0029] Figure 3 This is an embodiment of the discrete path tracking framework based on a DRL controller according to the present invention.

[0030] Figure 4 This is a schematic diagram of the improved line-of-sight guidance law (I-LOS) of this invention.

[0031] Figure 5 This is the two-stage exploration strategy of the present invention.

[0032] Figure 6 This invention relates to a DRL controller framework based on I-TD3.

[0033] Figure 7 This refers to the average round reward of the sliding window during I-TD3 training in this invention.

[0034] Figure 8 This is an I-TD3 tracking trajectory according to an embodiment of the present invention.

[0035] Figure 9 This refers to the deviation of I-TD3 in one embodiment of the present invention. Detailed Implementation

[0036] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0037] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0038] A path-following control method for underwater vehicles is proposed. The controlled object of this invention is an underactuated autonomous underwater vehicle. Therefore, the dynamic model of the controlled object and the discrete path-following problem are defined first.

[0039] To fully describe the motion characteristics of the underwater vehicle in three-dimensional space, a northeast-east coordinate system (NED) is used. and body fixed reference frame A dual-coordinate framework is constructed. Based on the typical parameters of REMUS AUV (a series of autonomous underwater vehicles developed by the Woods Hole Oceanographic Institution and later commercially produced by Hydra-Ed), the following six-DOF Fossen model is established.

[0040] The kinematic equations are shown in equation (1):

[0041] (1)

[0042] The dynamic equation is shown in equation (2):

[0043] (2)

[0044] in, This represents the position and attitude vector of the underwater vehicle, where x, y, and z are the position coordinates of the AUV. , , These represent roll angle, pitch angle, and yaw angle, respectively. Represents the linear velocity and angular velocity vectors. For longitudinal velocity, For lateral velocity, For heave speed, For the roll angular velocity, The pitch angular velocity, Yaw angular velocity; This is the coordinate system transformation matrix; The inertia matrix includes the added mass; For the Coriolis centripetal matrix; Here is the fluid dynamics damping matrix; Additional forces caused by gravity and buoyancy; The force and torque caused by the actuator, where, For longitudinal force, It is a lateral force. For heave force, Rolling torque, Pitch moment, Yaw moment, Depends on a given AUV , , , The REMUS AUV is equipped with a cruciform rudder and a rear thruster, in which... and These represent the angles of the rudder and elevator, respectively. Indicates thrust. Control input is subject to... , and Constraints.

[0045] Table 1 summarizes the motion state, related forces and moments, and their coordinate system, while Equation 1 and Figure 1 This demonstrates the coordinate transformation relationship.

[0046] Table 1. Motion state, forces, and torques of the AUV in the reference coordinate system

[0047]

[0048] In summary, the 6-DOF AUV model described in equation (2) is detailed in equations (3) and (4).

[0049] (3)

[0050] (4)

[0051] in and These represent the gravitational force and buoyancy acting on the AUV, respectively. It's about quality; and Representing coordinate systems Coordinates of the center of gravity and the center of buoyancy; , , This represents the corresponding axial moments of inertia. , , , , , , , , , , , , , , , , , , , , , This represents the system's hydrodynamic coefficients. When developing the control model, only the AUV's velocity, angular velocity, position, attitude, thrust, and rudder deflection angle are used.

[0052] It should be noted that the above model forms the basis for training and validating the framework and algorithm proposed in this invention. The intelligent controller developed in this invention does not rely on explicit AUV model equations, but instead employs an end-to-end black-box approach, using only input-output data for path tracking. AUV state information, including position, attitude, and velocity, is assumed to be reliably estimated through onboard navigation (such as INS / DVL), while waypoint sequences are provided by the upstream planner or pre-loaded and can be accessed during execution.

[0053] The control task of this invention is to achieve discrete path following with attitude adjustment objectives. In actual AUV operations, the output of the upstream path planner (including RRT, A*, ACO, PSO, etc.) is typically a sequence of discrete waypoints. The resulting path lacks explicit timestamps or prescribed speed curves. These waypoints are generated by the planner based on factors such as the AUV's motion characteristics, target location, and obstacle distribution, meeting overall mission requirements. Discrete path tracking control refers to guiding the AUV along a planned discrete path. .

[0054] The invention further considers imposing additional attitude requirements at each waypoint to ensure that the vehicle arrives at or passes through a designated location with the desired orientation. Therefore, each waypoint is defined as... ,in, It is a three-dimensional position vector , , These are the coordinates of the waypoints. , , These represent the roll angle, pitch angle, and yaw angle, respectively.Figure 2 The diagram shows a path consisting of a series of waypoints with specific attitude requirements. The AUV needs to pass through each waypoint sequentially with the appropriate attitude. This process can be divided into path segments corresponding to the number of waypoints. In each segment, the AUV travels to the current waypoint with the appropriate attitude, continuing sequentially until the final waypoint is reached.

[0055] Throughout the discrete path tracking process, the motion of the underactuated AUV is controlled by the control input. Adjustments were made to sequentially pass waypoints while meeting specified position and attitude specifications. To quantify waypoint passing performance and establish control objectives, two metrics were defined based on position and attitude: position deviation and position error. and attitude deviation The positional deviation of waypoint k Defined as:

[0056] (5)

[0057] Attitude deviation It is measured by the angle between the unit vector of the direction of the current AUV attitude and the current waypoint attitude, and is defined as:

[0058] (6)

[0059] in:

[0060] ;

[0061] ;

[0062] Therefore, the control objective is to reduce all waypoints under actuator constraints. and .

[0063] The underwater vehicle path-following control method provided by this invention is a complete technical solution executed sequentially according to time. Based on the clarified controlled object model and task definition, the method mainly includes the following steps:

[0064] Step S1: Obtain the discrete waypoint sequence and construct an external tracking framework.

[0065] First, obtain the discrete waypoint sequence generated or preloaded by the upstream path planner. Each waypoint It includes location information and attitude adjustment targets.

[0066] Building upon this, an external tracking framework is constructed to decompose the global discrete path following task into a series of continuous, independent waypoint-to-point tracking subtasks. The core idea of ​​this framework is that the controller only needs to focus on the current target waypoint at any given time. The system determines control actions based on the real-time motion status and relative geometry of the underwater vehicle. Once preset switching conditions are met, the system automatically switches the target to the next waypoint. This mechanism enables the method to track both offline-loaded static paths and dynamic waypoint sequences output in real time by the upstream planner, giving it high engineering adaptability.

[0067] Step S2: Design an improved line-of-sight guidance law to provide guidance constraints.

[0068] To effectively guide an underwater vehicle (AUV) to approach discrete waypoints in a desired attitude, this invention proposes an improved line-of-sight (I-LOS) guidance method specifically designed for discrete waypoint tracking. This method is specifically designed to handle the tracking of a single waypoint with attitude adjustment objectives within a certain time interval. Path tracking is decomposed into waypoint-by-waypoint segments, with each segment corresponding to the tracking of one waypoint. This design enables the AUV to approach each current waypoint with an appropriate attitude. Figure 3 As shown, the discrete path tracking task is completed when the AUV passes through all waypoints in sequence. Step S2 provides the theoretical basis and geometric constraints for the design of the reward function of the subsequent Markov Decision Process (MDP).

[0069] I-LOS geometry using single waypoint information, such as Figure 4 As shown. Definition To connect the AUV quality center to waypoint The line of position, Indicates the distance from the current location of the AUV (i.e., the quality center) to... The line of the central axis, angle Defined as waypoint central axis and The included angle, and The current center axis of the AUV and The included angle. It should be noted that, Figure 4 The two-dimensional diagram shown is a simplification of the three-dimensional diagram; all the lines and angles described are defined in three-dimensional space.

[0070] To ensure that the AUV arrives at the current waypoint with the target position and attitude. ,Right now and Only That's sufficient. Because when all three of the above values ​​are zero, the AUV coincides with the waypoint. Therefore, when the AUV tracks the waypoint... Sometimes, That's enough.

[0071] Step S3: Establish a Markov decision process model that includes the attitude adjustment objective.

[0072] To achieve end-to-end control based on deep reinforcement learning (DRL), this invention models the path following problem as a Markov decision process (MDP), which consists of tuples. Description. Specifically, at time t, the AUV observes the environment to obtain its state. and make decisions Then, according to and It transitions to the next state. and receive a reward The ultimate goal of DRL is to discover an effective control strategy. This enables the accumulation of long-term returns. Maximize. Here, This represents the discount factor, reflecting the importance of future rewards, and also indicates the initial time step of the current event. This step specifically defines the actions involved. ,state and rewards function.

[0073] Step S31: Define the action space A.

[0074] Action space A defines the control inputs directly executed by the actuators. Considering the actuator configuration of the underactuated AUV (horizontal rudder, vertical elevator, rear thruster), the control action is modeled as a combination of rudder angle and thrust, as shown in Equation (8):

[0075] (8)

[0076] in, For rudder angle, For elevator angle, This is the thrust command corresponding to the thruster's rotational speed.

[0077] Step S32: Define the state space S.

[0078] The state space S consists of the AUV's motion state, attitude, relative position between the AUV and waypoints, desired waypoint attitude, and actuator output. To provide the agent with sufficient observation information to make optimal decisions, the state space is constructed as a 17-dimensional vector, as shown in equation (9):

[0079] (9)

[0080] Step S33: Define the reward function R.

[0081] The reward function R is a key signal guiding the agent to learn to meet the control objective. Based on the I-LOS guidance principle described in step S2, this invention incorporates both position deviation and attitude deviation into the reward function framework. At time step t, the above variables... , , , They are respectively represented as , , , Their changes are described in differential form to capture the trend of state improvement:

[0082] (10)

[0083] In path planning, underwater obstacle avoidance is considered, and a safe distance for obstacle avoidance is given. Add an extra variable to the reward function of the DRL-based path tracing controller. For the obstacle avoidance problem, define a reward component. As shown below:

[0084] (11)

[0085] in This indicates the penalty imposed for the collision. Represents time t .

[0086] The reward function is defined as:

[0087] (12)

[0088] in, These are the weight coefficients for each optimization objective. Since most reinforcement learning algorithms use a Gaussian distribution and normalize the action space to... Within range. Action Normalized to , and state Only scaling is applied, with scaling factors shown in Table 2. For clarity, the normalization action and scaling state are represented as follows: and .

[0089] The design of this reward function explicitly includes the attitude adjustment target of waypoints in the optimization scope, which is an important technical feature that distinguishes this invention from existing schemes that only focus on position tracking accuracy.

[0090] Table 2 Scaling of State Variables

[0091]

[0092] Step S4: Train the control policy using the improved dual-delay deep deterministic policy gradient (TD3).

[0093] TD3 adopts an actor-commentator structure and maintains four... The network consists of two commentators and their corresponding objectives, and two policy networks (one actor and its objective). Specifically, the commentator network is parameterized as follows: and Its target network is and The policy network consists of... It indicates that its goal is .

[0094] To address the complex state space of discrete path tracking and the underestimation problem of standard TD3, the improved TD3 (I-TD3) algorithm is improved from three aspects: a two-stage exploration strategy, a reward-based dynamic adjustment two-level priority experience replay (RDTL-PER) mechanism, and a dynamic target Q-value estimation strategy.

[0095] Step S41: Implement a two-stage exploration strategy.

[0096] To address the issue of insufficient sample diversity in the early stages of training, this invention implements a phased exploration mechanism. In the initial training phase (when the number of training steps t is less than the initial random policy steps)... In the later stages of training (when the number of training steps t is greater than or equal to the number of initial random policy steps), uniform random sampling is performed within the action space to enrich the experience buffer and promote the diversity of training samples. In the previous stage, the algorithm transitioned to policy-guided exploration, where actions were generated by a participant network with added Gaussian noise, ensuring a balance between exploration and training stability. Detailed implementation details are as follows... Figure 5 As shown, the mathematical expressions for action generation are given by equations (13) and (14), while equation (15) defines the inverse scaling applied to the action.

[0097] (13)

[0098] (14)

[0099] (15)

[0100] in, and These represent the maximum and minimum actions, respectively, and c represents the constraint on the policy network output, which is usually set to 1. The function is the initial number of random policy steps. This means truncating variable b within the interval. Inside, Indicates distribution, This represents a normal distribution.

[0101] This strategy ensures stable convergence in the later stages of training while significantly improving exploration efficiency and sample quality in the early stages.

[0102] Step S42: Apply the reward-based dynamic two-level priority experience replay (RDTL-PER) mechanism.

[0103] In the early training of the AUV path tracking controller, many unsuccessful events occurred. To improve efficiency, RDTL-PER was designed, where high-reward transitions are prioritized for playback, and this priority gradually decreases until uniform sampling is adopted. Specifically, the system maintains a replay buffer M and two sampling methods, and the ratio of these two methods is dynamically adjusted. M contains a total of There are i samples. The sampling priority of the i-th sample is defined as follows: As shown in equation (16).

[0104] (16)

[0105] in It is a small positive number, ensuring , yes The upper limit, This represents the reward for the i-th sample. The replay pool follows the "first-in, first-out" principle. To improve sampling efficiency, this invention employs a SumTree data structure. Unlike traditional fixed-ratio priority experience replay, this invention uses a dynamic adjustment strategy, drawing N samples in each training iteration, where... It was obtained using Priority Experience Replay (PER). The value is dynamically adjusted based on t.

[0106] (17)

[0107] in express initial value, This represents the final value. To adjust the number of steps for sampling parameters, in time after, . This indicates rounding up. For priority playback in sampling, the original distribution will be altered. Therefore, the Importance Sampling Weight (ISW) technique is applied to correct the gradient of the playback transformation, and the derivation of the correction term is as follows.

[0108] In uniform sampling, the probability of selecting a single sample is:

[0109] (18)

[0110] In contrast, in Priority Experience Replay (PER), the sampling probability is defined as:

[0111] (19)

[0112] Therefore, the ISW correction term is defined as

[0113] (20)

[0114] in Adjust the degree of importance sampling weight to compensate for bias, and update it dynamically as follows.

[0115] (twenty one)

[0116] in, and They are The lower and upper limits, That is the maximum training time.

[0117] It is normalized, and the final value of ISW is given by the following formula.

[0118] (twenty two)

[0119] In summary, RDTL-PER is described by the following pseudocode, as shown in Table 3.

[0120] Table 3 RDTL-PER Pseudocode

[0121]

[0122] Step S43: Execute the dynamic objective Q-value estimation strategy.

[0123] After constructing the experience replay pool M, the actor and commentator networks are trained using the experience from the replay pool. During network updates, it is necessary to estimate the Q-value. A Q-network is used to approximate the action-value function. Specifically, at each time step t, Indicating in strategy Next execution state At that time, the action The expected long-term cumulative reward obtained. Thus, the Q-function approximately satisfies:

[0124] (twenty three)

[0125] To mitigate overestimation bias, the target value is calculated as the minimum of the two target Q-network estimates at each step. While this method effectively suppresses overestimation, it tends to underestimate the value function, leading to an overly conservative bias. To balance this trade-off, a dynamic weighting coefficient is introduced. It adaptively adjusts the target value, as shown in equation (26).

[0126] (twenty four)

[0127] (25)

[0128] (26)

[0129] in This represents the Q-value estimated by the network corresponding to the target policy. For the target value, This represents a set of samples obtained from sampling M. Representation of goal decision network According to sampling status The output action, and It is a time-varying weighting coefficient designed to balance the trend of overestimation in the early stages and underestimation in the later stages.

[0130] Time-varying weighting coefficients The value is determined by the following expression:

[0131] (27)

[0132] in and They represent The initial and final values, where , It is a time control factor.

[0133] Based on the above improvements, within the TD3 framework, agent training is then carried out to update the critic and actor networks. Each critic is trained by minimizing the mean squared error between their current estimate and the target value; the critic's loss function is:

[0134] (28)

[0135] in Indicates taking the expected value. This represents the samples obtained from sampling M, where N is the number of samples in a single sampling.

[0136] Parameters of the two critic networks and Updates are performed while minimizing the loss function using mini-batch gradient descent.

[0137] (29)

[0138] in, The policy network estimates the Q-value based on the sampled state-action pairs. As given in equation (26), This indicates calculating the gradient.

[0139] The optimization objective of the policy network is defined as the cumulative discounted reward. By combining the Q-network and mini-batch sampling from the replay buffer M, the optimization objective (i.e., the objective function) can be expressed as follows:

[0140] (30)

[0141] (31)

[0142] in The states sampled from M, , , …is the single-step reward calculated using the reward function. This is the discount factor.

[0143] For training stability, the policy network, its target network, and the target Q network are not updated in each iteration; they are updated at fixed time intervals. Update. Furthermore, update according to equations (29) and (31). , and Meanwhile, the corresponding target network is updated using the soft update mechanism shown in Equation (32) to further improve training stability.

[0144] (32)

[0145] in This represents the soft update parameter used to determine the target network update rate.

[0146] In summary, the pseudocode for the I-TD3 training process is shown in Table 4, and its overall framework is as follows: Figure 6 As shown.

[0147] Table 4 Training Algorithm for AUV Discrete Path Tracking Controller Based on I-TD3

[0148]

[0149]

[0150] Step S5: Perform real-time control and waypoint switching based on the trained policy network.

[0151] After completing the offline training described in step S4, a converged control policy network is obtained. When actually performing the discrete path following task, the system executes the following sub-steps in a loop.

[0152] Step S51: Observe the current state.

[0153] The current motion status of the AUV is obtained through the onboard navigation equipment, and combined with the current target waypoint. Based on the information, construct the current state vector according to the state space format defined in step S3. .

[0154] Step S52: Strategy reasoning and action execution.

[0155] The state vector Input to the trained policy network The normalized motion output is obtained, and after inverse scaling, the actual control command is obtained. The data is then sent to the underlying actuators to drive the AUV's movement, and the AUV updates its own motion state.

[0156] Step S53: Determine the waypoint switching conditions.

[0157] Since discrete path tracking involves tracking waypoints one by one, it is necessary to define a condition to terminate the tracking of the current waypoint and trigger the transition to the next waypoint. Because the AUV considered in this invention is underactuated, continuing to track the current waypoint is meaningless once it has sufficiently approached or begun to move away from the waypoint. Therefore, when the condition of formula (7) is satisfied between the AUV and the waypoint, the tracking of the current waypoint is considered complete, and the switching requirement is met.

[0158] (7)

[0159] in It is the position of the AUV at time t. The AUV begins tracking waypoints. The location.

[0160] Step S54: Update the target waypoint.

[0161] Once the above formula (7) is satisfied, the system automatically increments the target waypoint index k to k+1. If the end point of the path has been reached, the task ends; otherwise, it returns to step S51 to continue execution. By repeatedly executing the above sub-steps, all waypoints in the path sequence are traversed, and finally, high-precision pose synchronization following of the entire path is achieved.

[0162] To verify the effectiveness of the proposed method, simulation experiments were conducted. The simulation software framework was built using Python 3.12, PyTorch 2.5.1, and CUDA 12.4, and ran on a hardware platform equipped with a 12th generation Intel(R) Core(TM) i5-12490F CPU (3.00 GHz), an NVIDIA GeForce RTX 3060 GPU, and 16.0 GB of RAM. The parameters used in the AUV kinematic model adopted the REMUS AUV hydrodynamic coefficients published in the literature. To more realistically simulate the dynamic behavior of the AUV, the Runge-Kutta integral method was used to update the system state. Table 5 shows the network architecture and hyperparameter settings used in this invention.

[0163] Table 5 Simulation Training Parameter Settings

[0164]

[0165]

[0166] Based on the above parameter settings, follow the path output by the path planner, such as... Figure 7 The reward convergence curve for I-TD3 training shows that the algorithm can converge effectively and remain stable in the later stages.

[0167] Table 6 shows the average position deviation (m) and attitude deviation (rad) at each waypoint switching time during different training phases.

[0168]

[0169] Figure 8 Table 6 shows the embedded I-TD3 controller obtained after training. Figure 3 The performance of tracking the complete path is shown in the example frame. It can be seen that as training progresses, the path tracking performance of the frame improves in terms of both position and pose deviations.

[0170] Therefore, the proposed algorithm can satisfy the requirement of AUV to follow the output path of the planner.

[0171] Finally, it is necessary to state that the above embodiments are only used to further illustrate the technical solution of the present invention in detail, and should not be construed as limiting the scope of protection of the present invention. Any non-essential improvements and adjustments made by those skilled in the art based on the above content of the present invention shall fall within the scope of protection of the present invention.

Claims

1. A path-following control method for an underwater vehicle, characterized in that, Includes the following steps: Step S1: Obtain the discrete waypoint sequence and construct an external tracking framework; The discrete waypoint sequence containing position information and attitude adjustment targets generated by the path planner is obtained, and the external tracking framework decomposes the global discrete path following task into a series of waypoint-to-point tracking subtasks. Step S2: Design an improved line-of-sight guidance law to provide guidance constraints; Step S3: Establish a Markov decision process model that includes the attitude adjustment objective, and define the action space, state space and reward function, wherein the reward function includes position deviation term and attitude deviation term; Step S4: The control policy network is trained offline using an improved dual-delay deep deterministic policy gradient algorithm; The improved dual-delay depth deterministic strategy gradient algorithm includes a two-stage exploration strategy, a reward-based dynamic two-level priority experience replay mechanism, and a dynamic target Q-value estimation strategy. Step S5: Perform real-time control and waypoint switching based on the trained policy network; Using the trained control strategy network, control commands are generated based on the current state of the underwater vehicle to drive the actuators. After the preset waypoint switching conditions are met, the vehicle switches to the next target waypoint until the full path following is completed.

2. The path following control method according to claim 1, characterized in that: In step S2, the improved line-of-sight guidance law constructs guidance constraints through the following geometric relationships: Definition To connect the AUV quality center to the waypoint The line of position, Indicates the distance from the current position of the AUV to... The line of the central axis, angle Defined as waypoint central axis and The included angle, angle The current center axis of the AUV and The angle between the two points, and the waypoints for AUV tracking. Sometimes, Approaching .

3. The path following control method according to claim 2, characterized in that: In step S3, the reward function Defined as: ； ； ； in, Here are the weight coefficients for each optimization objective, and t is the time step. The penalty imposed for collision behavior. Given a safe distance for obstacle avoidance, This indicates that the AUV arrives at time t. Central axis The distance.

4. The path following control method according to claim 1, characterized in that: In step S4, the two-stage exploration strategy includes: in the early stage of training, randomly sampling actions from a uniform distribution to fill the experience replay pool and increase sample diversity; in the middle and late stages of training, transitioning to a policy-guided exploration mode that superimposes Gaussian noise on the output of the policy network.

5. The path following control method according to claim 1, characterized in that: In step S4, the reward-based dynamic two-level priority experience replay mechanism includes: defining a sampling priority for each sample based on the level of reward obtained by the experience sample; and extracting a preset number of samples in each training iteration, wherein the number of samples sampled based on priority is dynamically adjusted and gradually decreases with the number of training steps until it eventually degenerates into uniform sampling.

6. The path following control method according to claim 5, characterized in that: The reward-based dynamic two-level priority experience replay mechanism also includes: applying importance sampling weights to correct the gradient estimation bias introduced by priority replay.

7. The path following control method according to claim 1, characterized in that: In step S4, the dynamic target Q-value estimation strategy includes: introducing a weighting coefficient that adapts to the training time, and calculating the target Q-value as a weighted sum of the minimum and maximum values ​​of the two target Q-network outputs, so as to suppress the value overestimation bias in the early stage of training and overcome the value underestimation bias in the later stage of training.

8. The path following control method according to claim 1, characterized in that: In step S5, the waypoint switching condition is: ； in It is the position of the AUV at time t. The AUV begins tracking waypoints. The position at that time The current waypoint being followed The location.

9. The path following control method according to claim 8, characterized in that: In step S3, the AUV is equipped with a rudder cross and a rear thruster, and the state space is a multi-dimensional vector that contains at least... ,in, , , , , , , , , These are the longitudinal speed, lateral speed, heave speed, roll rate, pitch rate, yaw rate, roll angle, pitch angle, and yaw angle of an AUV. , These represent the AUVs at waypoints. The pitch and yaw angles are given, and x, y, and z are the position coordinates of the AUV in the northeast coordinate system. Indicates thrust. and These represent the angles of the rudder and elevator, respectively.

10. The path following control method according to claim 1, characterized in that: The AUV is an underactuated autonomous underwater vehicle, and its dynamic model adopts a six-degree-of-freedom nonlinear model. The control strategy network is an end-to-end black box network. During the training and inference phases, the control strategy network only uses the observation vector in the state space as input and outputs action commands, without using the hydrodynamic parameters or dynamic equations of the underwater vehicle as network input or as part of the network structure.

Images