Ship autonomous tracking control method integrating deep reinforcement learning and sight line algorithm

Abstract

The invention provides a ship autonomous tracking control method fusing deep reinforcement learning and a sight line algorithm, and belongs to the technical field of ship intelligent control. A cooperative control framework EEATD3 integrating a sight guidance algorithm and drift angle compensation and deep reinforcement learning splits coupling links such as interference compensation and course adjustment in a path tracking problem, so that a deep reinforcement learning controller does not need to undertake a complex adjustment task of global state coupling in path tracking. A rudder angle instruction output strategy is accurately learned only according to real-time states such as deviation between a target course and an actual course. Meanwhile, a composite reward function is designed, various targets such as course error, rudder angle change rate and rudder angle amplitude change are comprehensively considered and verified through experiments, and experimental results show that the method effectively improves the precision and stability of trajectory tracking, balances the smoothness of rudder angle adjustment, and improves the accuracy and stability of trajectory tracking. And under complex disturbance of wind, waves, flow and the like, stronger robustness is shown.

Description

Technical Field

[0001] This invention belongs to the field of ship intelligent control technology, and in particular relates to a ship autonomous tracking control method that integrates deep reinforcement learning and line-of-sight algorithms. Background Technology

[0002] With the increasing prevalence of global trade integration and marine resource development, the technology of various equipment for exploring the ocean is constantly advancing. Unmanned surface vessels (USVs), with their advantages of high autonomy, strong environmental adaptability, and high operational efficiency, have attracted widespread attention due to their potential for application in marine resource development, intelligent shipping, and global trade transportation. In these applications, autonomous trajectory tracking and control is the most fundamental and critical technology. However, due to the strong nonlinearity and coupling characteristics of USVs, as well as the presence of large inertia, model parameter perturbations, and external environmental disturbances such as wind, waves, and currents, drift angles (the angle between the ship's actual velocity direction and its heading) can occur. Achieving high-performance autonomous trajectory tracking and control is by no means easy. Summary of the Invention

[0003] To address the aforementioned issues, this invention proposes a collaborative control framework EEATD3 that integrates line-of-sight guidance (LOS algorithm), drift angle compensation, and deep reinforcement learning. This framework separates the coupled components such as interference compensation and heading adjustment in the path tracking problem, allowing the deep reinforcement learning controller to avoid the complex adjustment tasks of global state coupling in path tracking. Instead, it only needs to accurately learn the rudder angle command output strategy based on real-time states such as the deviation between the target heading and the actual heading.

[0004] This invention proposes a ship autonomous tracking control method that integrates deep reinforcement learning and line-of-sight algorithms, comprising the following steps: Step 1: Construct a Nomoto ship model to simulate the ship's steering characteristics; Step 2: Calculate the target heading angle using the LOS guidance algorithm and compensate for the drift angle of the target heading angle. Step 3: Input the set route, current heading angle, historical heading angle, rudder angle, historical rudder angle data and target heading angle into the EEATD3 network architecture to obtain the real-time ship rudder angle; The EEATD3 network architecture is based on the TD3 reinforcement learning architecture, adopts a hierarchical feature learning network, and integrates a multi-head self-attention feature mechanism and a residual learning module. The EEATD3 network architecture encodes angles in the state space using sine and cosine and integrates historical temporal features to model the temporal dependencies between state features. The EEATD3 network architecture adopts a composite reward function, including heading error rewards, rudder angle rewards, and stability rewards. The EEATD3 network architecture randomly generates training paths and adds velocity perturbations to the Nomoto ship model to simulate the impact of wind, waves, and currents on the ship for model training.

[0005] Preferably, the Nomoto ship model has the following specific formula:

[0006] in, For the ship's heading angle, The first derivative of the ship's heading angle. The second derivative of the ship's heading angle. For rudder angle command, Characterizing a ship's turning ability, The time constant for heading stability is given; after transformation, the current heading angle can be obtained. as follows:

[0007] in, and These represent the heading angle at the previous moment and the heading angles at the two moments before that, respectively. , and These represent the rudder angle at the current moment, the rudder angle at the previous moment, and the rudder angle at the two moments before that, respectively. , and For model hyperparameters; the position of the ship at any given time. and Specifically as follows:

[0008] in, The velocity vector of the ship. This represents the ship's position at time 0.

[0009] Preferably, step 2 calculates the target heading angle using the LOS guidance algorithm. The specific calculation method is as follows:

[0010] in, The aiming point in the LOS algorithm. This indicates the ship's current position.

[0011] Preferably, the drift angle The calculation method is as follows:

[0012] in, The lateral drift speed of the ship, The velocity vector of the ship; Thus, the compensated target heading angle is obtained. :

[0013] in, The target heading angle is calculated using the LOS guidance algorithm.

[0014] Preferably, the EEATD3 network architecture is based on the deep reinforcement learning algorithm framework TD3, mainly composed of an online policy network (Actor) and an online value network (Critic). The Actor network takes the current ship's state information as input, and outputs the ship's control rudder angle after passing through multiple linear layers, normalization layers, and activation functions. The Critic network takes the current ship's state information as input, and extracts corresponding features through three feature extraction networks: heading error, motion state, and historical rudder angle. The current ship's actions are processed by an action feature extraction network to obtain action features. The action feature extraction network consists of linear layers, normalization layers, and activation functions. After concatenation, the action features are processed through linear mapping and a multi-head self-attention mechanism to obtain fused features. The formula for the overall process is as follows:

[0015] in These represent the heading error, motion state, historical rudder angle, and features extracted by the motion network, respectively. They are two learnable tensors. The function is a multi-head attention function. The features are fused; these fused features are concatenated by a feature fusion encoder and an error augmentation bypass to obtain the Q-value in reinforcement learning. A higher Q-value indicates that the action is more consistent with the current situation. The feature fusion encoder consists of a linear layer, a normalization layer, a dropout layer, and an activation function. The error augmentation bypass consists of a linear layer, a normalization layer, and an activation function.

[0016] Preferably, the EEATD3 network architecture specifically includes an online policy network. Two online value networks and And the target network corresponding to the online policy network. The target networks corresponding to the two online value networks and ,in Represents a state. Representative action, These represent the training parameters in the network.

[0017] Online policy network based on current state Get the current optimal action value Add Gaussian noise The details are as follows:

[0018] in, The new state is obtained through interaction with the environment, representing the standard deviation of motion noise. and rewards The obtained data Store in the experience replay pool Then from the experience replay pool Randomly extract batches of data The target policy network is based on Calculate the optimal action and add Gaussian noise. Obtain the target motion value with noise The details are as follows:

[0019] in, The standard deviation of the strategy noise. Policy noise cutoff value; Bi-objective value network and according to and Calculate the expected return and The dual online Critic network is then updated by minimizing the loss function. and The parameters and loss function are as follows:

[0020]

[0021] in, The discount factor is used; the online policy network is responsible for policy optimization, and its optimization objective is to maximize the Q-value predicted by the first online value network.

[0022] Preferably, the EEATD3 network architecture performs sine and cosine encoding on the angles in the state space and fuses historical time-series features, as shown in the following formula:

[0023] in express The bow angle of the ship at any given time. express The target heading angle of the ship at any given time. express The angle of error between the ship's heading angle at a given moment and the target heading angle, indicated by the subscript. They represent The values ​​of each parameter at each time point, express The value of the rudder angle at that moment; The motion space is:

[0024] in, The range is The overall action space is constrained by continuous interval constraints to limit the feasible domain of the rudder angle.

[0025] Preferably, the heading error reward includes a heading error reward and an error change rate reward; The rudder angle rewards include rudder angle amplitude rewards, rudder angle change rate rewards, and additional penalty items; The stability bonus includes: a bonus awarded for the system approaching the target course and maintaining a stable state; and a special bonus set for rudder angle stability. Ultimately, the overall composite reward function constructs a phased control strategy based on the magnitude difference of the current heading error through differentiated weight configuration, thereby achieving dynamic collaborative optimization of each sub-reward item.

[0026] Compared with the prior art, the present invention has the following beneficial effects: This invention proposes a hierarchical feature learning network that integrates multi-head self-attention feature mechanisms and residual learning modules. An enhanced state representation is designed to encode angles in the state space using sine and cosine and fuse historical temporal features, thereby more effectively modeling long-range dependencies between state features and improving the network's learning ability. Finally, a composite reward function is designed, comprehensively considering multiple objectives such as heading error, rudder angle change rate, and rudder angle amplitude change. Experiments are conducted to verify the proposed method, which effectively improves the accuracy and stability of trajectory tracking while balancing the smoothness of rudder angle adjustment. It also exhibits stronger robustness under complex disturbances such as wind, waves, and currents. The main contributions are as follows: 1. A collaborative control architecture integrating LOS guidance, drift compensation and deep reinforcement learning (DRL) is proposed. By integrating prior knowledge and perturbation preprocessing, the learning complexity of DRL is reduced and the sample efficiency and policy interpretability are improved.

[0027] 2. Design an enhanced state representation and hierarchical feature learning network architecture. Use a multi-branch fully connected network (MLP) to decouple heterogeneous features and combine multi-head self-attention mechanism and residual connection to improve the model's representation and generalization ability to dynamic environments.

[0028] 3. Construct a composite reward function to comprehensively optimize heading error, rudder angle change rate, and stability, balancing tracking accuracy and actuator performance. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the ship's motion coordinate system; Figure 2 A schematic diagram of the LOS method with a fixed foresight circle; Figure 3 This is a diagram of a trajectory tracking framework based on deep reinforcement learning (DRL). Figure 4 The overall architecture of the EEATD3 network; Figure 5 To illustrate the tracking performance of the learning method and PID control on different trajectories; Figure 6 This is a diagram illustrating the effect of actual ship path tracking control. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are for illustrative purposes only, and the scope of protection of this invention is not limited thereto.

[0031] I. Constructing the Nomoto Ship Model This invention takes unmanned surface vessels as the research object and selects the Nomoto ship bow roll response model to rudder angle as the interactive environment for reinforcement learning, as detailed below:

[0032] in, For the ship's heading angle, The first derivative of the ship's heading angle. The second derivative of the ship's heading angle. For rudder angle command, Characterizing a ship's turning ability, Let be the heading stability time constant.

[0033] To achieve high-precision simulation of ship heading dynamics in reinforcement learning, the Tustin bilinear transform is used to discretize the heading angle to match the discrete decision step size of RL, preserving the amplitude-frequency characteristics of the continuous system and reducing phase distortion. The specific transfer function is as follows:

[0034] in, Represents a variable in the complex frequency domain. Represents the Laplace transform of the heading angle. This represents the Laplace transform of the rudder angle.

[0035] Then a bilinear transformation is performed, introducing a differential operator substitution, as follows:

[0036] in, This indicates the sampling period. After modification, the heading angle at the current moment can be obtained. as follows:

[0037] in, and These represent the heading angle at the previous moment and the heading angles at the two moments before that, respectively. , and These represent the rudder angle at the current moment, the rudder angle at the previous moment, and the rudder angle at the two moments before that, respectively. , and The model coefficients are as follows:

[0038] in, Characterizing a ship's turning ability, Let be the heading stability time constant.

[0039] A coordinate system is established to describe the ship's motion, as shown in Figure 1. Using a static horizontal plane coordinate system, Using the ship's coordinate system, and The ship's current position. For heading angle, For the bow turning speed, This represents the maximum rudder angle. The ship's position at any given time. and Specifically as follows:

[0040] in, The velocity vector of the ship. This represents the ship's position at time 0.

[0041] II. LOS Guidance like Figure 2 The diagram shows the LOS method with a fixed foresight circle, where... For the total speed of the ship, For lateral error, Forward sight distance. Through radius. 2.5 times the length of the ship The point closest to the target point among the intersections of the forward-looking circle and the trajectory is taken as the aiming point. Calculate the target heading angle Specifically as follows:

[0042] in, The aiming point in the LOS algorithm. This indicates the ship's current position.

[0043] However, in actual navigation, due to interference factors such as ocean currents and sea winds, ships will experience lateral drift. This results in a drift angle. The following dynamic relationship is satisfied:

[0044] in, The lateral drift speed of the ship, This represents the ship's velocity vector. The drift angle compensation method is as follows:

[0045] in, The target heading angle is calculated using the LOS guidance algorithm.

[0046] III. EEATD3 Network Architecture Design 1. Strengthen learning This invention uses the TD3 algorithm with an Actor-Critic architecture to construct a heading angle controller, which includes an online policy network. Two online value networks and And the target network corresponding to the online policy network. The target networks corresponding to the two online value networks and ,in Represents a state. Representative action, These represent the training parameters in the network.

[0047] Online policy network based on current state Get the current optimal action value Add Gaussian noise The details are as follows:

[0048] in, The new state is obtained through interaction with the environment, representing the standard deviation of motion noise. and rewards The obtained data Store in the experience replay pool Then from the experience replay pool Randomly extract batches of data The target policy network is based on Calculate the optimal action and add Gaussian noise. Obtain the target motion value with noise The details are as follows:

[0049] in, The standard deviation of the strategy noise. This is the policy noise cutoff value. (Dual-objective value network) and according to and Calculate the expected return and The dual online Critic network is then updated by minimizing the loss function. and The parameters and loss function are as follows:

[0050]

[0051] in, The discount factor is used. The Actor network is responsible for policy optimization, and its optimization objective is to maximize the Q-value predicted by the first Critic network. Maximizing the Q-value is equivalent to minimizing the negative Q-value, therefore the loss function is as follows:

[0052] The parameters of the target network are updated incrementally using a soft update method, as follows:

[0053]

[0054] in, This is the soft update coefficient.

[0055] 2. Path tracking control system like Figure 3As shown, the trajectory tracking framework based on deep reinforcement learning (DRL) proposed in this invention incorporates prior knowledge of Loss of Orientation (LOS) guidance, mainly comprising a training phase and an inference phase. To simplify learning complexity and utilize mature ship guidance principles, the framework embeds LOS guidance prior knowledge into the environment design: the DRL agent does not need to learn the dual tasks of trajectory planning and heading control from scratch, but instead pre-calculates the target heading based on the USV's real-time position and reference trajectory using LOS guidance. The DRL agent only needs to focus on learning heading tracking capability, i.e., how to adjust rudder angle actions to make the USV's actual heading follow the target heading generated by LOS guidance. In the training phase, the interactive environment is modeled based on Nomoto dynamics, and the initial heading is randomly sampled, with noise introduced to simulate external disturbances; the collected empirical data is stored in an experience replay buffer for iteratively updating the Actor-Critic network, thereby optimizing the control strategy centered on heading tracking. The integration of this LOS guidance prior knowledge greatly simplifies the training task: the agent does not need to explore the complex mapping relationship of "position-target heading", but only needs to focus on the relatively simple "heading-rudder angle" mapping, which significantly reduces the exploration space and speeds up the convergence.

[0056] During the inference phase, the trained agent is applied to the simulation model or the actual ship. The framework further constructs a two-layer architecture based on LOS guidance (core prior knowledge): The first layer is the guidance layer, which directly uses the LOS guidance method to calculate the desired course and generates commands with deviation correction. It adjusts the desired course according to the real-time deviation between the actual course and the target course, which is a logical extension of the LOS prior knowledge. The second layer is the control execution layer, where the DRL controller, specifically trained for course tracking, translates the desired course command into rudder angle actions. In the simulation scenario, speed disturbances are introduced to evaluate the robustness of the course tracking performance. In actual deployment, the rudder angle command directly drives the actuator through closed-loop feedback, ensuring that the USV's actual course stably follows the desired course generated by LOS.

[0057] By incorporating prior knowledge of LOS guidance, the target heading calculation task is separated from the DRL agent, and training is simplified to focus solely on heading tracking. This framework not only reduces the agent's exploration complexity and alleviates the burden of multi-target learning (no need to learn trajectory planning and heading control simultaneously), but also allows the DRL controller to concentrate on optimizing the "heading-rudder angle" mapping relationship. This not only accelerates the convergence speed of the stable control strategy, but also ensures trajectory tracking accuracy based on the reliable principle of LOS guidance, ultimately achieving efficient and accurate USV trajectory tracking.

[0058] 3. Reinforcement Learning Controller Design Because the heading angle has an inherent periodicity, directly using the original angle value as the state input will result in discontinuous numerical jumps when the angle crosses boundaries, increasing the learning difficulty of the neural network and potentially leading to unstable control strategy output or high-frequency chattering. Therefore, this invention employs a trigonometric function coordinate representation, decomposing the original angle value into sine and cosine values ​​as independent state terms. This naturally encodes the periodicity of the angle, ensuring a smooth transition at any position and eliminating the negative impact of numerical discontinuities on network learning. Meanwhile, ship motion exhibits inertial characteristics, and its dynamic processes (such as yaw and turning) depend not only on the current instantaneous state but also on the evolutionary trends of historical states. Relying solely on single-step state information makes it difficult for an agent to accurately distinguish between instantaneous disturbances and continuous dynamic changes. Therefore, this invention introduces historical time-step information from the previous moment into state construction. Through state representations containing temporal context, it provides the DRL agent with key clues to the dynamic evolution of the system, enabling it to implicitly capture the dynamic characteristics of the carrier's motion (such as acceleration, changing trends, and oscillation phase) and environmental disturbance patterns, thereby enhancing the controller's ability to perceive, predict, and adapt to dynamic processes. Therefore, the state space... The specific design is as follows:

[0059] in express The bow angle of the ship at any given time. express The target heading angle of the ship at any given time. express The angle of error between the ship's heading angle at a given moment and the target heading angle, indicated by the subscript. They represent The values ​​of each parameter at each time point, express The value of the rudder angle at any given moment.

[0060] The large inertia, slow response characteristics, and rudder angle saturation nonlinearity of ship maneuvering place stringent requirements on autopilot control. Improper command design can easily lead to actuator saturation, stall, and oscillation, threatening navigation safety and control stability. Therefore, the present invention designs the motion space as follows:

[0061] in, The range is By limiting the feasible domain of the rudder angle through continuous interval constraints, the feasibility of commands is ensured and control efficiency is improved.

[0062] To achieve high-precision, low-fluctuation, and robust control of rudder angle by an intelligent agent in ship heading control, and to balance rudder angle stability and heading tracking accuracy, this invention constructs a multi-factor comprehensive reward function system containing three types of sub-functions. .

[0063] Error-related rewards include heading error rewards and error change rate rewards. The heading error reward focuses on the deviation between the current and target headings, employing a "larger deviation, smaller reward" mechanism to force the agent to prioritize reducing deviations. Simultaneously, it utilizes the smoothing properties of an exponential function to avoid strategy oscillations caused by discrete penalties, aligning with the ship's gradual heading adjustment needs. The error change rate reward, through directional constraints and deviation thresholds, only rewards are given when the error change direction is correct and the current error is large. This accelerates adjustments when deviations are significant and emphasizes control precision when deviations are small, avoiding rudder angle oscillations during small error phases. Specifically:

[0064]

[0065]

[0066] in The angle representing the error between the ship's heading angle and the target heading angle. This represents the change in the error angle. The function represents a symbolic function.

[0067] Rudder angle rewards include rudder angle amplitude rewards, rudder angle change rate rewards, and additional penalties. The rudder angle amplitude reward uses the current rudder angle as input and encourages the use of smaller rudder angles to avoid excessive maneuvers. The rudder angle change rate reward is based on the change in rudder angle between adjacent time steps, guiding towards smooth control. The additional penalty applies punishment for sudden rudder angle reversals to ensure control stability. Similar to error rewards, rudder angle rewards employ a dynamic weighting adjustment strategy to optimize control performance at different heading error stages, as detailed below:

[0068]

[0069]

[0070]

[0071] in Indicates the current rudder angle. This indicates the amount of change in the rudder angle.

[0072] The stability bonus consists of two parts: the first is the bonus awarded when the system approaches the target heading and maintains a stable state; the second is a specific bonus for rudder angle stability, as detailed below:

[0073]

[0074] Finally, the total reward function constructs a phased control strategy based on the magnitude difference of the current heading error through differentiated weight configuration, thereby achieving dynamic collaborative optimization of each sub-reward item, as follows:

[0075] In the application of reinforcement learning to ship path tracking control, the method of state information processing significantly impacts algorithm performance. Addressing the limitations of traditional Actor-Critic architectures in feature processing, this invention proposes an improved Actor-Critic architecture. This architecture significantly enhances control performance through structured feature processing modules and network optimization. Its overall structure is as follows: Figure 4 As shown.

[0076] To enhance the algorithm's feature learning ability while avoiding increased computational costs during deployment, the core improvement strategy revolves around the Critic network. Specifically, performance optimization is achieved through the decoupling of the feature extraction module and the self-attention mechanism. The former focuses on efficiently separating and capturing key features, while the latter specifically strengthens the modeling of relationships between features.

[0077] IV. Experimental Results and Analysis To verify the effectiveness of the proposed algorithm, experiments were conducted in both a simulation environment and a real-ship experiment. Regarding the parameter configuration of the reinforcement learning architecture, both the Actor and Critic networks adopted... A dual-hidden-layer structure is used, with a uniform learning rate of 0.0001 to ensure the synchronization of parameter updates between the two networks. Soft update coefficients are employed. Set it to 0.002, combined with a discount factor of 0.95. Effectively balances immediate rewards and long-term benefits. Motion noise standard deviation. With the standard deviation of policy noise Set them to 0.2 and 0.1 respectively, and set the policy noise cutoff value. The value is limited to 0.5 to suppress parameter oscillations while ensuring the policy's exploratory nature. Policy updates employ a two-step delay mechanism, and the experience replay buffer capacity is set to... Each training iteration randomly selects 128 samples in a batch. The training process is set to a maximum of 10,000 rounds and a maximum of 300 steps per round. This dual constraint ensures the algorithm's convergence stability and training efficiency.

[0078] Regarding the initial setup of the simulation environment, the initial heading angle of the ship is defined as follows: Initial speed is and give the initial direction as The initial size is The drift speed, and add a range of values ​​to the drift speed. speed disturbance and The directional perturbation was tested on Track1 and Track2 respectively.

[0079] In this experiment, the performance of each model was verified by comparing the root mean square error (RMSE) and mean absolute error (MAE) of cross track error and heading error, as well as the mean rudder angle and mean rudder rate during the tracking process. Furthermore, to comprehensively evaluate the overall capabilities of each model, the following comprehensive evaluation indicators were designed:

[0080] in, and These represent the root mean square errors of the lateral error and the heading error, respectively. and These represent the mean rudder angle amplitude and the mean rudder angle change rate, respectively. , , and These are the maximum acceptable lateral error, heading error, maximum rudder angle amplitude, and maximum acceptable rudder angle change rate, respectively, with values ​​of 10m, 5°, 35°, and 3° / s.

[0081] To comprehensively evaluate the performance of the EEATD3 algorithm, it was compared with classic PID control and DDPG across trajectories. The results are shown in Table 1 and 2. Figure 5 Data shows that it has advantages in path tracking accuracy, servo operation efficiency and overall control performance. Table 1. Actual parameters of different methods on different trajectories.

[0082] In terms of tracking accuracy, EEATD3 shows a significant improvement over PID: Track 1 lateral error RMSE decreased from 17.36 to 13.42, a reduction of 22.7%; heading error MAE decreased from 10.72 to 5.65, a reduction of 47.3%; Track 2 lateral error MAE improved from 9.35 to 8.05, a reduction of 13.9%. Compared to DDPG, EEATD3 has obvious advantages: Track 1 lateral error RMSE of 13.42 is lower than DDPG's 15.66, and heading error RMSE of 14.28 is 18.9% lower than DDPG's 17.60; Track 2 lateral error RMSE of 17.75 is 11.8% lower than DDPG's 20.12, and heading error MAE of 6.26 is only 52.5% of DDPG's 11.92. In terms of servo operating efficiency, EEATD3 has its own characteristics compared with PID and DDPG: its average rudder angle amplitude is 7.21° on both tracks, higher than PID's 5.12°, but less than 50% of DDPG's 15.15°; on Track 2, it is 6.37°, higher than PID's 4.89°, but also less than 50% of DDPG's 15.36°. The average rudder angle change rate is 3.94° / s, slightly higher than PID's 3.06° / s, but lower than DDPG's 4.26° / s, balancing operational smoothness and control flexibility.

[0083] In terms of overall control performance, the CEI of EEATD3 performs excellently on both tracks. Track 1 is 1.62, which is 21.0% lower than PID's 2.05 and 17.3% lower than DDPG's 1.96. Track 2 is 2.13, which is 17.4% lower than PID's 2.58 and 13.1% lower than DDPG's 2.45, fully demonstrating its superior overall performance.

[0084] In real-world marine environments, control systems must simultaneously address multiple uncertainties, including wave disturbances, sensor measurement noise, actuator response delays, and model parameter mismatches. To comprehensively evaluate the generalization capability of the proposed cooperative guidance architecture under actual conditions, this invention completed the engineering deployment of the EEATD3 controller on an experimental vessel and conducted systematic real-ship path tracking tests in open sea areas. The test routes included typical scenarios such as straight-line tracking, high-angle curve maneuvering, adaptation to both upstream and downstream conditions, and suppression of cross-current interference. The test site and results are as follows: Figure 6 As shown, the ship is approximately 35m long.

[0085] During the testing phase, a GPS / IMU integrated navigation system was used to collect the ship's attitude information in real time and simultaneously monitor the rudder's operating status. Key indicators such as lateral error and heading error were obtained after data processing. Detailed core performance indicators for each stage are shown in Table 2. In the small drift angle straight-line control scenario, the controller demonstrated excellent precision control capabilities, with lateral trajectory errors (RMSE) as low as 4.04 meters and 8.65 meters, and heading errors (RMSE) as low as 0.50° and 0.49°. However, in the turning and high drift angle straight-line conditions, due to the increased complexity of ship motion, the various error indicators increased. Of particular note is that the maximum average rudder angle under all operating conditions occurred during the large drift angle straight-line phase, with a value of only 2.93°, indicating that the controller has low load requirements on the actuators and possesses good engineering applicability.

[0086] Table 2 Core parameters for each stage of the live-ship experiment

[0087] The EEATD3 controller proposed in this invention constructs an innovative hierarchical feature processing architecture, which collaboratively enhances ship trajectory tracking performance through three key modules: a feature decoupling and embedding module using an independent fully connected network to achieve efficient extraction of heterogeneous features such as pose error and error derivative; an interactive modeling enhancement module based on a multi-head self-attention mechanism to accurately capture the long-range dependency between rudder angle and heading; and an error enhancement mechanism using a bypass structure to dual-focus on key control objectives. Real-ship verification results show that the controller achieves high-precision tracking under straight-line conditions with small drift angles while maintaining low actuator load operation, and also exhibits good convergence stability under steering operations and complex conditions with high drift angles.

[0088] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0089] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A ship autonomous tracking control method integrating deep reinforcement learning and gaze-only algorithm, characterized in that, Includes the following steps: Step 1: Construct a Nomoto ship model to simulate the ship's steering characteristics; Step 2: Calculate the target heading angle using the LOS guidance algorithm and compensate for the drift angle of the target heading angle. Step 3: Input the set route, current heading angle, historical heading angle, rudder angle, historical rudder angle data and target heading angle into the EEATD3 network architecture to obtain the real-time ship rudder angle; The EEATD3 network architecture is based on the TD3 reinforcement learning architecture, adopts a hierarchical feature learning network, and integrates a multi-head self-attention feature mechanism and a residual learning module. The EEATD3 network architecture encodes angles in the state space using sine and cosine and integrates historical temporal features to model the temporal dependencies between state features. The EEATD3 network architecture adopts a composite reward function, including heading error rewards, rudder angle rewards, and stability rewards. The EEATD3 network architecture randomly generates training paths and adds velocity perturbations to the Nomoto ship model to simulate the impact of wind, waves, and currents on the ship for model training.

2. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: The specific formula for the Nomoto ship model is as follows: in, For the ship's heading angle, The first derivative of the ship's heading angle. The second derivative of the ship's heading angle. For rudder angle command, Characterizing a ship's turning ability, The time constant for heading stability is given; after transformation, the current heading angle can be obtained. as follows: in, and These represent the heading angle at the previous moment and the heading angles at the two moments before that, respectively. , and These represent the rudder angle at the current moment, the rudder angle at the previous moment, and the rudder angle at the two moments before that, respectively. , and For model hyperparameters; the position of the ship at any given time. and Specifically as follows: in, The velocity vector of the ship. This represents the ship's position at time 0.

3. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: Step 2 calculates the target heading angle using the LOS guidance algorithm. The specific calculation method is as follows: in, The aiming point in the LOS algorithm. This indicates the ship's current position.

4. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: The drift angle The calculation method is as follows: in, The lateral drift speed of the ship, The velocity vector of the ship; Thus, the compensated target heading angle is obtained. : in, The target heading angle is calculated using the LOS guidance algorithm.

5. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: The EEATD3 network architecture is based on the deep reinforcement learning algorithm framework TD3, and mainly consists of an online policy network Actor and an online value network Critic. The Actor network takes the current state information of the ship as input, and outputs the control rudder angle of the ship after passing through multiple linear layers, normalization layers and activation functions. The Critic network takes the current ship's state information as input and extracts corresponding features through three feature extraction networks: heading error, motion state, and historical rudder angle. The current ship's actions are processed by an action feature extraction network to obtain action features. The action feature extraction network consists of linear layers, normalization layers, and activation functions. After concatenation, the action features are processed through linear mapping and a multi-head self-attention mechanism to obtain fused features. The formula for the overall process is as follows: in These represent the heading error, motion state, historical rudder angle, and features extracted by the motion network, respectively. They are two learnable tensors. The function is a multi-head attention function. It is a feature of fusion; The fused features are concatenated by a feature fusion encoder and an error augmentation bypass to obtain the Q-value in reinforcement learning. A higher Q-value indicates that the action is more consistent with the current situation. The feature fusion encoder consists of a linear layer, a normalization layer, a dropout layer, and an activation function. The error augmentation bypass consists of a linear layer, a normalization layer, and an activation function.

6. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 5, characterized in that: The EEATD3 network architecture specifically includes an online policy network. Two online value networks and And the target network corresponding to the online policy network. The target networks corresponding to the two online value networks and ,in Represents a state. Representative action, These represent the training parameters in the network; Online policy network based on current state Get the current optimal action value Add Gaussian noise The details are as follows: in, The new state is obtained through interaction with the environment, representing the standard deviation of motion noise. and rewards The obtained data Store in the experience replay pool Then from the experience replay pool Randomly extract batches of data The target policy network is based on Calculate the optimal action and add Gaussian noise. Obtain the target motion value with noise The details are as follows: in, The standard deviation of the strategy noise. Policy noise cutoff value; Bi-objective value network and according to and Calculate the expected return and The dual online Critic network is then updated by minimizing the loss function. and The parameters and loss function are as follows: in, The discount factor is used; the online policy network is responsible for policy optimization, and its optimization objective is to maximize the Q-value predicted by the first online value network.

7. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: The EEATD3 network architecture performs sine and cosine encoding on angles in the state space and fuses historical time-series features, as shown in the following formula: in express The bow angle of the ship at any given time. express The target heading angle of the ship at any given time. express The angle of error between the ship's heading angle at a given moment and the target heading angle, indicated by the subscript. They represent The values ​​of each parameter at each time point, express The value of the rudder angle at that moment; The motion space is: in, The range is The overall action space is constrained by continuous interval constraints to limit the feasible domain of the rudder angle.

8. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: The heading error reward includes heading error reward and error change rate reward; The rudder angle rewards include rudder angle amplitude rewards, rudder angle change rate rewards, and additional penalty items; The stability bonus includes: a bonus awarded for the system approaching the target course and maintaining a stable state; and a special bonus set for rudder angle stability. Ultimately, the overall composite reward function constructs a phased control strategy based on the magnitude difference of the current heading error through differentiated weight configuration, thereby achieving dynamic collaborative optimization of each sub-reward item.

Images