A ship autonomous tracking control method fusing deep reinforcement learning and line-of-sight algorithm

Abstract

This invention provides an autonomous tracking control method for unmanned surface vessels (USVs) that integrates Kalman filtering and deep reinforcement learning, belonging to the field of intelligent ship and autonomous navigation control technology. Addressing issues such as increased drift angle and frequent steering in real-world marine environments, it constructs a control framework that combines Kalman filtering-assisted line-of-sight guidance with deep reinforcement learning. By introducing a Kalman filtering-based line-of-sight guidance algorithm to predict and compensate for drift angle in real time, a target heading is generated, ensuring the USV's speed and direction align with the desired path. A deep reinforcement learning controller learns and outputs the optimal rudder angle control strategy based on state information such as the deviation between the target heading and the actual heading. A composite reward function is designed that considers heading error, drift angle changes, and rudder angle stability, significantly suppressing the frequent steering issues found in traditional control methods. This invention outperforms existing traditional control technologies in terms of track tracking accuracy, rudder angle adjustment smoothness, and robustness under complex sea conditions.

Description

Technical Field

[0001] This invention belongs to the field of ship intelligent control technology, and in particular relates to a ship autonomous tracking control method that integrates deep reinforcement learning and line-of-sight algorithms. Background Technology

[0002] With the continuous advancement of artificial intelligence, automatic control, communication and navigation, and sensor technologies, intelligent ship technology has become an important direction for the development of modern ship technology. Intelligent ships not only pursue improved navigation efficiency but also emphasize navigation safety, autonomous decision-making, environmental perception, and mission execution capabilities. Especially in harsh sea conditions, complex waterways, and near-shore maneuvering scenarios, traditional human-based control methods are increasingly unable to meet the requirements of high precision and high reliability. Therefore, the development of autonomous tracking control technology has become a key component of intelligent ship technology.

[0003] In intelligent navigation and control systems for ships, autonomous tracking control technology can generally be divided into two core components: tracking algorithms and control algorithms. The main function of the tracking algorithm is to calculate the ideal path or course information that the ship should follow during navigation based on preset mission objectives and environmental constraints, serving as the system's reference input. The control algorithm, on the other hand, calculates control commands in real time based on the dynamic deviation between the ship's current motion state (such as position, heading, and speed) and the reference trajectory. This allows for the adjustment of actuators such as rudder angles, thereby achieving precise control of the ship's motion and ensuring stable and accurate tracking of the target trajectory in dynamic environments.

[0004] As research progresses, researchers are using deep learning as a tool to improve control performance in complex sea conditions and nonlinear systems. One approach is to improve upon traditional algorithms such as PID controllers, including using deep reinforcement learning to adaptively select relevant PID parameters. To save time in adjusting PID parameters, some studies have introduced an improved simulated annealing algorithm. The overall controller design can overcome external disturbances such as wind, waves, and currents. Another approach is to directly construct a deep learning controller, including using a three-degree-of-freedom ship model and a deep Q-network to optimize tracking accuracy. A near-end strategy optimization algorithm was implemented for the ship tracking problem, which, combined with a line-of-sight guidance system, ensures that the ship follows a predetermined path while also improving the ship's adaptability to unknown and complex environments.

[0005] Although some progress has been made in the research on ship tracking control algorithms at home and abroad, experiments have revealed two problems with existing ship tracking methods. First, the control accuracy of traditional methods is not high and there is a problem of frequent ruddering. Second, in actual sea areas, ships are affected by wind and waves, especially when the ship's course is perpendicular to the direction of the water flow. During the tracking process, the ship will deviate, that is, the difference between the course angle and the heading angle increases, resulting in a large lateral error. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a ship autonomous tracking control method that integrates deep reinforcement learning and Kalman filtering, comprising the following steps: Step 1, construct a first-order kt ship model to simulate the ship's steering characteristics; Step 2: Using the current target route as input, calculate the target heading angle using the KLOS algorithm; The KLOS algorithm uses Kalman filtering to model the drift angle and adds a drift angle factor to the LOS algorithm to correct the ship's speed direction. Step 3: Input the current heading angle, historical heading angle, historical rudder angle, and historical rudder angle data along with the target heading angle into the TD3 reinforcement learning network architecture to obtain the real-time ship rudder angle; The TD3 reinforcement learning network architecture adopts a composite reward function, which constructs rewards from three aspects: heading reward, heading change rate reward, and rudder angle reward. The TD3 reinforcement learning network architecture randomly generates training paths and gradually increases the training difficulty using a course-based learning method.

[0007] Preferably, the first-order kt model of the ship has the following specific formula:

[0008] in, For the ship's heading angle, The first derivative of the ship's heading angle. The second derivative of the ship's heading angle. For rudder angle command, Characterizing a ship's turning ability, The time constant for heading stability is given; after transformation, the current heading angle is obtained. as follows:

[0009] in, and These represent the heading angle at the previous moment and the heading angles at the two moments before that, respectively. , and These represent the rudder angle at the current moment, the rudder angle at the previous moment, and the rudder angle at the two moments before that, respectively. , and For model hyperparameters; the position of the ship at any given time. and Specifically as follows:

[0010] in, The velocity vector of the ship. Let be the position of the ship at time t. For the ship's heading angle, For a time slice.

[0011] Preferably, in step 2, the target heading angle is calculated using the KLOS algorithm with the current target route as input. First, the LOS angle is calculated using the LOS algorithm, and the calculation formula is as follows:

[0012] in, This refers to the aiming point position in the LOS algorithm. The ship's current position; Then calculate The drift angle of the ship at that moment is:

[0013] in, The lateral drift speed of the ship, Let be the ship's velocity vector; let the drift angle and the rate of change of drift angle be used as state vectors, and update these two physical quantities. The formula for the update process is as follows:

[0014] in for The prior state estimate at time t. for Predicted value at time, Let be the state transition matrix, set as a discrete first-order integral model, indicating that the current drift angle can be obtained from the previous drift angle and the rate of change of drift angle; express The posterior estimate of the covariance at time t. for The prior estimate of covariance at time t, Indicates random noise; In the prediction phase, the specific formula is as follows:

[0015] in This is the Kalman gain matrix, used to measure the contribution of the current state value to the correction of the state estimate. These are the current observations. It is the transformation matrix from the state vector to the measurement vector. It is the covariance matrix of the measured noise; The overall rudder angle and rate of change are input into a Kalman filter. While smoothing the input, the output drift angle value is corrected in real time based on the current drift angle observation matrix. This drift angle value is then used to generate the ship's target heading angle in real time. For time The revised formula is as follows:

[0016] in They represent Predicted values ​​of drift angle and rate of change of drift angle at time. This represents the target heading angle output by the KLOS algorithm.

[0017] Preferably, the TD3 reinforcement learning network architecture is based on the TD3 deep reinforcement learning algorithm framework and consists of an online policy network (Actor) and an online value network (Critic). The Actor network takes the current state information of the ship as input, and outputs the actual control rudder angle of the ship after passing through multiple fully connected layers and activation functions. The Critic network takes five features as input: the current heading angle, the historical heading angle, the historical rudder angle, the current rudder angle data, and the target heading angle. After passing through multiple fully connected layers and activation functions, it outputs a score for the judgment of the current rudder angle action. The higher the score, the better the current rudder angle action is compared to the overall system.

[0018] Preferably, the TD3 reinforcement learning network architecture encodes angles in the state space and fuses historical temporal features, as shown in the following formula:

[0019] in express The bow angle of the ship at any given time. express The target heading angle of the ship at any given time. express The angle of error between the ship's heading angle at a given moment and the target heading angle, indicated by the subscript. They represent The values ​​of each parameter at each time point, express The value of the rudder angle at that moment; The motion space is:

[0020] in, The range is The overall action space takes each integer rudder angle as the output based on the feasible domain of the rudder angle.

[0021] Preferably, the heading error reward includes a heading error reward and an error change rate reward; The rudder angle rewards include rudder angle amplitude rewards, rudder angle change rate rewards, and additional penalty items; The stability bonus includes: a bonus awarded for the system approaching the target course and maintaining a stable state; and a special bonus set for rudder angle stability. Ultimately, the overall composite reward function constructs a phased control strategy based on the magnitude difference of the current heading error through differentiated weight configuration, thereby achieving dynamic collaborative optimization of each sub-reward item.

[0022] Preferably, the TD3 reinforcement learning network architecture randomly generates training paths and uses a course learning method to gradually increase the training difficulty; it uses the ship's turning angle as a random training target, starts training with small-amplitude turns, and gradually increases the ship's turning angle after the results stabilize, finally obtaining a final model that can adapt to all angles.

[0023] Compared with the prior art, the present invention has the following beneficial effects: The algorithm of this invention first obtains the target heading angle using a given path and the KLOS algorithm, and then uses a deep reinforcement learning algorithm to provide the rudder angle required by the vessel at this time. In the overall architecture, the improved line-of-sight guidance algorithm KLOS can predict the future drift angle and adjust the control commands before the vessel deviates significantly, so that the vessel's speed direction, rather than its bow, is aligned with the path. This is applicable to situations affected by external factors such as crosswinds and currents, increasing the robustness of the overall algorithm. In addition, the invention utilizes deep reinforcement learning to design relevant reward functions for factors such as changes in drift angle, alleviating the problem of frequent rudder adjustments. Extensive experiments were conducted on simulation platforms and actual fishing vessels to verify the correctness and effectiveness of the algorithm in practical engineering applications. Attached Figure Description

[0024] Figure 1 Here is a schematic diagram of the ship's motion coordinate system; (a) without drift angle; (b) with drift angle. Figure 2 A diagram illustrating the overall framework of a ship's autonomous tracking control algorithm; Figure 3 The overall framework of the TD3 network; Figure 4 The following are actual control charts for three angles as target heading angles in the embodiment; where (a) is the actual control chart for 30 degrees; (b) is the actual control chart for 60 degrees; and (c) is the actual control chart for 120 degrees. Figure 5 The image shows the performance of the traditional LOS algorithm and the KLOS algorithm on the same line of sight. Figure 6 This is a diagram showing the tracking performance of the final model RL_all in this invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are for illustrative purposes only, and the scope of protection of this invention is not limited thereto.

[0026] Example 1: I. Construction of Mathematical Model for Ships This invention simplifies the ship model into a dynamic model and a steering model. This invention establishes the ship's dynamic model by setting up a ship motion coordinate system, such as... Figure 1 As shown in the figure, Using a static horizontal plane coordinate system, For the first Target route segment It refers to the ship's heading. It is the direction of the ship's speed. For the ship's heading angle, The velocity direction angle, The deviation between the ship's heading angle and velocity angle is denoted as the drift angle in this paper. Figure (a) shows a simplified motion model where the ship's velocity and heading coincide, i.e., the drift angle is 0. Figure (b) shows the case where the ship's drift angle is not 0, which in reality is usually caused by wind and waves in the external environment. The ship's position at any given time... and It can be calculated using the following recursive model: (1) The ship's simulated turning model is based on the k-t model and calculates the ship's current heading angle through a recursive relationship. The model uses a discretized recursive formula to calculate the ship's heading angle, with a range of values ​​ranging from [value missing]. The formula is as follows: (2) in, , and These are the ship's heading angles at the previous moment, two moments before that, and the current moment, respectively. , and These are the rudder angle values ​​at the previous moment, the rudder angle values ​​at the two moments before, and the current rudder angle value, respectively. , and These are the coefficients of the discrete model.

[0027] II. Design of Autonomous Tracking Control Algorithm for Ships The overall framework of the ship autonomous tracking control algorithm proposed in this invention is as follows: Figure 2 As shown. The initial input to the algorithm is the target flight path. The sequence of target points is fed into the KLOS guidance algorithm to calculate and generate the desired heading angle. Then generate the desired heading angle And the return value of the ship's rudder angle model: current heading angle. The heading angle difference was calculated. This is used as input to the reinforcement learning algorithm, which then calculates and outputs the rudder angle via the reinforcement learning controller. This allows the actual heading angle to converge towards the desired heading angle. The simulation experiment uses an unmanned vessel rudder angle model to reflect the turning characteristics of an actual ship.

[0028] 1. KLOS Algorithm In traditional LOS methods, the target heading angle at the current point is obtained using the fixed forward look distance method. The fixed forward sight distance method is usually based on the ship's length. Draw a forward-looking circle two to three times the length of the ship, intersecting the trajectory. Select the point closer to the target point as the aiming point. Calculate the target heading angle from the forward viewpoint. The specific formula is as follows: (3) In the process of establishing the simulation model, this invention uses the kt model to calculate the ship's heading and makes a simple assumption about the ship's velocity direction. With bow direction Similarly, the overall LOS algorithm diagram is as follows: Figure 1 As shown in (a), where It refers to the first The starting and ending points of each segment. However, in actual ship experiments, it was found that the ship's heading and speed direction did not coincide, as shown in the diagram below. Figure 1 As shown in (b). This phenomenon is mainly due to two reasons: First, the ship is affected by the surrounding wind and waves, causing the drift angle to occur. Second, when the ship turns, due to the ship's inertia, the heading direction always changes significantly first, and then the speed direction changes. This leads to instability in the direct control methods of speed direction or heading, further causing the ship to frequently steer.

[0029] In practical applications on ships, all sensors are inevitably affected by environmental interference and hardware errors, resulting in noise in the measurement data. This noise makes the raw data fluctuate greatly and unstable, making it difficult to use directly for control or decision-making.

[0030] To address drift angle and sensor interference issues, this invention proposes an improved LOS algorithm based on Kalman filtering, the KLOS algorithm. This algorithm uses Kalman filtering to model the drift angle and adds the influence of the drift angle to the LOS algorithm, thereby correcting the ship's velocity direction. The Kalman filter can not only correct the state based on current observations but also predict the drift angle at the next moment based on the system model. This is crucial for systems requiring proactive responses, such as autonomous tracking.

[0031] This invention uses the drift angle and the rate of change of drift angle as state vectors to update and predict these two physical quantities. The formula for the update step is as follows: (4) in for The prior state estimate at time t. for Predicted value at time, The state transition matrix is ​​set as a discrete first-order integral model in this invention, indicating that the current drift angle can be obtained from the previous drift angle and the rate of change of drift angle. express The posterior estimate of the covariance at time t. for The prior estimate of covariance at time t, This represents random noise.

[0032] In the prediction phase, the specific formula is as follows: (5) in This is the Kalman gain matrix, used to measure the contribution of the current state value to the correction of the state estimate. These are the current observations. It is the transformation matrix from the state vector to the measurement vector. It is the covariance matrix of the measured noise.

[0033] By inputting the overall rudder angle and rate of change into a Kalman filter, the input can be smoothed while the output drift angle value is corrected in real time based on the current drift angle observation matrix. This drift angle value is then used to generate the ship's target heading angle in real time. For time The revised formula is as follows: (6) in They represent Predicted values ​​of drift angle and rate of change of drift angle at time.

[0034] The KLOS algorithm can predict future drift angles. Before the ship deviates significantly, it can adjust control commands and increase the predicted drift angle to correct the LOS target point calculation, so that the ship's speed direction, rather than its bow direction, is aligned with the path. It is suitable for situations affected by external factors such as crosswinds and currents, and increases the overall robustness of the algorithm.

[0035] 2. Deep reinforcement learning algorithm TD3 (Twin Delayed Deep Deterministic Policy Gradient) is a reinforcement learning algorithm particularly suitable for decision-making problems in continuous action spaces. In the scenario of ship tracking, TD3 can learn a policy through interaction with the environment, enabling the unmanned vessel to make appropriate rudder angle control based on the current state, thereby navigating stably and accurately along the target trajectory.

[0036] TD3 is an Actor-Critic architecture algorithm that includes a policy (Actor) network. Two value (Critic) networks and and their corresponding target networks , and The Critic network is used for value assessment. Two Critic networks use the same mathematical objective but have different parameter initializations. The Actor network is responsible for policy optimization, and its optimization objective is to maximize the predictions of the first Critic network. Value, the overall architecture of the TD3 network is as follows Figure 3 As shown. Based on the TD3 reinforcement learning framework, this invention designs a corresponding reward function to address problems such as frequent rudder input and unstable control, thereby reducing the lateral error of the ship's tracking while minimizing rudder input, thus achieving precise control and tracking.

[0037] To address the specific needs of ships, this invention first designs a reinforcement learning state space, incorporating the target heading angle, current heading angle, heading angle error, and rudder angle value as fundamental features. Furthermore, to capture the system's dynamic characteristics and changing trends, historical data is added to each current angle. The current data and historical data together constitute the overall state space. : (7) in Indicates the current heading angle. Indicates the target heading angle. Indicates the difference between the current heading angle and the target heading angle, using the subscript. express The values ​​of each parameter at any given time.

[0038] For the action space in reinforcement learning In the actual control of ships, this invention typically involves the rudder angle. To implement control, the range of rudder angle values ​​is determined based on the actual characteristics of the ship. To achieve more stable control and prevent frequent steering errors, the output of reinforcement learning... Take the integer as the output result; the overall action space. As shown below: (8) To achieve precise and smooth rudder angle control, a comprehensive reward function was designed based on the TD3 architecture. It consists of three sub-reward functions, and is designed for heading error. Different reward strategies are used to control the rudder angle at different stages.

[0039] First, in order to minimize the angle error as quickly as possible, the first reward It is constructed using an exponential function, where This represents the error between the current heading and the target heading. The range of values ​​is The overall formula is as follows: (9) This award encourages agents to quickly reduce heading errors.

[0040] In addition to reducing angular error, the error rate of the actual system was further considered to reward effective error reduction in a more refined manner. This represents the rate of change of heading error, and this reward is only given when the direction of error change is correct and the current error is large. This invention uses... The judgment condition is expressed as follows: (10) in This represents the sign function. Based on... This invention provides different reward strategies for different error ranges, and the specific calculation formulas are as follows: (11) In addition to rapidly reducing angular errors, a reward system for rudder angle usage was designed. Based on experience in manually maneuvering the ship, the overall navigation process should avoid using large rudder angles and instead use small rudder angles as much as possible. Large rudder angles are subject to exponential penalties. The overall formula is as follows: (12) To achieve smooth control, penalties are applied to large changes in rudder angle, and moderate penalties are applied to moderate changes in rudder angle, resulting in a smoother rudder angle change curve during ship turns. The overall formula for the reward, which is the difference in rudder angle values ​​between two adjacent time steps, is as follows: (13) In addition, a stability bonus is given when the ship is in a steady state. That is, when the angular error is small, the ship is encouraged to avoid ruddering as much as possible to maintain stability. The overall formula for the bonus is as follows: (14) Based on the above three types of rudder angle-related rewards, the rudder angle-related reward formula is as follows: (15) Based on the magnitude of the current heading error, different weight configurations are used to form a phased control strategy, and the total reward function is shown below: (16) The overall reward function minimizes heading error while reducing the angle and frequency of rudder input, making the overall tracking more consistent with the experience of human ship handling.

[0041] Example 2: I. Experimental Setup The overall model training in this embodiment is based on the ship kt model, and the steering gain coefficient is set in the experiment. Steering time constant The steering time constant is usually used to reflect the inertia of a ship's steering system. The model time step is defined as the overall system executing a rudder command once every second. In a real ship environment, the overall system reads real-time information such as the bow angle and heading angle, inputs it into the overall system, and outputs the rudder angle value.

[0042] In the TD3 reinforcement learning model, this embodiment sets the dimensions of the fully connected layers of the action policy network to 256,128 and the learning rate to 0.0005. The dimensions of the fully connected layers of the target action evaluation network are 256,128 and the learning rate is 0.001. This embodiment sets the maximum step size of reinforcement learning to 300, the total number of rounds to 10,000, the number of training rounds per batch to 128, and the cache size in reinforcement learning to 100,000.

[0043] In terms of training strategy design, this invention introduces a course learning mechanism, constructing a sequence of training tasks from easy to difficult to achieve stable convergence and performance improvement of the model. Specifically, the ship's turning angle is used as the core difficulty adjustment parameter, and the turning amplitude is divided into multiple level intervals. In the initial stage, only small-amplitude turning paths are generated, enabling the ship to complete the tracking task under conditions of low lateral acceleration and low rate of change of heading. This ensures that the policy network can first learn basic stability control behaviors, including attitude maintenance, heading fine-tuning, and error convergence mechanisms.

[0044] Once the system's cumulative reward value, path tracking error, and heading stability indicators reach the preset convergence threshold at the current difficulty level, the training system automatically increases the difficulty level, gradually increasing the range of steering angle and curvature changes on the target path. The difficulty increase process employs a graded, progressive approach, enabling the model to expand its adaptability to larger heading deviations and more complex dynamic changes based on its existing strategy. This gradual training mechanism avoids policy oscillations or convergence failures caused by directly training under high-difficulty tasks.

[0045] In the final stage, the training path covers the entire angular range, including sharp turns, multiple continuous curvature changes, and complex trajectory scenarios, enabling the policy network to stably control the ship across the entire heading space. The final control model obtained after multi-stage training maintains robustness and generalization ability under different steering angles, initial heading deviations, and external disturbances, thereby achieving adaptive tracking control for paths with arbitrary steering angles.

[0046] II. Comparative Experiment During model training, this embodiment uses the kt model to obtain the heading and yaw angle information used in the deep learning environment. However, during inference on actual ships, this method directly uses the actual heading and yaw angle information read by GPS and inertial navigation equipment as input for deep reinforcement learning.

[0047] This embodiment utilizes a reinforcement learning strategy to provide a real-time rudder angle for a given target heading angle and plots the overall control curve. This embodiment also adds classic PID control as a comparison method, setting different target heading angles to specifically examine the rise time of the reinforcement learning controller and the traditional PID controller. steady state time and overshoot To compare control strategies, 30 degrees, 60 degrees, and 120 degrees were set as target heading angles for comparative experiments.

[0048] Table 1 Comparison of Ascent Curves at Different Target Angles

[0049] like Figure 4 The image shows a comparison of the target heading angle tracking control effects at various angles. The black straight line represents the set target heading angle, the dashed lines on both sides represent the 2% stability band, the blue solid line represents the control effect of the deep reinforcement learning model, and the orange dashed line represents the control effect of the PID model. In this embodiment, the PID parameters are set as KP=0.95, KI=0.0001, and KD=4.2.

[0050] The experimental results are shown in Table 1. Comparing the performance indicators of reinforcement learning control (RL) and traditional PID control at different target angles (30°, 60°, 120°), the following conclusions can be drawn: In terms of rise time, RL control is significantly shorter than PID at all angles, exhibiting faster dynamic response characteristics. In terms of peak time, RL also outperforms PID, indicating a faster transition process before reaching steady state. Regarding overshoot, PID maintains a low level overall, demonstrating better stability, while RL achieves zero overshoot at 60° and also exhibits low overshoot at other angles, demonstrating its adaptive advantage under specific operating conditions. Overall, RL outperforms PID in dynamic response and convergence speed, while PID is more balanced in overshoot control. These comparative results demonstrate that reinforcement learning methods, while ensuring a faster response, have the potential to achieve overshoot-free control under certain operating conditions, thus improving the performance of ship tracking control.

[0051] In addition to the deep learning controller, this embodiment also includes comparative experiments to verify the effectiveness of the KLOS algorithm. A constant water flow variation is incorporated into the overall simulation model, with a flow velocity of 0.5 m / s and a northeast direction. Based on the reinforcement learning controller, this embodiment simultaneously tests the performance of the traditional LOS algorithm and the proposed KLOS algorithm on the same tracking path. The overall tracking performance is as follows: Figure 5 As shown in Table 2, the relevant indicators are as follows. Solid lines represent the actual trajectory of the ship, while dashed lines represent the target path. A total of four indicators are introduced, including lateral distance and rudder angle change. The specific calculation formulas for these indicators are as follows: (17) in express The lateral distance at any given moment and the actual steering value. This indicates the time taken to complete the entire tracking process.

[0052] from Figure 5As can be seen, when using only the LOS algorithm, the ship will deviate slightly from the course under the influence of the water flow, and will significantly interfere with the turning effect of the original algorithm when turning. However, after using the KLOS algorithm, the algorithm adopts more variable rudder angle control, but other indicators such as distance and rudder angle are significantly reduced. The ship can achieve more precise tracking under the influence of water flow, which improves the robustness of the overall tracking control model.

[0053] Table 2 Comparison metrics between LOS and KLOS algorithms

[0054] III. Ablation Experiment To test the effects of each component of the reinforcement learning reward, this embodiment designed relevant ablation experiments to further verify the impact of each reward component on the overall control effect. Two paths, Tr1 and Tr2, were set up, and reinforcement learning algorithms with different reward modules were allowed to follow the paths. A unified approach was adopted. The reward item enables the agent to reduce angular errors; only... The method for this term is called RL_base. Based on this, a rudder angle-related reward function is introduced. and The methods RL_rate and RL_rud are obtained respectively. The complete method proposed in this invention is called RL_all. Figure 6 The image clearly demonstrates the trajectory and the tracking performance of the final model RL_all. The black line represents the set path, and the red line represents the ship's tracking route in the simulation. In addition, a PID algorithm was added to specifically compare the tracking performance at the simulation level.

[0055] Table 3. Results of Deep Reinforcement Learning Ablation Experiment

[0056] Ablation experiments show, as illustrated in Table 3, that if only the basic shortening reward model (RL_base model) is used, the parameters for two different trajectories are poor. Although the ship can keep up with the path overall, it will frequently steer. The values ​​are too large, which does not conform to the principle of stable ship operation. After adding the rate term as a reward, the ship can receive different rewards according to different error ranges, which improves all four indicators compared to the RL_base model. After adding the rud term as a reward, the two indicators related to rudder angle are significantly reduced, slightly exceeding the traditional PID control algorithm. Overall, the ship tends to use small rudder angles and smoothly changing rudder angles to achieve turns. Combining the three overall rewards, the RL_all model achieves optimal performance in all four indicators, realizing stable control and tracking, and showing significant improvement in each indicator compared to the traditional PID control algorithm.

[0057] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0058] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for autonomous tracking control of ships that integrates deep reinforcement learning and Kalman filtering, characterized in that, Includes the following steps: Step 1, construct a first-order kt ship model to simulate the ship's steering characteristics; Step 2: Using the current target route as input, calculate the target heading angle using the KLOS algorithm; The KLOS algorithm uses Kalman filtering to model the drift angle and adds a drift angle factor to the LOS algorithm to correct the ship's speed direction. Step 3: Input the current heading angle, historical heading angle, historical rudder angle, and historical rudder angle data along with the target heading angle into the TD3 reinforcement learning network architecture to obtain the real-time ship rudder angle; The TD3 reinforcement learning network architecture adopts a composite reward function, which constructs rewards from three aspects: heading reward, heading change rate reward, and rudder angle reward. The TD3 reinforcement learning network architecture randomly generates training paths and gradually increases the training difficulty using a course-based learning method.

2. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: The specific formula for the first-order kt model of the ship is as follows: in, For the ship's heading angle, The first derivative of the ship's heading angle. The second derivative of the ship's heading angle. For rudder angle command, Characterizing a ship's turning ability, The time constant for heading stability is given; after transformation, the current heading angle is obtained. as follows: in, and These represent the heading angle at the previous moment and the heading angles at the two moments before that, respectively. , and These represent the rudder angle at the current moment, the rudder angle at the previous moment, and the rudder angle at the two moments before that, respectively. , and For model hyperparameters; the position of the ship at any given time. and Specifically as follows: in, The velocity vector of the ship. Let be the position of the ship at time t. For the ship's heading angle, For a time slice.

3. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: Step 2 uses the current target route as input to calculate the target heading angle using the KLOS algorithm. First, the LOS angle is calculated using the LOS algorithm, and the calculation formula is as follows: in, This refers to the aiming point position in the LOS algorithm. The ship's current position; Then calculate The drift angle of the ship at that moment is: in, The lateral drift speed of the ship, Let be the ship's velocity vector; let the drift angle and the rate of change of drift angle be used as state vectors, and update these two physical quantities. The formula for the update process is as follows: in for The prior state estimate at time t. for Predicted value at time, Let be the state transition matrix, set as a discrete first-order integral model, indicating that the current drift angle can be obtained from the previous drift angle and the rate of change of drift angle; express The posterior estimate of the covariance at time t. for The prior estimate of covariance at time t, Indicates random noise; In the prediction phase, the specific formula is as follows: in This is the Kalman gain matrix, used to measure the contribution of the current state value to the correction of the state estimate. These are the current observations. It is the transformation matrix from the state vector to the measurement vector. It is the covariance matrix of the measured noise; The overall rudder angle and rate of change are input into a Kalman filter. While smoothing the input, the output drift angle value is corrected in real time based on the current drift angle observation matrix. This drift angle value is then used to generate the ship's target heading angle in real time. For time The revised formula is as follows: in They represent Predicted values ​​of drift angle and rate of change of drift angle at time. This represents the target heading angle output by the KLOS algorithm.

4. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: The TD3 reinforcement learning network architecture is based on the TD3 deep reinforcement learning algorithm framework and consists of an online policy network (Actor) and an online value network (Critic). The Actor network takes the current state information of the ship as input, and outputs the actual control rudder angle of the ship after passing through multiple fully connected layers and activation functions. The Critic network takes five features as input: the current heading angle, the historical heading angle, the historical rudder angle, the current rudder angle data, and the target heading angle. After passing through multiple fully connected layers and activation functions, it outputs a score for the judgment of the current rudder angle action. The higher the score, the better the current rudder angle action is compared to the overall system.

5. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: The TD3 reinforcement learning network architecture encodes angles in the state space and fuses historical temporal features, as shown in the following formula: in express The bow angle of the ship at any given time. express The target heading angle of the ship at any given time. express The angle of error between the ship's heading angle at a given moment and the target heading angle, indicated by the subscript. They represent The values ​​of each parameter at each time point, express The value of the rudder angle at that moment; The motion space is: in, The range is The overall action space takes each integer rudder angle as the output based on the feasible domain of the rudder angle.

6. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: The heading error reward includes heading error reward and error change rate reward; The rudder angle rewards include rudder angle amplitude rewards, rudder angle change rate rewards, and additional penalty items; The stability bonus includes: a bonus awarded for the system approaching the target course and maintaining a stable state; and a special bonus set for rudder angle stability. Ultimately, the overall composite reward function constructs a phased control strategy based on the magnitude difference of the current heading error through differentiated weight configuration, thereby achieving dynamic collaborative optimization of each sub-reward item.

7. The ship autonomous tracking control method integrating deep reinforcement learning and line-of-sight algorithm as described in claim 1, characterized in that: The TD3 reinforcement learning network architecture randomly generates training paths and gradually increases the training difficulty using a course learning method. The ship's turning angle is used as a random training target. Training begins with small turns, and the turning angle is gradually increased after the results stabilize, eventually resulting in a final model that can adapt to all angles.

Images