Unmanned surface ship path following method based on DQN and backstepping control

Abstract

The invention discloses an unmanned surface ship path following method based on DQN and backstepping control, and the method comprises the following steps: designing a communication system based on an MAVLink protocol, and achieving the real-time transmission of state information between a piloting ship and a controlled ship; constructing a dynamic model of the pilot ship and the controlled ship, and observing the ship following performance in real time in the model training process by adopting a rendering window; optimizing a path following strategy through a reward function by using the constructed DQN model; under the condition of unstable communication, real-time feedback compensation is carried out by adopting backstepping control; under the condition that communication is seriously interfered, selecting a next following action by adopting a state prediction method; and by dynamically adjusting the weight of the DQN and backstepping control, path following in a complex marine environment is realized. According to the method, adaptive optimization of a path following strategy is realized, the problems of communication delay and data packet loss in a complex marine environment can be effectively solved, and the robustness and control precision of path following of the unmanned ship are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of ship control, and relates to a path following technology for an unmanned surface ship, and in particular to a path following method for an unmanned surface ship based on DQN and backstepping control. Background Art

[0002] In the marine environment, path following of unmanned vehicles (USVs) is a challenging task. With the increasing complexity of the marine environment and the diversification of operational requirements, the action capability of a single USV is limited, while multiple USVs can cover a larger sea area and perform a wider range of tasks. Therefore, path following and formation control have become research hotspots. As a common path following method, the pilot following strategy enables the controlled vessel to follow the pilot vessel to perform complex tasks.

[0003] In existing research, Kapitanyuk et al. used the guidance vector field method to realize the path following control of non-holonomic mobile robots. Fossen and Lekkas designed a line-of-sight path tracking controller based on the relative velocity of USV, established a kinematic model, generated relative velocity in the form of amplitude and phase, and used hydroacoustic measurement to convert the relative velocity into absolute velocity to realize the control law. Khamseh and Janabi-Sharifi established a USV model to synchronously control the quadcopter and the manipulator based on a linear quadratic regulator. Dong et al. used backstepping control based on state feedback to solve the USV tracking problem of curved and straight paths.

[0004] With the breakthrough of DeepMind project in deep reinforcement learning (DRL), reinforcement learning (RL) has been widely studied and applied in robotics, industrial automation, machine learning and other fields. RL is considered to be one of the core technologies for designing intelligent systems. Combined with deep learning, deep reinforcement learning can characterize and control extremely complex systems in dynamic environments. Therefore, it is reasonable to establish an effective path following control decision model based on DRL. In order to overcome the usability problem caused by the complexity of control law in traditional analytical methods, a concise DRL was developed using Deep Q-Network (DQN). Li et al. used RL to reduce the travel time at the bottleneck of highways and proposed a reinforcement learning model based on Q-learning to control the speed limit of vehicles under various traffic conditions. The use of reinforcement learning based on Q-learning can significantly reduce the training time. However, the training efficiency of Q-learning is a problem, and the improvement of USV path following by reinforcement learning based on Q-learning is rarely discussed in the literature.

[0005] In order to achieve communication between the pilot ship and the controlled ship, the lightweight MAVLink protocol is used. However, complex communication environments (electromagnetic interference, etc.) often lead to data packet loss and delay, affecting communication and control effects. Summary of the invention

[0006] Purpose of the invention: In order to overcome the shortcomings of the prior art, a path following method for unmanned surface ships based on DQN and backstepping control is provided. Combining deep reinforcement learning and control theory, it not only realizes the adaptive optimization of the path following strategy, but also effectively copes with the communication delay and data packet loss problems in complex marine environments, significantly improves the robustness and control accuracy of the unmanned ship path following, and is suitable for path planning tasks of collaborative operations of multiple unmanned ships.

[0007] Technical solution: To achieve the above purpose, the present invention provides a path following method for an unmanned surface ship based on DQN and backstepping control, comprising the following steps:

[0008] S1: Design a communication system based on the MAVLink protocol to achieve real-time transmission of status information between the pilot ship and the controlled ship;

[0009] S2: In the constructed model training environment, the dynamic models of the pilot ship and the controlled ship are constructed, and the rendering window is used to observe the ship following performance in real time during the model training process;

[0010] S3: Use the built deep reinforcement learning (DQN) model to optimize the path following strategy through the reward function;

[0011] S4: Under unstable communication conditions, backstepping control is used for real-time feedback compensation to improve control accuracy.

[0012] When the communication is seriously disturbed, packet loss or delay occurs, the state prediction method is used to enable the controlled ship to select the next following action through state prediction;

[0013] S5: Path following in complex ocean environments is achieved by dynamically adjusting the weights of DQN and backstepping control.

[0014] Furthermore, in step S1, the communication system uses the MAVLink protocol to achieve status data transmission between the pilot ship and the controlled ship through WIFI, including position information and attitude information (roll angle, pitch angle, yaw angle), and introduces an electromagnetic interference model through simulation to evaluate the communication robustness.

[0015] Furthermore, the operation of the communication system in step S1 includes:

[0016] A1: Pilot ship sends information: collects the status information data of the pilot ship and processes the data, uses the MAVLink protocol to generate the Payload data part containing key control information, and the pilot ship acts as a sending module to regularly send data packets containing position information and attitude information (roll angle, pitch angle, yaw angle) to ensure the accuracy and effectiveness of the data and adapt to complex marine environments;

[0017] A2: WIFI protocol simulation and transmission: MAVLink data is sent via WIFI protocol; the simulation takes into account channel attenuation, noise interference and multipath effects, uses a dual-path model to reflect signal reflection and refraction, evaluates packet loss rate and delay, and simulates real wireless communication characteristics;

[0018] A3: Introduction and control of electromagnetic interference: Design an adjustable electromagnetic interference model, adjust interference parameters, and test the robustness of communication under different noise levels; use WIFI protocol interference simulation, interference simulation packet loss rate and delay, and verify the stability of data transmission and decoding in complex environments;

[0019] The introduction of electromagnetic interference not only affects the transmission of data, but also makes the reception and decoding of MAVLink data packets more challenging by simulating interference with the WIFI protocol;

[0020] A4: Controlled ship receives information: The controlled ship extracts the position information and attitude data of the pilot ship through the MAVLink unpacking process. Abnormal data (such as interference or packet loss) is recorded and fed back to the motion control algorithm to analyze the impact of interference on motion control.

[0021] Furthermore, the dynamic model in step S2 is divided into a pilot ship dynamic model and a controlled ship dynamic model, wherein the pilot ship dynamic model includes:

[0022] The dynamic equation of the pilot ship is composed of inertia, Coriolis force, centrifugal force and damping force, and the equation form is as follows:

[0023]

[0024] Where M is the inertia matrix (including added mass effects), C(v) is the Coriolis and centrifugal force matrix, D(v) is the damping matrix, and τ is the control input (force and torque);

[0025] The expression of inertia matrix M is:

[0026]

[0027] Where m is the mass of the ship, represents the hydrodynamic additional mass coefficient, X g is the position of the ship's center of gravity in the hull coordinate system, Iz is the moment of inertia of the ship around the z-axis;

[0028] The expression of the Coriolis and centrifugal force matrix C(v) is:

[0029]

[0030] Among them, u, v, and r represent the longitudinal velocity, lateral velocity, and yaw angular velocity of the ship, respectively;

[0031] The damping matrix D(v) includes linear and nonlinear damping terms and is expressed as:

[0032]

[0033] Linear terms (such as X u ): represents the linear damping coefficient, describing the linear relationship between speed and resistance; nonlinear terms (such as X uu ): represents the nonlinear damping coefficient, describing the effect of the square or cube of velocity on the resistance; |u|, |v|, |r| represent the absolute values ​​of the longitudinal velocity, lateral velocity and yaw angular velocity of the ship, respectively, reflecting that the nonlinear resistance depends on the direction of motion;

[0034] The rotation matrix R(ψ) is used to transform the velocity in the hull coordinate system to the global coordinate system, and its expression is:

[0035]

[0036] The control input τ acts on the unmanned ship in the form of thrust and torque:

[0037]

[0038] The state update equation is divided into two parts: position update and velocity update;

[0039] Position update (global coordinate system):

[0040]

[0041] Where p = [x, y, ψ] T ,V=[u,v,r] T ;

[0042] Velocity update (hull coordinate system):

[0043]

[0044] This dynamic model describes the motion behavior of the unmanned ship in three degrees of freedom, including the inertial effect, Coriolis and centrifugal force effects, damping effect, and the influence of external control input on the motion state.

[0045] The dynamic model of the controlled ship adopts a relatively simplified model when implementing the reference trajectory simulation environment, focusing on the position update of the unmanned ship, and does not involve complex dynamic equations in detail. The main part of the model involves the update of attitude rotation and displacement. The following are the main formulas of the model and their explanations:

[0046] The dynamics model of the controlled vessel includes:

[0047] The position update is obtained by integrating the velocity, using Euler's method for numerical integration:

[0048] p new =p old +Δt·R(ψ)·v

[0049] where p = [x, y, ψ] T is the position vector, V = [u, v, r] T is the velocity vector, Δt is the time step;

[0050] The attitude rotation matrix R(ψ) is used to map the velocity [u, v, r] to the global coordinate system, and then update the position of the unmanned ship [x, y, ψ];

[0051] Speed ​​update formula:

[0052]

[0053] The velocity is calculated by dividing the displacement in the global coordinate system by Re-mapped back to the hull coordinate system.

[0054] Further, in step S2, the rendering window is based on the gym.envs.classic_control.rendering module, and a display area is set. The background is set to blue to simulate the color of the ocean environment, and different colors (such as red for pilot ships and blue for controlled ships) are used to distinguish ships. The position information of the ship is updated in real time through the Transform object, so that the ship can move dynamically in the rendering window. In the early stage of training, since the model has not yet converged, the controlled ship may deviate from the trajectory. Through the rendering window, this deviation can be observed and the model training parameters or reward function design can be adjusted in real time. As the training deepens, the rendering window will gradually display the process of the controlled ship approaching the pilot ship trajectory.

[0055] Furthermore, in step S3, the DQN model is trained using experience replay and epsilon-greedy strategy, including an evaluation network and a target network, wherein the parameters of the target network are synchronized regularly to improve training stability;

[0056] The goal of the DQN model is to find an optimal strategy so that the state-action value function Q(s, a) reaches the maximum value; the optimal state-action value function satisfies the Bellman equation:

[0057]

[0058] Among them, s is the current state, a is the current action, s' is the next state (determined by the state transition probability), and a' is the action in the next state; Q * (s,a) is the optimal state-action value function, which indicates the maximum value of the expected cumulative reward that the agent can obtain after taking action a in state s; E[·] is the mathematical expectation, which indicates the uncertainty and expectation calculation of the next state s'; r is the immediate reward, which indicates the reward obtained immediately after taking action a; γ is a weighting factor, ranging from [0,1], which is used to weigh the importance of current rewards and future rewards; Indicates the maximum Q value that can be obtained by selecting action a' in the next state;

[0059] In existing Q-learning, values ​​are stored by table lookup. However, when the state space and action space are large or continuous, it becomes infeasible to store the state-action value function in a Q table. DQN uses a deep neural network to approximate Q values, allowing it to learn efficiently in high-dimensional state space. The goal of the DQN model is to minimize the following loss function so that the Q network gradually approaches the optimal state-action value function:

[0060]

[0061] Among them, θ is the parameter of the current Q network, θ - is the parameter of the target network; the target network Q target The parameters of are fixed for a period of time to improve training stability.

[0062] Furthermore, the DQN model in step S3 is instantiated through the DQN class, which contains two deep neural networks, one is the current Q network eval_net, which is used to estimate the Q value of the current state; the other is the target Q network target_net, which is used to estimate the target Q value of the next state; the initial parameters of the two networks are consistent, and as the training proceeds, the parameters of the target network will be synchronized from the Q network at a certain interval;

[0063] In each time step, the epsilon-greedy strategy will be used to select actions. In the exploration phase (initial epsilon is large), the agent will randomly select actions more often to explore more possibilities; as the training progresses, the epsilon value gradually decreases, and the unmanned ship is more inclined to select the optimal action evaluated by the current Q network.

[0064] Specifically, in the select_action method, DQN will randomly select an action based on the value of epsilon or select an action through the maximum Q value of the Q network. This strategy allows the unmanned ship to balance between exploration and exploitation.

[0065] After each action is executed, the action selected in the current state, the reward obtained, and the next state are stored in the experience replay cache. When the cache is full, the new experience will overwrite the old experience. Through the experience replay mechanism, small batches of data are randomly extracted from the cache for training, which reduces the correlation between samples and makes model learning more stable.

[0066] In the DQN update method, a small batch of experience samples are first extracted from the experience replay buffer. For each sample, the Q value target is calculated according to the Bellman equation:

[0067]

[0068] Among them, θ- is the parameter of the target network;

[0069] Calculate the mean square error between the estimated Q value q_eval of the current Q network and the target Q value q_target, and minimize the error through back propagation. This process makes the estimated value of the Q network gradually approach the true Q value.

[0070] To improve stability, the parameters of the target network are not updated at every step, but are periodically synchronized with the parameters of the current Q network at a certain interval. This can reduce the fluctuation of the Q value during the update process and make the model more stable.

[0071] Furthermore, the reward function in step S3 is dynamically adjusted according to the difference between the states of the controlled vessel and the pilot vessel (such as position deviation, speed deviation and path deviation angle), and the closer the reward value is to zero, the better the path following effect is.

[0072] Through the calculate_reward function, a reward function based on state difference needs to be defined to measure the effect of the current action. Actions with larger rewards are considered good control strategies. The DQN network continuously optimizes the reward value and adjusts the control strategy, so that the agent gradually learns how to track the path and minimize errors.

[0073] The reward function evaluates the effect of the control strategy by calculating the difference between the state of the controlled ship and the state of the pilot ship, and evaluates the overall control effect within the round;

[0074]

[0075] The reward function includes:

[0076] Total reward for each round: The total reward output at the end of each round is the cumulative value of all time step rewards in that round; the higher the total reward value (the closer the negative value is to zero), the smaller the difference between the state of the controlled ship and the state of the pilot ship in that round, and the better the control effect;

[0077] Immediate reward: Immediate feedback calculated at each time step to evaluate the quality of the current action; calculated as the difference between the current state and the goal state;

[0078] Sliding average reward: The sliding average reward is obtained by calculating the average of the total rewards of a certain number of recent rounds; it reflects the average performance of the model in the recent period of time;

[0079]

[0080] Where N is the size of the sliding window and the total reward i is the total reward of the i-th round.

[0081] Furthermore, the backstepping control in step S4 realizes feedback compensation by calculating the error and recursively designing the controller, and generates control input according to the path deviation and real-time feedback under unstable communication conditions to correct the state of the controlled ship; specifically includes:

[0082] Calculate the error z1 based on the environmental state and the reference state:

[0083] z1=env-env r

[0084] Then we get the intermediate variables ω and

[0085] ω=k1·z1

[0086]

[0087] Then calculate the error z2:

[0088] z2=env-env r -ω

[0089] Generate control input τ:

[0090] τ=-k2·z2+kinetic model term;

[0091] The backstepping control strategy calculates the error and uses the system dynamics model to generate the control input τ so that the state of the controlled ship (env) approaches the reference ship (env r ) status.

[0092] Hybrid strategy: By recursively designing the controller, the control target is gradually approached; under unstable communication conditions, the control input is generated according to the path deviation and real-time feedback to correct the state of the controlled ship.

[0093] In the early stages of training, DQN has a long random action exploration time, which leads to slow convergence and high path following error. At the same time, DQN is limited by the discrete action space. High update frequency can easily cause Q value oscillation, while low frequency reduces control accuracy. It is also difficult to converge effectively when the uncertainty of state transition is large.

[0094] Backstepping control (BC) is a method of recursively designing controllers, which gradually approaches the final goal by decomposing the control target. After the introduction of BC, the DQN output is combined with the BC control result. BC smoothes the DQN action selection, improves system stability and shortens exploration time, especially in the early stages of training. BC provides strong dynamic stability, quickly responds to tracking errors, and corrects path deviations; DQN captures the long-term change trend of the path.

[0095] The combination of DQN and BC forms a "fast response-slow learning" balance: DQN optimizes the path in the long term, BC corrects the action in the short term, and the target synchronization mechanism avoids DQN instability, ultimately achieving faster convergence and excellent control performance in complex paths.

[0096] The state prediction method uses the latest valid state information of the pilot ship, including position, speed and acceleration, combined with its dynamic model (such as uniform linear motion or uniform acceleration model) to predict the state change in a short time; the prediction formula is as follows:

[0097]

[0098] Among them, s predicted is the predicted pilot ship state, s last valid The most recently received valid status, The speed of the pilot vessel (or the estimated rate of change of state), Δt is the estimated time difference (delay time).

[0099] Furthermore, the method for dynamically adjusting the weights of DQN and backstepping control in step S5 is: weight distribution ratio: tau = alpha*tau_dqn+(1-alpha)*tau_bc;

[0100] The weight distribution of DQN and backstepping control (such as alpha = 0.5) is determined through experimental analysis. Backstepping control is mainly used in the early stage of training, and gradually transitions to DQN dominance in the later stage to improve the control effect and convergence speed.

[0101] The present invention aims at the communication packet loss and delay problems under electromagnetic interference, and proposes a hybrid control method based on deep reinforcement learning (DQN) and backstepping control (BC). Specifically, DQN optimizes the path to follow under stable communication conditions. Backstepping control provides real-time compensation in the case of delay and packet loss. The anti-interference performance is improved by state prediction technology, and the weights between DQN and BC are adjusted to achieve adaptive control to ensure the stability of the system in a high-level interference environment. The present invention adopts the combination of backstepping control to improve the efficiency of Q learning training.

[0102] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0103] (1) Improved robustness and control accuracy: The present invention combines deep reinforcement learning (DQN) with backstepping control to effectively deal with communication delays and data packet loss problems in complex marine environments, and significantly improves the robustness and control accuracy of unmanned surface vessel path following.

[0104] (2) Realize adaptive control in dynamic environments: Use DQN to optimize the path following strategy and dynamically adjust the reward function to adapt to different interference conditions. At the same time, combine backstepping control to provide real-time feedback compensation to ensure that the controlled ship operates smoothly in a changing environment.

[0105] (3) Combining the “fast response-slow learning” strategy: DQN is responsible for capturing the long-term change trend of the path, and backstepping control provides short-term real-time correction. The dynamic balance between the two is achieved through weight distribution, shortening the exploration time in the early stage of training and improving the control accuracy in the later stage.

[0106] (4) Applicable to collaborative operations of multiple unmanned ships: Through the design of a communication system based on the MAVLink protocol, real-time state transmission and path planning between multiple unmanned ships are supported, making the present invention potentially applicable to collaborative operations of multiple unmanned ships, especially maintaining high performance under high interference conditions. The state prediction method used ensures that the control strategy enables the controlled ship to select the optimal action for piloting and following even under high interference conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] Figure 1 This is a schematic diagram of the pilot-following method;

[0108] Figure 2 Run a flow chart for a MAVLink-based communication system;

[0109] Figure 3 This is a schematic diagram of the ship dynamics model;

[0110] Figure 4Provides a visual display for the rendering window;

[0111] Figure 5 Comparison chart of reward value curves under different strategies;

[0112] Figure 6 Comparison chart of reward value curve and loss value curve under different weights;

[0113] Figure 7 Comparison chart of reward value curves under different interference levels. DETAILED DESCRIPTION

[0114] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0115] The present invention provides a path following method for an unmanned surface ship based on DQN and backstepping control, which adopts a pilot following method. Figure 1 As shown, the controlled ship follows the trajectory of the pilot ship by continuously adjusting its status (position, speed and heading angle). The pilot ship acts as a guide and its preset path provides a reference for the controlled ship. The controlled ship obtains the real-time status information of the pilot ship through the communication system and makes adjustments accordingly.

[0116] Based on the above-mentioned pilot-following method, the present invention provides an unmanned surface ship path following method based on DQN and backstepping control, comprising the following steps:

[0117] S1: Design a communication system based on the MAVLink protocol to achieve real-time transmission of status information between the pilot ship and the controlled ship;

[0118] The communication system uses the MAVLink protocol to achieve status data transmission between the pilot ship and the controlled ship through WIFI, including position information and attitude information (roll angle, pitch angle, yaw angle), and introduces an electromagnetic interference model through simulation to evaluate the communication robustness;

[0119] like Figure 2 As shown, the operation of the communication system includes the following steps:

[0120] A1: Pilot ship sends information: collects the status information data of the pilot ship and processes the data, uses the MAVLink protocol to generate the Payload data part containing key control information, and the pilot ship acts as a sending module to regularly send data packets containing position information and attitude information (roll angle, pitch angle, yaw angle) to ensure the accuracy and effectiveness of the data and adapt to complex marine environments;

[0121] A2: WIFI protocol simulation and transmission: MAVLink data is sent via WIFI protocol; the simulation takes into account channel attenuation, noise interference and multipath effects, uses a dual-path model to reflect signal reflection and refraction, evaluates packet loss rate and delay, and simulates real wireless communication characteristics;

[0122] A3: Introduction and control of electromagnetic interference: Design an adjustable electromagnetic interference model, adjust interference parameters, and test the robustness of communication under different noise levels; use WIFI protocol interference simulation, interference simulation packet loss rate and delay, and verify the stability of data transmission and decoding in complex environments;

[0123] The introduction of electromagnetic interference not only affects the transmission of data, but also makes the reception and decoding of MAVLink data packets more challenging by simulating interference with the WIFI protocol;

[0124] A4: Controlled ship receives information: The controlled ship extracts the pilot ship's position information and attitude data through the MAVLink unpacking process. Abnormal data (such as communication delay or packet loss) is recorded and fed back to the motion control algorithm to analyze the impact of interference on motion control. It should be noted here that abnormal data is reflected in the training process and is part of the algorithm training process.

[0125] S2: In the constructed model training environment (such as Figure 1 In a training environment with a fixed motion reference trajectory of a pilot ship as shown in FIG, a dynamic model of the pilot ship and the controlled ship is constructed, and a rendering window is used to observe the ship following performance in real time during the model training process;

[0126] The dynamic model is divided into the pilot ship dynamic model and the controlled ship dynamic model. The pilot ship dynamic model includes:

[0127] The dynamic equation of the pilot ship is composed of inertia, Coriolis force, centrifugal force and damping force, and the equation form is as follows:

[0128]

[0129] Where M is the inertia matrix (including added mass effects), C(v) is the Coriolis and centrifugal force matrix, D(v) is the damping matrix, and τ is the control input (force and torque);

[0130] The expression of inertia matrix M is:

[0131]

[0132] Where m is the mass of the ship, represents the hydrodynamic additional mass coefficient, X g is the position of the ship's center of gravity in the hull coordinate system, Iz is the moment of inertia of the ship around the z-axis;

[0133] The expression of the Coriolis and centrifugal force matrix C(v) is:

[0134]

[0135] Among them, u, v, and r represent the longitudinal velocity, lateral velocity, and yaw angular velocity of the ship, respectively;

[0136] The damping matrix D(v) includes linear and nonlinear damping terms and is expressed as:

[0137]

[0138] Linear terms (such as X u ): represents the linear damping coefficient, describing the linear relationship between speed and resistance; nonlinear terms (such as X uu ): represents the nonlinear damping coefficient, describing the effect of the square or cube of velocity on the resistance; |u|, |v|, |r| represent the absolute values ​​of the longitudinal velocity, lateral velocity and yaw angular velocity of the ship, respectively, reflecting that the nonlinear resistance depends on the direction of motion;

[0139] The rotation matrix R(ψ) is used to transform the velocity in the hull coordinate system to the global coordinate system, and its expression is:

[0140]

[0141] The control input τ acts on the unmanned ship in the form of thrust and torque:

[0142]

[0143] The state update equation is divided into two parts: position update and velocity update;

[0144] Position update (global coordinate system):

[0145]

[0146] Where p = [x, y, ψ] T ,V=[u,v,r] T ;

[0147] Velocity update (hull coordinate system):

[0148]

[0149] This dynamic model describes the motion behavior of the unmanned ship in three degrees of freedom, including the inertial effect, Coriolis and centrifugal force effects, damping effect, and the influence of external control input on the motion state.

[0150] The dynamic model of the controlled ship adopts a relatively simplified model when implementing the reference trajectory simulation environment, focusing on the position update of the unmanned ship, and does not involve complex dynamic equations in detail. The main part of the model involves the update of attitude rotation and displacement. The following are the main formulas of the model and their explanations:

[0151] The dynamics model of the controlled vessel includes:

[0152] The position update is obtained by integrating the velocity, using Euler's method for numerical integration:

[0153] p new =p old +Δt·R(ψ)·v

[0154] where p = [x, y, ψ] T is the position vector, V = [u, v, r] T is the velocity vector, Δt is the time step;

[0155] The attitude rotation matrix R(ψ) is used to map the velocity [u, v, r] to the global coordinate system, and then update the position of the unmanned ship [x, y, ψ];

[0156] Speed ​​update formula:

[0157]

[0158] The velocity is calculated by dividing the displacement in the global coordinate system by Re-mapped back to the hull coordinate system.

[0159] The ship dynamics model established in this embodiment is as follows: Figure 3 As shown in the figure, the dynamic model of USV is the basis for designing the control strategy. In order to simplify the calculation and reduce the complexity of the control system, the pilot ship and the controlled ship are described by a three-degree-of-freedom (3-DoF) dynamic model. Surge is a linear motion along the X-axis with a speed of u; Sway is a linear motion along the Y-axis with a speed of v; Yaw is a rotational motion around the Z-axis with a speed of r. The model includes the motion states of the lateral (x), longitudinal (y) and heading angle (ψ).

[0160] The rendering window is based on the gym.envs.classic_control.rendering module. The display area is set, the background is set to blue, simulating the color of the ocean environment, and different colors (such as red for pilot ships and blue for controlled ships) are used to distinguish ships. The position information of the ship is updated in real time through the Transform object, so that the ship moves dynamically in the rendering window. In the early stage of training, the controlled ship may deviate from the trajectory because the model has not yet converged. Through the rendering window, this deviation can be observed and the model training parameters or reward function design can be adjusted in real time. As the training progresses, the rendering window will gradually show the process of the controlled ship approaching the trajectory of the pilot ship.

[0161] In this embodiment, the rendering window is as follows Figure 4 As shown in the figure, the learning process is visualized in the rendering window, and the state difference between the pilot ship and the controlled ship, the execution of the control strategy and the reinforcement learning are directly observed. In the initial rounds, the state difference between the controlled ship and the pilot ship is large during the exploration phase, and the path following is offset. As the number of training rounds increases, the state of the controlled ship approaches that of the pilot ship, and the path following is achieved.

[0162] S3: Use the built deep reinforcement learning (DQN) model to optimize the path following strategy through the reward function;

[0163] The DQN model is trained using experience replay and epsilon-greedy strategies, including an evaluation network and a target network, where the parameters of the target network are regularly synchronized to improve training stability;

[0164] The goal of the DQN model is to find an optimal strategy so that the state-action value function Q(s, a) reaches the maximum value; the optimal state-action value function satisfies the Bellman equation:

[0165]

[0166] Among them, s is the current state, a is the current action, s' is the next state (determined by the state transition probability), and a' is the action in the next state; Q * (s,a) is the optimal state-action value function, which indicates the maximum value of the expected cumulative reward that the agent can obtain after taking action a in state s; E[·] is the mathematical expectation, which indicates the uncertainty and expectation calculation of the next state s'; r is the immediate reward, which indicates the reward obtained immediately after taking action a; γ is a weighting factor, ranging from [0,1], which is used to weigh the importance of current rewards and future rewards; Indicates the maximum Q value that can be obtained by selecting action a' in the next state;

[0167] In existing Q-learning, values ​​are stored by table lookup. However, when the state space and action space are large or continuous, it becomes infeasible to store the state-action value function in a Q table. DQN uses a deep neural network to approximate Q values, allowing it to learn efficiently in high-dimensional state space. The goal of the DQN model is to minimize the following loss function so that the Q network gradually approaches the optimal state-action value function:

[0168]

[0169] Among them, θ is the parameter of the current Q network, θ - is the parameter of the target network; the target network Q target The parameters of are fixed for a period of time to improve training stability.

[0170] The DQN model is instantiated through the DQN class, which contains two deep neural networks: one is the current Q network eval_net, which is used to estimate the Q value of the current state; the other is the target Q network target_net, which is used to estimate the target Q value of the next state; the initial parameters of these two networks are the same, and as the training progresses, the parameters of the target network will be synchronized from the Q network at regular intervals;

[0171] In each time step, the epsilon-greedy strategy will be used to select actions. In the exploration phase (initial epsilon is large), the agent will randomly select actions more often to explore more possibilities; as the training progresses, the epsilon value gradually decreases, and the unmanned ship is more inclined to select the optimal action evaluated by the current Q network.

[0172] Specifically, in the select_action method, DQN will randomly select an action based on the value of epsilon or select an action through the maximum Q value of the Q network. This strategy allows the unmanned ship to balance between exploration and exploitation.

[0173] After each action is executed, the action selected in the current state, the reward obtained, and the next state are stored in the experience replay cache. When the cache is full, the new experience will overwrite the old experience. Through the experience replay mechanism, small batches of data are randomly extracted from the cache for training, which reduces the correlation between samples and makes model learning more stable.

[0174] In the DQN update method, a small batch of experience samples are first extracted from the experience replay buffer. For each sample, the Q value target is calculated according to the Bellman equation:

[0175]

[0176] Among them, θ- is the parameter of the target network;

[0177] Calculate the mean square error between the estimated Q value q_eval of the current Q network and the target Q value q_target, and minimize the error through back propagation. This process makes the estimated value of the Q network gradually approach the true Q value.

[0178] To improve stability, the parameters of the target network are not updated at every step, but are periodically synchronized with the parameters of the current Q network at a certain interval. This can reduce the fluctuation of the Q value during the update process and make the model more stable.

[0179] Reward function: Dynamically adjusted according to the difference between the states of the controlled ship and the pilot ship (such as position deviation, speed deviation and path deviation angle). The closer the reward value is to zero, the better the path following effect is.

[0180] Through the calculate_reward function, a reward function based on state difference needs to be defined to measure the effect of the current action. Actions with larger rewards are considered good control strategies. The DQN network continuously optimizes the reward value and adjusts the control strategy, so that the agent gradually learns how to track the path and minimize errors.

[0181] The reward function evaluates the effect of the control strategy by calculating the difference between the state of the controlled ship and the state of the pilot ship, and evaluates the overall control effect within the round;

[0182]

[0183] The reward function includes:

[0184] Total reward for each round: The total reward output at the end of each round is the cumulative value of all time step rewards in that round; the higher the total reward value (the closer the negative value is to zero), the smaller the difference between the state of the controlled ship and the state of the pilot ship in that round, and the better the control effect;

[0185] Immediate reward: Immediate feedback calculated at each time step to evaluate the quality of the current action; calculated as the difference between the current state and the goal state;

[0186] Sliding average reward: The sliding average reward is obtained by calculating the average of the total rewards of a certain number of recent rounds; it reflects the average performance of the model in the recent period of time;

[0187]

[0188] Where N is the size of the sliding window and the total reward i is the total reward of the i-th round.

[0189] S4: Under unstable communication conditions, backstepping control is used for real-time feedback compensation to improve control accuracy.

[0190] Backstepping control achieves feedback compensation by calculating errors and recursively designing controllers. Under unstable communication conditions, control inputs are generated based on path deviations and real-time feedback to correct the state of the controlled ship. Specifically, it includes:

[0191] Calculate the error z1 based on the environmental state and the reference state:

[0192] z1=env-env r

[0193] Then we get the intermediate variables ω and

[0194] ω=k1·z1

[0195]

[0196] Then calculate the error z2:

[0197] z2=env-env r -ω

[0198] Generate control input τ:

[0199] τ=-k2·z2+kinetic model term;

[0200] The backstepping control strategy calculates the error and uses the system dynamics model to generate the control input τ so that the state of the controlled ship (env) approaches the reference ship (env r ) status.

[0201] Hybrid strategy: By recursively designing the controller, the control target is gradually approached; under unstable communication conditions, the control input is generated according to the path deviation and real-time feedback to correct the state of the controlled ship.

[0202] In the early stages of training, DQN has a long random action exploration time, which leads to slow convergence and high path following error. At the same time, DQN is limited by the discrete action space. High update frequency can easily cause Q value oscillation, while low frequency reduces control accuracy. It is also difficult to converge effectively when the uncertainty of state transition is large.

[0203] Backstepping control (BC) is a method of recursively designing controllers, which gradually approaches the final goal by decomposing the control target. After the introduction of BC, the DQN output is combined with the BC control result. BC smoothes the DQN action selection, improves system stability and shortens exploration time, especially in the early stages of training. BC provides strong dynamic stability, quickly responds to tracking errors, and corrects path deviations; DQN captures the long-term change trend of the path.

[0204] The combination of DQN and BC forms a "fast response-slow learning" balance: DQN optimizes the path in the long term, BC corrects the action in the short term, and the target synchronization mechanism avoids DQN instability, ultimately achieving faster convergence and excellent control performance in complex paths.

[0205] When the communication is seriously disturbed, packet loss or delay occurs, the state prediction method is used to enable the controlled ship to select the next following action through state prediction;

[0206] The state prediction method uses the latest valid state information of the pilot ship, including position, speed and acceleration, combined with its dynamic model (such as uniform linear motion or uniform acceleration model) to predict the state change in a short time; the prediction formula is as follows:

[0207]

[0208] Among them, s predicted is the predicted pilot ship state, s last_valid The most recently received valid status, The speed of the pilot vessel (or the estimated rate of change of state), Δt is the estimated time difference (delay time).

[0209] S5: Path following in complex ocean environments is achieved by dynamically adjusting the weights of DQN and backstepping control;

[0210] Control input weight distribution ratio:

[0211] tau=alpha*tau_dqn+(1-alpha)*tau_bc

[0212] The weight distribution of DQN and backstepping control (such as alpha = 0.5) is determined through experimental analysis. Backstepping control is mainly used in the early stage of training, and gradually transitions to DQN dominance in the later stage to improve the control effect and convergence speed.

[0213] Experimental environment: The simulation environment includes a 1000×1000 two-dimensional static map to simulate different interference conditions (such as packet loss rate and delay) to verify the robustness and adaptability of the method.

[0214] In order to verify the effectiveness and effect of the method of the present invention, the method of the present invention is compared with the existing control method in this embodiment, as follows:

[0215] Comparison of training of hybrid control strategy and DQN control strategy Figure 5 ,exist Figure 5 In the figure, the left picture shows that the communication system is not affected by additional interference, and the right picture adds 10s-20s of random delay and about 20% packet loss rate and other uncertain electromagnetic interference factors to simulate and complex environment. It can be observed from the left chart that in the interference-free environment, the average reward value of the hybrid strategy (blue curve) is higher, the fluctuation is smaller, and the control process is more stable; the reward value of the DQN control strategy (red curve) fluctuates greatly, and it takes longer to reach a similar reward level. The hybrid strategy enables the unmanned ship to achieve high-precision following at an early stage, reducing the training time. During the training process, the hybrid strategy reward function gradually approaches 0, the state of the controlled ship approaches the state of the pilot ship, and the path following error is small.

[0216] The comparison of the actions selected by DQN and the input weights calculated by backstepping control is shown in the figure below. Figure 6 As shown in the figure, the experiment shows that the training effect is best when the weight alpha = 0.5. In the early stage of training, DQN needs a lot of exploration. At this time, the introduction of 50% BC control performance can effectively reduce fluctuations, smooth the reward curve, and accelerate the convergence speed. Compared with DQN dominant control with alpha = 0.7, the weight balance of alpha = 0.5 significantly reduces the uncertainty of DQN output in the early stage. In the later training, DQN has learned an effective strategy. By integrating with BC, the action output is more stable, the risk of deviation from the target state is reduced, and the reward and loss fluctuations are reduced, which enhances the convergence and stability of the model. Low weight (alpha < 0.5) will weaken DQN's learning ability, resulting in a lack of flexibility in the face of complex environments; while high weight (alpha = 0.7) will cause DQN to be too dominant, control fluctuations will increase, and performance will be unstable in interference environments.

[0217] Comparison of communication and control coordination performance under different interference intensities Figure 7 , Figure 7 The figure shows the changes in the reward function under high interference (red curve) and low interference (blue curve) conditions. It can be seen that even under high interference conditions, the reward curve shows obvious oscillations in the early stage, but as the training process progresses, it gradually stabilizes and approaches the reward level under low interference conditions, and achieves a higher average reward value. This shows that the control strategy has strong robustness in interference environments. Even in the case of high data packet loss rate and large delay, the system can still effectively adjust the control parameters to maintain the stability of path following.

Claims

1. A path following method for unmanned surface ships based on DQN and backstepping control, characterized in that: The steps include: S1: Design a communication system based on the MAVLink protocol to achieve real-time transmission of status information between the pilot ship and the controlled ship; S2: In the constructed model training environment, the dynamic models of the pilot ship and the controlled ship are constructed, and the rendering window is used to observe the ship following performance in real time during the model training process; S3: Use the constructed DQN model to optimize the path following strategy through the reward function; S4: Under unstable communication conditions, backstepping control is used for real-time feedback compensation; When packet loss or delay occurs in communication, the state prediction method is used to enable the controlled ship to select the next following action through state prediction; S5: Path following in complex ocean environments is achieved by dynamically adjusting the weights of DQN and backstepping control.

2. According to claim 1, a path following method for an unmanned surface ship based on DQN and backstepping control is characterized in that: In step S1, the communication system uses the MAVLink protocol to achieve status data transmission between the pilot ship and the controlled ship through WIFI, including position information and attitude information, and introduces an electromagnetic interference model through simulation to evaluate communication robustness.

3. The unmanned surface ship path following method based on DQN and backstepping control according to claim 2 is characterized in that: The operation of the communication system in step S1 includes: A1: Pilot ship sends information: collects the pilot ship's status information data and processes the data, and uses the MAVLink protocol to generate the Payload data part containing key control information; A2: WIFI protocol simulation and transmission: MAVLink data is sent via WIFI protocol; the simulation takes into account channel attenuation, noise interference and multipath effects, uses a dual-path model to reflect signal reflection and refraction, evaluates packet loss rate and delay, and simulates real wireless communication characteristics; A3: Introduction and control of electromagnetic interference: design an adjustable electromagnetic interference model, adjust interference parameters, and use WIFI protocol interference simulation; A4: Controlled ship receives information: The controlled ship extracts the position information and attitude data of the pilot ship through the MAVLink unpacking process. Abnormal data is recorded and fed back to the motion control algorithm to analyze the impact of interference on motion control.

4. The unmanned surface ship path following method based on DQN and backstepping control according to claim 1, characterized in that: The dynamic model in step S2 is divided into a pilot ship dynamic model and a controlled ship dynamic model, wherein the pilot ship dynamic model includes: The dynamic equation of the pilot ship is composed of inertia, Coriolis force, centrifugal force and damping force, and the equation form is as follows: Where M is the inertia matrix, C(v) is the Coriolis and centrifugal force matrix, D(v) is the damping matrix, and τ is the control input; The expression of inertia matrix M is: Where m is the mass of the ship, represents the hydrodynamic additional mass coefficient, X g is the position of the ship's center of gravity in the hull coordinate system, I z is the moment of inertia of the ship around the z-axis; The expression of the Coriolis and centrifugal force matrix C(v) is: Among them, u, v, and r represent the longitudinal velocity, lateral velocity, and yaw angular velocity of the ship, respectively; The damping matrix D(v) includes linear and nonlinear damping terms and is expressed as: Linear term: represents the linear damping coefficient, describing the linear relationship between speed and resistance; Nonlinear term: represents the nonlinear damping coefficient, describing the effect of the square or cube of speed on resistance; The rotation matrix R(ψ) is used to transform the velocity in the hull coordinate system to the global coordinate system, and its expression is: The control input τ acts on the unmanned ship in the form of thrust and torque: The state update equation is divided into two parts: position update and velocity update; Location Updates: Where, p=[x, y, ψ] T ,V=[u,v,r] T ; Speed ​​Update: The dynamics model of the controlled vessel includes: The position update is obtained by integrating the velocity, using Euler's method for numerical integration: p new =p old +Δt·R(ψ)·v where p = [x, y, ψ] T is the position vector, V = [u, v, r] T is the velocity vector, Δt is the time step; The attitude rotation matrix R(ψ) is used to map the velocity [u, v, r] to the global coordinate system, and then update the position of the unmanned ship [x, y, ψ]; Speed ​​update formula: The velocity is calculated by dividing the displacement in the global coordinate system by Re-mapped back to the hull coordinate system.

5. The unmanned surface ship path following method based on DQN and backstepping control according to claim 1, characterized in that: In step S2, the rendering window is based on the gym.envs.classic_control.rendering module, the display area is set, the background is set to blue to simulate the color of the ocean environment, and different colors are used to distinguish ships. The position information of the ship is updated in real time through the Transform object, so that the ship moves dynamically in the rendering window.

6. The unmanned surface ship path following method based on DQN and backstepping control according to claim 1, characterized in that: In step S3, the DQN model is trained using experience replay and epsilon-greedy strategy, including an evaluation network and a target network, wherein the parameters of the target network are synchronized regularly to improve training stability; The goal of the DQN model is to find an optimal strategy so that the state-action value function Q(s, a) reaches the maximum value; the optimal state-action value function satisfies the Bellman equation: Among them, s is the current state, a is the current action, s' is the next state, and a' is the action in the next state; Q * (s,a) is the optimal state-action value function, which indicates the maximum value of the expected cumulative reward that the agent can obtain after taking action a in state s; E[·] is the mathematical expectation, which indicates the uncertainty and expectation calculation of the next state s'; r is the immediate reward, which indicates the reward obtained immediately after taking action a; γ is a weighting factor in the range of [0,1], which is used to weigh the importance of current rewards and future rewards; Indicates the maximum Q value that can be obtained by selecting action a' in the next state; The goal of the DQN model is to minimize the following loss function so that the Q network gradually approaches the optimal state-action value function: Among them, θ is the parameter of the current Q network, θ - are the parameters of the target network; the parameters of the target network Qtarget are fixed for a period of time to improve training stability.

7. The unmanned surface ship path following method based on DQN and backstepping control according to claim 6, characterized in that: The DQN model in step S3 is instantiated through the DQN class, which contains two deep neural networks, one is the current Q network eval_net, which is used to estimate the Q value of the current state; the other is the target Q network target_net, which is used to estimate the target Q value of the next state; In the DQN update method, a small batch of experience samples are first extracted from the experience replay buffer. For each sample, the Q value target is calculated according to the Bellman equation: Among them, θ- is the parameter of the target network; Calculate the mean square error between the estimated Q value q_eval of the current Q network and the target Q value q_target, and minimize the error through back propagation. This process makes the estimated value of the Q network gradually approach the true Q value.

8. The unmanned surface ship path following method based on DQN and backstepping control according to claim 1, characterized in that: In step S3, the reward function evaluates the effect of the control strategy by calculating the difference between the state of the controlled ship and the state of the pilot ship, and evaluates the overall control effect within the round; The reward function includes: Total reward for each round: The total reward output at the end of each round is the cumulative value of all time-step rewards in that round; Immediate reward: Immediate feedback calculated at each time step to evaluate the quality of the current action; calculated as the difference between the current state and the goal state; Sliding average reward: The sliding average reward is obtained by calculating the average of the total rewards of a certain number of recent rounds; Where N is the size of the sliding window and the total reward i is the total reward of the i-th round.

9. The unmanned surface ship path following method based on DQN and backstepping control according to claim 1, characterized in that: In step S4, the backstepping control realizes feedback compensation by calculating the error and recursively designing the controller, and generates control input according to the path deviation and real-time feedback under unstable communication conditions to correct the state of the controlled ship; Specifically include: Calculate the error z1 based on the environmental state and the reference state: z1=env-env r Then we get the intermediate variables ω and ω=k1·z1 Then calculate the error z2: z2=env-env r -ω Generate control input τ: τ=-k2·z2+kinetic model term; The state prediction method uses the latest valid state information of the pilot ship, including position, speed and acceleration, combined with its dynamic model to predict state changes in a short period of time; the prediction formula is as follows: Among them, s predicted is the predicted pilot ship state, s lastvalid The most recently received valid status, is the speed of the pilot vessel, and Δt is the estimated time difference.

10. The unmanned surface ship path following method based on DQN and backstepping control according to claim 1, characterized in that: In step S5, the weight distribution ratio of DQN and backstepping control is dynamically adjusted: tau=alpha*tau_dqn+(1-alpha)*tau_bc.

Images