Adaptive control method for trajectory tracking of unmanned ship

Abstract

The present invention provides an unmanned ship trajectory tracking adaptive control method, comprising the following steps: step S1: constructing an unmanned ship environmental interference model based on a wind model and a flow model; constructing an unmanned ship kinematic model based on position and heading; constructing an unmanned ship dynamic model based on lateral force, longitudinal force and vertical force; step S2: combining the unmanned ship environmental interference model, the unmanned ship kinematic model and the unmanned ship dynamic model to obtain an unmanned ship motion model under environmental interference; step S3: tracking the trajectory of the unmanned ship motion model under environmental interference through an MPC controller. This method not only improves the navigation stability and trajectory tracking accuracy of the unmanned ship under various complex climatic conditions, but also significantly enhances its adaptability to sudden environmental changes in practical applications.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment control, and in particular to an unmanned ship trajectory tracking adaptive control method. Background Art

[0002] With the rapid development of unmanned boat technology, it has shown a wide range of application potential in the fields of marine exploration, scientific research and resource development. However, there are still many challenges in realizing adaptive trajectory tracking control in complex marine environments. The variability and uncertainty of the marine environment pose a severe test to the control system of unmanned ships. Common environmental factors such as wind, waves, and currents have high uncertainty and dynamic change characteristics. These complex factors cause unpredictable fluctuations in the motion state of unmanned ships, making it difficult for traditional control strategies to achieve the desired effect. In addition, the motion model of unmanned ships in the marine environment is extremely complex, involving a variety of nonlinear dynamic characteristics and multi-degree-of-freedom motion, which makes accurate modeling a difficult task. Environmental factors such as waves, wind and currents have a significant impact on the dynamic behavior of unmanned ships. The randomness and time-varying nature of these factors further increase the complexity of modeling.

[0003] Therefore, a new control method is urgently needed to improve the adaptability and robustness of unmanned ship control systems in complex marine environments. Summary of the invention

[0004] The present invention proposes an unmanned ship trajectory tracking adaptive control method to solve the technical problem that the existing unmanned ship trajectory tracking control method is difficult to solve the influence of the complex and diverse marine environment on the unmanned ship, resulting in trajectory tracking failure.

[0005] In order to solve the above technical problems, the present invention provides an unmanned ship trajectory tracking adaptive control method, comprising the following steps:

[0006] Step S1: construct an unmanned ship environmental interference model based on the wind model and the flow model; construct an unmanned ship kinematic model based on the position and heading; and construct an unmanned ship dynamic model based on the lateral force, longitudinal force and vertical force;

[0007] Step S2: combining the unmanned ship environmental interference model, the unmanned ship kinematic model and the unmanned ship dynamic model to obtain an unmanned ship motion model under environmental interference;

[0008] Step S3: Tracking the trajectory of the unmanned ship motion model under the environmental interference through the MPC controller.

[0009] Preferably, the unmanned ship kinematic model and the unmanned ship dynamic model adopt the Fossen model.

[0010] Preferably, the expression of the unmanned ship environment interference model is:

[0011] ;

[0012] ;

[0013] In the formula, , , Indicates the forces and moments on the unmanned ship in the longitudinal, transverse and bow rolling directions; Indicates the air density; and They represent the orthogonal projection and side projection areas above the waterline of the unmanned ship respectively; and They represent the longitudinal and transverse speeds of the ship after being affected by the wind; L represents the length of the ship; Indicates the wind coefficient on the X-axis, i.e. the transverse direction of the ship; Indicates the wind coefficient along the Y axis, i.e. the longitudinal direction of the ship; It represents the wind moment coefficient around the Z axis, i.e. the vertical direction of the ship; Indicates wind direction; and It represents the relative longitudinal and lateral speeds of the unmanned boat after being affected by the water flow; u and v represent the longitudinal and lateral speeds of the unmanned boat respectively; and Respectively represent the direction and speed of water flow; Indicates the ship's heading.

[0014] Preferably, step S3 comprises:

[0015] Step S311: At time k, the state of the unmanned ship can be read ;

[0016] Step S312: performing prediction within a prediction step length of N steps using the unmanned ship kinematic model and the unmanned ship dynamic model;

[0017] Step S313: subtract the predicted state of the unmanned ship from the current reference trajectory to obtain an error prediction result;

[0018] Step S314: establishing an MPC quadratic programming model, and calculating a cost function based on the error prediction result, the prediction step size, the state weight matrix, and the control input weight matrix;

[0019] Step S315: Establish constraint conditions according to the physical constraints of the unmanned ship to minimize the cost function and obtain the optimal control sequence.

[0020] Preferably, in step S3, the state weight matrix of the MPC controller is adjusted by a reinforcement learning algorithm, and the prediction step size is optimized in real time based on the reward value of the reinforcement learning algorithm.

[0021] Preferably, the reinforcement learning algorithm adopts an actor-critic algorithm.

[0022] Preferably, the method of adjusting the state weight matrix of the MPC controller using the actor-critic algorithm comprises:

[0023] Step S321: The initial state weight matrix of MPC As input to the actor-critic algorithm;

[0024] Step S322: The action network in the actor-critic algorithm performs actions according to the input state and calculates the state weight matrix Adjustment: By adjusting the state weight matrix The three values ​​in are increased or decreased by 0.1 or remain unchanged to adjust the three values;

[0025] Step S323: After executing the action, a new state weight matrix is ​​generated , and apply it to the MPC optimizer. The optimizer will output a future input to feed back to the MPC prediction model and continue to generate new prediction states. After N steps of prediction, the MPC controller will solve the optimal control input and input it into the unmanned ship system.

[0026] Preferably, the method for optimizing the prediction step length in real time using the reward value of the actor-critic algorithm includes:

[0027] Step S331: Calculate the change in reward value in real time;

[0028] Step S332: adjusting the prediction step size based on the change amount.

[0029] Preferably, in step S331, the change in the reward value The expression is:

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] In the formula, Represents the change in reward value, Represents the reward value at the current moment, Indicates the reward value at the previous moment; represents the reward at time t; , represents the discount factor reflecting the value of the reward at different times; Indicates the transverse error of the ship; Indicates the longitudinal error of the ship; Indicates the ship's heading error.

[0037] Preferably, in step S332, the expression for adjusting the prediction step length is:

[0038] ;

[0039] ;

[0040] In the formula, Represents the change in reward value; N represents the prediction step; A represents the adjustment amplitude of the prediction step N.

[0041] The beneficial effects of the present invention include at least: the present invention constructs an unmanned ship environmental interference model based on a wind model and a flow model; an unmanned ship kinematic model based on position and heading; and an unmanned ship dynamic model based on lateral force, longitudinal force and vertical force. And combines them to obtain an unmanned ship motion model under environmental interference, which integrates the nonlinear dynamics and kinematic characteristics of ship motion and the influence of wind direction, intensity and water flow, and then uses an MPC controller to track the motion trajectory of the model, which not only improves the navigation stability and trajectory tracking accuracy of the unmanned ship under various complex climatic conditions, but also significantly enhances its adaptability to sudden environmental changes in practical applications. This flexible and intelligent control strategy has laid a solid technical foundation for the widespread application of unmanned ships in commercial shipping, marine exploration, environmental monitoring and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the method flow of Example 1;

[0043] Figure 2 A diagram of a real ship model and a schematic diagram of unmanned ship modeling according to an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the principle of the actor-critic algorithm;

[0045] Figure 4A control method flow chart of an AC-MPC adaptive controller is provided for Example 2;

[0046] Figure 5 A control principle diagram of an AC-MPC adaptive controller is provided for Example 2;

[0047] Figure 6 Convergence diagram of the reward value of the sine function trajectory reinforcement learning according to the embodiment of the present invention;

[0048] Figure 7 Comparison diagram of sine function trajectory tracking in an embodiment of the present invention;

[0049] Figure 8 A sine function trajectory tracking error diagram of an embodiment of the present invention;

[0050] Fig. 9 A graph showing a change in the Q value of a sine function trajectory weight parameter according to an embodiment of the present invention;

[0051] Fig.10 Comparison diagram of variable step length of sine function trajectory tracking in an embodiment of the present invention;

[0052] Fig.11 A variable step error diagram of a sinusoidal function trajectory tracking according to an embodiment of the present invention;

[0053] Fig.12 Comparison diagram of reward value convergence variable step length of sine function trajectory reinforcement learning in an embodiment of the present invention;

[0054] Fig.13 A graph showing the numerical variation of the sine function trajectory prediction step length N according to an embodiment of the present invention;

[0055] Fig.14 Convergence diagram of the reward value of square trajectory reinforcement learning in an embodiment of the present invention;

[0056] Fig.15 A comparison diagram of square trajectory tracking according to an embodiment of the present invention;

[0057] Fig.16 A square trajectory tracking error diagram of an embodiment of the present invention;

[0058] Fig.17 A diagram showing a change in the Q value of a square trajectory weight parameter according to an embodiment of the present invention;

[0059] Fig.18 A comparison diagram of variable step lengths for square trajectory tracking according to an embodiment of the present invention;

[0060] Fig.19 Square trajectory tracking variable step error diagram of an embodiment of the present invention;

[0061] Fig. 20Convergence of reward values ​​for square trajectory reinforcement learning with variable step lengths comparison diagram in an embodiment of the present invention;

[0062] Fig.21 A diagram showing the numerical variation of the square trajectory prediction step length N according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0064] Embodiment 1:

[0065] like Figure 1 As shown, an embodiment of the present invention provides an unmanned ship trajectory tracking adaptive control method, comprising the following steps:

[0066] Step S1: construct an unmanned ship environmental interference model based on the wind model and the flow model; construct an unmanned ship kinematic model based on the position and heading; and construct an unmanned ship dynamic model based on the lateral force, longitudinal force and vertical force;

[0067] Step S2: combining the unmanned ship environmental interference model, the unmanned ship kinematic model and the unmanned ship dynamic model to obtain the unmanned ship motion model under environmental interference;

[0068] Step S3: Track the trajectory of the unmanned ship motion model under environmental interference through the MPC controller.

[0069] In this embodiment, a three-degree-of-freedom ship Fossen model is first established, and then the model predictive control (MPC) method is used to control the ship to track the trajectory.

[0070] Specifically, the model predictive controller based on the ship Fossen model under environmental disturbance in this embodiment is described as follows.

[0071] Model predictive control is a computer control algorithm that has gradually developed in the process of industrial practice. The MPC controller consists of two parts: the prediction model and the optimizer. First, the MPC controller measures or reads the current state of the system, then the prediction model uses the measured state from the system to calculate the predicted state, and finally the optimizer uses the predicted state as the initial condition, calculates the future control input under the control objective and constraints, and sends the future input back to the model. After all prediction steps are completed, the first set of values ​​of the control input calculated by the prediction model is sent to the controlled system.

[0072] In this embodiment, the planar motion of the ship is described by the kinematic and dynamic models of three degrees of freedom (DOF). The Fossen model combines the nonlinear dynamic and kinematic characteristics of the ship's motion, and at the same time, uses the real ship model Cyber ​​Ship Ⅱ. The real ship model diagram of Cyber ​​Ship Ⅱ and the schematic diagram of unmanned ship modeling are shown in Figure 2 shown.

[0073] The Fossen motion model of the unmanned ship under environmental interference can be expressed as:

[0074] (1)

[0075] Where t represents the time, Represents the position vector in the inertial coordinate system, including the position coordinates and ship heading ; Represents the velocity vector in the ship coordinate system, including the surge velocity , sway speed and bow speed , It is the transformation matrix from the inertial coordinate system to the hull coordinate system, as follows:

[0076] (2)

[0077] matrix , , denote the inertia matrix, Coriolis center matrix and damping matrix respectively, and the variables Represents the forces and moments of the control input, including longitudinal forces and moments , lateral force and moment , yaw directional force and moment , Represents environmental disturbance forces and moments.

[0078] In the unmanned ship trajectory tracking control of this embodiment, a wind model and a flow model are added. The direction and strength of the wind will directly affect the course and speed of the ship. The flow model describes the influence of water flow in the ocean, but compared with the wind, its influence is relatively more stable and more predictable. The specific modeling is as follows:

[0079] Specifically, the direction is established and speed is The force and moment of steady wind acting on the ship are expressed as:

[0080] (3)

[0081] in, , , Represents the forces and moments on the unmanned ship in the longitudinal, transverse and bow rolling directions. and are the longitudinal and transverse speeds of the ship affected by the wind, is the air density, and They are the orthogonal projection and side projection areas above the waterline of the unmanned ship, L represents the length of the ship, It represents the wind coefficient on the X-axis, i.e. the transverse direction of the ship. Indicates the wind coefficient along the Y axis, i.e. the longitudinal direction of the ship. It represents the wind moment coefficient around the Z axis, i.e. the vertical direction of the ship.

[0082] Assume that the direction and speed of uniform water flow are and , the relative speed of the unmanned ship is:

[0083] (4)

[0084] Among them, u and v represent the longitudinal and lateral speeds of the unmanned ship respectively. and Represents the relative longitudinal and lateral speeds of the unmanned ship after being affected by the water flow.

[0085] Specifically, in step S3, a method for tracking the trajectory of the unmanned ship motion model under the environmental interference through the MPC controller is as follows.

[0086] At time k, read the state of the unmanned ship ,based on And the nonlinear dynamic model, that is, formula (1), is predicted within the prediction step length N steps, and the following formula is obtained:

[0087] (5)

[0088] (6)

[0089] in is the sampling time, It indicates the prediction of the kinematic state of the unmanned ship at the next moment. It indicates the prediction of the dynamic state of the unmanned ship at the next moment. represents the force and moment of wind disturbance at time t, Represents the force and torque of the flow disturbance at time t.

[0090] definition , is the state input and control input predicted in the i-th step in the k-time prediction model, the prediction result can be expressed as:

[0091] (7)

[0092] (8)

[0093] definition is the tracking error at time k, and the state of the unmanned ship is subtracted from the current reference trajectory:

[0094] (9)

[0095] Get the error prediction result:

[0096] (10)

[0097] After that, the MPC quadratic programming model is established and its cost function is calculated. The goal of this algorithm is to make the following weighted cost function Minimize:

[0098] (11)

[0099] Where N represents the prediction step length, the matrix and are positive definite symmetric matrices, representing the state weight matrix and the control input weight matrix respectively:

[0100] ;

[0101] Establish constraints based on the physical constraints of the unmanned ship and the actual situation:

[0102] (12)

[0103] (13)

[0104] in and Represent the maximum value and maximum rate of change of thruster force and torque respectively.

[0105] By minimizing the cost function, the optimal control sequence is obtained , at time k, only the first element in the control sequence is applied , and repeat the whole process at the next moment.

[0106] In this embodiment, the unmanned ship trajectory tracking technology based on the ship Fossen model and MPC control is combined and applied to the control system of the unmanned ship, which can achieve more accurate track tracking and navigation control. The Fossen model can accurately describe the motion characteristics of the ship, while the MPC control can make multi-step predictions based on the system model, optimize the control input, and ensure the stable operation of the system in complex sea conditions.

[0107] When dealing with environmental disturbances such as wind and current, the unmanned ship trajectory tracking technology based on the Fossen model and MPC control shows significant advantages. By adjusting the state weight parameters in the MPC controller, the performance of the controller can be effectively optimized to enhance its resistance to external disturbances. Specifically, the Fossen model accurately captures the dynamic response characteristics of the ship in a complex fluid environment, enabling the MPC controller to flexibly adjust the control strategy based on real-time state information. Through dynamic optimization of the state weight parameters, the controller can give priority to key state variables under current environmental conditions, thereby quickly generating optimal control inputs when faced with changes in wind speed and water flow direction.

[0108] Example 2

[0109] At present, in order to improve the adaptability and robustness of unmanned ship control systems in complex marine environments, a lot of research and practical work has been done based on various control methods and their improvement methods. There are two main methods for adaptive adjustment of control parameters in MPC controllers, including methods based on numerical optimization and methods based on reinforcement learning. Methods based on numerical optimization usually dynamically adjust control parameters through iterative algorithms such as genetic algorithms, particle swarm algorithms or gradient descent to optimize system performance indicators. The advantages of this type of method are mature theory, strong convergence, and low dependence on existing models; but its disadvantages are high computational complexity and slow optimization process. Reinforcement learning-based methods adjust control parameters through interactive learning with the environment, such as Q learning or deep reinforcement learning. These methods can gradually improve control performance in unknown environments and have strong adaptability and robustness, but their training process may require a lot of data and time. However, at present, for the control parameters in model predictive control, there are many parameters involved and there is no objective optimization standard for adaptive adjustment of control parameters. Therefore, when the MPC controller uses unreasonable control parameters to control the unmanned ship for trajectory tracking, it cannot cope with environmental interference, resulting in large trajectory tracking errors.

[0110] In the trajectory tracking of unmanned ships, MPC control still has the following limitations. For example, the values ​​of the weight matrices Q and R in the evaluation function have a great influence on the tracking effect, and it is difficult and time-consuming to manually adjust these two values. On the other hand, the size of the prediction step N in the prediction model also affects the solution rate of the MPC algorithm and the tracking effect. Therefore, this embodiment introduces a reinforcement learning algorithm, which can improve the performance of the controller by finding the optimal strategy through continuous learning and experimentation. In the complex unmanned ship trajectory tracking control task, the reinforcement learning algorithm is used to adaptively adjust the state weight Q and prediction step N in the MPC algorithm, enhance the anti-disturbance ability of the controller, and achieve faster and more stable trajectory tracking.

[0111] Specifically, the AC-MPC adaptive controller of this embodiment will be described below starting from the principle of the Actor-Critic method.

[0112] Deep reinforcement learning is based on neural networks and reinforcement learning, and has computing and decision-making capabilities. The actor-critic algorithm is one of the deep reinforcement learning algorithms. It is a reinforcement learning method that combines strategy evaluation (Critic) and strategy improvement (Actor). Its essence is to obtain the optimal strategy through the interaction between the environment and the intelligent agent.

[0113] Specifically, the AC algorithm can be described as a Markov Decision Process (MDP). MDP consists of the state space S, the action space A of the control strategy, the state transition probability P, and the reward function R. Among them, the state transition probability is:

[0114] (14)

[0115] p Indicates that under action a, state s is transferred to state The mathematical expression of this strategy can be expressed as:

[0116] (15)

[0117] Represents the probability distribution of action a generated in state s.

[0118] For a sequence , assuming that in the sequence The sum of the rewards at each moment is , then in the strategy When the expected value of the reward yes:

[0119] (16)

[0120] We expect to obtain the optimal strategy, that is, , It represents the expected value of the reward value. The input of the action network is state s, and the output is action. The input of the value network is state s, and the output is the action-value function. :

[0121] (17)

[0122] The main idea of ​​the actor-critic algorithm is to use the learning ability of neural networks to build an interactive model consisting of two networks, parameterize the strategy and evaluation function respectively, and learn and update the strategy and value function simultaneously until the optimal strategy is obtained. The algorithm diagram of the actor-critic is shown in the figure below: Figure 3 shown.

[0123] First, the action (Actor) network applies the initial control strategy to the system or environment, and the environment updates the state and calculates the corresponding reward value. The evaluation (Critic) network evaluates the value of the control strategy based on the state and reward value. Based on the evaluation of the strategy, the control strategy is improved by updating the action (Actor) network, and the value function generated by the new strategy is compared with the value function of the previous strategy to determine whether it is necessary to continue iterating. The actor-critic structure directly performs gradient ascent in the direction of maximizing the value function to solve the optimal strategy.

[0124] As a dynamic system with complex uncertainties, the trajectory tracking task of unmanned boats is particularly challenging when facing environmental disturbances. The selection of fixed controller parameters in traditional control strategies is difficult to effectively deal with unknown states and complex interferences. Therefore, it is urgent to introduce adaptive controller parameters to ensure that the unmanned boat exhibits excellent maneuverability and anti-interference capabilities during trajectory tracking. A notable feature of the actor-critic algorithm is that it can adaptively make optimization decisions based on the current state of the system. Therefore, the actor-critic algorithm in deep reinforcement learning is used in this embodiment to achieve dynamic adjustment of controller parameters. This method can effectively improve the responsiveness and robustness of unmanned boats in complex environments.

[0125] The control method of the AC-MPC adaptive controller provided in this embodiment is as follows: Figure 4 As shown, the principle is as Figure 5 As shown in Figure 1, the environment of the actor-critic algorithm is composed of an unmanned ship system and an MPC controller. By obtaining the initial state of the unmanned ship , is input into the prediction model of the MPC controller, and the actor-critic algorithm is used to adjust the state weight matrix in the MPC controller Therefore, the state space S consists of three dimensions:

[0126] (18)

[0127] The initial state weight parameters of MPC As input to the AC algorithm.

[0128] The Actor (action) network in the actor-critic algorithm performs actions based on the input state and adjusts the parameters Therefore, the action space A contains three actions:

[0129] (19)

[0130] By changing the state weight matrix Adjust the three values ​​by increasing them by 0.1, decreasing them by 0.1 or keeping them unchanged.

[0131] After executing the action, a new weight matrix is ​​generated , and applies it to the optimizer, which will output a future input to feed back to the prediction model and continue to generate new prediction states. After N steps of prediction, the MPC controller will solve the optimal control input and input it into the unmanned ship system.

[0132] At the same time, the reward is generated based on the difference between the updated system status of the unmanned ship and the reference trajectory. r :

[0133] (20)

[0134] in represents the error of the ship in the x direction, represents the error of the ship in the y direction, Indicates the ship's heading error.

[0135] Calculate cumulative rewards :

[0136] (twenty one)

[0137] in is a discount factor that reflects the value of the reward at different times, The reward R is input to the Critic (evaluation) network to generate the value function To evaluate the quality of the current strategy:

[0138] (twenty two)

[0139] in, is the current state, is the action being performed at the current moment, Indicates that in the current strategy Next, find the expected value of the future sequence.

[0140] Specifically, the update process of the actor-critic algorithm can be expressed by the following formula:

[0141] 1) For the evaluation (Critic) network: value-based update, using Temporal-Difference (TD) error to update:

[0142] (twenty three)

[0143] (twenty four)

[0144] in, is the current reward value, Is the current state The state value function under Indicates the next moment The state value function at time.

[0145] 2) For the Actor network: the policy gradient method is used, by parameterizing the network as And adjust the parameters in the direction of increasing the reward. The action (Actor) network update can be expressed by the following formula:

[0146] (25)

[0147] in, Representation parameters The update rate, is the learning rate, is the current state, Is the current strategy Next, select the action. represents the gradient of the policy parameters.

[0148] Afterwards, based on the difference between the current moment reward and the previous moment reward value, when the reward value increases, the controller can autonomously adjust the prediction step size N to use better weight parameters to predict more steps; on the contrary, when the reward value decreases, the controller can select a smaller prediction step size N to reduce the application of inappropriate weight parameters and prediction models. In this embodiment, the specific adjustment method of the prediction step size N is based on the following formula:

[0149] First calculate the change in reward value:

[0150] (26)

[0151] in, Represents the change in reward value, Represents the reward value at the current moment, Represents the reward value at the previous moment.

[0152] Calculate the value of the prediction step size N:

[0153] (27)

[0154] The definition of x value is determined by the change of reward value:

[0155] (28)

[0156] A represents the adjustment range of the prediction step size N. Since the value of N in the scenario of this embodiment should not be too large or too small, the value of A is set to 2 or 3.

[0157] In this embodiment, the Actor-Critic (AC) method in reinforcement learning is used to dynamically adjust the state weight parameters in the MPC controller, thereby realizing an adaptive resistance mechanism to environmental interference. Specifically, the AC method evaluates the effectiveness of the current state weight parameters through the reward value generated at each step. When the reward value increases, it indicates that the selected state weight parameters perform well under the current environmental conditions. At this time, the system will increase the prediction step size of the MPC controller accordingly to make full use of these superior parameters, thereby improving the control performance. On the contrary, when the reward value decreases, it means that the effect of the selected state weight parameters is not good. At this time, the system will reduce the prediction step size and reduce the dependence on the parameter group. Through this forward optimization process, an adaptive feedback mechanism is formed between the state weight parameters and the prediction step size, so that the MPC controller can flexibly adjust its control strategy in a constantly changing environment to ensure the track tracking accuracy and navigation stability of the unmanned ship in complex sea conditions. This dynamic adjustment mechanism not only enhances the robustness of the control system, but also provides higher autonomy and reliability for the unmanned ship in various application scenarios, fully reflecting the innovation of the present invention and its extensive potential in practical applications.

[0158] When the reward value reaches a certain threshold or the number of training times reaches the set maximum value, the AC network will stop training and obtain the final weight parameter Q and prediction step size N. The working principle and parameter update process of the AC-MPC controller show that the actor-critic algorithm can input the updated weight parameter Q and prediction step size N into the MPC controller at each step, thereby improving the control performance of the MPC controller and reducing the trajectory tracking error.

[0159] In order to further verify the feasibility of the AC-MPC controller designed in this embodiment, simulation experiments based on two experimental scenarios were carried out, and the trajectory tracking control results of the unmanned ship with and without environmental interference were compared. The ship model used was "CyberShip II" and the basic parameters were shown in Table 1. The simulation experiments were carried out in MATLAB R2023a.

[0160] Table 1: Basic parameters of the unmanned ship CyberShip II

[0161]

[0162] The structures of the Actor network and Critic network used in the AC algorithm are shown in Table 2.

[0163] Table 2: Critic network and Actor network structure

[0164] Critic Network

[0165]

[0166] Actor Network

[0167]

[0168] The initial motion state of the unmanned ship, environmental disturbances, and some initial parameter settings of the AC-MPC controller are shown in Table 3. The initial ship heading is set to 0°, the initial values ​​of the three values ​​in the weight matrix parameter Q of the MPC controller are all 1, the hyperparameter learning rate in AC is set to 0.001, and the discount factor is set to 0.98. The main difference between the two scenarios is that the reference trajectories tracked by the ship are different, which are sine function curves and square curves. In each scenario, a control experiment is set to determine whether the AC algorithm is added for adaptive parameter adjustment.

[0169] Table 3 Initial parameter settings for unmanned ship trajectory tracking experiment

[0170]

[0171] The experimental results of unmanned ship trajectory tracking with two reference trajectories are analyzed as follows.

[0172] The reference trajectory of scenario 1 is a sine function, and the specific expression is as follows:

[0173] (29)

[0174] in and Represent the horizontal and vertical coordinates of the reference trajectory, and t represents the tracking time. The initial position of the ship is (0, 0). For other parameter settings, refer to Table 3. In scenario 1, the experiment uses reinforcement learning for training. The changes and convergence of the cumulative reward value of the feedback are shown in Figure 6 As shown in the figure, the comparison of using the original MPC controller and the AC-MPC adaptive controller to control the unmanned ship to track the trajectory with and without environmental interference is shown in the figure. Figure 7 As shown, the tracking error analysis is as follows Figure 8 The changes of the three values ​​in the weight parameter Q adjusted by the AC method are shown in Fig. 9 As shown, Fig. 9(a) is the case without environmental interference. Fig. 9 (b) in the figure shows the situation with environmental interference. Under environmental interference, two methods, fixed prediction step size and adaptive prediction step size, are used to conduct unmanned ship trajectory tracking experiments. The trajectory comparison is shown in the figure below. Fig.10 As shown, the tracking error is Fig.11 As shown in Figure 2, the convergence of the reinforcement learning reward value when using a fixed prediction step size and adaptively selecting the prediction step size is compared. Fig.12 As shown, Fig.12 (a) shows the case of using a fixed prediction step size. Fig.12 (b) in the figure shows the case of using adaptive prediction step size. The change of prediction step size N is as follows Fig.13 shown.

[0175] Figure 6 By visualizing the reward value and average reward value during the reinforcement learning training process, it is shown that the AC method selected by the present invention eventually converges the reward value through continuous training, and begins to converge at about the 150th round of training, with the reward convergence value around -50.

[0176] Figure 7 This shows that both the original MPC controller and the AC-MPC adaptive controller can achieve trajectory tracking, but in the sinusoidal function trajectory, the tracking error tends to increase at the vertex of the sinusoidal function. Figure 7 and Figure 8 It can be seen that after adding environmental disturbance, the control errors of the two controllers have increased to varying degrees compared with the absence of environmental disturbance. However, under environmental disturbance, the AC-MPC adaptive controller can control the unmanned ship closer to the reference trajectory than the original MPC controller. Especially in Figure 7 From the local trajectory comparison in , we can see that at the inflection point of the sine function, the unmanned ship needs to make a large angle turn. After the turn, the unmanned ship under the control of the original MPC controller deviates seriously to the lower left due to the influence of the wind and flow in the lower left direction, while the AC-MPC adaptive controller increases the state weight parameter in time, thereby effectively resisting interference and getting closer to the reference trajectory.

[0177] from Figure 8 It can be seen that under environmental disturbance, the peak tracking error generated by the designed AC-MPC adaptive controller after stable tracking is 0.25m, which is smaller than the 0.45m generated by the MPC controller.

[0178] Fig. 9The changes of three values ​​in the state weight parameters adjusted by the reinforcement learning method are shown. From the comparison of the figures, it can be seen that after adding environmental interference, the AC-MPC adaptive controller will appropriately increase the state weight parameters so that the unmanned ship can better track the reference trajectory, which is also in line with the actual situation and expected results.

[0179] Fig.10 The comparison of the unmanned ship trajectory tracking method using a fixed prediction step size N=5 and using the reinforcement learning method to autonomously select the prediction step size under environmental interference is shown. The tracking error is compared. Fig.11 As shown. Combined Fig.10 , Fig.11 and Fig.13 It can be seen that, compared with the fixed step size and the variable step size, before the unmanned ship turns at the inflection point of the sine function, the error increases and the reward value decreases, so the AC-MPC adaptive controller selects a smaller prediction step to reduce the use of unreasonable prediction models. After adjustment and turning, the error decreases and the reward value increases. The AC-MPC adaptive controller will select a larger prediction step for prediction to increase the use of more accurate prediction models, thereby further reducing the tracking error. After such a forward optimization process, the peak tracking error of the unmanned ship after stable tracking is 0.02m, which is less than the fixed step size of 0.11m.

[0180] Fig.12 It is shown that when using a fixed prediction step size Fig.12 (a) and variable step length as Fig.12 (b) The convergence of reward values ​​in two cases. As can be seen from the figure, when using a variable step-size controller, due to the unreasonable prediction model, the prediction step size is reduced, the number of optimizer solutions is reduced, and the solution speed is accelerated, which can make the reinforcement learning training process converge faster. The reward value of the adaptive controller with a variable step size converges around the 100th round, which is less than the 150th round of the fixed step size controller. Fig.13 The changes in the prediction step size N selected by the AC-MPC adaptive controller during the training process are shown. The above simulation results prove that the designed AC-MPC adaptive controller can adaptively adjust the state weight parameters and prediction step size of the MPC at a faster speed to achieve the effect of minimizing the tracking error.

[0181] The reference trajectory of scene 2 is a square with a side length of 20. The specific expression is as follows:

[0182] (30)

[0183] In scenario 2, the changes and convergence of the cumulative reward value of reinforcement learning training feedback are as follows: Fig.14As shown in the figure, the comparison of using the original MPC controller and the AC-MPC adaptive controller to control the unmanned ship to track the trajectory with and without environmental interference is shown in the figure. Fig.15 As shown, the tracking error analysis is as follows Fig.16 The changes of the three values ​​in the weight parameter Q adjusted by the AC method are shown in Fig.17 As shown, Fig.17 (a) is the case without environmental interference. Fig.17 (b) in the figure shows the situation with environmental interference. Under environmental interference, two methods, fixed prediction step size and adaptive prediction step size, are used to conduct unmanned ship trajectory tracking experiments. The trajectory comparison is shown in the figure below. Fig.18 As shown, the tracking error is Fig.19 As shown in Figure 2, the convergence of the reinforcement learning reward value when using a fixed prediction step size and adaptively selecting the prediction step size is compared. Fig. 20 As shown, Fig. 20 (a) shows the case of using a fixed prediction step size. Fig. 20 (b) in the figure shows the case of using adaptive prediction step size. The change of prediction step size N is as follows Fig.21 shown.

[0184] Fig.14 By visualizing the reward value and average reward value during the reinforcement learning training process, it is shown that the AC method selected by the present invention eventually converges the reward value through continuous training, and begins to converge at about the 140th round of training, with the reward convergence value around -20.

[0185] Fig.15 This shows that both the original MPC controller and the AC-MPC adaptive controller can achieve trajectory tracking, but in the square trajectory, the tracking error tends to become larger at the four vertices of the square. Fig.15 and Fig.16 It can be seen that after adding environmental disturbance, the control errors of the two controllers increase to varying degrees compared to when there is no environmental disturbance. Fig.15 It can be seen from the local enlarged image that even in the straight-line trajectory tracking stage, both controllers have certain tracking errors when controlling the unmanned ship. This is because the controller itself has some static errors when facing environmental interference. However, at this stage, the error of the AC-MPC adaptive controller is still smaller than that of the original MPC controller. At the second turn, the original MP slave controller has a certain overshoot, which increases the tracking error of the unmanned ship. The AC-MPC adaptive controller can reduce the unmanned ship's trajectory tracking error by increasing the state weight parameters in time, resisting environmental interference and reducing the overshoot. At the third turn, the AC-MPC adaptive controller can also make the unmanned ship adjust faster to get closer to the reference trajectory.

[0186] from Fig.16 It can be seen that under environmental disturbance, the peak tracking error generated by the designed AC-MPC adaptive controller after stable tracking is 0.32m, which is smaller than the 0.42m generated by the MPC controller.

[0187] Fig.17 The changes of three values ​​in the state weight parameters adjusted by the reinforcement learning method are shown. From the comparison of the figures, it can be seen that after adding environmental interference, the AC-MPC adaptive controller will appropriately increase the state weight parameters so that the unmanned ship can better track the reference trajectory, which is also in line with the actual situation and expected results.

[0188] Fig.18 The comparison of the unmanned ship trajectory tracking method using a fixed prediction step size N=5 and using the reinforcement learning method to autonomously select the prediction step size under environmental interference is shown. The tracking error is compared. Fig.19 As shown. Combined Fig.18 and Fig.19 It can be seen that at the second turning point, compared with the AC-MPC adaptive controller with a fixed step size, the AC-MPC adaptive controller with a variable step size will not have an anti-overshoot situation. At the third turning point, when the error increases, the AC-MPC adaptive controller with a variable step size selects a smaller prediction step size due to the decrease in the reward value, reducing the use of unreasonable state weight parameters and prediction models. When completing the turn and entering the straight-line tracking part, a larger prediction step size can be selected to reduce the tracking error, allowing the unmanned ship to quickly track the straight-line trajectory. When using a variable step size, the peak tracking error of the unmanned ship after stable tracking is 0.26m, which is less than the fixed step size of 0.32m.

[0189] Fig. 20 It is shown that when using a fixed prediction step size Fig. 20 (a) and variable step length as Fig. 20 (b) The convergence of the reward value in the two cases. As can be seen from the figure, when using a variable step-size controller, the reward value of the reinforcement learning training reaches convergence at the 100th round, which is less than the 130 rounds of the fixed step-size, and converges faster. Fig.21 The changes in the prediction step size N selected by the AC-MPC adaptive controller during the training process are shown. The above simulation results prove that in the square trajectory, the designed AC-MPC adaptive controller can adaptively adjust the state weight parameters and prediction step size of the MPC at a faster speed to achieve the effect of minimizing the tracking error.

[0190] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. As long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification.

[0191] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An unmanned ship trajectory tracking adaptive control method, characterized in that: The following steps are involved: Step S1: construct an unmanned ship environmental interference model based on the wind model and the flow model; construct an unmanned ship kinematic model based on the position and heading; and construct an unmanned ship dynamic model based on the lateral force, longitudinal force and vertical force; Step S2: combining the unmanned ship environmental interference model, the unmanned ship kinematic model and the unmanned ship dynamic model to obtain an unmanned ship motion model under environmental interference; Step S3: Tracking the trajectory of the unmanned ship motion model under the environmental interference through the MPC controller; In step S3, the state weight matrix of the MPC controller is adjusted by the actor-critic algorithm, and the prediction step length is optimized in real time based on the reward value of the reinforcement learning algorithm; The method for optimizing the prediction step length in real time using the reward value of the actor-critic algorithm includes: Step S331: Calculate the change in reward value in real time; Step S332: adjusting the prediction step length based on the change amount; In step S331, the change in the reward value The expression is: ； ； ； ； ； ； In the formula, represents the change in reward value, Represents the reward value at the current moment, Indicates the reward value at the previous moment; represents the reward at time t; represents the reward at time t+1; represents the reward at time t+2; represents the reward at time t+3; represents the reward at time t+k+1; , represents the discount factor reflecting the value of the reward at different times; Indicates the transverse error of the ship; Indicates the longitudinal error of the ship; Indicates the ship's heading error; Represents the position coordinates at time t; represents the ship's heading at time t; and represent the horizontal and vertical coordinates of the reference trajectory respectively; In step S332, the expression for adjusting the prediction step length is: ； ； In the formula, Represents the change in reward value; N represents the prediction step; A represents the adjustment amplitude of the prediction step N.

2. The unmanned ship trajectory tracking adaptive control method according to claim 1, characterized in that: The unmanned ship kinematic model and the unmanned ship dynamic model adopt the Fossen model.

3. The unmanned ship trajectory tracking adaptive control method according to claim 2, characterized in that: The expression of the unmanned ship environment interference model is: ； ； In the formula, , , Indicates the forces and moments on the unmanned ship in the longitudinal, transverse and bow rolling directions; Indicates the air density; and They represent the orthogonal projection and side projection areas above the waterline of the unmanned ship respectively; and They represent the longitudinal and transverse speeds of the ship after being affected by the wind; L represents the length of the ship; Indicates the wind coefficient on the X-axis, i.e. the transverse direction of the ship; Indicates the wind coefficient along the Y axis, i.e. the longitudinal direction of the ship; It represents the wind moment coefficient around the Z axis, i.e. the vertical direction of the ship; Indicates wind direction; and It represents the relative longitudinal and lateral speeds of the unmanned boat after being affected by the water flow; u and v represent the longitudinal and lateral speeds of the unmanned boat respectively; and Respectively represent the direction and speed of water flow; Indicates the ship's heading.

4. The unmanned ship trajectory tracking adaptive control method according to claim 1, characterized in that: Step S3 includes: Step S311: At time k, the state of the unmanned ship can be read ; Step S312: performing prediction within a prediction step length of N steps using the unmanned ship kinematic model and the unmanned ship dynamic model; Step S313: subtract the predicted state of the unmanned ship from the current reference trajectory to obtain an error prediction result; Step S314: establishing an MPC quadratic programming model, and calculating a cost function based on the error prediction result, the prediction step size, the state weight matrix, and the control input weight matrix; Step S315: Establish constraint conditions according to the physical constraints of the unmanned ship to minimize the cost function and obtain the optimal control sequence.

5. The unmanned ship trajectory tracking adaptive control method according to claim 1, characterized in that: The method of adjusting the state weight matrix of the MPC controller using the actor-critic algorithm includes: Step S321: The initial state weight matrix of MPC As input to the actor-critic algorithm; Step S322: The action network in the actor-critic algorithm performs actions according to the input state and calculates the state weight matrix Adjustment: Adjust the three values ​​in the state weight matrix Q by increasing or decreasing them by 0.1 or keeping them unchanged; Step S323: After executing the action, a new state weight matrix is ​​generated , and apply it to the MPC optimizer. The optimizer will output a future input to feed back to the MPC prediction model and continue to generate new prediction states. After N steps of prediction, the MPC controller will solve the optimal control input and input it into the unmanned ship system.

Images