An autonomous surface ship elastic train formation control method based on reinforcement learning

Abstract

The application discloses an autonomous water surface ship elastic train formation control method based on reinforcement learning, comprising the following steps: obtaining a virtual ship reference signal according to a constructed time-varying adjustment mechanism; obtaining error data of a following ship based on the virtual ship reference signal and a nonlinear mathematical model, so as to construct a ship virtual control law; performing filtering processing on the virtual control law of the forward direction and the bow swing direction of the ship to obtain a filtering signal; defining a dynamic error according to the filtering signal to obtain a dynamic error derivative, and using a neural network to approximate the dynamic error derivative to obtain an optimized dynamic error derivative; constructing a weight update law of an actor neural network and a critic neural network according to the error data; and based on the weight update law, designing a formation controller and an adaptive law according to the optimized dynamic error derivative combined with the dynamic error. The application solves the technical problem that the existing method lacks communication state identification and cannot effectively realize adaptive compensation for model uncertainty and unknown environmental disturbance.

Description

Technical Field

[0001] This invention relates to autonomous surface vehicles (ASMEs). This research relates to the field of motion control technology, and in particular to an autonomous surface vessel elastic train formation control method based on reinforcement learning. Background Technology

[0002] against The planar motion control problem primarily involves the propeller and rudder, which provide forward thrust and turning torque, respectively. During closed-loop formation control, control commands are calculated in real-time by the control system based on the relative relationship between the ship's attitude and the navigator's state information. This information is exchanged via a communication network topology, driving the formation towards a reference position. However, when performing missions in island and reef waters... Intermittent communication environments are inevitable, leading to the loss of navigator signals and disrupting the continuity of formation control. Furthermore, ships are also affected by uncertainties in model parameters and unknown environmental disturbances during actual navigation.

[0003] In the process of performing ship formation control tasks, the communication system is One of the core components of swarm collaborative control is the intermittent loss of signals, which can lead to a decline or even failure of swarm control performance, causing ships to deviate from the predetermined swarm route, resulting in collisions or swarm breakup. On the one hand, while existing control methods can mitigate the impact of communication interruptions to some extent, traditional strategies typically focus on simple extrapolation or static compensation when ships encounter intermittent, prolonged, and non-periodic communication interruptions caused by complex waters such as islands and reefs. On the other hand, current research lacks control strategies that can balance environmental disturbance suppression and swarm accuracy in communication-constrained environments; achieving this goal is crucial for ensuring the continuity of ship swarm missions.

[0004] Existing ship formation control algorithms have the following two shortcomings:

[0005] 1) Existing communication compensation methods lack real-time perception and recognition of communication status. If such algorithms are applied, they will lead to inaccurate state prediction and reduced formation flexibility in the face of complex intermittent communication situations, making it difficult to promote and apply the algorithms in actual complex marine engineering projects.

[0006] 2) Considering The formation task requires robustness to the operating environment. Existing algorithms cannot effectively compensate for model uncertainties and unknown environmental disturbances while handling state loss caused by intermittent communication. Summary of the Invention

[0007] This invention provides an autonomous flexible train formation control method for surface vessels based on reinforcement learning to overcome the above-mentioned technical problems.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] A reinforcement learning-based autonomous flexible train formation control method for surface vessels includes the following steps:

[0010] S1: Acquire autonomous vessels for establishing ship train formations The nonlinear mathematical model; the ship train formation includes a pilot ship and several follower ships arranged according to a pre-set train topology, and the pilot ship is communicatively connected to each follower ship.

[0011] S2: Construct a time-varying adjustment mechanism for continuously acquiring the pilot ship's status reference signal in an intermittent communication environment, and acquire the virtual ship reference signal based on the time-varying adjustment mechanism;

[0012] S3: Obtain error data of the following ship based on virtual ship reference signals and nonlinear mathematical models; and the error data includes heading error and position error;

[0013] Based on the error data, a virtual control law for the ship's forward direction and bow roll direction is constructed, and the virtual control law for the ship's forward direction and bow roll direction is filtered to obtain the filtered signal;

[0014] S4: According to the definition of the filtered signal The dynamic error is determined by taking the derivative of the dynamic error, and then a neural network is used to approximate the dynamic error derivative to obtain the optimized dynamic error derivative.

[0015] S5: By introducing the Actor-critic reinforcement learning algorithm, the weight update laws of the actor neural network and the critic neural network are constructed based on the error data;

[0016] S6: Based on the weight update law, a formation controller and an adaptive law are designed according to the optimized dynamic error derivative and dynamic error. The control of the autonomous surface vessel elastic train formation is realized according to the formation controller and the adaptive law.

[0017] Furthermore, the formula for obtaining the nonlinear mathematical model described in S1 is as follows:

[0018]

[0019] In the formula: express Position coordinates; express The bow angle and ; Represents the state vector; express The forward speed; express lateral drift speed; express The bow roll rate; They represent Hydrodynamic added mass in the forward, lateral, and yaw degrees of freedom; They represent Nonlinear terms in forward, lateral, and yaw degrees of freedom; Indicates the effect on External environmental disturbances and torques on the forward, drift, and yaw degrees of freedom; This represents the actual control inputs of the ship's propulsion system; express The first derivative; Describes the velocity vector and .

[0020] Furthermore, step S2 specifically includes the following steps:

[0021] S21: Construct a time-varying adjustment mechanism for continuously acquiring the pilot ship's state reference signal in an intermittent communication environment, wherein the expression of the time-varying adjustment mechanism is:

[0022]

[0023] In the formula: , Indicates adjustable design parameters; Indicates the first i Follower ship to pilot ship The time derivative of the state estimate; Indicates based on estimated heading angle The rotation matrix; Indicates the first i The estimated velocity vector of the lead ship for each following ship; Indicates the first i The estimated state of the navigator as observed by the following ships; Indicates the first j The following ships' state estimates of the lead ship and ; Indicates the first i The estimated position coordinates of the lead ship from the following ships; Indicates the first i The estimated heading angle of the lead ship from the following ships; Indicates the first i The following ship and the first jAdjacency matrix elements of the communication topology diagram of the following ships; This indicates the true state of the pilot vessel, namely its actual position coordinates and heading angle; Indicates the number of ships following;

[0024] S22: Based on the navigator's state estimate observed using a time-varying adjustment mechanism, obtain a virtual ship reference signal. The formula for obtaining the virtual ship reference signal is as follows:

[0025]

[0026] In the formula: Indicates the first i The position coordinates of the virtual ship corresponding to each following ship; Indicates the first i Reference value of the heading angle of the virtual ship corresponding to each following ship; This indicates the pilot vessel's actual heading angle; i Indicates the number of the accompanying ship; Indicates adjacent ships in a train formation The expected distance between them.

[0027] Furthermore, step S3 specifically includes the following steps:

[0028] S31: Obtain error data of the following ship based on virtual ship reference signals and a nonlinear mathematical model; and the error data includes heading error and position error, the expression of which is:

[0029]

[0030] In the formula: express The horizontal axis direction error, express The error in the ordinate direction; express The heading angle error; Represents the rotation matrix related to the heading angle;

[0031] S32: Stabilization error and Based on the error data, the virtual control laws for the ship's forward direction and bow direction are constructed as follows:

[0032]

[0033] In the formula: An intermediate parameter representing the error; A virtual control law representing the ship's forward direction and bow roll direction; Indicates a positive design parameter; Indicates the first iThe relative bearing between the follower and the corresponding virtual ship; express The first derivative;

[0034] S33: The virtual control laws for the ship's forward direction and bow roll direction are filtered to obtain a filtered signal, and the formula for obtaining the filtered signal is:

[0035]

[0036] In the formula: This represents a time constant that is greater than zero. The signal after filtering indicates the direction of travel or pitching. This indicates the signal after filtering in the forward direction; This indicates the signal after filtering in the bow-roll direction; This represents the initial value of the filtered signal on the forward or yaw degrees of freedom. This represents the initial value of the virtual control law on the forward or yaw degrees of freedom; express The first derivative.

[0037] Furthermore, step S4 specifically includes the following steps:

[0038] S41: According to the definition of the filtered signal The dynamic error is:

[0039]

[0040] In the formula: Indicates the forward speed error; Indicates the error in bow roll rate;

[0041] S42: Taking the derivative of the dynamic error, we obtain the derivative of the dynamic error as follows:

[0042]

[0043]

[0044] In the formula: express The first derivative; Indicates intermediate parameters; express The first derivative;

[0045] S43: The optimized dynamic error derivative is obtained by approximating the dynamic error derivative using a neural network:

[0046]

[0047] In the formula: , This represents the transpose of the neural network weight matrix; Represents the Gaussian function; This represents the approximation error.

[0048] Furthermore, S5 specifically includes the following steps:

[0049] S51: Define the long-run cost function based on error data for:

[0050]

[0051] In the formula: Indicates an adjustable discount factor; Indicates about time The integral variable; Indicates the first i The instantaneous cost function of each following ship; Indicates a positive design constant; Indicates intermediate parameters;

[0052] S52: By introducing the Actor-critic reinforcement learning algorithm, the weight update law of the actor neural network is obtained based on the long-term cost function. Weight update law of the critic neural network for:

[0053]

[0054] In the formula: This represents an estimate of the long-run cost function; The basis function vector of the neural network representing the bow roll degree of freedom; This represents the estimated values ​​of the neural network weights; The adaptive learning rate represents the forward degrees of freedom; The adaptive learning rate represents the bow roll degree of freedom; The adaptive error cost function represents the forward degrees of freedom; The adaptive error cost function represents the bow roll degree of freedom; The basis function vectors of a neural network representing forward degrees of freedom; Parameters used to compensate for the uncertainty of ship inertia; The basis function vectors of the neural network representing the bow roll degree of freedom.

[0055] Furthermore, the design formula for the formation controller and adaptive law described in S6 is as follows:

[0056]

[0057] In the formula: Indicates a positive design parameter; express initial value and ; This represents the estimated weights of the actuator's neural network. Indicates dynamic error; This represents the actual control inputs of the ship's propulsion system; express The first derivative is Adaptive update law for forward and yaw degrees of freedom; express Adaptive parameter estimates for forward and yaw degrees of freedom.

[0058] Beneficial Effects: This invention provides a reinforcement learning-based method for the flexible formation control of autonomous surface vessels (ASVs). By introducing a time-varying adjustment mechanism, it can more realistically reflect the intermittent communication characteristics between ASVs in complex waters around islands and reefs, and achieve autonomous estimation and dynamic compensation of the navigator's state during communication interruptions, ensuring the continuity of formation control. Through an adaptive control approach based on Actor-critic reinforcement learning, that is, using a critic neural network to evaluate the system cost function online and synchronously driving the actor neural network to optimize the control strategy to construct the formation controller and adaptive law, it effectively realizes real-time adaptive compensation for model uncertainties and unknown environmental disturbances during the flexible formation of autonomous surface vessels, significantly improving the robustness of the formation system in intermittent communication environments. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart of the autonomous flexible train formation control method for surface vessels based on reinforcement learning according to the present invention.

[0061] Figure 2 This is a block diagram of the flexible train formation control structure based on RDLP and reinforcement learning in this embodiment;

[0062] Figure 3 This is a schematic diagram of the ship train formation guidance frame in this embodiment;

[0063] Figure 4 This is a trajectory comparison curve diagram in this embodiment;

[0064] Figure 5 This is a comparison chart of error variation curves in this embodiment;

[0065] Figure 6 This is a comparison chart of control input change curves in this embodiment;

[0066] Figure 7 This is a simulation diagram of the ship train formation trajectory in this embodiment;

[0067] Figure 8 For this embodiment Tracking error curve;

[0068] Figure 9 For this embodiment Input change curve;

[0069] Figure 10 This is a graph showing the observation error in this embodiment. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] This embodiment provides an autonomous surface vessel flexible train formation control method based on reinforcement learning, such as... Figures 1 to 2 As shown, the steps include:

[0072] S1: Acquire autonomous vessels for establishing ship train formations The nonlinear mathematical model; the ship train formation includes a pilot ship and several follower ships arranged according to a pre-set train topology, and the pilot ship is communicatively connected to each follower ship.

[0073] Specifically, the formula for obtaining the nonlinear mathematical model is as follows:

[0074] ,

[0075] In the formula: express Position coordinates; express The bow angle and ; Represents the state vector; express The forward speed; express lateral drift speed; express The bow roll rate; They represent Hydrodynamic added mass in the forward, lateral, and yaw degrees of freedom; They represent Nonlinear terms in forward, lateral, and yaw degrees of freedom; Indicates the effect on External environmental disturbances and torques on the forward, drift, and yaw degrees of freedom; This represents the actual control inputs of the ship's propulsion system; express The first derivative; Describes the velocity vector and ;

[0076] S2: Construct a time-varying adjustment mechanism for continuously acquiring the pilot ship's state reference signal in an intermittent communication environment, and acquire the virtual ship reference signal based on the time-varying adjustment mechanism. Specific steps include:

[0077] S21: Construct a time-varying adjustment mechanism for continuously acquiring the pilot ship's state reference signal in an intermittent communication environment, wherein the expression of the time-varying adjustment mechanism is:

[0078] ,

[0079] In the formula: , Indicates adjustable design parameters; Indicates the first i Follower ship to pilot ship The time derivative of the state estimate; Indicates based on estimated heading angle The rotation matrix; Indicates the first i The estimated velocity vector of the lead ship for each following ship; Indicates the first i The estimated state of the navigator as observed by the following ships; Indicates the first j The following ships' state estimates of the lead ship and ; Indicates the first i The estimated position coordinates of the lead ship from the following ships; Indicates the first i The estimated heading angle of the lead ship from the following ships; Indicates the first i The following ship and the first jAdjacency matrix elements of the communication topology diagram of the following ships; This indicates the true state of the pilot vessel, namely its actual position coordinates and heading angle; Indicates the number of accompanying ships; in this embodiment The expected formation trajectory can be generated by the flexible train formation mechanism, while RDLP, as the core estimation model, is used to provide continuous navigator state reference in intermittent communication environments to solve the practical problem that follower ships cannot obtain the navigator's signal when communication is interrupted.

[0080] S22: Based on the navigator's state estimate observed using a time-varying adjustment mechanism, a virtual ship reference signal is obtained. For multi-follower systems, this is typically used when navigating narrow channels in island and reef waters. Figure 3 The illustrated ship convoy pattern shows each follower sequentially following the navigator through a narrow channel area. Subsequently, a reference signal for the virtual ship is calculated based on geometric spatial relationships. The formula for obtaining the virtual ship reference signal is as follows:

[0081] ,

[0082] In the formula: Indicates the first i The position coordinates of the virtual ship corresponding to each following ship; Indicates the first i Reference value of the heading angle of the virtual ship corresponding to each following ship; This indicates the pilot vessel's actual heading angle; i Indicates the number of the accompanying ship; Indicates adjacent ships in a train formation The expected distance between them.

[0083] In this embodiment, in order to ensure To accurately maintain train formation configuration under intermittent communication conditions and improve the robustness of the control system to environmental disturbances, a flexible train formation control method based on RDLP and reinforcement learning was designed. This method mainly consists of two steps: the first step is to design a virtual controller; the second step is to design a reinforcement learning controller based on an Actor-critic neural network, specifically including the following steps:

[0084] S3: Obtain error data of the following ship based on virtual ship reference signals and nonlinear mathematical models; and the error data includes heading error and position error; construct virtual control laws for the ship's forward direction and bow roll direction based on the error data, and filter the virtual control laws for the ship's forward direction and bow roll direction to obtain the filtered signal;

[0085] The specific steps include:

[0086] S31: Obtain error data of the following ship based on virtual ship reference signals and a nonlinear mathematical model; and the error data includes heading error and position error, the expression of which is:

[0087] ,

[0088] In the formula: express The horizontal axis direction error, express The error in the ordinate direction; express The heading angle error; Represents the rotation matrix related to the heading angle;

[0089] S32: Stabilization error and The virtual control laws for the ship's forward direction and bow roll direction are constructed as follows:

[0090] ,

[0091] In the formula: An intermediate parameter representing the error; A virtual control law representing the ship's forward direction and bow roll direction; Indicates a positive design parameter; Indicates the first i The relative bearing between the follower and the corresponding virtual ship; express The first derivative;

[0092] S33: To avoid the complexity explosion caused by repeated differentiation of the virtual controller, DSC technology is introduced to filter the virtual control laws of the ship's forward and yaw directions, obtaining a filtered signal. The formula for obtaining the filtered signal is as follows:

[0093] ,

[0094] In the formula: This represents a time constant that is greater than zero. The signal after filtering indicates the direction of travel or pitching. This indicates the signal after filtering in the forward direction; This indicates the signal after filtering in the bow-roll direction; This represents the initial value of the filtered signal on the forward or yaw degrees of freedom. This represents the initial value of the virtual control law on the forward or yaw degrees of freedom; express The first derivative.

[0095] S4: According to the definition of the filtered signal The dynamic error is determined by taking the derivative of the dynamic error, and then a neural network is used to approximate the dynamic error derivative to obtain the optimized dynamic error derivative.

[0096] The specific steps include:

[0097] S41: According to the definition of the filtered signal The dynamic error is:

[0098] ,

[0099] In the formula: Indicates the forward speed error; Indicates the error in bow roll rate;

[0100] S42: Taking the derivative of the dynamic error, we obtain the derivative of the dynamic error as follows:

[0101] ,

[0102] ,

[0103] In the formula: express The first derivative; Indicates intermediate parameters; express The first derivative;

[0104] S43: The optimized dynamic error derivative is obtained by approximating the dynamic error derivative using a neural network:

[0105] The neural network approximation formula is: ,

[0106] ,

[0107] In the formula: , This represents the transpose of the neural network weight matrix; Represents the Gaussian function; This represents the approximation error.

[0108] S5: In order to achieve real-time adaptive compensation for model uncertainty and unknown environmental disturbances, this embodiment introduces Actor-critic reinforcement learning technology. The performance index is evaluated by the critic network and the actor network is driven to optimize the control strategy online. That is, by introducing the Actor-critic reinforcement learning algorithm, the weight update law of the actor neural network and the critic neural network is constructed based on the error data.

[0109] The specific steps include:

[0110] S51: Define the long-run cost function based on error data for:

[0111] ,

[0112] In the formula: Indicates an adjustable discount factor; Indicates about time The integral variable; Indicates the first i The instantaneous cost function of each following ship; Indicates a positive design constant; Indicates intermediate parameters;

[0113] S52: By introducing the Actor-critic reinforcement learning algorithm, the weight update law of the actor neural network is obtained based on the long-term cost function. Weight update law of the critic neural network for:

[0114] ,

[0115] In the formula: This represents an estimate of the long-run cost function; The basis function vector of the neural network representing the bow roll degree of freedom; This represents the estimated values ​​of the neural network weights; The adaptive learning rate represents the forward degrees of freedom; The adaptive learning rate represents the bow roll degree of freedom; The adaptive error cost function represents the forward degrees of freedom; The adaptive error cost function represents the bow roll degree of freedom; The basis function vectors of a neural network representing forward degrees of freedom; Parameters used to compensate for the uncertainty of ship inertia; The basis function vector of the neural network representing the bow roll degree of freedom;

[0116] S6: In this embodiment, considering the intermittent communication environment in island and reef waters, based on the weight update law, a formation controller and adaptive law are designed according to the derivative of the optimized dynamic error and the dynamic error. The control of the autonomous surface vessel flexible train formation is achieved based on the formation controller and adaptive law.

[0117] Specifically, the design formula for the formation controller and the adaptive law is as follows:

[0118] ,

[0119] In the formula: Indicates a positive design parameter; express initial value and ; This represents the estimated weights of the actuator's neural network. Indicates dynamic error; This represents the actual control inputs of the ship's propulsion system; express The first derivative is Adaptive update law for forward and yaw degrees of freedom; express Adaptive parameter estimates for forward and yaw degrees of freedom.

[0120] In this embodiment, an RDLP predictor with a time prediction mechanism is introduced to dynamically estimate and compensate for lost leader signals. This proactively maintains formation coordination logic during communication interruptions, reducing the formation's stability from excessive reliance on a continuous and stable communication link and significantly improving the system's communication resilience. Subsequently, using Actor-critic reinforcement learning, the critic network evaluates performance metrics and drives the actor network to optimize control strategies online, enabling adaptive compensation for model parameter uncertainties and external environmental disturbances in the control system. The method described in this embodiment not only ensures formation tracking accuracy but also significantly enhances the system's operational reliability in intermittent communication environments.

[0121] To verify the effectiveness of the method described in this embodiment... When communication is interrupted, it can generate continuous control signals and transmit pitch input commands to the actuator through the control system to enable... The goal is to achieve the desired train formation state while ensuring good adaptive performance of the entire control system under model uncertainties and unknown environmental disturbances. This embodiment utilizes computer simulation experiments to conduct numerical comparison simulations in a simulated island and reef marine environment. The effectiveness and superiority of the method described in this embodiment are verified by comparing it with Algorithm A (i.e., the traditional robust adaptive formation control algorithm). The main differences between the method described in this embodiment and Algorithm A are shown in Table 1.

[0122] Table 1. Similarities and differences between the method described in this embodiment and Algorithm A

[0123]

[0124] The method described in this embodiment and Algorithm A were compared and simulated on an industrial computer. Figures 4 to 6 The main comparison results are presented. Figure 4 express Comparison of trajectory results in path tracking control tasks under two different algorithms. Figure 4 Although both algorithms achieved satisfactory path tracking, the algorithm proposed in this invention utilizes an RDLP predictor and employs actor-critic reinforcement learning to achieve online evaluation of the system performance cost function and optimize the control strategy. In contrast, the trajectory of Algorithm A, which uses a traditional guidance mechanism, exhibits a larger deviation. Therefore, the method described in this embodiment is superior in ensuring the stability of train formations in communication-constrained environments. Figure 5 The error curves of the two algorithms are described. As can be seen from the figure, the method described in this embodiment has a relatively smaller and more stable error, and has a greater advantage in terms of path tracking accuracy. Figure 6 The control input variation curves of the two algorithms are shown. As can be seen from the graph, compared to the interference compensation method using an interference observer in Algorithm A, the method described in this embodiment, through the online learning capability of reinforcement learning, exhibits a smoother and more stable control input. In summary, the method described in this embodiment demonstrates excellent robustness in handling intermittent communication problems in complex island and reef waters.

[0125] also, Figures 7 to 10 Numerical results of the method described in this embodiment in a marine environment are presented. Figure 7 The train convoy trajectory using the algorithm of this invention in narrow island and reef waters is described. It can be seen that... Starting from a random initial formation, the system, under the guidance of the navigation law, sequentially follows the pilot ship to form a standard train formation and traverses the island and reef group. Figure 8 The tracking error shown indicates that the method described in this embodiment maintains high tracking accuracy throughout the entire navigation process, with both position and heading errors kept within a small range. Figure 9 The control input amplitude is within a reasonable range and satisfies the actuator constraints. Figure 10 The RDLP observation error shows a convergence trend, with two error peaks corresponding to steering maneuvers during navigation, proving the observer's effective tracking capability of the pilot ship's state.

[0126] Compared with existing technologies, the method described in this embodiment, based on the design and numerical simulation of an elastic train formation controller using RDLP and reinforcement learning, demonstrates superior performance. The beneficial effects in the field of motion control are as follows:

[0127] 1. Compared with existing formation control algorithms that handle intermittent communication, the collaborative method of RDLP and ship train formation mechanism proposed in this embodiment shows significant advantages in several aspects. First, by introducing a time-varying adjustment mechanism into the communication topology, the pilot ship's state estimation process possesses the core characteristics of communication state awareness and dynamic compensation, enabling it to autonomously recover guidance signals during communication interruptions and proactively ensuring the continuity of the formation. Second, this method considers the constraints of smooth waypoint switching and passage through island and reef waters, and the guidance mechanism designed on this basis is more in line with... Engineering practice requirements for operating in complex waters.

[0128] 2. With the increasing demands for precision in intelligent unmanned swarm operations during marine resource development, robust intelligent control technology is a control objective pursued globally. The adaptive control scheme based on Actor-critic reinforcement learning proposed in this embodiment precisely addresses this issue. In formation control, performance degradation is caused by model parameter uncertainty and unknown environmental disturbances. Considering the control energy efficiency requirements under complex sea states, the method described in this embodiment uses a critic network to evaluate the cost function online and drive the actor network to optimize the control strategy in real time, which is in line with the requirements of formation control. The formation mission is developing towards more refined and intelligent control.

[0129] 3) Numerical simulations verified that the method described in this embodiment is effective. In train formation control tasks, it achieves excellent control performance, not only solving the problems of formation breakup and collision avoidance failure caused by guidance signal interruption in intermittent communication environments, but also significantly improving the system's adaptability and reliability in dynamic and complex environments by introducing RDLP elastic prediction and reinforcement learning optimization mechanisms. The field of collaborative control offers more efficient, secure, and stable solutions. The method described in this embodiment has broad application prospects in modern intelligent shipping, marine scientific research, and operation and maintenance of large-scale deep-sea engineering projects, and can accelerate... To transform towards greater resilience, intelligence, and security.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for autonomous water surface vessel elastic platoon control based on reinforcement learning, characterized in that, Specifically comprising steps: S1: obtaining an autonomous ship for establishing a ship train formation ; the ship train formation comprises a leading ship and a plurality of following ships arranged according to a preset train topology structure, and the leading ship is in communication connection with each following ship respectively; S2: Constructing a time-varying adjustment mechanism for continuously obtaining a reference signal of a virtual ship in an intermittent communication environment, and obtaining the reference signal of the virtual ship according to the time-varying adjustment mechanism; S3: Obtaining error data of a following ship based on the reference signal of the virtual ship and a nonlinear mathematical model, and the error data includes a heading error and a position error; Constructing a virtual control law of a ship heading direction and a yaw direction according to the error data, and performing filtering processing on the virtual control law of the ship heading direction and the yaw direction to obtain a filtered signal; S4: defining a dynamics error from the filtered signal derivative of the dynamics error, and using a neural network to approximate the dynamics error derivative to obtain an optimized dynamics error derivative; S5: Introducing an Actor-critic reinforcement learning algorithm, and constructing a weight update law of an actor neural network and a critic neural network according to the error data; S6: Designing a formation controller and an adaptive law according to an optimized derivative of a kinetic error based on the weight update law, and realizing control of an autonomous water surface ship elastic train formation according to the formation controller and the adaptive law.

2. The autonomous surface vessel elastic platoon control method based on reinforcement learning according to claim 1, characterized in that, The formula for obtaining the nonlinear mathematical model in S1 is: In the formula: express Position coordinates; express The bow angle and ; Represents the state vector; express The forward speed; express lateral drift speed; express The bow roll rate; They represent Hydrodynamic added mass in the forward, lateral, and yaw degrees of freedom; They represent Nonlinear terms in forward, lateral, and yaw degrees of freedom; Indicates the effect on External environmental disturbances and torques on the forward, drift, and yaw degrees of freedom; This represents the actual control inputs of the ship's propulsion system; express The first derivative; Describes the velocity vector and .

3. The method of claim 2, wherein, The S2 specifically comprises steps: S21: Constructing a time-varying adjustment mechanism for continuously obtaining a reference signal of a virtual ship in an intermittent communication environment, and the expression of the time-varying adjustment mechanism is: In the formula: , Indicates adjustable design parameters; Indicates the first i Follower ship to pilot ship The time derivative of the state estimate; Indicates based on estimated heading angle The rotation matrix; Indicates the first i The estimated velocity vector of the lead ship for each following ship; Indicates the first i The estimated state of the navigator as observed by the following ships; Indicates the first j The following ships' state estimates of the lead ship and ; Indicates the first i The estimated position coordinates of the lead ship from the following ships; Indicates the first i The estimated heading angle of the lead ship from the following ships; Indicates the first i The following ship and the first j Adjacency matrix elements of the communication topology diagram of the following ships; This indicates the true state of the pilot vessel, namely its actual position coordinates and heading angle; Indicates the number of ships following; S22: Obtaining the reference signal of the virtual ship according to a state estimation value of a leader observed based on the time-varying adjustment mechanism, and the formula for obtaining the reference signal of the virtual ship is: In the formula: represents the position coordinates of the virtual ship corresponding to the i first following ship; represents the heading angle reference value of the virtual ship corresponding to the i first following ship; represents the real heading angle of the lead ship; i indicates the number of the following ship; indicates the desired distance between adjacent ships in the train formation.

4. The autonomous surface vessel elastic platoon control method based on reinforcement learning according to claim 3, characterized in that, The S3 specifically comprises steps: S31: Obtaining error data of a following ship based on the reference signal of the virtual ship and a nonlinear mathematical model, and the error data includes a heading error and a position error, and the expression is: In the formula: express The horizontal axis direction error, express The ordinate direction error; express The heading angle error; Represents the rotation matrix related to the heading angle; S32: stabilizing error and According to the error data, a virtual control law of the ship's forward direction and yaw direction is constructed as follows: wherein: an intermediate variable representing an error; a virtual control law representing the heading of the vessel and the yaw direction; a positive design parameter; a first derivative of the i a relative bearing between the i-th follower and the corresponding virtual vessel; a first derivative of the a first derivative of the S33: Performing filtering processing on a virtual control law of a ship heading direction and a yaw direction to obtain a filtered signal, and the formula for obtaining the filtered signal is: wherein: represents a time constant greater than zero; represents a filtered signal in the forward direction or yaw direction, represents a filtered signal in the forward direction; represents a filtered signal in the yaw direction; represents an initial value of a filtered signal in the forward degree of freedom or yaw degree of freedom; represents an initial value of a virtual control law in the forward degree of freedom or yaw degree of freedom; represents a first derivative of 5. The autonomous surface vessel elastic platoon control method based on reinforcement learning according to claim 4, characterized in that, The S4 specifically comprises steps: S41: defining a filter signal according to the dynamics error of the dynamics error is: In the formulae: denotes the forward speed error; denotes the yaw angle velocity error; S42: Deriving a kinetic error to obtain a kinetic error derivative as: wherein: denotes the first derivative of denotes an intermediate variable denotes the first derivative of denotes the first derivative of​ S43: Approximating the kinetic error derivative by using a neural network to obtain an optimized kinetic error derivative as: wherein: , denotes the transpose of a neural network weight matrix; denotes a Gaussian function; denotes an approximation error.

6. The autonomous surface vessel elastic platoon control method based on reinforcement learning according to claim 5, characterized in that, The S5 specifically comprises steps: S51: Define long-term cost function from error data f: wherein: denotes an adjustable discount factor; denotes the integral variable with respect to time ; denotes the current price function of the th i following ship; denotes a positive design constant; denotes an intermediate variable; S52: Obtain the weight update law of the actor neural network according to the long-term cost function by introducing the Actor-critic reinforcement learning algorithm and the weight update law of the critic neural network is: wherein: represents an estimate of the long-term cost function; represents a basis function vector of the neural network for the yaw degree of freedom; represents an estimate of the neural network weights; represents an adaptive learning rate for the forward degree of freedom; represents an adaptive learning rate for the yaw degree of freedom; represents an adaptive error cost function for the forward degree of freedom; represents an adaptive error cost function for the yaw degree of freedom; represents a basis function vector of the neural network for the forward degree of freedom; a parameter for compensating for ship inertia uncertainty; represents a basis function vector of the neural network for the yaw degree of freedom.

7. The autonomous surface vessel elastic platoon control method based on reinforcement learning according to claim 6, characterized in that, The design formula of the formation controller and the adaptive law in S6 is: wherein: denotes a positive design parameter; denotes an initial value of ; denotes an estimate of the actuator neural network weights; denotes a dynamics error; denotes an actual control input of the ship propulsion system; denotes a first derivative of an adaptive update law in the forward and yaw degrees of freedom; denotes an adaptive parameter estimate in the forward and yaw degrees of freedom.

Images