A multi-unmanned ship formation capture control method based on reinforcement learning

Abstract

The present invention discloses a multi-unmanned ship formation capture control method based on reinforcement learning, including constructing a multi-unmanned ship motion system affected by environmental interference, constructing a capture formation error variable; based on the capture formation error variable, using the backstepping method to construct a virtual controller of the unmanned ship; based on the reinforcement learning method combined with a neural network, constructing a leading ship formation capture controller; based on the adaptive backstepping method combined with Lyapunov theory and a neural network, constructing a follower ship formation capture controller; according to the leading ship formation capture controller and the follower ship formation capture controller, the formation capture control of the target ship is realized. The method solves the problem that the multi-unmanned ship formation is disturbed by complex marine environments such as wind, waves, and currents during the efficient execution of the capture task by the multi-unmanned ship formation, resulting in insufficient control performance of the multi-unmanned ship, and the inability to fully reflect the collaborative relationship between the chasing unmanned ships, resulting in difficulty in controlling the multi-unmanned ship formation, and difficulty in achieving control at the lowest possible cost.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-unmanned ship formation capture control method based on reinforcement learning. Background Art

[0002] In recent years, with the development of new technologies such as the Internet of Things, big data, and artificial intelligence, unmanned ship (USV) technology has been rapidly developed and commercialized worldwide. The Chinese government attaches great importance to the development of artificial intelligence industries such as unmanned ships. Its application areas include civil fields such as marine resource exploration, seabed topography mapping, offshore safety inspections, and marine ranch operation and maintenance, as well as military fields such as anti-mine warfare, anti-submarine warfare, maritime security, and surface warfare. However, the capabilities of a single unmanned ship are limited and it is difficult to meet the needs of complex tasks. Compared with a single unmanned ship, a multi-unmanned ship formation has stronger robustness, redundancy, and adaptability. Multi-unmanned ship formations have been widely used in marine environment detection, target collaborative capture, cluster confrontation, emergency rescue, and other aspects. In maritime security and military applications, the coordinated capture of unmanned ship formations plays an important role in ensuring national security. Therefore, the capture of multi-unmanned ship formations is an important research direction.

[0003] At present, there are still severe challenges in the process of multiple unmanned ship formations efficiently executing the capture mission: for example, when multiple unmanned ships are performing the capture mission, they ignore the interference of complex marine environments such as wind, waves, and currents, resulting in insufficient control performance of multiple unmanned ships. Secondly, when multiple unmanned ships rely on division of labor and cooperation to perform the capture mission, due to the large number of unmanned ships and the lack of effective communication between them, the collaborative relationship between the chasing unmanned ships cannot be fully reflected, resulting in difficulty in controlling the multiple unmanned ship formations. The final capture mission of the multiple unmanned ship formations requires the control target to be achieved at the lowest possible cost, which brings challenges to the optimization of the tracking trajectory of multiple unmanned ships. Summary of the invention

[0004] The present invention provides a multi-unmanned ship formation capture control method based on reinforcement learning to overcome the above technical problems.

[0005] In order to achieve the above object, the technical solution of the present invention is:

[0006] A multi-unmanned ship formation capture control method based on reinforcement learning, specifically comprising the following steps:

[0007] S1: A multi-unmanned ship motion system is constructed under the earth coordinate system and the attached coordinate system, which is affected by environmental interference, and the control tasks of each unmanned ship in the formation are divided by the leader-following method;

[0008] S2: Construct the capture formation error variable according to the multi-unmanned ship motion system and the control tasks of each unmanned ship in the formation;

[0009] S3: Based on the error variables of the encirclement formation, the backstepping method is used to construct the virtual controller of the unmanned ship to obtain the optimal derivative of the first Lyapunov function of the unmanned ship;

[0010] S4: Based on the reinforcement learning method, the optimal derivative of the first Lyapunov function of the unmanned ship is combined with the neural network to construct the leader ship formation capture controller;

[0011] The control law used by the leader ship formation capture controller includes the leader ship optimal control law, the ship feedforward adaptive law and the leader ship weight update law;

[0012] S5: Based on the adaptive backstepping method, according to the optimal derivative of the first Lyapunov function of the unmanned ship, and combining the Lyapunov theory with the neural network, a follower ship formation capture controller is constructed;

[0013] And the following ship formation capture controller includes an adaptive controller and an adaptive control law;

[0014] S6: According to the leading ship formation capture controller and the following ship formation capture controller, the formation capture control of the target ship is realized.

[0015] Beneficial effects: The present invention provides a multi-unmanned ship formation capture control method based on reinforcement learning. Through the reinforcement learning algorithm, it ensures that the multi-unmanned ship formation can efficiently track and capture the target ship at the lowest possible cost, providing a high-quality solution for the multi-unmanned ship formation to perform collaborative capture tasks under the influence of environmental interference. The leader-following method is used to divide the multi-unmanned ship formation into a leader ship and a follower ship. The leader ship is responsible for tracking the target ship with the optimal trajectory, and the follower ship is responsible for ensuring the capture formation. The optimal controller and weight update law of the leader ship are designed based on the reinforcement learning method, and the adaptive controller and adaptive law of the follower ship are designed based on the adaptive backstepping method. It ensures that the multi-unmanned ship formation can achieve unmanned ship formation control at the lowest possible control cost when performing the capture task, fully reflecting the collaborative cooperation relationship between the capture unmanned ships, and greatly improving the control efficiency and accuracy of the unmanned ship formation capture. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1It is a flow chart of the multi-unmanned ship formation capture control method based on reinforcement learning of the present invention;

[0018] Figure 2 A communication topology diagram of a multi-unmanned ship formation provided in this embodiment;

[0019] Figure 3 This is a simulation diagram of the capture effect of the multi-unmanned ship formation provided in this embodiment at 0 seconds;

[0020] Figure 4 This is a simulation diagram of the capture effect of a multi-unmanned ship formation provided in this embodiment at 50 seconds;

[0021] Figure 5 This is a simulation diagram of the capture effect of a multi-unmanned ship formation at 100 seconds provided in this embodiment;

[0022] Figure 6 This is a simulation diagram of the capture effect of a multi-unmanned ship formation at 200 seconds provided in this embodiment;

[0023] Figure 7 This is a simulation diagram of the capture error result of the leader unmanned ship provided in this embodiment;

[0024] Figure 8 This is a simulation diagram of the capture error result of the first follower unmanned boat provided in this embodiment;

[0025] Fig. 9 This is a simulation diagram of the capture error result of the second follower unmanned boat provided in this embodiment;

[0026] Fig.10 This is a simulation diagram of the capture error result of the third follower unmanned ship provided in this embodiment;

[0027] Fig.11 This is a control input simulation diagram of the leader unmanned ship provided in this embodiment;

[0028] Fig.12 This is a control input simulation diagram of the first follower unmanned ship provided in this embodiment;

[0029] Fig.13 This is a control input simulation diagram of the second follower unmanned ship provided in this embodiment;

[0030] Fig.14 This is a control input simulation diagram of the third follower unmanned ship provided in this embodiment;

[0031] Fig.15 This is the neural network weight convergence diagram provided in this embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] This embodiment provides a multi-unmanned ship formation capture control method based on reinforcement learning, such as Figure 1 As shown, the specific steps include:

[0034] S1: A multi-unmanned ship motion system is constructed under the earth coordinate system and the attached coordinate system, which is affected by environmental interference, and the control tasks of each unmanned ship in the formation are divided by the leader-following method;

[0035] The specific steps include:

[0036] S11: Constructing a multi-unmanned ship motion system affected by environmental interference in the earth coordinate system and the attached coordinate system, and the first The ship motion model expression of an unmanned ship is

[0037] ,

[0038] ,

[0039] ,

[0040] ,

[0041] Where: represents the position vector in the geodetic coordinate system, and ; Indicates the horizontal and vertical coordinates of the ship's position; Indicates the ship's heading; represents the velocity vector in the body coordinate system, and ; They represent the ship's angular velocity of pitch, angular velocity of roll and angular velocity of bow respectively; represents a reversible rotation matrix that satisfies ; represents the identity matrix; represents the inertia matrix; represents the Coriolis matrix; indicates unmodeled dynamics; represents the nonlinear term affected by environmental disturbances; represents control input; represents an unknown bounded environmental disturbance, and , ; Indicates the total number of ships in the unmanned ship fleet; , , , , Indicates the additional mass of the ship; Represents unknown bounded environmental disturbance The upper bound of

[0042] S12: Due to , and are all nonlinear terms related to the environment and ship characteristics, the ship motion model is nonlinearly rewritten to obtain the final multi-unmanned ship motion system, and the final multi-unmanned ship motion system The ship motion model expression of an unmanned ship is

[0043] ,

[0044] Where: represents the nonlinear dynamic term of the unmanned ship, and ;

[0045] S13: Use the leader-following method to divide the control tasks of each unmanned ship in the formation, specifically:

[0046] The target ship’s trajectory signal is defined as ;

[0047] The relative formation distance between the target ship and the multi-unmanned ship motion system is ;

[0048] Set up multiple unmanned ship formations Follower ships with If there is a leader ship, the task of the multi-unmanned ship formation to surround the target ship is described as , ;

[0049] The leader ship is used to achieve optimized trajectory tracking of the target ship, and the follower ship is used to ensure the coordinated capture formation;

[0050] S2: Construct the capture formation error variable according to the multi-unmanned ship motion system and the control tasks of each unmanned ship in the formation;

[0051] The error variable of the constructed capture formation is

[0052] ,

[0053] Where: A virtual controller representing the unmanned ship to be solved; represents the adjacency matrix and ; Indicates The unmanned ship can receive The decision variable of the information of the unmanned ship, if The unmanned ship can receive If the information of unmanned ships is ,otherwise ; Represents the signal variable between each unmanned ship and the target ship. If If an unmanned ship can receive the target ship's track signal, then ,otherwise ; represents the error variable of the leader ship of the roundup formation; represents the error variable of the follower ship of the encirclement formation; Indicates The relative distance between the unmanned ship and the target ship; Indicates The relative distance between the unmanned ship and the target ship, and ;

[0054] S3: Based on the error variables of the encirclement formation, the backstepping method is used to construct the virtual controller of the unmanned ship to obtain the optimal derivative of the first Lyapunov function of the unmanned ship;

[0055] The specific steps include:

[0056] S31: Based on the ship motion model in step S12, the dynamic formation error of the capture formation error variable is obtained, and its expression is:

[0057] ,

[0058] ,

[0059] Where: represents the in-degree of each unmanned ship in the formation, and ;

[0060] S32: Using the backstepping method to construct the first Lyapunov function of the unmanned ship according to the error variable of the capture formation, and the first Lyapunov function The expression is

[0061] ,

[0062] S33: According to the first Lyapunov function With the dynamic formation error, the derivative of the first Lyapunov function is obtained, and its expression is

[0063] ,

[0064] S34: Construct a virtual controller for the unmanned ship, whose expression is

[0065] ,

[0066] Where: represents a positive constant used to design a virtual controller;

[0067] S35: Bring the constructed virtual controller of the unmanned ship into step S33 to rewrite the derivative of the first Lyapunov function and obtain the optimal derivative of the first Lyapunov function as follows:

[0068] ,

[0069] S4: Based on the reinforcement learning method, the optimal derivative of the first Lyapunov function of the unmanned ship is combined with the neural network to construct the leader ship formation capture controller;

[0070] The control law used by the leader ship formation capture controller includes the leader ship optimal control law, the ship feedforward adaptive law and the leader ship weight update law;

[0071] The specific steps include:

[0072] S41: Based on the reinforcement learning method, the second Lyapunov function is constructed according to the first Lyapunov function through the step-by-step design method of the feedforward and feedback controller. Its expression is:

[0073] ,

[0074] ,

[0075] Where: represents the second Lyapunov function; Indicates the intermediate parameter quantity; Indicates To The error variable of the follower ship in the roundup formation of ships The error matrix of ; Indicates the total number of ships in the multi-unmanned ship formation; Indicates the number of follower ships; represents the parameter matrix; represents the weight error of the neural network, and ; represents the ideal weights of the neural network; represents the estimated value of the neural network weights and represents transpose;

[0076] S42: Obtain the derivative of the second Lyapunov function according to the optimal derivative of the first Lyapunov function of the unmanned ship for

[0077] ,

[0078] , ,

[0079] Where: Indicates that it is used for design is a diagonal matrix composed of positive constants of the virtual controllers of the leader ships, and

[0080] express The upper bound of express The matrix of the first error variables of the leader ships, and ; express The diagonal matrix of the leading ships in degrees, and Indicates that leader ships, each ship and the target ship's signal variables form a diagonal matrix and ; express The diagonal matrix composed of the rotation matrices of the leader ships, and

[0081] ;

[0082] And the derivative of the second Lyapunov function Rewrite as

[0083] ,

[0084] Where: Represented by the feedforward controller and feedback controller The control input consists of and ; represents the unknown bounded environmental disturbance term of the leader ship; express Virtual controller of the leader ship;

[0085] S43: Define intermediate variables ,and ;

[0086] According to the intermediate variable The derivative of the second Lyapunov function is rewritten Transform into

[0087] ,

[0088] Where: Indicates about is a nonlinear function of ;

[0089] S44: Using Neural Networks to Approximate Nonlinear Functions , whose expression is

[0090] ,

[0091] Where: represents the ideal weights of the neural network; Represents the activation function of the neural network; represents the approximation error of the neural network, and express The upper bound of

[0092] Then the derivative of the second Lyapunov function in step S43 is , further rewritten as

[0093] ,

[0094] S45: To ensure the derivative of the second Lyapunov function further rewritten Less than or equal to 0, construct a ship feedforward controller Feed-forward adaptive law for ships , whose expression is

[0095] ,

[0096] ,

[0097] Where: represents the positive definite matrix used to design the feedforward controller, express The upper bound of

[0098] S46: Based on the ship feedforward controller Feed-forward adaptive law for ships The derivative of the second Lyapunov function in step S44 is Rewrite as

[0099] ,

[0100] Where: represents the approximation error of the neural network;

[0101] According to Young's inequality, we can get

[0102] ,

[0103] In this embodiment, according to Lyapunov's stability theory, there is an uncertainty term in the inequality obtained according to Young's inequality. , resulting in the instability of the closed-loop system. Therefore, in this embodiment, step S47 needs to be performed to further design the optimal controller To ensure the stability of the closed-loop system;

[0104] S47: In order to ensure the stability of the closed-loop system, according to Lyapunov stability theory, the uncertainty in S46 is considered , constructing the optimal control law of the leading ship based on the nonlinear affine system of the ship ;

[0105] And the nonlinear affine system expression is

[0106] ,

[0107] Where: , in order to facilitate control design;

[0108] And define the cost function based on the nonlinear affine system , whose expression is

[0109] ,

[0110] Where: represents the integral variable; represents an additional term associated with the uncertain dynamics, and ; represents the basic utility function and ; , represents a positive definite matrix; represents a positive constant matrix;

[0111] S48: According to the cost function Get the optimal cost function for

[0112] ,

[0113] And according to the optimal cost function , solve the constructed Hamilton-Jacobi-Bellman equation to obtain the optimal control law of the leading ship ;

[0114] The Hamilton-Jacobi-Bellman equation is expressed as

[0115] ,

[0116] Where: It means to find partial derivatives; represents the intermediate parameter quantity and , Represents unknown bounded environmental disturbance The upper bound of

[0117] The optimal control law of the leading ship The expression is

[0118] ,

[0119] S49: Approximating the Optimal Cost Function Using Evaluation Neural Networks , whose expression is

[0120] ,

[0121] Where: represents the ideal weights for evaluating neural networks; Represents the activation function of the evaluation neural network; Represents the approximation error of the evaluation neural network; for The abbreviation of

[0122] Get the optimal cost function The partial derivative of

[0123] ,

[0124] Based on the evaluation neural network, according to the optimal cost function The partial derivatives of rewrite the optimal control law of the leading ship, the nonlinear affine system and the Hamilton-Jacobi-Bellman equation as

[0125] ,

[0126] ,

[0127] ,

[0128] Where: , , represents the intermediate parameter quantity, and , , ,

[0129] Define the optimal cost function Estimated value of for

[0130] ,

[0131] Where: Represents an estimate of the ideal weights for evaluating a neural network;

[0132] Get the optimal cost function The partial derivative of the estimate of for

[0133] ,

[0134] And according to the optimal cost function The partial derivative of the estimate of , obtain the estimated value of the optimal control law of the leading ship, the estimated value of the nonlinear radiation system and the estimated value of the Hamilton-Jacobi-Bellman equation, which is expressed as

[0135] ,

[0136] ,

[0137] ,

[0138] Construct the error function of the Hamilton-Jacobi-Bellman equation, which is expressed as

[0139] ,

[0140] ,

[0141] Define reinforcement function: In order to strengthen the learning process, this embodiment will define the following reinforcement function to design an additional term, which is expressed as

[0142] , and there exists a positive definite matrix Make Established;

[0143] Evaluation of the error function corresponding to the error value of the Hamilton-Jacobi-Bellman equation designed based on the reinforcement function. Weight update law of the neural network , which is the leader ship weight update law, and its expression is

[0144] ,

[0145] Where: represents adaptive learning parameters; Indicates adjustable parameters; represents the reinforcement function; express The first derivative of represents the error of the Hamilton-Jacobi-Bellman equation.

[0146] The update law of the weight estimation error of the evaluation network is

[0147] ,

[0148] S5: Based on the adaptive backstepping method, according to the optimal derivative of the first Lyapunov function of the unmanned ship, and combining the Lyapunov theory with the neural network, a follower ship formation capture controller is constructed;

[0149] The following ship formation capture controller includes an adaptive controller and an adaptive control law;

[0150] The specific steps include:

[0151] S51: Based on the adaptive backstepping method, the third Lyapunov function is constructed according to the first Lyapunov function of the unmanned ship , whose expression is

[0152] ,

[0153] Where: express The matrix of the second error variables of the follower ships, and ; represents a positive definite constant matrix; represents the weight error of the adaptive neural network and ; represents the ideal weights of the adaptive neural network; represents an estimate of the ideal weights of the adaptive neural network and ; represents an intermediate variable;

[0154] S52: Obtain the derivative of the third Lyapunov function according to the optimal derivative of the first Lyapunov function of the unmanned ship for

[0155] ,

[0156] ,

[0157] ,

[0158] Where: Indicates that it is used for design is a diagonal matrix composed of positive constants of the virtual controllers of the follower ships, and , , express The upper bound of express The matrix of the first error variables of the follower ships, and ; express The diagonal matrix of follower ships' in-degrees and ; Indicates that A diagonal matrix consisting of the signal variables of each follower ship and the target ship, and ; express The diagonal matrix composed of the rotation matrices of the follower ships, and ;

[0159] And the derivative of the third Lyapunov function Rewrite as

[0160] ,

[0161] Where: represents the control input of the follower ship, ; represents the unknown bounded environmental disturbance term of the follower ship; express The virtual controller of the follower ship, How to obtain The principle is the same;

[0162] S53: Using adaptive neural networks Approximation processing, its expression is

[0163] ,

[0164] Where: represents the ideal weights of the adaptive neural network; Represents the activation function of the adaptive neural network; represents the approximation error of the adaptive neural network, express The upper bound of

[0165] Then the derivative of the third Lyapunov function , further rewritten as

[0166] ,

[0167] Where: represents the unknown bounded environmental disturbance of the follower ship, and ; express The first-order derivative of the virtual controller of the follower ship, and ;

[0168] S54: In order to ensure the derivative of the third Lyapunov function after further rewriting Less than or equal to 0, the adaptive controller and adaptive control law of the follower ship are constructed, and its expression is

[0169] ,

[0170] ,

[0171] Where: represents the control input of the adaptive controller; represents the adaptive control law; represents the positive constant matrix used to design the follower ship adaptive controller.

[0172] This example is for the subsequent stability proof, can further obtain

[0173] ,

[0174] S55: Based on Young’s inequality, we can get

[0175] ,

[0176] S6: According to the leading ship formation capture controller and the following ship formation capture controller, the formation capture control of the target ship is realized.

[0177] This embodiment also includes step S7: for a multi-unmanned ship formation system considering environmental interference, based on Lyapunov theory, it is proved that the designed controller, adaptive law and weight update law can ensure that all signals in the closed-loop system are ultimately uniformly bounded by selecting appropriate parameters:

[0178] S71: First, it is proved that for the leader ship system, the designed optimization controller and weight update law can ensure that the closed-loop system state and weight estimation errors are eventually uniformly bounded;

[0179] Choose the following Lyapunov function

[0180] ,

[0181] By combining the evaluation neural network weight update law in S49 , can get The derivative of

[0182] ,

[0183] From the estimated value of the nonlinear radiation system in S49 After mathematical transformation, we can get , It can be further deduced as

[0184] ,

[0185] Consider the nonlinear radiation system obtained by using the evaluation neural network in S49 Estimation of nonlinear radiation systems , can get , It can be further deduced as

[0186] ,

[0187] S72: Since on a compact set, , , and All are bounded and satisfying , , and , and are all positive numbers. And, there exists a positive definite matrix Make

[0188] Therefore, according to Young's inequality, we can get

[0189] ,

[0190] in, , , They are all normal numbers.

[0191] S73: In order to prove the stability of the entire multi-unmanned ship system, consider the following Lyapunov function

[0192]

[0193] Combining S46, S55, and S72, according to Young's inequality, The derivative of satisfies

[0194] ,

[0195] definition ,

[0196] ,

[0197] ,

[0198] Can further obtain,

[0199] ,

[0200] S74: Therefore, as long as the appropriate parameters are selected to meet , , One of the three conditions, It is thus proved that the designed controller, adaptive law and weight update law can ensure that all signals in the closed-loop system are ultimately uniformly bounded by selecting appropriate parameters.

[0201] In order to verify the feasibility and effectiveness of the multi-unmanned ship capture control method based on reinforcement learning in this embodiment, this embodiment uses MATLAB for computer simulation research. The simulation object is a 1:70 scaled supply ship CyberShip II, and the parameters are set as follows: , , , , , , , , target ship trajectory , , , , , Controller Design Parameters , . Adaptive law design parameters . Adaptive learning parameters , adjustable parameters . System state initial value , , , , .

[0202] The simulation results are shown in the figure: Figure 2 The communication topology diagram of multiple unmanned ship formations shows the information transmission relationship between unmanned ships, where 0 is the target ship, 1 is the leader in the multi-unmanned ship formation, and 2, 3, and 4 are followers in the unmanned ship formation. Figures 3 to 6 The following are the position diagrams of the multi-unmanned ship formation and the target ship at 0 seconds, 50 seconds, 100 seconds, and 200 seconds, respectively, indicating that the multi-unmanned ship formation can stably capture the target ship. Figures 7 to 10The following are the tracking error diagrams of each unmanned ship. It can be seen that the tracking error of each ship can quickly converge to a bounded tight set of zero. Figures 11 to 14 The control input change curve for each unmanned ship is used, and the control signal can converge quickly and smoothly. Fig.15 This is the convergence diagram of the neural network weights. It can be seen that the weights of each neural network can converge quickly.

[0203] The simulation results show that the controller designed by the multi-unmanned ship formation capture control method based on reinforcement learning can efficiently and stably track and capture the target ship, and the tracking error can quickly converge to a bounded tight set of zero. The reinforcement learning algorithm is used to ensure that the multi-unmanned ship formation can efficiently track and capture the target ship at the lowest possible cost, providing a high-quality solution for the multi-unmanned ship formation to perform collaborative capture tasks under the influence of environmental interference.

[0204] Compared with the prior art, the beneficial effects of this embodiment are:

[0205] 1. The reinforcement learning-based multi-unmanned ship formation capture control method provided in this embodiment uses a reinforcement learning algorithm to ensure that the multi-unmanned ship formation can efficiently track and capture the target ship at the lowest possible cost, providing a high-quality solution for the multi-unmanned ship formation to perform collaborative capture tasks under the influence of environmental interference;

[0206] 2. This embodiment uses the leader-following method to divide the multi-unmanned ship formation into a leader ship and a follower ship. The leader ship is responsible for tracking the optimal trajectory of the target ship, and the follower ship is responsible for ensuring the capture formation. The optimal controller and weight update law of the leader ship are designed based on the reinforcement learning method, and the adaptive controller and adaptive law of the follower ship are designed based on the adaptive backstepping method. Finally, the capture control of the target ship by the multi-unmanned ship formation is realized;

[0207] 3. This embodiment conducts simulation experiments based on the MATLAB platform and realizes the optimal trajectory tracking and encirclement of the target ship trajectory by the fleet based on the reinforcement learning method. After considering reinforcement learning, the system controller is more in line with navigation practice, which further verifies the effectiveness and rationality of the reinforcement learning-based multi-unmanned ship formation encirclement control method provided in this embodiment;

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-unmanned ship formation capture control method based on reinforcement learning, characterized in that: The specific steps include: S1: A multi-unmanned ship motion system is constructed under the earth coordinate system and the attached coordinate system, which is affected by environmental interference, and the control tasks of each unmanned ship in the formation are divided by the leader-following method; S2: Construct the capture formation error variable according to the multi-unmanned ship motion system and the control tasks of each unmanned ship in the formation; S3: Based on the error variables of the encirclement formation, the backstepping method is used to construct the virtual controller of the unmanned ship to obtain the optimal derivative of the first Lyapunov function of the unmanned ship; The specific steps include: S31: Based on the ship motion model in step S12, the dynamic formation error of the capture formation error variable is obtained, and its expression is: Where: d i represents the in-degree of each unmanned ship in the formation, and η 2d,i represents the virtual controller of the unmanned ship to be solved; a i,j The decision variable that indicates that the i-th unmanned ship can receive information from the j-th unmanned ship; a i,0 represents the signal variable between each unmanned ship and the target ship; e i,1 represents the error variable of the leader ship of the siege formation; e i,2 represents the error variable of the follower ship of the siege formation; Δ i represents the relative distance between the i-th unmanned ship and the target ship; Δ j represents the relative distance between the jth unmanned ship and the target ship, and i,j∈[1,2,…,N]; represents a reversible rotation matrix; N represents the total number of ships in the unmanned ship formation; v i ,v j represents the velocity vector in the attached coordinate system, and v i =[u i ,v i ,r i ] T ;u i ,v i ,r i They represent the ship's angular velocity of pitch, roll and pitch respectively; τ i represents the control input; τ i,w represents unknown bounded environmental disturbance; f i (v i ) represents the nonlinear dynamic term of the unmanned ship; η i represents the position vector in the geodetic coordinate system; η 1f A track signal indicating the target ship; represents the inertia matrix; S32: Using the backstepping method to construct the first Lyapunov function of the unmanned ship according to the error variable of the capture formation, and the first Lyapunov function V i,1 The expression is S33: According to the first Lyapunov function V i,1 With the dynamic formation error, the derivative of the first Lyapunov function is obtained, and its expression is S34: Construct a virtual controller for the unmanned ship, whose expression is Where: c i,1 represents a positive constant used to design a virtual controller; S35: Bring the constructed virtual controller of the unmanned ship into step S33 to rewrite the derivative of the first Lyapunov function and obtain the optimal derivative of the first Lyapunov function as follows: S4: Based on the reinforcement learning method, the optimal derivative of the first Lyapunov function of the unmanned ship is combined with the neural network to construct the leader ship formation capture controller; The control law used by the leader ship formation capture controller includes the leader ship optimal control law, the ship feedforward adaptive law and the leader ship weight update law; S5: Based on the adaptive backstepping method, according to the optimal derivative of the first Lyapunov function of the unmanned ship, and combining the Lyapunov theory with the neural network, a follower ship formation capture controller is constructed; And the following ship formation capture controller includes an adaptive controller and an adaptive control law; S6: According to the leading ship formation capture controller and the following ship formation capture controller, the formation capture control of the target ship is realized.

2. According to the reinforcement learning-based multi-unmanned ship formation capture control method of claim 1, it is characterized in that: The S1 specifically includes the following steps: S11: A multi-unmanned ship motion system affected by environmental interference is constructed in the earth coordinate system and the attached coordinate system, and the ship motion model expression of the i-th unmanned ship in the multi-unmanned ship motion system is: Where: η i represents the position vector in the geodetic coordinate system, and x i ,y i Indicates the horizontal and vertical coordinates of the ship's position; Indicates the ship's heading; v i represents the velocity vector in the attached coordinate system, and v i =[u i ,v i ,r i ] T ;u i ,v i ,r i They represent the ship's angular velocity of pitch, angular velocity of roll and angular velocity of bow respectively; represents a reversible rotation matrix that satisfies I represents the identity matrix; represents the inertia matrix; C i (v i ) represents the Coriolis matrix; g i Denotes unmodeled dynamics; D i represents the nonlinear term affected by environmental interference; τ i represents the control input; τ i,w represents unknown bounded environmental disturbance, and ||τ i,W ||≤λ W , I∈[1,2,…,N]; N represents the total number of ships in the unmanned ship fleet; m 11 ,M 22 ,M 23 ,M 32 ,M 33 represents the additional mass of the ship; W represents the unknown bounded environmental disturbance τ i,w The upper bound of S12: Rewrite the ship motion model nonlinearly to obtain the final multi-unmanned ship motion system, and the ship motion model expression of the i-th unmanned ship in the final multi-unmanned ship motion system is: Where: f i (v i ) represents the nonlinear dynamic term of the unmanned ship, and S13: Use the leader-following method to divide the control tasks of each unmanned ship in the formation, specifically: Define the target ship’s trajectory signal as η 1d ; The relative formation distance between the target ship and the multi-unmanned ship motion system is Δ i ; Assuming that there are M follower ships and NM leader ships in the multi-unmanned ship formation, the task of the multi-unmanned ship formation to capture the target ship is described as The leader ship is used to achieve optimized trajectory tracking of the target ship, and the follower ship is used to ensure the coordinated capture formation.

3. According to the reinforcement learning-based multi-unmanned ship formation capture control method of claim 2, it is characterized in that: The error variable of the roundup formation constructed in S2 is expressed as Where: η 2d,i A virtual controller representing the unmanned ship to be solved; represents the adjacency matrix and a i,j The decision variable that indicates that the i-th unmanned ship can receive information from the j-th unmanned ship; a i,0 represents the signal variable between each unmanned ship and the target ship; e i,1 represents the error variable of the leader ship of the siege formation; e i,2 represents the error variable of the follower ship of the siege formation; Δ i represents the relative distance between the i-th unmanned ship and the target ship; Δ j represents the relative distance between the jth unmanned ship and the target ship, and i,j∈[1,2,…,N].

4. According to the reinforcement learning-based multi-unmanned ship formation capture control method of claim 3, it is characterized in that: The S4 specifically comprises the following steps: S41: Based on the reinforcement learning method, the second Lyapunov function is constructed according to the first Lyapunov function through the step-by-step design method of the feedforward and feedback controller. Its expression is: Where: V n,2 represents the second Lyapunov function; V n,1 Indicates the intermediate parameter quantity; e n,2 The error variable e represents the follower ship of the M+1th to Nth ship capture formation. i,2 The error matrix of e n,2 =[e M+1,2 ,e M+2,2 ,…,e N,2 ] T ; N represents the total number of ships in the multi-unmanned ship formation; M represents the number of follower ships; γ n represents the parameter matrix; represents the weight error of the neural network, and ω n represents the ideal weights of the neural network; represents the estimated value of the neural network weights and T stands for transpose; S42: Obtain the derivative of the second Lyapunov function according to the optimal derivative of the first Lyapunov function of the unmanned ship for Where: c n,1 represents a diagonal matrix of positive constants used to design the virtual controllers of NM leader ships, and c n,1 =diag{c M+1,1 ,c M+2,1 ,…,c N,1 }, Indicates c n,1 The upper bound of e n,1 represents the matrix of the first error variables of the NM leader ships, and e n,1 =[e M+1,1 ,e M+2,1 ,…,e N,1 ] T ; represents the diagonal matrix composed of the in-degree of NM leader ships, and a n represents the diagonal matrix composed of the signal variables of each ship and the target ship for NM leader ships and a n =diag{a M+1,0 ,a M+2,0 ,…,a N,0 }; represents the diagonal matrix composed of the rotation matrices of the NM leader ships, and And the derivative of the second Lyapunov function Rewrite as Where: τ n Represented by the feedforward controller and feedback controller The control input consists of And τ n =[τ M+1 ,τ M+2 ,…,τ N ]; τ n,w represents the unknown bounded environmental disturbance term of the leader ship; Represents the virtual controllers of NM leader ships; S43: Define intermediate variable h n (e n,2 ), and h n (e n,2 ) = f n (v n )-f n (η 2d,n ); According to the intermediate variable h n (e n,2 ) The derivative of the rewritten second Lyapunov function Transform into Where: k n (η 2d,n ) represents the 2d,n is a nonlinear function of S44: Using neural networks to approximate nonlinear functions k n (η 2d,n ), whose expression is Where: n represents the ideal weights of the neural network; Represents the activation function of the neural network; ε n (η 2d,n ) represents the approximation error of the neural network, and Represents ε n (η 2d,n )’s upper bound; Then the derivative of the second Lyapunov function in step S43 is Further rewritten as S45: To ensure the derivative of the second Lyapunov function further rewritten Less than or equal to 0, construct a ship feedforward controller Feed-forward adaptive law for ships Its expression is Where: c n,2 represents the positive definite matrix used to design the feedforward controller, Indicates c n,2 The upper bound of S46: Based on the ship feedforward controller Feed-forward adaptive law for ships The derivative of the second Lyapunov function in step S44 is Rewrite as Where: ε n (η 2d,n ) represents the approximation error of the neural network; S47: According to Lyapunov stability theory, consider the uncertain terms in S46 Constructing the optimal control law of the leading ship based on the nonlinear affine system of ships And the nonlinear affine system expression is Where: And define the cost function J(e according to the nonlinear affine system n,2 (t)), whose expression is Where: γ represents the integral variable; Θ(e n,2 (γ)) represents an additional term related to the uncertain dynamics, and Θ(e n,2 (γ))>0; represents the basic utility function and represents a positive definite matrix; R represents a positive constant matrix; S48: According to the cost function J(e n,2 (t)) Get the optimal cost function J * (e n,2 (t)) is And according to the optimal cost function J * (e n,2 (t)), solve the constructed Hamilton-Jacobi-Bellman equation to obtain the optimal control law of the leading ship The Hamilton-Jacobi-Bellman equation is expressed as Where: Indicates partial derivative; Θ(e n,2 ) represents the intermediate parameter quantity and λ w represents the unknown bounded environmental disturbance τ i,w The upper bound of The optimal control law of the leading ship The expression is S49: Using evaluation neural network to approximate the optimal cost function J * (e n,2 ), whose expression is Where: c represents the ideal weights for evaluating neural networks; φ(e n,2 ) represents the activation function of the evaluation neural network; ε c (e n,2 ) represents the approximation error of the evaluation neural network; J * (e n,2 ) is J * (e n,2 (t)) is an abbreviation; Get the optimal cost function J * (e n,2 ) has a partial derivative of Based on the evaluation neural network, according to the optimal cost function J * (e n,2 ) rewrites the optimal control law of the leading ship, the nonlinear affine system and the Hamilton-Jacobi-Bellman equation as Where: Ξ, Π, ε HJB represents the intermediate parameter quantity, and Define the optimal cost function J * (e n,2 ) for Where: Represents an estimate of the ideal weights for evaluating a neural network; Get the optimal cost function J * (e n,2 ) for And according to the optimal cost function J * (e n,2 ) Obtain the estimated value of the optimal control law of the leading ship, the estimated value of the nonlinear radiation system, and the estimated value of the Hamilton-Jacobi-Bellman equation, which is expressed as Construct the error function of the Hamilton-Jacobi-Bellman equation, which is expressed as Define the reinforcement function and design the weight update law of the evaluation neural network corresponding to the error value of the error function for eliminating the Hamilton-Jacobi-Bellman equation based on the reinforcement function That is, the leader ship weight update law, which is expressed as Where: α1 represents the adaptive learning parameter; α2 represents the adjustable parameter; J s (e n，2 ) represents the reinforcement function; Indicates J s (e n,2 ) first-order derivative; ε HJB represents the error of the Hamilton-Jacobi-Bellman equation.

5. According to the reinforcement learning-based multi-unmanned ship formation capture control method of claim 1, it is characterized in that: The S5 specifically includes the following steps: S51: Based on the adaptive backstepping method, the third Lyapunov function V is constructed according to the first Lyapunov function of the unmanned ship. m，2 , whose expression is Where: e m,2 represents the matrix of the second error variables of the M follower ships, and e m,2 =[e 1,2 ,e 2,2 ,…,e M,2 ] T ; γ m represents a positive definite constant matrix; represents the weight error of the adaptive neural network and ω m represents the ideal weights of the adaptive neural network; represents an estimate of the ideal weights of the adaptive neural network and V m,1 represents an intermediate variable; S52: Obtain the derivative of the third Lyapunov function according to the optimal derivative of the first Lyapunov function of the unmanned ship for m∈[1,2,…,M] Where: c m,1 represents a diagonal matrix of positive constants used to design the virtual controllers of M follower ships, and c m,1 =diag{c 1,1 ,c 2,1 ,…,c M,1 }, Indicates c m,1 The upper bound of e m,1 represents the matrix of the first error variables of the M follower ships, and e m,1 =[e 1,1 ,e 2,1 ,…,e M,1 ] T ; represents the diagonal matrix of the in-degree of M follower ships and a m represents the diagonal matrix composed of the signal variables of each ship and the target ship among the M follower ships, and a m =diag{a 1,0 ,a 2,0 ,…,a M,0 }; represents the diagonal matrix composed of the rotation matrices of the M follower ships, and And the derivative of the third Lyapunov function Rewrite as Where: τ m represents the control input of the follower ship, τ m =[τ1,τ2,…,τ M ]; τ m,w represents the unknown bounded environmental disturbance term of the follower ship; represents the virtual controllers of the M follower ships; S53: Using adaptive neural network to m (v m ) approximation processing, its expression is Where: m represents the ideal weights of the adaptive neural network; represents the activation function of the adaptive neural network; ε m (v m ) represents the approximation error of the adaptive neural network, Represents ε m (v m )’s upper bound; Then the derivative of the third Lyapunov function Further rewritten as Where: τ m,w represents the unknown bounded environmental disturbance of the follower ship, and τ m,w =[τ 1,w ,τ 2,w ,…,τ M,w ]; represents the first-order derivative of the virtual controller of the M follower ships, and S54: In order to ensure the derivative of the third Lyapunov function after further rewriting Less than or equal to 0, the adaptive controller and adaptive control law of the follower ship are constructed, and its expression is Where: τ m represents the control input of the adaptive controller; represents the adaptive control law; c m,2 represents the positive constant matrix used to design the follower ship adaptive controller.

Images