Unmanned surface vehicle-oriented zero-sum game self-triggering anti-disturbance reinforcement learning method and device

Abstract

The application discloses a zero-sum game self-triggered anti-disturbance reinforcement learning method and device for unmanned surface vehicle, and relates to the technical field of autonomous control of marine unmanned system. The method comprises the following steps: constructing a state space equation of the unmanned surface vehicle; defining the control input and the disturbance input as target opposite game players, constructing an augmented state of the unmanned surface vehicle, constructing an HJI equation, and obtaining a Nash feedback strategy of both parties; designing a self-triggered sampling mechanism, and calculating a next trigger time; constructing an evaluation network, an execution network and a disturbance network; constructing an evaluation network target function based on an experience replay mechanism, and designing a weight update rate of the evaluation network; taking the strategy structure induced by the evaluation network as a reference, constructing a target function of the execution network and the disturbance network; constructing a time projection operator, and constructing a weight update rate of the execution network and the disturbance network; and outputting an approximate Nash equilibrium strategy of the execution network and the disturbance network. By using the application, online solution can be realized, the calculation cost is low, and the redundancy of on-board calculation and communication is reduced.

Description

Technical Field

[0001] This invention relates to the field of autonomous control technology for marine unmanned systems, and in particular to a zero-sum game self-triggered anti-disturbance reinforcement learning method and device for unmanned surface vessels. Background Technology

[0002] In engineering scenarios such as autonomous navigation and formation coordination of unmanned surface vessels (USVs), maritime patrol and target tracking and interception, near-shore collision avoidance and narrow waterway navigation in ports, and the stability of marine observation platforms against wind, waves and currents, USV systems operate in highly uncertain marine environments for extended periods. They are often simultaneously affected by external disturbance forces / torques caused by the coupling of wind, waves and currents, thrust attenuation and cavitation effects of the propulsion system, load changes and mass parameter drift, sensor noise and communication delays, and may even face adversarial environmental inputs (such as human interference, induced target maneuvers, or sudden changes in severe sea conditions). In order to minimize the impact of disturbances while ensuring the closed-loop stability and navigation safety of USVs, the USV controller and the environmental disturbance party can be regarded as two opposing players in a game, thus establishing a zero-sum game model: the controller minimizes trajectory errors, energy consumption, and maneuvering costs by adjusting inputs such as thruster thrust, rudder angle, or multi-thrust differential thrust, while the disturbance party aims to maximize these performance indicators to characterize the most unfavorable wind, wave and current disturbances and unmodeled dynamics.

[0003] For general nonlinear unmanned surface vessel (USV) dynamics systems, the optimal solution to a zero-sum game is typically given by nonlinear HJI (Hamilton-Jacobi-Issacs) partial differential equations. However, traditional numerical methods (such as grid discretization and dynamic programming iteration) suffer from exponentially increasing computational complexity with increasing state dimension when the USV state dimension is high and the model includes nonlinear hydrodynamic terms and coupling constraints (such as velocity-heading-sideslip coupling, propulsion / controller saturation, and formation relative attitude constraints). This makes them difficult to deploy online on USV platforms where onboard computing resources are limited, power supply is constrained, and real-time response is required. Furthermore, to meet the high reliability and precision requirements of maritime navigation, traditional control implementations often employ extremely short-period sampling and continuous communication. This significantly increases bandwidth consumption in "vessel-to-vessel / vessel-to-shore" links, high-frequency navigation sensor readouts, and energy consumption from frequent controller updates, further limiting their application in multi-vessel formations, long-distance communication, and long-endurance missions. Summary of the Invention

[0004] To address the technical problems of existing technologies, such as the difficulty in solving the nonlinear HJI equations of zero-sum game for unmanned surface vessels online and the high computational cost, as well as the redundancy of onboard computing and communication and high energy consumption caused by periodic control / learning updates, this invention provides a self-triggered anti-disturbance reinforcement learning method and apparatus for zero-sum game for unmanned surface vessels. The technical solution is as follows: On the one hand, a zero-sum game self-triggered disturbance-resistant reinforcement learning method for unmanned surface vessels is provided. This method is implemented by a zero-sum game self-triggered disturbance-resistant reinforcement learning device for unmanned surface vessels, and includes: S1. Based on the parameters and velocity parameters of the unmanned surface vessel (USV), construct the kinematic model and dynamic model of the USV respectively. Based on the kinematic model and dynamic model, construct the state-space equation of the USV under the input affine form. S2. Define the control input and disturbance input as opposing players in the game, construct the augmented state of the unmanned surface vessel, construct the zero-sum game value function of the unmanned surface vessel, construct the game objective, construct the Hamilton-Jacobi-Isaks equation, and obtain the Nash feedback strategy of the two players. S3. Based on the state space equation and Nash feedback strategy, design a self-triggering sampling mechanism to calculate the next triggering time using the unmanned surface vessel state and learning parameters at the current triggering time, thereby realizing discrete triggering update; S4. Construct evaluation network, execution network and perturbation network to approximate the zero-sum game value function, control strategy and perturbation strategy of the unmanned surface vessel, respectively; S5. Construct the current evaluation error based on the current evaluation-execution-perturbation network, construct the historical evaluation error based on the experience replay mechanism, construct the objective function of the evaluation network based on the current evaluation error and the historical evaluation error, and design the weight update rate of the evaluation network. S6. Taking the policy structure induced by the evaluation network as a reference, construct the learning error at the current time and the learning error at the historical time for the execution network and the perturbation network respectively, and construct the objective function for the execution network and the objective function for the perturbation network. S7. Construct a discrete-time projection operator to act on the weights of the execution network, and combine it with the objective function of the execution network to construct the weight update rate of the execution network; construct a continuous-time projection operator to act on the weights of the perturbation network, and combine it with the objective function of the perturbation network to construct the weight update rate of the perturbation network. S8. When the network weights converge or reach the preset performance index, output the approximate Nash equilibrium strategy corresponding to the execution network and the perturbation network for online autonomous navigation control of unmanned surface vessels.

[0005] On the other hand, a zero-sum game self-triggered disturbance resistance reinforcement learning device for unmanned surface vessels is provided. This device is applied to the zero-sum game self-triggered disturbance resistance reinforcement learning method for unmanned surface vessels. The device includes: The state space construction unit is used to construct the kinematic model and dynamic model of the unmanned surface vessel (USV) based on its parameters and velocity parameters, and to construct the state space equation of the USV under the input affine form based on the kinematic model and dynamic model. An augmentation unit is used to define the control input and disturbance input as opposing players in a game, construct the augmented state of the unmanned surface vessel, construct the zero-sum game value function of the unmanned surface vessel, construct the game objective, construct the Hamilton-Jacobi-Isaks equation, and obtain the Nash feedback strategy of the two players. The sampling unit is used to design a self-triggering sampling mechanism based on the state space equation and Nash feedback strategy, and to calculate the next triggering time at the current triggering time using the unmanned surface vessel state and learning parameters, so as to realize discrete triggering update. The network construction unit is used to construct the evaluation network, execution network, and perturbation network, which are used to approximate the zero-sum game value function, control strategy, and perturbation strategy of the unmanned surface vessel, respectively. The first update rate construction unit is used to construct the current time evaluation error based on the current time evaluation-execution-perturbation network, construct the historical time evaluation error based on the experience replay mechanism, construct the evaluation network objective function based on the current time evaluation error and the historical time evaluation error, and design the weight update rate of the evaluation network. The objective function construction unit is used to construct the learning error at the current time and the learning error at the historical time for the execution network and the perturbation network, respectively, with reference to the policy structure induced by the evaluation network; and to construct the objective function for the execution network and the objective function for the perturbation network. The second update rate construction unit is used to construct a discrete-time projection operator applied to the weights of the execution network, and to construct the weight update rate of the execution network in combination with the objective function of the execution network; and to construct a continuous-time projection operator applied to the weights of the perturbation network, and to construct the weight update rate of the perturbation network in combination with the objective function of the perturbation network. The output unit is used to output the approximate Nash equilibrium strategy corresponding to the execution network and the perturbation network after the network weights converge or reach the preset performance index, for online autonomous navigation control of unmanned surface vessels.

[0006] On the other hand, a zero-sum game self-triggered anti-disturbance reinforcement learning device for unmanned surface vessels is provided. The zero-sum game self-triggered anti-disturbance reinforcement learning device for unmanned surface vessels includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the methods described above for the zero-sum game self-triggered anti-disturbance reinforcement learning method for unmanned surface vessels is implemented.

[0007] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described zero-sum game self-triggered disturbance-resistant reinforcement learning methods for unmanned surface vessels.

[0008] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention proposes a zero-sum game-based self-triggered reinforcement learning-based disturbance rejection control method for autonomous navigation and collaborative missions of unmanned surface vessels (USVs). By constructing a three-network structure of "evaluation-execution-disturbance" for the USV's propulsion-steering execution link, and under the background of nonlinear hydrodynamic coupling and uncertain disturbances of the USV, the method directly approximates the value function and saddle point strategy in the "state-control-disturbance" data space, thereby avoiding the need for high-dimensional nonlinear HJI (Hyperdynamic Interference) on the USV. The equations are solved explicitly numerically. By reusing key samples from the UAV's historical navigation process (such as straight-line steady state, sharp turn maneuvers, collision avoidance maneuvers, and strong current and wave conditions) through empirical playback, a learning error fused with historical information is constructed, reducing the dependence on continuous excitation and accelerating convergence. By introducing a projection operator to constrain the weights of the three networks within a pre-defined bounded set, the divergence of weights caused by sudden changes in sea state, parameter drift, or measurement noise is suppressed, improving the stability of the UAV's online learning and the feasibility of onboard implementation. Furthermore, by combining a self-triggering mechanism, the next update time is predicted online from the UAV's current navigation state and network parameters. Control quantities such as thruster thrust / rudder angle and network parameters are updated only at discrete trigger times, avoiding continuous high-frequency monitoring, frequent communication, and redundant updates at equal periods, significantly reducing the onboard computing and communication burden. Finally, under complex sea conditions such as wind, waves, and currents and the action of potential adversarial inputs, an approximate saddle point strategy for the UAV's zero-sum game is obtained, achieving a unity of closed-loop stability, trajectory tracking accuracy, and robust disturbance suppression performance. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of a zero-sum game self-triggered anti-disturbance reinforcement learning method for unmanned surface vessels provided by an embodiment of the present invention; Figure 2 This is a flowchart of a zero-sum game self-triggered anti-disturbance reinforcement learning method for unmanned surface vessels provided in an embodiment of the present invention; Figure 3 This is a block diagram of a zero-sum game self-triggered anti-disturbance reinforcement learning device for unmanned surface vessels provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a zero-sum game self-triggered anti-disturbance reinforcement learning device for unmanned surface vessels provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0012] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0013] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0014] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0015] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0016] This invention provides a zero-sum game-based self-triggered reinforcement learning method for unmanned surface vessels (USVs). This method can be implemented using a zero-sum game-based self-triggered reinforcement learning device for USVs, which can be a terminal or a server. This invention proposes a zero-sum game-based self-triggered reinforcement learning method for USVs, taking a single USV performing a stationary / low-speed track tracking mission in nearshore sea conditions as an example. The USV acquires attitude and velocity information through its onboard GNSS / IMU and achieves control through thrusters / rudders or differential propulsion. Environmental factors such as wind, waves, and currents are treated as disturbance inputs and constitute a zero-sum adversarial game with the control. The aim is to address key issues such as: the difficulty and high computational cost of solving the nonlinear HJI equations of the zero-sum game for USVs; redundancy in onboard computation and communication and high energy consumption due to periodic control / learning updates; the need for continuous monitoring of triggering conditions during event triggering, which still brings high-frequency state access and computational burden; and the tendency for network weights to diverge during online learning, leading to closed-loop instability and difficulty in engineering implementation.

[0017] like Figure 1 The flowchart shown is for a zero-sum game self-triggered disturbance-resistant reinforcement learning method for unmanned surface vessels. Figure 2 The flowchart shown is for a zero-sum game self-triggered disturbance-resistant reinforcement learning method for unmanned surface vessels. The processing flow of this method may include the following steps: S1. Based on the parameters and velocity parameters of the unmanned surface vessel (USV), construct the kinematic model and dynamic model of the USV respectively. Based on the kinematic model and dynamic model, construct the state-space equation of the USV under the input affine form.

[0018] In one feasible implementation, the kinematic and dynamic models of the unmanned surface vessel (USV) are constructed based on its two-dimensional coordinates and heading angle in the inertial coordinate system, and its longitudinal velocity, lateral velocity, and yaw rate in the hull coordinate system. Based on the control input, disturbance input, control allocation matrix, system state vector, kinematic model, and dynamic model, the state-space equations of the USV in the input affine form are constructed.

[0019] Taking unmanned surface vessels as the controlled object, a continuous-time nonlinear state-space model is established based on their actual motion mechanism in the horizontal plane (coupling of position, heading and velocity). External environmental factors such as wind, waves and currents, as well as possible adversarial inputs, are uniformly classified into "disturbance input channels" to provide an objectified dynamic basis for subsequent zero-sum game modeling.

[0020] Optionally, the specific processing procedure of S1 may include the following steps S11-S15: S11. Based on the three-degree-of-freedom motion of the unmanned surface vessel in the horizontal plane, establish an inertial coordinate system and a hull coordinate system.

[0021] In one feasible implementation, considering the three-degree-of-freedom motion of the unmanned surface vessel in the horizontal plane, an inertial coordinate system {I} and a hull coordinate system {B} are established.

[0022] S12. Obtain the parameters and speed parameters of the unmanned surface vessel (USV). The USV parameters include its two-dimensional position and heading angle in the inertial coordinate system, and the speed parameters include its longitudinal velocity, lateral velocity, and bow roll rate in the hull coordinate system.

[0023] In one feasible implementation, let Indicates that the unmanned surface vessel is in Two-dimensional position of coordinate system and heading angle ; Indicates that the unmanned surface vessel is in Longitudinal velocity of the coordinate system lateral speed Bow roll rate .

[0024] S13. Based on the rotation matrix from the hull coordinate system to the inertial coordinate system, the velocity parameters, and the heading angle, construct the kinematic model of the unmanned surface vessel.

[0025] In one feasible implementation, the kinematic model of the unmanned surface vessel (physically corresponding to "pose evolution under coordinate transformation") can be written as follows (1): (1) in, Let be the rotation matrix from the hull to the inertial frame.

[0026] S14. Based on the equivalent mass-moment of inertia matrix, the derivative of velocity parameters with respect to time, the Coriolis / eccentric matrix, the viscous and quadratic damping equivalent matrix, the generalized control force generated by propulsion / rudder control, the resultant torque, and the generalized disturbance force generated by the equivalent antagonistic input, construct the dynamic model of the unmanned surface vessel.

[0027] The equivalent mass-moment-of-inertia matrix is ​​constructed based on the equivalent mass of the unmanned surface vessel (USV) in the longitudinal direction, the equivalent mass in the lateral direction, and the equivalent moment of inertia of the USV about its roll axis. The Coriolis / eccentric matrix is ​​constructed based on the equivalent mass of the USV in the longitudinal direction, the equivalent mass in the lateral direction, the longitudinal velocity, and the lateral velocity. The viscous and quadratic damping equivalent matrix is ​​constructed based on the longitudinal velocity, the linear and quadratic damping coefficients of the longitudinal motion, the lateral velocity, the linear and quadratic damping coefficients of the lateral motion, the yaw rate, and the linear and quadratic damping coefficients of the rolling motion.

[0028] In one feasible implementation, the dynamic model of the unmanned surface vessel (physically corresponding to "velocity evolution under hydrodynamic / inertial conditions") can be written as follows (2): (2) in, The equivalent mass-moment-of-inertia matrix (may include additional mass); This represents the derivative of the velocity parameter with respect to time. These represent the equivalent mass of the unmanned surface vessel (USV) in the longitudinal direction, the equivalent mass of the USV in the lateral direction, and the equivalent moment of inertia of the USV about its rocking axis, respectively. Coriolis / eccentric matrix The equivalent matrix for viscosity and quadratic damping is... , , These represent the linear damping coefficients for longitudinal motion, lateral motion, and rocking motion, respectively. , , These represent the secondary damping coefficients for longitudinal motion, lateral motion, and rocking motion, respectively. The generalized control force generated by propulsion / rudder control; It refers to the generalized disturbance force generated by the resultant torque of wind, waves, currents, etc., as well as equivalent antagonistic inputs.

[0029] S15. Construct the control allocation matrix. Based on the control input, system state, disturbance input, drift term, control gain matrix, and disturbance gain matrix, construct the state space equation of the unmanned surface vessel in the input affine form, as shown in equation (3) below: (3) in, Indicates control input, Indicates the system status. This indicates a disturbance input. The drift term is represented by the kinematic model, the Coriolis / eccentric matrix, the viscous and quadratic damping equivalent matrix, the velocity parameters, and the equivalent mass-moment of inertia matrix. The control gain matrix is ​​constructed based on the zero matrix, the equivalent mass-moment of inertia matrix, and the control allocation matrix. The disturbance gain matrix is ​​represented by the zero matrix and the equivalent mass-moment of inertia matrix.

[0030] In one feasible implementation, to be consistent with the zero-sum game learning structure in the embodiments of the present invention, the control input is defined as Corresponding to the longitudinal resultant thrust of the unmanned surface vessel With bow roll control torque (On a twin-thruster boat, this can be achieved through differential thrust from the left and right thrusters); let the disturbance input... These represent the equivalent disturbances of the environment in the longitudinal, lateral, and bow-and-screw directions, respectively. The control allocation matrix is ​​taken as follows (4): (4) Define the system state vector as The dynamics of the unmanned surface vessel can then be organized into the input affine form uniformly adopted in this invention, as shown in equation (3) above, where: (5) The above In the unmanned surface vessel working area The local Lipschitz condition is satisfied. Describe the natural evolution of unmanned surface vessels under uncontrolled and undisturbed conditions, caused by kinematic and hydrodynamic damping. Describes the effect of propulsion / torque input on the acceleration of the unmanned surface vessel; Describe the equivalent injection methods of disturbances such as wind, waves, and currents into each channel of the unmanned surface vessel.

[0031] S2. Define the control input and disturbance input as opposing players in a game, construct the augmented state of the unmanned surface vessel, construct the zero-sum game value function of the unmanned surface vessel, construct the game objective, construct the Hamilton-Jacobi-Isaks equation, and obtain the Nash feedback strategy of the two players.

[0032] In one feasible implementation, the available executable quantities such as propulsion / rudder control of the unmanned surface vessel (USV) are defined as decision variables for the controlling player, and environmental disturbances and adversarial inputs are defined as decision variables for the disturbing player. An integral performance index is constructed that includes indicators such as trajectory / fixed-point error, attitude error, speed, and control energy, so that the adversarial relationship of "minimizing by the controlling player and maximizing by the disturbing player" is consistent with the mission objective of the USV. Based on this, a zero-sum game optimization problem and the corresponding HJI (Hamilton-Jacobi-Issacs) equation are formed.

[0033] Optionally, the specific processing procedure of S2 may include the following steps S21-S26: S21. Construct the augmented state of the unmanned surface vessel based on the reference pose, reference velocity, pose error, and velocity error.

[0034] In one feasible implementation, based on the unified dynamics model of the unmanned surface vessel (USV) in step S1 under the affine form of the input, the control input and disturbance input are regarded as opposing players in a game. The USV's mission is stationary maintenance / track tracking; for a unified description, the reference pose is... and reference speed And define pose error and speed error ,in Indicates the heading angle difference mapped to (To conform to the physical meaning of actual heading). Definition To enhance the status of unmanned surface vessels.

[0035] S22. Based on the unified dynamic model, the augmented state of the unmanned surface vessel, the drift term of the augmented system, the control gain matrix, and the disturbance gain matrix, construct the tracking dynamic equations of the unmanned surface vessel.

[0036] In one feasible implementation, the dynamic equation for unmanned surface vessel tracking is constructed as follows (6): (6) in: (7) S23. Define the zero-sum game value function of the unmanned surface vessel based on the integral summation of the instantaneous cost, where the instantaneous cost is constructed based on the augmented state weight function of the unmanned surface vessel, the symmetric matrix, and the degree of suppression of the disturbance by the controller.

[0037] In one feasible implementation, the zero-sum game value function of the unmanned surface vessel is defined as follows (8): (8) Instantaneous cost middle, An augmented state weight function is used for the unmanned surface vessel to reflect the requirements for velocity and angular velocity stability; the symmetric matrix R>0 represents the input weights of the controlling player, characterizing the energy consumption / execution cost of propulsion and steering. To determine the degree to which the controlling player suppresses the disruptive player, This is the discount factor.

[0038] S24. Based on game theory and the zero-sum game value function of unmanned surface vessels, the game objective is defined as follows (9): (9) in, Represents the zero-sum game value function of the unmanned surface vessel. Indicates the augmentation status of the unmanned surface vessel. Representing state The set to which it resides, where u represents the control input and d represents the disturbance input. This represents the set of permissible control inputs u. This represents the set of perturbation inputs d with finite energy.

[0039] In one feasible implementation, according to game theory, the goal of the controlling party is to minimize the value function (8), and the goal of the perturbing party is to maximize the value function (8). Therefore, a zero-sum game problem can be defined, and the specific formula is as shown in the above formula (9).

[0040] S25. Based on the game objective and the Nash saddle point feedback strategy, construct the Nash equilibrium conditions.

[0041] One feasible implementation is the Nash saddle point feedback strategy for zero-sum games involving unmanned surface vessels. The Nash equilibrium condition is satisfied, as shown in equation (10): (10) S26. Construct the Hamilton-Jacobi-Isaks equation and obtain the Nash feedback strategies of the controller and the disturbance in the Nash strategy based on the stationary point condition.

[0042] In one feasible implementation, the output of the above zero-sum game It can be obtained by solving the HJI equation, which is specifically shown in equation (11): (11) Based on equation (11), the feedback structure between the control side and the disturbance side in the Nash strategy (i.e., the Nash feedback strategy) can be obtained according to the stationary point condition, as follows: (12) (13) Equation (12) reflects the relationship between the unmanned surface vessel's propulsion / torque control and the gradient of the value function: when the position / heading / velocity error increases, the control will be enhanced along the direction of suppressing the error; Equation (13) reflects the direction selection of the disturbance party in each channel for the "most unfavorable disturbance".

[0043] S3. Based on the state-space equation and Nash feedback strategy, a self-triggered sampling mechanism is designed to calculate the next trigger time using the unmanned surface vessel's state and learning parameters at the current trigger time, thereby realizing discrete trigger update.

[0044] In one feasible implementation, a self-triggered sampling and update structure is designed around the "measurement-control-communication" link of the unmanned surface vessel (USV): the USV state is frozen once at each trigger moment for strategy calculation, and the next trigger moment is calculated based on the current state change trend and learned parameters, thereby avoiding continuous online monitoring of trigger conditions and realizing adaptive sparse updates oriented towards sea state changes.

[0045] Optionally, the specific processing procedure of S3 may include the following steps S31-S32: S31. Define the sampling state based on the real state, which is obtained by continuous evolution of the dynamic equation for unmanned surface vessel tracking.

[0046] In one feasible implementation, to reduce the onboard computing and communication burden, this embodiment of the invention defines a sampling state on the unmanned surface vessel: (14) in This is the current sampling time. It is a neighbor The next sampling time, The sampled state represents the unmanned surface vessel's state (including pose and velocity) frozen between two triggers, used by the network to calculate and maintain control output within the interval; the real state... The unmanned surface vessel augmentation system continues to evolve dynamically (6) in step S2.

[0047] S32. Design a self-triggered sampling mechanism based on the sampling state. The self-triggered sampling mechanism is activated at the current time. Calculate the next sampling time based on system information. As shown in equation (15): (15) in, , , yes Convex functions yes Class function, Sampling error The upper bound of the squared norm, where the sampling error is , It is the lower bound of the square of the state norm of the unmanned surface vessel:

[0048] For design parameters, for any and satisfy:

[0049] In one feasible implementation, to reduce communication costs and improve computational efficiency, a sampling state is defined. Based on the sampling state in (14), this invention designs a self-triggered sampling mechanism at the current moment. Based on system information, the next sampling time is intelligently calculated. Specifically, as shown in equation (15) above.

[0050] Unmanned surface vessel Consistent at all times Compared with current measurements Therefore, the solution can be obtained. The root is used as the next trigger interval to achieve adaptive updates of "stable sea conditions - sparse triggering, severe sea conditions - encrypted triggering".

[0051] S4. Construct evaluation network, execution network, and perturbation network to approximate the zero-sum game value function, control strategy, and perturbation strategy of the unmanned surface vessel, respectively.

[0052] In one feasible implementation, an evaluation network is designed to approximate the zero-sum game value function of the unmanned surface vessel (USV). Basis functions / features directly related to the USV's navigation error, heading angle, and speed are selected to ensure that the approximate value function reflects the USV's mission performance and robust disturbance suppression requirements. An execution network and a disturbance network are designed to approximate the strategies of the controller and the disturbance party, respectively. The output of the execution network is mapped to actual implementable controllable quantities such as the USV's thrust / torque or the differential thrust of the dual-thruster system. The output of the disturbance network is used to characterize the effect of equivalent environmental disturbances / adversarial inputs on the USV, thereby forming an objectified closed-loop structure of reinforcement learning: "open-loop learning - closed-loop application".

[0053] Optionally, the specific processing procedure of S4 may include the following steps S41-S44: S41. Select basis functions related to the unmanned surface vessel's navigation error, heading angle, and speed. Based on the ideal weights, parameterized residuals, and basis functions of the evaluation network, parameterize the zero-sum game value function of the unmanned surface vessel.

[0054] In one feasible implementation, to avoid directly solving the HJI equation (11) of the unmanned surface vessel in step S2 numerically, this embodiment of the invention employs a reinforcement learning structure to... Perform function approximation. Select a set of basis functions. Value function Parameterization, specifically as shown in equation (16): (16) in To evaluate the ideal weights of the network, For parameterized residuals.

[0055] S42. Construct an evaluation network based on the weights of the evaluation network and the augmented state of the unmanned surface vessel (USV) to approximate the parameterized zero-sum game value function of the USV.

[0056] In one feasible implementation, an evaluation network is constructed to approximate... Specifically, it is shown in formula (17): (17) in For the first The weights of the evaluation network. To reflect the direct impact of the unmanned surface vessel's attitude / heading / speed on mission performance, for example, the following can be selected: (18) S43. Construct the execution network of the controller based on the weights of the execution network and the sampled state to approximate the optimal strategy of the controller. The output of the execution network is mapped to the control quantity that the unmanned surface vessel can implement.

[0057] S44. Construct a perturbation network for the perturberer based on the weights of the perturbation network and the true state. This network is used to approximate the optimal strategy of the perturberer. The output of the perturbation network is used to characterize the equivalent environmental perturbation or adversarial input.

[0058] In one feasible implementation, based on steps (12) and (13) of S2, basis functions are constructed for the unmanned surface vessel (USV) controller and the disturbance party, respectively. and An execution network and a perturbation network are constructed to approximate the player policy in the Nash equilibrium solution. The control-side execution network uses the sampled state from step S3. The input and output controllable quantities (longitudinal thrust and yaw moment) are as follows: (19) The disturbance side disturbance network is in the real state. The input-output equivalent perturbation generalized force is given by the following equation (20): (19) (20) in To execute the network weights, The weights are used for the perturbation network. The execution network determines the thrust / torque based on the UAV's attitude and velocity, while the perturbation network expresses the characteristics of the equivalent environmental effect that increase with velocity. Based on (19) and (20), the UAV system operates according to the dynamic (6) in step S2.

[0059] S5. Construct the current evaluation error based on the current evaluation-execution-perturbation network, construct the historical evaluation error based on the experience replay mechanism, construct the objective function of the evaluation network based on the current evaluation error and the historical evaluation error, and design the weight update rate of the evaluation network.

[0060] In one feasible implementation, the "evaluation" error signal in the form of HJI residuals is constructed using the current "evaluation-execution-disturbance" network output of the unmanned surface vessel, and experience replay is introduced to extract representative information from historical navigation data to form a learning objective that simultaneously includes current and historical data, thereby improving the learning efficiency of the evaluation network and reducing its dependence on continuous incentives.

[0061] Optionally, the specific processing procedure of S5 may include the following steps S51-S54: S51. Based on the evaluation network, regression factors, and instantaneous cost, construct the evaluation error at the current moment. The regression factors are constructed from the gradient of the augmented state of the unmanned surface vessel with respect to the basis functions, the true state, the dynamic equation of the unmanned surface vessel tracking, the sampled state, and the basis functions.

[0062] In one feasible implementation, based on the evaluation-execution-perturbation network (17), (19), (20) of step S3, the evaluation error at the current moment is constructed as follows: (21) (twenty one) in The instantaneous cost of the unmanned surface vessel (corresponding to position / heading / speed errors and propulsion energy consumption, disturbance countermeasures). The regression factor is as follows (22): (twenty two) in, The gradient of the basis function with respect to the augmented state of the unmanned surface vessel is shown; the part in parentheses represents the direction of the actual state evolution of the unmanned surface vessel under the current control and disturbance estimation.

[0063] S52. Select historical moments based on the experience playback mechanism, and construct the evaluation error of historical moments based on the evaluation error of the current moment.

[0064] In one feasible implementation, to relax the continuous stimulus conditions using historical navigation data, a historical moment is selected based on an experience playback mechanism. , The evaluation error is constructed as follows (23): (twenty three) S53. Based on the evaluation error at the current moment, the evaluation error at historical moments, and the regression factor, construct the objective function of the evaluation network.

[0065] In one feasible implementation, the objective function of the evaluation network is constructed based on the current evaluation error (21) and the historical evaluation error (23), as shown in the following equation (24): (twenty four) S54. Based on the objective function of the evaluation network and the gradient descent method, design the weight update rate of the evaluation network.

[0066] In one feasible implementation, the update law of the evaluation network is designed according to gradient descent, as shown in equation (25): (25) in To evaluate the network learning rate, the above design enables the unmanned surface vessel to use both current sea state data and historical navigation data to drive the value function approximation, thereby improving learning efficiency and reducing dependence on continuous stimulation.

[0067] S6. Taking the policy structure induced by the evaluation network as a reference, construct the learning error at the current time and the learning error at the historical time for the execution network and the perturbation network respectively, and construct the objective function for the execution network and the objective function for the perturbation network.

[0068] In one feasible implementation, the learning error of the execution / perturbation network is constructed with reference to the policy structure induced by the evaluation network, and the historical error of the experience replay is combined to drive the network training, so that the unmanned surface vessel control strategy gradually approaches the saddle point strategy of the zero-sum game, while enhancing the suppression performance against disturbances to the marine environment.

[0069] Optionally, the specific processing procedure of S6 may include the following steps S61-S64: S61. The optimal control strategy and the optimal disturbance strategy are obtained by the evaluation network.

[0070] In one feasible implementation, the purpose of the execution network of the unmanned surface vessel controller and the perturbation network of the perturbation party is to approximate the player strategy induced by the evaluation network in step S3. The optimal control strategy and the optimal perturbation strategy are specifically as follows: (26) and (27): (26) (27) S62. Based on the difference between the execution network and the optimal control strategy, construct the learning error of the execution network at the current moment. Based on the difference between the perturbation network and the optimal perturbation strategy, construct the learning error of the perturbation network at the current moment.

[0071] In one feasible implementation, the learning error of the execution network and the perturbation network at the current moment is constructed by combining the controlling player given by the execution network (19) and the perturbation player given by the perturbation network (20) in step S3, as shown in equations (28) and (29) below: (28) (29) in, .

[0072] S63. Select historical moments based on the experience playback mechanism, construct the learning error of the historical moments of the execution network based on the learning error of the current moment of the execution network, and construct the learning error of the historical moments of the perturbation network based on the learning error of the current moment of the perturbation network.

[0073] In one feasible implementation, the experience replay mechanism in step S5 is used to select historical moments for the execution network. , Selecting historical moments for the perturbation network , The learning errors of the execution network and the perturbation network at historical moments are constructed respectively, as shown in equations (30) and (31): (30) (31) S64. Based on the learning error of the execution network at the current moment and the learning error at the historical moment, construct the objective function of the execution network. Based on the learning error of the perturbation network at the current moment and the learning error at the historical moment, construct the objective function of the perturbation network.

[0074] In one feasible implementation, the execution network and perturbation network based on the reinforcement learning of unmanned surface vessels learn the errors at the current and historical moments, and construct objective functions respectively, as shown in equations (32) and (33) below: (32) (33) The physical meaning of the above error and objective function is: to make the thrust / torque output by the unmanned surface vessel execution network gradually approach the saddle point control structure induced by the gradient of the value function, and at the same time make the disturbance network gradually approach the "most unfavorable disturbance" structure, thereby improving the robust disturbance suppression capability of the unmanned surface vessel in the sense of adversarial.

[0075] S7. Construct a discrete-time projection operator to act on the weights of the execution network, and combine it with the objective function of the execution network to construct the weight update rate of the execution network; construct a continuous-time projection operator to act on the weights of the perturbation network, and combine it with the objective function of the perturbation network to construct the weight update rate of the perturbation network.

[0076] In one feasible implementation, a projection operator is introduced into the execution / perturbation network parameter update to constrain the network weights within a preset bounded set, thereby avoiding the instability of unmanned surface vessel control and the impracticality of engineering caused by weight divergence during the learning process. It is also combined with a self-triggered update mechanism to perform discrete updates only at the triggering time.

[0077] To avoid the unrealizable control caused by the divergence of network weights during the online learning process of the unmanned surface vessel, a discrete-time projection operator is introduced, as shown in equation (34): (34) in, This represents the current weight vector. This represents the upper bound (constraint radius) of the weight norm.

[0078] The discrete-time projection operator is applied to the weights of the unmanned surface vessel execution network to construct the weight update law, as shown in equation (35): (35) in , To strengthen the implementation of online learning rules, To impose constraints on the network weights, such that... ; Indicates the execution of the network objective function Its weight The gradient.

[0079] The continuous-time projection operator is constructed as follows (36): (36) in, They are vectors of the same dimension. It is a one-dimensional convex function. , It is an n-dimensional positive definite matrix.

[0080] Applying the continuous-time projection operator to the perturbation network weights, a weight update law is constructed as follows (37): (37) in To disrupt the learning laws of the network for unmanned surface vessels. , To constrain the weights of the reinforcement learning perturbation network, such that... .

[0081] By using the projection constraints of (35) and (37), the weights of the unmanned surface vessel can be guaranteed to be bounded when it performs online learning in complex sea conditions, thereby enhancing the closed-loop stability and engineering feasibility.

[0082] S8. When the network weights converge or reach the preset performance index, output the approximate Nash equilibrium strategy corresponding to the execution network and the perturbation network for online autonomous navigation control of unmanned surface vessels.

[0083] In one feasible implementation, according to steps S1-S7, after the weights of the "evaluation-execution-perturbation" reinforcement learning network converge, the execution network and the perturbation network can respectively output the unmanned surface vessel control strategy. With perturbation strategy This allows us to obtain an approximate Nash saddle solution in the zero-sum game sense. In actual operation, the unmanned surface vessel will... The thrust is mapped to the combined thrust and differential thrust (or thrust and torque) of the propulsion system and applied to achieve robust suppression of disturbances and adversarial inputs such as wind, waves and currents. At the same time, relying on the self-triggering mechanism, the network parameters are updated only at discrete moments to achieve a balance between high performance and low resource consumption.

[0084] This invention proposes a zero-sum game-based self-triggered reinforcement learning-based disturbance rejection control method for autonomous navigation and collaborative missions of unmanned surface vessels (USVs). By constructing a three-network structure of "evaluation-execution-disturbance" for the USV's propulsion-steering execution link, and under the background of nonlinear hydrodynamic coupling and uncertain disturbances of the USV, the method directly approximates the value function and saddle point strategy in the "state-control-disturbance" data space, thereby avoiding the need for high-dimensional nonlinear HJI (Hyperdynamic Interference) on the USV. The equations are solved explicitly numerically. By reusing key samples from the UAV's historical navigation process (such as straight-line steady state, sharp turn maneuvers, collision avoidance maneuvers, and strong current and wave conditions) through empirical playback, a learning error fused with historical information is constructed, reducing the dependence on continuous excitation and accelerating convergence. By introducing a projection operator to constrain the weights of the three networks within a pre-defined bounded set, the divergence of weights caused by sudden changes in sea state, parameter drift, or measurement noise is suppressed, improving the stability of the UAV's online learning and the feasibility of onboard implementation. Furthermore, by combining a self-triggering mechanism, the next update time is predicted online from the UAV's current navigation state and network parameters. Control quantities such as thruster thrust / rudder angle and network parameters are updated only at discrete trigger times, avoiding continuous high-frequency monitoring, frequent communication, and redundant updates at equal periods, significantly reducing the onboard computing and communication burden. Finally, under complex sea conditions such as wind, waves, and currents and the action of potential adversarial inputs, an approximate saddle point strategy for the UAV's zero-sum game is obtained, achieving a unity of closed-loop stability, trajectory tracking accuracy, and robust disturbance suppression performance.

[0085] Figure 3This is a block diagram of a zero-sum game self-triggered disturbance-resistant reinforcement learning device for unmanned surface vessels (USVs) provided in an embodiment of the present invention. This device is used in a zero-sum game self-triggered disturbance-resistant reinforcement learning method for USVs. (Refer to...) Figure 3 The device includes: The state space construction unit 310 is used to construct the kinematic model and dynamic model of the unmanned surface vessel (USV) based on its parameters and velocity parameters, and to construct the state space equation of the USV in the input affine form based on the kinematic model and dynamic model. Augmentation unit 320 is used to define the control input and disturbance input as opposing players in a game, construct the augmented state of the unmanned surface vessel, construct the zero-sum game value function of the unmanned surface vessel, construct the game objective, construct the Hamilton-Jacobi-Isaks equation, and obtain the Nash feedback strategy of the two players. The sampling unit 330 is used to design a self-triggering sampling mechanism based on the state space equation and Nash feedback strategy, and to calculate the next triggering time at the current triggering time using the unmanned surface vessel state and learning parameters, so as to realize discrete triggering update. Network building unit 340 is used to build evaluation network, execution network and perturbation network, which are used to approximate the zero-sum game value function, control strategy and perturbation strategy of unmanned surface vessel, respectively; The first update rate construction unit 350 is used to construct the current time evaluation error based on the current time evaluation-execution-perturbation network, construct the historical time evaluation error based on the experience replay mechanism, construct the evaluation network objective function based on the current time evaluation error and the historical time evaluation error, and design the weight update rate of the evaluation network. The objective function construction unit 360 is used to construct the learning error at the current time and the learning error at the historical time of the execution network and the perturbation network, respectively, with reference to the policy structure induced by the evaluation network; and to construct the objective function of the execution network and the objective function of the perturbation network. The second update rate construction unit 370 is used to construct a discrete-time projection operator applied to the weights of the execution network, and to construct the weight update rate of the execution network in combination with the objective function of the execution network; and to construct a continuous-time projection operator applied to the weights of the perturbation network, and to construct the weight update rate of the perturbation network in combination with the objective function of the perturbation network. The output unit 380 is used to output the approximate Nash equilibrium strategy corresponding to the execution network and the perturbation network after the network weights converge or reach the preset performance index, for online autonomous navigation control of unmanned surface vessels.

[0086] Figure 4 This is a schematic diagram of a zero-sum game self-triggered anti-disturbance reinforcement learning device for unmanned surface vessels provided in an embodiment of the present invention, as shown below. Figure 4 As shown, a zero-sum game self-triggered disturbance-resistant reinforcement learning device for unmanned surface vessels can include the above-mentioned... Figure 3 The illustrated zero-sum game self-triggered disturbance-resistant reinforcement learning device for unmanned surface vessels (USVs) is shown. Optionally, the zero-sum game self-triggered disturbance-resistant reinforcement learning device 410 for USVs may include a first processor 2001.

[0087] Optionally, the zero-sum game self-triggered disturbance-resistant reinforcement learning device 410 for unmanned surface vessels may also include a memory 2002 and a transceiver 2003.

[0088] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0089] The following is combined with Figure 4 The components of the 410 zero-sum game self-triggered disturbance-resistant reinforcement learning device 410 for unmanned surface vessels are described in detail: The first processor 2001 is the control center of the zero-sum game self-triggered anti-disturbance reinforcement learning device 410 for unmanned surface vessels. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0090] Optionally, the first processor 2001 can execute various functions of the zero-sum game self-triggered anti-disturbance reinforcement learning device 410 for unmanned surface vessels by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0091] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0092] In a specific implementation, as one example, the zero-sum game self-triggered anti-disturbance reinforcement learning device 410 for unmanned surface vessels may also include multiple processors, such as... Figure 4The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0093] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0094] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the zero-sum game self-triggered anti-disturbance reinforcement learning device 410 for unmanned surface vessels. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0095] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0096] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0097] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently, and it can be connected to the interface circuit of the zero-sum game self-triggered anti-disturbance reinforcement learning device 410 for unmanned surface vessels. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0098] It should be noted that, Figure 4 The structure of the zero-sum game self-triggered anti-disturbance reinforcement learning device 410 for unmanned surface vessels shown in the figure does not constitute a limitation on the router. The actual zero-sum game self-triggered anti-disturbance reinforcement learning device for unmanned surface vessels may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0099] Furthermore, the technical effect of the zero-sum game self-triggered anti-disturbance reinforcement learning device 410 for unmanned surface vessels can be referred to the technical effect of the zero-sum game self-triggered anti-disturbance reinforcement learning method for unmanned surface vessels described in the above method embodiments, and will not be repeated here.

[0100] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0101] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0102] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0103] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0104] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0105] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0106] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0108] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0109] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0111] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A zero-sum game-based self-triggered disturbance-resistant reinforcement learning method for unmanned surface vessels, characterized in that, The method includes: S1. Based on the parameters and velocity parameters of the unmanned surface vessel (USV), construct the kinematic model and dynamic model of the USV respectively. Based on the kinematic model and dynamic model, construct the state-space equation of the USV under the input affine form. S2. Define the control input and disturbance input as opposing players in the game, construct the augmented state of the unmanned surface vessel, construct the zero-sum game value function of the unmanned surface vessel, construct the game objective, construct the Hamilton-Jacobi-Isaks equation, and obtain the Nash feedback strategy of the two players. S3. Based on the state space equation and Nash feedback strategy, design a self-triggering sampling mechanism to calculate the next triggering time using the unmanned surface vessel state and learning parameters at the current triggering time, thereby realizing discrete triggering update; S4. Construct evaluation network, execution network and perturbation network to approximate the zero-sum game value function, control strategy and perturbation strategy of the unmanned surface vessel, respectively; S5. Construct the current evaluation error based on the current evaluation-execution-perturbation network, construct the historical evaluation error based on the experience replay mechanism, construct the objective function of the evaluation network based on the current evaluation error and the historical evaluation error, and design the weight update rate of the evaluation network. S6. Taking the policy structure induced by the evaluation network as a reference, construct the learning error at the current time and the learning error at the historical time for the execution network and the perturbation network respectively, and construct the objective function for the execution network and the objective function for the perturbation network. S7. Construct a discrete-time projection operator to act on the weights of the execution network, and combine it with the objective function of the execution network to construct the weight update rate of the execution network; construct a continuous-time projection operator to act on the weights of the perturbation network, and combine it with the objective function of the perturbation network to construct the weight update rate of the perturbation network. S8. When the network weights converge or reach the preset performance index, output the approximate Nash equilibrium strategy corresponding to the execution network and the perturbation network for online autonomous navigation control of unmanned surface vessels.

2. The zero-sum game self-triggered disturbance-resistant reinforcement learning method for unmanned surface vessels according to claim 1, characterized in that, S1 constructs a kinematic model and a dynamic model of the unmanned surface vessel (USV) based on its parameters and velocity parameters. Then, based on these kinematic and dynamic models, it constructs a state-space model of the USV under an affine input, including: S11. Based on the three-degree-of-freedom motion of the unmanned surface vessel in the horizontal plane, establish an inertial coordinate system and a hull coordinate system; S12. Obtain the parameters and speed parameters of the unmanned surface vessel (USV). The USV parameters include the USV's two-dimensional position and heading angle in the inertial coordinate system, and the speed parameters include the USV's longitudinal velocity, lateral velocity, and bow roll rate in the hull coordinate system. S13. Based on the rotation matrix from the hull coordinate system to the inertial coordinate system, the velocity parameters, and the heading angle, construct the kinematic model of the unmanned surface vessel; S14. Based on the equivalent mass-moment of inertia matrix, the derivative of velocity parameters with respect to time, the Coriolis / eccentric matrix, the viscous and quadratic damping equivalent matrix, the generalized control force generated by propulsion / rudder control, the resultant torque, and the generalized disturbance force generated by the equivalent antagonistic input, construct the dynamic model of the unmanned surface vessel. The equivalent mass-moment of inertia matrix is ​​constructed based on the equivalent mass of the unmanned surface vessel (USV) in the longitudinal direction, the equivalent mass of the USV in the lateral direction, and the equivalent moment of inertia of the USV about its rocking axis; the Coriolis / centrifugal matrix is ​​constructed based on the equivalent mass of the USV in the longitudinal direction, the equivalent mass of the USV in the lateral direction, the longitudinal velocity, and the lateral velocity; the viscous and quadratic damping equivalent matrix is ​​constructed based on the longitudinal velocity, the linear damping coefficient and the quadratic damping coefficient of the longitudinal motion, the lateral velocity, the linear damping coefficient and the quadratic damping coefficient of the lateral motion, the bow angular velocity, and the linear damping coefficient and the quadratic damping coefficient of the rocking motion. S15. Construct the control allocation matrix. Based on the control input, system state, disturbance input, drift term, control gain matrix, and disturbance gain matrix, construct the state space equation of the unmanned surface vessel in the input affine form, as shown in equation (1) below: （1） in, Indicates control input, Indicates the system status. Indicates a disturbance input; The drift term is represented by the kinematic model, Coriolis / eccentric matrix, viscous and quadratic damping equivalent matrix, velocity parameters, and equivalent mass-moment of inertia matrix. The control gain matrix is ​​constructed based on the zero matrix, the equivalent mass-moment-of-inertia matrix, and the control assignment matrix. The disturbance gain matrix is ​​represented by the zero matrix and the equivalent mass-moment of inertia matrix.

3. The zero-sum game self-triggered disturbance-resistant reinforcement learning method for unmanned surface vessels according to claim 1, characterized in that, The S2 defines the control input and disturbance input as opposing players in a game, constructs the augmented state of the unmanned surface vessel (USV), constructs the USV zero-sum game value function, constructs the game objective, constructs the Hamilton-Jacobi-Isaks equation, and obtains the Nash feedback strategies of both players, including: S21. Construct the augmented state of the unmanned surface vessel based on the reference pose, reference velocity, pose error, and velocity error; S22. Based on the unified dynamics model, the augmented state of the unmanned surface vessel, the drift term of the augmented system, the control gain matrix, and the disturbance gain matrix, construct the tracking dynamic equations of the unmanned surface vessel. S23. Define the zero-sum game value function of the unmanned surface vessel based on the integral summation of the instantaneous cost, wherein the instantaneous cost is constructed based on the augmented state weight function of the unmanned surface vessel, the symmetric matrix, and the degree of suppression of the disturbance by the control party. S24. Based on game theory and the zero-sum game value function of unmanned surface vessels, the game objective is defined as follows (2): （2） in, Represents the zero-sum game value function of the unmanned surface vessel. Indicates the augmented state of the unmanned surface vessel. Representing state The set to which it resides, where u represents the control input and d represents the disturbance input. This represents the set of permissible control inputs u. The set representing the perturbation input d with finite energy; S25. Construct the Nash equilibrium conditions based on the game objective and the Nash saddle point feedback strategy; S26. Construct the Hamilton-Jacobi-Isaks equation and obtain the Nash feedback strategies of the control and disturbance sides in the Nash strategy based on the stationary point condition.

4. The zero-sum game self-triggered disturbance-resistant reinforcement learning method for unmanned surface vessels according to claim 1, characterized in that, Based on the state-space equation and Nash feedback strategy, S3 designs a self-triggered sampling mechanism to calculate the next trigger time using the unmanned surface vessel's state at the current trigger time, achieving discrete trigger updates, including: S31. Define the sampling state according to the real state, wherein the real state is obtained by continuous evolution based on the dynamic equation of the unmanned surface vessel tracking. S32. Design a self-triggering sampling mechanism based on the sampling state. The self-triggering sampling mechanism at the current time... Calculate the next sampling time based on system information. As shown in equation (3): （3） in, , , yes Convex functions yes Class function, It is the upper bound of the square of the sampling error norm. It is the lower bound of the square of the state norm of the unmanned surface vessel.

5. The zero-sum game self-triggered disturbance-resistant reinforcement learning method for unmanned surface vessels according to claim 1, characterized in that, The S4 construction evaluation network, execution network, and perturbation network are used to approximate the unmanned surface vessel's zero-sum game value function, control strategy, and perturbation strategy, respectively, including: S41. Select basis functions related to the unmanned surface vessel's navigation error, heading angle, and speed. Based on the ideal weights, parameterized residuals, and basis functions of the evaluation network, parameterize the unmanned surface vessel's zero-sum game value function. S42. Construct an evaluation network based on the weights of the evaluation network and the augmented state of the unmanned surface vessel (USV) to approximate the parameterized zero-sum game value function of the USV. S43. Construct the execution network of the controller based on the weights of the execution network and the sampled state to approximate the optimal strategy of the controller. The output of the execution network is mapped to the control quantity that the unmanned surface vessel can implement. S44. Construct a perturbation network for the perturberer based on the weights of the perturbation network and the true state. This network is used to approximate the optimal strategy of the perturberer. The output of the perturbation network is used to characterize the equivalent environmental perturbation or adversarial input.

6. The zero-sum game self-triggered disturbance-resistant reinforcement learning method for unmanned surface vessels according to claim 1, characterized in that, S5 constructs the evaluation error for the current moment based on the evaluation-execution-perturbation network at the current moment, constructs the evaluation error for historical moments based on the experience replay mechanism, constructs the objective function of the evaluation network based on the evaluation error at the current moment and the evaluation error at historical moments, and designs the weight update rate of the evaluation network, including: S51. Based on the evaluation network, regression factor and instantaneous cost, construct the evaluation error at the current moment. The regression factor is constructed from the gradient of the augmented state of the unmanned surface vessel with respect to the basis function, the true state, the dynamic equation of the unmanned surface vessel tracking, the sampled state and the basis function. S52. Select historical moments based on the experience playback mechanism, and construct the evaluation error of historical moments based on the evaluation error of the current moment. S53. Based on the evaluation error at the current moment, the evaluation error at historical moments, and the regression factor, construct the objective function of the evaluation network; S54. Based on the objective function of the evaluation network and the gradient descent method, design the weight update rate of the evaluation network.

7. The zero-sum game self-triggered disturbance-resistant reinforcement learning method for unmanned surface vessels according to claim 1, characterized in that, S6, with reference to the policy structure induced by the evaluation network, constructs the learning errors at the current time and at historical time for both the execution network and the perturbation network, and constructs the objective functions for the execution network and the perturbation network, including: S61. The optimal control strategy and the optimal disturbance strategy are obtained by the evaluation network; S62. Based on the difference between the execution network and the optimal control strategy, construct the learning error of the execution network at the current moment; based on the difference between the perturbation network and the optimal perturbation strategy, construct the learning error of the perturbation network at the current moment. S63. Select historical moments based on the experience playback mechanism, construct the learning error of the historical moments of the execution network based on the learning error of the current moment of the execution network, and construct the learning error of the historical moments of the perturbation network based on the learning error of the current moment of the perturbation network. S64. Based on the learning error of the execution network at the current moment and the learning error at the historical moment, construct the objective function of the execution network. Based on the learning error of the perturbation network at the current moment and the learning error at the historical moment, construct the objective function of the perturbation network.

8. A zero-sum game self-triggered disturbance-resistant reinforcement learning device for unmanned surface vessels (USVs), wherein the zero-sum game self-triggered disturbance-resistant reinforcement learning device for USVs is used to implement the zero-sum game self-triggered disturbance-resistant reinforcement learning method for USVs as described in any one of claims 1-7, characterized in that, The device includes: The state space construction unit is used to construct the kinematic model and dynamic model of the unmanned surface vessel (USV) based on its parameters and velocity parameters, and to construct the state space equation of the USV under the input affine form based on the kinematic model and dynamic model. An augmentation unit is used to define the control input and disturbance input as opposing players in a game, construct the augmented state of the unmanned surface vessel, construct the zero-sum game value function of the unmanned surface vessel, construct the game objective, construct the Hamilton-Jacobi-Isaks equation, and obtain the Nash feedback strategy of the two players. The sampling unit is used to design a self-triggering sampling mechanism based on the state space equation and Nash feedback strategy, and to calculate the next triggering time at the current triggering time using the unmanned surface vessel state and learning parameters, so as to realize discrete triggering update. The network construction unit is used to construct the evaluation network, execution network, and perturbation network, which are used to approximate the zero-sum game value function, control strategy, and perturbation strategy of the unmanned surface vessel, respectively. The first update rate construction unit is used to construct the current time evaluation error based on the current time evaluation-execution-perturbation network, construct the historical time evaluation error based on the experience replay mechanism, construct the evaluation network objective function based on the current time evaluation error and the historical time evaluation error, and design the weight update rate of the evaluation network. The objective function construction unit is used to construct the learning error at the current time and the learning error at the historical time for the execution network and the perturbation network, respectively, with reference to the policy structure induced by the evaluation network; and to construct the objective function for the execution network and the objective function for the perturbation network. The second update rate construction unit is used to construct a discrete-time projection operator applied to the weights of the execution network, and to construct the weight update rate of the execution network in combination with the objective function of the execution network; and to construct a continuous-time projection operator applied to the weights of the perturbation network, and to construct the weight update rate of the perturbation network in combination with the objective function of the perturbation network. The output unit is used to output the approximate Nash equilibrium strategy corresponding to the execution network and the perturbation network after the network weights converge or reach the preset performance index, for online autonomous navigation control of unmanned surface vessels.

9. A self-triggered, disturbance-resistant reinforcement learning device for zero-sum game theory for unmanned surface vessels, characterized in that, The zero-sum game self-triggered anti-disturbance reinforcement learning device for unmanned surface vessels includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images