Unmanned surface vessel game confrontation control method and device based on LW-PPO, program and storage medium

Abstract

The invention relates to an unmanned surface vessel game confrontation control method and device based on LW-PPO, a program and a storage medium, and belongs to the technical field of USV intelligent control and deep reinforcement learning. The method comprises the following steps: performing Markov game process modeling on a game environment and initializing a role state; acquiring a previous moment hidden state vector and a current moment state vector, and inputting the current moment hidden state vector and the current moment state vector into a liquid neural network for time sequence feature extraction to obtain a current moment hidden state vector; inputting the vector into a strategy network to generate a selected action; the selected action is converted into acceleration and steering rate control quantity through a PID controller; performing kinematics updating according to the control quantity and the current state to obtain an updated state vector; interacting with the environment to judge whether to end, and if not, circularly executing; liquid neural network explicit modeling USV continuous time dynamics is introduced, and the time sequence feature extraction capability is enhanced; and traditional KL divergence constraint is replaced by a smooth regular term, so that the strategy updating stability is improved.

Description

Technical Field

[0001] This invention relates to the field of USV intelligent control and deep reinforcement learning technology, and in particular to a game-based adversarial control method, device, program and storage medium for unmanned surface vessels based on LW-PPO. This method is applicable to the autonomous decision-making and adversarial control of USVs in maritime scenarios such as maritime information perception and game-based adversarial activities. Background Technology

[0002] With the increasing scale and complexity of marine activities, unmanned surface vessels (USVs) are playing a vital role in tasks such as marine monitoring, emergency rescue, and port inspection. When performing these missions, USVs must make autonomous navigation, avoidance, and intervention decisions under complex and ever-changing marine environments and potential adversarial risks, placing higher demands on the real-time performance, stability, and robustness of their intelligent control algorithms.

[0003] Traditional USV control methods largely rely on manually designed rules and classic controllers such as PID controllers to achieve path tracking and obstacle avoidance. These methods have advantages in terms of simple structure and ease of implementation, but they are difficult to deal with complex game-theoretic scenarios in high-dimensional state spaces, especially when facing opponents with competitive strategies, lacking adaptability and strategy evolution capabilities.

[0004] The development of Deep Reinforcement Learning (DRL) has provided a new technological paradigm for autonomous decision-making in USV (Underworld Vehicle) games. By training agents through extensive interactions in a simulated environment, DRL algorithms can automatically learn near-optimal game strategies. However, directly applying DRL to USV adversarial games still presents the following challenges:

[0005] (1) Environmental non-stationarity: In the game adversarial environment, the opponent's strategy evolves continuously with training. From the perspective of a single agent, the environment transition probability is dynamic. The KL divergence or importance sampling constraint that traditional PPO relies on is prone to failure when the policy distribution support set is offset or even non-overlapping, leading to policy update oscillation or even divergence.

[0006] (2) Insufficient time-series dynamic modeling: The evolution of USV heading and velocity has continuous time characteristics. The traditional Actor-Critic structure using Multilayer Perceptron (MLP) is essentially still a static mapping, which cannot explicitly model time-series dependence and physical response, and is difficult to accurately characterize complex maneuvering behavior under delayed feedback.

[0007] (3) Sparse rewards and slow policy convergence: In complex game adversarial scenarios, strong feedback is only obtained when endgame events such as defeating or being defeated occur. Simple endgame reward design makes it difficult to guide agents to learn phased approach, avoidance and intervention strategies.

[0008] Therefore, there is an urgent need for a USV game adversarial control method that combines continuous-time dynamics modeling capabilities, policy update stability, and curriculum reward design to improve the convergence speed, training stability, and robustness of agents in non-stationary adversarial environments. Summary of the Invention

[0009] The purpose of this invention is to address the problems of insufficient temporal modeling ability, unstable policy updates, and difficulty in fast convergence of existing deep reinforcement learning methods for USV in highly adversarial and multi-stage game tasks. Combining LNN and Wasserstein distance, this invention proposes an unmanned surface vessel game adversarial control method, device, program, and storage medium based on LW-PPO, and designs a USV game adversarial training environment and course reward mechanism to improve the autonomous game adversarial ability of USV in complex marine environments.

[0010] This invention proposes a game-theoretic control method, device, program, and storage medium for unmanned surface vessels based on LW-PPO, the core of which includes the following technical solutions:

[0011] A game-theoretic control method for unmanned surface vessels based on LW-PPO includes the following steps:

[0012] Step 1: Model the Markov game process based on the game environment; set the hidden state of each role and initialize it.

[0013] Step 2: Obtain the hidden state vector of the previous time step containing the states of all game players and the current time step state vector containing the velocity, position coordinates and heading angle of all game players at the current time step, and input them into the liquid neural network for temporal feature extraction to obtain the hidden state vector of the current time step.

[0014] Step 3: Introduce a policy network and input the current hidden state vector into the policy network to obtain the selected action.

[0015] Step 4: Based on the selected action, convert it through the controller to obtain the action control quantity tuple.

[0016] Step 5: Based on the action control tuple and the current state vector, perform kinematic updates to obtain the updated state vector.

[0017] Step 6: Interact the updated state vector with the environment to determine if the game is over; if so, output the game result; otherwise, use the updated state vector as the state vector for the next moment and return to step 2.

[0018] Furthermore, the hidden state vector at the current time step 2 The specific calculation method includes: calculating the hidden state vector at the current moment based on the continuous-time differential equation of the liquid neural network and discretizing it;

[0019]

[0020] in, Let be the hidden state vector from the previous time step. For differential calculations, The learnable time decay constant, This is the state vector at the current moment.

[0021] Further, step 3 specifically includes the policy network generating action probabilities based on the temporal features extracted by the liquid neural network, and selecting the action with the highest probability as the selected action.

[0022] Furthermore, the method by which the controller converts the selected action in step 4 specifically includes:

[0023]

[0024] in, For motion control quantity, This is a dynamic control variable element. For acceleration control, For steering rate control quantity, This is the error value. For the error integral term, For the error differential term, These are the proportional parameter, integral parameter, and differential gain parameter, respectively.

[0025] Furthermore, the kinematic update described in step 5 specifically includes velocity update, position coordinate update, and heading angle update;

[0026]

[0027]

[0028]

[0029]

[0030] in, For the updated speed, The speed at the current moment, For the updated heading angle, The heading angle at the current moment. For modulo operation, The updated position coordinates, The coordinates are the current position.

[0031] The updated velocity, position coordinates, and heading angle are combined to form the updated state vector.

[0032] A computer device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0033] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0034] A computer program product includes computer instructions that, when executed by a processor, implement the steps of the method described above.

[0035] The beneficial effects of this invention are as follows: By introducing a liquid neural network, this invention explicitly models the time decay and historical information dependence during USV maneuvers, overcoming the limitations of traditional MLPs in processing continuous-time dynamic systems; by using Wasserstein distance to replace the simple KL divergence constraint, it can still obtain a continuous and bounded difference measure even when the support sets of the old and new policies are severely offset, alleviating policy oscillation and divergence problems in adversarial environments and improving the stability of policy updates; through the course reward design combined with the synergistic effect of LNN and Wasserstein, the policy can form a stable approach-avoidance-intervention multi-stage policy within a finite number of interaction rounds, which converges faster and fluctuates less than the baseline PPO, improving convergence speed and sample efficiency; in obstacle-containing environments, the LW-PPO policy can automatically learn the balance between obstacle avoidance paths and intervention decisions, avoiding collision endpoints while maintaining a high win rate, adapting to obstacles such as reefs and buoys in real waters, and thus having better generalization ability for obstacles and complex scenarios. Attached Figure Description

[0036] Figure 1 This is a flowchart of the method of the present invention.

[0037] Figure 2 The Actor-Critic network structure diagram of LNN is introduced for this invention.

[0038] Figure 3 This is a structural diagram of the LW-PPO algorithm in this invention.

[0039] Figure 4 The results show the comparative experiments of the LW-PPO and PPO algorithms in this embodiment of the invention.

[0040] Figure 5 This is a comparison chart of the ablation experimental results of LW-PPO, LNN-PPO, Wasserstein-PPO and PPO in the embodiments of the present invention.

[0041] Figure 6 This is a simulation result diagram of an adversarial game environment in an embodiment of the present invention.

[0042] Figure 7 This is a simulation result diagram of an obstacle-based game adversarial environment in an embodiment of the present invention. Detailed Implementation

[0043] refer to Figure 1 A game-based adversarial control method for unmanned surface vessels based on LW-PPO includes the following steps:

[0044] Step 1: Model the Markov game process based on the game environment; set the hidden state of each role and initialize it.

[0045] Step 2: Obtain the hidden state vector of the previous time step containing the states of all game players and the current time step state vector containing the velocity, position coordinates and heading angle of all game players at the current time step, and input them into the liquid neural network for temporal feature extraction to obtain the hidden state vector of the current time step.

[0046] The hidden state vector at the current moment The specific calculation method includes: calculating the hidden state vector at the current moment based on the continuous-time differential equation of the liquid neural network and discretizing it;

[0047]

[0048] in, Let be the hidden state vector from the previous time step. For differential calculations, The learnable time decay constant, This is the state vector at the current moment.

[0049] Step 3: Introduce a policy network and input the current hidden state vector into the policy network to obtain the selected action.

[0050] Step 3 specifically includes the policy network generating action probabilities based on the temporal features extracted by the liquid neural network, and selecting the action with the highest probability as the selected action.

[0051] Step 4: Based on the selected action, convert it through the controller to obtain the action control quantity tuple.

[0052] The method by which the controller converts a selected action specifically includes:

[0053]

[0054] in, For motion control quantity, This is a dynamic control variable element. For acceleration control, For steering rate control quantity, This is the error value. For the error integral term, For the error differential term, These are the proportional parameter, integral parameter, and differential gain parameter, respectively.

[0055] Step 5: Based on the action control tuple and the current state vector, perform kinematic updates to obtain the updated state vector.

[0056] The kinematic update specifically includes velocity update, position coordinate update, and heading angle update;

[0057]

[0058]

[0059]

[0060]

[0061] in, For the updated speed, The speed at the current moment, For the updated heading angle, The heading angle at the current moment. For modulo operation, The updated position coordinates, The coordinates of the current position at that moment;

[0062] The updated velocity, position coordinates, and heading angle are combined to form the updated state vector.

[0063] Step 6: Interact the updated state vector with the environment to determine if the game is over; if so, output the game result; otherwise, use the updated state vector as the state vector for the next moment and return to step 2.

[0064] Example

[0065] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The present invention provides a method, device, program, and storage medium for game-based adversarial control of unmanned surface vessels based on LW-PPO, such as... Figure 1 As shown, the method includes the following steps:

[0066] (1) Construction of the game-theoretic adversarial framework

[0067] Markov Decision Processes (MDPs) are a crucial concept in reinforcement learning. When using reinforcement learning to solve real-world problems, the problem must first be abstracted into an MDP. An MDP consists of five tuples. Composition. In this quintuple, It is the set of all environmental states. A set of actions that a USV can perform to actively control a character. It is a state transition function, representing the state transition. Execute action Then transition to the next state The probability, The reward function represents the USV of the actively controlling character in the state. Execute action The instant reward obtained afterward It is a discount factor, with a value between [0, 1], used to measure the importance of future rewards.

[0068] The decision-making problem of USVs in adversarial marine environments involves USVs with active control roles acting simultaneously and competing with each other in a shared environment, with conflicting objectives. This complex interaction process is best modeled formally using Markov Games (MG), also known as Stochastic Games. Markov Games are an extension of standard MDPs to adversarial USV game scenarios.

[0069] In a USV (Underworld) game adversarial environment, the strategy of the active USV player evolves continuously throughout the game. This means that from the perspective of the active USV player, the dynamic characteristics of the environment (i.e., state transition probabilities) are constantly changing. The non-stationarity of the target is the main reason why traditional DRL algorithms struggle to converge or generalize in USV adversarial scenarios where the active control role is involved. Therefore, adopting the MG framework and designing a stabilization mechanism to address its non-stationarity is crucial for achieving robust strategies.

[0070] The entire USV game process can be formalized into a six-tuple. ,in It is the state space of the game against the environment. It is the motion space of the USV. It is a state transition function. It is a reward function used to measure the reward obtained by the USV (Underworld Vehicle) that actively controls the character and performs actions. It is a discount factor.

[0071] To effectively train and evaluate USV game adversarial strategies, this study customized a training environment based on the OpenAI Gym framework. This environment follows a standard reinforcement learning interface, encapsulating USV dynamics, adversarial interactions, and obstacle-containing scenarios, ensuring the standardization, repeatability, and compatibility of the training process with existing DRL algorithm libraries.

[0072] This environment is a discrete practical environment in two-dimensional space simulating USV (Unmanned Vehicle) game-playing adversarial scenarios in a real ocean environment. It is used to test and verify the performance of USVs with active control roles when making behavioral decisions using reinforcement learning algorithms. Most traditional environments only consider the simple movement and route planning of unmanned vessels. However, in a real ocean environment, USVs must consider not only their own movement but also factors such as approach-avoidance strategies, intervention angles, and intervention success rates. Therefore, this custom game-playing adversarial environment optimizes traditional USV adversarial scenarios by integrating dynamic intervention judgment, multi-dimensional resource management, and a rule-based decision-making model for USVs with random control roles, thus constructing a game scenario with high policy complexity. The environment adopts a discrete action space and a refined multi-objective reward mechanism, supporting USV game-playing adversarial scenarios and dynamic policy optimization. It provides a scalable framework for USV approach-avoidance policy learning by active control roles and collaborative research on USVs with multiple active control roles.

[0073] During environment initialization, a 100×100 two-dimensional grid map is generated. The coordinates of the actively controlled character's USV are strictly constrained within the map boundaries to prevent logical anomalies caused by out-of-bounds behavior. To improve the algorithm's generalization ability to different scenes, the initial positions of the actively controlled character's USV are randomly distributed in the lower area of ​​the map. The heading angle is fixed at 90°, which is due north. The initial position of the USV controlled by the random character is randomly distributed at the top of the map. The initial heading angle is 270°, which is due south. All characters' USVs are set with the same initial attribute parameters, including health, battery capacity, maximum intervention distance, maximum intervention angle range, and basic intervention success rate. All characters' USVs are in a transparent state.

[0074] In each round, both actively controlled USVs and randomly controlled USVs make action decisions based on current situational information, engaging in a game with the ultimate goal of defeating the other USV. The heading and speed control of all USVs is implemented using a unified PID controller. A map boundary is set in the environment; USVs that move too close to the map boundary will incur penalties. Furthermore, USVs have energy and health attributes. Energy is consumed through movement and intervention; when energy is depleted or health reaches 0, it indicates that one USV has been defeated or stopped, and the other side wins. The initial environment concept diagram is shown in the figure.

[0075] The USV of randomly controlled characters implements the basic game strategy through a decision tree: when the distance between the USV of all characters in the game is less than a threshold, a large-angle turn is randomly selected to avoid intervention; otherwise, the characters continue to approach the target at full speed and prepare to intervene, simulating an alternating approach and avoidance strategy.

[0076] The observation space is designed as a 6-dimensional continuous vector, containing the real-time position and heading angle information of all roles' USVs. Intervention determination employs a distance-angle dual-constraint model: when the randomly controlled role's USV is within the intervention range of the actively controlled role's USV (i.e., distance ≤ 10 units and heading angle deviation ≤ 30°), the intervention success rate is dynamically calculated as follows:

[0077]

[0078] in To determine the real-time USV distance for all roles in the game, when This means that when all the USVs of the game players are 5 meters apart, the success rate of intervention reaches 100%, simulating the advantage of close-range intervention. Successful intervention triggers the endgame decision for the randomly controlled USV.

[0079] To verify the generalization ability of the LW-PPO algorithm, this paper designs an adversarial game environment with obstacles in the USV (Underworld Virtualization) framework. The spatial constraints introduced by the obstacles mean that the game process is no longer just a local optimization of distance or orientation, but requires a trade-off between path feasibility and the objective, which increases the requirements for strategy expression and planning capabilities.

[0080] In obstacle-prone scenarios, USVs actively controlling a character need to balance short-term action selection with long-term trajectory planning, such as obstacle avoidance paths, which helps generate more robust navigation strategies. Real-world waters often contain impassable areas such as reefs, islands, and buoys. Therefore, this experiment inserted an obstacle handling module into the original environment modeling layer and environment interaction logic layer. The specific implementation is as follows:

[0081] A square obstacle zone, a 10x10 grid block, is inserted in the center of the map to represent the coral reef area on the sea surface. The USV, which is actively controlled by the character, cannot directly "perceive" the obstacle distribution and must learn obstacle avoidance strategies through trial and error. When a collision is detected, the environment will reduce the collision character's health to 0, thus immediately triggering the endgame judgment.

[0082] The USV is modeled as an actively controlled USV with a multi-dimensional state space existing in a two-dimensional space. Its core attributes are divided into mobility attributes, survivability attributes, and intervention attributes, detailed in Table 1:

[0083]

[0084] Based on the USV attributes in the table, we will now introduce the USV state space, action space, and reward function in detail:

[0085] ①State space:

[0086] The state space describes how the USV perceives its current environment. The design of the USV's state space is crucial to the performance of deep reinforcement learning algorithms.

[0087] In this environment, the state space of all roles' USVs can be represented by six tuples. The state space represents the USV (United States Value) of all players in the game within the map. coordinates and and heading angle This status information can be used for USV environmental perception, path planning, and decision-making.

[0088] ②Motion space:

[0089] A PID controller (Proportional-Integral-Derivative controller) is a feedback-based closed-loop control algorithm that adjusts the control input in real time to make the system output track the desired target value. Its name comes from its three core components:

[0090] Proportional (P): Adjusts the control quantity proportionally based on the current error (the difference between the target value and the actual value).

[0091] Integral (I): Eliminates steady-state error by accumulating historical errors.

[0092] Derivative (D): Predicts future error trends and suppresses system oscillations.

[0093] The output formula of the PID controller is:

[0094]

[0095] in, It controls the output. It is the error between the set value and the actual measured value. , and These are the proportional, integral, and differential coefficients, respectively.

[0096] In a USV, the PID controller is mainly used for two core tasks: speed control and heading angle control. The former aims to control the actual speed of the USV. Tracking the expected speed The latter adjusts the USV's heading angle. To match the target heading angle Taking heading angle control as an example, in the proportional gain process, once a deviation occurs between the actual heading angle and the desired heading angle, the heading angle controller immediately generates a heading angle control signal to reduce the deviation. The integral part eliminates steady-state error by accumulating historical errors. The derivative part introduces a heading angle correction signal by predicting future error trends to suppress heading angle oscillations.

[0097] The movement of the USV is primarily determined by the control output, which is a tuple containing acceleration and steering torque (or angle change). Specifically, the first element of the control output is the speed control output, i.e., acceleration. The speed ratio ranges from 0 to 1, representing the proportion of speed. The second element is the steering angle change. After adjusting the speed ratio, the PID controller adjusts the acceleration to the target speed. The speed update formula is:

[0098]

[0099] in The acceleration output by the PID control. For speed.

[0100] After adjusting the steering angle, the formula for updating the heading angle using PID control to adjust the steering rate is as follows:

[0101]

[0102] This refers to the steering rate output by the PID control. A small steering angle of 15 degrees is suitable for fine-tuning the heading, while a large steering angle of 45 degrees is used for rapid avoidance or approach. After updating the speed and heading angles, the USV position coordinate update formula is as follows:

[0103]

[0104]

[0105] Based on this, the PID controller calculates the control output according to the error between the control input and the actual measured value, so as to achieve precise control of the USV, making the speed and heading angle of the USV tend to the set value, and finally translating into different actions of the USV, such as going straight, turning left, and turning right. The USV's action space consists of 9 discrete actions. To prevent the USV from turning indiscriminately when selecting an action, different gears are used to increase the probability of going straight. The action space design is shown in Table 2:

[0106]

[0107] ③ Reward function

[0108] The final reward is directly related to the outcome of the game. A winner in the actively controlled character's USV receives a huge reward, while a winner in the randomly controlled character's USV incurs a huge penalty. If all characters' USVs are defeated simultaneously, there is no additional reward. In the design of the reward function for the reinforcement learning algorithm, this study adopts an improved course learning approach, guiding the actively controlled character's USV to learn game strategies in stages: long-distance approach, policy distance avoidance (policy distance set to twice the maximum intervention distance), and close-range precise intervention. The reward function achieves a natural transition between the three-stage strategies by dynamically adjusting the multi-objective weights. The final reward function design is as follows:

[0109] Long-distance approach phase (USV distance for all roles > strategy distance): When the USV distance for all roles in the game is greater than the strategy distance, a long-distance approach reward is applied to the USV of the actively controlled role. This reward emphasizes spatial proximity and heading pre-alignment, using historical distance changes and angle factors to correct the USV orientation of the actively controlled role in the long-distance phase. The dynamic long-distance approach reward encourages the USV of the actively controlled role to quickly approach the USV of the randomly controlled role and enter the strategy distance in a decaying form. At the same time, an angle factor is applied to guide the heading to gradually align with the USV of the randomly controlled role through a decay function. This phase prevents ineffective wandering by applying an additional time penalty term, forming a behavior pattern of rapid USV approach.

[0110] Strategic Distance Avoidance Phase (Maximum Intervention Distance < Distance between the Active and Randomly Controlled USVs < Strategic Distance): When the distance between the Active and Randomly Controlled USVs lies between the maximum intervention distance and the strategic distance, the Active's USV needs to dynamically balance approach and avoidance. During this phase, the Active's USV receives an angle advantage reward, a risk situation penalty, and an avoidance maneuver reward. The reward mechanism prioritizes survival and prepares for intervention by adjusting the heading angle within the intervention distance. The angle advantage is designed as follows: if the relative angle deviation between the Active's USV heading and the Randomly Controlled USV is less than the intervention angle, the reward increases linearly as the angle deviation decreases. A distance decay factor is introduced, with higher rewards for closer proximity. The risk situation penalty is designed as follows: if the Randomly Controlled USV's intervention angle already covers the Active's USV, a dynamic penalty is applied based on the relative angle deviation between the Randomly Controlled USV's heading and the Active's USV to prevent successful intervention by the Randomly Controlled USV. The specific design of the evasion maneuver reward is as follows: if the angular deviation of the randomly controlled USV relative to the actively controlled USV exceeds its intervention range, a positive reward is given to incentivize the actively controlled USV to quickly escape the potential intervention area. Simultaneously, a boundary penalty is applied to limit the USV's range of motion and avoid the risk of hitting the edge. During this phase, the reward function trains the USV to form a game strategy that coordinates intervention and evasion, avoiding a passive stance and preparing for the close-range intervention phase.

[0111] Close-range precision intervention phase (distance between the active and random USVs ≤ maximum intervention distance): When the distance between the active and random USVs is less than or equal to the maximum intervention distance, the active USV must quickly complete precision intervention. This phase primarily encourages the active USV to quickly terminate the random USV by enhancing the angle advantage reward from the strategy distance avoidance phase, while retaining risk situation penalties and avoidance maneuver rewards to prevent the active USV from becoming passive. Furthermore, the final reward is both the ultimate feedback on victory or defeat and the core driving force of the close-range precision intervention phase. High victory or defeat rewards strongly incentivize the active USV to adopt unreliable strategies, prioritizing heading adjustments to maximize the probability of successful intervention, even temporarily ignoring risk penalties or boundary risks, thus preventing the USV from falling into a dilemma between intervention and avoidance. By balancing and coordinating the final reward, high-precision angle advantage reward, and risk penalty mechanism, the USV is trained to maximize intervention efficiency and minimize its own risk. Specific reward function settings are shown in Table 3.

[0112]

[0113] Experiments show that this hierarchical reward structure enables the USV of the actively controlled character to master basic approach strategies in the early stages of training, gradually develop dynamic avoidance capabilities in the middle stages, and finally converge to precise intervention strategies in the later stages, verifying the effectiveness of the course learning concept in the design of the reward function.

[0114] (2) Design of Actor-Critic structure based on liquid neural network

[0115] LNN is a neural network architecture with continuous-time recursive properties. Its core advantage lies in its mathematical foundation based on continuous-time dynamical systems and Ordinary Differential Equations (ODEs). Unlike traditional MLPs and RNNs, which update hidden states using discrete iterative methods, LNNs describe the evolution of the internal state of neurons as a continuous-time differential process. The mathematical model of LNNs is typically based on differential equations of the following form:

[0116]

[0117] in, This represents the state of a neuron over time, i.e., the hidden state; It is a decay constant that controls the speed at which neurons respond to external inputs and the rate at which they forget historical information; It is a function that represents the external input pair. The influence of neuronal state.

[0118] LNNs memorize and decay past information in a more compact, parameterized form through their internal "liquid neurons." This differential equation-driven framework makes LNNs more robust, enabling them to dynamically adjust their internal states and effectively capture complex temporal patterns.

[0119] To address the shortcomings of traditional PPO algorithms in USV game adversarial scenarios, such as insufficient temporal modeling capabilities and limited policy stability in non-stationary environments, this paper proposes the LW-PPO algorithm. Its core lies in enhancing the algorithm's adaptability and robustness in adversarial game environments by strengthening temporal dynamics modeling capabilities and optimizing policy update constraint mechanisms.

[0120] In the baseline experiment, the Actor-Critic network used by the PPO algorithm was implemented using an MLP. The policy network Actor and the value network Critic were activated by two fully connected hidden layers of size 64 and the Tanh activation function, respectively, to extract state features.

[0121] state After being input into the Actor network, the features are first linearly transformed to 64 dimensions, then activated by the Tanh activation function; subsequently, they are linearly transformed again to 64 dimensions and the Tanh activation function is applied; finally, the features are mapped to the action space dimension and normalized using Softmax, outputting a set of action selection probabilities. The Critic value network also uses states... As input, state features are extracted through two consecutive linear transformations, and finally mapped to 1D through an activation function, directly outputting the value estimate of the corresponding state. .

[0122] In this experiment, LNN is used to model the evolution of the hidden state of the neural network as a continuous-time dynamic system with a learning time constant. The core idea can be expressed in the following continuous-time form:

[0123]

[0124] in The hidden state vector, It is a learnable time decay constant, and It is a non-linear activation function tanh(). In implementation, Euler discretization is commonly used to obtain the following formula:

[0125]

[0126] The actions performed by USVs in adversarial situations, such as adjustments to course and speed, are physical processes that evolve continuously over time. Traditional PPO algorithms, using MLP architectures, cannot effectively capture the temporal relationships between states. LNN's ODE structure, however, can explicitly model the continuous amplitude changes in temporal information and system response. This capability significantly improves the model's fitting accuracy to the continuous maneuvers and nonlinear interactions of USVs, especially in scenarios with delayed feedback or low sampling rates, where LNNs still maintain sensitivity to and memory of system dynamics.

[0127] In summary, to address the challenges of modeling strong temporal dependencies and continuous dynamics in USV game adversarial systems, this paper introduces LNNs as core temporal modeling units into the policy and value networks of the PPO, replacing the input layer of the traditional multilayer perceptron (MLP). The specific network architecture is as follows: Figure 2 .

[0128] LiquidLayer takes the state dimension of the USV as input and outputs 64-dimensional features. It utilizes a recursive mechanism of "liquid units" to model the temporal dependencies of the input state sequence, then passes these temporally rich features to subsequent fully connected layers to output policy distributions or value estimates. By embedding LNNs into the network's lower layers, the model can explicitly capture the time decay characteristics of USV-environment interactions, improving the fitting accuracy for delayed responses, continuous maneuvers, and nonlinear interactions between USVs with active control roles. This provides more accurate temporal feature support for policy decision-making in complex game scenarios.

[0129] (3) Introduce Wasserstein distance as a policy smoothing regularization term.

[0130] In the non-stationary environment of USV game adversarial competition, the evolution of the USV strategy of the active player leads to new strategies. and old strategies The distribution frequently shifts. This paper introduces WD as a policy smoothing regularization term into the pruning objective function of PPO and addresses the limitations of the traditional PPO algorithm in the following way:

[0131] Overcoming KL divergence failure: In adversarial scenarios, PPO updates themselves may cause the action distribution of the old and new policies to shift in critical states. In this case, KL divergence as a metric will result in numerical divergence or gradient anomalies. By using WD as the policy smoothing regularization term, WD can provide a continuous and meaningful measure of difference through "minimum transport cost," thereby suppressing the resulting policy oscillations at the optimization level and improving training stability and convergence efficiency under rule constraints and strong temporal dynamics.

[0132] Providing smooth constraints: WD provides a continuous and interpretable penalty signal for policy changes. This restricts policy jumps across large scales. Even with abrupt changes in the environment, WD can still calculate the cost of policy adjustment and provide smooth gradients, significantly improving the training stability and robustness of the algorithm in dynamic and non-stationary environments.

[0133] The specific implementation mechanism is as follows: In a discrete action space, accurately calculating high-dimensional WD is computationally expensive. This study calculates the new strategy by combining the old and new strategies. and old strategies In the set of states The Wasserstein distance is approximated by the L1 distance of the cumulative distribution function (CDF) of the action probability distributions on the given surface. In one-dimensional space, the first-order Wasserstein distance is exactly equal to the area between two empirical CDF curves.

[0134] To suppress sample noise fluctuations during training, the calculated... Using the exponential moving average (EMA) mechanism and historical Wasserstein distance Perform blending and smoothing.

[0135]

[0136] in, These are the mixing coefficients. The mixed Wasserstein distance regularization term is incorporated into the total loss function of PPO. This provides a soft constraint on the policy update magnitude while preserving the original importance sampling and pruning mechanism of PPO.

[0137]

[0138] in, It is a coefficient used to balance value loss and Wasserstein regularization.

[0139] In summary, this study proposes a game-theoretic adversarial control method for unmanned surface vessels based on the LW-PPO algorithm. The specific work is as follows:

[0140] ① By embedding LNN as a temporal modeling unit in the Actor-Critic framework of PPO, the limitations of traditional MLP in handling continuous-time dynamics and strongly temporally correlated game scenarios are overcome, and the model's ability to capture temporal evolution information is enhanced.

[0141] ② Introduce a policy smoothing regularization term based on Wasserstein distance into the target loss function of PPO to provide a continuous and interpretable penalty signal for policy updates, thereby improving training stability and robustness in adversarial non-stationary environments.

[0142] LW-PPO improves learning efficiency and dynamic fitting ability through LNN, while utilizing the Wasserstein distance-stabilized update policy to achieve adaptability and reliability in the complex adversarial environment of USV. The flowchart of the LW-PPO algorithm is shown below. Figure 3 .

[0143] (4) Training and implementation of USV game adversarial strategies

[0144] Using a custom adversarial game environment, the USV of the active player was trained using the improved LW-PPO algorithm and compared with the basic PPO algorithm to verify the performance improvement of the LW-PPO algorithm on various evaluation metrics and to validate the algorithm's performance. Subsequently, a series of ablation experiments were conducted to verify the main innovations. Based on PPO and its improved method LW-PPO, reward value, number of rounds, policy entropy, adversarial win rate, and total loss value were proposed as evaluation metrics for the experiments.

[0145] First, a comparative experiment was conducted between the LW-PPO algorithm (which incorporates all optimization methods) and the PPO algorithm to verify the effectiveness of the LW-PPO algorithm. We first verified the reliability of the custom game adversarial environment and strategy design by analyzing the round reward changes during training in a custom USV adversarial environment using the PPO algorithm. Then, we trained the LW-PPO algorithm in an obstacle-based adversarial environment to verify its generalization ability and compared the training results with those of the LW-PPO algorithm in the original environment. The experimental results are as follows: Figure 4 .

[0146] The ablation experiments were conducted on LNN-PPO, which only incorporates LNN as the temporal modeling unit, and Wasserstein-PPO, which only incorporates Wasserstein distance. These experiments illustrate the optimization method proposed in this paper for the PPO algorithm, and analyze in depth the independent effectiveness and synergistic effect of the two innovative mechanisms. The experimental results are as follows: Figure 5 .

[0147] Comparative experiments with the PPO algorithm demonstrate that LW-PPO significantly outperforms the baseline PPO in terms of convergence speed, policy efficiency, and training stability. Furthermore, the ablation experiments used LNN-PPO (which only incorporates LNNs as temporal modeling units) and Wasserstein-PPO (which only incorporates Wasserstein distance) to further demonstrate the independent effectiveness and synergistic effect of the two innovative mechanisms.

[0148] In the simulation platform, the LW-PPO strategy receives the observed state from the simulation environment through the decision execution module, outputs discrete action commands for the USV, and converts them into low-level control variables via the PID control module to drive the USV to achieve autonomous maneuvering and game-theoretic confrontation. Experiments show that the method of this invention can effectively complete autonomous approach, avoidance, and intervention decisions for USVs with random maneuvering roles in complex marine environments, demonstrating good engineering feasibility and application value. Specific simulation results are as follows: Figure 6 , Figure 7 .

[0149] In summary, this invention proposes an adversarial control method for unmanned surface vessels based on the LW-PPO algorithm by constructing a USV game-theoretic training environment, introducing a liquid neural network to model continuous-time dynamics, and adding Wasserstein distance to PPO for policy smoothing. This method achieves efficient learning and stable convergence of USV game-theoretic strategies in adversarial non-stationary environments and can be widely applied to various USV application scenarios such as maritime information perception, game-theoretic confrontation, and maritime safety.

[0150] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A game-theoretic control method for unmanned surface vessels based on LW-PPO, characterized in that, Includes the following steps: Step 1: Model the Markov game process based on the game environment; set the hidden state of each role and initialize it; Step 2: Obtain the hidden state vector of the previous time step containing the states of all game players and the current time step state vector containing the velocity, position coordinates and heading angle of all game players at the current time step, and input them into the liquid neural network for temporal feature extraction to obtain the hidden state vector at the current time step. Step 3: Introduce a policy network and input the current hidden state vector into the policy network to obtain the selected action; Step 4: Based on the selected action, convert it through the controller to obtain the action control quantity tuple; Step 5: Based on the motion control tuple and the current state vector, perform kinematic updates to obtain the updated state vector; Step 6: Interact the updated state vector with the environment to determine if the game is over; if so, output the game result; otherwise, use the updated state vector as the state vector for the next moment and return to step 2.

2. The game-playing adversarial control method for unmanned surface vessels based on LW-PPO according to claim 1, characterized in that, Step 2 describes the hidden state vector at the current time. The specific calculation method includes: calculating the hidden state vector at the current moment based on the continuous-time differential equation of the liquid neural network and discretizing it; in, Let be the hidden state vector from the previous time step. For differential calculations, The learnable time decay constant, This is the state vector at the current moment.

3. The game-based adversarial control method for unmanned surface vessels based on LW-PPO according to claim 1, characterized in that: Step 3 specifically includes the policy network generating action probabilities based on the temporal features extracted by the liquid neural network, and selecting the action with the highest probability as the selected action.

4. The game-playing adversarial control method for unmanned surface vessels based on LW-PPO according to claim 1, characterized in that, Step 4 describes the method by which the controller converts the selected action. Specifically, this includes: in, For motion control quantity, This is a dynamic control variable group. For acceleration control quantity, For steering rate control quantity, This is the error value. For the error integral term, For the error differential term, These are the proportional parameter, integral parameter, and differential gain parameter, respectively.

5. The LW-PPO-based game-playing adversarial control method for unmanned surface vessels according to claim 4, characterized in that, Step 5, the kinematic update, specifically includes velocity update, position coordinate update, and heading angle update; in, For the updated speed, The speed at the current moment, For the updated heading angle, The heading angle at the current moment. For modulo operation, The updated position coordinates, The coordinates of the current position at that moment; The updated velocity, position coordinates, and heading angle are combined to form the updated state vector.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5.

8. A computer program product comprising computer instructions, characterized in that: When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1 to 5.

Images