Training a pursuer neural network agent

Abstract

The present invention relates to a computer-implemented method of training a pursuer neural network agent. The method comprising: providing an evader neural network agent trained to update iteratively a plurality of dynamic parameters of an evader, a pursuer neural network agent initialised to update iteratively a plurality of approximators for a pursuer, and an ordinary differential equation solver configured to update a plurality of dynamic parameters of the pursuer based on the updated approximators; sampling initial dynamic parameters from a set of initial dynamic parameters of the evader; running a simulation involving the pursuer neural network agent pursuing the evader neural network agent based on the sampled initial dynamic parameters; computing a loss based on the dynamic parameters of the pursuer from the ordinary differential equation solver and the dynamic parameters of the evader from the evader neural network agent; and optimising weights of the pursuer neural network agent based on the computed loss.

Description

[0001] TRAINING A PURSUER NEURAL NETWORK AGENT FIELD

[0001] The subject-matter of the present disclosure relates to computer-implemented methods of training a pursuer neural network agent, transitory or non-transitory computer-readable media, and pursuer vehicles. BACKGROUND

[0002] The need for advanced security measures to protect critical infrastructure and maritime assets has revealed the inadequacies of traditional defense approaches in the marine environment. Integrating artificial intelligence (AI) and autonomous systems into Unmanned Surface Vehicle (USV) operations is essential to enhance the effectiveness and adaptability of maritime security strategies. These technologies significantly improve the strategic capabilities of USVs. AI-aided tactics for infrastructure protection using USVs are a critical application of these technologies, effectively modeled as a pursuit- evasion game where the USV is the pursuer and an intruder vessel is the evader. Although pursuit-evasion strategies are still in their early stages of application for USVs, they have been extensively explored in other domains. Research in this area typically divides methods into two categories: conventional control techniques and machine learning approaches.

[0003] The conventional approach involves applying classical control and optimization theory. One such methodology is the use of the Model Predictive Control (MPC) framework to design a controller. A study in utilized MPC for the USV pursuit and evasion game scenario and similar applications have been made in other robotic platforms. However, implementing the MPC methods in pursuit-evasion games necessitates numerous simplifications, resulting in diminished performance in real-world applications.

[0004] Another method in the first category applies game theory, where the problem is treated as a min-max optimization problem and addressed based on cost definitions. Pursuit-evasion, a subtopic of game theory, is extensively studied across various application areas. For maritime domain, some prior art methods analyze how game theory is applied to various USV tasks. Other prior art methods investigate the pursuit- evasion game problem for USVs based on the threat potential field (TPF). Similarly, game theory methodology has been investigated and applied for other platforms like robotics, unmanned aerial vehicles and unmanned underwater vehicles. Some other studies studied the pursuit-evasion of two aircraft in a horizontal planed and proposed a solution based on extremal trajectory maps. Another study solved the problem through modelling the aircraft control input limits, creating the Hamiltonian equation and solving the optimization problem numerically. Other studies looked into a multiplayer pursuit- evasion game consisting of prey, predator, and protector for collision avoidance problem. Other studies handled the pursuit-evasion problem for two satellites considering their continuous thrust reachable domains. Other multiplayer differential game theory application looked at enabling multiple UAVs to engage in multiple targets. However, analytical solutions in game theory are feasible for simplified cases but challenging to implement in real-time. Deriving these solutions requires many assumptions based on extensive calculus, leading to reduced effectiveness in real-world applications, similar to classical methods

[0005] In pursuit-evasion problems, our study’s second category explores machine learning methodologies. Recent advancements in computational technology enable the training and deployment of machine learning models for real-world applications. Many studies use machine learning, particularly Reinforcement Learning (RL), for pursuit- evasion problems. RL is a trial-and-error-based learning approach where a machine autonomously learns optimal behavior or policies within a specific environment by maximizing or minimizing cumulative rewards or costs. An application of RL on maritime applications is to design a path following controller as the USV dynamics are usually not highly accurate and expose to high level of uncertainties. For the pursuit evasion problem, there have been several applications for training a pursuer, training an evader, co-operative hunting. RL for pursuit evasion game is also widely studied for robotics and UAVs for single / multi agent use cases. One advantage of RL methods is their suitability for fast real-time inference. However, a major limitation of RL methods is their training efficiency,which heavily depends on the design of reward functions, requiring careful reward engineering. Additionally, in continuous systems, the rapid expansion of the state space exploration complicates the search for global optimum solutions, representing a significant challenge in RL-based approaches.

[0006] It is an aim of the present invention to address such problems and improve on the prior art. SUMMARY

[0007] According to an aspect of the present disclosure, there is provided a computer- implemented method of training a pursuer neural network agent, the method comprising: providing an evader neural network agent trained to update iteratively a plurality of dynamic parameters of an evader, a pursuer neural network agent initialised to update iteratively a plurality of approximators for a pursuer, and an ordinary differential equation solver configured to update a plurality of dynamic parameters of the pursuer based on the updated approximators; sampling initial dynamic parameters from a set of initial dynamic parameters of the evader; running a simulation involving the pursuer neural network agent pursuing the evader neural network agent based on the sampled initial dynamic parameters; computing a loss based on the dynamic parameters of the pursuer from the ordinary differential equation solver and the dynamic parameters of the evader from the evader neural network agent; and optimising weights of the pursuer neural network agent based on the computed loss. Initial dynamic parameters may also be called initial conditions.

[0008] Incorporating the ordinary differential equation solver into the loop results in the weights of the pursuer neural network agent being more optimised to pursue an evader performing unseen manoeuvres.

[0009] In an embodiment, the computer-implemented method further comprises: separating the set of initial dynamic parameters of the pursuer into a plurality of mini batches, wherein sampling initial dynamic parameters from a set of initial dynamic parameters of the evader, running the simulation involving the pursuer neural network agent, and computing a loss based on the output from the pursuer neural network agent are performed for each mini batch of the plurality of mini batches of initial dynamic parameters of the pursuer.

[0010] In an embodiment, the computer-implemented method further comprises: computing a mean loss using the loss computed for each mini batch, wherein optimising the weights of the pursuer neural network agent based on the computed loss comprises optimising the weights of the pursuer neural network agent using the mean computed loss.

[0011] In an embodiment, running the simulation comprises: performing forward propagation of the evader neural network agent to update the plurality of dynamic parameters based on the sampled initial dynamic parameters; and performing forward propagation of the pursuer neural network agent to update the plurality of dynamic parameters based on initial dynamic parameters of the pursuer, the updated dynamic parameters from the evader neural network agent.

[0012] In an embodiment, the computer-implemented method further comprises: providing initial dynamic parameters of the pursuer; dividing the initial dynamic parameters of the pursuer into mini batches; running the simulation for each mini batch; and computing the loss for each mini batch.

[0013] In an embodiment, the computer-implemented method further comprises: computing an average loss using the computed loss for all mini batches, wherein optimising the weights comprises optimising the weights of the pursuer neural network agent to minimise the average loss.

[0014] In an embodiment, the dynamic parameters of each of the evader and the pursuer comprise at least one of: position, heading angle, and velocity.

[0015] In an embodiment, computing a loss based on the dynamic parameters of the pursuer from the ordinary differential equation solver and the dynamic parameters of the evader from the evader neural network agent, comprises: computing a plurality of component losses; and computing the loss as a weighted sum of the plurality of component losses.

[0016] In an embodiment, computing the loss as a weighted sum of the plurality of component losses uses a formula: calculating a first component loss ℒிof the plurality of component losses associated with an objective of the pursuer in following the evaderusing ℒி ൌ ^^^^^^^ െ ห^^^⃗^^^^^ െ ^^^⃗ா^^^^ห^, where ^^^^^^ is a desired tracking range, ^^^⃗^^^^^ ൌ^^^^^^^ ^^ா^^^^ ^^^^^^^^൩ and ^^^⃗ா^^^^ ൌ ^^^ா^^^^൩, where ^^^⃗^^^^^ is a position of the pursuer, ^^^^^^^ is a position of the pursuer along an x direction, ^^^^^^^ is position of the pursuer along a y direction, ^^^⃗ா^^^^ is a position of the evader, ^^ா^^^^ is a position of the evader along the x direction, and ^^ா^^^^ is a position of the evader along the y direction; calculating a second component loss ℒ^ாof the plurality of component losses using shows the ∗௧^output of the pursuer neural network agent during the simulation, and N is a number of iterations during the simulation; calculating a third component loss ℒௌ^of the plurality of component losses using where ^^^^^^^^^∗^ denotes the rectified linear unit (ReLU) function, ^^ is the pursuer’s heading angle and ^^௨^^^^, ^^^^௪^^are the upper and lower limits of the heading angle, ^^௫^,^௬^^and ^^௫ா,^^௬ாare the velocity components of the pursuer and evader along the ^^ and ^^ directions, respectively; calculating a fourth component loss ℒ^^of the plurality of component lossesusing where ^^^^^^^^ stands for "DesiredCollision Avoidance Range", calculating a fifth component loss ℒ^^of the plurality ofcomponent losses using ℒ^^ ൌ where ^^^௫ா^ி^^^^ and ^^^௬ா^ி^^^^ are ^^ and ^^ components of the relative position between thepursuer and the evader, expressed in the evader’s body frame, EBF, ^^^^ stands for "Desired Separation Distance", and ^^^^ is a mask function which assigns a value of 0 when the pursuer is positioned behind the evader, and a value of 1 when the pursuer is ahead of the evader.

[0017] In an embodiment, computing the loss as a weighted sum of the plurality ofcomponent losses uses ℒ௧^௧^^ ൌ ^^ிℒி ^ ^^^ாℒ^ா ^ ^^ௌ^ℒௌ^ ^ ^^^^ℒ^^ ^ ^^^^ℒ^^, where ^^ிis first a weight associated with the first loss component ℒி, ^^^ாis a second weight associated with the second loss component ℒ^ா, ^^ௌ^is a third weight associated with the third loss component ℒௌ^, ^^^^is a fourth weight associated with the fourth loss component ℒ^^, and ^^^^is a fifth weight associated with the fifth loss component ℒ^^.

[0018] In an embodiment, the dynamic parameters include a heading angle, a position, and a velocity.

[0019] In an embodiment, the evader neural network agent is trained to output a constant velocity and a constant heading angle of the evader.

[0020] In an embodiment, the method further comprises: implementing a barrier function as a safety constraint at each time step of the simulation.

[0021] In an embodiment, implementing the safety constraint comprises: implementing the safety constraint as a quadratic problem formulation

[0022] According to an aspect of the present disclosure, there is provided a transitory, or non-transitory, computer-readable medium having instructions stored thereon that, when executed by at least one processor, causes the at least one processor to perform the computer-implemented method of any preceding aspect or embodiment.

[0023] According to an aspect of the present disclosure, there is provided a pursuer vehicle comprising: at least one control surface for moving the pursuer vehicle; at least one sensor for sensing at least one dynamic parameter of an evader vehicle; at least one processor; and storage having the pursuer neural network agent trained according a method of any preceding aspect or embodiment stored thereon, the pursuer neural network agent causing the at least one processor to control the at least one control surface to move the pursuer vehicle in response to the at least one dynamic parameter of the evader vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0024] The subject-matter of the present disclosure is best described with reference to the accompanying figures, in which:

[0025] Figure 1 shows a perspective view illustrating scenario relating to infrastructure invasion;

[0026] Figure 2 shows illustrations of example unmanned surface vehicles (USVs) with vehicle dynamic limit trends;

[0027] Figure 3 shows an ordinary different equation solver according to one or more embodiments;

[0028] Figure 4 shows a flowchart detailed policy development associated with scientific machine learning;

[0029] Figure 5 shows a plurality of graphs illustrating time-elapsed trajectories of pursuer and evader from a scenario, e.g. scenario 1;

[0030] Figure 6 shows a graph of relative distance between a pursuer and an evader over time for another scenario, e.g. scenario 1;

[0031] Figure 7 shows graphs of scientific machine learning policy control actions over time for scenario 1;

[0032] Figure 8 shows a plurality of graphs illustrating a time evolution of the policy’s commanded reference control actions and the realised actions in scenario 2;

[0033] Figure 9 shows a graph illustrating relative distance between the pursuer and the evader over time for scenario 2;

[0034] Figure 10 shows a plurality of graphs illustrating a time evolution of the scientific machine learning policy control actions over time for scenario 2;

[0035] Figure 11 shows a plurality of graphs illustrating time-elapsed trajectories of pursuer and evader from scenario 2;

[0036] Figure 12a shows Case-1 Trajectory without CBF layer of Scenario-1;

[0037] Figure 12b shows Case-1 Trajectory without CBF layer of Scenario-1;

[0038] Figure 13 shows a relative distance between pursuer and evader over time for case-1;

[0039] Figure 14 shows policy control actions over time for case-1;

[0040] Figure 15 shows Monte-Carlo simulation trajectorys for SciML+HOCBF and Pure APF; and

[0041] Figure 16 shows tracjetory-1 and trajectory-2 for case-2 with different initial conditions. DESCRIPTION OF EMBODIMENTS

[0042] In this disclosure, the problem being addressed is how to develop an AI-aided tactics generator for uncrewed surface vessels (USVs) for protection of critical national infrastructure 10 and maritime assets 12 in face of surface vehicle attacks as illustrated in Figure 1.

[0043] The general mission profile for the pursuer USV includes initially either waiting near the protected platform or patrolling. When an approaching evader is localised and identified, the pursuer USV initiates the pursuing policy to intercept and neutralize the evader.

[0044] To accomplish this mission, an AI-aided defense strategy must overcome several challenges. The pursuer policy must navigate the USV to neutralize the evader in minimum time for protection. The steering commands must adhere to the vehicle’s dynamic limits. Given the safety-critical nature of the mission, a safe distance must be maintained while tracking to avoid collision with the evader. When facing a weaponized evader USV, it is necessary to approach from behind (astern approach) to avoid defeat. Additionally, the defense strategy must be capable of tracking and countering intelligent maneuvers by the evader.

[0045] The USVs are assumed to exhibit point mass characteristics. The equations motion model for an unmanned surface vehicle relative to inertial reference frame centered at the sea surface, considering 2 degrees of freedom, are expressed as follows: ^^^ ൌ ^^cos^^^^^ ൌ ^^sin^^

[0046] ^^^ ൌ ^^௩(1) ^^^ ൌ ^^

[0047] In this motion model, ^^ and ^^ denote the vehicle’s position coordinates, ^^ represents represents velocity magnitude, and ^^ indicates the heading angle. The control inputs ^^௩and ^^ correspond to longitudinal acceleration and angular velocity, respectively.

[0048] The maneuver capabilities of both pursuer and evader unmanned surface vehicles, defined by their ability to change velocity and heading, are constrained by dynamic models that are functions of current velocity of the vehicle. Figure 2 features illustrative examples of these dynamic constraints alongside several USVs currently on the market, intended purely for illustrative purposes.

[0049] In this section, we present the methodology and framework for developing AI- aided defense tactics for a pursuer USV against an evader. The baseline pursuing policy is formulated using scientific machine learning.

[0050] The principle of integrating governing physics into the training processes of machine learning models is to enhance their generalization capabilities. In accordance with this approach, the concept of scientific machine learning, implemented using Universal Ordinary Differential Equations (UODEs), has been proposed and formalized in for the purpose of scientific model discovery.

[0051] General training framework for SciML is illustrated in Figure 3.

[0052] The main idea of the UODEs involves formulating an ordinary differential equation that incorporates a mechanistic model with a universal approximator to estimate unknown dynamics. Specifically, a simple first-order ODE defined by the ground truth modelൌ ^^^^^, ^^^ can be represented in UODE form as ൌ ^^^^^^, ^^^ ^ ^^௨^∗^, where theknown part of the ground truth system is denoted with the function of ^^^^^^, ^^^ and ^^௨^∗^represents the unknown dynamics that we want to discover. This unknown component is approximated with a universal approximator (also merely called an approximator),specifically a neural network, expressed as ^^௨^∗^ ^ ^^^^^^^, ^^ேே^. During each forwardpropagation, the ODE system is simulated by solving it, concluding with the calculation of a scalar loss function, such as the mean square error between the ODE solution and real-world experimental data that represents the ground truth model. In backward propagation, continuous-time gradients of the network parameters with respect to the loss function are computed using adjoint sensitivity methods. After gradient computation, network parameters are updated using gradient descent methods.

[0053] In addressing the USV tactic generation problem, we have reformulated our pursuit-evasion problem and treating it as a scientific model discovery problem similar to framework described. The motion model presented earlier is deconstructed into components of known and unknown dynamics. Pursuer’s defence tactic generation policy meaning the controls of the vehicle is conceptualized as a learnable, unknown dynamical model. It is modelled utilizing an artificial neural network parameterized with ^^ேேincluding the network weights and biases.

[0055] In Equation 2, ^^^^^^ represents the feature vector of the neural network policy model, which will be designed accordingly to performance objectives described in subsequent sections.

[0056] Consistent with the scientific model discovery approach described, the mechanistic motion model in Equation is reformulated to include both known and unknown dynamics as follows:

[0058] Where, "*" shows the ∗௧^output of the neural network policy model.

[0059] In the UODEs framework detailed in Equation 3, learning the decision-making policy model for the pursuer involves training the artificial neural network. To do so, we need to define a performance index to enforce the pursuer for following the evader while also meeting the mission objectives. Accordingly, we have developed the following loss function to serve this purpose.

[0060] ℒ௧^௧^^ ൌ ^^ிℒி ^ ^^^ாℒ^ா ^ ^^ௌ^ℒௌ^ ^ ^^^^ℒ^^ ^ ^^^^ℒ^^ (4)

[0061] In Equation 4, we have presented the total loss function that includes various components crafted for specific objectives and aggregated using a weighted sum approach. The component ℒிis associated with the pursuer’s objective to follow the evader. ℒ^ாaccounts for the control effort loss. ℒௌ^addresses the pursuer’s state constraints, including maximum and minimum state limits. ℒ^^focuses on avoiding collisions between the pursuer and the evader. Additionally, we have used ℒ^^component to guide the pursuer to execute the astern approach maneuver whileintercepting the evader. The coefficients ^^ி , ^^^ா , ^^^^ , ^^^^ and ^^^^ scale the importance ofthese components within the optimization process.

[0062] In other words, methods described herein, specifically in relation to Algorithm 5, include computing a plurality of component losses, a computing the loss (i.e. the total loss) as a weighted sum of the plurality of component losses. The plurality of component losses includes a first component loss ℒி, a second component loss ℒ^ா, a third component loss ℒௌ^, a fourth component loss ℒ^^, and a fifth component loss ℒ^^.Similarly, the coefficients ^^ி , ^^^ா , ^^^^ , ^^^^ and ^^^^ may be understood to mean the first tofifth weights, respectively.

[0063] The primary objective of the pursuer is to follow and intercept the evader in the shortest possible time. To align with this goal, we have considered that minimizing the distance between the pursuer and the evader is crucial. However, it is also crucial for the pursuer to maintain a safe tracking distance from the evader to avoid collisions. Consequently, the loss component ℒிcan be expanded as follows: ^^^^^^^ ^^ா^^^^

[0065] Where, ^^^⃗^^^^^ ൌ ^ ^^^^^^^൩ and ^^^⃗ா^^^^ ൌ ^ ^^ா^^^^൩ represent the position vectors of thepursuer and evader, respectively. ^^^^^^ stands for the "Desired Tracking Range".

[0066] To align with the objective of minimizing capture time, we have included the minimization of angular velocity control effort in our loss definition. Minimizing control effort helps guide the pursuer to take the most efficient action, steering it toward the evader along the shortest possible path.

[0068] For state constraints, we have implemented continuous upper and lower bounds on the heading angle to prevent the pursuer from rotating around itself with divergent behavior during training iterations. This approach helps to ensure a more stable training process. Additionally, we have included endpoint constraints on velocity and its direction to further guide the pursuer’s movement and enhance the effectiveness of the training.

[0070] Where, ^^^^^^^^^∗^ denotes the rectified linear unit (ReLU) function, which outputs the input value if the input is greater than zero; otherwise, it outputs zero. ^^ is the pursuer’s heading angle and ^^௨^^^^, ^^^^௪^^are the upper and lower limits of the heading angle. Furthermore, ^^௫^,^௬^^and ^^௫ா,^^௬ாare the velocity components of the pursuer and evader along the ^^ and ^^ directions, respectively.

[0071] Considering safety requirements, we have implemented the collision avoidance loss function between the pursuer and the evader as follows:

[0073] Where, ^^^^^^^^ stands for "Desired Collision Avoidance Range" which is determined by considering the size information of the example USVs involved.

[0074] To successfully execute an astern approach for intercepting the evader, we have designed a loss function that operates within the evader’s body reference frame. This frame of reference provides key information about the relative positioning of the pursuer, specifically indicating whether the pursuer is ahead of or behind the evader. This orientation is essential for adjusting the pursuer’s strategy to either close in from behind or adjust its path to move into an astern position effectively.

[0076] Where, ^^^௫ா^ி^^^^ and ^^^௬ா^ி^^^^ are the ^^ and ^^ components of the relative positionvector between the pursuer and the evader, expressed in the evader’s body frame (EBF). ^^^^ stands for "Desired Separation Distance". Additionally, ^^^^ is a mask function which assigns a value of 0 when the pursuer is positioned behind the evader, and a value of 1 when the pursuer is ahead of the evader.

[0077] ^^^^ ൌ ^1,0,

[0078] From Equation 9, astern approach loss function ℒ^^consists of two components. The first component aims to guide the pursuer to decrease the distance to the evader and follow from behind, penalizing the pursuer when it is positioned in front of the evader. The second component ensures that the pursuer maintains a safe separation (^^^^^^) from the target line when it is in front of the evader, addressing safety requirements. Together, these components effectively constitute the astern approach maneuver.

[0079] To incorporate vehicle dynamics as control constraints, we have approximated the system dynamic limits using shallow neural networks (One Hidden Layer Networks) ,^^^^:ℝ^^ →ℝ^.These networks take the vehicle velocity as input and output thecorresponding vehicle dynamic limits. The training utilizes a dataset derived from real- world experiments involving an experimental USV. Furthermore, physical constraints are included as a Lagrangian penalty in the optimization objective, following the physics- informed neural networks framework, to ensure the limit functions behave well and generalize effectively. An example objective function for training the longitudinal acceleration upper limit model is provided below.

[0081] The equation utilizes ^^^^^^^^ to denote the mean squared error between, ^^^௬^௧^^௧^^and ^^^ representing the synthetic data from the vehicle dynamics model and the network approximation value, respectively. The coefficient serves as a loss weight, balancing the importance within the objective function. The ^^^^^^^^^^ function is employed to penalize the model, preventing negative values for the acceleration upper limit.

[0082] After approximating the vehicle dynamics models with shallow neural networks, we integrated the system dynamics into the scientific machine learning training process by employing a neuron-wise adaptive tanh activation function in the final layer of the policy network.

[0084] The main idea of adaptive activation functions is to make them trainable to enhance their ability to model nonlinear behavior. In Equation 12, ^^ is the trainable parameter used to adjust the slope of the tanh activation function. Moreover, Ameya et al. highlighted that adding a non-trainable scaling factor, ^^, into the equation contributes to the convergence speed. These adaptive activation functions can be applied either layer-wise, where a single trainable parameter is used uniformly across the layer, or neuron-wise, with each neuron having its own independent trainable parameter, matching the number of neurons in the layer.

[0085] This choice of activation function allows for dynamic scaling and adjustment of outputs, ensuring that the network’s response remains within the physically plausible limits dictated by the vehicle’s dynamics.

[0086] Employing the neuron-wise adaptive Tanh activation function in the final layer of the policy network effectively bounds the output variables within the range ^−1,1^. Subsequently, a dynamic re-scaling operation maps this ^−1,1^ interval to the current minimum and maximum limits as determined by the output of the trained vehicle dynamic’s limit networks. This mechanism matches the real-world limit values, thus integrating the policy outputs directly with the system’s dynamic constraints and preventing any violation of these limits. The re-mapping operation is given by the following formula.

[0088] ௪^ೌ^^^^௧^^ା௪^^^^^^௧^^(14) ^ ଶ

[0089] Where, ^^^^^௫,^^^^^^,^^^^௫,^^^^^are corresponding trained neural network models used for approximating the maximum and minimum longitudinal acceleration and angular velocity limits of the vehicle, respectively.

[0090] After establishing our performance index through specific loss functions and integrating vehicle dynamics, we conducted our scientific machine learning optimization processes using the Julia programming language. This choice was made due to Julia’s high performance in numerical and scientific computing, effectively facilitating complex computations and model training.

[0091] To perform the optimization loop, we defined optimization scenarios that establish the initial conditions and movement behaviors for both the pursuer and the evader.

[0092] For training purposes, we have considered to train our pursuer agent against a simple evader, which lacks intelligence or the capability to perform maneuvers during simulations. This simplification means that throughout a simulation, the evader maintains a constant speed and heading angle.

[0093] Specifically: ^ The initial position of the evader is stochastically sampled from all quadrants within a 500-meter length square centered at the origin, denoted as ∈ ^^^െ500^^, 500^^^.^ The initial velocity magnitude of the evader is sampled from a uniform distribution ^ ^ ^^ா^ ∈ ^^^1^, 17^ ^ and remains constant throughout a single simulation. ^ The initial heading angle of the evader is sampled from a uniform distribution ^^^ா∈ ^^^െ180∘, 180∘^ and remains constant throughout a single simulation.

[0094] In this way, the evader is trained to output a constant velocity and a constant heading angle of the evader.

[0095] For the pursuer, we created a structured initial conditions grid: ^ The initial position of the pursuer is distributed over 10 equally spaced intervals on circles centered at the origin with radius of 50 meters and 100 meters, resulting in a total of 20 distinct initial positions. ^ The initial velocity magnitude of the pursuer is divided into 5 evenly spaced samples ^ ^ within the range ^1 ^, 17^ ^ ^ The initial heading angle of the pursuer is divided into 10 evenly spaced samples spanning ^െ180∘, 180∘^

[0096] This configuration of initial conditions for the pursuer generates a grid by taking permutations of the states, resulting in 2000 distinct initial condition combinations.

[0097] The overall policy optimization framework, which utilizes SciML training for our pursuit-evasion formulation, is illustrated in Figure 2 and detailed in Algorithm 5.

[0098] Algorithm 5

[0099] Input: ODE Solver(ODE, IC’s),

[0100] Epoch Size: N, Mini-Batch Size: M

[0101] Data: Initial Conditions Grid for Pursuer

[0102] Initialization:Initilization of Network Weights, Optimization Variable:, ^^ேே^^^^^^^^ℎ ൌ 1,2, … , ^^

[0103] Generate Minibatches ^^^^^^^^^^^^^^^^ℎ ൌ 1,2, …

[0104] Pursuer IC’s ^^^^ Minibatch ^

[0105] ^^ா^^^^^ ← Sample the Evader’s Initial Condition ^

[0106] ^^^^^^ா^,^^^^^^^ ← Run the ^^^^^^^^^^^^^^^^^^^^^^^^^^ , ^^ ^^^^^, ^^ேே^^

[0107] Compute Loss, 108] Compute Gradient,ௗℒ^ [ ^^ௗ^ಿಿ^← ℒ^ , ^^^^^^^ ,^^^^^^^ , ^^ேே^

[0109] Compute Mean Loss, Update Network Weights, ^^ேே ^ ← ^^௨^ௗ^௧^ௗ

[0112] The SciML training flowchart in Figure 4 and corresponding Algorithm 5 can be briefly explained as follows: The initial conditions’ training grid for pursuer states is given as input to the algorithm. The first step of the algorithm involves initializing the weights of the neural network that models the decision-making policy. After successful initialization, mini batches are randomly formed from the initial condition grid created for the pursuer agent. Then, a simulation is run for each initial condition in the mini batches. For each mini-batch simulation, the initial condition states for the evader are sampled stochastically as explained. Subsequently, the defined loss functions are computed along with the policy gradient with respect to total loss given in Equation 4. After completing all the initial conditions in a mini batch, the average loss and gradient calculations are performed. This makes conducting gradient descent over the policy network possible. After updating the network weights for all mini batches, a new epoch starts. At the beginning of a new epoch, mini batches are created randomly. The same steps are followed until convergence is achieved.

[0113] In other words, Algorithm 5 summarises a computer-implemented method of training a pursuer neural network agent. The method may be summarised as providing an evader neural network agent trained to update iteratively a plurality of dynamic parameters of an evader, a pursuer neural network agent initialised to update iteratively a plurality of approximators for a pursuer, and an ordinary differential equation solver configured to update a plurality of dynamic parameters of the pursuer based on the updated approximators. The dynamic parameters of each of the evader and the pursuer comprise at least one of position, heading angle, and velocity.

[0114] The method further comprises providing initial dynamic parameters of the pursuer. The initial dynamic parameters may also be called initial conditions, ICs. The method may comprise dividing the initial dynamic parameters of the pursuer into mini batches.

[0115] The method may comprise sampling the initial dynamic parameters from a set of initial dynamic parameters of the evader, running a simulation involving the pursuer neural network agent pursuing the evader neural network agent based on the sampled initial dynamic parameters, computing a loss based on the dynamic parameters of the pursuer from the ordinary differential equation solver and the dynamic parameters of the evader from the evader neural network agent, and optimising weights of the pursuer neural network agent based on the computed loss. More specifically, the method may comprise separating the set of initial dynamic parameters of the pursuer into a plurality of mini batches, wherein sampling initial dynamic parameters from a set of initial dynamic parameters of the evader, running the simulation involving the pursuer neural network agent, and computing a loss based on the output from the pursuer neural network agent are performed for each mini batch of the plurality of mini batches of initial dynamic parameters of the pursuer.

[0116] Since the simulation may be run for each mini batch, the loss may also be computed for each mini batch. For example, the method may comprise computing an average loss using the computed loss for all mini batches, wherein optimising the weights comprises optimising the weights of the pursuer neural network agent to minimise the average loss. More specifically, computing a mean loss using the loss computed for each mini batch, wherein optimising the weights of the pursuer neural network agent based on the computed loss comprises optimising the weights of the pursuer neural network agent using the mean computed loss.

[0117] running the simulation comprises: performing forward propagation of the evader neural network agent to update the plurality of dynamic parameters based on the sampled initial dynamic parameters; and performing forward propagation of the pursuer neural network agent to update the plurality of dynamic parameters based on initial dynamic parameters of the pursuer, the updated dynamic parameters from the evader neural network agent.

[0118] This algorithm may be provided as a set of instructions stored on non-transitory computer-readable media that, when executed by at least once processor, causes the at least one processor to perform the computer-implemented method described here. The instructions may also be provided as transitory computer-readable media when being loaded onto the non-transitory storage media.

[0119] The resulting pursuer neural network agent may be installed onto a pursuer vehicle comprising at least one control surface for moving the pursuer vehicle, at least one sensor for sensing at the at least one dynamic parameter of an evader vehicle, at least one processor, and storage having the pursuer neural network agent stored thereon. In this way, the pursuer neural network agent may cause the at least one processor to control the at least one control surface to move the pursuer vehicle in response to the at least one dynamic parameter of the evader vehicle

[0120] In this section, we provide a brief overview of the final form of the policy architecture, hyperparameters used in the training processes and present the results from various simulation scenarios. Our simulations includes several cases: a simple evader heading directly towards infrastructure, a scenario incorporating actuator delay, and a scenario featuring a maneuvering evader. These settings help to evaluate the robustness and efficiency of our developed policy under different conditions.

[0121] As outlined in Equation 2, we have modeled pursuer agent’s policy by using a neural network architecture. The loss components, defined in Equations 5-9 based onmission requirements, contribute to designing the feature vector ^^^^^^ ∈ ℛ^^. This vectorcontains essential information for the agent’s decision-making at time ^^.

[0123] The vectors ^^^^⃗ ூ^ଶா ^^^^, ^^^^⃗^ாଶ^ாி^^^^ denote the relative distance between the pursuer and the evader, expressed in the inertial reference frame and the evader body fixed reference frame, respectively. ^^^^ଶா^^^^ indicates the time derivative of the relative position vector, highlighting the dynamic distance changes between pursuer and evader overtime. The velocity vectors of the pursuer and evader are denoted as ^^⃗^^^^^^ and ^^⃗^ா^^^^, with||^^⃗^^^^^^|| specifying the magnitude of the pursuer’s velocity. Lastly, ^^^^ைௌುమಶ^^^^ representsthe rate of change of the line-of-sight angle between the pursuer and the evader, capturing the angular dynamics of their engagement.

[0124] The specific details of the policy architecture, including layer configurations, activation functions, and other number of parameters, are summarized in Table 1

[0125] Table 1: Neural network polic architecture

[0126] Table 1 shows that the neural network architecture consists of three layers, eachwith 16 neurons (^^^ ൌ ^^ଶ ൌ 16), and includes Scaling and Re-Scaling layers at the inputand output stages, respectively. Min-max scaling is applied for the scaling operation, and re-scaling follows the method described in Section 3.4 at Equation 14. The activationfunction for each layer is the ^^^^^^ℎ function, except for the last layer, which uses the ^^^^ െ^^^^^^ℎ function—a neuronwise adaptive tanh function.

[0127] SciML’s policy training process involves tuning hyper-parameters such as cost coefficients, mini-batch size, simulation time, and learning rate. We have used the grid search algorithm for this purpose. The complete training setup and the optimized hyper- parameter values are summarized in Table 2.

[0128] Table 2: Training Setup and Hyperparameters

[0002]

[0129] Simple evader case 1

[0130] In this section, we present the simulation results of a pursuer using the SciML policy against a simple evader, which directly approaches the infrastructure at a constand speed and heading without any strategic maneuvering. The initial conditions for both the pursuer and evader are summarized in Table 3.

[0131] Table 3: Scenario-1 Initial Conditions Setup

[0132] Figure 5 displays the time-elapsed trajectories of both the pursuer and evader, with the pursuer’s path in blue and the evader’s in orange, as the evader heads straight for the infrastructure. Figure 6 illustrates the changing distance between the pursuer and evader over time. The simulation results show that the pursuer, guided by our trained policy, successfully maneuvers towards the capture zone and maintains tracking of the evader. A key aspect of performance highlighted here is that the pursuer executes an astern approach to maintain safety. Additionally, Figure 6 indicates that the pursuer keeps a safe distance, minimizing collision risks. These findings confirm that the trained policy meets the defined performance requirements effectively.

[0133] Figure 7 shows the SciML policy’s commanded reference longitudinal acceleration and angular velocity over time. The blue lines represent the control action values generated by the policy network, while the red dashed lines show the system dynamics’ upper and lower boundaries. The data are normalized to the range of ^െ1,1^, with ^1 and െ1 representing the upper and lower limits, respectively. This normalization is due to confidentiality reasons. The results demonstrate that the trained policy operates within the vehicle’s dynamic limits, with no commands exceeding these boundaries, which is a direct outcome of the vehicle dynamics integration strategy employed. Additionally, the action values from our policy exhibit a bang-bang control type, typical of minimum time control laws, indicating that the policy functions as an optimal controller.

[0134] Simple evader scenario 2

[0135] In this scenario, we followed the same setup as in Scenario-1, with the initial conditions of a simple evader summarized in Table 5. However, to assess the generalization capability of our approach, we introduced actuator delay into the system dynamics as a form of uncertainty. It is important to note that the policy optimization process did not originally account for actuator delay.

[0136] Table 4: Pursuer and simple evader elapsed time trajectories for scenario-1

[0137] The reference longitudinal acceleration and angular velocity commanded by the SciML policy are subject to a first-order dynamic actuator delay, modelled as follows:^^^^ைெ ൌ ^^^^^^^^^^^^^^,^^ேே^ ^^ ൌ ^ଶ^^ைெ ^^^^ ^^^^^^^, ^^ேே^

[0138] ^^^^^^^ೀಾି^^^௫ୀఛ (16) ^^^ ൌ^௪^ೀಾି௪^ఛ

[0139] In Equation 16, ^^^^ೀಾand ^^^ைெrepresent the commanded control actions from the SciML policy. The variables ^^^and ^^denote the actual realized control actions, whichare affected by actuator delay characterized by a time constant ^^ ൌ 1 seconds.

[0140] Figure 5 displays the time-elapsed trajectories of the pursuer and evader in Scenario-2, with the pursuer’s path marked in blue and the evader’s in orange. Figure 7 tracks the relative distance between the pursuer and evader over time. Additionally, Figure 8 illustrates the time evolution of the policy’s commanded reference control actions and the realized actions in Scenario-2. The blue lines represent the reference control action values generated by the policy network, while the orange lines show the realized control actions affected by actuator delay dynamics. Despite the the presence of significant actuator delay, which was not accounted for during the policy training, the simulation results show that the pursuer, guided by our trained policy, effectively approaches the evader and skillfully executes the astern approach maneuver. This demonstrates the generalization capability and the robustness of the policy in handling dynamics not explicitly included in its training.

[0141] Figure 8 illustrates the time evolution of the policy’s commanded reference control actions and the realized actions in Scenario-2. The blue lines represent the reference control action values generated by the policy network, while the orange lines show the realized control actions affected by actuator delay dynamics. The red dashed lines indicate the boundaries set by the system dynamics. Despite the presence of actuator delays, the trained policy demonstrates effective generalization capability. The commands generated by the policy adapt well to the dynamics encountered, effectively fulfilling the mission requirements.

[0142] Although there is a performance degradation compared to the Scenario-1 results, it remains within acceptable levels, as no unstable behavior is observed. The pursuer effectively tracks the evader and manages to avoid collision risks. However, unlike in Scenario-1, the pursuer does not maintain a consistent distance within the capture zone after executing an astern approach, due to the actuator delay impacting its ability to accelerate promptly. This performance loss could be eliminated by incorporating delay dynamics into the training processes, which would allow the policy to better handle delays.

[0143] Manoeuvring Evader: Scenario-3

[0144] To further assess the generalization capability of our trained SciML policy, we conducted tests against an evader executing an S-curve-like trajectory, representing an out-of-distribution scenario. Since, this type of evader trajectory was not encountered during the training phase; instead, as previously mentioned, the policy was trained against simpler evaders maintaining constant velocity and heading throughout a single simulation. This testing scenario helps evaluate how well the policy adapts to complex and unexpected maneuvers from the evader.

[0145] Figure 11 illustrates the elapsed time trajectories of the pursuer and evader, showing the paths from the initial to the final time. The pursuer’s path is depicted by the blue line, while the evader’s path, following a predefined S-curve-like trajectory, is shown in orange. Despite the pursuer’s policy not being specifically trained to handle maneuvering evaders, the simulation results demonstrate the policy’s exceptional generalization ability. It successfully intercepts the evader and maintains effective tracking throughout the scenario.

[0146] In this research, we addressed the USV tactic generation problem for protection of critical national infrastructure and maritime assets in face of surface vehicle attacks. We have conceptualized the pursuit-evasion problem as a scientific model discovery problem, wherein the tactic generation policy is treated as a learnable, unknown dynamical model that governs the relative system state.

[0147] Our approach has shown outstanding performance through extensive simulations, even in scenarios that significantly deviate from the training domain. Moreover, our approach effectively mitigates various challenges, including achieving near-optimal minimum time-to-capture, diminishing the reliance on extensive training datasets, enhancing convergence speed, and lowering computational demands. We illustrate the results across a wide range of applicable scenarios.

[0148] These results underscore the potential of scientific machine learning to revolutionize the field of pursuit-evasion and, by extension, the broader domain of security and defense. The methodology holds promise for various security and defense mechanisms, offering a scalable solution to many similar pursuit-evasion problems including one to many and many to many scenarios in both defensive and offensive setting. Currently, we are focused on implementation of this tactic generator on actual USV platforms to demonstrate the real-life performance of the proposed approach.

[0149] Scientific Machine Learning (SciML), which embeds physical priors into learning architectures, offers improved generalization and interpretability, as demonstrated in tasks like precision landing and nonlinear control. However, like RL, SciML-based methods often lack formal safety guarantees.

[0150] To address this gap, we propose a novel framework that integrates SciML with Control Barrier Functions to our knowledge, the first such integration for maritime pursuit- evasion. Our method enables provably safe, real-time obstacle avoidance during pursuit, even under partial observability and occlusion, while accounting for realistic USV dynamics and supporting non-convex obstacle representations.

[0151] This section of the description builds on the preceding section where we applied SciML to PE by embedding known USV dynamics into a universal ODE and learning unknown control terms through neural networks. While that approach focused on inter- agent collision avoidance and time-optimal capture, it did not address navigation in obstacle-rich environments.

[0152] Here, we extend that framework with a CBF-based safety layer to ensure provably safe navigation in complex, real-world maritime settings while maintaining pursuit performance. The synergy between SciML’s generalization and CBF’s formal safety assurances results in a robust and scalable approach to autonomous pursuit in cluttered maritime domains.

[0153] Problem Statement and Modelling

[0154] In this section, we present the methodology and framework for developing AI- aided defense tactics for a pursuer USV against an evader in a multi-obstacle environment. The baseline pursuing policy is formulated using scientific machine learning, and obstacle avoidance is integrated using the control barrier function approach.

[0155] Unmanned Surface Vehicle Model

[0156] To maintain tractability in the pursuit-evasion framework, each USV is modeled as a 2-DOF point mass with planar position, heading, and velocity states. Control inputs are longitudinal and rotational accelerations, constrained by physical limits (e.g., thruster capabilities). The point-mass dynamics is described by

[0157] ^^^ ൌ ^^cos^^, ^^^ ൌ ^^sin^^,^ ^(17) ^^ ൌ ^^௩, ^^ ൌ ^^,

[0158] where ^^, ^^ ∈ ℝ denote the position coordinates, ^^ ∈ ℝ is the linear velocitymagnitude, and ^^ ∈ ℝ is the heading angle. The control inputs ^^௩ and ^^ represent thelongitudinal acceleration and angular velocity, respectively.

[0159] Both the pursuer and the evader are subject to velocity-dependent maneuver limits: as speed rises, admissible surge acceleration and turn rate drop sharply. We calibrated these limits with proprietary Pacific 24 sea trial data (not publicly releasable); only the functional relationship - higher speed means lower agility - is required for reproduction of the results.

[0160] Method

[0161] In this section, we present the methodology and framework for developing AI- aided defense tactics for a pursuer USV against an evader. The baseline pursuing policy is formulated using scientific machine learning.

[0162] Scientific Machine Learning (SciML) preliminaries

[0163] Scientific machine learning (SciML) offers an alternative approach to classical machine learning control by embedding the agent’s governing physics directly into the training loop. Instead of learning a control policy purely from data or rewards, SciML treats the unknown portion of a physical model (in this case, the pursuer’s decision- making / control) as a learnable component within a system of differential equations. Formally, the known USV motion dynamics form the “mechanistic” part of the ordinary differential equation (ODE), while a neural network (NN) approximates the unknown terms.

[0164] In our previous work, we cast the pursuer dynamics as a Universal Differential Equation (UDE) such that: the pursuer’s state evolves through a combination of i) a physics-based model (referring to mechanistic model) and (ii) a NN-based policy that outputs the desired acceleration and angular velocity. This UDE framework is described as:

[0166] where the terms ^^^^⋅^ and ^^௨^⋅^ represent the known and unknown components of the system dynamics. The term ℕℕ^⋅^ denotes the neural network-based control policy trained via gradient descent, ^^^^^^ is the input feature vector, ^^ேேdenote the weights of the neural network, and ℕℕ^^^^⋅^ indicates the ^^th output of the policy network.

[0167] After formulating the UDE, we optimise the network weights with the composite physics-informed loss given in Equation 19 comprising: tracking loss (^^்்) to minimize deviation from the desired tracking range, control effort loss (^^^ா) to penalize high angular velocity, state constraint loss (^^ௌ^) to enforce state upper and lower limits, collision avoidance loss (^^^^) to maintain safe distance, and astern approach loss (^^^^) to guide pursuit from behind the evader.

[0168] ^^ ൌ ^^்் ^^்் ^ ^^^ா ^^^ா ^ ^^ௌ^ ^^ௌ^ ^ ^^^^ ^^^^ ^ ^^^^ ^^^^. (19)

[0169] where

[0176] Here ^^ indexes the simulation steps (^^ ^ 1 in total), ^^^^→^ is the relative positionvector, ^^^the desired range, ^^୰^^the reference yaw rate, ^^^^,^^௨^state bounds, ^^^^^thecollision threshold, and ^^̃^^^^௫ , ^̃^^^^௬ ^ the evader-body-frame coordinates used for asternpursuit. ^^ is a mask function which assigns a value of 0 when the pursuer is positioned behind the evader, and a value of 1 when the pursuer is ahead of the evader. All ^^^⋅^are positive weighting scalars.

[0177] The SciML policy training pseudo-algorithm shared in Algorithm 3.1.

[0178] SciML minibatch training algorithm

[0179] Input: ODE Solver(ODE, IC’s), Epoch Size: N, Mini-Batch Size: M Data: Initial Conditions Grid for Pursuer Initialization:Initilization of Network Weights, ^^ேேOptimization Variable:, ^^ேே ^^^^^^^^ℎ ൌ 1,2, … , ^^ Generate Minibatches ^^^^^^^^^^^^^^^^ℎ ൌ1,2, … Pursuer IC’s ^^^^ Minibatch ^^ா^^^^^← Sample the Evader’s Initial Condition ^ா^ ^^^^^^^,^^^^^^^ ← Run the ^^^^^^^^^^^^^^^^^^^^^^^^^^ , ^^ ^^^^^, ^^ேே^ Compute Loss, ^^^ ← ^^^^^^^ Computet, ^ௗ^^^ Gradien^ௗ^ಿಿ^← ^^^ , ^^^^^^^ ,^^^^^^^ , ^^ேே^

[0180] i ^^^^ Minibatch ^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^ ^^^^^^ℎ ^^^^^^^^ ^^^^^^^^^^^^^^^^^^:

[0181] ^^௨^ௗ^௧^ௗUpdate Network Weights, ^^ேே ^ ← ^^௨^ௗ^௧^ௗ

[0182] Algorithm 3.1 begins with a predefined grid of pursuer initial conditions. The neural network that represents the control policy is randomly initialised, after which the grid is split into mini-batches. For each batch, an evader state is sampled heading and speed held constant to emulate a non-manoeuvring target and, with fixed network weights, the resulting UDE is integrated in parallel to generate trajectory roll-outs. The composite loss of Eq. (19) is then evaluated on every trajectory, and its gradient with respect to the network parameters is obtained by back-propagating through the differentiable solver. A first-order optimiser updates the weights, a new set of mini- batches is drawn, and the process repeats until the loss converges. The resulting policy satisfies the pursuit-evasion objectives while respecting dynamic and safety constraints.

[0183] The control policy is realised by the feed-forward network reported in Table 5, which employs adaptive tanh activations. At each sampling instant, the network receives the input state vector as

[0185] and longitudinal acceleration and angular turn rate commands that are subsequently rescaled to honour the USV’s speed-dependent acceleration and turn-rate envelopes. This embedded dynamic filter guarantees real-time feasibility while preserving the expressive power of the network. In earlier work, the resulting policy achieved reliable interception and target-collision avoidance across a broad set of initial conditions, yet it lacked guarantees in cluttered waters. To address this limitation, we now augment the policy with a control-barrier-function (CBF) layer that enforces provable obstacle avoidance without materially altering baseline performance.

[0186] Table 5: Neural network policy architecture Layer Type # units Activation Input Scaling 11 - Layer 1 Dense 16 Tanh Layer 2 Dense 16 Tanh Layer 3 Dense 16 NA-Tanh Output Re-Scaling 2 -

[0187] Obstacle Avoidance with Control Barrier Functions

[0188] To ensure obstacle avoidance, we have integrated our baseline policy, derived using SciML, with the control barrier function approach. This combination allows us to enforce safety constraints dynamically.

[0189] Preliminaries

[0190] Considering a nonlinear affine system of the form,

[0191] ^^^ ൌ ^^^^^^ ^ ^^^^^^^^, ^^ ∈ ℝ^, ^^ ∈ ℝ^, (20)

[0192] let ^^ ൌ ^^^: ℎ^^^^ ^ 0^ be the safe set. ℎ is a control barrier function (CBF) if ∃ ^^ ∈^^^such that

[0193] sup^^^ℎ ^ ^^^ℎ ^^ ^ ^^^ℎ^ ^ 0, (21)^^

[0194] where ^^^ℎ and ^^^ℎ are Lie derivatives of ℎ along ^^ and ^^. Safety is enforced via the quadratic program

[0195] min^^^ െ ^^ ୃ^^^^ ^^^^^ െ ^^^^ s. t. ^10^. (22)

[0196] If ^^^ℎ ൌ 0 (relative degree ^^ ^ 1), the high-order CBF (HOCBF) frameworkimposes

[0198] with the recursive sequence

[0199] ^^^ ൌ ℎ, ^^^ା^ ൌ ^^^ ^ ^ ^^^ା^^^^^^, ^^ ൌ 0, … , ^^ െ 1. (24)

[0200] Here ^^^ and ^^ is the relative degree.

[0201] Obstacle Avoidance with Control Barrier Functions

[0202] To ensure obstacle avoidance, we have integrated our baseline policy, derived using SciML, with the control barrier function approach. This combination allows us to enforce safety constraints dynamically.

[0203] Preliminaries

[0204] Considering a nonlinear affine system of the form,

[0205] ^^^ ൌ ^^^^^^ ^ ^^^^^^^^, (25)

[0206] where ^^ ∈ ℝ^ defines the system states, ^^ ∈ ℝ^ are the control inputs, and^^^^^^ ∈ ℝ^ and ^^^^^^ ∈ ℝ^ൈ^ describes the nonlinear dynamics.

[0207] A Control Barrier Function (CBF) is defined for a system to enforce safetyconstraints by ensuring that the system states remain within a safe set, ^^ ⊂ ℝ^. Acontinuous differentiable function ℎ^^^^ is a CBF if there exists an extended class ^^ function ^^ such that for all ^^ in the safe set ^^,

[0208] sup^^^^ℎ^^^^ ^ ^^^^ℎ^^^^^^ ^ ^^^ℎ^^^^^ ^ 0, (26)^^

[0209] where ^^^^ℎ^^^^ and ^^^^ℎ^^^^ denote the Lie derivatives of ℎ along ^^ and ^^, respectively .

[0210] Typically, the barrier function is implemented as a safety constraint in a quadratic program (QP) at each time step with an objective of finding a control input that minimally deviates from a predetermined baseline control.

[0212] In this way, the method may comprise implementing a barrier function as a safety constraint at each time step of the simulation. Implementing the safety constraint may comprise: implementing the safety constraint as a quadratic problem formulation.

[0213] From the CBF definition and the quadratic program (QP) formulation, it becomesclear that when ^^^^ℎ^^^^ ൌ 0 it is impractical to find a safe controller using (26) or toimplement a QP as defined in (31). This issue arises because the chosen function ^^^^^^ possesses a high relative degree with respect to the system dynamics given in (25). A function ℎ that is continuously differentiable is said to have a control input relative degree ^^ with respect to system if the first occurrence of the control input ^^ in the derivatives of ℎ along the system’s dynamics happens in its ^^th derivative.

[0214] Prior methods enable the synthesis of a controller that ensures system forwardinvariance, denoted as ∀^^ ∈ ^^ for ^^ ^ ^^^, using a time-invariant HOCBF. The synthesisprocess effectively maintains the system’s state within the safe set ^^ beyond the initial time ^^^. sup ^^^ℎ^^^^ ^ ^^ ^^ ^ି^ℎ^^^^^^ ^ ^^^ℎ

[0215] ^^ ^^ ^^ ^^^^^^^ (28) ^^^^^^^^ି^^^^^^ ^ 0,

[0216] In the expression, ^^^^^ℎ^^^^ refers to ^^th Lie derivative of ℎ^^^^ along ^^ and ^^ ^^^ି^^^ ^^ ℎ^^^^ is used for representing Lie derivative along the ^^ following ^^^ െ 1^th Liederivate of ℎ^^^^ along ^^. The term ^^^ℎ^^^^^ indicates the remaining Lie derivatives along ^ ^^^ with degree less than or equal to ^^^ െ 1^th . The functions, ^^^:ℝ→ℝ, form a sequence, with their corresponding sets defined subsequently.

[0218] Where, ^^ is an extended class of ^^ function. ^^ denotes the relative degree of the barrier function with respect to system.

[0219] Obstacle Avoidance for Pursuit-Evasion Problem

[0220] Referring to the system definition in Equation 18, the functions ^^^^^^ and g(x) aredefined as follows ^^^^^^ ൌ ^^^cos^^, ^^sin^^, 0,0^ୃ and ^^^^^^ ൌ ^0 ୃଶ, ^^ଶ^ .

[0221] Given the assumption that obstacles are perfect circles for ease of implementation and to serve as a proof of concept, the safe set can be mathematically formulated as follows:

[0222] Consider an obstacle represented as a perfect circle with a centre at ^^^௫ , ^^௬^ andradius ^^ . The safe set ^^ for a point ^^^, ^^^ representing the position of the pursuer inrelation to this obstacle can be defined by the inequality:

[0223] ^^ ൌ ^^^^, ^^^ ∈ ℝଶ: ℎ^^^, ^^^ ^ 0^ (30)

[0224] where ℎ^^^, ^^^ ൌ ^^^ െ ^^ ଶ ଶ ଶ௫^ ^ ^^^ െ ^^௬^ െ ^^ .

[0225] The condition ℎ^^^, ^^^ ^ 0 ensures that the pursuer remains outside the obstacle,maintaining safety by avoiding collisions. However, upon analysing the chosen CBFcandidate, it becomes apparent that ^^^ℎ^^^, ^^^ ൌ 0, this characteristic categorizes thebarrier function as a "high relative degree" function. Specifically, the barrier function is ofthe second order relative to the system, with ^^ ൌ 2. Hence, we have followed to HOCBFapproach and implemented QP-HOCBF to derive a safe controller.

[0226] The core objective of this QP-HOCBF optimization is to identify a control commands (reference longitudinal and angular accelerations) that minimally deviates from the reference accelerations commanded from our baseline policy ,^^ௌ^^ெ^, while simultaneously ensuring that the system remains within the safe set defined by HOCBF, ^^, thus avoiding obstacles.

[0227] Furthermore, we have extended QP-HOCBF formulation to accommodate multi- obstacle scenarios by incorporating additional constraints for each obstacle,ℎ^^^^, ^^^, ℎଶ^^^, ^^^ℎ^^^^, ^^^, where ^^ denotes the total number of obstacles in theenvironment.

[0228] Additionally, in safety-critical systems, accounting for input constraints is vital, as saturation may impair the performance of automatic control subsystems. Nevertheless, incorporating control bounds into a QP can occasionally yield infeasible solutions. To mitigate this, we relaxed the QP-HOCBF optimization by introducing slack variables, ensuring the feasibility of solutions under all conditions.

[0229] The overall QP-HOCBF optimization problem is detailed below:

[0230]

[0231] Where, ^^ is a positive definite matrix that typically serves to weight the terms in the control inputs, ensuring a convex formulation. The variables ^^௨and ^^^are slack variables used to relax the upper and lower constraints on controls, respectively, allowing a feasible solution space when strict control bounds might not be directly achievable. are the coefficients used to balance the importance of the corresponding componentswithin the objective function. ^^ ∈ ℝଶ and ^^ ∈ ℝଶ௨^^^^ ^^௪^^ denotes the upper and lowerbounds for the controls.

[0232] Obstacle Avoidance for Non-Convex Shapes

[0233] The HOBF formulation (and the QP-based HOCBF in Equation (31) assumes circular obstacle shapes. However, in real maritime environments, obstacles often deviate from circular forms, and their locations may be initially unknown. For real†time detection, our USVs are equipped with a LiDAR sensor. This sensor emits laser rays in a known angular configuration, and each ray’s intersection point provides the obstacle’s range and bearing. Collecting these intersection points across all rays yields a discrete representation of the obstacle boundaries surrounding the vessel.

[0234] In this study, as a proof of concept, we have assumed that USVs are equipped with a LiDAR sensor which has a detection range of 60 meters with an angular configuration of േ60∘laser rays. As a practical solution for unknown, non-convex obstacles, the agent constructs virtual circular barriers centered at each laser-ray intersection point, and applies the same QP-HOCBF formulation from Equation 31. The radius of these virtual obstacles, ^^௩^^௧௨^^is chosen based on the USV’s dynamic characteristic; here, it is set to half the sensor range (30m), providing a practical safety margin for real-time obstacle avoidance.

[0235] Results

[0236] In this section, we present experimental results to evaluate our avoidance algorithm’s performance in a multi-obstacle environment. Specifically, we consider two main scenarios: (1) initially known, convex-shaped obstacles; and (2) initially unknown, irregularly shaped obstacles.

[0237] Case-1 Circular Obstacles

[0238] In this scenario, nine circular-shaped obstacles are placed in known locations. The SciML policy is combined with a HOCBF layer, configured such that the pursuer considers only the two nearest obstacles at any given time which reduces computational overhead for real-time applications. This allows the agent to navigate safely through the environment while maintaining its objective of capturing the evader.

[0239] With reference to Figs. 12a and 12b, the pursuer-evader trajectories may be compared under two settings: in Fig. 12a, the pursuer uses only the baseline SciML policy, while in Fig.12b, a HOCBF layer is integrated for obstacle avoidance. The evader employs a HOCBF-enhanced P-controller for illustration. Without the HOCBF layer, the SciML policy guides the pursuer through obstacles due to the absence of explicit obstacle avoidance, except for inter-agent collision avoidance encoded during policy training. In contrast, with the HOCBF layer, the pursuer adjusts its trajectory to avoid obstacles while effectively tracking the evader, even in complex environments.

[0240] With reference to Fig.13, the relative distance between the pursuer and evader steadily decreases and stabilizes within the safe capture zone, confirming successful and safe interception under known multi-obstacle conditions.

[0241] With reference to Fig. 14, the time evolution of both the SciML policy’s commanded reference accelerations (dashed blue lines) and the accelerations adjusted by the HOCBF layer (solid blue lines) for obstacle avoidance may be compared. The dashed red lines denote the vehicle’s dynamic control limits. Whenever the pursuer nears an obstacle, the HOCBF layer deviates from the SciML outputs, while still remaining within the feasible acceleration bounds. Otherwise, it closely follows the original policy commands, demonstrating that the HOCBF layer intervenes only as needed to ensure safe navigation.

[0242] To evaluate the proposed algorithm’s robustness under varying initial conditions and enable fair comparison with conventional methods, we conducted an extensive Monte Carlo study with 500 randomized trials. The baseline Artificial Potential Field (APF) method was carefully tuned through systematic parameter optimization to ensure equitable comparison grounds.

[0243] The artificial potential field (APF) formulation integrates target attraction and velocity-adaptive repulsion through:

[0245] where represent unit vectors from pursuer to evader and ^^-th obstacle respectively, ^^ୟ^^and ^^୰^୮govern attraction / repulsion balance, ^^^^^^denoteseffective obstacle radius with safety margin, ^^ ^ 1 controls repulsion gradient steepness,and introduces velocity-dependent tangential guidance for smooth obstacle circumvention. Control inputs for velocity and heading are derived from the resultant potential field through proportional-derivative (PD) techniques, ensuring smooth trajectory generation.

[0246] With reference to Fig. 15, the Monte-Carlo trajectories (solid lines denote the pursuer trajectories) may be visualised, while Table 2 summarizes key metrics.

[0247] Table 6: Performance Comparison Between Proposed and Baseline Methods Metric SciML + HOCBF Pure APF Number of Collisions 0 275 Mean Capture Time (s) 48.3 41.2

[0248] Note: Results averaged over 500 Monte Carlo trials with randomized initial conditions

[0249] Fig.15 and Table 6 show that the SciML + HOCBF policy outperforms the baseline APF. Across 500 Monte Carlo trials, it achieved zero collisions, whereas the APF-based controller failed in nearly half the runs. Although the SciML+HOCBF approach exhibits a slightly higher average capture time, this is expected since the statistics are computed only over successful captures. The gain stems from two factors: (i) the SciML training embeds USV dynamics and actuation limits, allowing the controller to slow down before high-rate turns, and (ii) the HOCBF layer projects these commands onto a safety-invariant set with minimal deviation. In contrast, the pure APF lacking both dynamic awareness and formal safety guarantees accelerates aggressively in cluttered scenes, causing repulsive-force deadlocks and collisions in roughly half the trials.

[0250] Case-2 Non-Convex Obstacles

[0251] In this scenario, two nonconvex-shaped obstacles are placed in the environment with unknown locations. The pursuer relies on an onboard LiDAR sensor for obstacle detection. Similar to Case 1, after identifying the intersection points between LiDAR rays and obstacles, only the two nearest points are used to reduce the computational load of the HOCBF. Virtual circular obstacles are then defined at these locations for avoidance.

[0252] Fig. 16 highlights the effectiveness of our proposed approach in a dynamic maritime environment containing unknown, non-convex obstacles. Despite having no prior knowledge of the obstacles’ locations or shapes, it continuously interprets sensor data to create virtual circular obstacles at critical detection points. This enables smooth, collision-free path adjustments while maintaining an effective pursuit velocity toward the evader. Moreover, the framework’s consistent success across varying initial conditions underscores its robustness which is an essential feature for real-world maritime deployments.

[0253] These results also demonstrate the strong generalization capability of the SciML training framework, attributed to the integration of known physics into the training process. In practice, the control adjustments made by the HOCBF layer which acts as a kind of disturbance to the original SciML policy were never encountered during training. Despite this, the SciML policy maintains robust performance, enabling the overall system to operate in harmony under unseen conditions.

[0254] This work introduced a novel framework combining Scientific Machine Learning with Control Barrier Functions for safe and effective maritime pursuit-evasion in cluttered environments. By embedding known USV dynamics into a universal ODE and layering HOCBF-based obstacle avoidance, our approach achieves real-time performance and provable safety even under partial observability and non-convex constraints. Extensive simulations demonstrate robust interception and reliable avoidance of both known and dynamically detected obstacles. Future work will explore hardware deployment and extensions to multi-agent and adversarial settings.

Claims

CLAIMS 1. A computer-implemented method of training a pursuer neural network agent, the method comprising: providing an evader neural network agent trained to update iteratively a plurality of dynamic parameters of an evader, a pursuer neural network agent initialised to update iteratively a plurality of approximators for a pursuer, and an ordinary differential equation solver configured to update a plurality of dynamic parameters of the pursuer based on the updated approximators; sampling initial dynamic parameters from a set of initial dynamic parameters of the evader; running a simulation involving the pursuer neural network agent pursuing the evader neural network agent based on the sampled initial dynamic parameters; computing a loss based on the dynamic parameters of the pursuer from the ordinary differential equation solver and the dynamic parameters of the evader from the evader neural network agent; and optimising weights of the pursuer neural network agent based on the computed loss.

2. The computer-implemented method of Claim 1, further comprising: separating the set of initial dynamic parameters of the pursuer into a plurality of mini batches, wherein sampling initial dynamic parameters from a set of initial dynamic parameters of the evader, running the simulation involving the pursuer neural network agent, and computing a loss based on the output from the pursuer neural network agent are performed for each mini batch of the plurality of mini batches of initial dynamic parameters of the pursuer.

3. The computer-implemented method of Claim 2, further comprising: computing a mean loss using the loss computed for each mini batch, wherein optimising the weights of the pursuer neural network agent based on the computed loss comprises optimising the weights of the pursuer neural network agent using the mean computed loss.

4. The computer-implemented method of any preceding claim, wherein running the simulation comprises: performing forward propagation of the evader neural network agent to update the plurality of dynamic parameters based on the sampled initial dynamic parameters; and performing forward propagation of the pursuer neural network agent to update the plurality of dynamic parameters based on initial dynamic parameters of the pursuer, the updated dynamic parameters from the evader neural network agent.

5. The computer-implemented method of any preceding claim, further comprising: providing initial dynamic parameters of the pursuer; dividing the initial dynamic parameters of the pursuer into mini batches; running the simulation for each mini batch; and computing the loss for each mini batch.

6. The computer-implemented method of Claim 5, further comprising: computing an average loss using the computed loss for all mini batches, wherein optimising the weights comprises optimising the weights of the pursuer neural network agent to minimise the average loss.

7. The computer-implemented method of any preceding claim, wherein the dynamic parameters of each of the evader and the pursuer comprise at least one of: position, heading angle, and velocity.

8. The computer-implemented method of any preceding claim, wherein computing a loss based on the dynamic parameters of the pursuer from the ordinary differential equation solver and the dynamic parameters of the evader from the evader neural network agent, comprises: computing a plurality of component losses; and computing the loss as a weighted sum of the plurality of component losses.

9. The computer-implemented method of Claim 8, wherein computing the loss as a weighted sum of the plurality of component losses uses a formula:calculating a first component loss ℒிof the plurality of component losses associated with an objective of the pursuer in following the evader using ℒிൌ^^^^^^^ െ ห^^^⃗^^^^^ െ ^^^⃗ா^^^^ห^, where ^^^^^^ is a desired tracking range, ^^^⃗^^^^^ ൌand^^^⃗ா^^^^ ൌwhere ^^^⃗^^^^^ is a position of the pursuer, ^^^^^^^ is a position of thepursuer along an x direction, ^^^^^^^ is position of the pursuer along a y direction, ^^^⃗ா^^^^ is a position of the evader, ^^ா^^^^ is a position of the evader along the x direction, and ^^ா^^^^ is a position of the evader along the y direction; calculating a second component loss ℒ^ாof the plurality of component losses using shows the∗௧^output of the pursuer neural network agent during the simulation, and N is a number of iterations during the simulation; calculating a third component loss ℒௌ^of the plurality of component losses usingdenotes the rectified linear unit (ReLU) function, ^^ is the pursuer’s heading angle and ^^௨^^^^, ^^^^௪^^are the upper and lower limits of the heading angle, ^^௫^,^^௬^and ^^௫ா,^^௬ாare the velocity components of the pursuer and evader along the ^^ and ^^ directions, respectively; calculating a fourth component loss ℒ^^of the plurality of component lossesusingstands forCollision Avoidance Range", calculating a fifth component loss ℒ^^of the plurality of component lossesusingand ^^^௬ா^ி^^^^ are ^^ and ^^ components of the relative position between the pursuer and the evader, expressed in the evader’s body frame, EBF, ^^^^ stands for "Desired Separation Distance", and ^^^^ is a mask function which assigns a value of 0 when the pursuer is positioned behind the evader, and a value of 1 when the pursuer is ahead of the evader.

10. The computer-implemented method of Claim 9, wherein computing theloss as a weighted sum of the plurality of component losses uses ℒ௧^௧^^ ൌ ^^ிℒி ^^^^ாℒ^ா ^ ^^ௌ^ℒௌ^ ^ ^^^^ℒ^^ ^ ^^^^ℒ^^, where ^^ி is first a weight associated with thefirst loss component ℒி, ^^^ாis a second weight associated with the second loss component ℒ^ா, ^^ௌ^is a third weight associated with the third loss component ℒௌ^, ^^^^is a fourth weight associated with the fourth loss component ℒ^^, and ^^^^is a fifth weight associated with the fifth loss component ℒ^^.

11. The computer-implemented method of any preceding claim, wherein the dynamic parameters include a heading angle, a position, and a velocity.

12. The computer-implemented method of Claim 11, wherein the evader neural network agent is trained to output a constant velocity and a constant heading angle of the evader.

13. The computer-implemented method of any preceding claim, further comprising: implementing a barrier function as a safety constraint at each time step of the simulation.

14. The computer-implemented method of Claim 13, wherein implementing the safety constraint comprises: implementing the safety constraint as a quadratic problem formulation.

15. A pursuer vehicle comprising: at least one control surface for moving the pursuer vehicle; at least one sensor for sensing at least one dynamic parameter of an evader vehicle; at least one processor; and storage having the pursuer neural network agent trained according to any of Claims 1 to 12 stored thereon, the pursuer neural network agent causing the at least one processor to control the at least one control surface to move the pursuer vehicle in response to the at least one dynamic parameter of the evader vehicle.

Images