Training a pursuer neural network agent

Abstract

A method of training a pursuer neural network agent. The method comprises: Providing an evader neural network agent trained to update iteratively a plurality of dynamic parameters of an evader, a purs

Description

FIELD

[01] The subject-matter of the present disclosure relates to computer-implemented methods of training a pursuer neural network agent, transitory or non-transitory computer-readable media, and pursuer vehicles. BACKGROUND

[02] The need for advanced security measures to protect critical infrastructure and maritime assets has revealed the inadequacies of traditional defense approaches in the marine environment. Integrating artificial intelligence (Al) and autonomous systems into Unmanned Surface Vehicle (USV) operations is essential to enhance the effectiveness and adaptability of maritime security strategies. These technologies significantly improve the strategic capabilities of USVs. Al-aided tactics for infrastructure protection using USVs are a critical application of these technologies, effectively modeled as a pursuitevasion game where the USV is the pursuer and an intruder vessel is the evader. Although pursuit-evasion strategies are still in their early stages of application for USVs, they have been extensively explored in other domains. Research in this area typically divides methods into two categories: conventional control techniques and machine learning approaches.

[03] The conventional approach involves applying classical control and optimization theory. One such methodology is the use of the Model Predictive Control (MPC) framework to design a controller. A study in utilized MPC for the USV pursuit and evasion game scenario and similar applications have been made in other robotic platforms. However, implementing the MPC methods in pursuit-evasion games necessitates numerous simplifications, resulting in diminished performance in real-world applications.

[04] Another method in the first category applies game theory, where the problem is treated as a min-max optimization problem and addressed based on cost definitions. Pursuit-evasion, a subtopic of game theory, is extensively studied across various application areas. For maritime domain, some prior art methods analyze how game theory is applied to various USV tasks. Other prior art methods investigate the pursuitevasion game problem for USVs based on the threat potential field (TPF). Similarly, game theory methodology has been investigated and applied for other platforms like robotics, unmanned aerial vehicles and unmanned underwater vehicles. Some other studies studied the pursuit-evasion of two aircraft in a horizontal planed and proposed a solution based on extremal trajectory maps. Another study solved the problem through modelling the aircraft control input limits, creating the Hamiltonian equation and solving the optimization problem numerically. Other studies looked into a multiplayer pursuitevasion game consisting of prey, predator, and protector for collision avoidance problem. Other studies handled the pursuit-evasion problem for two satellites considering their continuous thrust reachable domains. Other multiplayer differential game theory application looked at enabling multiple UAVs to engage in multiple targets. However, analytical solutions in game theory are feasible for simplified cases but challenging to implement in real-time. Deriving these solutions requires many assumptions based on extensive calculus, leading to reduced effectiveness in real-world applications, similar to classical methods

[05] In pursuit-evasion problems, our study’s second category explores machine learning methodologies. Recent advancements in computational technology enable the training and deployment of machine learning models for real-world applications. Many studies use machine learning, particularly Reinforcement Learning (RL), for pursuitevasion problems. RL is a trial-and-error-based learning approach where a machine autonomously learns optimal behavior or policies within a specific environment by maximizing or minimizing cumulative rewards or costs. An application of RL on maritime applications is to design a path following controller as the USV dynamics are usually not highly accurate and expose to high level of uncertainties. For the pursuit evasion problem, there have been several applications for training a pursuer, training an evader, co-operative hunting. RL for pursuit evasion game is also widely studied for robotics and UAVs for single / multi agent use cases. One advantage of RL methods is their suitability for fast real-time inference. However, a major limitation of RL methods is their training efficiency,which heavily depends on the design of reward functions, requiring careful reward engineering. Additionally, in continuous systems, the rapid expansion of the state space exploration complicates the search for global optimum solutions, representing a significant challenge in RL-based approaches.

[06] It is an aim of the present invention to address such problems and improve on the prior art. SUMMARY

[07] According to an aspect of the present disclosure, there is provided a computer-implemented method of training a pursuer neural network agent, the method comprising: providing an evader neural network agent trained to update iteratively a plurality of dynamic parameters of an evader, a pursuer neural network agent initialised to update iteratively a plurality of approximators fora pursuer, and an ordinary differential equation solver configured to update a plurality of dynamic parameters of the pursuer based on the updated approximators; sampling initial dynamic parameters from a set of initial dynamic parameters of the evader; running a simulation involving the pursuer neural network agent pursuing the evader neural network agent based on the sampled initial dynamic parameters; computing a loss based on the dynamic parameters of the pursuer from the ordinary differential equation solver and the dynamic parameters of the evader from the evader neural network agent; and optimising weights of the pursuer neural network agent based on the computed loss. Initial dynamic parameters may also be called initial conditions.

[08] Incorporating the ordinary differential equation solver into the loop results in the weights of the pursuer neural network agent being more optimised to pursue an evader performing unseen manoeuvres.

[09] In an embodiment, the computer-implemented method further comprises: separating the set of initial dynamic parameters of the pursuer into a plurality of mini batches, wherein sampling initial dynamic parameters from a set of initial dynamic parameters of the evader, running the simulation involving the pursuer neural network agent, and computing a loss based on the output from the pursuer neural network agent are performed for each mini batch of the plurality of mini batches of initial dynamic parameters of the pursuer.

[10] In an embodiment, the computer-implemented method further comprises: computing a mean loss using the loss computed for each mini batch, wherein optimising the weights of the pursuer neural network agent based on the computed loss comprises optimising the weights of the pursuer neural network agent using the mean computed loss.

[11] In an embodiment, running the simulation comprises: performing forward propagation of the evader neural network agent to update the plurality of dynamic parameters based on the sampled initial dynamic parameters; and performing forward propagation of the pursuer neural network agent to update the plurality of dynamic parameters based on initial dynamic parameters of the pursuer, the updated dynamic parameters from the evader neural network agent.

[12] In an embodiment, the computer-implemented method further comprises: providing initial dynamic parameters of the pursuer; dividing the initial dynamic parameters of the pursuer into mini batches; running the simulation for each mini batch; and computing the loss for each mini batch.

[13] In an embodiment, the computer-implemented method further comprises: computing an average loss using the computed loss for all mini batches, wherein optimising the weights comprises optimising the weights of the pursuer neural network agent to minimise the average loss.

[14] I n an embodiment, the dynamic parameters of each of the evader and the pursuer comprise at least one of: position, heading angle, and velocity.

[15] In an embodiment, computing a loss based on the dynamic parameters of the pursuer from the ordinary differential equation solver and the dynamic parameters of the evader from the evader neural network agent, comprises: computing a plurality of component losses; and computing the loss as a weighted sum of the plurality of component losses.

[16] In an embodiment, computing the loss as a weighted sum of the plurality of component losses uses a formula: calculating a first component loss £F of the plurality of component losses associated with an objective of the pursuer in following the evader using £F = |dt / ? - |PP(t) - PB(t)||, where DTR is a desired tracking range, Pp(t) = xp(ty yP(t) and PE(t) = xE(ty yEW) , where PP(t) is a position of the pursuer, xP(t) is a position of the pursuer along an x direction, yp(t) is position of the pursuer along a y direction, PE(t) is a position of the evader, xE(t) is a position of the evader along the x direction, and yE(t) is a position of the evader along the y direction; calculating a second component loss £CE of the plurality of component losses using r -lyw,„2 i . . 2> where NN^(X(ty shows the *tfl output = ^{j£^2Kx(tywNN)) of the pursuer neural network agent during the simulation, and N is a number of iterations during the simulation; calculating a third component loss £sc of the plurality of component £sc ^SCl ( / ^w(0(t) Gupper | + 1G(t^ "I" ^Zowerl|) losses using / 2 2\ where +*sc2((yntf)-VxE(trf) + (7 / (^)-7 / (^)) ) relu(*) denotes the rectified linear unit (ReLU) function, G is the pursuer’s heading angle and Gupper, Oiower are the upper and lower limits of the heading angle, Vx, Vy and 7 / , 1 / are the velocity components of the pursuer and evader along the x and y directions, respectively; calculating a fourth component loss £CA of the plurality of component losses using £CA = |z>C^7? - \relu(PP(t) - P£(t))||, where DCAR stands for "Desired Collision Avoidance Range", calculating a fifth component loss £AA of the plurality of component losses using £AA = ||(re / u(-P™F(r))|| + MQ * \\(relu(DSD - |PyBF(t)|)||, where RBBF(Q and RyBF(Q are x and y components of the relative position between the pursuer and the evader, expressed in the evader’s body frame, EBF, DS stands for "Desired Separation Distance", and MQ is a mask function which assigns a value of 0 when the pursuer is positioned behind the evader, and a value of 1 when the pursuer is ahead of the evader.

[17] In an embodiment, computing the loss as a weighted sum of the plurality of component losses uses £total = AF£F + ACE£CE + Asc£sc + Aca£ca + Aaa£aa, where AF is first a weight associated with the first loss component £F, ACE is a second weight associated with the second loss component £CE, Asc is a third weight associated with the third loss component £sc, Aca is a fourth weight associated with the fourth loss component £CA, and Aaa is a fifth weight associated with the fifth loss component £AA.

[18] In an embodiment, the dynamic parameters include a heading angle, a position, and a velocity.

[19] In an embodiment, the evader neural network agent is trained to output a constant velocity and a constant heading angle of the evader.

[20] According to an aspect of the present disclosure, there is provided a transitory, or non-transitory, computer-readable medium having instructions stored thereon that, when executed by at least one processor, causes the at least one processor to perform the computer-implemented method of any preceding aspect or embodiment.

[21] According to an aspect of the present disclosure, there is provided a pursuer vehicle comprising: at least one control surface for moving the pursuer vehicle; at least one sensor for sensing at least one dynamic parameter of an evader vehicle; at least one processor; and storage having the pursuer neural network agent trained according a method of any preceding aspect or embodiment stored thereon, the pursuer neural network agent causing the at least one processor to control the at least one control surface to move the pursuer vehicle in response to the at least one dynamic parameter of the evader vehicle. BRIEF DESCRIPTION OF DRAWINGS

[22] The subject-matter of the present disclosure is best described with reference to the accompanying figures, in which:

[23] Figure 1 shows a perspective view illustrating scenario relating to infrastructure invasion;

[24] Figure 2 shows illustrations of example unmanned surface vehicles (USVs) with vehicle dynamic limit trends;

[25] Figure 3 shows an ordinary different equation solver according to one or more embodiments;

[26] Figure 4 shows a flowchart detailed policy development associated with scientific machine learning;

[27] Figure 5 shows a plurality of graphs illustrating time-elapsed trajectories of pursuer and evader from a scenario, e.g. scenario 1;

[28] Figure 6 shows a graph of relative distance between a pursuer and an evader over time for another scenario, e.g. scenario 1;

[29] Figure 7 shows graphs of scientific machine learning policy control actions over time for scenario 1;

[30] Figure 8 shows a plurality of graphs illustrating a time evolution of the policy’s commanded reference control actions and the realised actions in scenario 2;

[31] Figure 9 shows a graph illustrating relative distance between the pursuer and the evader over time for scenario 2;

[32] Figure 10 shows a plurality of graphs illustrating a time evolution of the scientific machine learning policy control actions over time for scenario 2;

[33] Figure 11 shows a plurality of graphs illustrating time-elapsed trajectories of pursuer and evader from scenario 2. DESCRIPTION OF EMBODIMENTS

[34] In this disclosure, the problem being addressed is how to develop an Al-aided tactics generator for uncrewed surface vessels (USVs) for protection of critical national infrastructure 10 and maritime assets 12 in face of surface vehicle attacks as illustrated in Figure 1.

[35] The general mission profile for the pursuer USV includes initially either waiting near the protected platform or patrolling. When an approaching evader is localised and identified, the pursuer USV initiates the pursuing policy to intercept and neutralize the evader.

[36] To accomplish this mission, an Al-aided defense strategy must overcome several challenges. The pursuer policy must navigate the USV to neutralize the evader in minimum time for protection. The steering commands must adhere to the vehicle’s dynamic limits. Given the safety-critical nature of the mission, a safe distance must be maintained while tracking to avoid collision with the evader. When facing a weaponized evader USV, it is necessary to approach from behind (astern approach) to avoid defeat. Additionally, the defense strategy must be capable of tracking and countering intelligent maneuvers by the evader.

[37] The USVs are assumed to exhibit point mass characteristics. The equations motion model for an unmanned surface vehicle relative to inertial reference frame centered at the sea surface, considering 2 degrees of freedom, are expressed as follows: x = 7cos0 y = Vsin0

[38] V = av (1) e = w

[39] In this motion model, x and y denote the vehicle’s position coordinates, V represents represents velocity magnitude, and 6 indicates the heading angle. The control inputs av and w correspond to longitudinal acceleration and angular velocity, respectively.

[40] The maneuver capabilities of both pursuer and evader unmanned surface vehicles, defined by their ability to change velocity and heading, are constrained by dynamic models that are functions of current velocity of the vehicle. Figure 2 features illustrative examples of these dynamic constraints alongside several USVs currently on the market, intended purely for illustrative purposes.

[41] In this section, we present the methodology and framework for developing Al-aided defense tactics for a pursuer USV against an evader. The baseline pursuing policy is formulated using scientific machine learning.

[42] The principle of integrating governing physics into the training processes of machine learning models is to enhance their generalization capabilities. In accordance with this approach, the concept of scientific machine learning, implemented using Universal Ordinary Differential Equations (UODEs), has been proposed and formalized in for the purpose of scientific model discovery.

[43] General training framework for SciML is illustrated in Figure 3.

[44] The main idea of the UODEs involves formulating an ordinary differential equation that incorporates a mechanistic model with a universal approximator to estimate unknown dynamics. Specifically, a simple first-order ODE defined by the ground truth model = / (f, t) can be represented in UODE form as = gktf, t) + gu(*), where the known part of the ground truth system is denoted with the function of gk^, t) and #u(*) represents the unknown dynamics that we want to discover. This unknown component is approximated with a universal approximator (also merely called an approximator), specifically a neural network, expressed as gu(*) « NN(X, WNN). During each forward propagation, the ODE system is simulated by solving it, concluding with the calculation of a scalar loss function, such as the mean square error between the ODE solution and real-world experimental data that represents the ground truth model. In backward propagation, continuous-time gradients of the network parameters with respect to the loss function are computed using adjoint sensitivity methods. After gradient computation, network parameters are updated using gradient descent methods.

[45] In addressing the USV tactic generation problem, we have reformulated our pursuit-evasion problem and treating it as a scientific model discovery problem similar to framework described. The motion model presented earlier is deconstructed into components of known and unknown dynamics. Pursuer’s defence tactic generation policy meaning the controls of the vehicle is conceptualized as a learnable, unknown dynamical model. It is modelled utilizing an artificial neural network parameterized with including the network weights and biases.

[46] WW = WNN) (2)

[47] In Equation 2, X(r) represents the feature vector of the neural network policy model, which will be designed accordingly to performance objectives described in subsequent sections.

[48] Consistent with the scientific model discovery approach described, the mechanistic motion model in Equation is reformulated to include both known and unknown dynamics as follows:

[49] Sfc (*(£)) Su(X(t),M / NN) Vcos0’ Vsin0 0 .0 + O o o' l-^OOO 1..............................................................................................1 ■3£^^(X(t-),WNNy (3)

[50] Where, In "*" shows the *th output of the neural network policy model.

[51] In the UODEs framework detailed in Equation 3, learning the decision-making policy model for the pursuer involves training the artificial neural network. To do so, we need to define a performance index to enforce the pursuer for following the evader while also meeting the mission objectives. Accordingly, we have developed the following loss function to serve this purpose.

[52] £total — Ap£p + ^CE^CE + ^SC-^SC + ^CA^CA + &AA&AA (4)

[53] In Equation 4, we have presented the total loss function that includes various components crafted for specific objectives and aggregated using a weighted sum approach. The component £F is associated with the pursuer’s objective to follow the evader. £CE accounts for the control effort loss. £sc addresses the pursuer’s state constraints, including maximum and minimum state limits. £CA focuses on avoiding collisions between the pursuer and the evader. Additionally, we have used £AA component to guide the pursuer to execute the astern approach maneuver while intercepting the evader. The coefficients Af,Ace,Apc,Aca and Aaa scale the importance of these components within the optimization process.

[54] In other words, methods described herein, specifically in relation to Algorithm 5, include computing a plurality of component losses, a computing the loss (i.e. the total loss) as a weighted sum of the plurality of component losses. The plurality of component losses includes a first component loss £F, a second component loss £CE, a third component loss £sc, a fourth component loss £CA, and a fifth component loss £AA. Similarly, the coefficients Af,Ace,Apc,Aca and Aaa may be understood to mean the first to fifth weights, respectively.

[55] The primary objective of the pursuer is to follow and intercept the evader in the shortest possible time. To align with this goal, we have considered that minimizing the distance between the pursuer and the evader is crucial. However, it is also crucial for the pursuer to maintain a safe tracking distance from the evader to avoid collisions. Consequently, the loss component £F can be expanded as follows:

[56] Lf = \DTR- \PP(t) -Pe(t)|| (5)

[57] Where, Pp(t) '^(01 yP(0 and PE(t) xE(ty yE(t) represent the position vectors of the pursuer and evader, respectively. DTR stands for the "Desired Tracking Range".

[58] To align with the objective of minimizing capture time, we have included the minimization of angular velocity control effort in our loss definition. Minimizing control effort helps guide the pursuer to take the most efficient action, steering it toward the evader along the shortest possible path. r _ 1 VW ..,2 ^CE — wref

[59] "2 (6)

[60] For state constraints, we have implemented continuous upper and lower bounds on the heading angle to prevent the pursuer from rotating around itself with divergent behavior during training iterations. This approach helps to ensure a more stable training process. Additionally, we have included endpoint constraints on velocity and its direction to further guide the pursuer’s movement and enhance the effectiveness of the training. £sc ^SCl ( ^upper | + 0(0 + ^Zowerl|)

[61] z 2 2X (7) +^2((^)-^)) +{VyP^-VyE(tf^ )

[62] Where, relu^ denotes the rectified linear unit (ReLU) function, which outputs the input value if the input is greater than zero; otherwise, it outputs zero. 0 is the pursuer’s heading angle and 0upper, 0iower are the upper and lower limits of the heading angle. Furthermore, VP,VP and Vx, Vy are the velocity components of the pursuer and evader along the x and y directions, respectively.

[63] Considering safety requirements, we have implemented the collision avoidance loss function between the pursuer and the evader as follows:

[64] £ca = \DCAR - \relu(PP(t) - PE(t»\\ (8)

[65] Where, DCAR stands for "Desired Collision Avoidance Range" which is determined by considering the size information of the example USVs involved.

[66] To successfully execute an astern approach for intercepting the evader, we have designed a loss function that operates within the evader’s body reference frame. This frame of reference provides key information about the relative positioning of the pursuer, specifically indicating whether the pursuer is ahead of or behind the evader. This orientation is essential for adjusting the pursuer’s strategy to either close in from behind or adjust its path to move into an astern position effectively.

[67] ^aa = ||(reM-^BF(0)ll +M() *\\(relu(DSD - |^BF(t)|)|| (9)

[68] Where, RfBF(t) and RBBF(t) are the x and y components of the relative position vector between the pursuer and the evader, expressed in the evader’s body frame (EBF). DS stands for "Desired Separation Distance". Additionally, MQ is a mask function which assigns a value of 0 when the pursuer is positioned behind the evader, and a value of 1 when the pursuer is ahead of the evader. {i tfnEBF (■<-> a 1, W_u (10) 0, otherwise

[70] From Equation 9, astern approach loss function £AA consists of two components. The first component aims to guide the pursuer to decrease the distance to the evader and follow from behind, penalizing the pursuer when it is positioned in front of the evader. The second component ensures that the pursuer maintains a safe separation (DSD) from the target line when it is in front of the evader, addressing safety requirements. Together, these components effectively constitute the astern approach maneuver.

[71] To incorporate vehicle dynamics as control constraints, we have approximated the system dynamic limits using shallow neural networks (One Hidden Layer Networks) These networks take the vehicle velocity as input and output the corresponding vehicle dynamic limits. The training utilizes a dataset derived from real-world experiments involving an experimental USV. Furthermore, physical constraints are included as a Lagrangian penalty in the optimization objective, following the physics-informed neural networks framework, to ensure the limit functions behave well and generalize effectively. An example objective function for training the longitudinal acceleration upper limit model is provided below.

[72] J MSE(ysynthetic y) + Arelu( y) (11)

[73] The equation utilizes MSEQ to denote the mean squared error between, ysynthetic and y representing the synthetic data from the vehicle dynamics model and the network approximation value, respectively. The coefficient serves as a loss weight, balancing the importance within the objective function. The reluQ function is employed to penalize the model, preventing negative values for the acceleration upper limit.

[74] After approximating the vehicle dynamics models with shallow neural networks, we integrated the system dynamics into the scientific machine learning training process by employing a neuron-wise adaptive tanh activation function in the final layer of the policy network. pnax_p-nax

[75] Adaptive Tanh- = gnax+e_nax (12)

[76] The main idea of adaptive activation functions is to make them trainable to enhance their ability to model nonlinear behavior. In Equation 12, a is the trainable parameter used to adjust the slope of the tanh activation function. Moreover, Ameya et al. highlighted that adding a non-trainable scaling factor, n, into the equation contributes to the convergence speed. These adaptive activation functions can be applied either layer-wise, where a single trainable parameter is used uniformly across the layer, or neuron-wise, with each neuron having its own independent trainable parameter, matching the number of neurons in the layer.

[77] This choice of activation function allows for dynamic scaling and adjustment of outputs, ensuring that the network’s response remains within the physically plausible limits dictated by the vehicle’s dynamics.

[78] Employing the neuron-wise adaptive Tanh activation function in the final layer of the policy network effectively bounds the output variables within the range [^1,1]. Subsequently, a dynamic re-scaling operation maps this [a"’l,l] interval to the current minimum and maximum limits as determined by the output of the trained vehicle dynamic’s limit networks. This mechanism matches the real-world limit values, thus integrating the policy outputs directly with the system’s dynamic constraints and preventing any violation of these limits. The re-mapping operation is given by the following formula. Kiir 2

[79] (13) 2 Wref =

[80] ! wma\v(ty)+wmin(y(ty) (14)

[81] Where, .a^71 ,wmax ,wmin are the corresponding trained neural network models used for approximating the maximum and minimum longitudinal acceleration and angular velocity limits of the vehicle, respectively.

[82] After establishing our performance index through specific loss functions and integrating vehicle dynamics, we conducted our scientific machine learning optimization processes using the Julia programming language. This choice was made due to Julia’s high performance in numerical and scientific computing, effectively facilitating complex computations and model training.

[83] To perform the optimization loop, we defined optimization scenarios that establish the initial conditions and movement behaviors for both the pursuer and the evader.

[84] For training purposes, we have considered to train our pursuer agent against a simple evader, which lacks intelligence or the capability to perform maneuvers during simulations. This simplification means that throughout a simulation, the evader maintains a constant speed and heading angle.

[85] Specifically: • The initial position of the evader is stochastically sampled from all quadrants within a 500-meter length square centered at the origin, denoted as e 11(-500171,500m). • The initial velocity magnitude of the evader is sampled from a uniform distribution Vtf e U(1-,17 j) and remains constant throughout a single simulation. • The initial heading angle of the evader is sampled from a uniform distribution Bq e 71(-180^180°) and remains constant throughout a single simulation.

[86] In this way, the evader is trained to output a constant velocity and a constant heading angle of the evader.

[87] For the pursuer, we created a structured initial conditions grid: • The initial position of the pursuer is distributed over 10 equally spaced intervals on circles centered at the origin with radius of 50 meters and 100 meters, resulting in a total of 20 distinct initial positions. • The initial velocity magnitude of the pursuer is divided into 5 evenly spaced samples within the range [1-,17-] • The initial heading angle of the pursuer is divided into 10 evenly spaced samples spanning [-180°, 180°]

[88] This configuration of initial conditions for the pursuer generates a grid by taking permutations of the states, resulting in 2000 distinct initial condition combinations.

[89] The overall policy optimization framework, which utilizes SciML training for our pursuit-evasion formulation, is illustrated in Figure 2 and detailed in Algorithm 5.

[90] Algorithm 5

[91] Input: ODE Solver(ODE, IC’s),

[92] Epoch Size: N, Mini-Batch Size: M

[93] Data: Initial Conditions Grid for Pursuer

[94] lnitialization:lnitilization of Network Weights, eNN Optimization Variable:, WNN epoch = 1,2,..., N

[95] Generate Minibatches Minibatch = 1,2,...

[96] Pursuer IC’s in Minibatch

[97] xE(t0) <- Sample the Evader’s Initial Condition

[98] x(t)i,u(t)i <- Run the simulation^xE(t0),WNN)

[99] Compute Loss, £t <- %(t);

[100] Compute Gradient, ( d£i ) \dWNN / 1

[101] Compute Mean Loss, £mean = Compute Mean Gradient-. L J \dWNNJmean M l l\dWNNJt Gradient Descent with Adam Optimizer:

[103] ^updated tmean, Update Network Weights, WNN ^dWNNJmean ^updated

[104] The SciML training flowchart in Figure 4 and corresponding Algorithm 5 can be briefly explained as follows: The initial conditions’ training grid for pursuer states is given as input to the algorithm. The first step of the algorithm involves initializing the weights of the neural network that models the decision-making policy. After successful initialization, mini batches are randomly formed from the initial condition grid created for the pursuer agent. Then, a simulation is run for each initial condition in the mini batches. For each mini-batch simulation, the initial condition states for the evader are sampled stochastically as explained. Subsequently, the defined loss functions are computed along with the policy gradient with respect to total loss given in Equation 4. After completing all the initial conditions in a mini batch, the average loss and gradient calculations are performed. This makes conducting gradient descent over the policy network possible. After updating the network weights for all mini batches, a new epoch starts. At the beginning of a new epoch, mini batches are created randomly. The same steps are followed until convergence is achieved.

[105] In other words, Algorithm 5 summarises a computer-implemented method of training a pursuer neural network agent. The method may be summarised as providing an evader neural network agent trained to update iteratively a plurality of dynamic parameters of an evader, a pursuer neural network agent initialised to update iteratively a plurality of approximators for a pursuer, and an ordinary differential equation solver configured to update a plurality of dynamic parameters of the pursuer based on the updated approximators. The dynamic parameters of each of the evader and the pursuer comprise at least one of position, heading angle, and velocity.

[106] The method further comprises providing initial dynamic parameters of the pursuer. The initial dynamic parameters may also be called initial conditions, ICs. The method may comprise dividing the initial dynamic parameters of the pursuer into mini batches.

[107] The method may comprise sampling the initial dynamic parameters from a set of initial dynamic parameters of the evader, running a simulation involving the pursuer neural network agent pursuing the evader neural network agent based on the sampled initial dynamic parameters, computing a loss based on the dynamic parameters of the pursuer from the ordinary differential equation solver and the dynamic parameters of the evader from the evader neural network agent, and optimising weights of the pursuer neural network agent based on the computed loss. More specifically, the method may comprise separating the set of initial dynamic parameters of the pursuer into a plurality of mini batches, wherein sampling initial dynamic parameters from a set of initial dynamic parameters of the evader, running the simulation involving the pursuer neural network agent, and computing a loss based on the output from the pursuer neural network agent are performed for each mini batch of the plurality of mini batches of initial dynamic parameters of the pursuer.

[108] Since the simulation may be run for each mini batch, the loss may also be computed for each mini batch. For example, the method may comprise computing an average loss using the computed loss for all mini batches, wherein optimising the weights comprises optimising the weights of the pursuer neural network agent to minimise the average loss. More specifically, computing a mean loss using the loss computed for each mini batch, wherein optimising the weights of the pursuer neural network agent based on the computed loss comprises optimising the weights of the pursuer neural network agent using the mean computed loss.

[109] running the simulation comprises: performing forward propagation of the evader neural network agent to update the plurality of dynamic parameters based on the sampled initial dynamic parameters; and performing forward propagation of the pursuer neural network agent to update the plurality of dynamic parameters based on initial dynamic parameters of the pursuer, the updated dynamic parameters from the evader neural network agent.

[110] This algorithm may be provided as a set of instructions stored on non-transitory computer-readable media that, when executed by at least once processor, causes the at least one processor to perform the computer-implemented method described here. The instructions may also be provided as transitory computer-readable media when being loaded onto the non-transitory storage media.

[111] The resulting pursuer neural network agent may be installed onto a pursuer vehicle comprising at least one control surface for moving the pursuer vehicle, at least one sensor for sensing at the at least one dynamic parameter of an evader vehicle, at least one processor, and storage having the pursuer neural network agent stored thereon. In this way, the pursuer neural network agent may cause the at least one processor to control the at least one control surface to move the pursuer vehicle in response to the at least one dynamic parameter of the evader vehicle

[112] In this section, we provide a brief overview of the final form of the policy architecture, hyperparameters used in the training processes and present the results from various simulation scenarios. Our simulations includes several cases: a simple evader heading directly towards infrastructure, a scenario incorporating actuator delay, and a scenario featuring a maneuvering evader. These settings help to evaluate the robustness and efficiency of our developed policy under different conditions.

[113] As outlined in Equation 2, we have modeled pursuer agent’s policy by using a neural network architecture. The loss components, defined in Equations 5-9 based on mission requirements, contribute to designing the feature vector X(t) e 5?11. This vector contains essential information for the agent’s decision-making at time t.

[114] X(t) = Rp2E^ II^COII Ve^

[115] The vectors RP2E(t), Rp^e^) denote the relative distance between the pursuer and the evader, expressed in the inertial reference frame and the evader body fixed reference frame, respectively. RP2E(t) indicates the time derivative of the relative position vector, highlighting the dynamic distance changes between pursuer and evader over time. The velocity vectors of the pursuer and evader are denoted as VP(t) and with ||Pp(t)|| specifying the magnitude of the pursuer’s velocity. Lastly, AL0Sp2E(t) represents the rate of change of the line-of-sight angle between the pursuer and the evader, capturing the angular dynamics of their engagement.

[116] The specific details of the policy architecture, including layer configurations, activation functions, and other number of parameters, are summarized in Table 1

[117] Table 1: Neural network polic architecture .Layer ijp i units AOfestion Input Scaling 1 1 Laver 1 Dense 11^111 lank Layer 2 Dense n I * w WnB Layer 3 'Dense Output Re-Scalmg nx* 2- 2

[118] Table 1 shows that the neural network architecture consists of three layers, each with 16 neurons (¾ = n2 = 16), and includes Scaling and Re-Scaling layers at the input and output stages, respectively. Min-max scaling is applied for the scaling operation, and re-scaling follows the method described in Section 3.4 at Equation 14. The activation function for each layer is the tanh function, except for the last layer, which uses the NA -tank functiona€”a neuronwise adaptive tanh function.

[119] SciML’s policy training process involves tuning hyper-parameters such as cost coefficients, mini-batch size, simulation time, and learning rate. We have used the grid 5 search algorithm for this purpose. The complete training setup and the optimized hyperparameter values are summarized in Table 2.

[120] Table 2: Training Setup and Hyperparameters Object Tsh:5 (TsHnums 5'4 Runge-Kutta method? Adjoint Method —* Inleqxdating Adjoint ODE Solver At — Variole Time * Absolute / Rduuvc lokidsive* = 10”^ .................. Cost Ceettdeats              ......Aca..... I '       Ascn - on — 0<d: Cost Parameters                _ .,.. P5p = 2(i,w ...................................................... Opfimher Mlm-Baidi Size. .V “ 1 .txammg Rak\ -- 0.0003 Mmtieotum Wcw 0 ™ (0,.9,0.999)

[121] Simple evader case 1 10

[122] In this section, we present the simulation results of a pursuer using the SciML policy against a simple evader, which directly approaches the infrastructure at a constand speed and heading without any strategic maneuvering. The initial conditions for both the pursuer and evader are summarized in Table 3.

[123] Table 3: Scenario-1 Initial Conditions Setup Vehicle         Inidai Conditions । — n ™ j i H j > ? j 700, Evader 1 7, (X j .... ■■ — 15 '

[124] Figure 5 displays the time-elapsed trajectories of both the pursuer and evader, with the pursuer’s path in blue and the evader’s in orange, as the evader heads straight for the infrastructure. Figure 6 illustrates the changing distance between the pursuer and evader over time. The simulation results show that the pursuer, guided by our trained policy, successfully maneuvers towards the capture zone and maintains tracking of the evader. A key aspect of performance highlighted here is that the pursuer executes an astern approach to maintain safety. Additionally, Figure 6 indicates that the pursuer keeps a safe distance, minimizing collision risks. These findings confirm that the trained policy meets the defined performance requirements effectively.

[125] Figure 7 shows the SciML policy’s commanded reference longitudinal acceleration and angular velocity over time. The blue lines represent the control action values generated by the policy network, while the red dashed lines show the system dynamics’ upper and lower boundaries. The data are normalized to the range of [-1,1], with +1 and -1 representing the upper and lower limits, respectively. This normalization is due to confidentiality reasons. The results demonstrate that the trained policy operates within the vehicle’s dynamic limits, with no commands exceeding these boundaries, which is a direct outcome of the vehicle dynamics integration strategy employed. Additionally, the action values from our policy exhibit a bang-bang control type, typical of minimum time control laws, indicating that the policy functions as an optimal controller.

[126] Simple evader scenario 2

[127] In this scenario, we followed the same setup as in Scenario-1, with the initial conditions of a simple evader summarized in Table 5. However, to assess the generalization capability of our approach, we introduced actuator delay into the system dynamics as a form of uncertainty. It is important to note that the policy optimization process did not originally account for actuator delay.

[128] Table 4: Pursuer and simple evader elapsed time trajectories for scenario-1 SCENARIO-1 CONDITIONS $Frt.'P Vehicle Irthhil ( oiHlitiuns_____ .F<} i / o = 0 Pursuer           1¼ ~ .tn - -- fW.w Evader C, = IE — - 130'

[129] The reference longitudinal acceleration and angular velocity commanded by the SciML policy are subject to a first-order dynamic actuator delay, modeled as follows: aXc0M = wC0M = ^^2\X(t),WNN-)

[130] ax= (16) . _ (wCOM-w') W — ----- T

[131] In Equation 16, aXc0M and wCOM represent the commanded control actions from the SciML policy. The variables ax and wdenote the actual realized control actions, which are affected by actuator delay characterized by a time constant t = 1 seconds.

[132] Figure 5 displays the time-elapsed trajectories of the pursuer and evader in Scenario-2, with the pursuer’s path marked in blue and the evader’s in orange. Figure 7 tracks the relative distance between the pursuer and evader over time. Additionally, Figure 8 illustrates the time evolution of the policy’s commanded reference control actions and the realized actions in Scenario-2. The blue lines represent the reference control action values generated by the policy network, while the orange lines show the realized control actions affected by actuator delay dynamics. Despite the the presence of significant actuator delay, which was not accounted for during the policy training, the simulation results show that the pursuer, guided by our trained policy, effectively approaches the evader and skillfully executes the astern approach maneuver. This demonstrates the generalization capability and the robustness of the policy in handling dynamics not explicitly included in its training.

[133] Figure 8 illustrates the time evolution of the policy’s commanded reference control actions and the realized actions in Scenario-2. The blue lines represent the reference control action values generated by the policy network, while the orange lines show the realized control actions affected by actuator delay dynamics. The red dashed lines indicate the boundaries set by the system dynamics. Despite the presence of actuator delays, the trained policy demonstrates effective generalization capability. The commands generated by the policy adapt well to the dynamics encountered, effectively fulfilling the mission requirements.

[134] Although there is a performance degradation compared to the Scenario-1 results, it remains within acceptable levels, as no unstable behavior is observed. The pursuer effectively tracks the evader and manages to avoid collision risks. However, unlike in Scenario-1, the pursuer does not maintain a consistent distance within the capture zone after executing an astern approach, due to the actuator delay impacting its ability to accelerate promptly. This performance loss could be eliminated by incorporating delay dynamics into the training processes, which would allow the policy to better handle delays.

[135] Manoeuvring Evader: Scenario-3

[136] To further assess the generalization capability of our trained SciML policy, we conducted tests against an evader executing an S-curve-like trajectory, representing an out-of-distribution scenario. Since, this type of evader trajectory was not encountered during the training phase; instead, as previously mentioned, the policy was trained against simpler evaders maintaining constant velocity and heading throughout a single simulation. This testing scenario helps evaluate how well the policy adapts to complex and unexpected maneuvers from the evader.

[137] Figure 11 illustrates the elapsed time trajectories of the pursuer and evader, showing the paths from the initial to the final time. The pursuer’s path is depicted by the blue line, while the evader’s path, following a predefined S-curve-like trajectory, is shown in orange. Despite the pursuer’s policy not being specifically trained to handle maneuvering evaders, the simulation results demonstrate the policy’s exceptional generalization ability. It successfully intercepts the evader and maintains effective tracking throughout the scenario.

[138] In this research, we addressed the USV tactic generation problem for protection of critical national infrastructure and maritime assets in face of surface vehicle attacks. We have conceptualized the pursuit-evasion problem as a scientific model discovery problem, wherein the tactic generation policy is treated as a learnable, unknown dynamical model that governs the relative system state.

[139] Our approach has shown outstanding performance through extensive simulations, even in scenarios that significantly deviate from the training domain. Moreover, our approach effectively mitigates various challenges, including achieving near-optimal minimum time-to-capture, diminishing the reliance on extensive training datasets, enhancing convergence speed, and lowering computational demands. We illustrate the results across a wide range of applicable scenarios.

[140] These results underscore the potential of scientific machine learning to revolutionize the field of pursuit-evasion and, by extension, the broader domain of security and defense. The methodology holds promise for various security and defense mechanisms, offering a scalable solution to many similar pursuit-evasion problems including one to many and many to many scenarios in both defensive and offensive setting. Currently, we are focused on implementation of this tactic generator on actual USV platforms to demonstrate the real-life performance of the proposed approach.

[141] As a future study, several considerations arise. Firstly, although we have demonstrated the robustness of our trained policy in effectively capturing an maneuvering evader, testing and refining our algorithm against an intelligent evader would be a valuable next step. This could involve training our agent to confront such evaders using a more sophisticated area defense strategy. Additionally, developing an intelligent mission planner that dynamically switches tactics based on the evadera€™s behavior could greatly enhance operational adaptability and effectiveness. Another important area for further study involves training a pursuer to navigate multi-obstacle environments, balancing obstacle avoidance with the primary goal of capturing the evader. These considerations will help advance the capabilities of our systems in complex and dynamically changing scenarios.

Claims

1. A computer-implemented method of training a pursuer neural network agent, the method comprising:providing an evader neural network agent trained to update iteratively a plurality of dynamic parameters of an evader, a pursuer neural network agent initialised to update iteratively a plurality of approximators for a pursuer, and an ordinary differential equation solver configured to update a plurality of dynamic parameters of the pursuer based on the updated approximators;sampling initial dynamic parameters from a set of initial dynamic parameters of the evader;running a simulation involving the pursuer neural network agent pursuing the evader neural network agent based on the sampled initial dynamic parameters;computing a loss based on the dynamic parameters of the pursuer from the ordinary differential equation solver and the dynamic parameters of the evader from the evader neural network agent; andoptimising weights of the pursuer neural network agent based on the computed loss.

2. The computer-implemented method of Claim 1, further comprising:separating the set of initial dynamic parameters of the pursuer into a plurality of mini batches,wherein sampling initial dynamic parameters from a set of initial dynamic parameters of the evader, running the simulation involving the pursuer neural network agent, and computing a loss based on the output from the pursuer neural network agent are performed for each mini batch of the plurality of mini batches of initial dynamic parameters of the pursuer.

3. The computer-implemented method of Claim 2, further comprising: computing a mean loss using the loss computed for each mini batch, wherein optimising the weights of the pursuer neural network agent based on the computed loss comprises optimising the weights of the pursuer neural network agent using the mean computed loss.

4. The computer-implemented method of any preceding claim, wherein running the simulation comprises:performing forward propagation of the evader neural network agent to update the plurality of dynamic parameters based on the sampled initial dynamic parameters; andperforming forward propagation of the pursuer neural network agent to update the plurality of dynamic parameters based on initial dynamic parameters of the pursuer, the updated dynamic parameters from the evader neural network agent.

5. The computer-implemented method of any preceding claim, further comprising:providing initial dynamic parameters of the pursuer;dividing the initial dynamic parameters of the pursuer into mini batches; running the simulation for each mini batch; andcomputing the loss for each mini batch.

6. The computer-implemented method of Claim 5, further comprising: computing an average loss using the computed loss for all mini batches, wherein optimising the weights comprises optimising the weights of the pursuer neural network agent to minimise the average loss.

7. The computer-implemented method of any preceding claim, wherein the dynamic parameters of each of the evader and the pursuer comprise at least one of: position, heading angle, and velocity.

8. The computer-implemented method of any preceding claim, wherein computing a loss based on the dynamic parameters of the pursuer from the ordinary differential equation solver and the dynamic parameters of the evader from the evader neural network agent, comprises:computing a plurality of component losses; andcomputing the loss as a weighted sum of the plurality of component losses.

9. The computer-implemented method of Claim 8, wherein computing the loss as a weighted sum of the plurality of component losses uses a formula:calculating a first component loss £F of the plurality of component lossesassociated with an objective of the pursuer in following the evader using £F =\dtr - |PP(t) - PE(t)11, where DTR is a desired tracking range, Pp(t)xP(ty yP(t) and, where PP(t) is a position of the pursuer, xP(t) is a position of thepursuer along an x direction, yP(t) is position of the pursuer along a y direction, P£(t) is a position of the evader, xE(t) is a position of the evader along the x direction, and yE(t) is a position of the evader along the y direction;calculating a second component loss £CE of the plurality of component losses r — w2^CE — „ £o WREF , .using N. 2, where NN&(X(t), WNNy "*" shows the = -So*th output of the pursuer neural network agent during the simulation, and N is a number of iterations during the simulation;calculating a third component loss £sc of the plurality of component losses£sc Asci f |reZu(0(t) ^upper I + I+ ^lower 1| Jusing 2 2 where relu(*)+^sC2((yxp(tf)-vxEQtf)) +(vy\tf)-vyE(tf)))denotes the rectified linear unit (ReLU) function, 6 is the pursuer’s heading angle and Supper, Slower are the upper and lower limits of the heading angle, Vx , Vp and VE, VE are the velocity components of the pursuer and evader along the x and y directions, respectively;calculating a fourth component loss £CA of the plurality of component losses using £CA = Si - |reZu(PP(t) - PB(t))||, where DCAR stands for "Desired Collision Avoidance Range",calculating a fifth component loss £AA of the plurality of component losses using £AA = \ \(relu(-REBF(t))|| + MQ * \\(relu(DSD - |^SF(t)l)ll> where REBF(t) and RyBF(f) are x and y components of the relative position between the pursuer and the evader, expressed in the evader’s body frame, EBF, DS stands for "Desired Separation Distance", and M0 is a mask function which assigns a value of 0 when the pursuer is positioned behind the evader, and a value of 1 when the pursuer is ahead of the evader.

10. The computer-implemented method of Claim 9, wherein computing the loss as a weighted sum of the plurality of component losses uses £totai = XF£F + kCE£CE + ^sc^sc + ^-ca^ca + ^aa^aa^ where XF is first a weight associated with the first loss component £F, XCE is a second weight associated with the second loss component £CE, Asc is a third weight associated with the third loss component £SCt XCA is a fourth weight associated with the fourth loss component £CA, and Aaa is a fifth weight associated with the fifth loss component £AA.

11. The computer-implemented method of any preceding claim, wherein the dynamic parameters include a heading angle, a position, and a velocity.

12. The computer-implemented method of Claim 11, wherein the evader neural network agent is trained to output a constant velocity and a constant heading angle of the evader.

13. A transitory, or non-transitory, computer-readable medium having instructions stored thereon that, when executed by at least one processor, causes the at least one processor to perform the computer-implemented method of any preceding claim.

14. A pursuer vehicle comprising:at least one control surface for moving the pursuer vehicle;at least one sensor for sensing at least one dynamic parameter of an evader vehicle;at least one processor; andstorage having the pursuer neural network agent trained according to any of Claims 1 to 12 stored thereon, the pursuer neural network agent causing the at least one processor to control the at least one control surface to move the pursuer vehicle in response to the at least one dynamic parameter of the evader vehicle.27

Images