Simulation-based testing for robotic systems

Abstract

Performance testing of mobile robot planners is considered. A particular aim is herein to find a perception error or set of perception errors for a scenario that yields 'good' performance on a predetermined perception performance metric but has an unexpected impact on downstream performance (such as planner performance), in a reduced number of simulated runs. Of particular interest are instances of good perception performance that nevertheless lead to planner failure (e.g. breaking of a safety rule) or some other salient planning outcome. A core aim is to identify perception errors that are salient in this sense in a computationally efficient manner, by maximizing the saliency of perception errors for a given number of simulated runs. This is achieved through a systematic search for salient perception error based on a predetermined perception performance metric in combination with selectively performed simulation rollouts and rules-based planning performance evaluation.

Description

SIMULATION-BASED TESTING FOR ROBOTIC SYSTEMS TECHNICAL FIELD

[0001] The present disclosure pertains to methods for evaluating the performance of trajectory planners and other robotic systems in simulated scenarios, and computer programs and systems for implementing the same. Trajectory planners are capable of autonomously planning ego trajectories for fully / semi-autonomous vehicles or other mobile robots. Example applications include ADS (Autonomous Driving System) and ADAS (Advanced Driver Assist System) performance testing. BACKGROUND

[0002] There have been major and rapid developments in the field of autonomous vehicles. An autonomous vehicle (AV) is a vehicle which is equipped with sensors and control systems which enable it to operate without a human controlling its behavior. An autonomous vehicle is equipped with sensors which enable it to perceive its physical environment, such sensors including for example cameras, radar and lidar. Autonomous vehicles are equipped with suitably programmed computers which are capable of processing data received from the sensors and making safe and predictable decisions based on the context which has been perceived by the sensors. An autonomous vehicle may be fully autonomous (in that it is designed to operate with no human supervision or intervention, at least in certain circumstances) or semi-autonomous. Semi-autonomous systems require varying levels of human oversight and intervention, such systems including Advanced Driver Assist Systems and level three Autonomous Driving Systems. There are different facets to testing the behavior of the sensors and control systems aboard a particular autonomous vehicle, or a type of autonomous vehicle. Other mobile robots are being developed, for example for carrying freight supplies in internal and external industrial zones. Such mobile robots would have no people on board and belong to a class of mobile robot termed UAV (unmanned autonomous vehicle). Autonomous air mobile robots (drones) are also being developed.

[0003] To be viable, it is generally assumed that autonomous driving must achieve a minimum level of safety that significantly exceeds that of human drivers. Human driving is estimated to cause of the order 10-6 severe accidents per hour. It has been suggested that autonomous driving systems will need to reduce this by at least three orders of magnitude tobe accepted in society, implying a minimum safety level of the order of 10-9 severe accidents per hour needs to be guaranteed.

[0004] Ensuring that the performance of a system is satisfactory under uncertain or variable operating circumstances is an important step towards deploying such systems safely in the real world. This is especially important in safety critical applications such as autonomous driving, where finding rare but catastrophic failures has been identified as an important challenge.

[0005] Sensor processing may be evaluated in real-world physical facilities. Similarly, the control systems for autonomous vehicles may be tested in the physical world, for example by repeatedly driving known test routes, or by driving routes with a human on-board to manage unpredictable or unknown context. Physical world testing will remain an important factor in the testing of autonomous vehicles capability to make safe and predictable decisions. However, physical world testing is expensive and time-consuming. Therefore, during the development process, simulation-based testing is often used in addition to real world testing in order to more quickly test autonomous driving systems at a reduced cost, with greater speed and increased safety.

[0006] For mobile robot testing, a test scenario instance (or ‘run’) includes a simulated mobile robot (the ego agent) and some physical context that the ego agent is required to navigate. The physical context of a driving scenario generally includes a road layout and may include one or more ‘challenger’ objects (static objects and / or dynamic agents, such as vehicles, pedestrians, cyclists, animals etc.). An instance of a test scenario is realized in a simulation environment, with the robotic system under testing in control of the ego agent. The performance of the robotic system may be assessed in terms of the behavior of the ego agent in the scenario instance and / or in terms of decisions taken internally within the robotic system over the course of the simulation. The robotic system has knowledge of its location at any instant of time, understands its context, and is required to make safe and predictable decisions about how to navigate its environment to reach a pre-programmed destination. Simulation-based testing may, for example, utilize parameterized scenarios, where different instances (or ‘runs’) of the scenario may be realized in a simulation environment with different parameter combinations.

[0007] Simulation-based testing comes with its own challenges. Generating and evaluating a simulated run requires significant computational resources. As the safety of autonomous systems increases, the number of required simulation instances also increases.

[0008] An end-to-end testing checks planner behaviour in a scenario-based simulation with simulated perception errors. Specific traffic situations may be tested in simulation, or recorded data may be replayed through the entire system. The system can be adjusted based on these tests to ensure robustness to typical perception errors in a wide variety of scenarios, before the system is deployed in the real world.

[0009] Steps can be taken to improve the efficiency of individual simulated runs. One approach generates high-fidelity sensor data, which is fed to a perception system. Alternatively, perception error models may be used to directly inject realistic perception errors into simulator ground truth, enabling higher-level components such as planners to be tested in the presence of perception error, without requiring synthetic sensor data that may be computationally expensive and / or difficult to generate. Nevertheless, an autonomous driving system operating at a required level of safety, a ‘naïve’ simulation strategy would still require of the order of 109 simulated driving hours on average to encounter a single severe accident. Therefore, there remains a need to increase the efficiency of simulation-based testing.

[0010] The identification of useful and representative scenarios which can be used to effectively test autonomous vehicles has emerged as a separate task from the overall estimation of failure probability for the system.

[0011] Because severe failures in realistic systems are expected to be rare, adaptive search approaches have been often used to more quickly identify failure cases which can be used to adjust the system’s behaviour. SUMMARY

[0012] When testing a modular autonomous driving system, a perception system may be tested separately to the system as a whole. End-to-end performance of the system can be measured directly, by checking the number of driving rule violations either in simulation or in the real world. However, the perception system might be tested using a metric which is not strongly correlated with this end-to-end performance, particularly in ‘long-tail’ scenarios where rule breaking behaviour is expected. Performance testing of mobile robot planners under perception error is considered more generally.

[0013] A particular aim is herein to find a perception error or set of perception errors for a scenario that yields ‘good’ performance on a predetermined perception performance metric, but has an unexpected impact on downstream performance (such as planner performance), in a reduced number of simulated runs. Of particular interest are instances of good perception performance that nevertheless lead to planner failure (e.g. breaking of a safety rule) or some other salient planning outcome.

[0014] A core aim is to identify perception errors that are salient in this sense in a computationally efficient manner, by maximizing the saliency of perception errors for a given number of simulated runs (or, equivalently, minimizing the number of runs that are required to achieve a given level of perception error saliency). This is achieved through a systematic search for salient perception error based on a predetermined perception performance metric in combination with selectively performed simulation rollouts and rules-based planning performance evaluation.

[0015] Planning performance may be evaluated using one or more planning performance rules (such as driving rules(s) pertaining to safety). In this context, the aim is to find perception error which, when passed to a planner, result in rule breaking behaviour, despite having good perception performance. The output of the planner can be a mobile robot action (such as a control signal or motion state), or a series of such actions. In this context, planning performance is evaluated by applying a planning performance rule (or rules) to the mobile robot action output by the planner.

[0016] A first aspect herein provides a computer-implemented method of performance testing a mobile robot planner in simulation in the presence of perception errors, the method comprising: receiving a ground truth state of a test scenario; generating a first perturbed scenario state based on the ground truth state; computing a first perception performance score for the first perturbed scenario state based on a perception scoring function applied to the first perturbed scenario state and the ground truth state; responsive to determining that the first perception performance score exceeds a first minimum perception performance requirement, selecting the first perturbed scenario for use in testing the mobile robot planer; performing a simulation rollout process based on the selected first perturbed scenario, the first perception performance score and the first minimum perception performance requirement by: generating a first mobile robot action using the mobile robot planner under testing applied to the selected first perturbed scenario state, evaluating the first mobile robot action based on a planningperformance evaluation rule to obtain a first planning performance evaluation result, determining a second minimum perception performance requirement as a function of: the first perception performance score, the first minimum perception performance requirement and the first planning performance evaluation result, selecting a second perturbed scenario state having a second perception performance score that exceeds the second minimum perception performance requirement, and repeating the simulation rollout process based on the selected second perturbed scenario state, the second perception performance score and the second perception performance requirement; wherein the simulation rollout process is repeatedly performed until reaching a termination condition, the method further comprising outputting a performance evaluation result pertaining to the mobile robot planner under testing based on a final minimum perception performance requirement as determined on reaching the termination condition.

[0017] In embodiments, the first mobile robot action may be evaluated to determine whether a saliency condition is satisfied; wherein the second minimum perception performance requirement may be determined so to as match: the first minimum performance requirement in the event the saliency condition is not satisfied, or the first perception performance score in the event the saliency condition is satisfied.

[0018] The saliency condition may be failure on the planning performance evaluation rule.

[0019] An initial perception error may be received, wherein the first perturbed scenario state may be generated by perturbing the groundtruth scenario state based on a first perception error, the first perception error determined based on the initial perception error; wherein the second perturbed scenario state may be generated based on a second perception error, the second perception error determined based on the first perception error.

[0020] The first and second perception errors may be determined using a random error step generator of the form ^(^|^^), wherein the first perception error ^^may be sampled from ^(^|^^), where ^^is the initial perception error, and the second perception error ^^is sampled from ^(^|^^).

[0021] The first minimum perception performance requirement may be determined based on the initial perception error.

[0022] The initial perception error may be determined in an initial search phase by repeatedly running simulation rollouts with different perception errors to obtain different planningperformance evaluation results, the initial perception error selected as a minimized perception error for which a salient planning performance evaluation result is obtained.

[0023] Each performance evaluation result may comprise a planning performance score, the initial perception error selected as a minimized perception error that results in a planning performance score approximately equal to a planning performance threshold.

[0024] The ground truth state may form part of a time series of ground truth states, and the first perturbed scenario state forms part of a first time series of perturbed scenario states generated based on the time series of ground truth states, wherein the first perception performance score may pertain to the first time series of perturbed scenario states as a whole; wherein the second perturbed scenario state may form part of a second time series of perturbed scenario states, wherein the second perception performance score may pertain to the second time series of perturbed scenario states as a whole.

[0025] Each perception error may comprise a time series of error components, wherein the initial search phase may comprise: performing multiple simulation rolls out based on multiple time series of error components, and selecting as the initial perception error the time series of error components having a minimum error duration.

[0026] Each state may comprise a plurality of agent states, wherein the first perturbed scenario state may be generated based on a sampled perception error for each agent state of the ground truth state.

[0027] The initial perception error may be selected as the time series of error components having a minimum duration of false negative agent detection.

[0028] The second perception performance score may be computed based on a perception scoring function applied to the first perturbed scenario state and the ground truth state.

[0029] The ground truth state may be an initial ground truth state, wherein the second perception performance score may be determined based on a perception scoring function applied to the first perturbed scenario state and a first ground truth state dependent on the first mobile robot action.

[0030] The perception scoring function may have the form of a learned perception error model (PEM) for modelling a real perception system.

[0031] A second aspect herein provides a computer system comprising one or more computers configured to implement any of the above aspects or embodiments.

[0032] A third aspect herein provides transitory or non-transitory media embodying computer-readable instructions, which are configured when executed on one or more processors to cause the one or more processors to implement any of the above aspects or embodiments.

[0033] In this manner, a perturbation to the scenario ground truth is only ‘accepted’ as a basis for a simulated run (or ‘simulation rollouts') when it exceeds a current minimum perception performance requirement, i.e. when it yields sufficiently good perception performance. For example, the current requirement might be defined by a threshold and a perturbed scenario may only be accepted when it yields a perception performance score above this threshold (in the case that a higher score demoted better perception performance) or below this threshold (in the case that a higher score denotes worse performance). Simulation rollouts are not performed perturbed scenarios that fail to meet the current minimum perception performance requirement. The current minimum perception performance required is, in turn dynamically adapted based on the evaluation of planner performance over the course of the simulated runs. Over a number of iterations, the search is guided towards scenario perturbations that perform increasingly better on the perception scoring function (the perception performance metric), but nevertheless resulting in salient outcomes in terms of planner behaviour as measured using the planning performance evaluation rule. BRIEF DESCRIPTION OF FIGURES

[0034] Particular embodiments will now be described, by way of example only, with reference to the following schematic figures, in which:

[0035] Figure 1 shows a functional block diagram of a system for perception component evaluation.

[0036] Figure 2 shows a function block diagram of system configuration that may be used for testing with a Perception Error Model (PEM).

[0037] Figure 3 shows an example system configuration used for adversarial perception error search.

[0038] Figure 4 shows a graphical representation of a boundary attack perception error search algorithm.

[0039] Figure 5 shows an example mobile robot system configuration at runtime.

[0040] Figure 6 shows schematic diagrams of example test scenarios.

[0041] Figure 7 shows a birds eye view representation of a conversion from vehicles in an ego frame of reference to an arc segment representation.

[0042] Figure 8 shows metrics comparing a PEM and BEVFusion detector on nuScenes test set.

[0043] Figure 9 shows measured correlation of the log likelihood of the PEM against the NDS and NDS-t metrics on the nuScenes val dataset.

[0044] Figure 10 shows an example of a neural network architecture for a Neural Surrogate.

[0045] Figure 11 shows a histogram of the obtained log likelihood for a perception error model on held out data from a dataset used to train the PEM.

[0046] Figure 12 shows a plot of the cumulative largest m achieved for various methods in a failing scenario against number of rollouts made.

[0047] Figure 13 shows example frames from the observed perception errors for an overtaking scenario (A), a right turn scenario (B) and a left turn scenario (C).

[0048] Figure 14 shows a plot of the cumulative largest PEM log likelihood achieved for various methods method in a failing scenario against the number of rollouts made.

[0049] Figure 15 shows a histogram of the obtained log likelihood for a perception error model on held out data from a dataset used to train the PEM.

[0050] Figure 16 shows results of applying random perturbations of increasing strength to the obtained rule breaking errors.

[0051] Figure 17 shows a schematic function block diagram of an autonomous vehicle stack.

[0052] Figure 18 shows a schematic overview of an autonomous vehicle testing paradigm.

[0053] Figure 19 shows a schematic block diagram of a scenario extraction pipeline.

[0054] Figure 20 shows a schematic block diagram of a testing pipeline.

[0055] Figure 21 shows further details of a possible implementation of the testing pipeline.

[0056] Figure 22 shows a rendering component for rendering a graphical user interface for displaying a test output. DETAILED DESCRIPTION

[0057] Performance testing of mobile robots in simulated scenarios is considered.

[0058] Testing methodologies may involve a careful elucidation of the operational design domain (ODD) in which the system will be deployed, and specification of the desired behaviour of the system in the ODD via the definition of driving rules. This allows the behaviour of the system as a whole to be assessed, which is crucial for safe deployment. Individual components making up the system can also be tested separately, however testing a perception module separately with common metrics like mean average precision might only be weakly correlated with the impact of perception mistakes on the planning system. The system as a whole as well as the individual subsystems should be fine-tuned on recorded data and in simulation prior to deployment. Then once the system is deployed in the real world further data can be collected which can be used to improve the system in the future. The safety of the system as a whole can be defined and improved by assessing how often the driving rules are broken.

[0059] Experimental results are provided which demonstrate the existence of sets of erroneous perception system outputs which score highly in common perception metrics, but nevertheless cause the system to violate driving rules. Such perception errors are termed ‘adversarial’ perception errors herein. The existence of these adversarial perception errors has implications for how these systems are built and tested and is therefore highly relevant to practitioners in the field of autonomous vehicles. In fact, the adversarial perception errors score much more highly on the perception metrics than actual outputs from the perception system on an independent test dataset. The described approach may be characterized as a form of ‘adversarial attack’ on the planner under testing, the objective being to try to cause the planner to fail under good perception conditions.

[0060] Motivated by these insights, an efficient search algorithm is described, which yields the most adversarial perception failure modes for the system in simulation, where the saliencyof these modes are specified by the user specified perception metric. The algorithm has been tested in the Carla simulator

[0013] on a state-of-the-art planner

[0014] , and judge the saliency of the identified errors using nuScenes detection score and other metrics, and performance of the method has been compared with existing approaches.

[0061] In certain embodiments, adversarial attack methods are leveraged to provide an algorithm based on boundary attack to identify the adversarial perception errors which is particularly applicable for black box systems.

[0062] A typical mobile robotic system includes perception functionality (the ability to receive and interpret high-fidelity sensor signals, such as image data, radar data, lidar data and / or other forms of spatial sensor data, and produce lower-fidelity perception outputs, such as object bounding boxes, detected object locations / orientations, segmentation masks etc.) and motion planning functionality (the ability to generate a motion plan for a mobile robot). A motion plan may take the form of a control signal or sequence of control signals for directly controlling a mobile robot actor system, or a higher-level motion plan that is inputted to a controller. The following examples consider robotic systems that are ‘modular’ in so far as perception is separable from one or more downstream tasks (such as motion planning), meaning that the perception system can, in testing, be substituted with a surrogate perception model, also referred to as a perception error model (PEM). A surrogate perception model is used to inject perception error into scenario groundtruth generated by a simulator directly, resulting in a ‘perturbed’ perception output that approximates the (imperfect) output of the perception system in a comparable real-world scenario, but without the need to generate high- fidelity synthetic sensor data based on sensor model(s). For example, perception errors or perception outputs may be sampled from a distribution representative of the perception system. This approach enables the motion planning functionality to be rigorously tested in the presence of realistic perception error, but with much increased computational efficiency compared with simulation-based testing approaches using synthetic sensor data. Moreover, for certain types of sensor data that are inherently difficult to model (such as radar) or for certain types of perception component that are highly sensitive to discrepancies between synthetic data and real data (such as convolutional neural networks applied to images), the surrogate-based approach can yield improved testing accuracy, as it is able to produce more realistic perception outputs than synthetic data approaches based on sensor modelling.

[0063] As discussed, in the aforementioned context, an issue of perception error ‘saliency’ arises. Occasionally, a perception error that is considered ‘good’ perception performance may have a major impact on a downstream task such as motion planning. In other words, ‘good’ performance on a perception metric does not necessarily equate with ‘good’ performance on a downstream metric, such as driving safety / performance (and vice versa). Numerical perception metrics may be used to quantify perception performance in isolation. Note that, in the present context, the performance of a perception system is not being evaluated; rather, a perception metric is used to characterize the output of a surrogate perception model used in testing. In this context, a perception error metric is used to quantify ‘goodness’ of a perturbed perception output relative to the corresponding simulated ground truth. Examples of established perception performance metrics in the field of autonomous driving include the nuScenes detection score (used to evaluate object detection performance) and the nuScenes tracking score (used to evaluate object tracking performance). Note, these are domain-specific metrics tailored to autonomous driving, in the expectation they will be highly correlated with driving performance. Nevertheless, it has been found that good perception performance on such metrics can still cause unexpectedly poor planner performance from time-to-time. An instance of good performance on an established domain- specific perception performance metric that yields an unexpectedly salient planning outcome is particularly informative. ‘Good’ performance may alternatively be assessed using a perception scoring function in the form of a perception error model (PEM) that statistically models a real perception system. This approach is described in further detail below.

[0064] Perception performance may be measured with respect to a minimum perception performance requirement. The minimum perception performance requirement may be quantitative, such as a minimum level of precision to be achieved, or a maximum number of false negative detections allowed. The minimum perception performance requirement may also be qualitative, such as whether or not an external agent was detected.

[0065] A salient outcome might, for example, be failure on a planning performance evaluation rule. In the following examples, planner performance is scored numerically, with failure defined in terms of a threshold on the planning performance score. Planning performance scores are normalized in the following examples, with a failure threshold of zero, though it will be appreciated this is an arbitrary choice.

[0066] One aim in particular is to find perception errors yielding good perception performance scores, but which have a material impact on downstream performance, and to do so in a computationally efficient manner by reducing the number of simulated runs that need to be performed to find such perception errors. Among other things, the described techniques can be used to efficiently find perception errors that score highly on established perception performance but which lead to safety critical outcomes. Such errors, once identified, can be used to guide the design or tuning of real perception systems (to reduce instances of such errors in the real world) and / or to guide the design / tuning of downstream components (such as motion planners) to make them more robust to such perception errors.

[0067] In the following, autonomous vehicles are considered by way of example. However, the description applies equally to any form of mobile robot system. Autonomous Vehicle Simulation

[0068] At any given time, the agents in a driving environment can be described by a set of states s in a d-dimensional spacewhich are perceived by the system underThese states s contain relevant properties of every agent in the scene (e.g., position, velocity etc.) as well as sensor data like LiDAR point clouds and RGB images in the case of real driving scenarios. Correspondingly, contains perceived properties of set of the agents which may or may not really exist in the scene. An action is determined by a mobile robot planner, denoted, based on the perceived state deterministically with policy . Assuming deterministic simulator physics, and assuming actions of the other agents are deterministic given a particular state, then it follows that for any action the state transition is also deterministic.

[0069] In many cases non-deterministic agent actions can be made deterministic by parameterising the agent behaviour in some way, e.g. by specifying the aggressiveness and direction of turns by an agent in a particular scenario. Therefore, the above assumptions are not as overly restrictive. However, there will be some cases in which agent behaviours cannot be made deterministic.

[0070] To enable an AD system under testing to perceive the environment, a function is defined to map from the environmental state to the perceived state In the real world this is achieved with a perception system e.g., a lidar based object detector, so the actions are produced as. Therefore, a deterministic ‘rollout’ of T states starting from may be defined as. Perception Testing

[0071] Sets of ground truth and perceived agents are defined asand . A real valued perception scoring functionis defined as a figure of merit for the perceived states, so that the perception system can be evaluated. For example, m could be for example the nuScenes detection score or the nuScenes tracking score

[0016] . In some situations, a ground truth state may only have one agent at T=1. In this situation, a single planned action for the agent is evaluated.

[0072] Figure 1 shows a system configuration for perception component evaluation. This configuration was also used to train a PEM and also to create histograms shown in Figure 10. The detector 102 uses images and lidar data from a real-world recorded dataset 101 to output detections. Perception evaluation data 103 is formed of the detections from the detector 102, together with ground truth detections from the recorded dataset 101. The PEM training step 104 takes as input the perception evaluation data 103. Perception Error Models

[0073] Perception error model (PEM) have previously been used to approximate f using a probability distribution, conditioned on an augmented state , which is cheaper to produce in simulation than the actual state s, because expensive-to-compute sensor data is not included in [9, 17]. The approximation is probabilistic because the augmented state does not include all the information required to predict the perceived state exactly. To simplify the notation, herein a PEM is denoted as a distribution of perceived states, conditioned on s.

[0074] Figure 2 shows a system configuration used for perception component evaluation, and to train a Perception Error Model 202. The perception error model 202 takes as input theworld state and produces as output detections from the world state. A tracker 203 takes as input the detections from the perception error model 202 and outputs tracked detections. A planner 204 takes as input tracked detections from the tracker 203. In a planner rule evaluation step 205, a planner rule is applied to the actions planned by the planner 204. In a simulator update stap 201, actions planned by the planner 204 are used to update a world state in an iterative process.

[0075] When simulating with the PEM, the probability of transitioning to statefrom s isstarting from a T-step simulation rollout is defined as , with a rollout probability defined as:

[0076] To create training data for the PEM, a tupleof input-output is created for every frame by running the perception system on a labelled sensor dataset, which is processed with an association algorithm to obtain an input-output tuple for each agent in the scene. The association algorithm produces training data for the surrogate detector of the form[17, 9]. These input-output per agent tuples can be used to train a PEM which factorises over each agent in each scene. The PEM is trained by optimising the parameters of a probabilistic neural network to minimise a losswhere represents the likelihood, represents the Boolean output which is true if the object was detected, andrepresents a real-valued output describing the centre position of the detectedin Eqn.7 is equivalent to the binary cross-entropy when using a Bernoulli distribution to predict false negatives.

[0077] Assuming Cartesian components of the positional error to be independent, the first term in Equation 7 is determined aswhere µ and log(σ) are the outputs of the fully connected neural network.

[0078] The PEM is evaluated by comparing properties of samples from p(sˆ | s), such as mean average precision, with corresponding properties of the original perception system outputs. Of course, the above loss function is merely one example, and similar loss functions can be defined for many different distributions and model architectures. Planner testing

[0079] Given a simulation rollout, a safety metric ofmay be defined such that denotes violation of the metric, for example in an autonomous vehicle context this could be a (thresholded) closest distance of approach of ego to any other agent. The safety metric is essentially a driving rule which is violated when the metric is less than zero. The rate of failures is quantified using the probabilistic threshold robustness (PTR) of the system, defined aswhere is the indicator function. In the case that the simulator, ego and other agents behave deterministically, and the perceived state is also deterministic, the failure probability will either be zero or one because only one state rollout is possible. Otherwise, if the perceived state is produced by a PEM, then solving the integral in Eqn.2 will involve taking the expectation over the PEM. Estimating in Eq. (2) using a vanilla Monte Carlo simulation is known to be computationally expensive since identifying a failure rate lower than ^ will typically require at least 1 / ^ tests

[0019] . ApproachMotivation

[0080] The planner and perception system are typically tested using different metrics. Therefore, it is important to understand if there are situations in which the perception system scores highly on perception metrics whilst the same outputs of the perception system could result in the planner performing uncharacteristically poorly by breaking planning rules. However, when testing a typical system, it is expected that such systems should be robust to high probability perception errors, and indeed may even have their performance tuned using these typical perception errors [9-20]. Therefore, a search-based approach is required to identify erroneous perception system outputs which are critical for system safety. Furthermore, it is known from Hohenbichler’s lemma that the most likely failure scenarios are strongly related to the overall system failure probability and hence determining these failures is of great interest to engineers [21, 22]. An important distinction is drawn between the most likely failure scenarios, and overall most likely scenarios; the former of which should have low probability and not be evident from vanilla Monte Carlo sampling, as the system is expected to be robust to high-probability failures. Generally, the quality of a perception system is measured using perception metrics such as mean average precision, and therefore it could be considered a requirement for downstream parts of the system that they are robust to perception system outputs with metric values equal or greater than those of the upstream perception system on a test dataset. These perception metrics are not necessarily strongly correlated with the planner safety metrics of interest, particularly in the low probability-high severity areas of the scenario space that are crucial to safety critical systems such as autonomous vehicles. Therefore, a goal is to find erroneous perception system outputs which score highly on perception metrics whilst still resulting in a planner failure, termed adversarial perception errors herein. Theory

[0081] A robust method for identifying adversarial perception errors is described, which is applicable for black box systems, i.e. those for which gradients are not available. As noted, the aim is to efficiently discover perception errors which result in rule breaking behaviour whilst also scoring highly on perception metrics, as intuitively these should be more unexpected and therefore more revealing to system designers / test engineers. A fuzzy set

[0023] for conditional on y is specified using a perception metric function. An α cut of the set is specified as. This fuzzy set can be seen as a specification ofthe perception system. The set of implies a set of possible rollouts for each α cut which will be given by

[0082] To perform simulation with the fuzzy set, the method finds the largest α cut of the fuzzy set which still contains a rollout which results in failure, i.e.

[0083] The contained by this set represent the failure modes and can be obtained as

[0084] In order to parametrically generate arbitrary setsan error injection function,is used to apply an error set, e, to any ground truth state s to yield a perceived state. Using a parametric error injector function provides a simple human interpretable method of specifying the set of possible detections based on the ground truth perception. The error set e comprises d variables which cause I to transform the ground truth state in some way, e.g. remove or add a particular agent or apply a spatial transformation to an agent as in the following examples. During simulation rollouts, the error injection function is applied to each ground truth state independently, to obtain the set of perceived states aswhere the entire error set is denoted as The error set is parameterized in this example as, where is a Cartesian additive error for agent j,is an orientation additive error for agent j, and is a binary false negative indicator which can be set to causes agent j to be completely removed from the output of I. Of course, many other parameterisations are possible. Figure 3 shows a system configuration used for adversarial perception error search. A search algorithm 303 outputs parameterised perception errors. In an adversarial error injection step302, the perception errors from the search algorithm 303 are injected to a world state (ground truth state) to produce perceived detections. A perception metric 304 is computed from the world state (ground truth state) and the perceived detections. A tracker 305 takes as input the perceived detections and outputs tracked detections. A planner 306 takes as input the tracked detections from the tracker 305 and outputs planned actions. In a planner rule evaluation step 307, a planner rule is applied to the actions from the planner 306. A simulator 301 takes as input the planned actions from the planner 306 to output the world state for the next time step. Implementation

[0085] To solve Eqn.4 approximately, an initial heuristic search algorithm is provided in Algorithm 1, followed by a random search algorithm to iteratively improve the solution in Algorithm 2. Algorithm 2 is based on ‘open loop' simulation. An alternative ‘closed loop’ algorithm is provided in Algorithm 3.

[0086] Algorithm 1 is run in order to seed Algorithm 2 or Algorithm 3. Algorithm 1 is constructed heuristically based on intuition about how the space is structured. The heuristic search Algorithm leverages the observation that, for most driving rules, it is straightforward for a human to identify a perception error which will cause a rule violation, and also straightforward to adjust this error so that it is less severe according to some human interpretable perception metric. For example, it is apparent that, during highway driving, not perceiving slow agents which a driver is approaching in the lane of travel could cause collisions. It is also evident that an average precision perception metric would have its value increased by reducing the number of agents which are not perceived. Therefore, it is possible to find a good initialisation for the search algorithm, which is particularly useful considering the large size of the search space and the lack of gradients to guide the search.

[0087] However, it seems implausible that a heuristic guided algorithm could find a global maximiser of the perception metric which causes rule violations, because the rule violation indicator function,, with respect to the perception error, e, is expected to have a complex structure and be difficult for humans to interpret in detail. Therefore, a random search process in Algorithm 2 (or Algorithm 3) guides the algorithm closer to a global optimum, whilst still taking advantage of the initialisation provided by the heuristic search. Initialising the random search from no perception errors would not be effective because thesystem under test is expected to be robust to small perception errors, so most steps would be rejected due to not finding any rule violations. Given the lack of gradient information from the planner under ‘attack’, a similarly strong initialisation is particularly beneficial Alternative (e.g. more complex) search algorithms could be used.

[0088] The heuristic search of Algorithm 1 could be replaced by any other algorithm which yields good initialisation points for the random search.

[0089] Figure 4 shows a graphical representation of a boundary attack perception error search algorithm. A boundary 401 separates the unsafe region 403 and the safe region 402. The random search iterations of the boundary attack perception error search algorithm find instances along the boundary 401 where the perception metric is maximised in the unsafe region. The saliency of the perception errors is measured using a perception metric, and therefore the algorithm attempts to maximise the perception metric by making the perception error as close to the ground truth perception of the scene as possible according to the perception metric, whilst being inside the unsafe region 401 where a planning rule is broken.

[0090] To perform rollouts more efficiently an open loop simulation approach is used in Algorithm 2. In this context, open loop means the state of the other agents, s, is only computed once (in the world frame), with no perception errors, and the injection function is applied to these "ground truth states" to obtain sˆ which can be used to recompute the Ego plans. Although the states s will be frozen in the world frame they will change in the Ego- centric frame since the plans of Ego will depend on the applied perception errors. The advantage of this algorithm is that the expense of repeatedly performing rollouts with the full simulator to obtain τ will be avoided. Another advantage is that simulator non-determinism can be avoided. However, the algorithm may be inaccurate if the behaviour of other agents has a strong dependence on the behaviour of ego, because in this algorithm the recorded behaviour of the other agents is simply “replayed”, and therefore a closed loop algorithm could be used.

[0091] A closed loop random search algorithm is shown in Algorithm 3. The heuristic search in Algorithm 1 can also be modified to use closed loop rollouts rather than the open loop simulations.

[0092] Despite its simplicity, the open loop algorithm of Algorithm 2 has been demonstrated to be highly effective. Nevertheless, the closed loop version of Algorithm 3 may be more suitable for certain types of scenario.

[0093] The heuristic search algorithm finds rollouts such that using a bisection approach, which is possible since it is known that that using will typically cause , because for most sensible systems and safety metrics, not making any detections will result in a collision. Then random steps are made to higher α cuts of the perception set such that r(τ) ≤ 0. To guide the random search more efficiently, before running a rollout, the algorithm attempts to find proposal steps which are expected to yield increased α. Proposal steps are proposed using a distribution conditional on the previous error, which will with a fixed probability flip the fn variable and introduce additive noise to the x variable. Of course, other proposal distributions are possible. Conditioning on the perception from the previous iteration is used because y is not expected to change significantly between random search steps due to the typically small changes of e, and it is expected thatfor the obtained errors to independent samples of from a dataset (e.g. by using concentration inequalities) could be interpreted as reasoning about the plausibility of the errors rather than realism, where plausibility refers to an inability to prove that something is unrealistic. This is because the comparison is effectively a one-tailed hypothesis test with an attempt to reject the null hypothesis that the detector could produce an error this severe. Therefore, coverage of the described testing is guaranteed, in the sense of Conformal Prediction

[0024] , meaning this approach will find the most plausible perception outputs resulting in failure.

[0095] In the field of computer vision, ‘adversarial attack’ algorithms have been developed to find ‘small’ image perturbations that nevertheless result in classification errors. Here, an adversarial attack approach is applied to a different problem, namely finding instances of good perception performance that lead to planner failure. This uses a modified version of the boundary attach algorithm described in section 2.2 of

[0012] .

[0096] In Algorithms 2 and 3 perception errors are sampled from a proposal step generator, which is a conditional probability distribution of the form. The proposal stepgenerator serves a similar purpose to the proposal distribution from

[0012] . It is constructed so as to guide the search towards e which have a reasonable chance of rule breaking (r < 0) and a higher perception metric value (m) than the previous error. It does not necessarily need to learn from data, as it designed intuitively to satisfy these properties.

[0097] In onecauses a (typically small) percentage of False Negatives to be randomly flipped fromand (typically small) random perturbations to be added to the position errors of the ^^^^. For example, FNs may be randomly flipped inand position errors may be added towith a fixed probability that is predetermined, or with a probability derived as a predetermined function of ^^^^.

[0098] In Algorithms 2 and 3, it can be seen that the perception score threshold ^^applied in step ^ is determined as the following function of the previous thresholdand the perception score:^(^)≥ 0 (planner passes) if ^(^) < 0 (planner fails) with ^^= ^^∀^ in Algorithm 2 (open loop).

[0099] The output, ^^^^^^^(the highest perception performance threshold that led to planner failure) is highly informative, as it quantifies ‘how far’ it was possible to increase perception performance and still obtain a planning failure outcome.

[0100] In the above, it is assumed that, when the planner is run on the groundtruth ^^or ^^directly (i.e. without any perception error), a pass outcome is obtained, meaning that a failure obtained with the addition of perception error can be attributed to the perception error. An additional check may be performed to verify that a pass outcome is obtained for a given ground truth ^^.

[0101] Note that the Algorithms can be run on a single time step, i.e. with ^ = 1 in the simplest case. Algorithms 2 and 3 can also be run with ^^^^^^^^^^^^^^^= 1, and Nstepsincreased to compensate. The reason for introducing the separation is that it gives a simple intuition for roughly how many evaluations of the perception metric and planner will be made. Example AV stack:

[0102] To provide relevant context to the described embodiments, further details of an example form of AV stack will now be described.

[0103] Figure 17 shows a highly schematic block diagram of an AV runtime stack 1700. The run time stack 1700 is shown to comprise a perception (sub-)system 1702, a prediction (sub- )system 1704, a planning (sub-)system (planner) 1706 and a control (sub-) system (controller) 1708. As noted, the term (sub-)stack may also be used to describe the aforementioned components 1702-1708.

[0104] In a real-world context, the perception system 1702 receives sensor outputs from an on-board sensor system 1710 of the AV, and uses those sensor outputs to detect external agents and measure their physical state, such as their position, velocity, acceleration etc. The on-board sensor system 1710 can take different forms but generally comprises a variety of sensors such as image capture devices (cameras / optical sensors), lidar and / or radar unit(s), satellite-positioning sensor(s) (GPS etc.), motion / inertial sensor(s) (accelerometers,gyroscopes etc.) etc. The onboard sensor system 1710 thus provides rich sensor data from which it is possible to extract detailed information about the surrounding environment, and the state of the AV and any external actors (vehicles, pedestrians, cyclists etc.) within that environment. The sensor outputs typically comprise sensor data of multiple sensor modalities such as stereo images from one or more stereo optical sensors, lidar, radar etc. Sensor data of multiple sensor modalities may be combined using filters, fusion components etc.

[0105] The perception system 1702 typically comprises multiple perception components which co-operate to interpret the sensor outputs and thereby provide perception outputs to the prediction system 1704.

[0106] In a simulation context, depending on the nature of the testing – and depending, in particular, on where the stack 1700 is “sliced” for the purpose of testing (see below) – it may or may not be necessary to model the on-board sensor system 1700. With higher-level slicing, simulated sensor data is not required therefore complex sensor modelling is not required.

[0107] The perception outputs from the perception system 1702 are used by the prediction system 1704 to predict future behaviour of external actors (agents), such as other vehicles in the vicinity of the AV.

[0108] Predictions computed by the prediction system 1704 are provided to the planner 1706, which uses the predictions to make autonomous driving decisions to be executed by the AV in a given driving scenario. The inputs received by the planner 1706 would typically indicate a drivable area and would also capture predicted movements of any external agents (obstacles, from the AV’s perspective) within the drivable area. The driveable area can be determined using perception outputs from the perception system 1702 in combination with map information, such as an HD (high definition) map.

[0109] A core function of the planner 1706 is the planning of trajectories for the AV (ego trajectories), taking into account predicted agent motion. This may be referred to as trajectory planning. A trajectory is planned in order to carry out a desired goal within a scenario. The goal could for example be to enter a roundabout and leave it at a desired exit; to overtake a vehicle in front; or to stay in a current lane at a target speed (lane following). The goal may, for example, be determined by an autonomous route planner (not shown).

[0110] The controller 1708 executes the decisions taken by the planner 1706 by providing suitable control signals to an on-board actor system 1712 of the AV. In particular, the planner 1706 plans trajectories for the AV and the controller 1708 generates control signals to implement the planned trajectories. Typically, the planner 1706 will plan into the future, such that a planned trajectory may only be partially implemented at the control level before a new trajectory is planned by the planner 1706. The actor system 1712 includes “primary” vehicle systems, such as braking, acceleration and steering systems, as well as secondary systems (e.g. signalling, wipers, headlights etc.).

[0111] Note, there may be a distinction between a planned trajectory at a given time instant, and the actual trajectory followed by the ego agent. Planning systems typically operate over a sequence of planning steps, updating the planned trajectory at each planning step to account for any changes in the scenario since the previous planning step (or, more precisely, any changes that deviate from the predicted changes). The planning system 1706 may reason into the future, such that the planned trajectory at each planning step extends beyond the next planning step. Any individual planned trajectory may, therefore, not be fully realized (if the planning system 1706 is tested in isolation, in simulation, the ego agent may simply follow the planned trajectory exactly up to the next planning step; however, as noted, in other real and simulation contexts, the planned trajectory may not be followed exactly up to the next planning step, as the behaviour of the ego agent could be influenced by other factors, such as the operation of the control system 1708 and the real or modelled dynamics of the ego vehicle). In many testing contexts, the actual trajectory of the ego agent is what ultimately matters; in particular, whether the actual trajectory is safe, as well as other factors such as comfort and progress. However, the rules-based testing approach herein can also be applied to planned trajectories (even if those planned trajectories are not fully or exactly realized by the ego agent). For example, even if the actual trajectory of an agent is deemed safe according to a given set of safety rules, it might be that an instantaneous planned trajectory was unsafe; the fact that the planner 1706 was considering an unsafe course of action may be revealing, even if it did not lead to unsafe agent behaviour in the scenario. Instantaneous planned trajectories constitute one form of internal state that can be usefully evaluated, in addition to actual agent behaviour in the simulation. Other forms of internal stack state can be similarly evaluated.

[0112] The example of Figure 17 considers a “modular” architecture, with separable perception, prediction, planning and control systems 1702-1708. The sub-stack themselvesmay also be modular, e.g. with separable planning modules within the planning system 1706. For example, the planning system 1706 may comprise multiple trajectory planning modules that can be applied in different physical contexts (e.g. simple lane driving vs. complex junctions or roundabouts). This is relevant to simulation testing for the reasons noted above, as it allows components (such as the planning system 1706 or individual planning modules thereof) to be tested individually or in different combinations. For the avoidance of doubt, with modular stack architectures, the term stack can refer not only to the full stack but to any individual sub-system or module thereof.

[0113] The perception functions are separable from the motion planning and prediction functions. The extent to which the other stack functions are integrated or separable can vary significantly between different stack implementations – in some stacks, certain aspects may be so tightly coupled as to be indistinguishable. For example, in other stacks, planning and control may be integrated (e.g. such stacks could plan in terms of control signals directly), whereas other stacks (such as that depicted in Figure 17) may be architected in a way that draws a clear distinction between the two (e.g. with planning in terms of trajectories, and with separate control optimizations to determine how best to execute a planned trajectory at the control signal level). Similarly, in some stacks, prediction and planning may be more tightly coupled. Unless otherwise indicated, the perception, prediction planning and control terminology used herein does not imply any particular coupling or modularity of those aspects.

[0114] It will be appreciated that the term “stack” encompasses software, but can also encompass hardware. In simulation, software of the stack may be tested on a “generic” off- board computer system, before it is eventually uploaded to an on-board computer system of a physical vehicle. However, in “hardware-in-the-loop” testing, the testing may extend to underlying hardware of the vehicle itself. For example, the stack software may be run on the on-board computer system (or a replica thereof) that is coupled to the simulator for the purpose of testing. In this context, the stack under testing extends to the underlying computer hardware of the vehicle. As another example, certain functions of the stack 1710 (e.g. perception functions) may be implemented in dedicated hardware. In a simulation context, hardware-in-the loop testing could involve feeding synthetic sensor data to dedicated hardware perception components. Testing paradigm:

[0115] Figure 18 shows a highly schematic overview of a testing paradigm for autonomous vehicles. An ADS / ADAS stack 1700, e.g. of the kind depicted in Figure 17, is subject to repeated testing and evaluation in simulation, by running multiple scenario instances in a simulator 2702, and evaluating the performance of the stack 1700 (and / or individual subs- stacks thereof) in a test oracle 2752. The output of the test oracle 2752 is informative to an expert 1722 (team or individual), allowing them to identify issues in the stack 1700 and modify the stack 1700 to mitigate those issues (S1724). The results also assist the expert 1722 in selecting further scenarios for testing (S7126), and the process continues, repeatedly modifying, testing and evaluating the performance of the stack 1700 in simulation. The improved stack 1700 is eventually incorporated (S1725) in a real-world AV 1701, equipped with a sensor system 1710 and an actor system 1712. The improved stack 1700 typically includes program instructions (software) executed in one or more computer processors of an on-board computer system of the vehicle 1701 (not shown). The software of the improved stack is uploaded to the AV 1701 at step S1725. Step S1725 may also involve modifications to the underlying vehicle hardware. On board the AV 1701, the improved stack 1700 receives sensor data from the sensor system 1710 and outputs control signals to the actor system 1712. Real-world testing (S1728) can be used in combination with simulation-based testing. For example, having reached an acceptable level of performance though the process of simulation testing and stack refinement, appropriate real-world scenarios may be selected (S1730), and the performance of the AV 1701 in those real scenarios may be captured and similarly evaluated in the test oracle 2752.

[0116] Scenarios can be obtained for the purpose of simulation in various ways, including manual encoding. The system is also capable of extracting scenarios for the purpose of simulation from real-world runs, allowing real-world situations and variations thereof to be re-created in the simulator 2702.

[0117] Figure 19 shows a highly schematic block diagram of a scenario extraction pipeline. Data 1740 of a real-world run is passed to a ‘ground-truthing’ pipeline 1742 for the purpose of generating scenario ground truth. The run data 1740 could comprise, for example, sensor data and / or perception outputs captured / generated on board one or more vehicles (which could be autonomous, human-driven or a combination thereof), and / or data captured from other sources such external sensors (CCTV etc.). The run data is processed within the ground truthing pipeline 1742, in order to generate appropriate ground truth 1744 (trace(s) and contextual data) for the real-world run. As discussed, the ground-truthing process could bebased on manual annotation of the ‘raw’ run data 1740, or the process could be entirely automated (e.g. using offline perception method(s)), or a combination of manual and automated ground truthing could be used. For example, 3D bounding boxes may be placed around vehicles and / or other agents captured in the run data 1740, in order to determine spatial and motion states of their traces. A scenario extraction component 1746 receives the scenario ground truth 1744, and processes the scenario ground truth 1744 to extract a more abstracted scenario description 1748 that can be used for the purpose of simulation. The scenario description 1748 is consumed by the simulator 2702, allowing multiple simulated runs to be performed. The simulated runs are variations of the original real-world run, with the degree of possible variation determined by the extent of abstraction. Ground truth 1750 is provided for each simulated run.

[0118] In the present off-board content, there is no requirement for the traces to be extracted in real-time (or, more precisely, no need for them to be extracted in a manner that would support real-time planning); rather, the traces are extracted “offline”. Examples of offline perception algorithms include non-real time and non-causal perception algorithms. Offline techniques contrast with “on-line” techniques that can feasibly be implemented within an AV stack 1700 to facilitate real-time planning / decision making.

[0119] For example, it is possible to use non-real time processing, which cannot be performed on-line due to hardware or other practical constraints of an AV’s onboard computer system. For example, one or more non-real time perception algorithms can be applied to the real-world run data 1740 to extract the traces. A non-real time perception algorithm could be an algorithm that it would not be feasible to run in real time because of the computation or memory resources it requires.

[0120] It is also possible to use “non-causal” perception algorithms in this context. A non- causal algorithm may or may not be capable of running in real-time at the point of execution, but in any event could not be implemented in an online context, because it requires knowledge of the future. For example, a perception algorithm that detects an agent state (e.g. location, pose, speed etc.) at a particular time instant based on subsequent data could not support real-time planning within the stack 1700 in an on-line context, because it requires knowledge of the future (unless it was constrained to operate with a short look ahead window). For example, filtering with a backwards pass is a non-causal algorithm that can sometimes be run in real-time, but requires knowledge of the future.

[0121] The term “perception” generally refers to techniques for perceiving structure in the real-world data 1740, such as 2D or 3D bounding box detection, location detection, pose detection, motion detection etc. For example, a trace may be extracted as a time-series of bounding boxes or other spatial states in 3D space or 2D space (e.g. in a birds-eye-view frame of reference), with associated motion information (e.g. speed, acceleration, jerk etc.). In the context of image processing, such techniques are often classed as “computer vision”, but the term perception encompasses a broader range of sensor modalities. Testing pipeline:

[0122] Further details of the testing pipeline and the test oracle 2752 will now be described. The examples that follow focus on simulation-based testing. However, as noted, the test oracle 2752 can equally be applied to evaluate stack performance on real scenarios, and the relevant description below applies equally to real scenarios. The following description refers to the stack 1700 of Figure 17 by way of example. However, as noted, the testing pipeline 2700 is highly flexible and can be applied to any stack or sub-stack operating at any level of autonomy.

[0123] Figure 20 shows a schematic block diagram of the testing pipeline, denoted by reference numeral 2700. The testing pipeline 2700 is shown to comprise the simulator 2702 and the test oracle 2752. The simulator 2702 runs simulated scenarios for the purpose of testing all or part of an AV run time stack 1700, and the test oracle 2752 evaluates the performance of the stack (or sub-stack) on the simulated scenarios. As discussed, it may be that only a sub-stack of the run-time stack (or some portion or portions thereof) is tested, but for simplicity, the following description refers to the (full) AV stack 1700 throughout. However, the description applies equally to a sub-stack in place of the full stack 1700. The term “slicing” is used herein to the selection of a set or subset of stack components for testing.

[0124] As described previously, the idea of simulation-based testing is to run a simulated driving scenario that an ego agent must navigate under the control of the stack 1700 being tested. Typically, the scenario includes a static drivable area (e.g. a particular static road layout) that the ego agent is required to navigate, typically in the presence of one or more other dynamic agents (such as other vehicles, bicycles, pedestrians etc.). To this end, simulated inputs 2703 are provided from the simulator 2702 to the stack 1700 under testing.

[0125] The slicing of the stack dictates the form of the simulated inputs 2703. By way of example, Figure 20 shows the prediction, planning and control systems 1704, 1706 and 1708 within the AV stack 100 being tested. To test the full AV stack of Figure 17, the perception system 1702 could also be applied during testing. In this case, the simulated inputs 2703 would comprise synthetic sensor data that is generated using appropriate sensor model(s) and processed within the perception system 1702 in the same way as real sensor data. This requires the generation of sufficiently realistic synthetic sensor inputs (such as photorealistic image data and / or equally realistic simulated lidar / radar data etc.). The resulting outputs of the perception system 1702 would, in turn, feed into the higher-level prediction and planning systems 1704, 1706.

[0126] By contrast, so-called “planning-level” simulation would essentially bypass the perception system 1702. The simulator 2702 would instead provide simpler, higher-level inputs 2703 directly to the prediction system 1704. In some contexts, it may even be appropriate to bypass the prediction system 1704 as well, in order to test the planner 1706 on predictions obtained directly from the simulated scenario (i.e. “perfect” predictions).

[0127] Between these extremes, there is scope for many different levels of input slicing, e.g. testing only a subset of the perception system 1702, such as “later” (higher-level) perception components, e.g. components such as filters or fusion components which operate on the outputs from lower-level perception components (such as object detectors, bounding box detectors, motion detectors etc.).

[0128] Whatever form they take, the simulated inputs 2703 are used (directly or indirectly) as a basis for decision-making by the planner 1708. The controller 1708, in turn, implements the planner’s decisions by outputting control signals 1709. In a real-world context, these control signals would drive the physical actor system 1712 of AV. In simulation, an ego vehicle dynamics model 2704 is used to translate the resulting control signals 1709 into realistic motion of the ego agent within the simulation, thereby simulating the physical response of an autonomous vehicle to the control signals 1709.

[0129] Alternatively, a simpler form of simulation assumes that the ego agent follows each planned trajectory exactly between planning steps. This approach bypasses the control system 1708 (to the extent it is separable from planning) and removes the need for the ego vehicle dynamic model 2704. This may be sufficient for testing certain facets of planning.

[0130] To the extent that external agents exhibit autonomous behaviour / decision making within the simulator 2702, some form of agent decision logic 2710 is implemented to carry out those decisions and determine agent behaviour within the scenario. The agent decision logic 2710 may be comparable in complexity to the ego stack 1700 itself or it may have a more limited decision-making capability. The aim is to provide sufficiently realistic external agent behaviour within the simulator 2702 to be able to usefully test the decision-making capabilities of the ego stack 1700. In some contexts, this does not require any agent decision making logic 2710 at all (open-loop simulation), and in other contexts useful testing can be provided using relatively limited agent logic 2710 such as basic adaptive cruise control (ACC). One or more agent dynamics models 2706 may be used to provide more realistic agent behaviour if appropriate.

[0131] A scenario is run in accordance with a scenario description 2701a and (if applicable) a chosen parameterization 2701b of the scenario (generally denoted by ^ herein). A scenario typically has both static and dynamic elements which may be “hard coded” in the scenario description 2701a or configurable and thus determined by the scenario description 2701a in combination with a chosen parameterization 2701b. In a driving scenario, the static element(s) typically include a static road layout.

[0132] The dynamic element(s) typically include one or more external agents within the scenario, such as other vehicles, pedestrians, bicycles etc.

[0133] The extent of the dynamic information provided to the simulator 2702 for each external agent can vary. For example, a scenario may be described by separable static and dynamic layers. A given static layer (e.g. defining a road layout) can be used in combination with different dynamic layers to provide different scenario instances. The dynamic layer may comprise, for each external agent, a spatial path to be followed by the agent together with one or both of motion data and behaviour data associated with the path. In simple open-loop simulation, an external actor simply follows the spatial path and motion data defined in the dynamic layer that is non-reactive i.e. does not react to the ego agent within the simulation. Such open-loop simulation can be implemented without any agent decision logic 2710. However, in closed-loop simulation, the dynamic layer instead defines at least one behaviour to be followed along a static path (such as an ACC behaviour). In this case, the agent decision logic 2710 implements that behaviour within the simulation in a reactive manner, i.e. reactive to the ego agent and / or other external agent(s). Motion data may still be associatedwith the static path but in this case is less prescriptive and may for example serve as a target along the path. For example, with an ACC behaviour, target speeds may be set along the path which the agent will seek to match, but the agent decision logic 2710 might be permitted to reduce the speed of the external agent below the target at any point along the path in order to maintain a target headway from a forward vehicle.

[0134] As will be appreciated, scenarios can be described for the purpose of simulation in many ways, with any degree of configurability. For example, the number and type of agents, and their motion information may be configurable as part of the scenario parameterization 2701b.

[0135] The output of the simulator 2702 for a given simulation includes an ego trace 2712a of the ego agent and one or more agent traces 2712b of the one or more external agents (traces 2712). Each trace 2712a, 2712b is a complete history of an agent’s behaviour within a simulation having both spatial and motion components. For example, each trace 2712a, 2712b may take the form of a spatial path having motion data associated with points along the path such as speed, acceleration, jerk (rate of change of acceleration), snap (rate of change of jerk) etc.

[0136] Additional information is also provided to supplement and provide context to the traces 2712. Such additional information is referred to as “contextual” data 2714. The contextual data 2714 pertains to the physical context of the scenario, and can have both static components (such as road layout) and dynamic components (such as weather conditions to the extent they vary over the course of the simulation). To an extent, the contextual data 2714 may be "passthrough" in that it is directly defined by the scenario description 2701a or the choice of parameterization 2701b, and is thus unaffected by the outcome of the simulation. For example, the contextual data 2714 may include a static road layout that comes from the scenario description 2701a or the parameterization 2701b directly. However, typically the contextual data 2714 would include at least some elements derived within the simulator 2702. This could, for example, include simulated environmental data, such as weather data, where the simulator 2702 is free to change weather conditions as the simulation progresses. In that case, the weather data may be time-dependent, and that time dependency will be reflected in the contextual data 2714.

[0137] The test oracle 2752 receives the traces 2712 and the contextual data 2714, and scores those outputs in respect of a set of performance evaluation rules 2754. The performance evaluation rules 2754 are shown to be provided as an input to the test oracle 2752.

[0138] The rules 2754 are categorical in nature (e.g. pass / fail-type rules). Certain performance evaluation rules are also associated with numerical performance metrics used to “score” trajectories (e.g. indicating a degree of success or failure or some other quantity that helps explain or is otherwise relevant to the categorical results). The evaluation of the rules 2754 is time-based – a given rule may have a different outcome at different points in the scenario. The scoring is also time-based: for each performance evaluation metric, the test oracle 2752 tracks how the value of that metric (the score) changes over time as the simulation progresses. The test oracle 2752 provides an output 2756 comprising a time sequence 256a of categorical (e.g. pass / fail) results for each rule, and a score-time plot 2756b for each performance metric, as described in further detail later. The results and scores 2756a, 2756b are informative to the expert 1722 and can be used to identify and mitigate performance issues within the tested stack 1700. The test oracle 2752 also provides an overall (aggregate) result for the scenario (e.g. overall pass / fail). The output 2756 of the test oracle 2752 is stored in a test database 2758, in association with information about the scenario to which the output 2756 pertains. For example, the output 2756 may be stored in association with the scenario description 2710a (or an identifier thereof), and the chosen parameterization 2701b. As well as the time-dependent results and scores, an overall score may also be assigned to the scenario and stored as part of the output 2756. For example, an aggregate score for each rule (e.g. overall pass / fail) and / or an aggregate result (e.g. pass / fail) across all of the rules 2754. An overall score may be assigned to a given rule (aggregated over time), or to multiple rules (aggregated over time and over the multiple rules), which is one way to implement the driving performance metric ^(^) described above.

[0139] Figure 21 illustrates another choice of slicing and uses reference numerals 1700 and 1700S to denote a full stack and sub-stack respectively. It is the sub-stack 1700S that would be subject to testing within the testing pipeline 2700 of Figure 20.

[0140] A number of “later” perception components 1702B form part of the sub-stack 1700S to be tested and are applied, during testing, to simulated perception inputs 2703. The later perception components 1702B could, for example, include filtering or other fusion components that fuse perception inputs from multiple earlier perception components.

[0141] In the full stack 1700, the later perception components 1702B would receive actual perception inputs 2713 from earlier perception components 1702A. For example, the earlier perception components 1702A might comprise one or more 2D or 3D bounding box detectors, in which case the simulated perception inputs provided to the late perception components could include simulated 2D or 3D bounding box detections, derived in the simulation via ray tracing. The earlier perception components 1702A would generally include component(s) that operate directly on sensor data. With the slicing of Figure 21, the simulated perception inputs 2703 would correspond in form to the actual perception inputs 2713 that would normally be provided by the earlier perception components 1702A.

[0142] However, the earlier perception components 1702A are not applied as part of the testing. In some implementations, they may be used to train one or more perception error models 2708 that can be used to introduce realistic error, in a statistically rigorous manner, into the simulated perception inputs 2703 that are fed to the later perception components 1702B of the sub-stack 1700 under testing. In other examples, a perception error models may be constructed in a more heuristic fashion. Manually tuned PEMS may be used.

[0143] In a simulation context, “perfect” ground truth perception inputs 2703G are provided by the simulator, but these are used to derive more realistic (ablated) perception inputs 2703 with realistic error introduced by the perception error models(s) 2708. The perception error model(s) 2708 serve as a “surrogate model” (being a surrogate for the perception system 1702, or part of the perception system 1702A, but operating on lower-fidelity inputs). Non- trained perception error models may also be used.

[0144] Perception Error Models are surrogate models used in simulation to replicate the outputs of perception systems so that downstream tasks can be evaluated as realistically as possible.

[0145] A PEM may take the form of a machine learning (ML) model that is trained on training examples of the form (^^, ^^̂) where and ^^̂ (training label) is a real perceived state (or sequence of perceived states) obtained with a real perception system, such as the perception system 1702 (or part of it) and ^^(input to the PEM) is a corresponding ground truth state (or time sequence of groundtruth states). Note, in this this context, the ground truth is an input to the PEM, and the real perceived state provides the training label. To generate the PEM training data, the ground truth state may be determined though manual annotation, offline- processing (e.g. using a non-real time perception system with better performance than a real-time perception system) or a combination thereof. Alternatively, in place of ^^̂, the label may take the form of an error ^^measured between ^^and ^^̂. A learned PEM has one or more parameters ^ (such as weights) that are fitted to the training examples(^^, ^^̂)or(^^, ^^). A probabilistic PEM consumes simulator-generated groundtruth ^ and returns a distribution ^^(^̂|^) or ^^(^|^)from which different perceived states (or perception errors) may be sampled. The parameter(s) ^ may be parameter(s) of the distribution itself (such as variance), or parameter(s) of a machine learning model used to generate the distribution (for example, parameters of a neural network that consumes ^ and generates the distribution as a function of ^).

[0146] In the adversarial perception error search described above, perceived states / perception errors are not sampled from a PEM. Rather, they are obtained using the random step generator ^(^|^^^^). However, in certain implementations, a probabilistic PEM may be used as a perception scoring function, whereby an error ^ sampled from ^(^|^^^^) is scored based on a likelihood of the PEM given the sample: ℒ(^|^, ^)= ^^(^|^) (e.g. based on a logarithm of the likelihood). In this context, ‘good’ perception performance is assessed relative to actual planner performance: if ^^(^|^) is relatively low for a given error e then e is an outlier under the PEM (classed as ‘poor’ perception performance in this context). An outlier perception error might lead to planner failure, but this is not necessarily informative, particularly if the probability of the error is very low. However, if ^^(^|^) is relatively high for a given error e (classed as ‘good’ performance in this context, meaning that the error e aligns with ‘normal’ operation of the modelled perception system), and the error e causes planner failure, that is a highly salient test outcome (because a non-outlier perception error has resulted in planner failure).

[0147] By way of example, any of the following PEMs might be applied in the present context. Piazzoni et al.

[0032] present a PEM for the pose and class of dynamic objects, where the error distribution is conditioned on the weather variables, and use the model to validate an autonomous vehicle system in simulation on urban driving tasks. Piazzoni et al. [7] describe a similar approach using a time dependent model and a model for false negative detections. Time dependent perception PEMs have also been used by Berkhahn et al.

[0033] to model traffic intersections with a behaviour model and a stochastic process misperception model on velocity, and Hirsenkorn et al.

[0034] by creating a Kernel Density Estimator model of a filtered radar sensor, where the simulated sensor is modelled by a Markov process. Zec et al.

[0031] propose to model an off the shelf perception system using a Hidden Markov Model. Modern machine learning techniques have also been used to create PEMs, for example Krajewski et al.

[0036] create a probabilistic neural network model for a LiDAR sensor, Arnelid et al.

[0037] use Recurrent Conditional Generative Adversarial Networks to model the output of a fused camera and radar sensor system, and Suhre and Malik

[0038] describe an approach for simulating a radar sensor using conditional variational auto-encoders. In Mitra et al.

[0039] , an autoregressive neural network PEM is evaluated in simulation by visually comparing ego behaviour to the behaviour under no perception errors. In Sadeghi et al.

[0017] , the behaviour of a neural network PEM in the Carla simulator is studied in a large scale urban driving simulation and the behaviour compared to an object detector and simulation with ground truth perception. In Innes and Ramamoorthy [9] a neural network PEM is deployed in an emergency braking scenario in the Carla simulator, and an efficient importance sampling strategy is used to reduce the number of samples required to estimate the probability of collision. Philipp et al.

[0040] consider a low dimensional perception error model, and verify the amount of perceptual error which can be applied without a collision occurring in a simple scenario. Reeb et al. [8] describes an alternative to PEMs, where instead the input and output distributions of each component in a modular system under simulation are bounded relative to their real-world distributions, and hence the behaviour of the system as a whole is simulated truthfully.

[0148] A test orchestration component 2760 is responsible for selecting scenarios for the purpose of simulation. For example, the test orchestration component 2760 may select scenario descriptions 2701a and suitable parameterizations 2701b automatically, which may be based on the test oracle outputs 2756 from previous scenarios and / or other criteria.

[0149] Figure 22 shows a graphical user interface (GUI) 2203 used to display the final output of the algorithm 2201. The final output may be the final minimum perception performance requirement, such as the final perception score threshold ^^^^^^^. A rendering component 2202 receives the output of the algorithm and cases the output to be indicated on the GUI 2203.

[0150] References herein to components, functions, modules and the like, denote functional components of a computer system which may be implemented at the hardware level in various ways. A computer system comprises execution hardware which may be configured to execute the method / algorithmic steps disclosed herein and / or to implement amodel trained using the present techniques. The term execution hardware encompasses any form / combination of hardware configured to execute the relevant method / algorithmic steps. The execution hardware may take the form of one or more processors, which may be programmable or non-programmable, or a combination of programmable and non- programmable hardware may be used. Examples of suitable programmable processors include general purpose processors based on an instruction set architecture, such as CPUs, GPUs / accelerator processors etc. Such general-purpose processors typically execute computer readable instructions held in memory coupled to or internal to the processor and carry out the relevant steps in accordance with those instructions. Other forms of programmable processors include field programmable gate arrays (FPGAs) having a circuit configuration programmable through circuit description code. Examples of non- programmable processors include application specific integrated circuits (ASICs). Code, instructions etc. may be stored as appropriate on transitory or non-transitory media (examples of the latter including solid state, magnetic and optical storage device(s) and the like). The subsystems 1702-1708 of the runtime stack Figure 17 may be implemented in programmable or dedicated processor(s), or a combination of both, on-board a vehicle or in an off-board computer system in the context of testing and the like. The various components of Figure 20, such as the simulator 2702 and the test oracle 2752 may be similarly implemented in programmable and / or dedicated hardware. Experiments System Setup

[0151] Firstly, an autonomous vehicle system is constructed. Figure 5 shows the system configuration at runtime. A detector 501 takes as input image and lidar data and outputs detections. A tracker 502 takes as input the detections from the detector 501 and outputs tracked detections. A planner 503 takes as input the tracked detections from the tracker 502. This is how the system would be configured in the real world; our testing strategy aims to understand the performance of this system as a whole by understanding the impact of perception errors on downstream components. A BEVFusion detector was used for the perception part of our system

[0057] . The BEVFusion detector will output 2d bounding boxes with pose, size and velocity in the birds eye view space (orthographic view).

[0152] Tracker: Before BEVFusion detections were passed to the planner, they were first pre-processed by a Kalman filter birds eye view bounding box tracker with a constant velocity state dynamics model with association based on the Hungarian assignment algorithm for box distance, with association threshold 2 meters

[0058] . Tracks are considered confirmed after 1 observation and deleted after not being observed for a period of 1 second, which was chosen because this is the time between planning decisions so the percieved state provided to the planner can depend on all obervations since the planner was last called. The observation space of the filter is the 2d bounding box pose, size and velocity in the birds eye view representation. Only the false negative, orientation error and spatial position error properties are parameterised, and therefore the ground truth velocity and size are provided as tracker observations.

[0153] Planners and tasks: The following planners will be tested on the tasks listed below, which are expected to be contained in the operational design domain of their respective planners. ^ ObP: An optimisation based planner

[0014] . 1. A T-junction right turn, into nearside lane whilst avoiding traffic 2. Overtaking a stationary vehicle with a vehicle moving in the oncoming lane 3. Turning left at a T-junction, into far side lane across oncoming traffic ^ IDM: A path following planner based on the Intelligent Driver Model [15, 59] 1. Lane following a constant velocity vehicle 2. Lane following a vehicle which overtakes a stationary vehicle

[0154] Figure 6 shows diagrams of all test scenarios. Thevehicle boxes 612-616 represent ego vehicles , the vehicle boxes 601-606 each represent a moving vehicle and thevehicle boxes 607-611 represent vehicles that are stationary. Figure 6(a) shows vehicle 612 overtaking a stationary vehicle 607-608 with a vehicle 601 moving in the oncoming lane. Figure 6(b) shows the vehicle 613 turning left at a T-junction, into the far side lane across oncoming traffic 602-603. Figure 6(c) shows vehicle 614 doing a T-junction right turn, into nearside lane whilst avoiding traffic 604. Figure 6(d) shows the ego vehicle 615 lanefollowing a constant velocity vehicle 605. Figure 6(e) shows vehicle 616 lane following a vehicle 606 which overtakes a stationary vehicle 611.

[0155] Perception Error Model: Using a Perception Error Model (PEM), it is aimed to show that the present system is robust to typical perception errors. A PEM is trained as in Sadeghi et al.

[0017] and then sampled at test time. The perception error model is a lightweight feedforward network trained to predict existence of true positive detections and spatial errors on true positive detections. The PEM was trained to approximate the detection performance of the BEVFusion detector

[0057] on the nuScenes dataset

[0016] . This is achieved by running the BEVFusion detection on the nuScenes dataset and then associating the ground truth object labels to detections to obtain lists of true positive and false positive detections, which can be used for training when it is assumed that the PEM model factorises over objects.

[0156] A lightweight “low fidelity rendering” is also run to obtain percentage occlusion for each object. In order to calculate the percentage occlusion for each agent in the scene birds eye view representation of the scene is converted to a radial birds eye view representation in Ego coordinates by converting each agent into an arc segment in radial space which bounds the minimum distance of the agent from Ego and bounds the angular coordinate from Ego. Then the arcs are sorted by distance from Ego and checks are done for overlap between each arc segment and all closer arc segments. The percentage occlusion is calculated as the largest percentage of the arc segment which is intersected by any closer arc segment. Since the calculated value is not guaranteed to be equal to the percentage occlusion which would be calculated by any individual sensor using a high fidelity 3D calculation, the calculated quantity is termed low fidelity pseudo occlusion. Figure 7 shows how a birds eye view representation of a vehicle 700 in the Ego frame can be converted to arc segment representations 701-704.

[0157] The nuScenes validation set was split sequentially, with the first 90% of scenes used to train the model and the final 10% of scenes used for testing. Figure 8 shows metrics comparing the PEM and BEVFusion detector on nuScenes test set. The line 801 shows the Translation score of the PEM; the line 802 shows the precision score of the PEM; the line 803 shows the recall score of the PEM. The line 804 shows the translation score of the BEVFusion detector; the line 805 shows the precision score of the BEVFusion detector; line 806 shows the recall score of the BEVFusion detector. Figure 9 shows the correlation of the log likelihood of the PEM against the NDS and NDS-t metrics on the nuScenes val dataset.The PEM log likelihood shares a correlation coefficient of 0.4 with the NDS metric, and a correlation coefficient of 0.08 with the NDS-t metric. The NDS and NDS-t metrics share a correlation coefficeient of 0.46. No collisions were observed when taking 100 random samples from the PEM in each scenario; it is therefore claimed that the present system is robust to most typical errors produced by the perception error model.

[0158] The position error is not predicted by independent normal distributions, but instead by a Multivariate Student T distribution which allows for a more accurate characterisation of errors. The PEM is trained by using the log likelihood of this distribution in place of the log likelihood of the normal distributions. The inputs to the neural network were the position, extent, yaw of the agent concatenated with a one hot encoding of the object class of the object. Figure 10 shows the architecture of the network for Neural Surrogate. Fully connected layers are shown as fc blocks 1001-1015 where the number represents the output channel size. The blocks 1016-1020 represent tanh activation function layers. The blocks 1021-1025 represent the sum or pooling layers. The blocks 1026-1029 represent the dropout layers. The dropout probability was set to 0.2 to prevent overfitting. Other methods to form the perception error set – binned sets

[0159] Each x in s is divided into separate categories based on categories of the ground truth agents, e.g. how occluded the agent is from the ego vehicle’s sensors or the distance of the agent from ego. Functionsare defined which are only sensitive to the elements of each category, e.g.could calculate NDS on unoccluded objects closer than 10m only. The value of each is calculated from a labelled test set to obtain. Thenis obtained, i.e. situations where the performance in any category differs significantly from the performance observed on a test set are penalised.

[0160] The disadvantage of this procedure is that data is divided into smaller amounts when more categories are introduced and hence potentially a large amount of labelled sensor data will be required. The advantage is that the properties of the perception system in different situations will be properly understood. Adversarial Attacks using Perception Metrics

[0161] The effectiveness of the present algorithm is illustrated in a variety of settings. All results have been obtained exclusively using the recompute algorithm shown in Algorithm 2. The first heuristic (bisection) part of the present algorithm is evaluated separately without the random search component.

[0162] Perception Metric Functions: To specify the fuzzy setsused in the present search algorithms,the following functions were used for^ nuScenes detection score (NDS) a weighted combination of mean average precision and various true positive errors (translation error, orientation error, etc.)

[0016] . ^ NDS with continuous false negative penalty (NDS-t): Equal weighted sum of the NDS and a term penalising the longest fraction of the track which is a continuous false negative.

[0163] Figure 11 shows histograms of the perception metric functions on the nuScenes val dataset for the BEVFusion detector. The histogram 1101 corresponds to the NDS metric, and the histogram 1102 corresponds to the NDS-t metric. In all perception metric functions any other object categories than cars are removed, since cars are the only category of actor used in this study.

[0164] Random search proposal: In the random search part of the present algorithm a proposal distribution is used where false negatives from the original heuristic search are replaced with true positives for a random segment length of the false negative part of the track, where the new true positive detections will be shifted by a constant random spatial perturbation from their true location and have some random jitter applied to their orientation.

[0165] Results: Although the present system was robust to perception errors produced by sampling the PEM, it is nevertheless able to produce errors which score highly for various m whilst still causing failures. To do this the largestobtained for all rule failing scenarios tested by the present method, i.e. those for which r(τ) < 0, is evaluated. Figure 12 shows a plot of the cumulative largest m achieved for each method in a failing scenario against the number of rollouts made. The cumulative highest m is that obtained when attacking using the perception metrics to specify the fuzzy perception set (only includesfailing scenarios). The heuristic search is represented by solid line, whilst the random search is represented by dashed line. The plot 1201 shows the cumulative largest NDS score achieved for an ObP overtake scenario (1211), an IDM lane following scenario (1212), an ObP right turn scenario (1213), an IDM overtake follow scenario (1214), and an ObP left turn scenario (1215). The plot 1202 shows the cumulative largest NDS-t score achieved for an ObP overtake scenario (1221), an IDM lane following scenario (1222), an ObP right turn scenario (1223), an IDM overtake follow scenario (1224), and an ObP left turn scenario (1225). These are the scenarios shown in Figure 6.

[0166] It is observed that for all metrics the heuristic search obtains quite a strong result, which in most scenarios is improved by the random search. In Table 1 properties of the obtained errors are compared. It can be seen that in most cases the random search significantly reduces the number of false negatives whilst only slightly increasing the mean position and orientation errors. It is observed that in most cases using the NDS-t metric usually leads to a reduction in the Longest Continuous FN property. However this is not always the case, indicating that the random search is not always effective in searching the space.It is observed that the perception metric value of the samples is higher than many of those seen on an independent test set in Figure 11, and hence this evaluation does not enable the exclusion of the possibility that these planner failures could occur in reality, despite not being predicted by the PEM.

[0167] Figure 13 shows example frames from the observed perception errors for an overtaking scenario (A), a right turn scenario (B) and a left turn scenario (C). The vehicles 1301, 1303, and 1305 are ego vehicles, other vehicles ground truth positions are shown with outlines. Block rectangles 1302, 1304, and 1306 show vehicle positions with a perception error in position injected.. Solid blocks 1307 represent false negative errors.

[0168] In scenario A, the first two frames show examples of position errors for the obtained error for this scene. The final frame shows the ego vehicle 1301 colliding with the vehicle 1302 which was observed with position errors. In scenario B, the first two frames show examples of position errors for the obtained error for this scene. The final frames shows the ego vehicle 1303 colliding with the vehicle 1307 which is now a false negative. In scenario C, the first frame shows no perception errors. The next two frames show examples of position errors for the obtained error for this scene. The final frame shows the ego vehicle 1305 colliding with the vehicle 1306.Table 1: Summary of highest obtained m^errors for random search. For the heuristic algorithm, the multiple obtained errors which were then used to initialize the random search algorithm are shown.Adversarial Attack with Perception Error Model

[0169] It is also possible to run the attack algorithm using the log likelihood of the perception error model for, so that the α cuts of the fuzzy perception sets are defined as superlevel sets of probability density for the PEM. This is fundamentally different to the approach in the previous section because although both perception metrics and the PEM log likelihood are a function of formperception metrics provide a way to measure the quality of detections from a detector, but the log likelihood of a PEM merely attempts to score how similar detections are to those from a specific detector.

[0170] Results: Figure 14 shows a plot of the cumulative largest PEM log likelihood achieved for each method in a failing scenario against the number of rollouts made. Figure 14 shows the largest PEM log likelihood achieved for an ObP overtake scenario (1411), an IDM lane following scenario (1412), an ObP right turn scenario (1413), an IDM overtake follow scenario (1414), and an ObP left turn scenario (1415). For context, Figure 15 showshistograms of the obtained log likelihood for the perception error model on the nuScenes val dataset for the BEVFusion detector. The nuScenes val dataset was split in a 0.9:0.1 ratio to create train and test datasets for the PEM which are themselves independent to the nuScenes train dataset used to train BEVFusion. Histogram 1501 shows the log likelihood of the PEM model on scenes used for training the PEM. Histogram 1502 shows the log likelihood of the PEM model on scenes used to test the PEM model. It is observed that for all metrics the heuristic search obtains quite a strong result, which in most scenarios is improved by the random search. In Table 2 properties of the obtained errors are compared. It is observed that compared to using the NDS metric, the random search with PEM metric yields errors with larger PEM log likelihood and in some sense these errors may be considered more typical according to the PEM. It is noted that that increasing the PEM log likelihood does not necessarily correspond to increasing the perception metrics (NDS and NDS-t) but is instead a complicated function of object distance, occlusion, false negatives, true positives and other considerations. It can be observed for the PEM metric that the log likelihood of the samples is higher than many of those seen on an independent test set for the training data in Figure 15. From a hypothesis testing perspective, this indicates that the possibility of such errors occurring in reality using this PEM model and dataset cannot be excluded. Table 2: Summary of highest obtained PEM log likelihood errors with random search Analysis of the adversarial boundary Figure 16 shows the results of applying random perturbations of increasing strength to the obtained rule breaking errors from the previous section. This is achieved by randomly flipping the false negative property in the error with a specific probability (here termed the perturbation strength) and adding random spatial noise. Plot 1601 shows the value of the perception metric as a function of perturbation strength for the PEM log likelihood (1603), the NDS metric (1604), and the NDS-t metric (1605). It is observed that the percentage ofrule breaking instances increases quickly as the strength of the perturbation increases. Plot 1602 shows the value of the adversarial accuracy as a function of perturbation strength when using the PEM log likelihood (1606), the NDS metric (1607), and the NDS-t metric (1608). There is some noise in the plot leading to non-monotonic decrease in adversarial accuracy because the average of 10 random perturbations was used. This indicates that although small perturbations around the obtained attacks do not result in failures, larger perturbations do result in failures and hence the failure surface has a complex structure, which is different to that of the perception metric. Therefore, it is argued that the obtained attacks can be seen as adversarial in the same sense as image based adversarial attacks because they seem to be highly structured and localised in the perception space.

[0171] The method has thus been shown to be capable of identifying erroneous perception system outputs which cause failures in modular autonomous vehicles with widely used components. Surprisingly, these failures occur despite the identified perception errors appearing benign when analysed with common perception metrics. A modified Boundary Attack algorithm uses a combination of heuristic and random search to identify these failures for black box driving systems and simulators that do not provide gradients whilst only requiring access to the detections from the system and the plans executed by the system. Experimental results are provided to demonstrate that the algorithm works reasonably well in practice on a number of driving scenarios that are relevant to the industrial deployment of autonomous vehicle systems. It is observed that making small changes to these errors result in the autonomous vehicle system no longer failing; hence the present results can be seen as somewhat analogous to adversarial attacks for image classifiers.

[0172] Tt is clear that considering the impact of the performance of the perception system on the planner in autonomous vehicle testing is necessary because planner rule functions are not always highly correlated with perception metrics in severe failure cases. If testing for the planner and perception systems is performed separately then practitioners should take care to use a perception metric which is as correlated as possible with the planner behaviour in failure cases. If this cannot be achieved then the present algorithm proposed could be applied to identify adversarial perception errors which could be fixed either by ensuring the perception system does not exhibit these cases via analysis of real world data, or instead ensuring that the planner is robust to these perception errors, for example by making the planner more conservative in the errors it assumes are associated with the perception system.References

[0173] The following are incorporated by reference herein in their entirety: [1] Philip Koopman and Michael Wagner. Challenges in autonomous vehicle testing and validation. SAE International Journal of Transportation Safety, 4(1):15–24, 2016. [2] Philip Koopman and Frank Fratrik. How many operational design domains, objects, and events? Safeai@ aaai, 4, 2019. [3] Stefan Riedmaier, Thomas Ponn, Dieter Ludwig, Bernhard Schick, and Frank Diermeyer. Survey on scenario-based safety assessment of automated vehicles. IEEE access, 8:87456–87477, 2020. [4] Philip Koopman, Uma Ferrell, Frank Fratrik, and Michael Wagner. A safety standard approach for fully autonomous vehicles. In Computer Safety, Reliability, and Security: SAFECOMP 2019 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Turku, Finland, September 10, 2019, Proceedings 38, pages 326–332. Springer, 2019. [5] Jonah Philion, Amlan Kar, and Sanja Fidler. Learning to evaluate perception models using planner-centric metrics. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 14055–14064, 2020. [6] Philip Koopman and Michael Wagner. Toward a framework for highly automated vehicle safety validation. SAE Technical Paper, Tech. Rep, 2018. [7] Andrea Piazzoni, Jim Cherian, Martin Slavik, and Justin Dauwels. Modeling perception errors towards robust decision making in autonomous vehicles. In IJCAI, 2020. [8] David Reeb, Kanil Patel, Karim Barsim, Martin Schiegg, and Sebastian Gerwinn. Validation of composite systems by discrepancy propagation. arXiv preprint arXiv:2210.12061, 2022. [9] Craig Innes and Subramanian Ramamoorthy. Testing rare downstream safety violations via upstream adaptive sampling of perception error models. In 2023 IEEE International Conference on Robotics and Automation, 2023.

[0010] Anthony Corso, Peter Du, Katherine Driggs-Campbell, and Mykel J Kochenderfer. Adaptive stress testing with reward augmentation for autonomous vehicle validatio. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 163–168. IEEE, 2019.

[0011] Jonathan Sadeghi, Romain Mueller, and John Redford. An active learning reliability method for systems with partially defined performance functions. arXiv preprint arXiv:2210.02168, 2022.

[0012] Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representations, 2018.

[0013] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017.

[0014] Francisco Eiras, Majd Hawasly, Stefano V Albrecht, and Subramanian Ramamoorthy. A two-stage optimization-based motion planner for safe urban driving. IEEE Transactions on Robotics, 38(2):822–834, 2021.

[0015] Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical review E, 62(2):1805, 2000.

[0016] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.

[0017] Jonathan Sadeghi, Blaine Rogers, James Gunn, Thomas Saunders, Sina Samangooei, Puneet Kumar Dokania, and John Redford. A step towards efficient evaluation of complex perception tasks in simulation. arXiv preprint arXiv:2110.02739, 2021.

[0018] Hans-Georg Beyer and Bernhard Sendhoff. Robust optimization–a comprehensive survey. Computer methods in applied mechanics and engineering, 196(33-34):3190–3218, 2007.

[0019] Jonathan Uesato, Ananya Kumar, Csaba Szepesvari, Tom Erez, Avraham Ruderman, Keith Anderson, Krishnamurthy Dj Dvijotham, Nicolas Heess, and Pushmeet Kohli. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. In International Conference on Learning Representations, 2018.

[0020] Anthony Corso, Sydney Katz, Craig Innes, Xin Du, Subramanian Ramamoorthy, and Mykel J Kochenderfer. Risk-driven design of perception systems. Advances in Neural Information Processing Systems, 35:9894– 9906, 2022.

[0021] M Hohenbichler. An asymptotic formula for the probability of intersections. Berichte zur Zuverlassigkeitstheorie der Bauwerke, (69):21–48, 1984.

[0022] Karl Wilhelm Breitung. Univariate integrals, pages 45–50. Springer Berlin Heidelberg, Berlin, Heidelberg, 1994. ISBN 978-3-540-49033-3. doi: 10.1007 / BFb0073542.

[0023] Lotfi Asker Zadeh. Fuzzy sets as a basis for a theory of possibility. Fuzzy sets and systems, 1(1):3–28, 1978.

[0024] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.

[0025] Samira Pouyanfar, Muneeb Saleem, Nikhil George, and Shu-Ching Chen. Roads: Randomization for obstacle avoidance and driving in simulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.

[0026] Aravind Balakrishnan. Closing the modelling gap: Transfer learning from a low- fidelity simulator for autonomous driving. Master’s thesis, University of Waterloo, 2020.

[0027] Ahmad El Sallab, Ibrahim Sobh, Mohamed Zahran, and Nader Essam. Lidar sensor modeling and data augmentation with gans for autonomous driving. arXiv preprint arXiv:1905.07290, 2019.

[0028] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:13049–13061, 2020.

[0029] Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. AdvSim: Generating safety-critical scenarios forself-driving vehicles. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9909–9918, June 2021.

[0030] Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Are we making real progress in simulated environments? measuring the sim2real gap in embodied visual navigation. arXiv preprint arXiv:1912.06321, 2019.

[0031] Maria Lyssenko, Christoph Gladisch, Christian Heinzemann, Matthias Woehrle, and Rudolph Triebel. Towards safety-aware pedestrian detection in autonomous systems. In 2022 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), pages 293– 300, 2022. doi: 10.1109 / IROS47612.2022.9981309.

[0032] Andrea Piazzoni, Jim Cherian, Martin Slavik, and Justin Dauwels. Modeling sensing and perception errors towards robust decision making in autonomous vehicles. arXiv preprint arXiv:2001.11695, 2020.

[0033] Volker Berkhahn, Marcel Kleiber, Johannes Langner, Chris Timmermann, and Stefan Weber. Traffic dynamics at intersections subject to random misperception. IEEE Transactions on Intelligent Transportation Systems, pages 1–11, 2021. doi: 10.1109 / TITS.2020.3045480.

[0034] Nils Hirsenkorn, Timo Hanke, Andreas Rauch, Bernhard Dehlink, Ralph Rasshofer, and Erwin Biebl. Virtual sensor models for real-time applications. Advances in Radio Science, 14:31–37, 2016.

[0035] Edvin Listo Zec, Nasser Mohammadiha, and Alexander Schliep. Statistical sensor modelling for autonomous driving using autoregressive input- output hmms. In 201821st International Conference on Intelligent Transportation Systems (ITSC), pages 1331–1336. IEEE, 2018.

[0036] Robert Krajewski, Michael Hoss, Adrian Meister, Fabian Thomsen, Julian Bock, and Lutz Eckstein. Using drones as reference sensors for neural-networks-based modeling of automotive perception errors. In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 708–715, 2020. doi: 10.1109 / IV47402.2020.9304615.

[0037] Henrik Arnelid, Edvin Listo Zec, and Nasser Mohammadiha. Recurrent conditional generative adversarial networks for autonomous driving sensor modelling. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 1613–1618. IEEE, 2019.

[0038] Alexander Suhre and Waqas Malik. Simulating object lists using neural networks in automotive radar. In 201819th International Conference on Thermal, Mechanical and Multi- Physics Simulation and Experiments in Microelectronics and Microsystems (EuroSimE), pages 1–5. IEEE, 2018.

[0039] Pallavi Mitra, Apratirn Choudhury, Vimal Rau Aparow, Giridharan Kulandaivelu, and Justin Dauwels. Towards modeling of perception errors in autonomous vehicles. In 201821st International Conference on Intelligent Transportation Systems (ITSC), pages 3024–3029. IEEE, 2018.

[0040] Robin Philipp, Hedan Qian, Lukas Hartjen, Fabian Schuldt, and Falk Howar. Simulation-based elicitation of accuracy requirements for the environmental perception of autonomous vehicles. In Leveraging Applications of Formal Methods, Verification and Validation: 10th International Symposium on Leveraging Applications of Formal Methods, ISoLA 2021, Rhodes, Greece, October 17–29, 2021, Proceedings 10, pages 129–145. Springer, 2021.

[0041] Abraham M Hasofer and Niels C Lind. Exact and invariant second-moment code format. Journal of the Engineering Mechanics division, 100(1):111–121, 1974.

[0042] Rüdiger Rackwitz and Bernd Flessler. Structural reliability under combined random load sequences. Computers & structures, 9(5):489–494, 1978.

[0043] Bernd Möller and Michael Beer. Fuzzy randomness: uncertainty in civil engineering and computational mechanics. Springer Science & Business Media, 2004.

[0044] B Möller, W Graf, and M Beer. Fuzzy structural analysis using α-level optimization. Computational mechanics, 26(6):547–565, 2000.

[0045] Yu Inatsu, Shogo Iwazaki, and Ichiro Takeuchi. Active learning for distributionally robust level-set estimation. In International Conference on Machine Learning, pages 4574– 4584. PMLR, 2021.

[0046] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.

[0047] Mohammadhossein Bahari, Saeed Saadatnejad, Ahmad Rahimi, Mohammad Shaverdikondori, Amir Hossein Shahidzadeh, Seyed-Mohsen Moosavi-Dezfooli, andAlexandre Alahi. Vehicle trajectory prediction works, but not everywhere. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17123–17133, June 2022.

[0048] Rebecca Roelofs, Liting Sun, Benjamin Caine, Khaled S. Refaat, Benjamin Sapp, Scott M. Ettinger, and Wei Chai. Causalagents: A robustness benchmark for motion forecasting using causal relationships. ArXiv, abs / 2207.03586, 2022.

[0049] Anthony Corso, Robert Moss, Mark Koren, Ritchie Lee, and Mykel Kochenderfer. A survey of algorithms for black-box safety validation of cyber-physical systems. J. Artif. Int. Res., 72:377–428, October 2021. ISSN 1076-9757. doi: 10.1613 / jair.1.12716.

[0050] Xinhai Zhang, Jianbo Tao, Kaige Tan, Martin Törngren, José Manuel Gaspar Sánchez, Muhammad Rusyadi Ramli, Xin Tao, Magnus Gyllenhammar, Franz Wotawa, Naveen Mohan, et al. Finding critical scenarios for automated driving systems: A systematic literature review. arXiv preprint arXiv:2110.08664, 2021.

[0051] Aman Sinha, Matthew O’Kelly, Russ Tedrake, and John C Duchi. Neural bridge sampling for evaluating safety-critical autonomous systems. Advances in Neural Information Processing Systems, 33, 2020.

[0052] Halil Beglerovic, Michael Stolz, and Martin Horn. Testing of autonomous vehicles using surrogate models and stochastic optimization. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pages 1–6. IEEE, 2017.

[0053] Sai Vemprala and Ashish Kapoor. Adversarial attacks on optimization based planners. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9943– 9949. IEEE, 2021.

[0054] Mark Koren, Saud Alsaif, Ritchie Lee, and Mykel J Kochenderfer. Adaptive stress testing for autonomous vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1–7. IEEE, 2018.

[0055] Mark Koren and Mykel J Kochenderfer. Efficient autonomy validation in simulation with adaptive stress testing. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 4178–4183. IEEE, 2019.

[0056] Niklas Hanselmann, Katrin Renz, Kashyap Chitta, Apratim Bhattacharyya, and Andreas Geiger. King: Generating safety-critical driving scenarios for robust imitation via kinematics gradients. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVIII, pages 335–352. Springer, 2022.

[0057] Zhijian Liu, Haotian Tang, Alexander Amini, Xingyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation. In IEEE International Conference on Robotics and Automation (ICRA), 2023.

[0058] David A Forsyth and Jean Ponce. Computer Vision: A Modern Approach. Always learning. Pearson, 2012. ISBN 9780273764144.

[0059] Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with misconceptions about learning-based vehicle motion planning. arXiv preprint arXiv:2306.07962, 2023.

Claims

CLAIMS 1. A computer-implemented method of performance testing a mobile robot planner in simulation in the presence of perception errors, the method comprising: receiving a ground truth state of a test scenario; generating a first perturbed scenario state based on the ground truth state; computing a first perception performance score for the first perturbed scenario state based on a perception scoring function applied to the first perturbed scenario state and the ground truth state; responsive to determining that the first perception performance score exceeds a first minimum perception performance requirement, selecting the first perturbed scenario for use in testing the mobile robot planer; performing a simulation rollout process based on the selected first perturbed scenario, the first perception performance score and the first minimum perception performance requirement by: generating a first mobile robot action using the mobile robot planner under testing applied to the selected first perturbed scenario state, evaluating the first mobile robot action based on a planning performance evaluation rule to obtain a first planning performance evaluation result, determining a second minimum perception performance requirement as a function of: the first perception performance score, the first minimum perception performance requirement and the first planning performance evaluation result, selecting a second perturbed scenario state having a second perception performance score that exceeds the second minimum perception performance requirement, and repeating the simulation rollout process based on the selected second perturbed scenario state, the second perception performance score and the second perception performance requirement;wherein the simulation rollout process is repeatedly performed until reaching a termination condition, the method further comprising outputting a performance evaluation result pertaining to the mobile robot planner under testing based on a final minimum perception performance requirement as determined on reaching the termination condition.

2. The method of claim 1, the first mobile robot action is evaluated to determine whether a saliency condition is satisfied; wherein the second minimum perception performance requirement is determined so to as match: the first minimum performance requirement in the event the saliency condition is not satisfied, or the first perception performance score in the event the saliency condition is satisfied.

3. The method of claim 2, wherein the saliency condition is failure on the planning performance evaluation rule.

4. The method of claim 1, 2 or 3, comprising, the method further comprising: receiving an initial perception error, wherein the first perturbed scenario state is generated by perturbing the groundtruth scenario state based on a first perception error, the first perception error determined based on the initial perception error; wherein the second perturbed scenario state is generated based on a second perception error, the second perception error determined based on the first perception error.

5. The method of claim 4, wherein of the first and second perception errors are determined using a random error step generator of the form ^(^|^^), wherein the first perception error ^^is sampled from ^(^|^^),where ^^is the initial perception error, and the second perception error ^^is sampled from ^(^|^^).

6. The method of claim 4 or 5, wherein the first minimum perception performance requirement is determined based on the initial perception error.

7. The method of claim 4, 5 or 6, wherein the initial perception error is determined in an initial search phase by repeatedly running simulation rollouts with different perception errors to obtain different planning performance evaluation results, the initial perception error selected as a minimized perception error for which a salient planning performance evaluation result is obtained.

8. The method of claim 7, wherein each performance evaluation result comprises a planning performance score, the initial perception error selected as a minimized perception error that results in a planning performance score approximately equal to a planning performance threshold.

9. The method of any preceding claim, wherein the ground truth state forms part of a time series of ground truth states, and the first perturbed scenario state forms part of a first time series of perturbed scenario states generated based on the time series of ground truth states, wherein the first perception performance score pertains to the first time series of perturbed scenario states as a whole; wherein the second perturbed scenario state forms part of a second time series of perturbed scenario states, wherein the second perception performance score pertains to the second time series of perturbed scenario states as a whole.

10. The method of claim 9 when dependent on claim 8, wherein each perception error comprises a time series of error components, wherein the initial search phase comprises:performing multiple simulation rolls out based on multiple time series of error components, and selecting as the initial perception error the time series of error components having a minimum error duration.

11. The method of any preceding claim, wherein each state comprises a plurality of agent states, wherein the first perturbed scenario state is generated based on a sampled perception error for each agent state of the ground truth state.

12. The method of claim 11 when dependent on claim 10, wherein the initial perception error is selected as the time series of error components having a minimum duration of false negative agent detection.

13. The method of any preceding claim, wherein the second perception performance score is computed based on a perception scoring function applied to the first perturbed scenario state and the ground truth state.

14. The method of any of claims 1 to 12, wherein the ground truth state is an initial ground truth state, wherein the second perception performance score is determined based on a perception scoring function applied to the first perturbed scenario state and a first ground truth state dependent on the first mobile robot action.

15. The method of any preceding claim, wherein the perception scoring function has the form of a learned perception error model (PEM) for modelling a real perception system.

16. A computer system comprising one or more computers configured to implement the method of any preceding claim.

17. Transitory or non-transitory media embodying computer-readable instructions, which are configured when executed on one or more processors to cause the one or more processors to implement the method of any of claims 1 to 15.

Images