Support tools for mobile robots

Abstract

A computer-implemented method of generating a static layer and / or a dynamic layer for a test scenario for performance testing a robotic system in a simulation environment, the method comprising: inputting to a geometric constraint solver a scenario skeleton defining geometric elements and geometric constraints on the geometric elements, the geometric elements being modifiable based on a set of geometric variables, and the geometric constraints being characterised by one or more constraint parameters, thereby causing the geometric constraint solver to determine one or more functions of the constraint parameters for returning values of the geometric variables that satisfy the geometric constraints; receiving value(s) of the constraint parameter(s); using the received value(s) of the constraint parameter(s) and the one or more functions to compute values of the geometric variables satisfying the geometric constraints; and configuring based on the received value(s) of the constraint parameter(s), the computed value(s) of the geometric variables and the scenario skeleton, the static layer for the test scenario and / or the dynamic layer for the test scenario.

Description

Support Tools for Mobile RobotsTechnical Field

[0001] The present disclosure pertains to support tools for autonomous vehicles and other mobile robots.Background

[0002] There have been major and rapid developments in the field of autonomous vehicles. An autonomous vehicle (AV) is a vehicle which is equipped with sensors and control systems which enable it to operate without a human controlling its behaviour. An autonomous vehicle is equipped with sensors which enable it to perceive its physical environment, such sensors including for example cameras, radar and lidar. Autonomous vehicles are equipped with suitably programmed computers which are capable of processing data received from the sensors and making safe and predictable decisions based on the context which has been perceived by the sensors.

[0003] An autonomous vehicle may be fully autonomous (in that it is designed to operate with no human supervision or intervention, at least in certain circumstances) or semi- autonomous. Semi-autonomous systems require varying levels of human oversight and intervention. An Advanced Driver Assist System (ADAS) and certain levels of Autonomous Driving System (ADS) may be classed as semi-autonomous. A “level 5” vehicle is one that can operate entirely autonomously in any circumstances, because it is always guaranteed to meet some minimum level of safety. Such a vehicle would not require manual controls (steering wheel, pedals etc.) at all. By contrast, level 3 and level 4 vehicles can operate fully autonomously but only within certain defined circumstances (e.g. within geofenced areas). A level 3 vehicle must be equipped to autonomously handle any situation that requires an immediate response (such as emergency braking); however, a change in circumstances may trigger a “transition demand”, requiring a driver to take control of the vehicle within some limited timeframe. A level 4 vehicle has similar limitations; however, in the event the driver does not respond within the required timeframe, a level 4 vehicle must also be capable of autonomously implementing a “minimum risk maneuver” (MRM), i.e. some appropriate action(s) to bring the vehicle to safe conditions (e.g. slowing down and parking the vehicle). A level 2 vehicle requires the driver to be ready to intervene at any time, and it is the responsibility of the driver to intervene if the autonomous systems fail to respond properly atany time. With level 2 automation, it is the responsibility of the driver to determine when their intervention is required; for level 3 and level 4, this responsibility shifts to the vehicle’s autonomous systems and it is the vehicle that must alert the driver when intervention is required.

[0004] Physical world testing will remain an important factor in the testing of autonomous vehicles’ capability to make safe and predictable decisions. However, physical world testing is expensive and time-consuming. Moreover, as the safety of AV systems improves, the number of incidents per mile driven decreases. To achieve widespread acceptance of AVs, it is estimated that an AV should cause no more than 1 severe accident per 10A9 miles driven (compared with 1 such accident per 10A6 miles for the average human driver). Identifying and mitigating issues based solely on ‘miles driven’ in the real-world is simply not viable for an AV operating even several orders of magnitude below the minimum performance level that will ultimately be required.

[0005] As such, increasing reliance is placed on testing using simulated environments. If there is to be an increase in testing in simulated environments, it is desirable that such environments can reflect as far as possible real-world scenarios. Autonomous vehicles need to have the facility to operate in the same wide variety of circumstances that a human driver can operate in. Such circumstances can incorporate a high level of unpredictability. Increasing attention is being placed on the creation of simulation environments which can provide such testing in a manner that gives confidence that the test outcomes represent potential real behaviour of an autonomous vehicle.

[0006] Scenarios must be described with sufficient detail and precision in a form conducive to high-quality simulation, otherwise the results of testing will not sufficiently reflect real- world AV performance. A typical driving scenario includes a static road layout and various dynamic agents (other vehicles, pedestrians, cyclists, animals etc.) that an autonomous vehicle (the ego vehicle) is required to navigate. An ego vehicle may be required to predict the motion of other agents and plan safely within a complex road network. In an offline context, a scenario description may be required as an input to a simulator to facilitate simulation-based testing of an autonomous vehicle stack prior to deployment on a real-world vehicle.

[0007] By way of example, ASAM OpenSCENARIO (R) defines a file format for the description of dynamic driving scenario content. OpenSCENARIO may be used togetherwith the ASAM OpenDRIVE (R) schema for describing static road networks. The stated purpose of OpenDRIVE “is to provide a road network description that can be fed into simulations to develop and validate ADAS and AD [Autonomous Driving] features”; see ASAM OpenDRIVE VI.7.0 (and accompanying User Guide), Release Date 3 August 2021 [available at https: / / www.asam.net / standards / detail / opendrive / ],

[0008] Other mobile robots are being developed, for example for carrying freight supplies in internal and external industrial zones. Such mobile robots would have no people on board and belong to a class of mobile robot termed UAV (unmanned autonomous vehicle). Autonomous air mobile robots (drones) are also being developed.Summary

[0009] “Saliency” (ensuring that scenarios are actually useful for testing) is a core problem in scalable simulation-based testing of robotic systems. When testing the performance of a robotic system over a large number of scenarios, it is important to have some guarantee of saliency. It is straightforward to generate and run numerous simulation scenarios at scale. However, ensuring that those scenarios remain salient is much more challenging. One class of non-salient scenarios are those that are simply unrealistic. Performance testing a mobile robot in scenarios that are not sufficiently reflective of the real-world will not yield informative results. As another example, a parameterized scenario might be designed to test an ego agent’s response to a certain type of event (such as a cut-in by another agent) that does not actually occur for all param eterizations of the scenario. Test results obtained in nonsalient scenarios provide limited insights and, worse, can skew the results in a way that gives a misleading picture of real-world performance. A worst case scenario would be a batch of simulation test results on non-salient scenarios that indicate an AV system is performing to a given level of safety that it would not actually be able to achieve in the real world.

[0010] With large scale simulation of the kind required in AV testing it is not feasible to manually check each scenario for saliency. Recall that a ‘safe’ AV vehicle should cause no more than 1 severe accident per 10A9 miles driven. Validating this level of performance in simulation would require a number of simulations that is several orders of magnitude higher than 10A9.

[0011] Herein, a scenario design tool is provided that allows test scenarios to be generated in a flexible manner, with saliency constraints hard-coded at the design stage.

[0012] A first aspect of the present disclosure provides a computer-implemented method of generating a static layer and / or a dynamic layer for a test scenario for performance testing a robotic system in a simulation environment, the method comprising: inputting to a geometric constraint solver a scenario skeleton defining geometric elements and geometric constraints on the geometric elements, the geometric elements being modifiable based on a set of geometric variables, and the geometric constraints being characterised by one or more constraint parameters, thereby causing the geometric constraint solver to determine one or more functions of the constraint parameters for returning values of the geometric variables that satisfy the geometric constraints; receiving value(s) of the constraint parameter(s); using the received value(s) of the constraint parameter(s) and the one or more functions to compute values of the geometric variables satisfying the geometric constraints; and configuring based on the received value(s) of the constraint parameter(s), the computed value(s) of the geometric variables and the scenario skeleton, the static layer for the test scenario and / or the dynamic layer for the test scenario.

[0013] In embodiments, the method may comprise: receiving second value(s) of the constraint parameter independent variable(s); using the received second value(s) of the constraint param eter(s) and the one or more functions to compute second values of the geometric variables satisfying the geometric constraints; and configuring based on the received second value(s) of the constraint parameter(s), the computed second value(s) of the geometric variables and the scenario skeleton, a second static layer for the test scenario and / or a second dynamic layer for the test scenario.

[0014] In this manner, the same solution (the functions) can be used to generate multiple static / dynamic layers satisfying the constraints with different value(s) of the constraint parameter(s) (by simply ‘plugging in’ new values of the constraint parameters to obtain new values of the geometric variables) without having to re-solve the constraints. This allows multiple salient variations of a scenario (all satisfying a designer’s intent, as expressed in the constraints) to be generated with increased computational efficiently for testing purposes, and the solution to the constraint problem need only be determined once.

[0015] The method may comprise receiving initial value(s) of the geometric variables. The received value(s) of the constraint parameter(s) and the received initial values of the geometric variables may not satisfy the geometric constraints, and the received initialvalue(s) of the geometric variables may be provided to the geometric constraint solver for use in determining the functions of the constraint parameter(s).

[0016] The geometric constraints may be expressed in the form of a constraints graph, and the geometric solver may use a plurality of predetermined deduction rules to extract the function(s) from the constraints graph.

[0017] The geometric elements may comprise road elements and the geometric constraints may comprise constraints on the road elements. The received and computed values may be used to generate the static layer, which comprises a road layout satisfying the geometric constraints.

[0018] The scenario skeleton may be generated based on scenario creation inputs received at a graphical user interface.

[0019] The scenario skeleton may be generated based on a selected region of a map, and at least some of the geometric elements may correspond to respective map elements within the selected region of the map.

[0020] The method may comprise storing associations between at least some geometric elements and the respective map elements, and the static layer may be generated from the map based on the values of the geometric variables and the stored associations.

[0021] The geometric constraints may comprise inter-element geometric constraints between respective subsets of the geometric elements.

[0022] The method may comprise executing a test scenario comprising at least one of the static layer and the dynamic layer, with an ego agent of the test scenario controlled by a robotic planner under testing; and generating by a test oracle a set of test results for evaluating performance of the robotic planner in the test scenario.

[0023] The robotic planner may be tested in combination with one or more other components, e.g. one or more of: a perception system, a prediction system and a controller.

[0024] The method may comprise using the set of test results to identify and mitigate an issue in the robotic planner or another component tested in combination with the robotic planner.

[0025] Further aspects herein provide a computer system comprising one or more computers configured to implement the method of the first aspect or any embodiment thereof, andcomputer-readable instructions stored in transitory or non-transitory media and configured, when executed in a computer system, to implement the same.Brief Description of Figures

[0026] Illustrative embodiments will now be described, by way of example only, with reference to the following schematic figures, in which:

[0027] FIG. 1 A shows a schematic function block diagram of an autonomous vehicle stack;

[0028] FIG. IB shows a schematic overview of an autonomous vehicle testing paradigm;

[0029] FIG.1C shows a schematic block diagram of a scenario extraction pipeline;

[0030] FIG.2 shows a schematic block diagram of a testing pipeline;

[0031] FIG.3 shows a schematic block diagram of a visualization component for rendering a graphical user interface on which test results are displayed;

[0032] FIG.4 shows a view available within a graphical user interface;

[0033] FIG.5 shows an example road network;

[0034] FIG.6 shows part of a road network annotated with OpenDRIVE elements used to describe the road network;

[0035] FIG.7 shows a lane graph of a static road layout;

[0036] FIG.8 shows a schematic block diagram of a scenario design tool;

[0037] FIG. 9 illustrates how a scenario skeleton may be used to generate or modify a road layout within a map;

[0038] FIG. 10 shows an example of a geometric constraint problem with multiple solutions.Detailed Description

[0039] A scenario design tool is described herein, which is capable of building a dimensioned geometrical skeleton for a map (or, more generally, a static layer for a test scenario). The skeleton may be generated ‘from scratch’, or alternatively using an existing map fragment as a template. The map skeleton geometry is then added to a constraint solver with a required set of dimensions and constraints (imposed at the design stage), and solved in a default position.The scenario skeleton has variables that can be modified and the constraints introduce dependencies between those variables. The constraints are characterised by parameters, which are treated as independent variables (referred to as ‘dimensions’ or ‘constraint parameters’) and the geometric variables are treated as dependent variables (the ‘unknowns’). In a geometric context, the dependent variables may be referred to as ‘geometries’. The choice of independent variables is indicated to the constraint solver and the solver is tasked with solving for the dependent variables. The solver uses, primarily, a set of predetermined deduction rules to analytically derive a solution, which means determining each dependent variable as a function of one or more of the independent variables (constraint parameters / dimensions). Once that is done, the skeleton can be manipulated continuously, using its associated dimensions (e.g., distances, lengths, angles, etc.), and deformed in ways that maintain the designer’s desired invariants encapsulated in the constraints.

[0040] A constraint problem is defined by a set of variables, Y, and a set of constraints on the variables characterized by dimensions, X, which are quantities treated as independent variables. In the choice of X, the scenario designer is indicating that a user should retain control over those variables. For a given choice of {Y, X} the aim is to solve for the dependent variables Y = {j / ,- } (dependent variables / unknowns). Solving for Y ‘analytically’ implies finding some function for each dependent variable, ytE Y, such that yt= ft (X) satisfies the constraints. The solution is a set of functions F = {ft}, and is generally valid for multiple values of X (it may be valid for all possible values of X or only a subset of possible values). Once a solution has been found for a given choice of {Y, X], it can be used multiple times to generate variations of the skeleton (all of which satisfy the constraints) by ‘plugging in’ different values of X for which the solution is valid.

[0041] A constraint problem may be ‘under-defined’, or it may be well-defined but with multiple solution branches. As described in further detail below, under-defined problems and well-defined problems with multiple solutions can be handled in different ways, and the response of the tool to such a problem may be context-dependent. An analytical solution is generally valid for a range of X-values. Nevertheless, an initial choice of X or Y values may, in some cases, guide how the solver resolves an under-defined problem or a well-defined problem with multiple solution branches (e.g. favouring a solution that is valid for the initial choice of dimension values X, or a solution that is closer to the initial choice of geometry values K).

[0042] A binding between the dimensional map skeleton and the static layer is created, and as a user manipulates the map skeleton, the static layer (which is typically much richer in content than the skeleton) reacts to the changes dynamically. In this manner, the tool can generate a continuum of map variants that can then be used to form the basis of an Automated Driving System test suite. Preferably, the transformations are chosen so that the map topology (or static layer topology) is invariant under these transformations, making it relatively straightforward to execute the same dynamic layer(s) on each of the map variants without the need to materially change the dynamic layer(s).

[0043] A scenario ‘test suite’ refers to a set of concrete scenarios (whose geometry is defined) that are related to some degree (for example, a test suite might correspond to ‘all overtaking scenarios’ or some defined subset of overtaking scenarios, or a subset of scenarios with a particular road topology). In the present context, a scenario test suite may be defined by a scenario skeleton, with variations (concrete scenarios) generated from different parameterizations of the skeleton (all of which satisfy the constraints imposed at the design stage). Note that the term ‘scenario’ may be used herein to refer to a concrete scenario or a wider-ranging test suite, and the meaning shall be clear from the context.

[0044] Stationary parked vehicles and occluding objects that might be considered part of the map can be included in the map skeleton and so the approach is fairly powerful.

[0045] The design tool can also be applied to impose some level of constraint on dynamic agents in the scenario, as described in further detail below.

[0046] Test scenarios are considered, in which an ego agent is required to navigate a real or modelled physical context. The ego agent is a real or simulated mobile robot that moves under the control of the stack under testing. The physical context includes static and / or dynamic element(s) that the stack under testing is required to respond to effectively. For example, the mobile robot may be a fully or semi-autonomous vehicle under the control of the stack (the ego vehicle). The physical context may comprise a static road layout and a given set of environmental conditions (e.g. weather, time of day, lighting conditions, humidity, pollution / particulate level etc.) that could be maintained or varied as the scenario progresses. A dynamic scenario additionally includes one or more other agents (“external” agent(s), e.g. other vehicles, pedestrians, cyclists, animals etc.).

[0047] In an offline simulation context, a scenario description is provided to an offline simulator as input, in order to expose a stack under testing to a simulated scenario, and test its performance in the simulated scenario. This allows performance issues (such as safety issues) to be identified and mitigated prior to real-world deployment. In an online context, a perception system may be used to generate a scenario description that can be used as a basis for higher-level functions, such as motion prediction and planning, which might involve some form of online simulation to simulate possible futures and plan accordingly.

[0048] The present design tool is described below in the context of offline simulation, to generate scenario test suites for scalable performance testing.

[0049] A scenario description may be encoded using a scenario description language (SDL), or in any other form that can be consumed by whichever component(s) require it. As briefly discussed, the ASAM OpenDRIVE (R) standard defines a storage format for the static description of road networks and OpenSCENARIO (R) may be used to add dynamic content. Other forms of scenario description may be used, including bespoke languages and formats, and the present techniques are not limited to any particular SDL, storage format, schema or standard.

[0050] A “scenario run” or “scenario instance” refers to a concrete occurrence of an agent(s) navigating a physical context, optionally in the presence of one or more other agents. A single scenario description can give rise to multiple simulated runs, with different outcomes, not least because those outcomes depend on decisions taken by the stack under testing (as changes are made to the stack to improve performance, that may change the outcome on a given concrete scenario). The terms “run” and “instance” are used interchangeably in this context.

[0051] Figure 1 A shows, by way of context, a highly schematic block diagram of an AV runtime stack 100. The stack 100 may be fully or semi-autonomous. For example, the stack 100 may operate as an Autonomous Driving System (ADS) or Advanced Driver Assist System (ADAS).

[0052] The run time stack 100 is shown to comprise a perception system 102, a prediction system 104, a planning system (planner) 106 and a control system (controller) 108.

[0053] In a real -world context, the perception system 102 receives sensor outputs from an onboard sensor system 110 of the AV, and uses those sensor outputs to detect external agentsand measure their physical state, such as their position, velocity, acceleration etc. The onboard sensor system 110 can take different forms but generally comprises a variety of sensors such as image capture devices (cameras / optical sensors), lidar and / or radar unit(s), satellitepositioning sensor(s) (GPS etc.), motion / inertial sensor(s) (accelerometers, gyroscopes etc.) etc. The onboard sensor system 110 thus provides rich sensor data from which it is possible to extract detailed information about the surrounding environment, and the state of the AV and any external actors (vehicles, pedestrians, cyclists etc.) within that environment. The sensor outputs typically comprise sensor data of multiple sensor modalities such as stereo images from one or more stereo optical sensors, lidar, radar etc. Sensor data of multiple sensor modalities may be combined using filters, fusion components etc.

[0054] The perception system 102 typically comprises multiple perception components which co-operate to interpret the sensor outputs and thereby provide perception outputs to the prediction system 104.

[0055] In a simulation context, depending on the nature of the testing, it may or may not be necessary to model the on-board sensor system 100. For example, when only the planner 106 (or only the planner 106 and controller 108) are tested on a ‘perfect’ representation of the scenario (that is, directly on simulator ground truth), simulated sensor data is not required therefore complex sensor modelling is not required. Surrogate model(s) of the perception system 102 (or part(s) of it) can also be used to test planner performance (or planner and controller performance) in the presence of realistic perception errors, without the use of sensor models.

[0056] The perception outputs from the perception system 102 are used by the prediction system 104 to predict future behaviour of external actors (agents), such as other vehicles in the vicinity of the AV.

[0057] Predictions computed by the prediction system 104 are provided to the planner 106, which uses the predictions to make autonomous driving decisions to be executed by the AV in a given driving scenario. The inputs received by the planner 106 would typically indicate a drivable area and would also capture predicted movements of any external agents (obstacles, from the AV’s perspective) within the drivable area. The driveable area can be determined using perception outputs from the perception system 102 in combination with map information, such as an HD (high definition) map.

[0058] A core function of the planner 106 is the planning of trajectories for the AV (ego trajectories), taking into account predicted agent motion. This may be referred to as trajectory planning. A trajectory is planned in order to carry out a desired goal within a scenario. The goal could for example be to enter a roundabout and leave it at a desired exit; to overtake a vehicle in front; or to stay in a current lane at a target speed (lane following). The goal may, for example, be determined by an autonomous route planner 120, also referred to as a goal generator 120.

[0059] The controller 108 executes the decisions taken by the planner 106 by providing suitable control signals to an on-board actor system 112 of the AV. In particular, the planner 106 plans trajectories for the AV and the controller 108 generates control signals to implement the planned trajectories. Typically, the planner 106 will plan into the future, such that a planned trajectory may only be partially implemented at the control level before a new trajectory is planned by the planner 106. The actor system 112 includes “primary” vehicle systems, such as braking, acceleration and steering systems, as well as secondary systems (e.g. signalling, wipers, headlights etc.).

[0060] The example of Figure 1 A considers a relatively “modular” architecture, with separable perception, prediction, planning and control systems 102-108. The sub-stack themselves may also be modular, e.g. with separable planning modules within the planning system 106. For example, the planning system 106 may comprise multiple trajectory planning modules that can be applied in different physical contexts (e.g. simple lane driving vs. complex junctions or roundabouts). This is relevant to simulation testing for the reasons noted above, as it allows components (such as the planning system 106 or individual planning modules thereof) to be tested individually or in different combinations. For the avoidance of doubt, with modular stack architectures, the term stack can refer not only to the full stack but to any individual sub-system or module thereof.

[0061] The extent to which the various stack functions are integrated or separable can vary significantly between different stack implementations - in some stacks, certain aspects may be so tightly coupled as to be indistinguishable. For example, in other stacks, planning and control may be integrated (e.g. such stacks could plan in terms of control signals directly), whereas other stacks (such as that depicted in Figure 1 A) may be architected in a way that draws a clear distinction between the two (e.g. with planning in terms of trajectories, and with separate control optimizations to determine how best to execute a planned trajectory at thecontrol signal level). Similarly, in some stacks, prediction and planning may be more tightly coupled. At the extreme, in so-called “end-to-end” driving, perception, prediction, planning and control may be essentially inseparable. Unless otherwise indicated, the perception, prediction, planning, and control terminology used herein does not imply any particular coupling or modularity of those aspects.

[0062] A “full” stack typically involves everything from processing and interpretation of low-level sensor data (perception), feeding into primary higher-level functions such as prediction and planning, as well as control logic to generate suitable control signals to implement planning-level decisions (e.g. to control braking, steering, acceleration etc.). For autonomous vehicles, level 3 stacks include some logic to implement transition demands and level 4 stacks additionally include some logic for implementing minimum risk maneuvers. The stack may also implement secondary control functions e.g. of signalling, headlights, windscreen wipers etc.

[0063] Whilst the following description refers to the stack 100 in the context of testing, testing may be applied to individual components / portions of the stack, such as the perception, prediction, planning or control stacks 102, 104, 106 (alone or in various combinations), or individual component(s) thereof. A stack (or component) can refer purely to software, i.e. one or more computer programs that can be executed on one or more general-purpose computer processors. However, such terminology can also encompass hardware. In simulation, software of the stack may be tested on a “generic” off-board computer system, before it is eventually uploaded to an on-board computer system of a physical vehicle.However, in “hardware-in-the-loop” testing, the testing may extend to underlying hardware of the vehicle itself. For example, the stack software may be run on the on-board computer system (or a replica thereof) that is coupled to the simulator for the purpose of testing. In this context, the stack under testing extends to the underlying computer hardware of the vehicle. As another example, certain functions of the stack 100 (e.g. perception functions) may be implemented in dedicated hardware. In a simulation context, hardware-in-the loop testing could involve feeding synthetic sensor data to dedicated hardware perception components.

[0064] Within the stack 100, a scenario description 116 may be used as a basis for planning and prediction. The scenario description 116 is generated using the perception system 102, together with a high-definition (HD) map 114. By localizing the ego vehicle 114 on the map, it is possible to combine the information extracted in the perception system 102 (includingdynamic agent information) with the pre-existing environmental information contained in the HD map 114. The scenario description 116 is, in turn, used as a basis for motion prediction in the prediction system 104, and the resulting motion predictions 118 are used in combination with the scenario description 116 as a basis for planning in the planning system 106.

[0065] Figure IB shows a highly schematic overview of a testing paradigm for autonomous vehicles. An ADS / ADAS stack 100, e.g. of the kind depicted in Figure 1 A, is subject to repeated testing and evaluation in simulation, by running multiple scenario instances in a simulator 202, and evaluating the performance of the stack 100 (and / or individual subs-stacks thereof) in a test oracle 252. The output of the test oracle 252 is informative to an expert 122 (team or individual), allowing them to identify issues in the stack 100 and modify the stack 100 to mitigate those issues (S124). The results also assist the expert 122 in selecting further scenarios for testing (S126), and the process continues, repeatedly modifying, testing and evaluating the performance of the stack 100 in simulation. The improved stack 100 is eventually incorporated (S125) in a real-world AV 101, equipped with a sensor system 110 and an actor system 112. The improved stack 100 typically includes program instructions (software) executed in one or more computer processors of an on-board computer system of the vehicle 101 (not shown). The software of the improved stack is uploaded to the AV 101 at step S125. Step S125 may also involve modifications to the underlying vehicle hardware. On board the AV 101, the improved stack 100 receives sensor data from the sensor system 110 and outputs control signals to the actor system 112. Real-world testing (S128) can be used in combination with simulation-based testing. For example, having reached an acceptable level of performance through the process of simulation testing and stack refinement, appropriate real -world scenarios may be selected (SI 30), and the performance of the AV 101 in those real scenarios may be captured and similarly evaluated in the test oracle 252.

[0066] Figure 1C shows a highly schematic block diagram of the testing pipeline applied to a scenario description 148. A scenario generator 146 generates the scenario description 148 in a form that may be consumed by the simulator 202, allowing multiple simulated runs to be derived therefrom (for example, with different configurations of the AV stack 100). Ground truth 150 is provided for each simulated run.

[0067] The ground truth 150 for each run comprises a trace of the ego agent and any other dynamic agents in the scenario. A trace is a history of an agent’s location and motion over the course of a scenario. There are many ways a trace can be represented. Trace data will typically include spatial and motion data of an agent within the environment.

[0068] Further details of an example testing pipeline incorporating the test oracle 252 will now be described.

[0069] Figure 2 shows a schematic block diagram of the testing pipeline, denoted by reference numeral 200. The testing pipeline 200 is shown to comprise the simulator 202 and the test oracle 252. The simulator 202 runs simulated scenarios for the purpose of testing all or part of an AV run time stack 100, and the test oracle 252 evaluates the performance of the stack (or sub-stack) on the simulated scenarios. As discussed, it may be that only a sub-stack of the run-time stack is tested, but for simplicity, the following description refers to the (full) AV stack 100 throughout. However, the description applies equally to a sub-stack in place of the full stack 100. The term “slicing” is used herein to the selection of a set or subset of stack components for testing.

[0070] The idea of simulation-based testing is to run a simulated driving scenario that an ego agent must navigate under the control of the stack 100 being tested. Typically, the scenario includes a static drivable area (e.g. a particular static road layout) that the ego agent is required to navigate, typically in the presence of one or more other dynamic agents (such as other vehicles, bicycles, pedestrians etc.). To this end, simulated inputs 203 are provided from the simulator 202 to the stack 100 under testing.

[0071] The slicing of the stack dictates the form of the simulated inputs 203. By way of example, Figure 2 shows the prediction, planning and control systems 104, 106 and 108 within the AV stack 100 being tested. To test the full AV stack of Figure 1 A, the perception system 102 could also be applied during testing. In this case, the simulated inputs 203 would comprise synthetic sensor data that is generated using appropriate sensor model(s) and processed within the perception system 102 in the same way as real sensor data. This requires the generation of sufficiently realistic synthetic sensor inputs (such as photorealistic image data and / or equally realistic simulated lidar / radar data etc.). The resulting outputs of the perception system 102 would, in turn, feed into the higher-level prediction and planning systems 104, 106.

[0072] By contrast, so-called “planning-level” simulation would essentially bypass the perception system 102. The simulator 202 would instead provide simpler, higher-level inputs 203 directly to the prediction system 104. In some contexts, it may even be appropriate to bypass the prediction system 104 as well, in order to test the planner 106 on predictions obtained directly from the simulated scenario (i.e. “perfect” predictions).

[0073] Between these extremes, there is scope for many different levels of input slicing, e.g. testing only a subset of the perception system 102, such as “later” (higher-level) perception components, e.g. components such as filters or fusion components which operate on the outputs from lower-level perception components (such as object detectors, bounding box detectors, motion detectors etc.).

[0074] As an alternative to generating high-fidelity synthetic sensor data that is fed to the perception system 102, all or part of the perception system 102 may be modelled using one or more “surrogate” perception models. A surrogate model models the perception system 102 (or some component / components thereof), but operates more efficiently on lower-fidelity inputs derived from the simulation ground truth, without the need for sensor models. A surrogate model allows the simulated inputs 203 to be generated with realistic error (comparable to the error that would be produced by the perception system 102 in a comparable real-world situation) without the use of sensor models.

[0075] Whatever form they take, the simulated inputs 203 are used (directly or indirectly) as a basis for decision-making by the planner 106. The controller 108, in turn, implements the planner’s decisions by outputting control signals 109. In a real -world context, these control signals would drive the physical actor system 112 of AV. In simulation, an ego vehicle dynamics model 204 is used to translate the resulting control signals 109 into realistic motion of the ego agent within the simulation, thereby simulating the physical response of an autonomous vehicle to the control signals 109.

[0076] Alternatively, a simpler form of simulation assumes that the ego agent follows each planned trajectory exactly between planning steps. This approach bypasses the control system 108 (to the extent it is separable from planning) and removes the need for the ego vehicle dynamic model 204. This may be sufficient for testing certain facets of planning.

[0077] To the extent that external agents exhibit autonomous behaviour / decision making within the simulator 202, some form of agent decision logic 210 is implemented to carry outthose decisions and determine agent behaviour within the scenario. The agent decision logic 210 may be comparable in complexity to the ego stack 100 itself or it may have a more limited decision-making capability. The aim is to provide sufficiently realistic external agent behaviour within the simulator 202 to be able to usefully test the decision-making capabilities of the ego stack 100. In some contexts, this does not require any agent decision making logic 210 at all (open-loop simulation), and in other contexts useful testing can be provided using relatively limited agent logic 210 such as basic adaptive cruise control (ACC). One or more agent dynamics models 206 may be used to provide more realistic agent behaviour if appropriate.

[0078] A scenario is run in accordance with a scenario description 201, which typically has both static and dynamic elements. Scenario runs are orchestrated by a test orchestration component 260.

[0079] The static elements are encoded in a static layer 201a of the scenario description 201. The static element(s) typically include a road layout. For example, the static layer 201a may be encoded in the OpenDrive format, or a comparable road network SDL that describes both the geometry and the topology of the road layout.

[0080] The dynamic element(s) are encoded in a dynamic layer 201b, and typically include one or more external agents within the scenario, such as other vehicles, pedestrians, bicycles etc.

[0081] The extent of the dynamic information provided to the simulator 202 for each external agent can vary. For example, a scenario may be described by separable static and dynamic layers. A given static layer (e.g. defining a road layout) can be used in combination with different dynamic layers to provide different scenario instances. The dynamic layer may comprise, for each external agent, a spatial path to be followed by the agent together with one or both of motion data and behaviour data associated with the path. In simple open-loop simulation, an external actor simply follows the spatial path and motion data defined in the dynamic layer that is non-reactive i.e. does not react to the ego agent within the simulation. Such open-loop simulation can be implemented without any agent decision logic 210. However, in closed-loop simulation, the dynamic layer instead defines at least one behaviour to be followed along a static path (such as an ACC behaviour). In this case, the agent decision logic 210 implements that behaviour within the simulation in a reactive manner, i.e. reactive to the ego agent and / or other external agent(s). Motion data may still be associatedwith the static path but in this case is less prescriptive and may for example serve as a target along the path. For example, with an ACC behaviour, target speeds may be set along the path which the agent will seek to match, but the agent decision logic 210 might be permitted to reduce the speed of the external agent below the target at any point along the path in order to maintain a target headway from a forward vehicle.

[0082] The output of the simulator 202 for a given simulation includes an ego trace 212a of the ego agent and one or more agent traces 212b of the one or more external agents (traces 212). Each trace 212a, 212b is a complete history of an agent’s behaviour within a simulation having both spatial and motion components. For example, each trace 212a, 212b may take the form of a spatial path having motion data associated with points along the path such as speed, acceleration, jerk (rate of change of acceleration), snap (rate of change of jerk) etc.

[0083] Additional information is also provided to supplement and provide context to the traces 212. Such additional information is referred to as “contextual” data 214. The contextual data 214 pertains to the physical context of the scenario, and can have both static components (such as road layout) and dynamic components (such as weather conditions to the extent they vary over the course of the simulation).

[0084] The test oracle 252 receives the traces 212 and the contextual data 214, and scores those outputs in respect of a set of performance evaluation rules 254. The performance evaluation rules 254 are shown to be provided as an input to the test oracle 252.

[0085] The rules 254 are categorical in nature (e.g. pass / fail-type rules). Certain performance evaluation rules are also associated with numerical performance metrics used to “score” trajectories (e.g. indicating a degree of success or failure or some other quantity that helps explain or is otherwise relevant to the categorical results). The evaluation of the rules 254 is time-based - a given rule may have a different outcome at different points in the scenario. The scoring is also time-based: for each performance evaluation metric, the test oracle 252 tracks how the value of that metric (the ‘robustness’ score) changes over time as the simulation progresses. The test oracle 252 provides an output 256 comprising a time sequence 256a of categorical (e.g. pass / fail) results for each rule, and a score-time plot 256b for each performance metric, as described in further detail later. The results and scores 256a, 256b are informative to the expert 122 and can be used to identify and mitigate performance issues within the tested stack 100. The test oracle 252 also provides an overall (aggregate)result for the scenario (e.g. overall pass / fail). The output 256 of the test oracle 252 is stored in a test database 258, in association with information about the scenario to which the output 256 pertains.

[0086] Figure 3 shows a schematic block diagram of a visualization component 320. The visualization component 320 is shown having an input connected to the test database 258 for rendering the outputs 256 of the test oracle 252 on a graphical user interface (GUI) 300. The GUI is rendered on a display system 322.

[0087] Figure 4 shows an example view of the GUI 300. The view pertains to a particular scenario containing multiple agents, and is shown to comprise a scenario visualization 301 and a set of driving performance assessment results 302. In this example, the test oracle output 256 pertains to multiple external agents, and the results are organized according to agent. For each agent, a time-series of results is available for each rule applicable to that agent at some point in the scenario. Some form of visual indicator (such as coding) is used to differentiate between periods of pass / fail on a particular rule.

[0088] Figure 5 schematically depicts an example road network 500 of the kind encoded in the static layer 201a. The following description assumes the road network 500 is described using the OpenDRIVE schema, or a similar format that adopts certain definitions and conventions from OpenDRIVE. However, it will be appreciated that the principles extend more generally to other formats, and the described techniques are not limited to any particular data format or schema.

[0089] The road network 500 is shown to comprise first, second, third and fourth roads 502, 504, 506, 508 (Roads 1 to 4), which are described with <road> elements having road identifiers (IDs) 1, 2, 3 and 4 respectively. The roads 502-508 are interconnected via a junction 510 described by a <junction> element. Each of the roads 502-508 is defined by a single road reference line, denoted as a thick solid arrow, and contains a single center lane of width zero. The center lanes are not depicted separately, and for simplicity it is assumed that the center lane of each road 502-508 lies along the road reference line (although, as noted, it is possible to define a non-zero offset between the road reference line and the center lane). The road reference lines of the first and second roads 502, 504 are denoted by reference numerals 503 and 505 respectively. A road reference line is directional, and could be described more precisely as a longitudinal axis or “s-axis” of the road, with s-coordinates running along that axis, and t-coordinates running orthogonal to it. As depicted for thereference lines 503, 505, the positive t-direction is defined as extending to the left of the s- axis. Lanes are numbered relative to the center lane; the center lane is always numbered 0, with side-lanes to the left of the center lane (+t) assigned incrementing positive lane numbers (1, 2, 3, . . .) and lanes to the right (-t) assigned decreasing negative lane numbers (-1, -2, -3, ...).

[0090] Each road has a minimum of one side-lane. In this example, the first road 502 has only positively numbered side-lanes to the left of the center lane, whereas the second road 504 has two positive side-lanes to the left of the center lane, and one negative side-lane to the right. Roads may be divided into “lane sections” to accommodate sections of the road with different numbers of lanes (see below).

[0091] In the following description, “Lane n” means a lane with lane number “n” and “Road m” means a road with road ID “m”. Other road elements (such as junctions) are referred to in the same way.

[0092] Adjacent lane sections may be referred to as “Lane Section n” and “Lane Section n+1” (the lane section numbers n, n+1 does not form part of the OpenDRIVE schema, but are nevertheless derivable from the static layer 201a).

[0093] A left-hand traffic (LHT) road system is depicted in this example. However, the schema can be used for either LHT or right-hand traffic (RHT) networks. A “rule” attribute of each <road> element indicates whether the road is LHT (vehicles drive on the left) or RHT (vehicles drive on the right).

[0094] A global cartesian coordinate system is defined with the x-direction lying eastwards and the y-axis extending northwards (OpenDRIVE calls this the inertial coordinate system).

[0095] Lane numbers only indicate relative directions of traffic flow within a road: for any given road, traffic flows in the same direction for positively-numbered one-way side-lanes, and in the opposite direction for negatively-numbered one-way side-lanes. Bi-directional lanes support traffic flow in both directions irrespective of the road traffic rule. However, the lane number alone is not sufficient to infer the direction of traffic flow, as the direction of the s-axis can be (more or less) arbitrarily chosen and does not indicate driving direction. For example, in Figure 5, the s-axis 505 of the second road 504 extends towards the junction from the east, whilst the s-axis of the fourth road 508 extends in the opposite direction towards the junction 510 from the west. Along the second road 504, positive lane numbers thereforedenote a direction of traffic flow from east to west, whereas along the fourth road 508, east- to-west traffic flow is denoted by negative lane numbers. For a LHT road, lanes to the left of the road reference line (+t) carry traffic in the direction of the road reference line (+s), whereas lanes to the right of the road reference line (-t) carry traffic in the opposite direction (-s). In an RHT road, lanes to the left of the road center line (+t) carry traffic in the opposite direction of the road reference line (-s) and lanes to the right (-t) carry traffic in the direction of the road reference line (+s).

[0096] It is also possible to define a bidirectional side-lane, permitting traffic flow in both directions, by setting a @type attribute of the <lane> element to “bidirectional”. Bidirectional lanes are addressed in more detail below.

[0097] Therefore, in order to determine the absolute direction of traffic flow, the lane number is not sufficient; the direction of the road reference line (s-axis) must also be considered, as must the @type attribute.

[0098] Lanes are not necessarily drivable. For example, Lane 1 of Lane Section 2 of Road 2 is non-drivable. The outer boundaries of a road may also be defined by non-drivable lanes (such as lanes of a pavement / sidewalk type).

[0099] Link elements are used to explicitly define linkage between roads and lanes. With only two roads, links can be defined with link elements between roads directly, provided the links are unambiguous. More complex linkage requires the use of junction elements.

[0100] A <link> element can have one or both of a <successor> element and a <predecessor> element. The predecessor / successor of a road can be another road or a junction. The predecessor / successor of a lane is another lane. “Predecessor” and “successor” relationships are defined relative to the s-axis of the road in question (a road’s t-axis runs from its predecessor, if any, to its successor, if any), and do not denote driving direction. The s-axis of a road runs away from its predecessor (if any), towards its successor (if any).

[0101] Predecessor / successor relationships may be ‘asymmetrical’. For example, if Road n is a predecessor of Road m, that does not imply Road m is a successor of Road n; if the s-axis of Road n runs in the opposite direction to the s-axis of Road m, then Road m could also be a predecessor of Road n. As another example, if Road m is part of a junction, then Road m cannot be a predecessor or successor of Road n, because the junction would be the predecessor / successor of Road n instead (see below for further examples).

[0102] Roads with varying number of lanes are accommodated using lane sections. Within the OpenDRIVE schema, lanes are described within a <lanes> element of a <road> element. A <lanes> element may be split into multiple sections (lane sections) with a <laneSection> element. Each individual lane is defined by a <lane> element within the <laneSection> element. Each lane section has a fixed number of lanes, numbered as per the above convention. For example, the first road 502 is shown divided into two lane sections (Lane Sections 1 and 2 of Road 1): approaching the junction 510, the number of lanes is shown to increase from two to three. On entering Lane Section 2 from Lane Section 1 of the first road 502, a new rightmost lane is created, whose width gradually increases from zero. In Lane Section 1, the left and right most lanes (to the left of the center lane) are numbered 2 and 1 respectively, whilst in Lane Section 2 the left-most lane and new right-most lanes are numbered 3 and 1 respectively; what is now the middle lane is numbered 2.

[0103] Links between lanes in adjacent lane sections are described using <link> elements. In Lane Section 2, Lane 2 of Lane Section 1 is a predecessor of Lane 3 of Lane Section 2, and Lane 1 of Lane Section 1 is a predecessor of Lane 2 of Lane Section 2. That is to say, the lane element describing Lane 3 of Lane Section 2 would contain a link element indicating Lane 2 as its predecessor, and it is implicit that the link refers to the immediately preceding lane section (Lane Section 1) etc.

[0104] Likewise, in Lane Section 1, Lane 3 of Lane Section 2 is a successor of Lane 2 of Lane Section 1, and Lane 2 of Lane Section 2 is a successor of Lane 1 of Lane Section 1. That is, the lane element describing Lane 2 of Lane Section 1 would indicate Lane 3 as its successor, and it is implicit that the link refers to the next lane section (Lane Section 2) etc.

[0105] Because Lane 1 of Lane Section 2 has zero width initially, it is not linked to any lane of Lane Section 1. It is, however, possible for a lane to have multiple successors or predecessors in different circumstances, for example if a lane splits immediately into two lanes, each of non-zero width at the lane section boundary.

[0106] Within the junction element 510 of Figure 5, additional roads are defined, whose reference lines are depicted as thick, solid arrows. Fifth to ninth roads 512, 514, 516, 518, 520 (Roads 5 to 9) are depicted within the junction 510 in this example. Although side-lanes within the junctions 510 are not depicted in Figure 5, each road within the junction 510 is also required to have at least one side-lane of non-zero width.

[0107] In Figure 5, the junction 510 is a successor of the first, second and fourth roads 502, 504, 508 because their respective s-axes extend towards the junction 510. Therefore, the <road> elements describing those roads 502, 504, 508 would contain <link> elements with <successor> elements indicating the junction 510. The junction 510 is a predecessor of the third road 506 because its s-axis extends away from the junction, and the <road> element describing the third road 506 would therefore contain a <link> element with a <predecessor> element indicating the junction 510.

[0108] Within the junction element 510, connecting roads are described by <road> and <connection> elements.

[0109] The fifth road 512 (Road 5) is shown to connect the first and third roads 502, 506. The fifth road 512 is defined by a <road> element within the <junction> element 510, of which the first road 502 is a predecessor and the third road 506 is a successor (defined via <predecessor> and <successor> elements in the <road> element describing the fifth road 506). Similarly the sixth road 514 has the fourth road 508 as its successor and the second road 504 as its predecessor. The seventh road 516 connects the second and third roads 504, 506, with the second road 504 as its predecessor and the third road 506 as its successor. The eighth road 518 connects the second and first roads 504, 502, with the second road 504 as successor and the first road 502 as predecessor. Finally, the ninth road 520 connects the first and fourth roads 502, 508, with the fourth road 508 as its successor and the first road 502 as its predecessor. Again, the predecessor / successor relationships are not direct indicators of driving direction.

[0110] A <connection> element is used to indicate driving direction for traffic joining a junction. A connection element indicates an incoming road and a connecting road (but does not explicitly define an outgoing road), with traffic entering the junction from the incoming road onto the connecting road. For example, a first <connection> element would be provided that indicates the road with ID=1 (the first road 502) as an incoming road and the road with ID=5 (the fifth road 512) as a connecting road; this connection element indicates that traffic can enter the junction 510 from the first road 502 onto the fifth road 512. A second <connection> element would indicate the first road 502 as an incoming road and the eighth road 518 as a connecting road, etc. The seventh connecting road 516 is a two-way road in this example, carrying traffic from the second road 504 to the third road 506, and traffic in the opposite direction. Therefore, two <connection> elements would be used, one indicatingthe second road as incoming and the seventh road 516 as connecting, and the other indicating the third road 506 as incoming and the seventh road 516 as connecting. The OpenDRIVE specification strongly advises one not to use two-way connecting roads, although two-way connecting roads are possible using the schema. The seventh connecting road 516 goes against this advice, but does not violate it (and, in practice, it is more likely that two one-way connecting roads would be defined). The fourth road 508 is not an incoming road to the junction 510 (even though its s-axis extends towards the junction 510), because it is a oneway road that only carries traffic away from the junction 510.[oni] A connecting road with multiple lanes is used when lane changes are possible within the junction 510; if lane changes are not permitted, multiple single-lane roads would be used instead.

[0112] A “virtual” connection can also be described using a <connection> element without any connecting road. Virtual connections are limited to “virtual” junctions, which do not require a main road to be split up.

[0113] Links between lanes are described via link elements within the <lane> elements that describe those lanes, where predecessor and successor lanes are described in a similar manner to roads. In order for a first lane of a first road to be a predecessor or successor of a second lane of a second road, the second road must be a predecessor or successor of the first road. As the junction 510 is the successor of the first, second and fourth roads 502, 504, 508, these roads 502, 504, 508 have no direct successor roads; therefore, it is not possible for any of their lanes to have lane successors. Likewise, as the junction 510 is the predecessor of the third road 506, the third road 506 has no direct predecessor road; therefore, its lanes cannot have any predecessors. Within the junction 510, however, each of the connecting roads 512- 520 has both a direct successor road and a direct predecessor road; for example, the direct successor road of the fifth road 512 is the first road 502 and its direct predecessor is the third road 506. Consequently, lanes of the connecting roads 512-520 can be linked to lanes of both their predecessor and their successor roads. For example, the fifth road 512 would typically contain two positively-numbered side-lanes, 1 and 2. The side-lanes of Road 5 are not depicted in Figure 5, but are shown in Figure 6. The lane elements describing lanes 2 and 1 of Road 5 would, in turn, contain link elements, which indicate lane numbers “3” and “2” as their respective predecessors (and it is implicit that these are lanes of Road 5’s successor, namely Road 1), and lane numbers “2” and “1” as their respective successors (implicitlyreferring to Road 5’s successor, namely Road 3). A <connection> element also contains any lane.

[0114] Figure 6 shows part of the road network 500 of Figure 5, annotated with sections of OpenSCENARIO code that define certain elements of the road network.

[0115] Starting from any lane of any of the incoming roads 502, 506, 506, the <connection> elements within the junction describe all possible routes though the junction 510 (at the road level). For a given lane on a given road, obtaining a list of all routes through the junction would mean locating all of the <connection> elements that identify the road in question as an incoming road, via its road ID, and which also have a lane linked to the lane in question. To obtain further lane links, it would then be necessary to locate the <road> element of its connecting road, in order to identify the outgoing road (for the fifth road 512, the outgoing road would be the third road 506), and determine the lane linkage between the connecting road and the outgoing road.

[0116] A first code portion 602 contains a connection element (connection ID=0) that indicates the first road 502 (the road with ID 1) as an incoming road and the fifth road 512 (the road with ID 5) as a connecting road. First and second <laneLink> elements link Lane 3 of Road 1 (its incoming road) to Lane 2 of Road 5, and Lane 2 of Road 1 to Lane 1 of Road 5. A second portion of code 604 contains the <road> element describing Road 5 within the junction 510 (junction ID=0). A <link> element within the road element contains both <successor> and <predecessor> elements. The road link in the predecessor element indicated Road 1 as the predecessor to Road 5, which mirrors the lane links in the corresponding <connection> element (Connection 0). The successor element indicates the third road 506 (the road with ID 3) as successor to Road 5. A <lane> element is also shown within the road element for Road 5, which describes Lane 2 of Road 5 (other lane elements are omitted for conciseness); the lane element contains a link element, which in turn indicates Lane 2 as its successor and Lane 3 as its predecessor. To meaningfully interpret the lane links, it is necessary to consider the road link information; Road 1 is the predecessor to Road 5, therefore Lane 3 of Road 1 is the predecessor of Lane 2 of Road 5; Road 3 is the successor to Road 5, therefore Lane 2 of Road 3 is the successor to Lane 2 of Road 5. A third code portion 606 contains the road element describing Road 3. The junction 510 (junction ID=0) is indicated as the predecessor to Road 3, and a lane element is shown that describes Lane 2 of Road 3 (other lane elements are omitted for conciseness).

[0117] Figure 7 shows part of a directed side-lane graph 700 representing the road topology of the road network 500. The directed side-lane graph 700 is referred to simply as the lane graph 700 for conciseness. The lane graph 700 describes lane interconnections from Road 1 though the junction 510, and may be derived from the underlying OpenDrive document taking into account the considerations set out above. For example, first and second nodes 702, 704 correspond, respectively, to Lane 1 of Lane Section 1 of Road 1 and Lane 2 of Lane Section 2 of Road 1. An onward edge 706 indicates the topological relationship between those lanes, namely that the latter is an “onward” lane from the former (i.e. a vehicle can traverse from the former to the latter without performing a lane change maneuver). A directed edge from node 702 to node 704 has an edge type describing the connection between the corresponding lanes (“onward” in this case).

[0118] The lane graph 700 may be constructed by interpreting the code of the static layer 201a. Note that edges of the lane graph denote driving direction. To determine onward edges, lane links need to be considered, but driving direction also needs to be considered.

[0119] Edges are also provided for left-right relationships. For example, third and fourth nodes 708, 710 are depicted, representing Lanes 3 and 1, respectively, of Lane Section 2 of Road 1. A right edge 711 from the second node 704 to the fourth node 710 represents the possibility of moving from Lane 2 to Lane 1 of that lane section via a right lane change maneuver. A left edge 709 from the second node 704 to the third node 708 represents the possibility of moving from Lane 2 to Lane 3 via a left lane change maneuver. The left and right edges 709, 711 have respective types (“left” and “right”), again describing the nature of the lane connections. This information is not provided in <link> elements of the underlying description, but can be derived from the structure of the road.

[0120] A fifth node 714 represents Lane 1 in the only road section of Road 5, which is an onward lane from Lane 2 in Lane Section 2 or Road 1. Therefore, an onward edge 712 is directed from the second node 704 representing the former to the fifth node 714 representing the latter.

[0121] A dynamic layer 201b may be defined and executed with respect to the lane graph 700. For example, particular agents may be associated with particular nodes at certain times (such as the start of the scenario, or defined ‘keyframes’ or synchronization points). In simulation, this topological description of the scenario dynamics is converted to a concrete geometric realization. Therefore, changes can be made to the road network 500 and,provided those changes do not alter the relevant road topology of the lane graph 700 (typically some sub-graph or sub-graphs of the lane graph 700), it is straightforward to run the same dynamic layer 201b on the modified road network without modification to the dynamic layer.

[0122] The dynamic layer 201b can also contain geometric information, such as relative locations of agents at particular points in time (e.g. at the start of the scenario, or at some ‘synchronization’ point during the scenario).Constraints-based scenario design

[0123] FIG. 8 shows a highly schematic block diagram of a scenario design system 800, together with a sequence of illustrative processing steps performed in the scenario design system 800.

[0124] The scenario design system allows multiple test scenarios to be generated with ease from a bespoke set of geometric elements 802 and constraints 804 imposed on the geometric elements 802, referred to as a scenario “skeleton” (or constraint scheme). The constraints 804 can include individual geometric constraints on single geometric elements and interelement (e.g. pairwise) constraints between subsets (e.g. pairs) of geometric elements. The geometric elements 802 are controlled by geometric variables X(configurable in the sense that different values may be assigned to them), also referred to as skeleton variables or geometries.

[0125] The geometric elements 802 are selectable from a predefined set of geometric primitives 808, such as lines, points, arcs, spirals, polynomials etc., which may be configured by geometric variables such as location, angle, radius of curvature, polynomial coefficients etc. The geometric elements 802 selected for the skeleton 801 are used to represent scenario elements (static layer elements, such as reference lines, lane borders, road markings, buildings etc., and / or dynamic agents, such as vehicles, pedestrians, cyclists, animals etc.).

[0126] The skeleton variables Y are defined, at least in part, by the selected geometric elements 802. Their values, denoted by reference sign 812, may be modified according to the processing flow described below. The skeleton variables 802 are independent at this stage in so far as any values can be assigned to them initially. However, only certain combinations of values will satisfy the constraints 804 (meaning they are treated as dependent variables in the solution stage - see below). This is accommodated using a geometric constraint solver 810.

[0127] The skeleton 801 is created with an initial set of values assigned (Step 1) to the variables Y that are not required to satisfy the constraints 804.

[0128] As indicated above, at Step 1, the scenario designer additionally chooses a set of dimensions (denoted X), which are quantities to be treated by the constraint solver 810 as independent variables. The geometric variables Y are treated as dependent variables by the constraint solver 810. In choosing the dimensions X, the designer is giving control over those properties of the scenario to an end-user.

[0129] Note that dimensions X are not necessarily associated directly with variables that support the unknown geometry descriptions, e.g. point.x and point.y (denoting the x and y coordinates of ‘point’ respectively). Rather, a dimension may be associated with constraint s) that require a value(s), such as a Distance, an Angle, a Length (e.g., measured along a curve), an Offset (e.g., measured from a curve), etc. In this case, dimensions correspond to parameters that would commonly appear on the right-hand-side of a set of mathematical equations that represent a constraint scheme, e.g. (p.x -q.x )A2 + (p.y - q.y)A2 + (p.z - q.z)A2 = dA2. This equation expresses a constraint on the distance between p and q (defined in the left hand side) characterised by a distance dimension d (note, d is not assigned a specific value at this point; rather, this is the mechanism to introduce d as an independent variable). Here, p and q are (dependent) geometric variables and d is an (independent) constraint parameter. It is also possible to constrain a variable directly, e.g. by specifying p.x = a (fixing the x coordinate of p to the value a) or a < p.x < b (fixing the x coordinate to a range of values). The constraint solver 810 would then attempt to solve for the dependent variables (e.g., p, q) given the independent variables (e.g., d, a, b).

[0130] Any dimensions X of the constraint scheme are independent variables and are not constrained by the scheme itself.

[0131] The skeleton 801 is then provided (Step 2a) as a first input to the geometric constraints solver 810, along with an indication of the choice of dimensions X and the initial values of the dimensions X and geometries Y.

[0132] As discussed, the constraint solver 810 is primarily analytical. Using a set of predetermined deduction rules, it attempts to determine (Step 2b) an analytical (and therefore ‘reusable’) solution in the form of a set of function(s) F =that express the (unknown) geometries Y = { q , y2, ■ ■ ■ } in terms of the (known) dimensions X = {xx, x2, ■ ■ ■ }such that the constraints are satisfied. The solution is denoted by Y=F(X) in FIG. 8, and the solver aims to compute F (the mapping between the dimensions and the unknowns) as a set of mathematical expression(s). Those expressions may be embodied in memory as a computational graph 830 (see below).

[0133] In the present example, the designer specifies initial values of both the dimensions X and the unknowns Y at Step 1, but those initial values do not necessarily satisfy the constraints 804.

[0134] Once the solution has been found at Step 2b, it can be used at Step 3, together with the initial values of X, to assign new values to Y that do satisfy the constraints 804. A new value is assigned to each unknown ytat Step 3 by evaluating ft(X) for the initial values of X.

[0135] Note, the solution Y=F(X) that is determined as Step 2b is not necessarily unique; rather, there may be multiple solutions to the constraint problem (that is, multiple mappings F between the dimensions X and the unknowns Y that satisfy the constraints 804).

[0136] If the constraint problem has two (or more) ‘point-like’ solutions then each of those solutions is well-defined, and the problem is said to have two (or more) well-defined solution branches.

[0137] A constraint problem is said to be ‘under-defined’ if there is some (non-zero) number of unconstrained degrees of freedom left over after all of the constraints and dimensions have been satisfied, meaning the solution is space-like, i.e. there is a continuum of infinitely many solutions on a single solution branch (an under-defined problem may have one or multiple solution branches). In some cases, an under-defined problem may trigger an error output. In other cases, the tool may handle an under-defined problem by making a choice of solution automatically (and in that case, it may or may not flag that there are other possible solutions).

[0138] FIG. 10 depicts a simple example of a well-defined constraint problem with two solution branches, in which a point has location yt(an unknown in Y) constrained by distances , d2(constants) relative to two other points at locations x}and x2(dimensions in X). The solution to this constraints problem generally has exactly two solutions -= / i+(xi<%2) and yi = fi~ (xi<%2) respectively - defined by the intersection of two circles centred on x and x2with radiusand d2. As will be appreciated, this is a simpleexample chosen for illustration (e.g., in a more complex example, the distances could themselves be variables, or the locations of the other points could also be unknowns).

[0139] In some cases, it may be appropriate for the solver 810 to infer an intent of the designer, e.g. from an initial choice of dimension (X) values and / or some initial choice of Y- values. Following the example of FIG. 10, suppose the designer has chosen initial values for the unknown ytand the dimensions x}and x2. Given those values, the first solution1+returns a value that is closer to the designer’s initial value of ytthan the second solution f2+. Therefore, the first solution1+may be chosen automatically by the solver 810 at Step 2b. In some implementations, this is flagged to the designer, e.g. via a notification at the GUI 816.

[0140] Note, the solution is not tied to the initial choice of X-values and will be valid for other values of X. Moreover, the initial F-values are not necessarily required to satisfy the constraints. The initial choice of X or Y values simply guides how the solver resolves an under-defined problem or a well-defined problem with multiple solution branches (e.g. favouring a solution that is valid for the initial choice of dimension values X, or a solution that is closer to the initial choice of geometry values F).

[0141] As another example, there may be multiple solutions to a given problem, but only one (or some) of these solutions might be valid for the designer’s initial choice of X-values. In that case, the solver may choose the solution that is valid for the designer’s initial choice of X-values.

[0142] At the other extreme, a geometry ytmay be completely unconstrained, which would generally indicate a mistake at the design stage, either in the choice of constraints C or the choice of dimensions X).

[0143] In more detail, the constraints solver 810 analyses the scheme during a resolution phase to produce the mappings F in the form of a computational directed acyclic graph (DAG) 830 that is viable for any set of dimension values in non-empty ranges that contain the initial dimension (X) values. The resolution phase is followed by a computational phase, in which the computational DAG 830 is evaluated for a given choice of X-values. If the dimension (X) values are subsequently modified and the solution remains viable for the modified X-values, only the computational part of the solution procedure needs to be performed after; the computational DAG 830 itself does not change.

[0144] Occasionally, the solver 810 may be unable to derive an analytical solution. In that case, it may resort to a numerical method to try to find values of Y that satisfy the constraints for a given choice of X-values. This is generally much less efficient as, unlike an analytical solution, the numerical solver would have to be re-run every time the value of a dimension(s) changes.

[0145] The constraints 804 provide a definition of ‘saliency’ that is hard-coded into scenarios at the design stage. All scenarios generated based on the scenario skeleton 801 are guaranteed to be salient, in the sense of satisfying the constraints 804 imposed at the design stage.

[0146] Note that the constraints 804 are expressed in an intuitive, user-friendly way, in terms of primitive geometric relationships (such as ‘parallel’, 'coincident’, ‘distance’ etc.) between elements. The scenario designer is not required to manually derive or enter mathematical expressions that define the unknowns in terms of dimensions. Rather, the constraint solver 810 derives those expressions automatically by applying a series of deductions.

[0147] The scenario skeleton 801 is a ‘lightweight’ description of a scenario (or part of a scenario) designed around some aspect (or aspects) of the scenario to be varied in simulation testing. The aim is to define a minimum number of geometric elements 802 and constraints 804 that capture this variable aspect(s), whilst sensibly limiting the degree to which it can be modified to ensure that all variations of the scenario satisfy the designer’s definition of saliency. A combination of variable values 812 satisfying the constraints 804 of the skeleton 801 may be referred to as ‘salient’.

[0148] Having assigned, at Step 3, new values to the unknowns Y that satisfy the constraints 804, the skeleton 801 can then be used (Step 4) together with the initial values of X and the computed values of Y to generate a full scenario description 201 in an appropriate SDL (such as OpenDRIVE or OpenSCENARIO, or some variation thereof). To this end, the skeleton 801 and the updated values 812 are provided as inputs to the scenario generator 146.

[0149] The primitives used in the constraint solver 810 may or may not correspond to primitives used in the underlying SDL (e.g. OpenDRIVE). In general, constraint solvers make a wide range of geometrical primitives available, e.g., from simple points, lines, and circles, up to NURBS (non-uniform rational basis splines) and / or custom parametric curves.Providing the primitives used in the constraint solver primitives back onto the set used by the desired SDL (within an acceptable tolerance), the generation step 4 is feasible.

[0150] In this example, the full scenario description has a static layer 201a encoded in a specification (document) that conforms to the OpenDRIVE schema, or some variant of OpenDRIVE (or other structured scenario description format), and a dynamic layer 201b is encoded using OpenSCENARIO, or some variant thereof. As noted, different formats / schemas may be used as an alternative to OpenDRIVE / OpenSCENARIO.

[0151] The static layer 201a is a full detailed description of a road network, e.g. at the level of detail of FIGS. 5 or 6, or greater.

[0152] To generate further variations, one or more values of the dimension(s) X may be subsequently modified at Step 5. In Step 5, the X-values may be arbitrarily modified (as independent degrees of freedom). Assuming the previously determined computational graph remains viable, then updated Y values are determined at Step 6 using the existing computation DAG (it is simply a case of re-evaluating the existing DAG on the new X- values).

[0153] The result of Steps 5 and 6 is a new set of X and Y values satisfying the constraints 804 of the skeleton 801.

[0154] The new variables obtained in Steps 5 and 6 can, in turn, be used (Step 7) by the scenario generator 146 to generate a second scenario from the skeleton (in SDL), which is also guaranteed to be salient in the above sense.

[0155] Steps 5-7 can be repeated as many times as desired, to generate any desired number of scenarios from the skeleton 801 all satisfying the constraints 804. Continuing the road curvature angle, continuous (or incremental) changes could be made to the road curvature angle, and re-computing values for the remaining variable(s) each time the road curvature angle is modified using the existing computation DAG.

[0156] It is also possible that the dimension (X) values may be altered in a way that means the earlier solution (computational graph) is no longer viable. However, it is possible that a different solution exists, which is viable for the new dimension values. In that event, the constraints solver 810 may attempt to find a new solution (in a second resolution phase), inthe form of a second computational DAG (not shown in FIG. 8) that is viable for the new X- values.

[0157] Geometric constraint solvers have been developed in other fields of technology, such as early stage ‘sketching’ tools in computer-aided design (CAD). Existing solvers may be utilized in the present context. One example of a suitable geometric constraints solver is the D-Cubed 2D Dimensional Constraint Manager (2D DCM) licensed by Siemens (R). This is merely one example, and there are various existing solver tools that may be deployed in the present context. Such tools can take a problem definition (dimensions and constraints) and an initialization, and return a solution in the form of a reusable computational graph.

[0158] One class of solver takes a first input that encodes the geometric elements 802 and the geometric constraints 804 in the form of a constraint graph. Certain nodes of the graph represent geometric elements, other nodes represent constraints, and edges between the nodes define how the constraints are applied to the geometric elements (see FIG. 9 and the accompanying description for further details). Such solvers are typically based on a (potentially large) set of deductions rules that allow individual subgraphs within the constraints graph 804 to be solved. For example, a particular deduction rule may be provided for a situation in which a point is defined in terms of distance from two other points (cf. FIG. 10). The computational graph is built up as analytically solvable sub-graphs are identified within the constraints graph, until the whole constraints graph has been solved for the chosen dimensions or the solver determines the problem is underdefined.

[0159] To facilitate Steps 1-7, a skeleton generator 814 is provided, in the form of an application programming interface (skeleton API). The skeleton API 814 is shown to comprise a skeleton builder 814a, a skeleton binder 814b and a skeleton modifier 814c. Each of these components 814a, 814b, 814c is implemented as a set of one or more API functions that may be called by another system component built on top of the skeleton API 814, such as a graphical user interface (design GUI) 816.

[0160] The design GUI 816 provides a set of graphical tools that can be used to select the geometric elements 802 from the geometric primitives 808, and define the constraints 804 on the selected elements 802.

[0161] The skeleton binder 814b is responsible for storing associations (bindings) between the geometric elements 802 of the skeleton 810 and corresponding elements of the scenario description 201 (in the static and / or dynamic layer 201a, 201b).

[0162] In one use case, the skeleton 801 is generated initially based on a selected region within an existing scenario description. Existing scenario descriptions are held in a scenario database 818. As each geometric element 802 is added to the skeleton 801, an association is recorded between that element and a corresponding element (or elements) of the existing scenario description. The initial values of the geometric variables 804 can also be derived from existing scenario description as the elements 802 are added and the bindings are created. Then, when the values are subsequently modified in Steps 2 / 6, the corresponding elements of the existing scenario description can be modified accordingly, using the binding between the skeleton 801 and the scenario description 201, in a way that is always guaranteed to satisfy the constraints 801. Here, the aim is to build a skeleton 801 with the fewest geometric elements needed to define the aspect (or aspects) to be varied, and the way in which this control is constrained to preserve saliency.

[0163] Alternatively, a new scenario description may be generated ‘from scratch’ based on the skeleton 801 and the initial variable values, in which case the binding may be created from scratch at the point the scenario description 201 is generated. The binding can thereafter be used to effect changes in the variable values more easily.

[0164] A skeleton 801 could also be created initially, and the geometric element 802 could then be bound to an existing scenario description once the skeleton 801 has been created.

[0165] The system is highly flexible, allowing a user to add any and constrain elements of their choosing to the skeleton 801, depending on their specific testing needs.

[0166] One application of the scenario design system 800 is the generation of the static layer 201a, where the geometric elements 802 correspond to road elements. In this case, the tool can be used to generate or modify a map contained in the static layer 201a subject to the designer’s constraints.

[0167] The full scenario description 201 is generally much more detailed than the skeleton 801. For example, to exert and constrain a degree of control over, say, road angle, it may only be necessary to include the applicable road reference lines in the skeleton (to fully define and constrain the road curvature with respect to the road reference lines). Within thefull scenario description 801, there may be many additional elements, such as drivable / non- drivable side lines, that are defined relative to the road reference lines and, therefore, do not need to be included in the skeleton 801 to exhibit the desired control. Such elements can be omitted from the skeleton 801 because changes to the road reference lines will automatically propagate into all other elements of the static layer 201a that are defined relative to the road reference lines.

[0168] FIG. 9 illustrates how a curved road reference line might be defined in terms of first and second lines LI, L2, an arc Al, and six points Pl,. . P6 for defining segments of the lines and the arc. A single dimension, a, is considered in this example. This would involve geometric elements, constraints and variables such as: a. Constraining Pl and P2 to be coincident with the first line LI (a first line segment defined by Pl, P2 and L2 can, in turn, be bound to a corresponding first straight line segment of the road reference line in the full static layer 201a); b. Constraining P3 and P4 to be coincident with the second line L2 (similarly defining a second line segment, possibly at a different angle); c. Defining a dimension a in X as the angle between LI and L2 (the road curvature angle, which is the property the user wishes to manipulate in this example); d. Constraining P5 and P6 to be coincident with the arc Al, defining a curve segment (which, in turn, can be bound to a corresponding curved segment of the road reference line in the in the full static layer 201a); e. Constraining P2 to be coincident with P5, and P3 to be coincident with P6 (guaranteeing a continuous road reference line extending between Pl and P4); f. Constraining the gradient of Al at P5 to match the gradient of LI, and the gradient of Al at P6 to match the gradient of LI (guaranteeing a road reference line of continuous gradient between Pl and P4) g. Placing respective distance constraints between Pl and P2 and between P3 and P4, to limit the size of the map fragment.

[0169] Additional / al ternative constraints could be imposed to fit the road fragment described by the skeleton 201 to an existing map (e.g. fixing the locations of Pl and P4 so they correctly connect to a wider road network).

[0170] Part of a ‘constraints graph’ 900 is shown, which is a graph-based representation of the skeleton 801. The constraints graph 900 encapsulates the geometric elements 802, the constraints 804, and their variables in a set of nodes and edges. As can be seen, the graph 900 includes first nodes corresponding to the geometric elements Pl,. . ,,P6, LI, L2, A2, and second nodes defining their variables and constraints. Coincidence constraints are represented by coincidence nodes (coi), whereby a coincidence relationship between two geometric elements is encoded as a coincidence node connected to the corresponding element nodes in the graph. Distance and angular variables, and any constraints attached to those variables, can be similarly represented with additional nodes and connections in the constraints graph 900. For example, a gradient node (Gr) is shown connecting the nodes representing LI, P5 and Al to define the constraint that the gradient of Al at P5 must equal the gradient of LI. A distance node (Dist) is shown connecting the Pl and P2 nodes, defining a distance variable between those points Pl, P2, which can be constrained as desired.

[0171] The constraints graph 900 is provided to the solver 810, together with an indication that a is to be treated as an independent dimension, and an initial set of values 902 of the variables used to guide the solver 810. The solver 810 constructs a computational graph 930 for computing the dependent variables in a way that satisfies the constraints graph 900, which in turn can be provided to the scenario generator 146 to generate a full map description satisfying the constraints. This is simply a case of evaluating the computational graph 930 on a chosen value of the road curvature angle a to compute the dependent variables, and using the established bindings.

[0172] The curvature angle a is defined by an angular node (Ang) of the constraints graph 900 connecting the nodes representing LI and L2. Note, it is the scenario designer’s choice to define the variable a as an independent dimension. The design tool is flexible enough to accommodate whatever dimension definitions the designer wants.

[0173] The same constraints 900 could be used but with a different choice of dimensions. For example, a might become a dependent variable, and some other variable (such as the curvature of Al) could be chosen as an independent dimension. This would then be a different constraints problem, solved for the new dependent variables in the same manner.

[0174] Whilst FIG. 9 considers only a static layer skeleton, the same techniques can also be used to configure the dynamic layer 201b. In this case, agents at a given point in time may be treated as static elements, whose geometry at that point in time can be constrained in exactly the same way. Different variations of the dynamic layer 201 can then be realised by repeatedly solving for the constraints 804 in the matter described above. One example would be a set of parked vehicles (corresponding to certain geometric elements of the skeleton 801), whose starting locations are constrained in the skeleton 801.

[0175] For static agents, the locations would be fixed at all times throughout the scenario. However, the same techniques could be used to determine the starting locations of dynamic agents at the start of the scenario, or at specified ‘keyframes’ within the scenario. The latter implies that agent locations are determined at some point during the scenario. With openloop simulation (non-ego agents do not react to the ego agent), this is straightforward to implement, by interpolating the relevant agent variables (location, heading, speed, acceleration etc.) back to the start of the scenario (or to an earlier key frame).

[0176] Constrained keyframes can be constructed from a skeleton, subject to a given set of constraints on properties of the agents (e.g. position, heading, speed, acceleration etc.) provided to the constraints solver 810. This goes some way to addressing the problem of dynamic saliency; for example, a constrained key frame can be constructed to ensure that two non-ego agents interact in some specified manner in all variations of a scenario.

[0177] With closed-loop simulation (non-ego agents can react to ego agents), the agent locations determined by the geometric constraints solver 810 at a given keyframe could be used as goals locations / states towards which the non-ego agents plan (in this case, the strong guarantee of saliency is not provided, because non-ego agents may fail in that goal, and that could depend on the behaviour of the ego agent itself. However, it does provide a weaker guarantee that non-ego agents plan their actions in a salient manner).

[0178] The dynamic layer 201b is preferably defined at the level of road topology, which typically means associating agents with particular lanes or other road elements, whose interconnections are then encoded in the static layer 201a. For example, in one implementation, the dynamic layer 201b is defined on top of a lane graph of the static layer 201a, such as the lane graph 700 depicted in Figure 7. In this case, agents and behaviours may be associated with respective nodes of the lane graph 700, to define scenario dynamics at the level of road topology. To the extent modifications of the scenario do not alter the lanegraph 700 (or the subset of the lane graph 700 on which the agent dynamics are defined), the dynamic layer 201b can be implemented straightforwardly on different variations of the static layer 201a. An ‘agent topology’ refers to a definition of agent dynamics at the level of road topology.

[0179] In this case, the dynamic layer 201b may encompass a range of geometries. For example, the starting positions of a set of agents may be defined in terms of their starting lanes. Variations of the dynamic layer 210b can then be implemented with different agent starting locations within their respective lanes, or with different shapes / sizes of agents (or, more generally, with different agent geometries within the same agent topology).

[0180] Different scenarios can be created by pairing different dynamic layers with the same static layer 201a. This essentially means placing different agents and behaviours on top of the same road layout.

[0181] Moreover, two dynamic layers defined on the same road topology (or subset of road topology) are readily interchangeable with respect to a static layer having that topology. In the present context, this means that the static layer 201a may be modified to adjust quantities such as road curvature, lane width, junction intersection angles etc. and, to the extent such modifications do not alter the road topology (or the subset of the road topology on which the dynamic layer 201b is defined), the same dynamic layer 201b can be executed on the modified static layers.

[0182] References herein to components, functions, modules and the like, denote functional components of a computer system which may be implemented at the hardware level in various ways. A computer system comprises execution hardware which may be configured to execute the method / algorithmic steps disclosed herein and / or to implement a model trained using the present techniques. The term execution hardware encompasses any form / combination of hardware configured to execute the relevant method / algorithmic steps. The execution hardware may take the form of one or more processors, which may be programmable or non-programmable, or a combination of programmable and nonprogrammable hardware may be used. Examples of suitable programmable processors include general purpose processors based on an instruction set architecture, such as CPUs, GPUs / accelerator processors etc. Such general-purpose processors typically execute computer readable instructions held in memory coupled to or internal to the processor and carry out the relevant steps in accordance with those instructions. Other forms ofprogrammable processors include field programmable gate arrays (FPGAs) having a circuit configuration programmable through circuit description code. Examples of nonprogrammable processors include application specific integrated circuits (ASICs). Code, instructions etc. may be stored as appropriate on transitory or non-transitory media (examples of the latter including solid state, magnetic and optical storage device(s) and the like). The subsystems 102-108 of the runtime stack Figure 1 A may be implemented in programmable or dedicated processor(s), or a combination of both, on-board a vehicle or in an off-board computer system in the context of testing and the like. The various components of Figure 2, 4 and 5 such as the simulator 202 and the test oracle 252 may be similarly implemented in programmable and / or dedicated hardware.

Claims

Claims1. A computer-implemented method of generating a static layer and / or a dynamic layer for a test scenario for performance testing a robotic system in a simulation environment, the method comprising: inputting to a geometric constraint solver a scenario skeleton defining geometric elements and geometric constraints on the geometric elements, the geometric elements being modifiable based on a set of geometric variables, and the geometric constraints being characterised by one or more constraint parameters, thereby causing the geometric constraint solver to determine one or more functions of the constraint parameters for returning values of the geometric variables that satisfy the geometric constraints; receiving value(s) of the constraint parameter(s); using the received value(s) of the constraint parameter(s) and the one or more functions to compute values of the geometric variables satisfying the geometric constraints; and configuring based on the received value(s) of the constraint parameter(s), the computed value(s) of the geometric variables and the scenario skeleton, the static layer for the test scenario and / or the dynamic layer for the test scenario.

2. The method of claim 1, comprising receiving initial value(s) of the geometric variables, wherein the received value(s) of the constraint parameter(s) and the received initial values of the geometric variables do not satisfy the geometric constraints, wherein the received initial value(s) of the geometric variables are provided to the geometric constraint solver for use in determining the functions of the constraint parameter(s).

3. The method of claim 1 or 2, wherein the geometric constraints are expressed in the form of a constraints graph, wherein the geometric solver uses a plurality of predetermined deduction rules to extract the function(s) from the constraints graph.

4. The method of claim 1, 2 or 3, wherein the geometric elements comprise road elements and the geometric constraints comprise constraints on the road elements, wherein the received and computed values are used to generate the static layer, which comprises a road layout satisfying the geometric constraints.

5. The method of any preceding claim, wherein the scenario skeleton is generated based on scenario creation inputs received at a graphical user interface.

6. The method of any preceding claim, wherein the scenario skeleton is generated based on a selected region of a map, wherein at least some of the geometric elements correspond to respective map elements within the selected region of the map.

7. The method of claim 6, comprising storing associations between at least some geometric elements and the respective map elements, wherein the static layer is generated from the map based on the values of the geometric variables and the stored associations.

8. The method of any preceding claim, wherein the geometric constraints comprise interelement geometric constraints between respective subsets of the geometric elements.

9. The method of any preceding claim, comprising: executing a test scenario comprising at least one of the static layer and the dynamic layer, with an ego agent of the test scenario controlled by a robotic planner under testing; and generating by a test oracle a set of test results for evaluating performance of the robotic planner in the test scenario.

10. The method of claim 9, wherein the robotic planner is tested in combination with one or more other components.

11. The method of claim 10, wherein the one or more other components comprise one or more of: a perception system, a prediction system and a controller.

12. The method of any of claims 9 to 11, comprising using the set of test results to identify and mitigate an issue in the robotic planner or another component tested in combination with the robotic planner.

13. The method of any preceding claim, comprising: receiving second value(s) of the constraint parameter independent variable(s);using the received second value(s) of the constraint parameter(s) and the one or more functions to compute second values of the geometric variables satisfying the geometric constraints; and configuring based on the received second value(s) of the constraint parameter(s), the computed second value(s) of the geometric variables and the scenario skeleton, a second static layer for the test scenario and / or a second dynamic layer for the test scenario.

14. A computer system comprising: one or more computers configured to implement the method of any preceding claim.

15. Computer-readable instructions stored in transitory or non-transitory media and configured, when executed in a computer system, to implement the method of any of claims 1 to 13.

Images