Support tools for mobile robots

Abstract

A computer-implemented method of testing an autonomous vehicle stack, comprising: storing a scenario datastore comprising a plurality of scenarios and first metadata describing each scenario, the first metadata having been generated by a first autonomous vehicle planner; retrieving a set of scenarios from the scenario datastore; for each retrieved scenario: executing a test run of the retrieved scenario using a second autonomous vehicle planner different from the first autonomous vehicle planner; generating second metadata describing the retrieved scenario based on the test run; and storing the second metadata in the scenario database.

Description

Support Tools for Mobile RobotsTechnical Field[1] The present disclosure pertains to support tools for autonomous vehicles and other mobile robots.Background[2] There have been major and rapid developments in the field of autonomous vehicles. An autonomous vehicle (AV) is a vehicle which is equipped with sensors and control systems which enable it to operate without a human controlling its behaviour. An autonomous vehicle is equipped with sensors which enable it to perceive its physical environment, such sensors include for example cameras, radar and lidar. Autonomous vehicles are equipped with suitably programmed computers which are capable of processing data received from the sensors and making safe and predictable decisions based on the context which has been perceived by the sensors.[3] An autonomous vehicle may be fully autonomous (in that it is designed to operate with no human supervision or intervention, at least in certain circumstances) or semi- autonomous. Semi-autonomous systems require varying levels of human oversight and intervention. An Advanced Driver Assist System (ADAS) and certain levels of Autonomous Driving System (ADS) may be classed as semi-autonomous. A “level 5” vehicle is one that can operate entirely autonomously in any circumstances, because it is always guaranteed to meet some minimum level of safety. Such a vehicle would not require manual controls (steering wheel, pedals etc.) at all. By contrast, level 3 and level 4 vehicles can operate fully autonomously but only within certain defined circumstances (e.g. within geofenced areas). A level 3 vehicle must be equipped to autonomously handle any situation that requires an immediate response (such as emergency braking); however, a change in circumstances may trigger a “transition demand”, requiring a driver to take control of the vehicle within some limited timeframe. A level 4 vehicle has similar limitations; however, in the event the driver does not respond within the required timeframe, a level 4 vehicle must also be capable of autonomously implementing a “minimum risk maneuver” (MRM), i.e. some appropriate action(s) to bring the vehicle to safe conditions (e.g. slowing down and parking the vehicle). A level 2 vehicle requires the driver to be ready to intervene at any time, and it is the responsibility of the driver to intervene if the autonomous systems fail to respond properly atany time. With level 2 automation, it is the responsibility of the driver to determine when their intervention is required; for level 3 and level 4, this responsibility shifts to the vehicle’s autonomous systems and it is the vehicle that must alert the driver when intervention is required.[4] Physical world testing will remain an important factor in the testing of autonomous vehicles’ capability to make safe and predictable decisions. However, physical world testing is expensive and time-consuming. Moreover, as the safety of AV systems improves, the number of incidents per mile driven decreases. To achieve widespread acceptance of AVs, it is estimated that an AV should cause no more than 1 severe accident per 10A9 miles driven (compared with 1 such accident per 10A6 miles for the average human driver). Identifying and mitigating issues based solely on ‘miles driven’ in the real-world is simply not viable for an AV operating even several orders of magnitude below the minimum performance level that will ultimately be required.[5] As such, increasing reliance is placed on testing using simulated environments. If there is to be an increase in testing in simulated environments, it is desirable that such environments can reflect as far as possible real-world scenarios. Autonomous vehicles need to have the facility to operate in the same wide variety of circumstances that a human driver can operate in. Such circumstances can incorporate a high level of unpredictability. Increasing attention is being placed on the creation of simulation environments which can provide such testing in a manner that gives confidence that the test outcomes represent potential real behaviour of an autonomous vehicle.[6] Scenarios must be described with sufficient detail and precision in a form conducive to high-quality simulation, otherwise the results of testing will not sufficiently reflect real- world AV performance. A typical driving scenario includes a static road layout and various dynamic agents (other vehicles, pedestrians, cyclists, animals etc.) that an autonomous vehicle (the ego vehicle) is required to navigate. An ego vehicle may be required to predict the motion of other agents and plan safely within a complex road network. In an offline context, a scenario description may be required as an input to a simulator to facilitate simulation-based testing of an autonomous vehicle stack prior to deployment on a real-world vehicle.[7] By way of example, ASAM OpenSCENARIO (R) defines a file format for the description of dynamic driving scenario content. OpenSCENARIO may be used togetherwith the ASAM OpenDRIVE (R) schema for describing static road networks. The stated purpose of OpenDRIVE “is to provide a road network description that can be fed into simulations to develop and validate ADAS and AD [Autonomous Driving] features”; see ASAM OpenDRIVE VI.7.0 (and accompanying User Guide), Release Date 3 August 2021 [available at https: / / www.asam.net / standards / detail / opendrive / ],[8] Other mobile robots are being developed, for example for carrying freight supplies in internal and external industrial zones. Such mobile robots would have no people on board and belong to a class of mobile robot termed UAV (unmanned autonomous vehicle). Autonomous air mobile robots (drones) are also being developed.Summary[9] A first aspect of the present disclosure provides a computer-implemented method of generating a test scenario for performance testing a robotic system in a simulation environment, the method comprising:

[0010] receiving a dynamic element graph comprising a plurality of dynamic nodes representative of dynamic scenario agents, the dynamic element graph including one or more attachment edges connected at one end to one of the plurality of dynamic nodes and unconnected at the other end;

[0011] searching a static element graph datastore comprising a plurality of static element graphs, each static element graph comprising a plurality of static nodes representative of static scenario elements, to identify locations at which the attachment edges can be connected to the static element graphs; and

[0012] storing data representative of the identified locations.

[0013] The method may comprise generating the dynamic element graph by parsing a dynamic layer of a logical scenario description. The dynamic layer may be in an OpenSCENARIO format.

[0014] The method may comprise generating the static element graphs by parsing a static layer of a respective logical scenario description. The static layer may be in an OpenDRIVE format.

[0015] The dynamic layer and the static layer may be of the same logical scenario. Accordingly, one of the static element graphs in the static element graph data store may be derived from the same logical scenario as the dynamic element graph. The method may thus comprise relocating the dynamic layer on the static scenario.

[0016] The method may comprise generating a logical scenario description. The logical scenario description may include a static layer corresponding to the static element graph and a dynamic layer based on the dynamic element graph and one of the identified locations.

[0017] The method may comprise generating a plurality of concrete scenarios. Each concrete scenario may be generated by instantiating the logical scenario description with a plurality of parameter values. The method may further comprise storing the plurality of concrete scenarios in a concrete scenario datastore. The plurality of parameter values used to instantiate the logical scenario description may be determined by sampling a parameter space of the logical scenario.

[0018] The dynamic element graph may include dynamic edges connecting pairs of the dynamic nodes, each dynamic edge representative of a relationship between the pair of the dynamic nodes at a synchronisation point. The synchronisation point may be the beginning of the scenario.

[0019] The static element graph may include static edges connecting pairs of the static nodes, each static edge representative of a spatial relationship between a pair of static scenario elements represented by the pair of static nodes.

[0020] The method may comprise carrying out a brute-force search of available locations in the static element graph to identify the locations at which the attachment edges can be connected to the static element graph. The method may comprise extracting a subgraph of the dynamic element graph comprising the attachment edges, wherein

[0021] A second aspect of the present disclosure provides a computer-implemented method of testing an autonomous vehicle stack, comprising:

[0022] storing a scenario datastore comprising a plurality of scenarios and first metadata describing each scenario, the first metadata having been generated by a first autonomous vehicle planner;

[0023] retrieving a set of scenarios from the scenario datastore;

[0024] for each retrieved scenario: executing a test run of the retrieved scenario using a second autonomous vehicle planner different from the first autonomous vehicle planner; generating second metadata describing the retrieved scenario based on the test run; and storing the second metadata in the scenario database.

[0025] The first autonomous vehicle planner and second autonomous vehicle planner may be different versions of an autonomous vehicle planner under development.

[0026] The first autonomous vehicle planner may be a non-real time planner. The first autonomous vehicle planner may form part of a scenario execution environment. That is to say, the first autonomous vehicle planner may not form part of an autonomous vehicle stack.

[0027] The first and / or second metadata may comprise a count of one or more prespecified events occurring in the test run.

[0028] The first and / or second metadata may includes metadata relating to one or more of: junction types or road layouts encountered; collisions; prespecified manoeuvres carried out; other agents encountered in the test run; and driving rules broken in the test run.

[0029] The method may comprise generating a trace of the test run. The trace may represent a history of a location and motion of an agent over the course of the test run. The method may comprise generating, by a test oracle based on the trace, performance metrics reflective of the performance of the agent over the course of the test run. The method may comprise generating the second metadata from the performance metrics. The method may comprise storing the performance metrics in the scenario database.

[0030] The method may comprise receiving a query to retrieve scenarios meeting at least one criterion; and retrieving the set of scenarios from the scenario database having first metadata and / or second metadata matching the criterion. The method may comprise receiving user input specifying the query.

[0031] Retrieving the set of scenarios may comprise retrieving a plurality of scenarios from the scenario database; and sampling from the plurality of scenarios to generate the set of scenarios.

[0032] The method may comprise generating the first metadata by executing a test run of each of the plurality of scenarios, and storing the first metadata in the database.

[0033] The method may comprise determining a proportion of scenarios stored in the scenario database that have been executed by the second autonomous vehicle planner.

[0034] The methods of the first aspect and second aspect may be combined. In other words, the disclosure extends to examples in which the scenario datastore of the second aspect is populated with scenario descriptions generated according to the method of the first aspect.

[0035] According to a third aspect of the present disclosure, there is provided a computer system comprising one or more computers configured to implement any of the methods disclosed herein.

[0036] According to a fourth aspect of the present disclosure, there are provided computer- readable instructions stored in transitory or non-transitory media and configured, when executed in a computer system, to implement any of the methods disclosed herein.Brief Description of Figures

[0037] Illustrative embodiments will now be described, by way of example only, with reference to the following schematic figures, in which:

[0038] FIG. 1 A shows a schematic function block diagram of an autonomous vehicle stack;

[0039] FIG. IB shows a schematic overview of an autonomous vehicle testing paradigm;

[0040] FIG.1C shows a schematic block diagram of a scenario extraction pipeline;

[0041] FIG.2 shows a schematic block diagram of a testing pipeline;

[0042] FIG.3 shows a schematic block diagram of a visualization component for rendering a graphical user interface on which test results are displayed;

[0043] FIG.4 shows a view available within a graphical user interface;

[0044] FIG.5 shows an example road network;

[0045] FIG.6 shows part of a road network annotated with OpenDRIVE elements used to describe the road network;

[0046] FIG.7 shows a lane graph of a static road layout;

[0047] FIG.8 shows a schematic block diagram of a scenario generation tool; and

[0048] FIG. 9 shows an example scenario and corresponding graph.

[0049] FIG. 10 shows an example of graph matching in more detail.

[0050] FIG. 11 shows a schematic block diagram of an example scenario testing tool.

[0051] FIG. 12 shows a schematic flowchart of a searching technique.

[0052] FIG. 13 shows example OpenSCENARIO V2.0 definitions of the scenario of FIG. 9.Detailed Description

[0053] Test scenarios are considered, in which an ego agent is required to navigate a real or modelled physical context. The ego agent is a real or simulated mobile robot that moves under the control of the stack under testing. The physical context includes static and / or dynamic element(s) that the stack under testing is required to respond to effectively. For example, the mobile robot may be a fully or semi-autonomous vehicle under the control of the stack (the ego vehicle). The physical context may comprise a static road layout and a given set of environmental conditions (e.g. weather, time of day, lighting conditions, humidity, pollution / particulate level etc.) that could be maintained or varied as the scenario progresses. A dynamic scenario additionally includes one or more other agents (“external” agent(s), e.g. other vehicles, pedestrians, cyclists, animals etc.).

[0054] Herein, a scenario may be generally defined by a scenario skeleton (logical scenario), having (amongst other things) a plurality of parameters. Variations of the scenario (concrete scenarios) are generated from different parameterisations of the skeleton (e.g. varying external conditions, varying starting speeds of other vehicles etc). That is to say a concrete scenario is a particular instance of a logical scenario, derived from the logical scenario by selecting particular parameters. Note that the term ‘scenario’ may be used herein to refer to a concrete scenario or a logical scenario, and the meaning shall be clear from the context.

[0055] In existing approaches, users may create logical scenarios using a scenario editing tool. The created logical scenarios are stored in a database. For large test campaigns involving large numbers of simulations, at test time concrete scenarios are instantiated “on demand” from the logical scenarios, in a manner that explores the relevant parameter space. For example, ranges of the parameters may be sampled stochastically according to some predetermined distribution. This approach can be satisfactory where the number of scenariosis in the order of the thousands, and the instantiation time for each scenario is in the order of milliseconds.

[0056] Recently, it has become desirable to frame the testing process for AVs in terms of Operational Design Domain (ODD) coverage. In short, the ODD codifies the possible technical limitations of the AV. This may be for example codified in an ontology as discussed in International Patent Publication WO2021 / 245152 Al, which is incorporated herein by reference in its entirety. The aim of testing is then to adequately cover the ODD, which may involve running a wide range of tests across a very large search space.

[0057] In such ODD testing, the existing approaches may be unsatisfactory. Particularly, the constraint of generating scenarios on demand (i.e. in response to a user request or input) can become problematic when the number of tests grows, even if the generation time for a concrete scenario is in the order of milliseconds. Furthermore, the process of generating the concrete scenarios on the fly from logical scenarios by sampling the parameter space means there is no suitable way of determining in advance (or recording after the fact) whether the particular concrete scenario generated is helpful in ODD compliance (or compliance with design intention of the stack more generally). As a consequence, there is also no means of adequately tracking which parts of the parameter space result in salient and useful scenarios, and which parts do not.

[0058] The disclosure herein addresses these issues, by decoupling the generation of tests from the execution of tests. In other words, the concrete scenarios are generated in an “offline” process and stored for later execution. This removes the constraint for scenario instantiation to be in the order of milliseconds, and facilitates the creation of a large high- quality test suite of scenarios (i.e. including hundreds of thousands or millions of scenarios) that can be reused and run many times over the development lifetime of the AV stack.

[0059] In overview, a scenario generation tool is described herein, which generates new test scenarios from existing scenario representations. The tool represents dynamic agents (e.g. other vehicles, pedestrians etc) of an existing scenario as a graph. The graph is an abstract representation of the relationships between dynamic elements. Static elements (e.g. the road layout) of other existing scenarios are similarly represented as graphs. The dynamic graphs can be matched to the static graphs, to generate new combinations of static and dynamic elements, and consequently new scenarios.

[0060] The disclosure also provides techniques that take advantage of the stored tests, with different planners (e.g. different iterations of a stack under development) accessing the tests and generating and maintaining metadata relevant for searching the tests.

[0061] In an offline simulation context, a scenario description is provided to an offline simulator as input, in order to expose a stack under testing to a simulated scenario, and test its performance in the simulated scenario. This allows performance issues (such as safety issues) to be identified and mitigated prior to real-world deployment. In an online context, a perception system may be used to generate a scenario description that can be used as a basis for higher-level functions, such as motion prediction and planning, which might involve some form of online simulation to simulate possible futures and plan accordingly.

[0062] A scenario description may be encoded using a scenario description language (SDL), or in any other form that can be consumed by whichever component(s) require it. As briefly discussed, the ASAM OpenDRIVE (R) standard defines a storage format for the static description of road networks and OpenSCENARIO (R) may be used to add dynamic content. Other forms of scenario description may be used, including bespoke languages and formats, and the present techniques are not limited to any particular SDL, storage format, schema or standard.

[0063] A “scenario run” or “scenario instance” refers to a concrete occurrence of an agent(s) navigating a physical context, optionally in the presence of one or more other agents. A single scenario description can give rise to multiple simulated runs, with different outcomes, not least because those outcomes depend on decisions taken by the stack under testing (as changes are made to the stack to improve performance, that may change the outcome on a given concrete scenario). The terms “run” and “instance” are used interchangeably in this context.

[0064] Figure 1 A shows, by way of context, a highly schematic block diagram of an AV runtime stack 100. The stack 100 is an example of a robotic system as discussed herein. The stack 100 may be fully or semi-autonomous. For example, the stack 100 may operate as an Autonomous Driving System (ADS) or Advanced Driver Assist System (ADAS).

[0065] The runtime stack 100 is shown to comprise a perception system 102, a prediction system 104, a planning system (planner) 106 and a control system (controller) 108.

[0066] In a real -world context, the perception system 102 receives sensor outputs from an onboard sensor system 110 of the AV, and uses those sensor outputs to detect external agents and measure their physical state, such as their position, velocity, acceleration etc. The onboard sensor system 110 can take different forms but generally comprises a variety of sensors such as image capture devices (cameras / optical sensors), lidar and / or radar unit(s), satellitepositioning sensor(s) (GPS etc.), motion / inertial sensor(s) (accelerometers, gyroscopes etc.) etc. The onboard sensor system 110 thus provides rich sensor data from which it is possible to extract detailed information about the surrounding environment, and the state of the AV and any external actors (vehicles, pedestrians, cyclists etc.) within that environment. The sensor outputs typically comprise sensor data of multiple sensor modalities such as stereo images from one or more stereo optical sensors, lidar, radar etc. Sensor data of multiple sensor modalities may be combined using filters, fusion components etc.

[0067] The perception system 102 typically comprises multiple perception components which co-operate to interpret the sensor outputs and thereby provide perception outputs to the prediction system 104.

[0068] In a simulation context, depending on the nature of the testing, it may or may not be necessary to model the on-board sensor system 100. For example, when only the planner 106 (or only the planner 106 and controller 108) are tested on a ‘perfect’ representation of the scenario (that is, directly on simulator ground truth), simulated sensor data is not required therefore complex sensor modelling is not required. Surrogate model(s) of the perception system 102 (or part(s) of it) can also be used to test planner performance (or planner and controller performance) in the presence of realistic perception errors, without the use of sensor models.

[0069] The perception outputs from the perception system 102 are used by the prediction system 104 to predict future behaviour of external actors (agents), such as other vehicles in the vicinity of the AV.

[0070] Predictions computed by the prediction system 104 are provided to the planner 106, which uses the predictions to make autonomous driving decisions to be executed by the AV in a given driving scenario. The inputs received by the planner 106 would typically indicate a drivable area and would also capture predicted movements of any external agents (obstacles, from the AV’s perspective) within the drivable area. The driveable area can be determinedusing perception outputs from the perception system 102 in combination with map information, such as an HD (high definition) map.

[0071] A core function of the planner 106 is the planning of trajectories for the AV (ego trajectories), taking into account predicted agent motion. This may be referred to as trajectory planning. A trajectory is planned in order to carry out a desired goal within a scenario. The goal could for example be to enter a roundabout and leave it at a desired exit; to overtake a vehicle in front; or to stay in a current lane at a target speed (lane following). The goal may, for example, be determined by an autonomous route planner 116, also referred to as a goal generator 116.

[0072] The controller 108 executes the decisions taken by the planner 106 by providing suitable control signals to an on-board actor system 112 of the AV. In particular, the planner 106 plans trajectories for the AV and the controller 108 generates control signals to implement the planned trajectories. Typically, the planner 106 will plan into the future, such that a planned trajectory may only be partially implemented at the control level before a new trajectory is planned by the planner 106. The actor system 112 includes “primary” vehicle systems, such as braking, acceleration and steering systems, as well as secondary systems (e.g. signalling, wipers, headlights etc.).

[0073] The example of Figure 1 A considers a relatively “modular” architecture, with separable perception, prediction, planning and control systems 102-108. The sub-stack themselves may also be modular, e.g. with separable planning modules within the planning system 106. For example, the planning system 106 may comprise multiple trajectory planning modules that can be applied in different physical contexts (e.g. simple lane driving vs. complex junctions or roundabouts). This is relevant to simulation testing for the reasons noted above, as it allows components (such as the planning system 106 or individual planning modules thereof) to be tested individually or in different combinations. For the avoidance of doubt, with modular stack architectures, the term stack can refer not only to the full stack but to any individual sub-system or module thereof.

[0074] The extent to which the various stack functions are integrated or separable can vary significantly between different stack implementations - in some stacks, certain aspects may be so tightly coupled as to be indistinguishable. For example, in other stacks, planning and control may be integrated (e.g. such stacks could plan in terms of control signals directly), whereas other stacks (such as that depicted in Figure 1 A) may be architected in a way thatdraws a clear distinction between the two (e.g. with planning in terms of trajectories, and with separate control optimizations to determine how best to execute a planned trajectory at the control signal level). Similarly, in some stacks, prediction and planning may be more tightly coupled. At the extreme, in so-called “end-to-end” driving, perception, prediction, planning and control may be essentially inseparable. Unless otherwise indicated, the perception, prediction, planning and control terminology used herein does not imply any particular coupling or modularity of those aspects.

[0075] A “full” stack typically involves everything from processing and interpretation of low-level sensor data (perception), feeding into primary higher-level functions such as prediction and planning, as well as control logic to generate suitable control signals to implement planning-level decisions (e.g. to control braking, steering, acceleration etc.). For autonomous vehicles, level 3 stacks include some logic to implement transition demands and level 4 stacks additionally include some logic for implementing minimum risk maneuvers. The stack may also implement secondary control functions e.g. of signalling, headlights, windscreen wipers etc.

[0076] Whilst the following description refers to the stack 100 in the context of testing, testing may be applied to individual components / portions of the stack, such as the perception, prediction, planning or control stacks 104, 106, 108 (alone or in various combinations), or individual component(s) thereof. A stack (or component) can refer purely to software, i.e. one or more computer programs that can be executed on one or more general-purpose computer processors. However, such terminology can also encompass hardware. In simulation, software of the stack may be tested on a “generic” off-board computer system, before it is eventually uploaded to an on-board computer system of a physical vehicle. However, in “hardware-in-the-loop” testing, the testing may extend to underlying hardware of the vehicle itself. For example, the stack software may be run on the on-board computer system (or a replica thereof) that is coupled to the simulator for the purpose of testing. In this context, the stack under testing extends to the underlying computer hardware of the vehicle. As another example, certain functions of the stack 100 (e.g. perception functions) may be implemented in dedicated hardware. In a simulation context, hardware-in-the loop testing could involve feeding synthetic sensor data to dedicated hardware perception components.

[0077] Within the stack 100, a scenario description 116 may be used as a basis for planning and prediction. The scenario description 116 is generated using the perception system 102,together with a high-definition (HD) map 114. By localizing the ego vehicle 114 on the map, it is possible to combine the information extracted in the perception system 102 (including dynamic agent information) with the pre-existing environmental information contained in the HD map 114. The scenario description 116 is, in turn, used as a basis for motion prediction in the prediction system 104, and the resulting motion predictions 118 are used in combination with the scenario description 116 as a basis for planning in the planning system 106.

[0078] Figure IB shows a highly schematic overview of a testing paradigm for autonomous vehicles. An ADS / ADAS stack 100, e.g. of the kind depicted in Figure 1 A, is subject to repeated testing and evaluation in simulation, by running multiple scenario instances in a simulator 202, and evaluating the performance of the stack 100 (and / or individual subs-stacks thereof) in a test oracle 252. The output of the test oracle 252 is informative to an expert 122 (team or individual), allowing them to identify issues in the stack 100 and modify the stack 100 to mitigate those issues (S124). The results also assist the expert 122 in selecting further scenarios for testing (S126), and the process continues, repeatedly modifying, testing and evaluating the performance of the stack 100 in simulation. The improved stack 100 is eventually incorporated (S125) in a real-world AV 101, equipped with a sensor system 110 and an actor system 112. The improved stack 100 typically includes program instructions (software) executed in one or more computer processors of an on-board computer system of the vehicle 101 (not shown). The software of the improved stack is uploaded to the AV 101 at step S125. Step S125 may also involve modifications to the underlying vehicle hardware. On board the AV 101, the improved stack 100 receives sensor data from the sensor system 110 and outputs control signals to the actor system 112. Real-world testing (S128) can be used in combination with simulation-based testing. For example, having reached an acceptable level of performance through the process of simulation testing and stack refinement, appropriate real -world scenarios may be selected (SI 30), and the performance of the AV 101 in those real scenarios may be captured and similarly evaluated in the test oracle 252.

[0079] Figure 1C shows a highly schematic block diagram of the testing pipeline applied to a scenario description 201. A scenario generator 146 generates the scenario description 201 in a form that may be consumed by the simulator 202, allowing multiple simulated runs to be derived therefrom (for example, with different configurations of the AV stack 100). Ground truth 150 is provided for each simulated run.

[0080] The ground truth 150 for each run comprises a trace of the ego agent and any other dynamic agents in the scenario. A trace is a history of an agent’s location and motion over the course of a scenario. There are many ways a trace can be represented. Trace data will typically include spatial and motion data of an agent within the environment.

[0081] Further details of an example testing pipeline incorporating the test oracle 252 will now be described.

[0082] Figure 2 shows a schematic block diagram of the testing pipeline, denoted by reference numeral 200. The testing pipeline 200 is shown to comprise the simulator 202 and the test oracle 252. The simulator 202 runs simulated scenarios for the purpose of testing all or part of an AV run time stack 100, and the test oracle 252 evaluates the performance of the stack (or sub-stack) on the simulated scenarios. As discussed, it may be that only a sub-stack of the run-time stack is tested, but for simplicity, the following description refers to the (full) AV stack 100 throughout. However, the description applies equally to a sub-stack in place of the full stack 100. The term “slicing” is used herein to the selection of a set or subset of stack components for testing.

[0083] The idea of simulation-based testing is to run a simulated driving scenario that an ego agent must navigate under the control of the stack 100 being tested. Typically, the scenario includes a static drivable area (e.g. a particular static road layout) that the ego agent is required to navigate, typically in the presence of one or more other dynamic agents (such as other vehicles, bicycles, pedestrians etc.). To this end, simulated inputs 203 are provided from the simulator 202 to the stack 100 under testing.

[0084] The slicing of the stack dictates the form of the simulated inputs 203. By way of example, Figure 2 shows the prediction, planning and control systems 104, 106 and 108 within the AV stack 100 being tested. To test the full AV stack of Figure 1 A, the perception system 102 could also be applied during testing. In this case, the simulated inputs 203 would comprise synthetic sensor data that is generated using appropriate sensor model(s) and processed within the perception system 102 in the same way as real sensor data. This requires the generation of sufficiently realistic synthetic sensor inputs (such as photorealistic image data and / or equally realistic simulated lidar / radar data etc.). The resulting outputs of the perception system 102 would, in turn, feed into the higher-level prediction and planning systems 104, 106.

[0085] By contrast, so-called “planning-level” simulation would essentially bypass the perception system 102. The simulator 202 would instead provide simpler, higher-level inputs 203 directly to the prediction system 104. In some contexts, it may even be appropriate to bypass the prediction system 104 as well, in order to test the planner 106 on predictions obtained directly from the simulated scenario (i.e. “perfect” predictions).

[0086] Between these extremes, there is scope for many different levels of input slicing, e.g. testing only a subset of the perception system 102, such as “later” (higher-level) perception components, e.g. components such as filters or fusion components which operate on the outputs from lower-level perception components (such as object detectors, bounding box detectors, motion detectors etc.).

[0087] As an alternative to generating high-fidelity synthetic sensor data that is fed to the perception system 102, all or part of the perception system 102 may be modelled using one or more “surrogate” perception models. A surrogate model models the perception system 102 (or some component / components thereof), but operates more efficiently on lower-fidelity inputs derived from the simulation ground truth, without the need for sensor models. A surrogate model allows the simulated inputs 203 to be generated with realistic error (comparable to the error that would be produced by the perception system 102 in a comparable real-world situation) without the use of sensor models.

[0088] Whatever form they take, the simulated inputs 203 are used (directly or indirectly) as a basis for decision-making by the planner 106. The controller 108, in turn, implements the planner’s decisions by outputting control signals 109. In a real -world context, these control signals would drive the physical actor system 112 of AV. In simulation, an ego vehicle dynamics model 204 is used to translate the resulting control signals 109 into realistic motion of the ego agent within the simulation, thereby simulating the physical response of an autonomous vehicle to the control signals 109.

[0089] Alternatively, a simpler form of simulation assumes that the ego agent follows each planned trajectory exactly between planning steps. This approach bypasses the control system 108 (to the extent it is separable from planning) and removes the need for the ego vehicle dynamic model 204. This may be sufficient for testing certain facets of planning.

[0090] To the extent that external agents exhibit autonomous behaviour / decision making within the simulator 202, some form of agent decision logic 210 is implemented to carry outthose decisions and determine agent behaviour within the scenario. The agent decision logic 210 may be comparable in complexity to the ego stack 100 itself or it may have a more limited decision-making capability. The aim is to provide sufficiently realistic external agent behaviour within the simulator 202 to be able to usefully test the decision-making capabilities of the ego stack 100. In some contexts, this does not require any agent decision making logic 210 at all (open-loop simulation), and in other contexts useful testing can be provided using relatively limited agent logic 210 such as basic adaptive cruise control (ACC). One or more agent dynamics models 206 may be used to provide more realistic agent behaviour if appropriate.

[0091] A scenario is run in accordance with a scenario description 201, which typically has both static and dynamic elements. Scenario runs are orchestrated by a test orchestration component 260.

[0092] The static elements are encoded in a static layer 201a of the scenario description 201. The static element(s) typically include a road layout. For example, the static layer 201a may be encoded in the OpenDrive format, or a comparable road network SDL that describes both the geometry and the topology of the road layout.

[0093] The dynamic element(s) are encoded in a dynamic layer 201b, and typically include one or more external agents within the scenario, such as other vehicles, pedestrians, bicycles etc.

[0094] The scenario description 201 is a logical scenario description. That is to say, the description 201 includes a plurality of parameters. Example parameters may include weather or lighting parameters reflective of the time of day / level of illumination / weather conditions etc. Example parameters may also include the speeds of agents within the scenario.

[0095] Before input to the simulator 202, a concrete scenario 205 is instantiated from the scenario description, by selecting particular values for the parameters.

[0096] The extent of the dynamic information provided to the simulator 202 for each external agent can vary. For example, a scenario may be described by separable static and dynamic layers. A given static layer (e.g. defining a road layout) can be used in combination with different dynamic layers to provide different scenario instances. The dynamic layer may comprise, for each external agent, a spatial path to be followed by the agent together with one or both of motion data and behaviour data associated with the path. In simple open-loopsimulation, an external actor simply follows the spatial path and motion data defined in the dynamic layer that is non-reactive i.e. does not react to the ego agent within the simulation. Such open-loop simulation can be implemented without any agent decision logic 210. However, in closed-loop simulation, the dynamic layer instead defines at least one behaviour to be followed along a static path (such as an ACC behaviour). In this case, the agent decision logic 210 implements that behaviour within the simulation in a reactive manner, i.e. reactive to the ego agent and / or other external agent(s). Motion data may still be associated with the static path but in this case is less prescriptive and may for example serve as a target along the path. For example, with an ACC behaviour, target speeds may be set along the path which the agent will seek to match, but the agent decision logic 210 might be permitted to reduce the speed of the external agent below the target at any point along the path in order to maintain a target headway from a forward vehicle.

[0097] The output of the simulator 202 for a given simulation includes an ego trace 212a of the ego agent and one or more agent traces 212b of the one or more external agents (traces 212). Each trace 212a, 212b is a complete history of an agent’s behaviour within a simulation having both spatial and motion components. For example, each trace 212a, 212b may take the form of a spatial path having motion data associated with points along the path such as speed, acceleration, jerk (rate of change of acceleration), snap (rate of change of jerk) etc.

[0098] Additional information is also provided to supplement and provide context to the traces 212. Such additional information is referred to as “contextual” data 214. The contextual data 214 pertains to the physical context of the scenario, and can have both static components (such as road layout) and dynamic components (such as weather conditions to the extent they vary over the course of the simulation).

[0099] The test oracle 252 receives the traces 212 and the contextual data 214, and scores those outputs in respect of a set of performance evaluation rules 254. The performance evaluation rules 254 are shown to be provided as an input to the test oracle 252.

[0100] The rules 254 are categorical in nature (e.g. pass / fail-type rules). Certain performance evaluation rules are also associated with numerical performance metrics used to “score” trajectories (e.g. indicating a degree of success or failure or some other quantity that helps explain or is otherwise relevant to the categorical results). The evaluation of the rules 254 is time-based - a given rule may have a different outcome at different points in the scenario.The scoring is also time-based: for each performance evaluation metric, the test oracle 252 tracks how the value of that metric (the ‘robustness’ score) changes over time as the simulation progresses. The test oracle 252 provides an output 256 comprising a time sequence 256a of categorical (e.g. pass / fail) results for each rule, and a score-time plot 256b for each performance metric, as described in further detail later. The results and scores 256a, 256b are informative to the expert 122 and can be used to identify and mitigate performance issues within the tested stack 100. The test oracle 252 also provides an overall (aggregate) result for the scenario (e.g. overall pass / fail). The output 256 of the test oracle 252 is stored in a test database 258, in association with information about the scenario to which the output 256 pertains.

[0101] Figure 3 shows a schematic block diagram of a visualization component 320. The visualization component 320 is shown having an input connected to the test database 258 for rendering the outputs 256 of the test oracle 252 on a graphical user interface (GUI) 300. The GUI is rendered on a display system 322.

[0102] Figure 4 shows an example view of the GUI 300. The view pertains to a particular scenario containing multiple agents, and is shown to comprise a scenario visualization 301 and a set of driving performance assessment results 302. In this example, the test oracle output 526 pertains to multiple external agents, and the results are organized according to agent. For each agent, a time-series of results is available for each rule applicable to that agent at some point in the scenario. Some form of visual indicator (such as coding) is used to differentiate between periods of pass / fail on a particular rule.

[0103] Figure 5 schematically depicts an example road network 500 of the kind encoded in the static layer 201a. The following description assumes the road network 500 is described using the OpenDRIVE schema, or a similar format that adopts certain definitions and conventions from OpenDRIVE. However, it will be appreciated that the principles extend more generally to other formats, and the described techniques are not limited to any particular data format or schema.

[0104] The road network 500 is shown to comprise first, second, third and fourth roads 502, 504, 506, 508 (Roads 1 to 4), which are described with <road> elements having road identifiers (IDs) 1, 2, 3 and 4 respectively. The roads 502-508 are interconnected via a junction 510 described by a <junction> element. Each of the roads 502-508 is defined by a single road reference line, denoted as a thick solid arrow, and contains a single center lane ofwidth zero. The center lanes are not depicted separately, and for simplicity it is assumed that the center lane of each road 502-508 lies along the road reference line (although, as noted, it is possible to define a non-zero offset between the road reference line and the center lane). The road reference lines of the first and second roads 502, 504 are denoted by reference numerals 503 and 505 respectively. A road reference line is directional, and could be described more precisely as a longitudinal axis or “s-axis” of the road, with s-coordinates running along that axis, and t-coordinates running orthogonal to it. As depicted for the reference lines 503, 505, the positive t-direction is defined as extending to the left of the s- axis. Lanes are numbered relative to the center lane; the center lane is always numbered 0, with side-lanes to the left of the center lane (+t) assigned incrementing positive lane numbers (1, 2, 3, . . .) and lanes to the right (-t) assigned decreasing negative lane numbers (-1, -2, -3, ...).

[0105] Each road has a minimum of one side-lane. In this example, the first road 502 has only positively numbered side-lanes to the left of the center lane, whereas the second road 504 has two positive side-lanes to the left of the center lane, and one negative side-lane to the right. Roads may be divided into “lane sections” to accommodate sections of the road with different numbers of lanes (see below).

[0106] In the following description, “Lane n” means a lane with lane number “n” and “Road m” means a road with road ID “m”. Other road elements (such as junctions) are referred to in the same way.

[0107] Adjacent lane sections may be referred to as “Lane Section n” and “Lane Section n+1” (the lane section numbers n, n+1 does not form part of the OpenDRIVE schema, but are nevertheless derivable from the static layer 201a).

[0108] A left-hand traffic (LHT) road system is depicted in this example. However, the schema can be used for either LHT or right-hand traffic (RHT) networks. A “rule” attribute of each <road> element indicates whether the road is LHT (vehicles drive on the left) or RHT (vehicles drive on the right).

[0109] A global cartesian coordinate system is defined with the x-direction lying eastwards and the y-axis extending northwards (OpenDRIVE calls this the inertial coordinate system).[HO] Lane numbers only indicate relative directions of traffic flow within a road: for any given road, traffic flows in the same direction for positively-numbered one-way side-lanes,and in the opposite direction for negatively-numbered one-way side-lanes. Bi-directional lanes support traffic flow in both directions irrespective of the road traffic rule. However, the lane number alone is not sufficient to infer the direction of traffic flow, as the direction of the s-axis can be (more or less) arbitrarily chosen and does not indicate driving direction. For example, in Figure 5, the s-axis 505 of the second road 504 extends towards the junction from the east, whilst the s-axis of the fourth road 508 extends in the opposite direction towards the junction 510 from the west. Along the second road 504, positive lane numbers therefore denote a direction of traffic flow from east to west, whereas along the fourth road 508, east- to-west traffic flow is denoted by negative lane numbers. For a LHT road, lanes to the left of the road reference line (+t) carry traffic in the direction of the road reference line (+s), whereas lanes to the right of the road reference line (-t) carry traffic in the opposite direction (-s). In an RHT road, lanes to the left of the road center line (+t) carry traffic in the opposite direction of the road reference line (-s) and lanes to the right (-t) carry traffic in the direction of the road reference line (+s).[Hl] It is also possible to define a bidirectional side-lane, permitting traffic flow in both directions, by setting a @type attribute of the <lane> element to “bidirectional”. Bidirectional lanes are addressed in more detail below.

[0112] Therefore, in order to determine the absolute direction of traffic flow, the lane number is not sufficient; the direction of the road reference line (s-axis) must also be considered, as must the @type attribute.

[0113] Lanes are not necessarily drivable. For example, Lane 1 of Lane Section 2 of Road 2 is non-drivable. The outer boundaries of a road may also be defined by non-drivable lanes (such as lanes of a pavement / sidewalk type).

[0114] Link elements are used to explicitly define linkage between roads and lanes. With only two roads, links can be defined with link elements between roads directly, provided the links are unambiguous. More complex linkage requires the use of junction elements.

[0115] A <link> element can have one or both of a <successor> element and a <predecessor> element. The predecessor / successor of a road can be another road or a junction. The predecessor / successor of a lane is another lane. “Predecessor” and “successor” relationships are defined relative to the s-axis of the road in question (a road’s t-axis runs from itspredecessor, if any, to its successor, if any), and do not denote driving direction. The s-axis of a road runs away from its predecessor (if any), towards its successor (if any).

[0116] Predecessor / successor relationships may be ‘asymmetrical’. For example, if Road n is a predecessor of Road m, that does not imply Road m is a successor of Road n; if the s-axis of Road n runs in the opposite direction to the s-axis of Road m, then Road m could also be a predecessor of Road n. As another example, if Road m is part of a junction, then Road m cannot be a predecessor or successor of Road n, because the junction would be the predecessor / successor of Road n instead (see below for further examples).

[0117] Roads with varying number of lanes are accommodated using lane sections. Within the OpenDRIVE schema, lanes are described within a <lanes> element of a <road> element. A <lanes> element may be split into multiple sections (lane sections) with a <laneSection> element. Each individual lane is defined by a <lane> element within the <laneSection> element. Each lane section has a fixed number of lanes, numbered as per the above convention. For example, the first road 502 is shown divided into two lane sections (Lane Sections 1 and 2 of Road 1): approaching the junction 510, the number of lanes is shown to increase from two to three. On entering Lane Section 2 from Lane Section 1 of the first road 502, a new rightmost lane is created, whose width gradually increases from zero. In Lane Section 1, the left and right most lanes (to the left of the center lane) are numbered 2 and 1 respectively, whilst in Lane Section 2 the left-most lane and new right-most lanes are numbered 3 and 1 respectively; what is now the middle lane is numbered 2.

[0118] Links between lanes in adjacent lane sections are described using <link> elements. In Lane Section 2, Lane 2 of Lane Section 1 is a predecessor of Lane 3 of Lane Section 2, and Lane 1 of Lane Section 1 is a predecessor of Lane 2 of Lane Section 2. That is to say, the lane element describing Lane 3 of Lane Section 2 would contain a link element indicating Lane 2 as its predecessor, and it is implicit that the link refers to the immediately preceding lane section (Lane Section 1) etc.

[0119] Likewise, in Lane Section 1, Lane 3 of Lane Section 2 is a successor of Lane 2 of Lane Section 1, and Lane 2 of Lane Section 2 is a successor of Lane 1 of Lane Section 1. That is, the lane element describing Lane 2 of Lane Section 1 would indicate Lane 3 as its successor, and it is implicit that the link refers to the next lane section (Lane Section 2) etc.

[0120] Because Lane 1 of Lane Section 2 has zero width initially, it is not linked to any lane of Lane Section 1. It is, however, possible for a lane to have multiple successors or predecessors in different circumstances, for example if a lane splits immediately into two lanes, each of non-zero width at the lane section boundary.

[0121] Within the junction element 510 of Figure, additional roads are defined, whose reference lines are depicted as thick, solid arrows. Fifth to ninth roads 512, 514, 516, 518, 520 (Roads 5 to 9) are depicted within the junction 510 in this example. Although side-lanes within the junctions 510 are not depicted in Figure 5, each road within the junction 510 is also required to have at least one side-lane of non-zero width.

[0122] In Figure 5, the junction 510 is a successor of the first, second and fourth roads 502, 504, 508 because their respective s-axes extend towards the junction 510. Therefore, the <road> elements describing those roads 502, 504, 508 would contain <link> elements with <successor> elements indicating the junction 510. The junction 510 is a predecessor of the third road 506 because its s-axis extends away from the junction, and the <road> element describing the third road 506 would therefore contain a <link> element with a <predecessor> element indicating the junction 510.

[0123] Within the junction element 510, connecting roads are described by <road> and <connection> elements.

[0124] The fifth road 512 (Road 5) is shown to connect the first and third roads 502, 506. The fifth road 512 is defined by a <road> element within the <junction> element 510, of which the first road 502 is a predecessor and the third road 506 is a successor (defined via <predecessor> and <successor> elements in the <road> element describing the fifth road 506). Similarly the sixth road 514 has the fourth road 508 as its successor and the second road 504 as its predecessor. The seventh road 516 connects the second and third roads 504, 506, with the second road 504 as its predecessor and the third road 506 as its successor. The eighth road 518 connects the second and first roads 504, 502, with the second road 504 as successor and the first road 502 as predecessor. Finally, the ninth road 520 connects the first and fourth roads 502, 508, with the fourth road 508 as its successor and the first road 502 as its predecessor. Again, the predecessor / successor relationships are not direct indicators of driving direction.

[0125] A <connection> element is used to indicate driving direction for traffic joining a junction. A connection element indicates an incoming road and a connecting road (but does not explicitly define an outgoing road), with traffic entering the junction from the incoming road onto the connecting road. For example, a first <connection> element would be provided that indicates the road with ID=1 (the first road 502) as an incoming road and the road with ID=5 (the fifth road 512) as a connecting road; this connection element indicates that traffic can enter the junction 510 from the first road 502 onto the fifth road 512. A second <connection> element would indicate the first road 502 as an incoming road and the eighth road 518 as a connecting road, etc. The seventh connecting road 516 is a two-way road in this example, carrying traffic from the second road 504 to the third road 506, and traffic in the opposite direction. Therefore, two <connection> elements would be used, one indicating the second road as incoming and the seventh road 516 as connecting, and the other indicating the third road 506 as incoming and the seventh road 516 as connecting. The OpenDRIVE specification strongly advises one not to use two-way connecting roads, although two-way connecting roads are possible using the schema. The seventh connecting road 516 goes against this advice, but does not violate it (and, in practice, it is more likely that two one-way connecting roads would be defined). The fourth road 508 is not an incoming road to the junction 510 (even though its s-axis extends towards the junction 510), because it is a oneway road that only carries traffic away from the junction 510.

[0126] A connecting road with multiple lanes is used when lane changes are possible within the junction 510; if lane changes are not permitted, multiple single-lane roads would be used instead.

[0127] A “virtual” connection can also be described using a <connection> element without any connecting road. Virtual connections are limited to “virtual” junctions, which do not require a main road to be split up.

[0128] Links between lanes are described via link elements within the <lane> elements that describe those lanes, where predecessor and successor lanes are described in a similar manner to roads. In order for a first lane of a first road to be a predecessor or successor of a second lane of a second road, the second road must be a predecessor or successor of the first road. As the junction 510 is the successor of the first, second and fourth roads 502, 504, 508, these roads 502, 504, 508 have no direct successor roads; therefore, it is not possible for any of their lanes to have lane successors. Likewise, as the junction 510 is the predecessor of thethird road 506, the third road 506 has no direct predecessor road; therefore, its lanes cannot have any predecessors. Within the junction 501, however, each of the connecting roads 512- 510 has both a direct successor road and a direct predecessor road; for example, the direct successor road of the fifth road 512 is the first road 502 and its direct predecessor is the third road 506. Consequently, lanes of the connecting roads 512-520 can be linked to lanes of both their predecessor and their successor roads. For example, the fifth road 512 would typically contain two positively-numbered side-lanes, 1 and 2. The side-lanes of Road 5 are not depicted in Figure 5, but are shown in Figure 6. The lane elements describing lanes 2 and 1 of Road 5 would, in turn, contain link elements, which indicate lane numbers “3” and “2” as their respective predecessors (and it is implicit that these are lanes of Road 5’s successor, namely Road 1), and lane numbers “2” and “1” as their respective successors (implicitly referring to Road 5’s successor, namely Road 3). A <connection> element also contains any lane.

[0129] Figure 6 shows part of the road network 500 of Figure 5, annotated with sections of OpenSCENARIO code that define certain elements of the road network.

[0130] Starting from any lane of any of the incoming roads 502, 506, 506, the <connection> elements within the junction describe all possible routes though the junction 510 (at the road level). For a given lane on a given road, obtaining a list of all routes through the junction would mean locating all of the <connection> elements that identify the road in question as an incoming road, via its road ID, and which also have a lane linked to the lane in question. To obtain further lane links, it would then be necessary to locate the <road> element of its connecting road, in order to identify the outgoing road (for the fifth road 512, the outgoing road would be the third road 506), and determine the lane linkage between the connecting road and the outgoing road.

[0131] A first code portion 602 contains a connection element (connection ID=0) that indicates the first road 502 (the road with ID 1) as an incoming road and the fifth road 512 (the road with ID 5) as a connecting road. First and second <laneLink> elements link Lane 3 of Road 1 (its incoming road) to Lane 2 of Road 5, and Lane 2 of Road 1 to Lane 1 of Road 5. A second portion of code 604 contains the <road> element describing Road 5 within the junction 510 (junction ID=0). A <link> element within the road element contains both <successor> and <predecessor> elements. The road link in the predecessor element indicated Road 1 as the predecessor to Road 5, which mirrors the lane links in the corresponding<connection> element (Connection 0). The successor element indicates the third road 506 (the road with ID 3) as successor to Road 5. A <lane> element is also shown within the road element for Road 5, which describes Lane 2 of Road 5 (other lane elements are omitted for conciseness); the lane element contains a link element, which in turn indicates Lane 2 as its successor and Lane 3 as its predecessor. To meaningfully interpret the lane links, it is necessary to consider the road link information; Road 1 is the predecessor to Road 5, therefore Lane 3 of Road 1 is the predecessor of Lane 2 of Road 5; Road 3 is the successor to Road 5, therefore Lane 2 of Road 3 is the successor to Lane 2 of Road 5. A third code portion 606 contains the road element describing Road 3. The junction 510 (junction ID=0) is indicated as the predecessor to Road 3, and a lane element is shown that describes Lane 2 of Road 3 (other lane elements are omitted for conciseness).

[0132] Figure 7 shows part of a directed side-lane graph 700 representing the road topology of the road network 500. The directed side-lane graph 700 is referred to simply as the lane graph 700 for conciseness. The lane graph 700 describes lane interconnections from Road 1 though the junction 510, and may be derived from the underlying OpenDrive document taking into account the considerations set out above. For example, first and second nodes 702, 704 correspond, respectively, to Lane 1 of Lane Section 1 of Road 1 and Lane 2 of Lane Section 2 of Road 1. An onward edge 706 indicates the topological relationship between those lanes, namely that the latter is an “onward” lane from the former (i.e. a vehicle can traverse from the former to the latter without performing a lane change maneuver). A directed edge from node 702 to node 704 has an edge type describing the connection between the corresponding lanes (“onward” in this case).

[0133] The lane graph 700 may be constructed by interpreting the code of the static layer 201a. Note that edges of the lane graph denote driving direction. To determine onward edges, lane links need to be considered, but driving direction also needs to be considered.

[0134] Edges are also provided for left-right relationships. For example, third and fourth nodes 708, 710 are depicted, representing Lanes 3 and 1, respectively, of Lane Section 2 of Road 1. A right edge 711 from the second node 704 to the fourth node 710 represents the possibility of moving from Lane 2 to Lane 1 of that lane section via a right lane change maneuver. A left edge 709 from the second node 704 to the third node 708 represents the possibility of moving from Lane 2 to Lane 3 via a left lane change maneuver. The left and right edges 709, 711 have respective types (“left” and “right”), again describing the nature ofthe lane connections. This information is not provided in <link> elements of the underlying description, but can be derived from the structure of the road.

[0135] A fifth node 714 represents Lane 1 in the only road section of Road 5, which is an onward lane from Lane 2 in Lane Section 2 or Road 1. Therefore, an onward edge 712 is directed from the second node 704 representing the former to the fifth node 714 representing the latter.

[0136] A dynamic layer 201b may be defined and executed with respect to the lane graph 700. For example, particular agents may be associated with particular nodes at certain times (such as the start of the scenario, or defined ‘keyframes’ or synchronization points). In simulation, this topological description of the scenario dynamics is converted to a concrete geometrical realization. Therefore, changes can be made to the road network 500 and, provided those changes do not alter the relevant road topology of the lane graph 700 (typically some sub-graph or sub-graphs of the lane graph 700), it is straightforward to run the same dynamic layer 201b on the modified road network without modification to the dynamic layer.

[0137] The dynamic layer 201b can also contain geometric information, such as relative locations of agents at particular points in time (e.g. at the start of the scenario, or at some ‘synchronization’ point during the scenario).

[0138] FIG. 8 is a schematic block diagram of an example scenario generation system 800.

[0139] The scenario generation system 800 comprises a database 801 storing a plurality of logical scenario descriptions 201. Each scenario description 201 includes a static layer 201a and a dynamic layer 201b. The scenario descriptions 201 generally take the form described above with reference to Figures 1-7, and may have been manually created by domain experts.

[0140] The system 800 selects a particular scenario 201-1 (hereinafter the “source” scenario) from the database 801 to form the basis of the scenario generation. For example, it may be that a particular scenario 201-1 is selected by a user on the basis that it includes some particular elements that make it useful or salient in testing.

[0141] The system 800 generates a dynamic graph 820 from the dynamic layer 201b-l of the source scenario 201-1. The dynamic graph 820 is an abstraction of the dynamic layer 201b, representing agents in the source scenario as nodes, with edges representing the relationshipsbetween the nodes. The edges can also be thought of as constraints, in the sense that they constrain the actions or positions of the agents with respect to one another.

[0142] The dynamic graph 820 includes some edges that represent connections to the static layer 201a-l of the source scenario 201-1. These edges (referred to herein as attachment edges) represent the relationship between elements of the static layer 201a-l and the dynamic layer 201b-l in the source scenario. As the purpose of the dynamic graph 820 is to abstract the dynamic layer 201b-l alone, the attachment edges are left unconnected at the end that would attach to the static layer 201a-l.

[0143] The system 800 also generates a static graph 810 from the static layer 201a of another scenario. The static graph represents the static elements of the scenario (e.g. the elements discussed above in relation to Figure 5 such as junctions, sections of roads etc) as nodes, with edges encoding the spatial relationships between the static elements.

[0144] The system 800 then carries out a graph matching process 804, in which locations are found where the dynamic graph 820 can be attached to the static graph 810 via the attachment edges. In the example shown, this involves identifying points where the dynamic graph 820 is attachable to the static graph 810 of a different scenario, so as to generate a new combination of static and dynamic elements - i.e. a new logical scenario. Examples also include the situation where a dynamic graph 820 is attached to a different part of the static graph 810 of the same scenario - i.e. relocating the dynamic elements within the scenario to form a new logical scenario.

[0145] The system 800 then stores each identified location. This may involve simply storing which nodes of the static graph 810 the attachment edges are linked to. In other examples, this may involve generating a scenario description similar in format to the source scenario description 201. The generated scenario description includes the static layer corresponding to static graph 810, with a dynamic layer based on the identified location and the dynamic graph 820. In other words, the dynamic layer includes the elements of the graph 820 and further data linking the elements into the static layer at the identified location.

[0146] The system 800 then carries out a parameter expansion step 805. This comprises the generation of a concrete scenarios 205 from the logical scenario, by selecting particular parameter values. The parameter space may be sampled according to a sampling strategy.

[0147] The generated concrete scenarios are stored by the system 800 in a suitable data structure, such as a database 806.

[0148] Although Figure 8 illustrates that only the generated concrete scenarios are stored, in other examples the system 800 may store each generated graph 810 / 820 in a datastore. This may avoid the need to repeatedly generate graphs - e.g. every time a new static layer 201a is selected for matching a dynamic layer 201b against. Furthermore, although the process is illustrated as a single pipeline, it will be appreciated that in some examples the process may be divided into separate steps carried out at different times. For example, the system may first generate the graphs, then at some later point carry out the matching. After the matching, the system may then at some later point carry out the parameter expansion.

[0149] There now follows a more detailed discussion of the process of generating the static and dynamic graphs 810, 820 and the graph matching process 804.

[0150] Figure 9 is a highly schematic illustration of an example logical scenario 900 and associated graph 950.

[0151] The scenario 900 includes an ego agent 901 and a parked car 902 on a stretch of road 903. As discussed above, aspects of the logical scenario 900 may be specified by suitable parameters. For example, the position of the parked car 902 along the road 901 may be a parameter as indicated by arrow 904. Likewise, the amount by which the parked car extends into the road may be a parameter as indicated by arrow 905.

[0152] The graph 950 corresponds to this scenario 900. In particular, nodes 951-954 correspond to static elements of the scenario 900 (i.e. static graph 810), and nodes 956-957 correspond to dynamic elements (i.e. dynamic graph 820).

[0153] In more detail, nodes 951-954 each represent parts of the road 903. For example, each node 951-954 represents a particular lane section of the road. Edges 955 between the nodes 951-954 represent the relationship between lane sections, i.e. adjacency or that one section is subsequent to another (i.e. the “next” section). It will be appreciated that this is a highly simplified scenario and in practice a static layer 201a will include many more elements.

[0154] This part of the graph 950 is generated from the static layer 201a of the scenario description. Particularly, as discussed above in relation to Figure 5, the OpenDRIVE schemadefines elements of the road in a scenario in a hierarchical structure and defines the links between the elements. The OpenDRIVE definition of the scenario can therefore be parsed to extract the nodes 951-954 and edges 955.

[0155] Nodes 956-957 respectively correspond to the ego agent 901 and the parked car 903. Edge 958 defines the relationship between the nodes in the scenario (i.e. at the beginning of the scenario or a some other keyframe / synchronisation point), in this case that the ego agent 901 is ahead of the parked car 903. The edges 959 are the attachment edges, defining the relationship between the dynamic elements and the static elements. In this case, the edges 959 define that the ego 901 is centred on the lane of the road section represented by node 952 and the parked car 902 has an offset with respect to the road section represented by node 954.

[0156] The dynamic part of the graph is generated from the dynamic layer 201b. In one example, the OpenSCENARIO definition of the dynamic layer 201b is in an OpenSCENARIO VI. X format. This format defines the dynamic elements and the relationships therebetween in an XML format, and can be parsed in a similar manner to the static layer 201a to generate these parts of the graph 900.

[0157] In further examples, the definition of the dynamic layer 201b is in the OpenSCENARIO V2.X format. The OpenSCENARIO V2.X format is a domain specific language, providing the ability to define concrete, logical and also abstract scenarios. Abstract scenarios are formal, declarative descriptions of a scenario that conceptualises the intent of the scenario. Effectively, it is a level of abstraction beyond that of the logical scenario. The process in this circumstance is similar to that for OpenSCENARIO VI, in the sense that the definition can be readily parsed to determine the agents and their relationships, and a graph can be constructed therefrom.

[0158] Example OpenSCENARIO V2.X descriptions 1310, 1320 are illustrated in Figure 13. The descriptions 1310, 1320 are effectively a form of pseudocode. The first scenario definition 1310 is an example of a scenario with a concrete map. The definition 1310 includes statements 1311-1312 defining the ego 901 and parked car 903, and then a reference 1313 to a particular map file - “straight road two landes.xodr”). This is followed by a group of statements 1314 defining the action of the ego 901 with respect to the parked car 903. The second scenario 1320 has statements corresponding to statements 1311-1312 and 1314. However, instead of a concrete reference to a map, the second scenario 1320 includes statements 1323 requesting a map with a minimum of two lanes. Each description 1310,1320 sets out a definition of the dynamic layer 201b, from which the dynamic graph can be constructed.

[0159] Figure 10 schematically illustrates the relocation of the dynamic graph to a different part of the static graph. In this example, more of the static part of the graph is illustrated, with extra nodes and edges (generally indicated by reference numeral 960) illustrating the continuation of the road.

[0160] The system 800 performs a search of the static graph for locations where the attachment edges 959 can be connected to different static nodes. Various search strategies may be adopted to accomplish this. For example, a “brute-force” search could be carried out, in which the system 800 attempts to attach the dynamic graph 820 at every possible location of the static graph 810. In some examples, a subgraph may be extracted from the dynamic graph 820 that includes the attachment edges 959 and the nodes that they are attached to. The search task then involves finding that subgraph in the static layer. Of course, in the examples shown in the figures, the subgraph and the dynamic layer are the same, but in practice, the dynamic layer will typically be a larger graph.

[0161] The search may be constrained to attachment to other nodes of the same type - for example in this case other road tiles representing a left lane. As discussed above, as the scenario need not be generated “on demand”, relatively time-consuming techniques may be employed.

[0162] It will be appreciated that, although the schematic diagram of Figure 10 illustrates a trivial example in which two dynamic elements are attached to two consecutive static elements, in practice, more attachment edges may be present and need not be attached to consecutive static elements. Furthermore, it will be understood that the relocation of dynamic graph to a different static graph proceeds on substantially the same basis.

[0163] The system 800 can then iterate through the static layers 201a of the plurality of scenario descriptions in the database 801, generating new scenarios by combining the dynamic graph 820 with the static graphs 810 generated therefrom. It may be the case that some subset of the scenario descriptions 201 is selected, rather than iterating through the whole database 801.

[0164] Although in the discussion above, the problem is framed as searching through a plurality of static layers to find a suitable location for a dynamic layer, it will be understoodthat in some examples the opposite approach could be taken. That is to say, a single static layer could be selected, and the system may search through a plurality of dynamic layers in order to find a match.

[0165] Figure 11 illustrates a system 1100 for testing an AV. The system includes a database 806 storing concrete scenarios 830. The concrete scenarios 830 may have been generated according to the techniques discussed above with respect to Figures 8-10, but may equally have been generated by other techniques.

[0166] In order to facilitate search and sampling of the stored scenarios for AV testing, initial metadata describing the scenario 830 generated by the system 1100. For example, initial metadata is generated using a “lightweight” planner 841. This acts a means of bootstrapping the database 806 with an initial indication of events occurring in the scenario, by using a relatively unsophisticated AV stack that can rapidly execute the scenario 806 in a manner that uses less compute resource than a full AV planner.

[0167] The lightweight planner 841 may take the form of a simple “scenario runtime in control of the ego” simulation (i.e. where logic internal to the simulator 202 itself controls the ego, rather than the AV stack). In other examples, the lightweight planner 841 may a simple planner not forming part of the simulator 202 software, such as a non-real time planner that can execute the initial test run over a time period shorter than the real-world time would be for driving the scenario. In general, the lightweight planner 841 is a planner that is not undergoing performance testing and is being employed specifically for the purpose of generating the metadata.

[0168] In terms of the metadata, the test oracle 252 discussed above may analyse the trace output by the lightweight planner and generate the metadata. The metadata may include any of the performance metrics discussed above. In addition, the test oracle 252 may count the occurrence of pre-specified events in the trace. These may include for example the number of certain types of junctions or road layout encountered (e.g. number of T-junctions or roundabouts reached, the number of traffic lights encountered), whether a collision occurs, etc. They may also include certain manoeuvres carried out by the ego (e.g. numbers of overtakes), the appearance of other agents (e.g. number of bicycles encountered). Further examples include the number of driving rules broken in the run. Broadly, any metadata describing the scenario 830 that is useful in subsequent searching of and sampling from the database 806.

[0169] Once generated, the metadata is stored in the database 806 and associated with the scenario 830.

[0170] Subsequently, a first version 842 of a planner undergoing testing is tested using the scenario 830. For example, the scenario 830 may be retrieved as a result of a search as discussed in more detail below with respect to Figure 12. In a similar manner to the initial lightweight planner, the trace output by testing the first planner 841 is provided to the test oracle, which in turn generates metadata based on the first planner’s 841 run of the scenario 830. The metadata based on the first planner’s 841 scenario run is then stored in the database 806 in association with the scenario 830.

[0171] In some examples, the metadata replaces the currently stored metadata. This may be on the basis that the first planner 842 is likely to provide a more accurate assessment of what occurs in the scenario 830 than the initial bootstrapped metadata. However, it is primarily envisaged that the metadata from the first planner 842 is stored in addition to the existing metadata in the database 806. This may effectively form a history of runs of the scenario 830 and associated metadata. Of course, the output of the test oracle 252 for the first planner 842 is not merely employed for metadata generation but may also form the actual assessment of the performance of the first planner 842.

[0172] Following this, different planners (generally indicated by reference numeral 843) may likewise access the scenario 830, run the scenario 830 and then record metadata to the database 806. It is for example envisaged that various versions of a planner under development may over time access the scenarios 830 in the database 803, run the scenarios and update the metadata. This enables the usefulness of scenarios 830 to be tracked over time.

[0173] The flowchart in Figure 12 illustrates example steps in search and sampling techniques.

[0174] In a first step 1201, the user can provide user input to select the type of tests to run. For example, they may select a particular sampling plan, a weekly test, overnight testing, or pre-road release regression tests. Alternatively, the user may provide user input to select tests having particular features - e.g. roundabouts with many vehicles, busy T-junction with a traffic light, etc. The user may also provide an indication of the number of tests scenarios to be run. For example, the system 1100 may be configured to parse natural language queries(e.g. “busy T-junction with a traffic light, run 100k”) or receive input via other suitable interfaces (e.g. selection of elements on a form displayed to the user).

[0175] In a second step 1202, tests are retrieved from the database matching the user input. Particularly, the metadata stored in the database 806 provides a means of selecting tests that meet the criteria specified by the user. Depending on the type of tests selected by the user, the system 1100 may execute different searches. For example, overnight tests may run the same query each night but sample different scenarios from within the returned results each night. In such examples, it may be possible to track the parameter space coverage over time.

[0176] In addition, in a typical AV development process, it can be expected that there will not be large changes to the behaviour of the AV between overnight testing. Accordingly, by running a subset of all the scenarios overnight and sweeping the parameter space over the space of a week or two weeks, a larger amount of coverage instead of running the same scenarios and parameter values repeatedly.

[0177] In step S1203, the selected scenarios are executed, and the test oracle is used to assess the performance of the AV in the scenario.

[0178] In step 1204, the system 1100 carries out post-simulation analysis. This includes the above-described metadata update. Beyond updating the metadata to include the information derived from results from the test oracle for that scenario run, the post-simulation analysis may further include assessment or update of the existing metadata more generally. For example, it may become clear that a scenario no longer results in a particular event such as an overtake or roundabout entry because in several previous runs the event has not occurred. In such a case, it may be that previous metadata labelling the scenario as including the now irrelevant event is removed.

[0179] In addition, based on the database and the test oracle output, further metrics such as residual risk and ODD coverage may be estimated. A simple ODD coverage estimate may effectively be an estimate of what percentage of the scenarios available have been tested with this current version of the planner. More sophisticated measures may be based on event counters or rules usage. In other words, a coverage estimate may take into account how many times a particular event has occurred or how many rules have been broken during the testing of the stack.

[0180] Although the discussion herein refers to databases, it will be understood that these are merely examples of suitable data storage structures (i.e. datastores) for implementing the techniques discussed in. The disclosure is not in any sense limited to relational databases and includes diverse data storage structures, including NoSQL and non-relational databases.

[0181] In the discussion above, reference is made to nightly or weekly testing plans. It will be understood that these are examples to illustrate the concept of regular testing of a stack underdevelopment and are applicable to any suitable time period.

[0182] References herein to components, functions, modules and the like, denote functional components of a computer system which may be implemented at the hardware level in various ways. A computer system comprises execution hardware which may be configured to execute the method / algorithmic steps disclosed herein and / or to implement a model trained using the present techniques. The term execution hardware encompasses any form / combination of hardware configured to execute the relevant method / algorithmic steps. The execution hardware may take the form of one or more processors, which may be programmable or non-programmable, or a combination of programmable and nonprogrammable hardware may be used. Examples of suitable programmable processors include general purpose processors based on an instruction set architecture, such as CPUs, GPUs / accelerator processors etc. Such general-purpose processors typically execute computer readable instructions held in memory coupled to or internal to the processor and carry out the relevant steps in accordance with those instructions. Other forms of programmable processors include field programmable gate arrays (FPGAs) having a circuit configuration programmable through circuit description code. Examples of nonprogrammable processors include application specific integrated circuits (ASICs). Code, instructions etc. may be stored as appropriate on transitory or non-transitory media (examples of the latter including solid state, magnetic and optical storage device(s) and the like). The subsystems 102-108 of the runtime stack Figure 1 may be implemented in programmable or dedicated processor(s), or a combination of both, on-board a vehicle or in an off-board computer system in the context of testing and the like. The various components of Figure 2, such as the simulator 202 and the test oracle 252 may be similarly implemented in programmable and / or dedicated hardware.

Claims

Claims1. A computer-implemented method of testing an autonomous vehicle stack, comprising: storing a scenario datastore comprising a plurality of scenarios and first metadata describing each scenario, the first metadata having been generated by a first autonomous vehicle planner; retrieving a set of scenarios from the scenario datastore; for each retrieved scenario: executing a test run of the retrieved scenario using a second autonomous vehicle planner different from the first autonomous vehicle planner; generating second metadata describing the retrieved scenario based on the test run; and storing the second metadata in the scenario database.

2. The method of claim 1, wherein the first autonomous vehicle planner and second autonomous vehicle planner are different versions of an autonomous vehicle planner under development.

3. The method of claim 1 or 2, wherein the first autonomous vehicle planner is a non- real time planner.

4. The method of claim 1 or 2, wherein the first autonomous vehicle planner forms part of a scenario execution environment.

5. The method of any preceding claim, wherein the first and / or second metadata comprises a count of one or more prespecified events occurring in the test run.

6. The method of claim 5, wherein the first and / or second metadata includes metadata relating to: junction types or road layouts encountered; collisions;prespecified manoeuvres carried out; other agents encountered in the test run; and driving rules broken in the test run.

7. The method of any preceding claim, comprising: generating a trace of the test run, the trace representing a history of a location and motion of an agent over the course of the test run; generating, by a test oracle based on the trace, performance metrics reflective of the performance of the agent over the course of the test run; generating the second metadata from the performance metrics.

8. The method of claim 7, comprising storing the performance metrics in the database.

9. The method of any preceding claim, comprising: receiving a query to retrieve scenarios meeting at least one criterion; and retrieving the set of scenarios from the scenario database having first metadata matching the criterion.

10. The method of claim 9, comprising receiving user input specifying the query.

11. The method of any preceding claim, wherein retrieving the set of scenarios comprising: retrieving a plurality of scenarios from the scenario database; and sampling from the plurality of scenarios to generate the set of scenarios.

12. The method of any preceding claim, comprising: generating the first metadata by executing a test run of each of the plurality of scenarios, and storing the first metadata in the database.

13. The method of any preceding claim, comprising: determining a proportion of scenarios stored in the scenario database that have been executed by the second autonomous vehicle planner.

14. A computer system comprising one or more computers configured to implement the method of any preceding claim.

15. Computer-readable instructions stored in transitory or non-transitory media and configured, when executed in a computer system, to implement the method of any of claims 1 to 13.

Images