Defining a scenario to be run in a simulation environment for testing the behaviour of a runtime stack of a sensor-equipped robot

Abstract

A computer-implemented method of defining a scenario to be run in a simulation environment for testing the behaviour of a runtime stack of a sensor-equipped robot is disclosed The method comprises rendering a code editing interface on a display of a computer device, the code editing interface configured to receive an abstract scenario definition in a scenario definition language, the abstract scenario definition defining a scenario to be run in a first simulation environment for testing the behaviour of the runtime stack of the sensor-equipped robot having a role of an ego agent in the scenario; generating a concrete scenario description from the abstract scenario definition and a selected physical context compatible with the abstract scenario definition; providing the concrete scenario description to a second simulation environment; executing the concrete scenario description in the second simulation environment using a lightweight planner not forming part of the runtime stack to act as the ego agent; receiving from the second simulation environment an ego trace representative of actions of the ego agent during execution of the concrete scenario description; rendering, on the display of the computer device, a visualisation interface displaying a visual representation of the selected physical context; and rendering the actions of the ego agent on the visual representation of the selected physical context.

Description

[0001] Defining a scenario to be run in a simulation environment for testing the behaviour of a runtime stack of a sensor-equipped robot

[0002] Technical Field

[0003] [1] The present application relates to a computer-implemented method of defining a scenario to be run in a simulation environment for testing the behaviour of a runtime stack of a sensor-equipped robot.

[0004] Background

[0005] [2] There have been major and rapid developments in the field of autonomous vehicles. An autonomous vehicle (AV) is a vehicle which is equipped with sensors and control systems which enable it to operate without a human controlling its behaviour. An autonomous vehicle is equipped with sensors which enable it to perceive its physical environment, such sensors include for example cameras, radar and lidar. Autonomous vehicles are equipped with suitably programmed computers which are capable of processing data received from the sensors and making safe and predictable decisions based on the context which has been perceived by the sensors.

[0006] [3] An autonomous vehicle may be fully autonomous (in that it is designed to operate with no human supervision or intervention, at least in certain circumstances) or semi- autonomous. Semi-autonomous systems require varying levels of human oversight and intervention. An Advanced Driver Assist System (ADAS) and certain levels of Autonomous Driving System (ADS) may be classed as semi-autonomous. A “level 5” vehicle is one that can operate entirely autonomously in any circumstances, because it is always guaranteed to meet some minimum level of safety. Such a vehicle would not require manual controls (steering wheel, pedals etc.) at all. By contrast, level 3 and level 4 vehicles can operate fully autonomously but only within certain defined circumstances (e.g. within geofenced areas). A level 3 vehicle must be equipped to autonomously handle any situation that requires an immediate response (such as emergency braking); however, a change in circumstances may trigger a “transition demand”, requiring a driver to take control of the vehicle within some limited timeframe. A level 4 vehicle has similar limitations; however, in the event the driver does not respond within the required timeframe, a level 4 vehicle must also be capable of autonomously implementing a “minimum risk maneuver” (MRM), i.e. some appropriate action(s) to bring the vehicle to safe conditions (e.g. slowing down and parking the vehicle). A level 2 vehicle requires the driver to be ready to intervene at any time, and it is the responsibility of the driver to intervene if the autonomous systems fail to respond properly at any time. With level 2 automation, it is the responsibility of the driver to determine when their intervention is required; for level 3 and level 4, this responsibility shifts to the vehicle’s autonomous systems and it is the vehicle that must alert the driver when intervention is required.

[0007] [4] Other mobile robots are being developed, for example for carrying freight supplies in internal and external industrial zones. Such mobile robots would have no people on board and belong to a class of mobile robot termed UAV (unmanned autonomous vehicle). Autonomous air mobile robots (drones) are also being developed.

[0008] [5] Physical world testing will remain an important factor in the testing of autonomous vehicles’ capability to make safe and predictable decisions. However, physical world testing is expensive and time-consuming. Moreover, as the safety of AV systems improves, the number of incidents per mile driven decreases. To achieve widespread acceptance of AVs, it is estimated that an AV should cause no more than 1 severe accident per 10A9 miles driven (compared with 1 such accident per 10A6 miles for the average human driver). Identifying and mitigating issues based solely on ‘miles driven’ in the real-world is simply not viable for an AV operating even several orders of magnitude below the minimum performance level that will ultimately be required.

[0009] [6] As such, increasing reliance is placed on testing using simulated environments. If there is to be an increase in testing in simulated environments, it is desirable that such environments can reflect as far as possible real-world scenarios. Autonomous vehicles need to have the facility to operate in the same wide variety of circumstances that a human driver can operate in. Such circumstances can incorporate a high level of unpredictability. Increasing attention is being placed on the creation of simulation environments which can provide such testing in a manner that gives confidence that the test outcomes represent potential real behaviour of an autonomous vehicle.

[0010] [7] Scenarios must be described with sufficient detail and precision in a form conducive to high-quality simulation, otherwise the results of testing will not sufficiently reflect real- world AV performance. A typical driving scenario includes a static road layout and various dynamic agents (other vehicles, pedestrians, cyclists, animals etc.) that an autonomous vehicle (the ego vehicle) is required to navigate. An ego vehicle may be required to predict the motion of other agents and plan safely within a complex road network. In an offline context, a scenario description may be required as an input to a simulator to facilitate simulation-based testing of an autonomous vehicle stack prior to deployment on a real -world vehicle.

[0011] [8] There is a need to formally define scenarios in a way that can be readily executed in a simulator. For example, a formal language such as OpenSCENARIO 2.0 (recently renamed OpenSCENARIO DSL) can be used to define scenarios. It will be understood that all references herein to OpenSCENARIO 2.0 apply correspondingly to OpenSCENARIO DSL. The OpenSCENARIO 2.0 language can be used to define the dynamic content of scenarios at varying levels of abstraction. More abstract scenarios are “concretized” for simulation testing, for example by selecting values for parameterized variables associated with the scenario (e.g. distances, speeds etc) and selecting a particular road layout on which the scenario runs. When defining a scenario at a high level of abstraction, it can be difficult for the scenario author to appreciate from reading the scenario definition how it will actually execute in simulation.

[0012] SUMMARY

[0013] [9] A first aspect of the present disclosure provides a computer-implemented method of defining a scenario to be run in a simulation environment for testing the behaviour of a runtime stack of a sensor-equipped robot, the method comprising: rendering a code editing interface on a display of a computer device, the code editing interface configured to receive an abstract scenario definition in a scenario definition language, the abstract scenario definition defining a scenario to be run in a first simulation environment for testing the behaviour of the runtime stack of the sensor-equipped robot having a role of an ego agent in the scenario; generating a concrete scenario description from the abstract scenario definition and a selected physical context compatible with the abstract scenario definition; providing the concrete scenario description to a second simulation environment; executing the concrete scenario description in the second simulation environment using a lightweight planner not forming part of the runtime stack to act as the ego agent; receiving from the second simulation environment an ego trace representative of actions of the ego agent during execution of the concrete scenario description; rendering, on the display of the computer device, a visualisation interface displaying a visual representation of the selected physical context; and rendering the actions of the ego agent on the visual representation of the selected physical context.

[0014]

[0010] In embodiments, the scenario definition language may be OpenSCENARIO.

[0015]

[0011] In embodiments, the method may comprise receiving a challenger agent trace from the second simulation environment representative of the actions of a challenger agent. Additionally, the method may comprise rendering the actions of the challenger agent on the visual representation of the physical context.

[0016]

[0012] In embodiments, the method may comprise rendering, on the display, a user input element and, in response to user input at the user input element, generating the concrete scenario description and providing the concrete scenario description to the second simulation environment.

[0017]

[0013] In embodiments, the lightweight planner may be a rules-based planner.

[0018]

[0014] In embodiments, the lightweight planner may be a non-real-time planner.

[0019]

[0015] In embodiments, the lightweight planner may form part of the second simulation environment.

[0020]

[0016] In embodiments, the method may comprise determining a plurality of physical contexts compatible with the abstract scenario definition. Additionally, the method may comprise selecting the physical context from the plurality of physical contexts.

[0021]

[0017] In embodiments, the method may comprise receiving user input selecting the physical context compatible with the abstract scenario definition.

[0022]

[0018] In embodiments, generating the concrete scenario description may comprise selecting values for one or more parameters defined in the abstract scenario definition.

[0023]

[0019] In embodiments, the method may comprise: receiving, via the code editing interface, user input editing the abstract scenario definition to provide a second abstract scenario definition. The method may additionally comprise generating a second concrete scenario description from the second abstract scenario definition and a selected physical context compatible with the second abstract scenario definition and providing the second concrete scenario description to the second simulation environment. The method may comprise executing the second concrete scenario description in the second simulation environment using the lightweight planner, receiving from the second simulation environment a second ego trace representative of actions of the ego agent during execution of the second concrete scenario description and rendering the actions of the ego agent corresponding to the second ego trace on the visual representation of the selected physical context.

[0024]

[0020] In embodiments, the method may comprise rendering, on the visualization interface, playback controls. Additionally, the method may comprise rendering the action of the ego agent in response to user input at the playback controls.

[0025]

[0021] In embodiments, the method may comprise concurrently displaying both the code editing interface and the visualization interface.

[0026]

[0022] A second aspect of the present disclosure provides a computer system for defining a scenario to be run in a simulation environment for testing the behaviour of a runtime stack of a sensor-equipped robot, the system comprising: a display; a memory; at least one processor configured to execute instructions stored in the memory to cause the system to: render a code editing interface on the display, the code editing interface configured to receive an abstract scenario definition in a scenario definition language, the abstract scenario definition defining a scenario to be run in a first simulation environment for testing the behaviour of the runtime stack of the sensor-equipped robot having a role of an ego agent in the scenario; generate a concrete scenario description from the abstract scenario definition and a selected physical context compatible with the abstract scenario definition; provide the concrete scenario description to a second simulation environment; execute the concrete scenario description in the second simulation environment using a lightweight planner not forming part of the runtime stack to act as the ego agent; receive from the second simulation environment an ego trace representative of actions of the ego agent during execution of the concrete scenario description; render, on the display, a visualisation interface displaying a visual representation of the selected physical context; and render the actions of the ego agent on the visual representation of the selected physical context.

[0027]

[0023] Further optional features of the second aspect are as defined above in relation to the first aspect and may be combined in any combination.

[0028]

[0024] According to a third aspect, there is provided a tangible, non-transient computer readable medium storing instructions, which when executed by a computer device, cause the computer device to perform the steps of the method of the first aspect.

[0029]

[0025] Further optional features of the third aspect are as defined above in relation to the first and second aspects and may be combined in any combination.

[0030]

[0026] According to a fourth aspect, there is provided a tangible, non-transient computer readable medium storing instructions, which when executed by a computer device, cause the computer device to implement the system of the second aspect.

[0031]

[0027] Further optional features of the fourth aspect are as defined above in relation to the first, second and third aspects and may be combined in any combination.

[0032]

[0028] A problem addressed herein is to check that the abstract driving definitions, upon execution of the scenario, correspond to the intention of the author. Since OpenSCENARIO 2.0 supports defining scenarios in abstract terms, it is difficult to tell whether the scenario will function as expected in simulation. In other words, it is difficult to tell from the scenario definition in code whether a scenario that performs the intended sequence of events will be executed. Rapid visualization of the scenario assists the user in quickly understanding the behaviour of the agents in various concretised versions of the abstract scenario. This in turn ensures that only ‘good’ scenarios are considered for simulation with an AV stack.

[0033]

[0029] Consequently this creates a fast feedback loop environment which facilitates the generation of functional new and modified scenarios in this format.

[0034] Brief Description Of The Drawings

[0035]

[0030] To assist understanding of the present disclosure and to show how embodiments may be put into effect, reference is made by way of example to the accompanying drawings in which:

[0036]

[0031] FIG. 1 A shows a schematic function block diagram of an autonomous vehicle stack;

[0037]

[0032] FIG. IB shows a schematic overview of an autonomous vehicle testing paradigm;

[0038]

[0033] FIG.1C shows a schematic block diagram of a scenario extraction pipeline;

[0039]

[0034] FIG.2 shows a schematic block diagram of a testing pipeline;

[0040]

[0035] FIG.3 A shows an example of a scenario definition in a domain specific language.

[0041]

[0036] FIG. 3B shows example abstract scenario definitions, with different levels of abstraction;

[0037] FIG.4A shows a schematic block diagram of a system for defining a scenario;

[0042]

[0038] FIG.4B shows a schematic block diagram of an example simulation environment;

[0043]

[0039] FIG.5 shows an example road network;

[0044]

[0040] FIG.6 shows an example graphical user interface.

[0045]

[0041] In the drawings, corresponding reference characters indicate corresponding components. The skilled person will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. Also, common but well-understood elements that are useful or necessary in a commercially feasible embodiment are often not depicted in order to facilitate a less obstructed view of these various example embodiments.

[0046]

[0042] It will be appreciated that where reference numerals of the form xxx-a, xxx-b etc., are used to denote a particular instance of a feature of a drawing, the same reference numeral without a specified letter suffix (e.g., a, b, etc.) may denote a generic instance of the same feature.

[0047] Detailed Description

[0048]

[0043] The present disclosure provides a computer-implemented method of defining a scenario to be run in a simulation environment for testing the behaviour of a runtime stack of a sensor-equipped robot. A code editing interface receives an abstract scenario definition. The scenario definition is concretised and then executed using a lightweight planner, and the output of the simulation environment is visualised on a graphical user interface.

[0049]

[0044] The visualisation of the scenario definition in a rapid manner using a lightweight planner allows a user to determine whether the scenario definition executes in simulation as intended by the user. In other words, it provides a tool to determine whether the agents and events defined in the scenario definition behave as expected in simulation.

[0050]

[0045] To provide context for the discussion of the examples of the disclosure, a more general discussion of simulation testing methods for a full AV stack will be provided first, with particular reference to logical and concrete scenarios.

[0046] Test scenarios are considered in which an ego agent is required to navigate a real or modelled physical context. The ego agent is a real or simulated mobile robot that moves under the control of the stack under testing. The physical context includes static and / or dynamic element(s) that the stack under testing is required to respond to effectively. For example, the mobile robot may be a fully or semi-autonomous vehicle under the control of the stack (the ego vehicle). The physical context may comprise a static road layout and a given set of environmental conditions (e.g. weather, time of day, lighting conditions, humidity, pollution / particulate level etc.) that could be maintained or varied as the scenario progresses. A dynamic scenario additionally includes one or more other agents (“external” agent(s), e.g. other vehicles, pedestrians, cyclists, animals etc.).

[0051]

[0047] In an offline simulation context, a scenario description is provided to an offline simulator as input, in order to expose a stack under testing to a simulated scenario, and test its performance in the simulated scenario. This allows performance issues (such as safety issues) to be identified and mitigated prior to real-world deployment.

[0052]

[0048] Herein, the scenario description is a form of machine-readable scenario definition or narrative. The term “description” is used to distinguish the scenario description actually consumed by the simulator from the more abstract scenario definitions discussed below with respect to FIGs 3 A-6.

[0053]

[0049] A simulated scenario requires an ego agent to navigate a modelled physical context. The ego agent is a simulated mobile robot that moves under the control of the stack under testing. The physical context includes static and / or dynamic element(s) that the stack under testing is required to respond to effectively. For example, the mobile robot may be a fully or semi-autonomous vehicle under the control of the stack (the ego vehicle). The physical context may comprise a static road layout and a given set of environmental conditions (e.g. weather, time of day, lighting conditions, humidity, pollution / particulate level etc.) that could be maintained or varied as the scenario progresses.

[0054]

[0050] Firstly, FIG. 1 A shows, by way of context, a highly schematic block diagram of an AV runtime stack 100. The stack 100 is an example of a robotic system as discussed herein. The stack 100 may be fully or semi-autonomous. For example, the stack 100 may operate as an Autonomous Driving System (ADS) or Advanced Driver Assist System (ADAS).

[0051] The run time stack 100 is shown to comprise a perception system 102, a prediction system 104, a planning system (planner) 106 and a control system (controller) 108.

[0055]

[0052] In a real -world context, the perception system 102 receives sensor outputs from an onboard sensor system 110 of the AV, and uses those sensor outputs to detect external agents and measure their physical state, such as their position, velocity, acceleration etc. The onboard sensor system 110 can take different forms but generally comprises a variety of sensors such as image capture devices (cameras / optical sensors), lidar and / or radar unit(s), satellite-positioning sensor(s) (GPS etc.), motion / inertial sensor(s) (accelerometers, gyroscopes etc.) etc. The onboard sensor system 110 thus provides rich sensor data from which it is possible to extract detailed information about the surrounding environment, and the state of the AV and any external actors (vehicles, pedestrians, cyclists etc.) within that environment. The sensor outputs typically comprise sensor data of multiple sensor modalities such as stereo images from one or more stereo optical sensors, lidar, radar etc. Sensor data of multiple sensor modalities may be combined using filters, fusion components etc.

[0056]

[0053] The perception system 102 typically comprises multiple perception components which co-operate to interpret the sensor outputs and thereby provide perception outputs to the prediction system 104.

[0057]

[0054] In a simulation context, depending on the nature of the testing, it may or may not be necessary to model the on-board sensor system 100. For example, when only the planner 106 (or only the planner 106 and controller 108) are tested on a ‘perfect’ representation of the scenario (that is, directly on simulator ground truth), simulated sensor data is not required therefore complex sensor modelling is not required. Surrogate model(s) of the perception system 102 (or part(s) of it) can also be used to test planner performance (or planner and controller performance) in the presence of realistic perception errors, without the use of sensor models.

[0058]

[0055] The perception outputs from the perception system 102 are used by the prediction system 104 to predict future behaviour of external actors (agents), such as other vehicles in the vicinity of the AV.

[0056] Predictions computed by the prediction system 104 are provided to the planner 106, which uses the predictions to make autonomous driving decisions to be executed by the AV in a given driving scenario. The inputs received by the planner 106 would typically indicate a drivable area and would also capture predicted movements of any external agents (obstacles, from the AV’s perspective) within the drivable area. The driveable area can be determined using perception outputs from the perception system 102 in combination with map information, such as an HD (high definition) map.

[0059]

[0057] A core function of the planner 106 is the planning of trajectories for the AV (ego trajectories), taking into account predicted agent motion. This may be referred to as trajectory planning. A trajectory is planned in order to carry out a desired goal within a scenario. The goal could for example be to enter a roundabout and leave it at a desired exit; to overtake a vehicle in front; or to stay in a current lane at a target speed (lane following). The goal may, for example, be determined by an autonomous route planner 115, also referred to as a goal generator 115.

[0060]

[0058] The controller 108 executes the decisions taken by the planner 106 by providing suitable control signals to an on-board actor system 112 of the AV. In particular, the planner 106 plans trajectories for the AV and the controller 108 generates control signals to implement the planned trajectories. Typically, the planner 106 will plan into the future, such that a planned trajectory may only be partially implemented at the control level before a new trajectory is planned by the planner 106. The actor system 112 includes “primary” vehicle systems, such as braking, acceleration and steering systems, as well as secondary systems (e.g. signalling, wipers, headlights etc.).

[0061]

[0059] The example of FIG. 1A considers a relatively “modular” architecture, with separable perception, prediction, planning and control systems 102-108. The sub-stack themselves may also be modular, e.g. with separable planning modules within the planning system 106. For example, the planning system 106 may comprise multiple trajectory planning modules that can be applied in different physical contexts (e.g. simple lane driving vs. complex junctions or roundabouts). This is relevant to simulation testing for the reasons noted above, as it allows components (such as the planning system 106 or individual planning modules thereof) to be tested individually or in different combinations. For the avoidance of doubt, with modular stack architectures, the term stack can refer not only to the full stack but to any individual sub-system or module thereof.

[0062]

[0060] The extent to which the various stack functions are integrated or separable can vary significantly between different stack implementations - in some stacks, certain aspects may be so tightly coupled as to be indistinguishable. For example, in other stacks, planning and control may be integrated (e.g. such stacks could plan in terms of control signals directly), whereas other stacks (such as that depicted in FIG. 1A) may be architected in a way that draws a clear distinction between the two (e.g. with planning in terms of trajectories, and with separate control optimizations to determine how best to execute a planned trajectory at the control signal level). Similarly, in some stacks, prediction and planning may be more tightly coupled. At the extreme, in so-called “end-to-end” driving, perception, prediction, planning and control may be essentially inseparable. Unless otherwise indicated, the perception, prediction planning and control terminology used herein does not imply any particular coupling or modularity of those aspects.

[0063]

[0061] A “full” stack typically involves everything from processing and interpretation of low- level sensor data (perception), feeding into primary higher-level functions such as prediction and planning, as well as control logic to generate suitable control signals to implement planning-level decisions (e.g. to control braking, steering, acceleration etc.). For autonomous vehicles, level 3 stacks include some logic to implement transition demands and level 4 stacks additionally include some logic for implementing minimum risk maneuvers. The stack may also implement secondary control functions e.g. of signalling, headlights, windscreen wipers etc.

[0064]

[0062] Whilst the following description refers to the stack 100 in the context of testing, testing may be applied to individual components / portions of the stack, such as the perception, prediction, planning or control stacks 104, 106, 108 (alone or in various combinations), or individual component(s) thereof. A stack (or component) can refer purely to software, i.e. one or more computer programs that can be executed on one or more general-purpose computer processors. However, such terminology can also encompass hardware. In simulation, software of the stack may be tested on a “generic” off-board computer system, before it is eventually uploaded to an on-board computer system of a physical vehicle. However, in “hardware-in-the-loop” testing, the testing may extend to underlying hardware of the vehicle itself. For example, the stack software may be run on the on-board computer system (or a replica thereof) that is coupled to the simulator for the purpose of testing. In this context, the stack under testing extends to the underlying computer hardware of the vehicle. As another example, certain functions of the stack 110 (e.g. perception functions) may be implemented in dedicated hardware. In a simulation context, hardware-in-the loop testing could involve feeding synthetic sensor data to dedicated hardware perception components.

[0065]

[0063] Within the stack 100, a scenario description 116 may be used as a basis for planning and prediction. Note that this is an internal scenario description held by the stack 100, separate to and different from the scenario descriptions authored by a user discussed below. The scenario description 116 is generated using the perception system 102, together with a high-definition (HD) map 114. By localizing the ego vehicle on the map 114, it is possible to combine the information extracted in the perception system 102 (including dynamic agent information) with the pre-existing environmental information contained in the HD map 114. The scenario description 116 is, in turn, used as a basis for motion prediction in the prediction system 104, and the resulting motion predictions 118 are used in combination with the scenario description 116 as a basis for planning in the planning system 106.

[0066]

[0064] FIG. IB shows a highly schematic overview of a testing paradigm for autonomous vehicles. An ADS / ADAS stack 100, e.g. of the kind depicted in FIG. 1A, is subject to repeated testing and evaluation in simulation, by running multiple scenario instances in a simulator 202, and evaluating the performance of the stack 100 (and / or individual subs- stacks thereof) in a test oracle 252. The output of the test oracle 252 is informative to an expert 122 (team or individual), allowing them to identify issues in the stack 100 and modify the stack 100 to mitigate those issues (S124). The results also assist the expert 122 in selecting further scenarios for testing (SI 26), and the process continues, repeatedly modifying, testing and evaluating the performance of the stack 100 in simulation. The improved stack 100 is eventually incorporated (S125) in a real-world AV 101, equipped with a sensor system 110 and an actor system 112. The improved stack 120 typically includes program instructions (software) executed in one or more computer processors of an on-board computer system of the vehicle 101 (not shown). The software of the improved stack 120 is uploaded to the AV 101 at step S125. Step S125 may also involve modifications to the underlying vehicle hardware. On board the AV 101, the improved stack 120 receives sensor data from the sensor system 110 and outputs control signals to the actor system 112. Real -world testing (SI 28) can be used in combination with simulation-based testing. For example, having reached an acceptable level of performance through the process of simulation testing and stack refinement, appropriate real-world scenarios may be selected (SI 30), and the performance of the AV 101 in those real scenarios may be captured and similarly evaluated in the test oracle 252.

[0067]

[0065] FIG. 1C shows a highly schematic block diagram of the testing pipeline applied to a scenario description 148. A scenario generator 146 generates the scenario description 148 in a form that may be consumed by the simulator 202, allowing multiple simulated runs to be derived therefrom (for example, with different configurations of the AV stack 100). The scenario description 148 of FIG. 1C may represent a concrete scenario suitable for execution by the simulator, as discussed in more detail below. Ground truth 150 is provided for each simulated run.

[0068]

[0066] The ground truth 150 for each run comprises a trace of the ego agent and any other dynamic agents in the scenario. A trace is a history of an agent’s location and motion over the course of a scenario. There are many ways a trace can be represented. Trace data will typically include spatial and motion data of an agent within the environment.

[0069]

[0067] Further details of an example testing pipeline incorporating the test oracle 252 will now be described.

[0070]

[0068] FIG. 2 shows a schematic block diagram of the testing pipeline, denoted by reference numeral 200. The testing pipeline 200 is shown to comprise the simulator 202 and the test oracle 252. The simulator 202 runs simulated scenarios for the purpose of testing all or part of an AV run time stack 100, and the test oracle 252 evaluates the performance of the stack (or sub-stack) on the simulated scenarios. As discussed, it may be that only a sub-stack of the run-time stack is tested, but for simplicity, the following description refers to the (full) AV stack 100 throughout. However, the description applies equally to a sub-stack in place of the full stack 100. The term “slicing” is used herein to the selection of a set or subset of stack components for testing.

[0071]

[0069] The idea of simulation-based testing is to run a simulated driving scenario that an ego agent must navigate under the control of the stack 100 being tested. Typically, the scenario includes a static drivable area (e.g. a particular static road layout) that the ego agent is required to navigate, typically in the presence of one or more other dynamic agents (such as other vehicles, bicycles, pedestrians etc.). To this end, simulated inputs 203 are provided from the simulator 202 to the stack 100 under testing.

[0072]

[0070] The slicing of the stack dictates the form of the simulated inputs 203. By way of example, FIG. 2 shows the prediction, planning and control systems 104, 106 and 108 within the AV stack 100 being tested. To test the full AV stack of FIG. 1 A, the perception system 102 could also be applied during testing. In this case, the simulated inputs 203 would comprise synthetic sensor data that is generated using appropriate sensor model(s) and processed within the perception system 102 in the same way as real sensor data. This requires the generation of sufficiently realistic synthetic sensor inputs (such as photorealistic image data and / or equally realistic simulated lidar / radar data etc.). The resulting outputs of the perception system 102 would, in turn, feed into the higher-level prediction and planning systems 104, 106.

[0073]

[0071] By contrast, so-called “planning-level” simulation would essentially bypass the perception system 102. The simulator 202 would instead provide simpler, higher-level inputs 203 directly to the prediction system 104. In some contexts, it may even be appropriate to bypass the prediction system 104 as well, in order to test the planner 106 on predictions obtained directly from the simulated scenario (i.e. “perfect” predictions).

[0074]

[0072] Between these extremes, there is scope for many different levels of input slicing, e.g. testing only a subset of the perception system 102, such as “later” (higher-level) perception components, e.g. components such as filters or fusion components which operate on the outputs from lower-level perception components (such as object detectors, bounding box detectors, motion detectors etc.).

[0075]

[0073] As an alternative to generating high-fidelity synthetic sensor data that is fed to the perception system 102, all or part of the perception system 102 may be modelled using one or more “surrogate” perception models. A surrogate model models the perception system 102 (or some component / components thereof), but operates more efficiently on lower- fidelity inputs derived from the simulation ground truth, without the need for sensor models. A surrogate model allows the simulated inputs 203 to be generated with realistic error (comparable to the error that would be produced by the perception system 102 in a comparable real-world situation) without the use of sensor models.

[0074] Whatever form they take, the simulated inputs 203 are used (directly or indirectly) as a basis for decision-making by the plannerl06. The controller 108, in turn, implements the planner’s decisions by outputting control signals 109. In a real -world context, these control signals would drive the physical actor system 112 of AV. In simulation, an ego vehicle dynamics model 204 is used to translate the resulting control signals 109 into realistic motion of the ego agent within the simulation, thereby simulating the physical response of an autonomous vehicle to the control signals 109.

[0076]

[0075] Alternatively, a simpler form of simulation assumes that the ego agent follows each planned trajectory exactly between planning steps. This approach bypasses the control system 108 (to the extent it is separable from planning) and removes the need for the ego vehicle dynamic model 204. This may be sufficient for testing certain facets of planning.

[0077]

[0076] To the extent that external agents exhibit autonomous behaviour / decision making within the simulator 202, some form of agent decision logic 210 is implemented to carry out those decisions and determine agent behaviour within the scenario. The agent decision logic 210 may be comparable in complexity to the ego stack 100 itself or it may have a more limited decision-making capability. The aim is to provide sufficiently realistic external agent behaviour within the simulator 202 to be able to usefully test the decision-making capabilities of the ego stack 100. In some contexts, this does not require any agent decision making logic 210 at all (open-loop simulation), and in other contexts useful testing can be provided using relatively limited agent logic 210 such as basic adaptive cruise control (ACC). One or more agent dynamics models 206 may be used to provide more realistic agent behaviour if appropriate.

[0078]

[0077] A scenario is run in accordance with a scenario description 201, which typically has both static and dynamic elements. Scenario runs are orchestrated by a test orchestration component 260.

[0079]

[0078] The static elements are encoded in a static layer 201a of the scenario description 201. The static element(s) typically include a road layout. For example, the static layer 201a may be encoded in the OpenDrive format, or a comparable road network scenario description language (SDL) that describes both the geometry and the topology of the road layout.

[0079] The dynamic element(s) are encoded in a dynamic layer 201b, and typically include one or more external agents within the scenario, such as other vehicles, pedestrians, bicycles etc.

[0080]

[0080] In the context of AV stack testing of this figure, the scenario description 201 is a logical scenario definition. That is to say, the description 201 includes a plurality of parameters. Example parameters may include weather or lighting parameters reflective of the time of day / level of illumination / weather conditions etc. Example parameters may also include the speeds of agents within the scenario.

[0081]

[0081] Before input to the simulator 202, a concrete scenario 205 is instantiated from the logical scenario definition, by selecting particular values for the parameters.

[0082]

[0082] The extent of the dynamic information provided to the simulator 202 for each external agent can vary. A given static layer (e.g. defining a road layout) can be used in combination with different dynamic layers to provide different scenario instances. The dynamic layer may comprise, for each external agent, a spatial path to be followed by the agent together with one or both of motion data and behaviour data associated with the path. In simple open-loop simulation, an external actor simply follows the spatial path and motion data defined in the dynamic layer that is non-reactive i.e. does not react to the ego agent within the simulation. Such open-loop simulation can be implemented without any agent decision logic 210. However, in closed-loop simulation, the dynamic layer instead defines at least one behaviour to be followed along a static path (such as an ACC behaviour). In this case, the agent decision logic 210 implements that behaviour within the simulation in a reactive manner, i.e. reactive to the ego agent and / or other external agent(s). Motion data may still be associated with the static path but in this case is less prescriptive and may for example serve as a target along the path. For example, with an ACC behaviour, target speeds may be set along the path which the agent will seek to match, but the agent decision logic 210 might be permitted to reduce the speed of the external agent below the target at any point along the path in order to maintain a target headway from a forward vehicle.

[0083]

[0083] The output of the simulator 202 for a given simulation includes an ego trace 212a of the ego agent and one or more agent traces 212b of the one or more external agents (traces 212). Each trace 212a, 212b is a complete history of an agent’s behaviour within a simulation having both spatial and motion components. For example, each trace 212a, 212b may take the form of a spatial path having motion data associated with points along the path such as speed, acceleration, jerk (rate of change of acceleration), snap (rate of change of jerk) etc.

[0084]

[0084] Additional information is also provided to supplement and provide context to the traces 212. Such additional information is referred to as “contextual” data 214. The contextual data 214 pertains to the physical context of the scenario, and can have both static components (such as road layout) and dynamic components (such as weather conditions to the extent they vary over the course of the simulation).

[0085]

[0085] The test oracle 252 receives the traces 212 and the contextual data 214, and scores those outputs in respect of a set of performance evaluation rules 254. The performance evaluation rules 254 are shown to be provided as an input to the test oracle 252.

[0086]

[0086] The rules 254 are categorical in nature (e.g. pass / fail-type rules). Certain performance evaluation rules are also associated with numerical performance metrics used to “score” trajectories (e.g. indicating a degree of success or failure or some other quantity that helps explain or is otherwise relevant to the categorical results). The evaluation of the rules 254 is time-based - a given rule may have a different outcome at different points in the scenario. The scoring is also time-based: for each performance evaluation metric, the test oracle 252 tracks how the value of that metric (the ‘robustness’ score) changes over time as the simulation progresses. The test oracle 252 provides an output 256 comprising a time sequence 256a of categorical (e.g. pass / fail) results for each rule, and a score-time plot 256b for each performance metric, as described in further detail later. The results and scores 256a, 256b are informative to the expert 122 and can be used to identify and mitigate performance issues within the tested stack 100. The test oracle 252 also provides an overall (aggregate) result for the scenario (e.g. overall pass / fail). The output 256 of the test oracle 252 is stored in a test database 258, in association with information about the scenario to which the output 256 pertains.

[0087]

[0087] Next, an example definition of a road network that forms an example static layer 201a is schematically shown in FIG. 5. The following description assumes the road network 500 is described using the OpenDRIVE schema, or a similar format that adopts certain definitions and conventions from OpenDRIVE. However, it will be appreciated that the principles extend more generally to other formats, and the described techniques are not limited to any particular data format or schema.

[0088]

[0088] The road network 500 is shown to comprise first, second, third and fourth roads 502, 504, 506, 508 (Roads 1 to 4), which are described with <road> elements having road identifiers (IDs) 1, 2, 3 and 4 respectively. The roads 502-508 are interconnected via a junction 510 described by a <junction> element. Each of the roads 502-508 is defined by a single road reference line, denoted as a thick solid arrow, and contains a single center lane of width zero. The center lanes are not depicted separately, and for simplicity it is assumed that the center lane of each road 502-508 lies along the road reference line (although, as noted, it is possible to define a non-zero offset between the road reference line and the center lane). The road reference lines of the first and second roads 502, 504 are denoted by reference numerals 503 and 505 respectively. A road reference line is directional, and could be described more precisely as a longitudinal axis or “s-axis” of the road, with s-coordinates running along that axis, and t-coordinates running orthogonal to it. As depicted for the reference lines 503, 505, the positive t-direction is defined as extending to the left of the s- axis. Lanes are numbered relative to the center lane; the center lane is always numbered 0, with side-lanes to the left of the center lane (+t) assigned incrementing positive lane numbers (1, 2, 3, . . .) and lanes to the right (-t) assigned decreasing negative lane numbers (-1, -2, -3, ...).

[0089]

[0089] Each road has a minimum of one side-lane. In this example, the first road 502 has only positively numbered side-lanes to the left of the center lane, whereas the second road 504 has two positive side-lanes to the left of the center lane, and one negative side-lane to the right. Roads may be divided into “lane sections” to accommodate sections of the road with different numbers of lanes (see below).

[0090]

[0090] In the following description, “Lane n” means a lane with lane number “n” and “Road m” means a road with road ID “m”. Other road elements (such as junctions) are referred to in the same way.

[0091]

[0091] Adjacent lane sections may be referred to as “Lane Section n” and “Lane Section n+1” (the lane section numbers n, n+1 does not form part of the OpenDRIVE schema, but are nevertheless derivable from the static layer 201a).

[0092] A left-hand traffic (LHT) road system is depicted in this example. However, the schema can be used for either LHT or right-hand traffic (RHT) networks. A “rule” attribute of each <road> element indicates whether the road is LHT (vehicles drive on the left) or RHT (vehicles drive on the right).

[0092]

[0093] A global cartesian coordinate system is defined with the x-direction lying eastwards and the y-axis extending northwards (OpenDRIVE calls this the inertial coordinate system).

[0093]

[0094] Lane numbers only indicate relative directions of traffic flow within a road: for any given road, traffic flows in the same direction for positively-numbered one-way side-lanes, and in the opposite direction for negatively-numbered one-way side-lanes. Bi-directional lanes support traffic flow in both directions irrespective of the road traffic rule. However, the lane number alone is not sufficient to infer the direction of traffic flow, as the direction of the s-axis can be (more or less) arbitrarily chosen and does not indicate driving direction. For example, in FIG. 5, the s-axis 505 of the second road 504 extends towards the junction from the east, whilst the s-axis of the fourth road 508 extends in the opposite direction towards the junction 510 from the west. Along the second road 504, positive lane numbers therefore denote a direction of traffic flow from east to west, whereas along the fourth road 508, east-to-west traffic flow is denoted by negative lane numbers. For a LHT road, lanes to the left of the road reference line (+t) carry traffic in the direction of the road reference line (+s), whereas lanes to the right of the road reference line (-t) carry traffic in the opposite direction (-s). In an RHT road, lanes to the left of the road center line (+t) carry traffic in the opposite direction of the road reference line (-s) and lanes to the right (-t) carry traffic in the direction of the road reference line (+s).

[0094]

[0095] It is also possible to define a bidirectional side-lane, permitting traffic flow in both directions, by setting a @type attribute of the <lane> element to “bidirectional”. Bidirectional lanes are addressed in more detail below.

[0095]

[0096] Therefore, in order to determine the absolute direction of traffic flow, the lane number is not sufficient; the direction of the road reference line (s-axis) must also be considered, as must the @type attribute.

[0096]

[0097] Lanes are not necessarily drivable. For example, Lane 1 of Lane Section 2 of Road 2 is non-drivable. The outer boundaries of a road may also be defined by non-drivable lanes (such as lanes of a pavement / sidewalk type).

[0098] Link elements are used to explicitly define linkage between roads and lanes. With only two roads, links can be defined with link elements between roads directly, provided the links are unambiguous. More complex linkage requires the use of junction elements.

[0097]

[0099] A <link> element can have one or both of a <successor> element and a <predecessor> element. The predecessor / successor of a road can be another road or a junction. The predecessor / successor of a lane is another lane. “Predecessor” and “successor” relationships are defined relative to the s-axis of the road in question (a road’s t-axis runs from its predecessor, if any, to its successor, if any), and do not denote driving direction. The s-axis of a road runs away from its predecessor (if any), towards its successor (if any).

[0098]

[0100] Predecessor / successor relationships may be ‘asymmetrical’. For example, if Road n is a predecessor of Road m, that does not imply Road m is a successor of Road n; if the s- axis of Road n runs in the opposite direction to the s-axis of Road m, then Road m could also be a predecessor of Road n. As another example, if Road m is part of a junction, then Road m cannot be a predecessor or successor of Road n, because the junction would be the predecessor / successor of Road n instead (see below for further examples).

[0099]

[0101] Roads with varying number of lanes are accommodated using lane sections. Within the OpenDRIVE schema, lanes are described within a <lanes> element of a <road> element. A <lanes> element may be split into multiple sections (lane sections) with a <laneSection> element. Each individual lane is defined by a <lane> element within the <laneSection> element. Each lane section has a fixed number of lanes, numbered as per the above convention. For example, the first road 502 is shown divided into two lane sections (Lane Sections 1 and 2 of Road 1): approaching the junction 510, the number of lanes is shown to increase from two to three. On entering Lane Section 2 from Lane Section 1 of the first road 502, a new rightmost lane is created, whose width gradually increases from zero. In Lane Section 1, the left and right most lanes (to the left of the center lane) are numbered 2 and 1 respectively, whilst in Lane Section 2 the left-most lane and new right-most lanes are numbered 3 and 1 respectively; what is now the middle lane is numbered 2.

[0100]

[0102] Links between lanes in adjacent lane sections are described using <link> elements. In Lane Section 2, Lane 2 of Lane Section 1 is a predecessor of Lane 3 of Lane Section 2, and Lane 1 of Lane Section 1 is a predecessor of Lane 2 of Lane Section 2. That is to say, the lane element describing Lane 3 of Lane Section 2 would contain a link element indicating Lane 2 as its predecessor, and it is implicit that the link refers to the immediately preceding lane section (Lane Section 1) etc.

[0101]

[0103] Likewise, in Lane Section 1, Lane 3 of Lane Section 2 is a successor of Lane 2 of Lane Section 1, and Lane 2 of Lane Section 2 is a successor of Lane 1 of Lane Section 1. That is, the lane element describing Lane 2 of Lane Section 1 would indicate Lane 3 as its successor, and it is implicit that the link refers to the next lane section (Lane Section 2) etc.

[0102]

[0104] Because Lane 1 of Lane Section 2 has zero width initially, it is not linked to any lane of Lane Section 1. It is, however, possible for a lane to have multiple successors or predecessors in different circumstances, for example if a lane splits immediately into two lanes, each of non-zero width at the lane section boundary.

[0103]

[0105] Within the junction element 510 of FIG. 5, additional roads are defined, whose reference lines are depicted as thick, solid arrows. Fifth to ninth roads 512, 514, 516, 518, 520 (Roads 5 to 9) are depicted within the junction 510 in this example. Although sidelanes within the junctions 510 are not depicted in FIG. 5, each road within the junction 510 is also required to have at least one side-lane of non-zero width.

[0104]

[0106] In FIG. 5, the junction 510 is a successor of the first, second and fourth roads 502, 504, 508 because their respective s-axes extend towards the junction 510. Therefore, the <road> elements describing those roads 502, 504, 508 would contain <link> elements with <successor> elements indicating the junction 510. The junction 510 is a predecessor of the third road 506 because its s-axis extends away from the junction, and the <road> element describing the third road 506 would therefore contain a <link> element with a <predecessor> element indicating the junction 510.

[0105]

[0107] Within the junction element 510, connecting roads are described by <road> and <connection> elements.

[0106]

[0108] The fifth road 512 (Road 5) is shown to connect the first and third roads 502, 506. The fifth road 512 is defined by a <road> element within the <junction> element 510, of which the first road 502 is a predecessor and the third road 506 is a successor (defined via <predecessor> and <successor> elements in the <road> element describing the fifth road 506). Similarly the sixth road 514 has the fourth road 508 as its successor and the second road 504 as its predecessor. The seventh road 516 connects the second and third roads 504, 506, with the second road 504 as its predecessor and the third road 506 as its successor. The eighth road 518 connects the second and first roads 504, 502, with the second road 504 as successor and the first road 502 as predecessor. Finally, the ninth road 520 connects the first and fourth roads 502, 508, with the fourth road 508 as its successor and the first road 502 as its predecessor. Again, the predecessor / successor relationships are not direct indicators of driving direction.

[0107]

[0109] A <connection> element is used to indicate driving direction for traffic joining a junction. A connection element indicates an incoming road and a connecting road (but does not explicitly define an outgoing road), with traffic entering the junction from the incoming road onto the connecting road. For example, a first <connection> element would be provided that indicates the road with ID=1 (the first road 502) as an incoming road and the road with ID=5 (the fifth road 512) as a connecting road; this connection element indicates that traffic can enter the junction 510 from the first road 502 onto the fifth road 512. A second <connection> element would indicate the first road 502 as an incoming road and the eighth road 518 as a connecting road, etc. The seventh connecting road 516 is a two-way road in this example, carrying traffic from the second road 504 to the third road 506, and traffic in the opposite direction. Therefore, two <connection> elements would be used, one indicating the second road as incoming and the seventh road 516 as connecting, and the other indicating the third road 506 as incoming and the seventh road 516 as connecting. The OpenDRIVE specification strongly advises one not to use two-way connecting roads, although two-way connecting roads are possible using the schema. The seventh connecting road 516 goes against this advice, but does not violate it (and, in practice, it is more likely that two one-way connecting roads would be defined). The fourth road 508 is not an incoming road to thejunction 510 (even though its s-axis extends towards the junction 510), because it is a one-way road that only carries traffic away from thejunction 510.

[0108] [HO] A connecting road with multiple lanes is used when lane changes are possible within thejunction 510; if lane changes are not permitted, multiple single-lane roads would be used instead.

[0109] [Hl] A “virtual” connection can also be described using a <connection> element without any connecting road. Virtual connections are limited to “virtual” junctions, which do not require a main road to be split up.

[0112] Links between lanes are described via link elements within the <lane> elements that describe those lanes, where predecessor and successor lanes are described in a similar manner to roads. In order for a first lane of a first road to be a predecessor or successor of a second lane of a second road, the second road must be a predecessor or successor of the first road. As the junction 510 is the successor of the first, second and fourth roads 502, 504, 508, these roads 502, 504, 508 have no direct successor roads; therefore, it is not possible for any of their lanes to have lane successors. Likewise, as the junction 510 is the predecessor of the third road 506, the third road 506 has no direct predecessor road; therefore, its lanes cannot have any predecessors. Within the junction 501, however, each of the connecting roads 512-510 has both a direct successor road and a direct predecessor road; for example, the direct successor road of the fifth road 512 is the first road 502 and its direct predecessor is the third road 506. Consequently, lanes of the connecting roads 512- 520 can be linked to lanes of both their predecessor and their successor roads. For example, the fifth road 512 would typically contain two positively-numbered side-lanes, 1 and 2. The side-lanes of Road 5 are not depicted in FIG. 5. The lane elements describing lanes 2 and 1 of Road 5 would, in turn, contain link elements, which indicate lane numbers “3” and “2” as their respective predecessors (and it is implicit that these are lanes of Road 5’s successor, namely Road 1), and lane numbers “2” and “1” as their respective successors (implicitly referring to Road 5’s successor, namely Road 3). A <connection> element also contains any lane.

[0110]

[0113] The scenario descriptions described in relation to FIG.s 1A-2 considered both the static and dynamic layers of logical and concrete scenarios. However, scenario definitions can be formulated without a specific static layer.

[0111]

[0114] As noted above, concrete scenarios are scenarios at the lowest level of abstraction, or rather at no level of abstraction in the sense that their parameters are fully specified. In concrete scenarios, the parameters and variables of the scenario are determined to be a fixed value for any point in time. In other words, concrete scenario descriptions describe a single scenario with one specific scenery, e.g. one road network, and one chain of events with fixed parameters.

[0112]

[0115] As also noted above, logical scenarios do not specify all values for all parameters but instead define a range of values that can be selected for a particular parameter. In other words, logical scenarios present a parameterized representation of a set of scenarios.

[0116] In the context of AV stack testing described in relation to FIG.s 1A-2, a logical scenario may be generally defined by a scenario definition, having (amongst other things) a plurality of parameters. Variations of the scenario (concrete scenarios) are generated from different parameterizations of the definition (e.g. varying external conditions, varying starting speeds of other vehicles etc). That is to say a concrete scenario is a particular instance of a logical scenario, derived from the logical scenario by selecting particular parameters. In practice, the selection of parameters to generate a concrete scenario may be based on sampling a relevant distribution of parameter values.

[0113]

[0117] The disclosure presented herein considers abstract scenario definitions. An abstract scenario definition can be considered a formal scenario definition that conceptualizes scenarios to the level of the scenario intent. In other words, an abstract scenario definition enforces constraints on the events within the scenario. Abstract scenarios efficiently describe the relations between elements of the scenario, e.g. cause-effect between actors in a scenario description.

[0114]

[0118] Abstract scenarios are one level of abstraction above logical scenarios, and cannot be consumed directly by a simulator. However, whereas a logical scenario corresponds to a finite (albeit potentially very large) set of concrete scenarios, it is impossible (or at least extremely difficult) to convert all possible permutations of an unbounded abstract scenario to a finite set of logical scenarios. For example, an abstract scenario could specify that a cut in scenario occurs on a road layout with at least one lane. This particular abstract scenario definition can therefore be executed on a road layout with any number of lanes as long as there are more than one - i.e. from 2 to an infinite number of lanes. In order to convert this example abstract scenario definition into a finite set of logical scenarios, sampling methods and selection rules need to be applied to sample a finite number of lanes on which to perform the scenario. Additionally, on any given map there are endless numbers of locations or possible speeds. Their dependencies result in an infinite number of logical scenarios with different ranges.

[0115]

[0119] Concrete scenario descriptions are in a form that may be consumed by a simulator. Abstract and logical definitions can be concretized by applying them to a map and choosing parameter values from defined parameter ranges.

[0116]

[0120] An example of an abstract scenario definition 300 of the dynamic content of a scenario in a DSL, specifically in OpenSCENARIO 2.0, is illustrated in FIG. 3A.

[0117]

[0121] The scenario definition 300, written in a scenario definition language, is an example scenario involving an ego vehicle 305 and two challenger vehicles. The definition includes statements defining the ego vehicle 305, a lead vehicle 310 and other vehicle 315.

[0118]

[0122] The sut.vehicle in FIG. 3 A is referred to as the ego vehicle 305 herein. The ego vehicle 305 is representative of the stack under test (SUT) to be used in the simulation testing described with reference to FIG.s 1 A to 2. Variable parameters of the lead vehicle 310 are defined with respect to the ego vehicle 305. The lead vehicle 310 is defined to be in the same lane as the ego vehicle 305 at the end of the scenario using statement “lane(same_as: sut.vehicle, at: end)”. The lead vehicle 310 is separated by a parameterized distance range from the ego vehicle at the end of the scenario. This is expressed in statement “position([20m..200m], ahead_of: sut.vehicle, at: end)”.

[0119]

[0123] The other vehicle 315 is also defined to be in the same lane as the ego vehicle 305 at the end of the scenario using statement “lane(same_as: sut.vehicle, at: end)”. The position of the other vehicle 315 is defined with respect to the lead vehicle 310 at the end of the scenario. This is expressed in statement “position([20m..200m], ahead of: lead vehicle, at: end)”. The speed of the other vehicle at the end of the scenario is also defined using the statement “speed(0kph, at: end)”.

[0120]

[0124] The scenario in which the vehicles will participate in simulation is also defined. In the example embodiment of FIG. 3 A, the scenario is a cut out scenario 320, in which the lead vehicle 310 pulls out of the lane to reveal the other vehicle 315. It will be appreciated however that any type of scenario can be defined using the DSL. Parameters relating to the duration of the scenario are defined using the statement “cut out: parallel(duration: [Is. ,4s])”. The participants of the scenario are defined to be the ego vehicle 305, the lead vehicle 310 and the other vehicle 315. The position of the lead vehicle 310 is defined to be in a side lane relative to the ego vehicle 305 at the end of the scenario using the statement “lead_vehicle.drive() with: lane(side_of: sut.vehicle, side: side, at: end)”. The speed of the other vehicle 315 is also defined using the statement “other_car.drive() with: speed(speed: Okph)”.

[0121]

[0125] More examples of abstract scenario definitions are shown in FIG. 3B. These examples can be found on the ASAM OpenSCENARIO V2.0.0 user guide, Release Date 20 July 2022 [available at https: / / www.asam.net / static downloads / public / asam- openscenario / 2.0.0 / conceptual-overview / scenario-abstraction.html#sec-global- term i nol ogy -ab stract scenari o] , the contents of which are incorporated herein by reference.

[0122]

[0126] Example scenario definition 2, indicated generally by reference number 301, defines a scenario involving an emergency vehicle and an ego vehicle. The scenario is defined to be a cut in scenario by using the statement 330 “scenario sut. right_cut_in_emergency_vehicle : ” .

[0123]

[0127] The road layout on which the scenario is to be run is constrained within the scenario definition. This is expressed in statement 334 “my road: road with: keep(it.min_lanes>=2)”.

[0124]

[0128] In both the scenario definitions shown in FIG. 3B, the cut in is defined by imposing constraints on the position of the emergency vehicle relative to the ego vehicle (see 336). Specifically, the emergency vehicle is defined to cut in on the right of the ego vehicle as expressed in statements "do cut_in() with:”, “keep(side == right)” and “keep(road == my road)”. . Vehiclel is constrained to be the emergency vehicle as expressed in statement “keep(vehiclel == my_emergency_car)” 332.

[0125]

[0129] In scenario definition 3 (indicated by reference numeral 302), the scenario is more concrete as the type of emergency vehicle is defined to be a police vehicle 340. This is expressed using statement “my_police_car: police vehicle” Although more concrete, the scenario definition is still an abstract scenario description as not all the locations and attributes are specified for any given point at all times of the scenario execution.

[0126]

[0130] As will be appreciated, the abstract scenarios are not tied to a particular physical context (i.e. a particular static layer). None of the example abstract scenarios discussed above include a detailed definition of the static layer. However, in order to be concretized and executed, a suitable static layer needs to be selected in which the abstract scenario takes place. In this context, a suitable static layer is one that satisfies the constraints set out in the scenario definition of the dynamic content described above.

[0127]

[0131] For example, the scenario definition 300 in FIG. 3 A requires a physical context that would facilitate the execution of the cut out scenario. Consequently, it requires an environment in which there are at least two lanes, such that there is somewhere for the lead vehicle to cut out into. Furthermore, it implicitly requires a stretch of road long enough to include the vehicles in the positions specified in the definition.

[0128]

[0132] In another example, the scenario definitions 301, 302 in FIG. 3B explicitly require a road network that has at least two lanes in order to execute the scenario definition.

[0129]

[0133] The scenario definitions 300, 301, 302 described in relation to FIGs 3 A and 3B can be abstractions of the scenario descriptions executed in simulation in FIGs IB - 2. The scenario definitions in FIGs 3A and 3B can be concretized to arrive at the scenario descriptions described with reference to FIGs IB - 2.

[0130]

[0134] FIG. 4A shows a schematic block diagram of a system for defining a scenario.

[0131]

[0135] The system of FIG. 4A includes an interface component 400. In general, the interface component 400 takes the form of a software application which is used by a scenario designer to create and edit scenario definitions. It is to some extent similar to an integrated development environment (IDE) which is typically used in code editing, but with particular functionality that makes it useful for editing scenario definitions.

[0132]

[0136] The interface component comprises a graphical user interface (GUI) 406. The GUI 406 is rendered on a display, in connection with a computer system, to a user.

[0133]

[0137] A computer system comprises execution hardware which may be configured to execute the method / algorithmic steps disclosed herein. The term execution hardware encompasses any form / combination of hardware configured to execute the relevant method / algorithmic steps. The execution hardware may take the form of one or more processors, which may be programmable or non-programmable, or a combination of programmable and non-programmable hardware may be used. Examples of suitable programmable processors include general purpose processors based on an instruction set architecture, such as CPUs, GPUs / accelerator processors etc. Such general-purpose processors typically execute computer readable instructions held in memory coupled to or internal to the processor and carry out the relevant steps in accordance with those instructions. Other forms of programmable processors include field programmable gate arrays (FPGAs) having a circuit configuration programmable through circuit description code. Examples of non-programmable processors include application specific integrated circuits (ASICs). Code, instructions etc. may be stored as appropriate on transitory or non- transitory media (examples of the latter including solid state, magnetic and optical storage device(s) and the like).

[0138] The computer system may comprise a display, for example a monitor. The monitor can render the GUI 406, displaying it to the user.

[0134]

[0139] The interface component 400 can be a local application on a client device or can be hosted remotely, for example by a cloud computing platform. The interface component 400 outputs a scenario definition to a simulation environment 410 and receives vehicle traces according to the scenario definition from the simulation environment 410 as input.

[0135]

[0140] The interface component 400 includes two components, a code editing interface 402 and a visualization interface 404. The two components 402, 404 can be implemented as software modules. For example, the code editing interface 402 and the visualization interface 404 can be one software module or separate software modules. The software module / s can be local to a client device or can be hosted on a cloud computing platform.

[0136]

[0141] The code editing interface 402 receives code written by a user defining an abstract scenario definition. It is a development environment that enables the user to write, change and edit code defining the scenario. Example abstract scenario definitions have been described with reference to FIGs 3A and 3B.

[0137]

[0142] The code can be input via a keyboard, touch screen or any suitable input means that enables the user to input text to the code editing interface 402. When the user inputs the code into the code editing interface 402, it can be continuously parsed while the user writes and edits the scenario definition. This provides immediate feedback when keywords or syntax errors are introduced.

[0138]

[0143] For example, the brackets that indicate parameter ranges within the scenario definition can be highlighted one color (e.g., purple) and a defined speed for a vehicle can be highlighted another color (e.g., green). It will be appreciated that other words, phrases and characters within the scenario definition can also be highlighted depending on whether they are considered keywords and / or syntax errors.

[0139]

[0144] In addition to the abstract scenario definition, the scenario requires a map. In other words, the scenario definition needs a suitable physical context in which to be executed. The user may select the physical context in the form of a map from a drop-down list displayed on the GUI 406 in which to run the simulation. Alternatively, the interface component 400 may use the constraints within scenario definition to search for compatible maps and select one that meets the criteria specified in the scenario definition. If a number of compatible maps are available, the interface 400 may select one of the compatible maps. For example, a simplest or smallest available map may be selected with a view to reducing overhead, or a map may be selected at random.

[0140]

[0145] The code editing interface 402 (or some other element of the interface component 400 more generally) includes an input element that causes visualization of the code, as discussed in more detail below. For example, the interface may comprise a button (e.g., labelled “preview”). In other examples, some other action may prompt the visualization e.g. saving the scenario definition.

[0141]

[0146] The visualization interface 404 receives vehicle traces from the simulation environment 410 and visualizes the trajectories of the agents within the scenario on a visual representation of a physical context, for example a road map. This is displayed to a user via a graphical user interface (GUI) 406.

[0142]

[0147] The visualization interface 404 may use any suitable rendering techniques to generate the visualization from the vehicle traces. For example, a number of suitable simulation visualization libraries exist, which may be employed to generate a suitable pictorial representation of the traces on the road map. These include visualization tools such as Foxglove (https: / / foxglove.dev / ), which is a tool for visualization of robotics and simulation data. It will be appreciated, however, that many proprietary simulation platforms and systems provide suitable libraries or techniques for visualizing the output of a simulation, given a trace in an appropriate data format.

[0143]

[0148] The visualization interface 404 may render the map and the traces in any suitable representative format. For example, the visualization may be a 2D “top-down” representation of the map with the vehicles displayed as simple shapes (e.g. boxes) thereon. Alternatively, the map may be represented as a simple 2D plane with appropriate markings, with the vehicles being shown as simple 3D shapes (boxes, approximate car shapes). In other examples, further detail (e.g. suitable textures, detailed road topography etc) may be rendered. It will be appreciated that the purpose of the visualization is to give the user an indication of the content of defined abstract scenario, the preference is for simpler and more lightweight rendering techniques, rather than photorealistic rendering.

[0149] FIG. 6 shows an example code editing interface panel 602 and visualization interface panel 604 on a GUI 406 displayed to a user. The view of the visualization interface panel 604 pertains to a particular scenario defined in the code editing interface panel 602 as described above.

[0144]

[0150] The code editing interface panel 602 displays the code 615 written by the user as described with reference to FIG. 4A. The code 615 in this example corresponds to the code described with reference to FIG. 3 A. The user then presses a button on the GUI 406 or other suitable trigger to initiate the execution of the scenario definition to be rendered on the visualization interface panel 604.

[0145]

[0151] In the example of Figure 6, the scenario contains two agents in the scenario. The agents are shown as 3D objects on a 2D road layout. The road layout is representative of the physical context, or static road layout, that was selected for the scenario run.

[0146]

[0152] Additionally, the visualization interface panel 604 displays the speed of the simulation 610, for example, whether the user is viewing the evolution of the scenario in lx or 1.5x the real-time of the scenario. Although the visualization of the scenario can be viewed in realtime, the lightweight planner that was used to execute the scenario definition does not necessarily operate at real-time. This will be described in more detail below.

[0147]

[0153] The GUI 406 shown in FIG. 6 displays the code editing interface panel 602 and the visualization interface panel 604 to be side-by-side. It will be appreciated that the present disclosure is not limited to display the interface panels in this configuration at all times. For example, the interface panels 602, 604 could be arranged such that the code editing interface panel 602 is displayed above the visualization interface panel 604 on the screen of a user. The interface panels 602, 604 could also be open in different windows of an application. Additionally, one of the interface panels could be a pop up on the screen of a user.

[0148]

[0154] FIG.4B shows a schematic block diagram of an example simulation environment.

[0149]

[0155] The simulation environment 410 receives the abstract scenario definition 412, described with reference to FIG. 4A, and a physical context, e.g., a road network 414 as input. It will be appreciated that receiving the physical context also includes cases where a reference to (e.g. a storage location of) the physical context is passed to the simulation environment 410. The simulation environment 410 outputs vehicle traces having run a scenario according to the scenario definition 412 using a lightweight planner 416.

[0150]

[0156] Similar to the interface component 400 described with reference to FIG. 4A, the simulation environment 410 can be implemented as a software module forming part of a local application to a client device or hosted by a cloud computing platform. Whilst the figures show the simulation environment 410 as a separate component to the interface component 400, it will be understood that in some examples the simulation environment and interface component 400 may be part of the same application, for example running on the same computer.

[0151]

[0157] The simulation environment 410 is similar to the simulator 202 described with reference to FIG.2, however the simulation environment 410 does not use the ego stack 100. Instead, the simulation environment 410 uses the lightweight planner 416. The purpose of the simulator 202 in FIG. 2 is for testing the ego stack 100 whereas the simulation environment 410 only requires the scenario to run such that the user can determine whether the agents behave as defined in the scenario definition 412. That is to say, complex ego logic such as that present in an ego stack 100 under test is not required for the purpose of running the scenario in order to verify the scenario definition 412. The simulation environment 410 may be termed a “second” simulation environment, in order to distinguish it from the simulator 202, which may be considered a “first” simulation environment herein.

[0152]

[0158] In order to create a scenario description that can be consumed by the simulation environment, the abstract scenario definition 412 needs to be concretized. In other words, the parameter ranges that constrain the scenario need to be converted to parameter values and an appropriate map needs to be selected according to those values. For example, consider the scenario definition in FIG. 3B. This scenario constrains the road to have at least one lane and the ego vehicle is involved in a cut in scenario with another vehicle. The road map therefore needs to have one lane to satisfy the scenario constraints, but would also need to ensure that the ego vehicle can be involved in a cut in scenario with the other vehicle. For example, the road may have three lanes all parallel to each other such that it facilitates a cut in scenario. It will be appreciated that any permutation of road network that satisfies the scenario constraints and event can be used.

[0159] The map that is selected for the scenario is compatible with the scenario definition. Suitable physical contexts in the form of road maps can be selected by the user via the GUI 406 described with reference to FIG.6. Alternatively, the simulation environment 410 can analyze the concretized description of the dynamic part of the scenario definition 412 to suggest suitable maps available to the user for the particular scenario. In another example, the simulation environment 410 can automatically select a suitable road map 414 according to the parameter values. The selection of the road map 414 could be performed by randomly selecting from a list of suitable maps, or selecting a simplest or smallest available road map 414.

[0153]

[0160] It will be appreciated that parameter ranges of dynamic aspects of the scenario must also become parameter values in order to execute the scenario definition in the simulation environment. For example, the scenario definition described with reference to FIG. 3 A constrains the scenario to be a cut out scenario with a duration of between 1 and 4 seconds. In order to concretize the scenario definition to be consumed by the simulation environment, the duration must be defined such that it is a parameter value, e.g., 3 s.

[0154]

[0161] The parameter values can be selected from the parameter ranges by the user on the GUI 406 described with reference to FIG. 6. Alternatively, the parameter values can be selected by the simulation environment 410 from the range of values given in the scenario definition 412. For example, the simulation environment 410 may select the midpoint of a range, either extreme of a range or randomly select a value from the range.

[0155]

[0162] Although the concretization of the scenario definition 412 described above has been described in the context of the simulation environment 410, it will be appreciated that this process could also be performed at the interface component 400 described with reference to FIG. 4A. Additionally, an intermediate component within the system of FIG. 4A (not shown) could perform the concretization of the scenario. This additionally applies to the process of selecting a suitable physical context, e.g. a road map 414. If the concretization of the scenario definition 412 is performed by the interface component 400 or some intermediate component, the input into the simulation environment 410 will consequently become the concrete scenario description.

[0156]

[0163] Once a concrete scenario description has been generated from the abstract scenario definition 412, the scenario can be run using the lightweight planner 416.

[0164] Referring back to FIG. 2, the simulator 202 operates analogously, but the ego stack is replaced by the lightweight planner 416. The lightweight planner 416 may perform in a manner similar to the agent decision logic 210 in FIG. 2, in that it has more limited decisionmaking capability than the ego stack 100. In this context, the decisions of all the agents within the scenario, including the ego vehicle, may be controlled by the lightweight planner 416. In general, the lightweight planner 416 is a planner that is not undergoing performance testing and is being employed specifically for the purpose of generating scenario visualizations.

[0157]

[0165] The lightweight planner 416 need not work in real-time when executing the scenarios. The purpose of the lightweight planner 416 is to ensure that the scenario definition 412 produces scenarios in which the agents behave as expected by providing immediate feedback to the user. There is therefore no benefit or requirement to execute the scenarios in real-time thus enabling the lightweight planner 416 to rapidly execute the scenario.

[0158]

[0166] The lightweight planner 416 can be a rules-based planner that does not require ‘full’ planner logic. The rules based planner can be implemented as a simplified basic driver model. For example, when considering safety of lane changes, e.g., in the cut in scenario, the simplified rules-based planner might look for a gap in the traffic in the lane next to ego. Based on a predefined assumption that if the ego were to start performing a lane change in the current timestep, complete it within ‘x’ seconds, remain at its current speed, a determination is made as to whether the lane change maneuver is safe, e.g., would it impact any oncoming vehicles in that lane. Taking this as a simplified planner decision, the basic driver model could be configured to find gaps in traffic that a hypothetical planner may have decided to move into or not based on its perception. Other similar rules may exist for other maneuvers.

[0159]

[0167] The lightweight planner 416 may be integral to the simulation environment 410. That is to say, the simulation environment 410 may be provided with internal logic e.g. ordinarily used for controlling agents other than the ego vehicle. However, in other cases the lightweight planner 416 is external to the simulator, effectively replacing the ego stack 100 in FIG. 2.

[0160]

[0168] The output of the simulation environment 410 is vehicle traces for the agents included in the executed scenario. Each trace is a complete history of an agent’s behavior within a scenario having both spatial and motion components. For example, each trace may take the form of a spatial path having motion data associated with points along the path such as speed, acceleration, jerk (rate of change of acceleration), snap (rate of change of jerk) etc, as discussed above. If different road networks are used in relation to the same abstract scenario definition, the vehicle traces will reflect the different road networks selected by the simulation environment 410.

[0161]

[0169] In use, first the user writes an abstract scenario definition using the code editing interface 402. The user or the simulation environment 410 selects parameter values according to the scenario definition 412 and a suitable road network 414 to generate a concretized scenario description. The concretized scenario description is then executed by the lightweight planner 416 on the selected road network 414 in simulation to generate vehicle traces for all actors in the scenario. The vehicle traces are visualized on the visualization interface 404 and displayed to the user, such that the vehicle traces are dependent on the scenario definition 412 and the road network 414 received by the lightweight planner 416. Consequently, the user is able to determine whether the actors in the scenario behaved as the user had envisaged when writing the abstract scenario definition. If they did not, the user can edit the abstract scenario definition and repeat the process. Alternatively, if the visualization of the scenario exhibits the expected behavior of the agents, the user can use the scenario description in simulation-based testing of the ego stack, as described with reference to FIG. 2.

[0162]

[0170] Whilst the above examples consider AV stack testing, the techniques can be applied to test components of other forms of mobile robot, and the definition of suitable test scenarios for such mobile robots. Other mobile robots are being developed, for example for carrying freight supplies in internal and external industrial zones. Such mobile robots would have no people on board and belong to a class of mobile robot termed UAV (unmanned autonomous vehicle). Autonomous air mobile robots (drones) are also being developed.

[0163]

[0171] References herein to components, functions, modules and the like, denote functional components of a computer system which may be implemented at the hardware level in various ways. A computer system comprises execution hardware which may be configured to execute the method / algorithmic steps disclosed herein and / or to implement a model trained using the present techniques. The term execution hardware encompasses any form / combination of hardware configured to execute the relevant method / algorithmic steps. The execution hardware may take the form of one or more processors, which may be programmable or non-programmable, or a combination of programmable and nonprogrammable hardware may be used. Examples of suitable programmable processors include general purpose processors based on an instruction set architecture, such as CPUs, GPUs / accelerator processors etc. Such general-purpose processors typically execute computer readable instructions held in memory coupled to or internal to the processor and carry out the relevant steps in accordance with those instructions. Other forms of programmable processors include field programmable gate arrays (FPGAs) having a circuit configuration programmable through circuit description code. Examples of nonprogrammable processors include application specific integrated circuits (ASICs). Code, instructions etc. may be stored as appropriate on transitory or non-transitory media (examples of the latter including solid state, magnetic and optical storage device(s) and the like). The subsystems 102-108 of the runtime stack Figure 1A may be implemented in programmable or dedicated processor(s), or a combination of both, on-board a vehicle or in an off-board computer system in the context of testing and the like. The various components of Figure 2, such as the simulator 202 and the test oracle 252 may be similarly implemented in programmable and / or dedicated hardware.

Claims

CLAIMS1. A computer-implemented method of defining a scenario to be run in a simulation environment for testing the behaviour of a runtime stack of a sensor-equipped robot, the method comprising: rendering a code editing interface on a display of a computer device, the code editing interface configured to receive an abstract scenario definition in a scenario definition language, the abstract scenario definition defining a scenario to be run in a first simulation environment for testing the behaviour of the runtime stack of the sensor-equipped robot having a role of an ego agent in the scenario; generating a concrete scenario description from the abstract scenario definition and a selected physical context compatible with the abstract scenario definition; providing the concrete scenario description to a second simulation environment; executing the concrete scenario description in the second simulation environment using a lightweight planner not forming part of the runtime stack to act as the ego agent; receiving from the second simulation environment an ego trace representative of actions of the ego agent during execution of the concrete scenario description; rendering, on the display of the computer device, a visualisation interface displaying a visual representation of the selected physical context; and rendering the actions of the ego agent on the visual representation of the selected physical context.

2. The method of claim 1, wherein the scenario definition language is OpenSCENARIO.

3. The method of claim 1 or 2, comprising: receiving a challenger agent trace from the second simulation environment representative of the actions of a challenger agent, and rendering the actions of the challenger agent on the visual representation of the physical context.

4. The method of any preceding claim, comprising:rendering, on the display, a user input element; in response to user input at the user input element, generating the concrete scenario description and providing the concrete scenario description to the second simulation environment.

5. The method of any preceding claim, wherein the lightweight planner is a rules-based planner.

6. The method of any preceding claim, wherein the lightweight planner is a non-real-time planner.

7. The method of any preceding claim, wherein the lightweight planner forms part of the second simulation environment.

8. The method of any preceding claim, comprising: determining a plurality of physical contexts compatible with the abstract scenario definition; and selecting the physical context from the plurality of physical contexts.

9. The method of any preceding claim, comprising: receiving user input selecting the physical context compatible with the abstract scenario definition.

10. The method of any preceding claim, wherein generating the concrete scenario description comprises selecting values for one or more parameters defined in the abstract scenario definition.

11. The method of any preceding claim, comprising: receiving, via the code editing interface, user input editing the abstract scenario definition to provide a second abstract scenario definition; generating a second concrete scenario description from the second abstract scenario definition and a selected physical context compatible with the second abstract scenario definition;providing the second concrete scenario description to the second simulation environment; executing the second concrete scenario description in the second simulation environment using the lightweight planner; receiving from the second simulation environment a second ego trace representative of actions of the ego agent during execution of the second concrete scenario description; and rendering the actions of the ego agent corresponding to the second ego trace on the visual representation of the selected physical context.

12. The method of any preceding claim, comprising: rendering, on the visualization interface, playback controls; rendering the action of the ego agent in response to user input at the playback controls.

13. The method of any preceding claim, comprising concurrently displaying both the code editing interface and the visualization interface.

14. A computer system for defining a scenario to be run in a simulation environment for testing the behaviour of a runtime stack of a sensor-equipped robot, the system comprising: a display; a memory; at least one processor configured to execute instructions stored in the memory to cause the system to: render a code editing interface on the display, the code editing interface configured to receive an abstract scenario definition in a scenario definition language, the abstract scenario definition defining a scenario to be run in a first simulation environment for testing the behaviour of the runtime stack of the sensor-equipped robot having a role of an ego agent in the scenario; generate a concrete scenario description from the abstract scenario definition and a selected physical context compatible with the abstract scenario definition; provide the concrete scenario description to a second simulation environment;execute the concrete scenario description in the second simulation environment using a lightweight planner not forming part of the runtime stack to act as the ego agent; receive from the second simulation environment an ego trace representative of actions of the ego agent during execution of the concrete scenario description; render, on the display, a visualisation interface displaying a visual representation of the selected physical context; and render the actions of the ego agent on the visual representation of the selected physical context.

15. A tangible, non-transient computer readable medium storing instructions, which when executed by a computer device, cause the computer device to perform the method of any of claims 1 to 13 or implement the system of claim 14.

Images