Simulating AV behaviour

Abstract

A method of simulating behaviour of an autonomous vehicle to test a control stack for controlling the autonomous vehicle, the method comprising: providing to a simulator a scenario in which an ego vehicle is under observation; initiating operation of the simulator to execute the scenario and simulating an initial phase of a simulation of the scenario with the ego vehicle under the control of a first control stack implemented by the simulator; detecting that a handover condition is met in the scenario; and relinquishing control of the ego vehicle by the first control stack and causing a second control stack to be engaged to control the ego vehicle in the subsequent phase of the simulation, wherein the second control stack is the control stack under test.

Description

Simulating AV behaviour

[0001] Technical field

[0002] The present disclosure relates to the simulation of the behaviour of autonomous vehicles particularly but not exclusively for testing their behaviour.

[0003] Background

[0004] There have been major and rapid developments in the field of autonomous vehicles. An autonomous vehicle is a vehicle which is equipped with sensors and control systems which enable it to operate without a human controlling its behaviour. An autonomous vehicle is equipped with sensors which enable it to perceive its physical environment, such sensors including for example cameras, radar and lidar. Autonomous vehicles are equipped with suitably programmed computers which are capable of processing data received from the sensors and making safe and predictable decisions based on the context which has been perceived by the sensors. There are different facets to testing the behaviour of the sensors and control systems aboard a particular autonomous vehicle, or a type of autonomous vehicle.

[0005] An autonomous vehicle may be fully autonomous (in that it is designed to operate with no human supervision or intervention, at least in certain circumstances) or semi-autonomous. Semi-autonomous systems require varying levels of human oversight and intervention. An Advanced Driver Assist System (ADAS) and certain levels of Autonomous Driving System (ADS) may be classed as semi-autonomous. A “level 5” vehicle is one that can operate entirely autonomously in any circumstances, because it is always guaranteed to meet some minimum level of safety. Such a vehicle would not require manual controls (steering wheel, pedals etc.) at all. By contrast, level 3 and level 4 vehicles can operate fully autonomously but only within certain defined circumstances (e.g. within geofenced areas). A level 3 vehicle must be equipped to autonomously handle any situation that requires an immediate response (such as emergency braking); however, a change in circumstances may trigger a “transition demand”, requiring a driver to take control of the vehicle within some limited timeframe. A level 4 vehicle has similar limitations; however, in the event the driver does not respond within the required timeframe, a level 4 vehicle must also be capable of autonomously implementing a “minimum risk maneuver” (MRM), i.e. some appropriate action(s) to bring the vehicle to safe conditions (e.g. slowing down and parking the vehicle). A level 2 vehicle requires the driver to be readyto intervene at any time, and it is the responsibility of the driver to intervene if the autonomous systems fail to respond properly at any time. With level 2 automation, it is the responsibility of the driver to determine when their intervention is required; for level 3 and level 4, this responsibility shifts to the vehicle’s autonomous systems and it is the vehicle that must alert the driver when intervention is required.

[0006] Sensor processing may be evaluated in real- world physical facilities. Similarly, the control systems for autonomous vehicles may be tested in the physical world, for example by repeatedly driving known test routes, or by driving routes with a human on-board to manage unpredictable or unknown context.

[0007] Physical world testing will remain an important factor in the testing of autonomous vehicles capability to make safe and predictable decisions. However, physical world testing is expensive and time-consuming. Increasingly there is more reliance placed on testing using simulated environments. Autonomous vehicles need to have the facility to operate in the same wide variety of circumstances that a human driver can operate in. Such circumstances can incorporate a high level of unpredictability.

[0008] It is not viable to achieve from physical testing a test of the behaviour of an autonomous vehicle in all possible scenarios that it may encounter in its driving life. Increasing attention is being placed on simulating the behaviour of autonomous vehicles in simulation environments which can provide such testing in a manner that gives confidence that the test outcomes represent potential real behaviour of an autonomous vehicle.

[0009] For effective testing in a simulation environment, the autonomous vehicle under test (the ego vehicle) should have knowledge of its location at any instant of time, understand its context (based on simulated sensor input) and can make safe and predictable decisions about how to navigate its environment to reach a pre-programmed destination.

[0010] Simulation environments need to be able to represent real-world factors that may change. This can include weather conditions, road types, road structures, road layout, junction types etc. This list is not exhaustive, as there are many factors that may affect the operation of an ego vehicle.

[0011] Many simulation environments used for testing involve other actors. These are agents in the environment with which the ego vehicle may have to interact. There are various ways of controlling the behaviour of such other actors in the simulation environment in which the egovehicle is to operate. For example they may have a controlled trajectory in a simulated environment, controlled by the scenario from which the environment is simulated. In some cases, the other actors may have some limited autonomy to react to actions taken by the ego vehicle. Such actors may be other vehicles, although they could be other actor types, such as pedestrians, animals, bicycles et cetera.

[0012] A simulator is a computer program which when executed by suitable computer hardware enables a sensor equipped vehicle control module to be developed and tested in simulation, before its physical counterpart is built and tested. A simulator provides a sensor simulation system which models each type of sensor with which the autonomous vehicle may be equipped. A simulator also provides a three-dimensional environmental model which reflects the physical environment that an automatic vehicle may operate in. The 3-D environmental model defines at least the road network on which an autonomous vehicle is intended to operate, and other actors in the environment. In addition to modelling the behaviour of the ego vehicle, the behaviour of these actors also needs to be modelled.

[0013] Simulators generate test scenarios (or handle scenarios provided to them). A simulator can simulate many different scenarios in which the ego vehicle can be tested. Such scenarios can include different behaviours of actors. The large number of factors involved in each decision to which an autonomous vehicle must respond, and the number of other requirements imposed on those decisions (such as safety and comfort as two examples) mean it is not feasible to write a scenario for every single situation that needs to be tested. Nevertheless, attempts must be made to enable simulators to efficiently provide as many scenarios as possible, and to ensure that such scenarios are close matches to the real world. The purpose of the simulation is to test how the ego vehicle will operate in a similar situation in the physical world. An ego vehicle is controlled by an ego controller, which receives inputs from the sensors from the environment and makes decisions based on those inputs. The decisions are enacted by the ego vehicle in the simulation - for example the ego vehicle may slow down, speed up, stop, change trajectory or carry out any other kind of action which may resemble its action in a corresponding physical environment. The behaviour of the ego vehicle ( and hence the ego controller ) is analysed to assess how the ego controller performed in that environment and whether adjustments are needed to the controller. If testing done in simulation does not generate outputs which are faithful to the outputs generated in the corresponding physical world environment, then the value of simulation in testing the ego controller is markedly reduced.

[0014] Summary

[0015] When running certain simulations in a simulation platform, the inventors have observed a phenomenon perceived as “vehicle jumps”. It is the aim of a simulation to accurately represent the physical world, and the behaviour of an ego vehicle in the physical world. In the physical world, a vehicle may only move continuously from one location to another location. If in a simulation an ego vehicle is observed to be performing a discontinuous movement between a first location and a second location, this may represent a problem with the simulation or it may represent a problem with the ego controller. Either way, it is not something which represents the behaviour of a vehicle in the physical world. An ego controller operates on inputs provided by the (simulated) sensors and acts on those inputs to determine a trajectory which controls the motion of the ego vehicle. In order to represent the physical world, this trajectory should define a continuous path. If it does not, the ego vehicle may be seen to “jump” as a discontinuity in the path is represented by the trajectory which is determined by the ego controller in response to the inputs that it is receiving. In some cases, the jump in vehicle position that is requested by the trajectory determined by the ego controller is sufficiently large so as to cause the ego controller to produce an error.

[0016] When the jump in vehicle position requested by the trajectory is small, the ego vehicle may be seen visually to “stutter”, but with no error reported.

[0017] An objective of embodiments of the present invention is to seek to remove such “vehicle jumps” in simulation. It will be recognised that when vehicle jumps occur, analysing the simulation for the purpose of testing may not be useful, because it is evident that the simulation has not accurately represented the behaviour of vehicle in the physical world. It is another objective of embodiments of the present invention therefore to improve the quality of testing using simulations of autonomous vehicles , particularly but not exclusively at level 3 autonomy and above.

[0018] An aspect of the present invention provides a method of simulating behaviour of an autonomous vehicle to test a control stack for controlling the autonomous vehicle , the method comprising: providing to a simulator a scenario in which an ego vehicle is under observation; initiating operation of the simulator to execute the scenario and simulating an initial phase of a simulation of the scenario with the ego vehicle under the control of a first control stack implemented by the simulator;detecting that a handover condition is met in the scenario; and relinquishing control of the ego vehicle by the first control stack and causing a second control stack to be engaged to control the ego vehicle in a subsequent phase of the simulation, wherein the second control stack is the control stack under test.

[0019] In embodiments , the first control stack implemented by the simulator generates simulator trajectory messages which are used to control the ego vehicle until the handover condition is met . The second control stack receives simulator input from the scenario after simulator operation is initiated and before the handover condition is met and generates control stack trajectory messages using the simulator input, wherein the trajectory messages are not used to control the ego vehicle while the ego vehicle is under control of the first control stack.

[0020] In embodiments , when the handover condition is met in the scenario a handover message exchange is performed between the simulator and the second control stack. The handover message exchange may include a ready message which is issued from the second control stack when it is ready to take over control of the ego vehicle, and wherein the ready message is sent to the simulator indicating the ready status whereby the second control stack is engaged.

[0021] The handover condition may be an engage action point configurable by a creator of the scenario.

[0022] The engage action point may be selected from: a predetermined time from the start of the scenario; a position of the ego vehicle in the scenario; a position of another particular actor in the scenario; a state of a traffic control component; and distance travelled by the ego vehicle from a selected location in the scenario.

[0023] In embodiments , the simulator provides perception inputs according to a standard protocol to integration software which is configured to convert the perception inputs to a stack specific protocol for providing inputs to the second control stack when it is the ego vehicle controller The second control stack outputs trajectory messages for controlling theFive Al447076PCT ego vehicle after the handover point. . When the first control stack implemented by the simulator controls the ego vehicle , the signals sent to the vehicle are trajectory messages from the first control stack .

[0024] The invention provides in another aspect a computer system for simulating behaviour of autonomous vehicle, the computer system comprising: one or more hardware processor programmed to execute a simulator of a scenario in which an ego vehicle is under observation; and computer memory holding computer code executable to execute the method as defined above in any aspect.

[0025] The invention provides in another aspect a computer program product comprising computer code executable by one or more hardware processor to implement the method as defined above in any aspect .

[0026] Another aspect of the invention provides a computer-implemented method for generating a simulation environment for testing an autonomous vehicle, the method comprising: generating a scenario by rendering on a display of a computer device a timeline for execution of the scenario in the simulator; displaying on the timeline an initial phase of a simulation with an indication of a first control stack for controlling the ego vehicle in the simulation; detecting that a switch function has been selected by a user and causing display of a condition input field to be displayed to a user responsive to selection of the switch function; receiving into the condition field user input defining a handover condition in the scenario; and defining in the scenario that when the handover condition is reached in the scenario control of the ego vehicle by the first control stack will be caused to be handed over to a second control stack in a subsequent phase of the simulation, wherein the second control stack is a control stack of the ego vehicle under test.

[0027] The invention provides in another aspect a computer system for simulating behaviour of autonomous vehicle, the computer system comprising: one or more hardware processor programmed to execute a simulator of a scenario in which an ego vehicle is under observation; andcomputer memory holding computer code executable to execute the method as defined above in any aspect.

[0028] The invention provides in another aspect a computer program product comprising computer code executable by one or more hardware processor to implement the method as defined above in any aspect .

[0029] A further aspect of the invention provides a computer program product comprising computer code stored on a computer readable medium which when executed by a computer implements the steps of any of the above-defined methods.

[0030] Brief description of the drawings

[0031] Figure la is a schematic block diagram of a runtime stack for an autonomous vehicle;

[0032] Figure lb is a schematic block diagram of a testing pipeline;

[0033] Figure 1c shows a highly schematic block diagram of a ground truthing pipeline;

[0034] Figure Id shows a highly schematic block diagram of a core simulator interacting with an autonomous vehicle stack.

[0035] Figure 2 shows a highly schematic diagram of a testing pipeline;

[0036] Figure 3a shows a flowchart that represents an initialisation process for an ego stack in a scenario;

[0037] Figure 3b shows an exemplary user interface corresponding to the initialisation process of Figure 3a;

[0038] Figure 4 shows a flowchart representing an initialisation sequence for an ego stack in which a handover sequence is implemented;

[0039] Figure 5 shows a highly schematic diagram of message flows between an ego stack, a core simulator, and a scenario simulator;

[0040] Figure 6 shows an exemplary user interface corresponding to the initialisation process of Figure 4;

[0041] Figure 7 shows an exemplary user interface for configuring a handover sequence; and

[0042] Figure 8 shows a highly schematic block diagram of a computer system implementing a scenario builder.

[0043] Figure 9 shows three plan- view instances of an ego vehicle during a handover sequence.

[0044] In the drawings, corresponding reference characters indicate corresponding components. The skilled person will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. Also, common but well- understood elements that are useful or necessary in a commercially feasible embodiment are often not depicted in order to facilitate a less obstructed view of these various example embodiments.

[0045] It will be appreciated that where reference numerals of the form xxx-a, xxx-b etc., are used to denote a particular instance of a feature of a drawing, the same reference numeral without a specified letter suffix (e.g., a, b, etc.) may denote a generic instance of the same feature.

[0046] Detailed Description

[0047] Autonomous vehicles may be tested using a simulation platform which executes scenarios defining a simulated environment. The objective is to provide a simulation which accurately reflects the behaviour of an autonomous vehicle in the physical world.

[0048] When running certain simulations in a simulation platform, the inventors have observed a phenomenon perceived as “vehicle jumps”. It is the aim of a simulation to accurately represent the physical world, and the behaviour of an ego vehicle in the physical world. In the physical world, a vehicle may only move continuously from one location to another location. If in a simulation an ego vehicle is observed to be performing a discontinuous movement between a first location and a second location, this may represent a problem with the simulation or it may represent a problem with the ego controller. In testing a control stack or substitute of an ego controller, it is important to eliminate these “jumps”. An ego controller operates on inputs provided by the (simulated) sensors and acts on those inputs to determine a trajectory which controls the motion of the ego vehicle. In order to represent the physical world, this trajectory should define a continuous path. If it does not, the ego vehicle may be seen to “jump” as a discontinuity in the path is represented by the trajectory which is determined by the ego controller in response to the inputs that it is receiving. In some cases, the jump in vehicleposition that is requested by the trajectory determined by the ego controller is sufficiently large so as to cause the ego controller to produce an error.

[0049] When the jump in vehicle position requested by the trajectory is small, the ego vehicle may be seen visually to “stutter”, but with no error reported.

[0050] An objective of embodiments of the present invention is to seek to remove such “vehicle jumps” in simulation of vehicles at level 3 autonomy and above. It will be recognised that when vehicle jumps occur, analysing the simulation for the purpose of testing may not be useful, because it is evident that the simulation has not accurately represented the behaviour of vehicle in the physical world. It is another objective of embodiments of the present invention therefore to improve the quality of testing using simulations of autonomous vehicles.

[0051] Scenarios may be defined and edited in offline mode, where the ego vehicle is not controlled, and then exported for testing in the next stage of a testing pipeline which is described below.

[0052] A scenario comprises one or more agents (sometimes referred to as actors) travelling along one or more paths in a road layout. A road layout is a term used herein to describe any features that may occur in a driving scene, and in particular includes at least one track along which a vehicle is intended to travel in a simulation. That track may be a road or lane or any other driveable path. The scene may comprise one or more road features such as roundabouts or junctions. These other actors / agents are intended to represent real- world entities encountered by the ego vehicle in real-life driving situations. Agents may comprise non-ego vehicles or other road users such as cyclists and pedestrians.

[0053] Example AV Stack

[0054] Figure 1A shows a highly schematic block diagram of an AV runtime stack 100. The stack 100 may be fully or semi-autonomous. For example, the stack 100 may operate as an Autonomous Driving System (ADS) or Advanced Driver Assist System (ADAS).

[0055] The run time stack 100 is shown to comprise a perception (sub-)system 102, a prediction (sub-) system 104, a planning (sub-)system (planner) 106 and a control (sub) system (controller) 108.

[0056] In a real- world context, the perception system 102 receives sensor outputs from an onboard sensor system 110 of the AV, and uses those sensor outputs to detect external agents andmeasure their physical state, such as their position, velocity, acceleration etc. The on-board sensor system 110 can take different forms but generally comprises a variety of sensors such as image capture devices (cameras / optical sensors), lidar and / or radar unit(s), satellitepositioning sensor(s) (GPS etc.), motion / inertial sensor(s) (accelerometers, gyroscopes etc.) etc. The onboard sensor system 110 thus provides rich sensor data from which it is possible to extract detailed information about the surrounding environment, and the state of the AV and any external actors (vehicles, pedestrians, cyclists etc.) within that environment. The sensor outputs typically comprise sensor data of multiple sensor modalities such as stereo images from one or more stereo optical sensors, lidar, radar etc. Sensor data of multiple sensor modalities may be combined using filters, fusion components etc.

[0057] The perception system 102 typically comprises multiple perception components which co-operate to interpret the sensor outputs and thereby provide perception outputs to the prediction system 104.

[0058] The run time stack may comprise different sub stacks or slices. It is possible to select in a simulation context the extent to which the run time stack is to be analysed / tested based on examining different slices of the stack 100. In a simulation context, depending on the nature of the testing - and depending, in particular, on where the stack 100 is sliced - it may or may not be necessary to model the on-board sensor system 100.

[0059] The perception outputs from the perception system 102 are used by the prediction system 104 to predict future behaviour of external actors (agents), such as other vehicles in the vicinity of the AV.

[0060] Predictions computed by the prediction system 104 are provided to the planner 106, which uses the predictions to make autonomous driving decisions to be executed by the AV in a given driving scenario. The inputs received by the planner 106 would typically indicate a drivable area and would also capture predicted movements of any external agents (obstacles, from the AV’s perspective) within the drivable area. The driveable area can be determined using perception outputs from the perception system 102 in combination with map information, such as an HD (high definition) map.

[0061] A core function of the planner 106 is the planning of trajectories for the AV (ego trajectories), taking into account predicted agent motion. This may be referred to as trajectory planning. A trajectory is planned in order to carry out a desired goal within a scenario. The goal could for example be to enter a roundabout and leave it at a desired exit; to overtake avehicle in front; or to stay in a current lane at a target speed (lane following). The goal may, for example, be determined by an autonomous route planner 116, also referred to as a goal generator 116. The planner comprises an interpolated vehicle component that handles the trajectory messages to implement the desired motion of the AV.

[0062] The controller 108 executes the decisions taken by the planner 106 by providing suitable control signals to an on-board actor system 112 of the AV. In particular, the planner 106 plans trajectories for the AV and the controller 108 generates control signals to implement the planned trajectories. Typically, the planner 106 will plan into the future, such that a planned trajectory may only be partially implemented at the control level.

[0063] The actor system 112 includes “primary” vehicle systems, such as braking, acceleration and steering systems, as well as secondary systems (e.g. signaling, wipers, headlights etc.).

[0064] The example of Figure la considers a relatively “modular” architecture, with separable perception, prediction, planning and control systems 102-108. The sub-stack themselves may also be modular, e.g. with separable planning modules within the planning system 106. For example, the planning system 106 may comprise multiple trajectory planning modules that can be applied in different physical contexts (e.g. simple lane driving vs. complex junctions or roundabouts). This is relevant to simulation testing for the reasons noted above, as it allows components (such as the planning system 106 or individual planning modules thereof) to be tested individually or in different combinations. For the avoidance of doubt, with modular stack architectures, the term stack can refer not only to the full stack but to any individual sub- system or module thereof.

[0065] The extent to which the various stack functions are integrated or separable can vary significantly between different stack implementations - in some stacks, certain aspects may be so tightly coupled as to be indistinguishable. For example, in other stacks, planning and control may be integrated (e.g. such stacks could plan in terms of control signals directly), whereas other stacks (such as that depicted in Figure 1A) may be architected in a way that draws a clear distinction between the two (e.g. with planning in terms of trajectories, and with separate control optimizations to determine how best to execute a planned trajectory at the control signal level). Similarly, in some stacks, prediction and planning may be more tightly coupled. At the extreme, in so-called “end-to-end” driving, perception, prediction, planning and control may be essentially inseparable. Unless otherwise indicated, the perception, prediction planning andcontrol terminology used herein does not imply any particular coupling or modularity of those aspects.

[0066] A “full” stack typically involves everything from processing and interpretation of low- level sensor data (perception), feeding into primary higher-level functions such as prediction and planning, as well as control logic to generate suitable control signals to implement planning-level decisions (e.g. to control braking, steering, acceleration etc.). For autonomous vehicles, level 3 stacks include some logic to implement.

[0067] Transition demands and level 4 stacks additionally include some logic for implementing minimum risk maneuvers. The stack may also implement secondary control functions e.g. of signaling, headlights, windscreen wipers etc.

[0068] The term “stack” can also refer to individual sub-systems (sub-stacks) of the full stack, such as perception, prediction, planning or control stacks 104, 106, 108, which may be tested individually or in any desired combination. A stack can refer purely to software, i.e. one or more computer programs that can be executed on one or more general-purpose computer processors. It will be appreciated that the term “stack” encompasses software, but can also encompass hardware. In simulation, software of the stack may be tested on a “generic” off- board computer system, before it is eventually uploaded to an on-board computer system of a physical vehicle. However, in “hardware-in-the-loop” testing, the testing may extend to underlying hardware of the vehicle itself. For example, the stack software may be run on the on-board computer system (or a replica thereof) that is coupled to the simulator for the purpose of testing. In this context, the stack under testing extends to the underlying computer hardware of the vehicle. As another example, certain functions of the stack 100 (e.g. perception functions) may be implemented in dedicated hardware. In a simulation context, hardware-in- the loop testing could involve feeding synthetic sensor data to dedicated hardware perception components.

[0069] Within the stack 100, a scenario description 117 may be used as a basis for planning and prediction. The scenario description 117 is generated using the perception system 102, together with a high-definition (HD) map 114. By localizing the ego vehicle 114 on the map, it is possible to combine the information extracted in the perception system 104 (including dynamic agent information) with the pre-existing environmental information contained in the HD map 114. The scenario description 117 is, in turn, used as a basis for motion prediction inthe prediction system 104, and the resulting motion predictions 118 are used in combination with the scenario description 117 as a basis for planning in the planning system 106.

[0070] Example Testing Paradigm

[0071] Figure lb shows a highly schematic overview of a testing paradigm for autonomous vehicles. An ADS / ADAS stack 100, e.g., of the kind depicted in Figure la, is subject to repeated testing and evaluation in simulation, by running multiple scenario instances in a simulator 202, and evaluating the performance of the stack 100 (and / or individual subs-stacks thereof) in a test oracle 252. The output of the test oracle 252 is informative to an expert 122 (team or individual), allowing them to identify issues in the stack 100 and modify the stack 100 to mitigate those issues (S124). The results also assist the expert 122 in selecting further scenarios for testing (S126), and the process continues, repeatedly modifying, testing and evaluating the performance of the stack 100 in simulation. The improved stack 100 is eventually incorporated (S125) in a real- world AV 101, equipped with a sensor system 110 and an actor system 112. The improved stack 100 typically includes program instructions (software) executed in one or more computer processors of an on-board computer system of the AV 101 (not shown). The software of the improved stack is uploaded to the AV 101 at step S125. Step S125 may also involve modifications to the underlying vehicle hardware. On board the AV 101, the improved stack 100 receives sensor data from the sensor system 110 and outputs control signals to the actor system 112. Real-world testing (S128) can be used in combination with simulation-based testing. For example, having reached an acceptable level of performance through the process of simulation testing and stack refinement, appropriate real-world scenarios may be selected (S130), and the performance of the AV 101 in those real scenarios may be captured and similarly evaluated in the test oracle 252.

[0072] Scenarios can be obtained for the purpose of simulation in various ways, including manual encoding. The system is also capable of extracting scenarios for the purpose of simulation from real- world runs, allowing real- world situations and variations thereof to be recreated in the simulator 202.

[0073] Figure 1c shows a highly schematic block diagram of a scenario extraction pipeline. Data 140 of a real-world run is passed to a ‘ground- truthing’ pipeline 142 for the purpose of generating scenario ground truth. The run data 140 could comprise, for example, sensor data and / or perception outputs captured / generated on board one or more vehicles (which could be autonomous, human-driven or a combination thereof), and / or data captured from other sourcessuch external sensors (CCTV etc.). The run data is processed within the ground truthing pipeline 142, in order to generate appropriate ground truth 144 (“trace(s)” and contextual data) for the real-world run. The ground-truthing process could be based on manual annotation of the ‘raw’ run data 140, or the process could be entirely automated (e.g. using offline perception method(s)), or a combination of manual and automated ground truthing could be used. For example, 3D bounding boxes may be placed around vehicles and / or other agents captured in the run data 140, in order to determine spatial and motion states of their traces. A scenario extraction component 146 receives the scenario ground truth 144, and processes the scenario ground truth 144 to extract a scenario description 148 that can be used for the purpose of simulation. The scenario description 148 is consumed by the simulator 202, allowing multiple simulated runs to be derived therefrom. Ground truth 150 is provided for each simulated run.

[0074] A “trace” is a history of an agent’s location and motion over the course of a scenario. There are many ways a trace can be represented. Trace data will typically include spatial and motion data of an agent within the environment. The term is used in relation to both real scenarios (with real- world traces) and simulated scenarios (with simulated traces).

[0075] The term “perception” generally refers to techniques for perceiving structure in the real- world data 140, such as 2D or 3D bounding box detection, location detection, pose detection, motion detection etc. For example, a trace may be extracted as a time-series of bounding boxes or other spatial states in 3D space or 2D space (e.g. in a birds-eye-view frame of reference), with associated motion information (e.g. speed, acceleration, jerk etc.). In the context of image processing, such techniques are often classed as “computer vision”, but the term perception encompasses a broader range of sensor modalities.

[0076] Reference is made to Figure Id to explain the interaction between core simulator software 1000 and an ADAS stack system under test 1002. Open Standard Interface (OSI) messages are generated by the core simulator software 1000 to core_sim integration software 1001. The integration software 1001 generates stack specific interface messages to communicate to the ADAS stack system under test 1002. In certain embodiments, the core simulator software may be genie simulator software. In that case, the integration software may be genie_sim integration software.

[0077] Example Testing Pipeline

[0078] Further details of an example testing pipeline incorporating the test oracle 252 will now be described. The examples that follow focus on simulation-based testing. However, as noted,the test oracle 252 can equally be applied to evaluate stack performance on real scenarios, and the relevant description below applies equally to real scenarios. The following description refers to the stack 100 of Figure la by way of example. However, as noted, the testing pipeline 200 is highly flexible and can be applied to any stack or sub-stack operating at any level of autonomy.

[0079] Figure 2 shows a schematic block diagram of the testing pipeline, denoted by reference numeral 200. The testing pipeline 200 is shown to comprise the simulator 202 and the test oracle 252. The simulator 202 runs simulated scenarios for the purpose of testing all or part of an AV run time stack 100, and the test oracle 252 evaluates the performance of the stack (or sub-stack) on the simulated scenarios. As discussed, it may be that only a sub-stack of the runtime stack is tested, but for simplicity, the following description refers to the (full) AV stack 100 throughout. However, the description applies equally to a sub-stack in place of the full stack 100. The term “slicing” is used herein to the selection of a set or subset of stack components for testing.

[0080] The idea of simulation-based testing is to run a simulated driving scenario that an ego agent must navigate under the control of the stack 100 being tested. Typically, the scenario includes a static drivable area (e.g. a particular static road layout) that the ego agent is required to navigate, typically in the presence of one or more other dynamic agents (such as other vehicles, bicycles, pedestrians etc.). To this end, simulated inputs 203 are provided from the simulator 202 to the stack 100 under testing.

[0081] The slicing of the stack dictates the form of the simulated inputs 203. By way of example, Figure 2 shows the prediction, planning and control systems 104, 106 and 108 within the AV stack 100 being tested. To test the full AV stack of Figure 1A, the perception system 102 could also be applied during testing. In this case, the simulated inputs 203 would comprise synthetic sensor data that is generated using appropriate sensor model(s) and processed within the perception system 102 in the same way as real sensor data. This requires the generation of sufficiently realistic synthetic sensor inputs (such as photorealistic image data and / or equally realistic simulated lidar / radar data etc.). The resulting outputs of the perception system 102 would, in turn, feed into the higher- level prediction and planning systems 104, 106.

[0082] By contrast, so-called “planning-level” simulation would essentially bypass the perception system 102. The simulator 202 would instead provide simpler, higher-level inputs 203 directly to the prediction system 104. In some contexts, it may even be appropriate toFive Al447076PCT bypass the prediction system 104 as well, in order to test the planner 106 on predictions obtained directly from the simulated scenario (i.e. “perfect” predictions).

[0083] Between these extremes, there is scope for many different levels of input slicing, e.g. testing only a subset of the perception system 102, such as “later” (higher-level) perception components, e.g. components such as filters or fusion components which operate on the outputs from lower-level perception components (such as object detectors, bounding box detectors, motion detectors etc.).

[0084] As an alternative to synthetic sensor data, all or part of the perception system 102 may be modelled, e.g. using one or more perception error models to introduce realistic error into the simulated inputs 203. For example, Perception Statistical Performance Models (PSPMs) or, synonymously, “PRISMs” may be used. Further details of the principles of PSPMs, and suitable techniques for building and training such models, may be bound in International Patent Publication Nos. WO2021037763 W02021037760, WO2021037765, WO2021037761, and WO2021037766, each of which is incorporated herein by reference in its entirety.

[0085] Whatever form they take, the simulated inputs 203 are used (directly or indirectly) as a basis for decision-making by the planner 108. The controller 108, in turn, implements the planner’s decisions by outputting control signals 109. In a real- world context, these control signals would drive the physical actor system 112 of AV. In simulation, an ego vehicle dynamics model 204 is used to translate the resulting control signals 109 into realistic motion of the ego agent within the simulation, thereby simulating the physical response of an autonomous vehicle to the control signals 109.

[0086] Alternatively, a simpler form of simulation assumes that the ego agent follows each planned trajectory exactly between planning steps. This approach bypasses the control system 108 (to the extent it is separable from planning) and removes the need for the ego vehicle dynamic model 204. This may be sufficient for testing certain facets of planning.

[0087] To the extent that external agents exhibit autonomous behaviour / decision making within the simulator 202, some form of agent decision logic 210 is implemented to carry out those decisions and determine agent behaviour within the scenario. The agent decision logic 210 may be comparable in complexity to the ego stack 100 itself or it may have a more limited decision-making capability. The aim is to provide sufficiently realistic external agent behaviour within the simulator 202 to be able to usefully test the decision-making capabilities of the ego stack 100. In some contexts, this does not require any agent decision making logic 210 at all(open-loop simulation), and in other contexts useful testing can be provided using relatively limited agent logic 210 such as basic adaptive cruise control (ACC). One or more agent dynamics models 206 may be used to provide more realistic agent behaviour if appropriate.

[0088] A scenario is run in accordance with a scenario description 201, which typically has both static and dynamic elements. The static element(s) typically include a static road layout. Static road layouts may be defined in maps, which may be constructed with high precision to represent a real-world road layout. The dynamic element(s) typically include one or more external agents within the scenario, such as other vehicles, pedestrians, bicycles etc. Scenario runs are orchestrated by a test orchestration component 260.

[0089] The extent of the dynamic information provided to the simulator 202 for each external agent can vary. For example, a scenario may be described by separable static and dynamic layers. A given static layer (e.g. defining a road layout) can be used in combination with different dynamic layers to provide different scenario instances. The dynamic layer may comprise, for each external agent, a spatial path to be followed by the agent together with one or both of motion data and behaviour data associated with the path. In simple open-loop simulation, an external actor simply follows the spatial path and motion data defined in the dynamic layer that is non-reactive i.e. does not react to the ego agent within the simulation. Such open-loop simulation can be implemented without any agent decision logic 210. However, in closed-loop simulation, the dynamic layer instead defines at least one behaviour to be followed along a static path (such as an ACC behaviour). In this case, the agent decision logic 210 implements that behaviour within the simulation in a reactive manner, i.e. reactive to the ego agent and / or other external agent(s). Motion data may still be associated with the static path but in this case is less prescriptive and may for example serve as a target along the path. For example, with an ACC behaviour, target speeds may be set along the path which the agent will seek to match, but the agent decision logic 210 might be permitted to reduce the speed of the external agent below the target at any point along the path in order to maintain a target headway from a forward vehicle.

[0090] The output of the simulator 202 for a given simulation includes an ego trace 212a of the ego agent and one or more agent traces 212b of the one or more external agents (traces 212). Each trace 212a, 212b is a complete history of an agent’s behaviour within a simulation having both spatial and motion components. For example, each trace 212a, 212b may take theform of a spatial path having motion data associated with points along the path such as speed, acceleration, jerk (rate of change of acceleration), snap (rate of change of jerk) etc.

[0091] Additional information is also provided to supplement and provide context to the traces 212. Such additional information is referred to as “contextual” data 214. The contextual data 214 pertains to the physical context of the scenario, and can have both static components (such as road layout) and dynamic components (such as weather conditions to the extent they vary over the course of the simulation).

[0092] Scenarios for use by a simulation system as described above may be generated in a scenario builder. An example of a scenario builder is given later.

[0093] Reference is now made to Figure 3a to describe the sequence of events in a simulation platform which exhibits a problem with “vehicle jumps”.

[0094] At step S30, the components of the runtime stack 100 are started up.

[0095] At step S32, the simulator starts, including the core simulator 1000 and the integration software 1001.

[0096] At step S34, the stack starts to receive simulated perception inputs fed from the simulator. By way of example, these perception inputs may be in accordance with the association for standardisation of automation and measuring systems (AS AM) open simulation interface (OSI) format. The ASAM OSI is a format which provides compatibility between automated driving functions and simulation frameworks. Depending on the nature of the simulator software and the nature of the stack software, there may be a requirement for conversion between OSI, the simulator software and the stack software to put the messages into a format that the stack can process. In the present embodiments, this may be done by integration software 1001.

[0097] The perception inputs are provided in the form of messages. The messages which are provided to the stack are repeated with the initial state. For example, the initial state may comprise a certain ego velocity and a certain ego position. By way of illustration, an ego velocity may be 25 metres per second and its position may be a starting position in cartesian co-ordinates, x=0, y=0. It is further noted that at this point the ego stack is not receiving any other information apart from its own status. That is, as the scenario is not yet playing, no other actors are published at this stage. In this context, a published actor is an actor that is provided in a scenario as it plays. The scenario parameters may be provided to the stack in a ground truthmessage. At this point therefore the ground truth messages are empty other than the ego status. At step S36, the ego stack “settles down” and starts generating trajectory messages. Each trajectory message defines a trajectory to be acted on by the ego vehicle in the simulation. Note that at this point, the stack is receiving the same repeated initial ground truth message with only the ego state and no other agents.

[0098] At step S38, the simulator integration software 1001 is receiving the trajectory messages from the stack and sends a message “Car Stack Status = READY” to the core simulator 1000. The simulator software enters an ignition phase. At a subsequent step S40 (a period of time after step S38, where this period of time may be very short), the core simulator sends to the integration software a “Car Stack Engage” message. At this point, the scenario starts to run. A scenario status message is recorded, e.g., for future analysis of the trace data, with a status set to “playing”. The “Car Stack Engage” message is marked with a time stamp at the end of the period of time to step S40. The scenario status message has the same time stamp as the car stack engage message. At this point, the ground truth message which is being supplied to the stack is also populated with other actors in the scenario. Once a scenario is playing, all agents in the scenario are moving.

[0099] In step S40, the integration software engages the stack. In this step, the interpolated vehicle component of the stack starts accepting trajectory messages and stepping through them.

[0100] The above process presents two potential problems.

[0101] At step S34, the simulator is outputting only the status of the ego vehicle in the ground truth messages. The potential dynamic actors of the scenario are masked from view (they are not published) and so are not provided as detections to the stack.

[0102] Similarly, at step S34, the simulator is reporting the ego vehicle in the scene with the velocity that the ego will take at the start of the scenario, but the vehicle is not actually moving until engagement (in step S40). Presently, the problem which may arise from only the status of the ego vehicle being output in the ground truth messages initially is circumvented by the design of the scenarios which are being simulated. Currently, scenarios are not written to have the ego vehicle starting close to dynamic actors of the scene. That is, there is a bit of “lead time” in the simulation to allow the ego vehicle to “catch up” with the scene before it starts to perceive moving actors which might affect its decision making. This circumvention, however, means that it is not currently possible to verify the performance if a driver attempts to engage when presently on a trajectory to collide with a slower moving vehicle in front. The solutionafforded herein by embodiments of the present invention enables such a test to be carried out , as well as other scenarios where the ego vehicle may encounter other moving actors at the initiation of a scenario.

[0103] The second problem arises because the trajectory planner of the ego stack has been given a non-zero velocity (in the example 25 metres per second), and therefore the ego vehicle can reasonably assume that it is moving. However, in between steps, it has in fact remained at the same pose (because it is not yet engaged by the simulator). If the trajectory planner is making its plans to start from where the vehicle is predicted to be at the next step, perhaps including some small history to cover error, then the first point on the trajectory will potentially be far from the actual current vehicle position. This may lead to an invalid trajectory jump being caused immediately as the simulator engages the stack in S40.

[0104] Such a jump may be small, and the ego vehicle will only be seen visually to stutter, but without the report of an error. However, in other cases, the jump in vehicle position requested by the trajectory messages may be sufficiently large so as to cause the interpolated vehicle step to throw an error (Trajectory Broken) error. This may cause the simulator to exit and cease simulation altogether. Either way, no useful test may be carried out when jumps of this kind occur in a simulation.

[0105] A solution to these issues is discussed herein. These problems may be addressed by implementing a “hot handover” function. As used herein, a “hot handover” means having the ego vehicle moving and the scenario playing before the engagement of the stack occurs. This makes the test much more like a real- world test where a driver is operating the vehicle up to the point of engagement. Movement of the vehicle is accomplished by initially having the ego vehicle under the control of the simulator in scenario runtime.

[0106] The length of time that the vehicle should be moving prior to engagement may be an attribute of the particular vehicle stack under test. Some stacks may be stateless and may be able to engage “cold”, but this is unlikely in many scenarios. In the present embodiments, a designer of the scenario is able to specify a pre-engagement trajectory and a handover point. Figure 4 illustrates a new initialisation sequence which implements a hot handover. In Figure 4, steps S40 and S42 corresponds to steps S30 and S32 of the initialisation sequence of Figure 3. That is, in Step S40, the stack components are started up, and in step S42, the simulator starts up.

[0107] In step S44, the stack receives inputs. However, these inputs differ from the inputs in the initialisation sequence of Figure 3a. In the present initialisation sequence of Figure 4, all dynamic actors are present in the ground truth messages which are input to the stack. As described later , these messages are used by the simulator itself which controls the ego vehicle. Note that this contrasts with the situation described above in Figure 3a, where only the ego state was supplied in the ground truth message to the stack at the outset.

[0108] In step S46, the stack “settles down” and starts generating trajectory messages. Note that these trajectory messages may be incorrect due to the fact that dynamic objects have been reported to the stack, but are not actually moving because the scenario has not yet started.

[0109] In step S48, the simulator determines that the stack is ready and receives a Car Stack Status = READY message. In step S50, the simulator starts driving the ego vehicle in a manner consistent with the motion model of the ego stack , and playing the scenario.

[0110] At step S52, it is determined whether or not the simulation has reached an ENGAGE action point defined by the scenario designer. If it has not, the simulator continues to control the ego vehicle. If it has, the simulator sends a “Car Stack Engage” message and stops controlling the ego vehicle, as represented by step S54.

[0111] At step S56, the simulator then engages the stack of the ego vehicle and the interpolated vehicle component of the stack starts accepting the trajectories and stepping along them.

[0112] Figure 5 is a more detailed diagram about the sequence of events and message exchange described above with reference to Figure 4.

[0113] Figure 5 represents an ADAS (Advanced Driver Assist Stack), integration software (genie_sim) , and a core (scenario) simulator. Each is represented by a vertical line, wherein arrows extending horizontally from each line indicate outbound and inbound messaging between the three components, depending on a direction of the arrow. For example, the genie simintegration software 1001 is configured to message the scenario simulator 1000 to establish a connection therebetween. The scenario simulator is configured to communicate to the genie_ sim integration software that it is ready for simulation. The genie_sim may optionally reply to the scenario simulator with configuration details.

[0114] As part of a communication loop between the genie simulator and the scenario simulator, the genie may provide traffic update communications to the scenario simulator,Five Al447076PCT which may reply with ground truth information pertaining to the scenario. Note that these will not be further described herein as they are not pertinent to this invention

[0115] The ADAS stack may provide trajectory messages to the genie, indicating that the ADAS stack is ready. The genie_sim may communicate with the scenario simulator to confirm that the ADAS stack is ready, and the scenario simulator may begin scenario playback by messaging the genie_sim with an indication that the scenario is being played back.

[0116] The simulator can control the ego vehicle using a version of the ego stack fit for that purpose. For example, a “basic driver” controller may be applied by the simulator. For this purpose, the stack software of the basic driver controller is implemented as part of the simulator itself and is used to control the actions of the ego vehicle. The basic driver controller moves the vehicle step by step to decide what to do (e.g., brake to avoid collisions). To achieve this , the basic driver uses an interpolated vehicle controller provided in the simulator which matches or substantially matches the interpolated vehicle component of the stack of the ego vehicle . In summary ,It knows its target trajectory, and moves the vehicle along it reporting the position frame by frame to the stack.

[0117] Note that the interpolated vehicle component of that stack may be running - and planning what it would be doing - but the stack is not engaged, so the simulator is controlling movement of the vehicle at this stage .

[0118] When the simulator engages the stack at the engage point, there is a handshake mechanism to ensure that the simulator knows that the ego stack is now taking over responsibility for controlling the ego vehicle from the simulator itself.

[0119] In certain embodiments, the ego vehicle is controlled by the simulator in all situations until a Car Stack Engage message is sent. A pre-engagement trajectory may be specified via a route / path, or a default agent control may be implemented by the simulator (for example, a basic driver stack).

[0120] When scenarios are created, a designer of a scenario may define the “engage” point. For example, this could be defined as a handover time from the scenario start. The Engage Point could be entered into a handover time field on a scenario editor. The simulator will hand over control to the ego controller when this time is reached from the start of the scenario.Five Al447076PCT

[0121] In an alternative embodiment, a user may be given a Car Stack Engage Car Action that they can add at any point in the scenario. When this action is triggered (for example by a condition on an act), then the Car Stack Engage message will be sent. The first option (handover time field) has the advantage that it is visible to the author of scenarios so they can more readily determine at what point the ego vehicle is under its own control in the simulation. The car stack engage action also realises this same benefit, as it is seen by the scenario author. In both examples, a condition is selected which defines a point at which to hand over control.

[0122] The Car Stack Engage Start Action embodiment has the advantages that by introducing an explicit requirement of a hot handover action being added into the scenario, there is backwards compatibility and only scenarios which have been explicitly selected for hot handover will have that behaviour. Other scenarios may be executed by the simulator without needing to change the simulator.

[0123] This option gives total control to a creator of a scenario by allowing them to set the engagement point. Engagement is part of the “test” when a hot handover is implemented. In real life, when a hot handover is implemented, then the safety driver has to make a specific decision about where / when to engage. It is possible to consider that the scenario runtime takes the place of a safety driver in a simulation context.The handover condition could be defined as any configurable state of the scenario which is being implemented by the simulator. For example, examples of simulator states are given in the Asam Open scenario document at the following locations: https: / / publications.pages.asam.net / standards / ASAM_OpenSCENARIO / ASAM_Ope nSCENARIO_XML / latest / generated / content / EntityCondition.htmlThese examples include: endOfRoadCondition Condition checking for how long the reference entity has reached the end of the road. collisionCondition Condition checking whether the reference entity was involved in a collision.Five Al 447076PCT offroadCondition Condition checking for how long the reference entity has left the road. timeHeadwayCondition Condition checking the time headway between two entities. timeToCollisionCondition Condition checking the time to collision between two entities. accelerationcondition Condition checking the current acceleration of an entity. standStillCondition Condition checking for how long the reference entity has not moved. speedCondition Condition checking the current speed of the referenced entities. relativeSpeedCondition Condition checking the relative speed between two entity. traveledDistanceCondition Condition checking the total traveled distance of the reference entity since the start of the scenario. reachPo sitionCondition Condition checking whether the reference entity has reached a given position within a given uncertainty. distanceCondition Condition checking the distance between two entities or an entity and a position. relativeDistanceCondition Condition checking the relative distance between two entities. relativeClearanceCondition Condition checking the relative clearance of an entity. angleCondition Condition checking the angle of an entity. relativeAngleCondition Condition checking the angle of an entity relative to a reference entity.

[0124] Currently, in order to implement a simulation, a controller for each vehicle is assigned during initialisation. In some examples, for non-ego entities (the other actors), the controllers may be basic driver, external and hitched (for trailers). The controller for the ego vehicle is hidden, but may be similar to the external controller. To support hot handover, the selection of the ego controller is allowed in an initiation. A change controller action is introduced. To support a hot handover, a user sets up the ego as controlled using the basic controller, implemented as part of the simulator, and then uses the change controller action to switch it to the external controller at the desired point in the story board of the scenario, triggered by any existing condition action.

[0125] Figure 3b shows a user interface for the initialisation process of Figure 3a. After the initialisation period, the ego is created in the scenario editor with an ego controller. As mentioned with respect to Figure 3a, when the scenario is run this causes an OSI message to be sent for the Car Stack to engage.

[0126] Figure 6 shows a user interface of a scenario editor for creating a scenario which implements hot handover. Hot handover is possible by aligning the simulator to control the ego vehicle until one or more given condition is met and the ego controller stack itself is activated. Activation causes the car stack to be sent the car stack engage message over OSI as described above.

[0127] Figure 6 illustrates a user interface showing an initialisation period 600, and then a period 602 under which the car is under the control of the ego controller. A switch button 60 is provided on the user interface which has the effect of switching the ego controller to the simulator control (e.g., basic driver). The user interface also provides an ego controller box. When the switch button 60 is actuated a selection box of controller options appears in the same page just next to the switch button. The controller options include “basic driver”. When “basic driver” is selected, the heading of the ego controller box changes, and the displayed contents are configuration options for the “basic driver” selection.

[0128] Figure 7 illustrates a display on the user interface of the scenario editor which allows appropriate conditions of act to be included. An “add new act” button 70 enables a new act, Act 72, to be added. Start conditions are defined for Act 1. In this case, a length of time is set using an “add new start conditions” button 74 to define when the control of the ego vehicle will switch from the simulator controller to the ego controller. For example, as shown by the timeline 76 in Figure 7, in this example the time is set at 5 seconds.

[0129] In the above example, the ego vehicle is controlled by the simulator until the conditions in Act 1 are met, when it will switch to the ego controller which sends a Car Stack Engage message to the car stack over OSI. In this example, the conditions define a time of 5 seconds.

[0130] Figure 8 shows a highly schematic block diagram of a computer implementing a scenario builder, which comprises a display unit 510, a user input device 502, computer storage such as electronic memory 500 holding program code 504, and a scenario database 508. The user interfaces and their functionality described above with reference to Figure 7 can be incorporated into the scenario builder described herein.

[0131] The program code 504 is shown to comprise modules configured to receive user input and generate output to be displayed on the display unit 510. User input entered to a user input device 502 is received by an editor interface 512. A scenario model module 506 is then configured to receive the user input from the editor interface 512 and to generate a scenario to be simulated.

[0132] The scenario model data is sent to a scenario description module 7201, which comprises a static layer 7201a and a dynamic layer 7201b. Thestatic layer 7201a includes the static elements of a scenario, which would typically include a static road layout, and the dynamic layer 7201b defines dynamic information about external agents within the scenario, such as other vehicles, pedestrians, bicycles etc. Data from the scenario model 506 that is received by the scenario description module 7201 may then be stored in a scenario database 508 from which the data may be subsequently loaded and simulated. Data from the scenario model 506, whether received via the editor interface or the scenario database, is sent to the scenario runtime module 516, which is configured to perform a simulation of the parametrised scenario. Output data of the scenario runtime is then sent to the scenario visualisation module 514, which is configured to produce data in a format that can be read to produce a dynamic visual representation of the scenario. The output data of the scenario visualisation module 514 may then be sent to the display unit 510 whereupon the scenario can be viewed, for example in a video format.

[0133] Reference is made to Figure 9, which shows three plan-view instances 901a-c of an ego vehicle 903 during a handover sequence. The first instance 901a shows the ego vehicle 903 at a first point in time, before the handover is completed. Four simulation control arrows 905 are shown to project outwardly from the ego vehicle 903 in respective left, right, forward, and backward directions relative to the ego vehicle. The simulation control arrows 905 indicate that dynamic behaviours of the ego vehicle 903 are under control of the simulator.

[0134] A first UI element 915 corresponding to the first instance 901a is shown in Figure 9. The first UI element 915 comprises text indicating that the simulator is in control of the ego driver. In some examples the first UI element 915 may be surfaced in a scenario builder user interface to indicate simulator control of the ego vehicle at the first point in time.

[0135] A first boundary line 911 indicates a transition between the first instance 901a and a second instance 901b of the ego vehicle. The second instance 901b represents the ego vehicle 903 at a second point in time, wherein a handover sequence is in progress. That is, the second instance 901b represents the ego vehicle 903 during a ‘handshake’ between the simulator and the ego controller, at which point dynamic control of the ego vehicle 903 is transferred from the simulator to the ego controller.

[0136] With reference again to Figure 4, in particular step S54, the simulator sends a ‘Car Stack Exchange message and stops controlling the ego vehicle, passing control of the ego vehicle 903 to the ego stack (e.g., ADS). The second instance 901b may represent a point within ahandover sequence at which the stack exchange message is sent, and control is passed between the simulator and the ego vehicle stack.

[0137] Second and third UI elements 917, 919 corresponding to the second instance 901b are shown in Figure 9. The second UI element 917 comprises a field indicating the second point in time, at which the transfer in dynamic control occurs. In the example of Figure 9, the second UI element 917 indicates that transfer is to occur after 5 seconds in the scenario. When rendered on a scenario builder UI, the second UI element 917 may be an interactive UI element. The second UI element 917 may be configured to receive user input indicating a parameter value, and to use that value to define the scenario time at which point the transfer in dynamic control of the ego vehicle occurs.

[0138] The third UI element 919 comprises text indicating that the dynamic control of the ego vehicle is being switched from the simulator to the ego controller. In some examples the third UI element 919 may be surfaced in a scenario builder user interface to indicate transfer of control of the ego vehicle at the second point in time.

[0139] In the second instance 901b, four simulation control arrows 905 and four ego control arrows 907 are shown. One simulation arrow 905 and one ego control arrow 907 project outwardly from the ego vehicle 903 in each of a left, right, forward, and backward direction, relative to the ego vehicle 903. As in the first instance 901a, the simulation control arrows 905 indicate that dynamic behaviours of the ego vehicle 903 are under control of the simulator. The ego control arrows 907 indicate that dynamic behaviours of the ego vehicle 903 are under control of the ego controller. In each direction away from the ego vehicle, a handshake icon 909 is also shown between the respective simulation 905 and ego control 907 arrows. The handshake icons 909 indicate that control of the ego vehicle is being transferred at the second point in time represented by the second instance 901b.

[0140] A second boundary line 913 indicates a transition between the second instance 901b and a third instance 901c of the ego vehicle 903. The third instance 901c represents the ego vehicle 903 at a third point in time, wherein a handover sequence is completed and the ego vehicle 903 is under autonomous control of the ego controller.

[0141] In the third instance, control by the ego controller is represented by four ego control arrows 907, again projecting outwardly from the ego vehicle 903 in each of a left, right, forward, and backward direction, relative to the ego vehicle 903.

Claims

CLAIMS:

1. A method of simulating behaviour of an autonomous vehicle to test a control stack for controlling the autonomous vehicle, the method comprising: providing to a simulator a scenario in which an ego vehicle is under observation; initiating operation of the simulator to execute the scenario and simulating an initial phase of a simulation of the scenario with the ego vehicle under the control of a first control stack implemented by the simulator; detecting that a handover condition is met in the scenario; and relinquishing control of the ego vehicle by the first control stack and causing a second control stack to be engaged to control the ego vehicle in a subsequent phase of the simulation, wherein the second control stack is the control stack under test.

2. The method of claim 1 wherein the second control stack receives simulator input from the scenario after simulator operation is initiated and before the handover condition is met and generates control stack trajectory messages using the simulator input, wherein the trajectory messages are not used to control the ego vehicle while the ego vehicle is under control of the first control stack.

3. The method of claim 1 or 2 wherein the first control stack implemented by the simulator generates simulator trajectory messages which are used to control the ego vehicle until the handover condition is met.

4. The method of claim 1 or 2 wherein when the handover condition is met in the scenario a handover message exchange is performed between the simulator and the second control stack.

5. The method of claim 4 wherein the handover message exchange includes a ready message which is issued from the second control stack when it is ready to take over control of the ego vehicle, and wherein the ready message is sent to the simulator indicating the ready status whereby the second control stack is engaged.

6. The method of any preceding claim wherein the handover condition is an engage action point configurable by a creator of the scenario.Five Al447076PCT7. The method of claim 5 wherein the engage action point is selected from: a predetermined time from the start of the scenario; a position of the ego vehicle in the scenario; a position of another particular actor in the scenario; a state of a traffic control component; distance travelled by the ego vehicle from a selected location in the scenario.

8. The method of any preceding claim wherein the simulator provides perception inputs according to a standard protocol to integration software which is configured to convert the perception inputs to a stack specific protocol for controlling the ego vehicle .

9. A computer system for simulating behaviour of autonomous vehicle, the computer system comprising: one or more hardware processor programmed to execute a simulator of a scenario in which an ego vehicle is under observation; and computer memory holding computer code executable to execute the method of any preceding claim.

10. A computer program product comprising computer code executable by one or more hardware processor to implement the method of any of claims 1 to 8.

11. A computer-implemented method for generating a simulation environment for testing an autonomous vehicle, the method comprising: generating a scenario by rendering on a display of a computer device a timeline for execution of the scenario in the simulator; displaying on the timeline an initial phase of a simulation with an indication of a first control stack for controlling the ego vehicle in the simulation; detecting that a switch function has been selected by a user and causing display of a condition input field to be displayed to a user responsive to selection of the switch function; receiving into the condition field user input defining a handover condition in the scenario; and defining in the scenario that when the handover condition is reached in the scenario control of the ego vehicle by the first control stack will be caused to be handed over to asecond control stack in a subsequent phase of the simulation, wherein the second control stack is a control stack of the ego vehicle under test.

12. The method of claim 11 wherein the handover condition is an engage action point configured by a creator of the scenario.

13. The method of claim 11, wherein the engaged action point is selected from: a predetermined time from the start of the scenario; a position of the ego vehicle in the scenario; a position of another particular actor in the scenario; a state of a traffic control component; distance travelled by the ego vehicle from a selected location in the scenario.

14. A computer system for simulating behaviour of autonomous vehicle, the computer system comprising: one or more hardware processor programmed to execute a simulator of a scenario in which an ego vehicle is under observation; and computer memory holding computer code executable to execute the method of any of claims 11 to 13.

15. A computer program product comprising computer code executable by one or more hardware processor to implement the method of any of claims 11 to 13.

Images