Creating a simulation environment for AV behavior testing

Abstract

A computer-implemented method for generating scenarios to be executed in a simulation environment for testing the behavior of an autonomous vehicle includes rendering an interactive visualization for editing a scenario model on a display of a computing device. The scenario model includes one or more interactions between an ego-vehicle object and one or more dynamic opponent objects, each of which is defined as a set of time constraints and / or relationship constraints between the dynamic ego-object and at least one of the opponent objects. The scenario model includes a scene topology, and the interactive visualization includes scene objects in which the ego-vehicle and at least one opponent object are displayed in the scene topology. The scenario is associated with a timeline extending in a direction of travel of the ego-vehicle relative to the scene topology. The method includes rendering on the display a timing control for selecting a moment along the timeline in response to a user input, and generating on the display an interactive visualization of the scene topology and scene objects of the scenario to be displayed at the selected moment.

Description

[Technical Field]

[0001] The present disclosure relates to generating scenarios for use in a simulation environment for testing the behavior of autonomous vehicles. [Background technology]

[0002] The field of autonomous vehicles is large and rapidly developing. An autonomous vehicle is a vehicle equipped with sensors and control systems that enable it to operate without human control of its behavior. An autonomous vehicle is equipped with sensors that enable it to recognize its physical environment, including, for example, cameras, RADAR, and LiDAR. An autonomous vehicle is also equipped with a suitably programmed computer that can process data received from the sensors and make safe and predictable decisions based on the situation recognized by the sensors. There are also various aspects for testing the behavior of the sensors and control systems installed in a particular autonomous vehicle or a class of autonomous vehicles.

[0003] Sensor processing may be evaluated in real-world physical installations. Similarly, autonomous vehicle control systems may be tested in the physical world, for example by repeatedly driving known test routes or driving routes with humans on board to manage unpredictable or unknown situations.

[0004] Physical-world testing will continue to be an important factor in testing an autonomous vehicle's ability to make safe and predictable decisions. However, physical-world testing is expensive and time-consuming. Therefore, there is an increasing reliance on testing in simulated environments. As testing in simulated environments increases, it is desirable that such environments be as representative of real-world scenarios as possible. Autonomous vehicles need to be equipped to operate in the same wide variety of situations in which a human driver may operate. These situations may involve a high level of unpredictability.

[0005] Testing the behavior of an autonomous vehicle in all scenarios it may encounter while driving is not achievable from physical testing. Increasing attention is being paid to creating simulation environments that can provide such testing with confidence that the test results represent the potential real-world behavior of the autonomous vehicle.

[0006] For effective testing in a simulated environment, the autonomous vehicle under test (self-vehicle) must be able to know its location at any given moment, understand its situation (based on simulated sensor inputs), and make safe and predictable decisions about how to navigate the environment to reach a pre-programmed destination.

[0007] The simulated environment needs to be able to represent real-world factors that may change, which may include weather conditions, road types, road structures, road layouts, types of junctions, etc. This list is not exhaustive as there are many factors that may affect the behavior of the ego-vehicle. Summary of the Invention [Problem to be solved by the invention]

[0008] The present disclosure addresses particular challenges that may arise in simulating the behavior of actors in a simulated environment that the ego-vehicle will be operating in. Such actors may be other vehicles, but also other types of actors, such as pedestrians, animals, cyclists, etc.

[0009] A simulator is a computer program that, when executed by a suitable computer, enables the development and testing in simulation of sensor-equipped vehicle control modules prior to the construction and testing of physical equivalents. The simulator provides a sensor simulation system that models the various sensors that an autonomous vehicle may have. The simulator also provides a three-dimensional environment model that reflects the physical environment in which the autonomous vehicle may operate. The 3D environment model defines at least the road network in which the autonomous vehicle is intended to operate and other actors in the environment. In addition to modeling the behavior of the self-vehicle, the behavior of these actors must also be modeled.

[0010] Simulators generate test scenarios (or handle scenarios provided to them). As explained above, it is important for simulators to be able to generate many different scenarios in which the autonomous vehicle can be tested. Such scenarios may involve different behaviors of the actors. The large number of factors involved in each decision to which an autonomous vehicle must respond, and the many other requirements imposed on these decisions (safety and comfort, as two examples), mean that it is impossible to write a scenario for every situation that needs to be tested. Nevertheless, it is necessary to attempt to enable simulators to efficiently provide as many scenarios as possible and to ensure that these scenarios are as close to the real world as possible. If tests performed in a simulation do not generate outputs that are faithful to the outputs generated in the corresponding physical-world environment, the value of the simulation is significantly diminished.

[0011] Scenarios may be generated from live scenes recorded during real-life driving. Such scenes could be marked to identify actual driving paths and be used in the simulation. The test generation system could generate new scenarios, for example, by taking elements (such as road layouts and actor behaviors) from existing scenarios and combining them with other scenarios. Additionally or alternatively, scenarios may be generated randomly.

[0012] However, there is an increasing need to tailor scenarios to specific situations so that specific sets of factors can be generated for testing, and it is desirable that such scenarios can specify the behavior of actors. [Means for solving the problem]

[0013] One aspect of the present disclosure addresses these challenges. According to one aspect of the present invention, there is provided a computer-implemented method for generating scenarios to be executed in a simulation environment for testing the behavior of an autonomous vehicle, the method comprising: Rendering an interactive visualization for editing on a display of a computing device for a scenario model comprising one or more interactions between an ego-vehicle object and one or more dynamic opponent objects, each interaction being defined as a set of time constraints and / or relationship constraints between at least one of the dynamic ego-object and opponent objects, wherein the scenario model comprises a scene topology, the interactive visualization includes scene objects in which the ego-vehicle and at least one opponent object are displayed in the scene topology, and the scenario is associated with a timeline extending in the direction of travel of the ego-vehicle relative to the scene topology; Rendering on the display, in response to user input, timing controls for selecting a moment along the timeline; generating on a display an interactive visualization of the scene topology and scene objects of the scenario as they appear at the selected moment; A computer-implemented method is provided, comprising:

[0014] In some embodiments, the timing controls are implemented by "handles" rendered in a user interface that a user can interact with to move between time instances along a timeline. In this manner, a user can select a particular time instance at which an interactive visualization of the scenario's scene topology and scene objects is displayed. In certain cases, the method may further include rendering a visual representation of a timeline associated with the timing controls on the display. In this case, a user may interact with the handle to move the handle along the visually represented timeline. In other cases, a timeline need not be visually represented, and a user may still interact with the handle to move the handle between time instances to select a particular time instance.

[0015] In an alternative embodiment, a visual representation of a map version of a scenario may be presented to a user on a display. In the map version of the scenario, each time instance may be visually represented by a particular location on the map view. Each location visually represented on the map view may correspond to a particular time instance. A user may select a particular time instance by interacting with a location represented on the map view.

[0016] The method may further include presenting to the user on the display a map view in which at least one selectable location is rendered on the map corresponding to the selectable moment in time.

[0017] The method may further include, in response to a selection of a moment, rendering a dynamic visualization of the scene according to the scenario model from the selected moment.

[0018] The method may include displaying an interactive visualization at an initial moment prior to selection of the moment, and in response to selection of the selected moment, rendering on the display a new interactive visualization of the scene at the selected moment without rendering views of the scene at moments between the initial moment and the selected moment.

[0019] In this method, the step of rendering the dynamic visualization of the scene may further occur automatically in response to a selection of a moment.

[0020] This method is displaying to a user in an editing user interface of the computing device a set of time and / or relationship constraints that govern one or more of the interactions presented in the scenario, and accepting user input to edit one or more of the set of time and / or relationship constraints for each of the one or more of the interactions; regenerating and rendering on a display a new interactive visualization of the scenario comprising the one or more edited interactions; It may further include:

[0021] The selected moment may be later in the timeline than the first moment.

[0022] The selected moment may be earlier than the first moment in the timeline.

[0023] The method may further include defining, in the scenario, a start constraint that triggers the interaction, and rendering on the display a first visual representation of a set of moments prior to the start constraint and a second visual representation of a set of moments during the interaction.

[0024] The method may further comprise the step of presenting to the user on the display a playback control which, when selected by the user, causes the dynamic visualization of the scenario to be played back from the currently selected moment in time.

[0025] The method may further include presenting to the user on the display playback controls that, when selected by the user, cause the dynamic visualization of the scenario to be played back from the beginning of the scenario.

[0026] According to another aspect of the present invention, there is provided a computer system for generating scenarios to be executed in a simulation environment for testing the behavior of an autonomous vehicle, the computer system comprising: a user interface configured to display an interactive visualization for editing of a scenario model comprising one or more interactions between an ego-vehicle object and one or more dynamic opponent objects, each interaction being defined as a set of time constraints and / or relationship constraints between at least one of the dynamic ego-object and opponent objects, the scenario model comprising a scene topology, the interactive visualization including scene objects in which the ego-vehicle and at least one opponent object are displayed in the scene topology, and the scenario is associated with a timeline extending in a direction of travel of the ego-vehicle relative to the scene topology; rendering in the user interface a timing control for selecting a moment along the timeline in response to user input of a user engaging with the user interface; generating in a user interface an interactive visualization of the scene topology and scene objects of the scenario as they appear at the selected moment; a processor configured to: A computer system is provided, comprising:

[0027] The processor may be configured to generate an interactive visualization from the stored parameters of the scenario model.

[0028] Also provided is a transient or non-transient computer readable medium having stored thereon computer readable instructions that, when executed by one or more processors, perform any of the predetermined methods.

[0029] For a better understanding of the present invention and to show how the same may be carried into effect, reference will now be made by way of example to the accompanying drawings in which: [Brief explanation of the drawings]

[0030] [Figure 1] Figure 1 shows the interaction space of a simulation involving three vehicles. [Figure 2] 10A-10C illustrate graphical representations of cut-in operations performed by an actor vehicle. [Figure 3] FIG. 10 illustrates a graphical representation of a cutout operation performed by a subject vehicle. [Figure 4] FIG. 10 illustrates a graphical representation of a slow-down maneuver performed by an actor vehicle. [Figure 5] FIG. 1 is a high-level schematic block diagram of a computer implementing the scenario builder. [Figure 6] High-level schematic block diagram of an autonomous vehicle runtime stack. [Figure 7] High-level schematic block diagram of a testing pipeline for autonomous vehicle performance during simulation. [Figure 8] FIG. 10 illustrates a graphical representation of the path for an exemplary cut-in maneuver. [Figure 9a] FIG. 1 illustrates a first exemplary user interface for configuring a dynamic layer of a simulation environment according to a first embodiment of the present invention. [Figure 9b] FIG. 10 illustrates a second exemplary user interface for configuring a dynamic layer of a simulation environment according to a second embodiment of the present invention. [Figure 10a]9b shows a graphical representation of an exemplary dynamic layer configured in FIG. 9a with the TV1 node selected. [Figure 10b] 9b shows a graphical representation of an exemplary dynamic layer configured in FIG. 9a with the TV2 node selected. [Figure 11] 9b shows a graphical representation of the dynamic layer as set in FIG. 9a with no nodes selected. [Figure 12] Figure 1 shows a comprehensive user interface through which the dynamic layers of the simulation environment can be parameterized. [Figure 13] FIG. 10 illustrates an exemplary user interface in which the static layer of the simulation environment can be parameterized. [Figure 14a] 9b shows an exemplary user interface with functionality configured to enable and control dynamic visualization of the parameterized scenario, as shown in FIG. 9a, showing the scenario at the start of the first operation. [Figure 14b] 14a , showing the same exemplary user interface as FIG. 14a , but illustrating a scenario during parameterization operation where time has passed since the instance of FIG. 14a and the parameterized vehicle has moved to reflect a new position after that time. [Figure 14c] FIG. 14B is the same exemplary user interface as FIGS. 14A and 14B, but illustrates a scenario at the end of the parameterization operation where time has passed since the instance of FIG. 14B and the parameterized vehicle has moved to reflect a new position after that time. [Figure 14d] Figure 1 shows the user interface with a visual representation of the map view of the scenario. [Figure 15a] FIG. 1 is a high-level schematic diagram of the process by which the system identifies a parameterized road layout on a map. [Figure 15b] FIG. 15b shows a map with an overlay representing an instance of a parameterized road layout identified on the map in the process represented by FIG. 15a. DETAILED DESCRIPTION OF THE INVENTION

[0031] It is necessary to define scenarios that can be used to test the behavior of the ego-vehicle in a simulated environment. The scenarios are defined and edited in offline mode, without the ego-vehicle being controlled, and then exported for testing in the next stage of the test pipeline 7200, described below.

[0032] A scenario comprises one or more subjects (sometimes referred to as actors) traveling along one or more paths in a road layout. A road layout is a term used herein to describe any feature that may occur in a driving scene, and in particular includes at least one trajectory along which a vehicle is intended to travel during simulation. The trajectory may be a road, a lane, or any other drivable path. The road layout is displayed in the edited scenario as an image in which the subjects are instantiated. According to an embodiment of the present invention, the road layout (or other scene topology) is accessed from a scene topology database. The road layout defines lanes and other features and is rendered in the scenario. The scenario is viewed from the perspective of an ego-vehicle operating in the scene. Other subjects in the scene may include non-ego-vehicles or other road users, such as cyclists and pedestrians. The scene may include one or more road features, such as roundabouts or junctions. These subjects are intended to represent real-world entities encountered by the ego-vehicle in real-life driving situations. According to this specification, a user can generate interactions between the subjects and the ego-vehicle, which can be simulated after execution in a scenario editor.

[0033] This specification relates to a method and system for generating scenarios to obtain a large validation set for testing an ego-vehicle. The scenario generation approach described herein allows for more convenient parameterization and exploration of scenarios, as well as closed-loop reuse of scenarios.

[0034] In this system, a scenario is described as a set of interactions. Each interaction is defined relative to the actors in a scene and the static topology of the scene. Each scenario may comprise a static layer for rendering static objects in a visualization of the environment presented to the user on a display, and a dynamic layer for controlling the movement of moving actors in the environment. Note that the terms "agent" and "actor" are sometimes used interchangeably herein.

[0035] Each interaction is described relative to an actor and a static topology. Note that in this context, the self-vehicle is considered a dynamic actor. An interaction includes an operation or behavior performed on another actor or on a static topology.

[0036] In this context, the term "behavior" can be interpreted as follows: A behavior involves an entity (such as an actor in a scene). Given a higher level goal, the behavior interactively performs operations that move the entity towards the given goal. For example, an actor in a scene can be given a Follow Lane goal and an appropriate behavior model. The actor will try to achieve that goal (in the scenario generated in the editor and in the resulting simulation).

[0037] Behavior can be viewed as an opaque abstraction that allows users to impart intelligence to scenarios to obtain more realistic scenarios. By specifying a scenario as a set of interactions, the system allows multiple actors to collaborate with active behaviors to generate closed-loop behavior networks, such as traffic models.

[0038] In the present context, the term "maneuver" is considered to be a concrete physical action that an entity can exhibit to achieve a particular goal according to its behavioral model.

[0039] An interaction involves a specific operation (or set of operations) / behavior with conditions and goals that occur relative to two or more and / or one actor and a static scene.

[0040] In accordance with a feature of the present system, interactions can be evaluated retrospectively using temporal logic, which can be viewed as reusable logic blocks for sequencing scenarios, as described in detail herein.

[0041] The concept of interactions can be used to define "critical paths" of interactions that are important for a particular scenario. Scenarios can have a whole range of abstractions whose parameters can be defined. These variations of abstract scenarios are called scenario instances.

[0042] Scenario parameters are important for specifying scenarios or interactions within scenarios. The system allows for the parameterization of any scenario value. As discussed elsewhere herein when describing interactions, if a value is expected in a scenario, the parameter can be specified with a compatible parameter type and appropriate constraints.

[0043] To illustrate a specific example of the concepts described herein, reference is made to FIG. 1 . An ego-vehicle EV is instantiated on lane L1. An opposing vehicle TV1 is also instantiated and, according to a desired scenario, intends to cut in on the ego-vehicle EV. The interaction shown in FIG. 1 defines a cut-in maneuver that occurs when the opposing vehicle TV1 realizes a specific relationship constraint with respect to the ego-vehicle EV. In FIG. 1 , the relationship constraint is defined as a lateral distance (dy0) offset condition indicated by the dotted line dx0 with respect to the ego-vehicle. At this point, the opposing vehicle TV1 executes a lane change (Switch Lane) maneuver indicated by the arrow M ahead of the ego-vehicle EV. The interaction further defines a new behavior (in this case, a lane-following target) of the opposing vehicle after the cut-in maneuver. Note that this target applies to lane L1 (although the opposing vehicle may have previously had a lane-following target applied to lane L2). The box defined by the dashed line designates this set of operations as interaction I. The vehicle TV2, which is the second operating subject, is assigned a lane keeping target for keeping on the lane L3.

[0044] For the definition of the interaction the following parameters may be assigned:

[0045] Object: an abstract object type that can be written from any ontology class Longitudinal distance dx0: The distance measured along the length of the lane Lateral distance dy0: Distance measured laterally across the lane Velocity Ve, Vy: The speed (longitudinal or transverse) assigned to an object Acceleration Gx: Acceleration assigned to the object Lane: Topological descriptor of a single lane An interaction is specified as a set of time and relational constraints between the dynamic and static layers of a scenario. The dynamic layer represents the scene objects and their respective states, while the static layer represents the scene topology of the scenario. Scenarios are edited / generated, while the constraints that parameterize the layers can be both observed at runtime or written and executed at design time.

[0046] Examples of interactions are given in Table 1 below. [Table 1] Each interaction has a description that defines the particular interaction and the relationships involved. For example, a "cut-in" interaction, as shown in Figure 1, is an interaction in which an object (opposing actor) moves laterally from an adjacent lane into the self-vehicle lane and intersects with a nearby trajectory. The nearby trajectory is a trajectory that overlaps with another actor, but the other actor does not need to take action in response.

[0047] There are two relationships in this interaction: the first is between the counteracting agent and the self-lane, and the second is between the counteracting agent and the self-trajectory. These relationships may be governed by time and relationship constraints, as discussed in more detail below.

[0048] The time and relationship constraints for each interaction can be specified using one or more nodes that input parameters characterizing the interaction. According to the present disclosure, nodes that hold these parameters are stored in an interaction container for the interaction. A scenario can be constructed from a series of interactions by editing and connecting these nodes. This allows users to configure a scenario with a set of required interactions that can be tested in a runtime simulation without requiring complex editing. In conventional systems, when creating and editing a scenario, users must determine whether the interactions they need to test actually occur in the scenario created by the editing tool.

[0049] The system described herein allows a user creating and editing a scenario to specify interactions that are guaranteed to occur when the simulation is run, so that such interactions can be tested in the simulation. As described above, interactions are specified between static topology and dynamic actors.

[0050] The user can define specific interaction operations as given in the table above.

[0051] The user may define the parameters of the interaction and may limit the range of parameters in the interaction.

[0052] FIG. 2 shows an example of a cut-in operation. In this operation, the longitudinal distance dx0 between the host vehicle EV and the opposing vehicle TV1 may be set to a specific value or a range of values. The lateral inner distance dy0 between the host vehicle EV and the opposing vehicle TV1 may be set to a specific value or within a parameter range. The lateral movement (Vy) parameter of the leading vehicle may be set to a specific value or within a specific range. The lateral movement parameter may represent a cut-in speed. The leading vehicle speed (Vo0), which is the forward speed of the opposing vehicle, may be set as a specific specified value or within a parameter range. The host vehicle speed Ve0 may be set to a specific value or within a parameter range and is the speed of the host vehicle in the forward direction. The host vehicle lane (Le0) and the leading vehicle lane (Lv0) may be specified within a parameter range.

[0053] Figure 3 illustrates a cut-out interaction, with some of the parameters identified with reference to the cut-in interaction in Figure 2 above. Additionally, the forward vehicle is defined as the FA (Father Acting), and there are additional parameters associated with this forward vehicle. These include the forward longitudinal distance (dx0_f) and the forward vehicle's velocity.

[0054] Additionally, the vehicle speed (Vf0) may be set to a specific value or parameter range. The vehicle speed Vf0 is the speed of the leading vehicle ahead of the cutout, and in this case the lateral movement Vy of the leading vehicle is in the cutout direction rather than the cut-in direction. Additionally, the leading vehicle is defined as the forward actor (FA), and there are additional parameters associated with this leading vehicle. These include the forward longitudinal distance (dx0_f) and the leading vehicle's speed.

[0055] FIG. 4 shows deceleration interaction. In this case, the parameters Ve0, dx0, and Vo0 have the same definitions as in the cut-in interaction. These values ​​may be set specifically or within a parameter range. Furthermore, the maximum acceleration (Gx_max) may be set to a specific value or within a parameter range as the deceleration of the opposing action subject.

[0056] The steps for defining interactions are discussed in more detail below.

[0057] The user may set a configuration for the ego-vehicle that incorporates a target speed (e.g., a percentage or target speed for each speed limit zone in the road layout), maximum acceleration values, maximum jerk values, etc. In some embodiments, a default speed may be applied to the ego-vehicle as the speed limit for a particular speed limit zone in the road layout. The user may be able to override this default value with an acceleration / jerk value or set a start point and target speed for the ego-vehicle at the interaction cut-in point, which may then be used to calculate acceleration values ​​between the start point and the cut-in point. As described in more detail below, the editing tool allows the user to visualize the scenario after generating it in the editing tool so that the set parameters can be adjusted / explored. Herein, the speed of the ego-vehicle at the interaction point may also be referred to as the ego-vehicle's interaction point speed.

[0058] The opponent vehicle's interaction point speed can also be set. The opponent vehicle's default speed can be set as the road's speed limit or can be set to match the ego vehicle. In some situations, the ego vehicle may have a planning stack that is at least partially exposed at scenario runtime. Note that the latter option applies in situations where the ego vehicle's speed can be extracted from the stack at scenario runtime. The user can override the default speed with acceleration / jerk values ​​or set the opponent vehicle's starting point and speed, which are used to calculate the acceleration value between the starting point and the cut-in point. As with the ego vehicle, the user can adjust / explore these values ​​when the generated scenario is run in the editing tool. In the interaction container (with nodes) discussed herein, the opponent vehicle's values ​​can be set relative to the ego vehicle, allowing the user to set the opponent vehicle's speed / acceleration / jerk relative to the ego vehicle's values ​​at the interaction point.

[0059] In the above, reference is made to interaction points. An interaction point is defined for each interaction. For example, in the scenarios of Figures 1 and 2, a cut-in interaction point is defined. In some embodiments, this is defined at the point where the ego-vehicle and oncoming vehicle overlap laterally (based on the vehicle edges as projected paths forward and backward (part of this is also possible as the lateral overlap)). If this cannot be determined, it can be estimated based on lane width, vehicle width, and lateral positioning.

[0060] The interaction is further specified relative to the scene topology by setting the starting lane (L1 in Figure 1) for the ego vehicle. For the opposing vehicle, the starting lane (L2) and ending lane (L1) are set.

[0061] A cut-in gap may be specified. The time advance is the key parameter value on which the rest of the cut-in interaction is configured. If the user sets a cut-in point 2 seconds ahead of the ego vehicle, the target speed of the ego vehicle at the interaction point is used to calculate the cut-in gap distance. For example, at a speed of 50 miles per hour (22 m / s), a 2-second cut-in gap would set a cut-in distance of 44 meters.

[0062] FIG. 5 is a high-level schematic block diagram of a computer implementing the scenario builder, including a display unit 510, a user input device 502, computer storage such as electronic memory 500 holding program code 504, and a scenario database 508.

[0063] Program code 504 is shown to include four modules configured to accept user input and generate output that is displayed on display unit 510. User input entered into user input device 502 is received by node interface 512, as described herein with reference to Figures 9-13. Scenario model module 506 is then configured to receive the user input from node interface 512 and generate a simulated scenario.

[0064] Scenario model data is sent to a scenario description module 7201, which includes a static layer 7201a and a dynamic layer 7201b. The static layer 7201a contains the static elements of the scenario (typically including a static road layout), while the dynamic layer 7201b specifies dynamic information about external actors in the scenario, such as other vehicles, pedestrians, and cyclists. The scenario model 506 data received by the scenario description module 7201 may then be stored in a scenario database 508, from which it can be loaded and simulated. The scenario model 506 data, whether received via a node interface or the scenario database, is sent to a scenario runtime module 516, which is configured to run a simulation of the parameterized scenario. The output data of the scenario runtime is then sent to a scenario visualization module 514, which is configured to generate data in a format that can be read to generate a dynamic visual representation of the scenario. The output data of the scenario visualization module 514 may then be sent to a display unit 510, where the scenario can be viewed, for example in video format. Additionally, in some embodiments, the display unit 510 may display additional data related to the analyses performed on the simulation data by the program code modules 512, 506, 516, 514.

[0065] A simulation system that can use scenarios generated by the scenario builder described herein will now be described with reference to FIGS. 6 and 7.

[0066] 6 is a high-level schematic block diagram of a runtime stack 6100 for an autonomous vehicle (AV), also referred to herein as an egocentric vehicle (EV). The runtime stack 6100 is shown to include a perception system 6102, a prediction system 6104, a planner 6106, and a controller 6108.

[0067] In a real-world situation, the perception system 6102 would receive sensor outputs from the AV's onboard sensor system 6110 and use these sensor outputs to detect external entities and measure their physical state, such as their position, velocity, acceleration, etc. The onboard sensor system 6110 can take various forms but typically includes a variety of sensors, such as image capture devices (cameras / optical sensors), LiDAR and / or RADAR units, satellite positioning sensors (e.g., GPS), and motion sensors (e.g., accelerometers, gyroscopes), which collectively provide rich sensor data that may enable detailed information to be extracted about the surrounding environment, the state of the AV, and external actors within that environment (e.g., vehicles, pedestrians, cyclists, etc.). The sensor output typically includes sensor data from multiple sensor modalities, such as stereo imagery from one or more stereo optical sensors, LiDAR, RADAR, etc. Stereo imaging is used to collect dense depth data, such as by LiDAR / RADAR, and potentially provides more accurate but less dense depth data. More generally, depth data collection from multiple sensor modalities may be combined (e.g., using Bayesian or non-Bayesian or some other statistical process), preferably respecting the respective levels of uncertainty. For example, multiple stereo pairs of optical sensors may be positioned around the vehicle to provide full 360° depth awareness.

[0068] The recognition system 6102 comprises multiple recognition components that cooperate to interpret sensor outputs and thereby provide recognition output to the prediction system 6104. External entities may be detected and represented probabilistically to reflect the level of recognition uncertainty within the recognition system 6102.

[0069] In a simulation situation, modeling of the vehicle sensor system 6100 may or may not be necessary depending on the nature of the test (especially how the stack 6100 is sliced). Higher levels of slicing do not require simulated sensor data, and therefore do not require complex sensor modeling.

[0070] The recognition output from recognition system 6102 is used by prediction system 6104 to predict future behavior of external actors (subjects), such as other vehicles in the vicinity of the AV.

[0071] The predictions computed by prediction system 6104 are provided to planner 6106, which uses the predictions to make autonomous driving decisions to be executed by the AV in a given driving scenario. A scenario is represented as a set of scenario description parameters used by planner 6106. A typical scenario will define a drivable area and incorporate the predicted movement of any external entities (obstacles from the AV's perspective) within the drivable area. The drivable area may be determined using recognition output from perception system 6102 in combination with map information, such as an HD (high definition) map.

[0072] The central function of the planner 6106 is to plan a trajectory (ego-trajectory) for the AV, taking into account the subject's predicted motion. This is sometimes referred to as maneuver planning. The trajectory is planned to achieve desired goals within a scenario. Goals could be, for example, entering a roundabout and leaving at a desired exit, overtaking a vehicle ahead, or staying in the current lane at a target speed (lane following). Goals could be determined, for example, by an autonomous path planner (not shown).

[0073] The controller 6108 carries out the decisions made by the planner 6106 by providing suitable control signals to the AV's on-board actor system 6112. In particular, the planner 6106 plans maneuvers by the AV, and the controller 6108 generates the control signals to carry out those maneuvers.

[0074] 7 is a schematic block diagram of a test pipeline 7200. The test pipeline 7200 is shown to include a simulator 7202 and a test oracle 7252. The simulator 7202 runs a simulation for the purpose of testing all or part of the AV runtime stack.

[0075] As an example only, the description of the test pipeline 7200 will illustrate some of the underlying principles by reference to the runtime stack 6100 of Figure 6. As noted above, it is possible that only a sub-stack of the runtime stack is tested, but for simplicity, the following description will always refer to the AV stack 6100. Note that, depending on how it is sliced ​​for testing, only a subset of the AV stack 6100 of Figure 6 may actually be tested. Thus, in Figure 6, reference numeral 6100 may refer to the entire AV stack or just a sub-stack, depending on the context.

[0076] 7 shows prediction, planning, and control systems 6104, 6106, and 6108 in an AV stack 6100 under test, with simulated recognition inputs 7203 fed into the stack 6100 from a simulator 7202. However, this does not necessarily imply that the prediction system 6104 operates directly on these simulated recognition inputs 7203 (although this is one possible slicing, in which case the simulated recognition inputs 7203 would correspond in form to the final output of the recognition system 6102). If the entire recognition system 6102 is implemented in the stack under test (or at least includes one or more lower-level recognition components that operate on raw sensor data), the simulated recognition inputs 7203 will comprise simulated sensor data.

[0077] The simulated cognitive inputs 7203 are used as the basis for predictions and ultimately decision-making by the planner 6106. The controller 6108, in turn, implements the planner's decisions by outputting control signals 6109. In a real-world situation, these control signals would drive the AV's physical agent system 6112. The format and content of the control signals generated in testing are the same as in a real-world situation. However, in the test pipeline 7200, these control signals 6109 instead drive an egomotion model 7204, thereby simulating egomotion within the simulator 7202.

[0078] To the extent that external subjects exhibit autonomous behavior / decision-making within the simulator 7202, some form of subject decision logic 7210 implementation executes these decisions and drives the external subject motion within the simulator 7202 accordingly. The subject decision logic 7210 may be comparable in complexity to the autostack 6100 itself, or it may have more limited decision-making capabilities. The goal is to provide sufficiently realistic external subject behavior within the simulator 7202 so that the decision-making capabilities of the autostack 6100 can be effectively tested. In this regard, in some situations, no subject decision logic 7210 is required at all (open-loop simulation), while in other situations, the use of a relatively limited subject logic 7210, such as basic adaptive cruise control (ACC), may provide useful testing. As with the autostack 6100, any subject decision logic 7210 is driven by outputs from the simulator 7202, which are then used to derive inputs to the subject motion model 7206 as the basis for the subject behavior simulation.

[0079] As described above, a simulation of a driving scenario is executed according to a scenario description 7201 having both a static layer 7201a and a dynamic layer 7201b.

[0080] The static layer 7201a defines the static elements of the scenario, which will typically include static road layouts. The static layer 7201a of the scenario description 7201 is placed on a map 7205, which is a map loaded from the map database 7207. For any road layout in a specified static layer 7201a, the system may be able to recognize on a given map 7205 all segments of that map 7205 that have instances of the specified road layout in the static layer 7201a. For example, if a particular map is selected and the road layout of a "roundabout" is specified in the static layer 7201a, the system may find all instances of roundabouts on the selected map 7205 and load them as a simulation environment.

[0081] The dynamic layer 7201b defines dynamic information about external entities in the scenario, such as other vehicles, pedestrians, cyclists, etc. The extent of the dynamic information provided can vary. For example, for each external entity, the dynamic layer 7201b may comprise a spatial path or designated lane to be followed by the entity along with motion data and / or behavior data.

[0082] In a simple open-loop simulation, the external actor simply follows the spatial path and motion data defined in the dynamic layer that is non-reactive, i.e., does not react to the self-acting subject within the simulation. Such an open-loop simulation can be realized without the subject determination logic 7210.

[0083] However, in a "closed-loop" simulation, the dynamic layer 7201b instead prescribes at least one behavior (such as an ACC behavior) to be followed along a static path or lane. In this case, the subject decision logic 7210 implements that behavior in the simulation reactively, i.e., in response to the self-subject and / or other external subjects. The motion data may still be associated with the static path, but in this case it may be less prescriptive and may serve as targets along the path, for example. For example, the ACC behavior may set a target speed along the path that the subject attempts to match, but the subject decision logic 7210 may allow the external subject's speed to drop below the target at any point along the path in order to maintain a target progress from the vehicle ahead.

[0084] In this embodiment, the static layer provides the road network with lane definitions that are used instead of defining a "route." The actual lane definitions are stored in the static layer, while the dynamic layer contains the assignment of subjects to lanes as well as any lane maneuvers.

[0085] The output of simulator 7202 for a given simulation includes a self-trace 7212a of the self-subject and one or more subject traces 7212b (trace 7212) of one or more external subjects.

[0086] A trace is a complete history of the behavior of a subject within a simulation, having both spatial and motion components. For example, a trace may take the form of a spatial path with motion data associated with points along the path, such as velocity, acceleration, jerk (rate of change of acceleration), and jerk (rate of change of jerk).

[0087] Additional information is also provided to complement and provide context for the trace 7212. Such additional information is referred to as "environmental" data 7214 and may have both static components (such as road layout) and dynamic components (such as weather conditions to the extent that they change over the course of the simulation).

[0088] The environmental data 7214 may be "passthrough" to some extent, in that it is specified directly by the scenario description 7201 and is not affected by the results of the simulation. For example, the environmental data 7214 may include a static road layout derived directly from the scenario description 7201. However, the environmental data 7214 will typically include at least some elements derived within the simulator 7202. This could include, for example, simulated weather data, and the simulator 7202 is free to change weather conditions as the simulation progresses. In this case, the weather data may be time-dependent, and this time-dependency would be reflected in the environmental data 7214.

[0089] Test oracle 7252 receives trace 7212 and environmental data 7214 and scores these outputs against a set of predetermined numerical performance criteria 7254. Performance criteria 7254 encode what may be referred to herein as a "Digital Highway Code" (DHC). Some examples of suitable performance criteria are given below.

[0090] Scoring is time-based, and the test oracle 7252 tracks, for each performance criterion, how the value of that criterion (score) changes over time as the simulation progresses. The test oracle 7252 provides an output 7256 that includes a score-time plot for each performance criterion.

[0091] The criteria 7256 are useful to practitioners and the scores can be used to identify and mitigate performance issues within the test stack 6100.

[0092] As described above, the scenarios used by the simulation system may be generated in the scenario builder described herein. Returning to the example scenario given in Figure 1, Figure 8 shows how the interactions in the scenario are decomposed into nodes.

[0093] FIG. 8 illustrates an exemplary path of a cut-in operation that may be defined as an interaction herein. In this example, the interaction is defined as three separate interaction nodes. The first node is considered a "start operation" node, shown at point N1. This node specifies the time (seconds) to the interaction point and the speed of the opposing vehicle. The second node, N2, may specify the cut-in profile and path curvature, shown diagrammatically by a double-headed arrow. This node is designated N2. This node may also specify the lateral velocity Vy of the cut-in profile through the cut-in duration and speed change profile. As described below, the user may adjust the acceleration and jerk values ​​as needed. Node N3 is an end operation, specifying the time (seconds) from the interaction point and the speed of the opposing vehicle. As described below, a node container may be made available to the user for setting the start and end points of the cut-in operation as well as the option to set parameters.

[0094] FIG. 13 shows the user interface 900a of FIG. 9a, which includes a road toggle 901 and an actor toggle 903. In FIG. 9a, the actor toggle 903 is selected, which adds features and input fields to the user interface 900a configured to parameterize dynamic layers of the simulation environment, such as simulated vehicles and their behavior. In FIG. 13, the road toggle 901 is selected. As a result of this selection, the user interface 900a adds features and input fields configured to parameterize static layers of the simulation environment, such as road layouts. In the example of FIG. 13, the user interface 900a includes a set of pre-configured road layouts 1301. Selecting a particular one of the set of pre-configured road layouts 1301 causes the selected road layout to be displayed in the user interface 900a (in this example, at the bottom of the user interface 900a) and allows for further parameterization of the selected road layout 1301. Radio buttons 1303 and 1305, when selected, are configured to parameterize the side of the road on which the simulated vehicle travels. Selecting the "Left Side" radio button 1303 will cause the system to configure the simulation so that vehicles in the dynamic layer travel on the left side of the road defined in the static layer. Similarly, selecting the "Right Side" radio button 1305 will cause the system to configure the simulation so that vehicles in the dynamic layer travel on the right side of the road defined in the static layer. In some embodiments, when a particular radio button 1303 or 1305 is selected, the other is automatically deselected so that oncoming traffic lanes are not possible.

[0095] The user interface 900a of Figure 13 further displays an editable road layout 1306 representing the selected pre-configured road layout 1301. The editable road layout 1306 has a plurality of width input fields 1309 associated with it, with each particular width input field 1309 being associated with a particular lane in the road layout. Data may be entered into a particular width input field 1309 to parameterize the width of that corresponding lane. The lane widths are used to render the scenario in the scenario editor and to run the simulation at runtime.

[0096] The editable road layout 1306 also has an associated curvature field 1313 configured to modify the curvature of the selected preset road layout 1301. In the example of Figure 13, the curvature field 1313 is shown as a slider. By sliding the arrow along the bar, the curvature of the road layout can be edited.

[0097] Additional lanes may be added to the editable road layout 1306 using the lane generator 1311. In the example of Figure 13, if driving on the left implies driving to the right or left on the displayed editable road layout 1306, one or more lanes may be added to the left side of the road by selecting the lane generator 1311 above the editable road layout 1306. Similarly, one or more lanes may be added to the right side of the road by selecting the lane generator 1311 below the editable road layout 1306. For each lane added to the editable road layout 1306, an additional width input field 1309 is also added, configured to parameterize the width of the new lane.

[0098] Lanes in the editable road layout 1306 may also be removed upon selection of a lane remover 1307, with each lane in the editable road layout being associated with a unique lane remover 1307. Upon selection of a particular lane remover 1307, the lane associated with that particular lane remover 1307 is removed, and the width input field 1309 associated with that lane is also removed.

[0099] In this way, interactions can be defined by the user for a particular layout. The path of the opposing vehicle can be set to continue to the maneuver point at a constant speed required to start the maneuver. The path of the opposing vehicle after the maneuver ends should continue at a constant speed using the value reached at the end of the maneuver. The user can be provided with the option to set the start and end of the maneuver point and to confirm the corresponding values ​​at the interaction point, as described in more detail below.

[0100] Configuring a scenario with a set of defined interactions can enhance what the generated scenario can do in the post-simulation analysis phase. For example, analysis output can be organized around an interaction point. An interaction can be used as a consistent point in time across all scenarios explored with a particular operation. This provides a single reference point for comparison, but users can view analysis output a configurable number of seconds before and after this point (based on runtime duration). FIG. 12 illustrates a framework for configuring a general user interface 900a through which a simulation environment can be parameterized. The user interface 900a of FIG. 12 includes a scenario name field 1201, where a name can be assigned to the scenario. Additionally, a description of the scenario can be entered in a scenario description field 1203, and metadata related to the scenario (e.g., generation date) can be stored in a scenario metadata field 1205.

[0101] A self-object editing node N100 is provided for parameterizing the self-vehicle, the self-node N100 comprising fields 1202 and 1204 configured to define the interaction point lane and interaction point speed, respectively, of the self-vehicle relative to the selected static road layout.

[0102] A first actor vehicle can be configured in a vehicle 1 object edit node N102, which includes a start lane field 1206 and a start speed field 1214 configured to define the starting lane and starting speed, respectively, of the corresponding actor vehicle in the simulation. Other actor vehicles (vehicle 2 and vehicle 3) can be configured in corresponding vehicle nodes N106 and N108, which also include a start lane field 1206 and a start speed field 1214 configured for the same purpose as node N102 but for different corresponding actor vehicles. The user interface 900a of FIG. 12 also includes an actor node generator 905b that, if selected, generates additional nodes to generate additional actor vehicles to be executed in the scenario. The newly generated vehicle nodes may include fields 1206 and 1214 so that the new vehicles can be parameterized like other objects in the scenario.

[0103] In some embodiments, vehicle nodes N102, N106, and N108 of user interface 900a may further include a vehicle selection field F5, as described below with reference to FIG. 9a.

[0104] For each of the subject vehicle nodes N102, N106, and N108, a set of associated action nodes may be generated and assigned using the action node generator 905a, with each vehicle node associated with an associated action node generator 905a located (in this example) at the right end of the row for that vehicle node. An action node may have multiple fields configured to parameterize the action performed by the corresponding vehicle during a scenario execution or simulation. For example, vehicle node N102 is associated with action node N103, which includes an interaction point definition field 1208, a target lane / speed field 1210, and a motion constraint field 1212. The interaction point definition field 1208 of node N103 may itself have one or more input fields that may specify a point on the static scene topology of the simulated environment where the maneuver is to be performed by vehicle N102. Similarly, the target lane / speed field 1210 may have one or more input fields configured to specify the speed and target lane of the vehicle performing the action using a lane identifier. The action constraint field 1212 may include one or more input fields configured to further define aspects of the action to be performed. For example, the action constraint field 1212 may include the behavior selection field 909, as described with reference to FIG. 9a, and may be configured to select an operation or behavior type from a predetermined list, with the system configured to add input fields required to parameterize the selected operation or behavior type to the associated action node upon selection of a specific behavior type. In the example of FIG. 12, the vehicle 1 is assigned a second action node N105 that includes the same set of fields 1208, 1210, and 1212 as the first action node N103. Note that a third action node can be added to the user interface 900a upon selection of the action node generator 905a located to the right of the second action node N105.

[0105] 12 shows a second vehicle node N106, which also includes a starting lane field 1206 and a starting speed field 1214. The second vehicle node N106 is shown as having three associated action nodes N107, N109, and N111, each including a set of fields 1208, 1210, and 1212 that may parameterize their associated actions. An action node generator 905a is also present to the right of action node N111, and its selection will also generate additional action nodes configured to parameterize different behaviors of vehicle 2 during the simulation.

[0106] Also shown is a third vehicle node N108, which also includes a starting lane field 1206 and a starting speed field 1214, but which has only one action node N113 assigned to it. Action node N113 similarly includes a set of fields 1208, 1210, and 1212 that may parameterize the associated action, and upon selection of action node generator 905a to the right of action node N113, a second action node may be generated.

[0107] Action and vehicle nodes also have selectable node removers 907 that, when selected, remove the associated node from the user interface 900a, thereby removing the associated action or object from the simulated environment. Additionally, selection of a particular node remover 907 may also remove auxiliary or subordinate nodes of that particular node. For example, selection of a node remover 907 associated with a vehicle node (e.g., N106) may automatically remove the action node (e.g., N107) associated with that vehicle node without selecting that action node's node remover 907.

[0108] Upon registering inputs into all relevant fields of user interface 900a of Figure 12, the user may be able to see pre-simulation visual representations of their simulation environment, such as those described below with reference to Figures 10a, 10b, and 11, for the inputs configured in Figure 9a, and selection of a particular node may display the entered parameters, which appear as data overlays on the associated visual representations, such as Figures 10a and 10b.

[0109] FIG. 9a illustrates a specific example of how the framework of FIG. 12 can be used to provide a set of nodes for defining cut-in interactions. Each node may be presented to the user on the user interface of an editing tool, allowing the user to configure interaction parameters. N100 indicates a node that defines the behavior of the ego vehicle. The lane field F1 allows the user to define the lane on the scene topology in which the ego vehicle begins. The maximum acceleration field F2 allows the user to set the maximum acceleration using the up and down menu selection buttons. The speed field F3 allows the user to enter a fixed speed using the up and down buttons. The speed mode selector allows the speed to be set to a fixed value (shown as selected in node N100 of FIG. 9a) or to a percentage of the speed limit. The percentage of the speed limit is associated with its own field F4, which is configured by the user. Node 102 describes the opposing vehicle, which is selected from the dynamic object ontology using a drop-down menu shown in field F5. The lane in which the oncoming vehicle operates is selected using the lane field F6. The cut-in interaction node N103 has a field F8 for specifying the forward distance dx0 and a field F9 for specifying the lateral distance dy0. Fields F10 and F11 are provided to specify the maximum acceleration of the cut-in maneuver in the forward and lateral directions, respectively.

[0110] Node N103 has a title field F12 in which the nature of the interaction can be defined by selecting from several options in a drop-down menu. As each option is selected, the relevant fields of the node are displayed and the user can add parameters appropriate to that interaction.

[0111] The path of the opposing vehicle is also subject to a second node N105 that defines the speed change behavior. Node N105 comprises a field F13 for setting the distance ahead of the opposing vehicle that causes a speed change, a field F14 for setting the maximum acceleration, and respective speed limit fields F15 and F16 that behave as described with reference to the own vehicle node N100.

[0112] Another vehicle is further defined using object node N106, which provides the same configurable parameters as node N102 of the opposing vehicle. The second vehicle is associated with a lane keeping behavior defined by node N107, which has a field F16 for setting the forward distance relative to the ego vehicle and a field F17 for setting the maximum acceleration.

[0113] 9a further illustrates road toggle 901 and actor toggle 903. Road toggle 901 is a selectable feature of user interface 900a that, when selected, adds features and input fields to user interface 900a configured to parameterize a static layer of the simulation environment, such as a road layout (see description of FIG. 13). Actor toggle 903 is a selectable feature of user interface 900a that, when selected, adds features and input fields to user interface 900a configured to parameterize a dynamic layer of the simulation environment, such as a simulated vehicle and its behavior.

[0114] As described with reference to FIG. 12 , the node generator 905 is a selectable feature of the user interface 900 a that, when selected, generates additional nodes that can parameterize additional aspects of the dynamic layer of the simulation environment. An action node generator 905 a may be located at the far right of each actor vehicle row. When selected, such an action node generator 905 a allows parameterization of multiple actions for a simulation by assigning additional action nodes to their associated actor vehicles. Similarly, a vehicle node generator 905 b may be located below the bottom vehicle node. When selected, the vehicle node generator 905 b adds additional vehicles or other dynamic objects to the simulation environment, which can be further configured by assigning one or more action nodes to them using the associated action node generator 905 a. Action and vehicle nodes may also have a selectable node remover 907 that, when selected, removes the associated node from the user interface 900 a, thereby removing the associated behavior or object from the simulation environment. Additionally, selection of a particular node remover 907 may also remove auxiliary or subordinate nodes of that particular node. For example, selection of a node remover 907 associated with a vehicle node (such as N106) automatically removes an operation node (such as N107) associated with that vehicle node without the need to select the node remover 907 for that operation node.

[0115] Each vehicle node may further include a vehicle selection field F5, where a particular type of vehicle may be selected from a predetermined set of types, such as a drop-down list. Upon selection of a particular vehicle type from the vehicle selection field F5, the corresponding vehicle node may be populated with other input fields configured to parameterize parameters specific to the vehicle type. Additionally, selecting a particular vehicle may impose constraints on corresponding motion node parameters, such as maximum acceleration or speed.

[0116] Each action node may also include a behavior selection field 909. Upon selection of the behavior selection field 909 associated with a particular action node (e.g., N107), the node displays, e.g., in a drop-down list, a set of predefined behaviors and / or maneuvers that can be configured for the simulation. Upon selection of a particular action from the set of predefined behaviors, the system adds to the action node the input fields required to parameterize the selected behavior of the associated vehicle. For example, action node N107 is associated with the subject vehicle TV2 and includes a behavior selection field 909, with the "lane keeping" behavior selected. As a result of this particular selection, action node N107 is also added with a field F16 for setting the forward distance of the associated vehicle TV2 from the subject vehicle EV and a maximum acceleration field F17, which is indicated to enable parameterization of the selected behavior type of the subject vehicle TV2.

[0117] FIG. 9b illustrates another embodiment of the user interface of FIG. 9a. FIG. 9b includes the same vehicle nodes N100, N102, and N106, representing the subject vehicle EV, a first actor vehicle TV1, and a second actor vehicle TV2, respectively. The example of FIG. 9b provides a similar scenario to that of FIG. 9a, except that the first actor vehicle TV1, defined by node N102, is performing a "lane change" maneuver rather than a "cut-in" maneuver, the second actor vehicle TV2, defined by node N106, is performing a "speed maintenance" maneuver rather than a "lane keeping" maneuver, and is defined as a "large truck" as opposed to a "passenger car," and several exemplary parameters entered into the fields of user interface 900b are different from those of user interface 900a.

[0118] The user interface 900b of FIG. 9b includes several features not present in the user interface 900a of FIG. 9a. For example, the actor vehicle nodes N102 and N106, configured to parameterize actor vehicles TV1 and TV2, respectively, include a starting speed field F29 configured to specify the initial speed of each vehicle during the simulation. The user interface 900b further includes a scenario name field F26 in which a user can enter one or more characters to specify the name of the scenario being parameterized. Also included is a scenario description field F27, which is configured to accept additional characters and / or identifying characters that can help identify the scenario and distinguish it from other scenarios. A label field F28 is also present, which is configured to accept additional words and / or identifying characters that can help categorize and organize saved scenarios. In the example user interface 900b, a label titled "Environment | Main Road" has been added to field F28.

[0119] The user interface 900b of Figure 9b does not have several features of the user interface 900a of Figure 9a. For example, in the user interface 900b of Figure 9b, acceleration control is not defined for the ego-vehicle node N100. Furthermore, in the example of Figure 9b, the road toggle 901 and the actor toggle 903 are not present, and the user interface 900b is specifically configured to parameterize the vehicle and its behavior.

[0120] Additionally, options F4 and F18 in Figure 9a, which specify the vehicle speed as a percentage of the specified speed limit, are not available in user interface 900b; only fixed speed field F3 is configurable in this embodiment. Acceleration control fields, such as field F14 in speed change operation node N105, are also not present in user interface 900b in Figure 9b. The behavior constraints for speed change operations are parameterized using a different set of fields.

[0121] Additionally, a different set of fields has been added to the speed change operation node N105 assigned to the first subject vehicle TV1. The maximum acceleration field F14, fixed speed field F15, and speed limit percentage field F18 present in user interface 900a are not present in 900b. Instead, a target speed field F22, a relative position field F21, and a speed field F23 are present. The target speed field F22 is configured to accept user input regarding the desired speed of the associated vehicle at the end of the speed change operation. The relative position field F21 is configured to specify the point or other simulation entity from which the forward distance specified in field F13 is measured, while the forward distance field F13 is present in both user interfaces 900a and 900b. In the example of FIG. 9b, the relative position field F21 is specified as the ego vehicle, although other options may be selected, for example, via a drop-down menu. The speed field F23 specifies the speed or percentage of the operation. Since the maneuver defined by node N103 depends on speed (and not on position or lane), the speed field F23 represents an acceleration control to limit the rate at which the target speed as defined in field F22 can be reached.

[0122] In user interface 900b, because operation node N103 assigned to first subject vehicle TV1 is specified as a lane change operation, node N103 includes additional fields in the same node as in user interface 900a, which specifies a cut-in operation. Operation node N103 in FIG. 9b still includes forward distance field F8 and lateral distance field F9, but now includes a relative position field F30 configured to specify a point or other simulation entity from which the forward distance in field F8 is measured. In the example of FIG. 9b, relative position field F30 specifies the ego-vehicle as the reference point, although other options, such as selection from a drop-down menu, may be configurable. Thus, the operation activation condition is specified by measuring the forward and lateral distances specified in fields F8 and F9 from the point or entity specified in F30. The lane change operation node N103 of Figure 9b further comprises a target lane field F19 configured to specify the lane to be occupied by the associated vehicle after execution of the operation, and a speed field F20 configured to specify the movement constraints of the operation.

[0123] In FIG. 9b, the operation node N107 assigned to the second operating subject vehicle TV2 is defined as a "maintain speed" operation. Therefore, different fields have been added to the same node in the user interface 900a for defining the "maintain speed" operation. While the operation node N107 in FIG. 9b still includes the forward distance field F16, it does not include the maximum acceleration field F17 present in FIG. 9a. Instead, the node N107 in FIG. 9b includes a relative position field F31, which serves the same purpose as the relative position fields F21 and F30 and may also be editable via a drop-down menu. Additionally, the node N107 in FIG. 9b includes a target speed field F32 and a speed field F25. The target speed field F32 is configured to define a target speed to be maintained during the operation. The speed field F25 defines the speed or rate of the operation. Since the maneuver defined by node N105 depends on speed (and not on position or lane), the speed field F25 represents an acceleration control to limit the rate at which the target speed as defined in field F32 can be reached.

[0124] The fields added to nodes N103 and N107 differ between Figures 9a and 9b because the operations defined therein are different. However, it should be noted that user interface 900b may still add each node differently from user interface 900a, provided that the types of operations defined for these nodes are consistent between Figures 9a and 9b.

[0125] Similar to the user interface 900a of Figure 9a, the user interface 900b of Figure 9b includes a node generation button 905. However, the example of Figure 9b does not show the vehicle node generator 905b that was a feature of the user interface 900a of Figure 9a.

[0126] In the example of Figure 9b, the operation type fields, such as F12, may not be editable fields. In Figure 9a, field F12 is an editable field, so that selecting a particular operation type from its drop-down list adds associated input fields to the associated node for parameterizing that particular operation type. Instead, in the example of Figure 9b, the operation type may be selected upon node creation, such as by selecting node generator 905.

[0127] Figures 10a and 10b provide an example of the system's pre-simulation visualization capabilities. The system can generate graphical representations of static and dynamic layers to allow users to visualize parameterized simulations before they are run. This capability significantly reduces the chances that users will unintentionally program a desired scenario.

[0128] A user can view a graphical representation of the simulated environment at key moments in the simulation (e.g., at interaction states) without having to run the simulation and observe the presence of programming errors. Figures 10a and 10b also demonstrate the selection functionality of user interface 900a of Figure 9a. One or more nodes from the set of nodes provided in Figure 9a may be selectable, and the system, upon selection, provides a data overlay of the node's programmed behavior onto the graphical representation of the simulated environment.

[0129] For example, FIG. 10a shows a graphical representation of the simulated environment programmed in the user interface 900a of FIG. 9a, with the node titled "Vehicle 1" selected. As a result of this selection, the parameters and behaviors assigned to Vehicle 1 TV1 are visible as a data overlay on FIG. 10a. The symbol X2 indicates the point where the interaction condition defined for node N103 is satisfied. Because point X2 is defined by a distance entered in F8 and F9 rather than a coordinate, the symbol X1 defines the point from which the distance parameterized in F8 and F9 is measured (in all given examples, the ego-vehicle EV defines the X1 point). Additionally, the orange dotted line 1001 labeled "20 m" clearly indicates the longitudinal distance (the distance between X1 and X2) between the ego-vehicle EV and Vehicle 1 TV1 where the maneuver is activated. Other data overlay features, such as the vehicle's set speed or the destination point in the scenario, may also be represented.

[0130] Also, the cut-in operation parameterized at node N103 is visible as a curved orange line 1002 from symbol X2 to symbol X4, with the symbol type specified in the upper left corner of node N103. Similarly, the speed change operation specified at node N105 is shown as an orange line 1003 from symbol X4, where the cut-in ends, to symbol X3, with the symbol type specified in the upper left corner of node N105.

[0131] Upon selection of node N106 for "Vehicle 2," a data overlay assigned to vehicle 2 TV2 is shown in the same manner as in FIG. 10b. Note that while FIGS. 10a and 10b depict the same moment in time, only the vehicle node selected in user interface 900a of FIG. 9a differs, and thus the data overlays are different. Upon selection of node N106 for vehicle 2, a visual representation of a "Lane Keeping" maneuver assigned to vehicle 2 TV2 at node N107 is shown in FIG. 10b. The activation conditions for this vehicle maneuver, as defined in F16, are shown as a blue dotted line 1004 overlaid in FIG. 10b, along with symbols X2 and X1, representing the point at which the activation conditions are met and the point at which the distance defining the activation conditions is measured, respectively. The lane keeping maneuver is shown as a blue arrow 1005 overlaid in FIG. 10b, the end of which is similarly marked with the symbol defined in the upper left corner of node N107 (in this case, symbol X3).

[0132] In some embodiments, it may be possible to display data overlays for multiple vehicles simultaneously, or to display data overlays for only one operation rather than all operations assigned to a particular vehicle.

[0133] Also, in some embodiments, it may be possible to edit the type of symbol used to define the start or end point of an operation; in this case, the symbol in the upper left corner of the action node in FIG. 9a is a selectable and editable feature of the user interface 900.

[0134] In some embodiments, no data overlays are shown. FIG. 11 shows the same simulation environment as configured in user interface 900 of FIG. 9a, but with no nodes selected. As a result, none of the data overlays seen in FIG. 10a or FIG. 10b are present, and only the ego-vehicle EV, vehicle 1 TV1, and vehicle 2 TV2 are shown. The representations of FIGS. 10a, 10b, and 11 remain unchanged; only the data overlays change.

[0135] 14a, 14b, and 14c show pre-simulation graphical representations of interaction scenarios between three vehicles EV, TV1, and TV2, representing the ego vehicle, the first actor vehicle, and the second actor vehicle, respectively. Each figure also includes a scrubbing timeline 1400 configured to enable dynamic visualization of the parameterized scenario prior to simulation. For all of FIGS. 14a, 14b, and 14c, the node for vehicle TV1 is selected in the node edit user interface (e.g., FIG. 9b) so that a data overlay related to the operation of vehicle TV1 is shown on the graphical representation.

[0136] The scrubbing timeline 1400 includes a scrubbing handle 1407 that can be controlled in both directions along the timeline. The scrubbing timeline 1400 also has associated playback amount controls 1401, 1402, and 1404 (play button 1401, rewind button 1402, and fast-forward button 1404). The play button, upon selection, may be configured to play a pre-simulation dynamic representation of the parameterized scenario, with playback beginning from the position of the scrubbing handle 1407 at the time of selection. The rewind button 1402, upon selection, may be configured to move the scrubbing handle 1407 to the left, thereby causing the graphical representation to show a corresponding earlier moment. The rewind button 1402, upon selection, may also be configured to move the scrubbing handle 1407 back to a key moment in the scenario, such as the nearest time an operation began, causing the graphical representation of the scenario to adjust to match the new point in time. Similarly, fast-forward button 1404 is configured to cause the graphical representation to show a corresponding later instant in time by selecting it and moving scrubbing handle 1407 to the right. Fast-forward button 1404 may also be configured to cause selection to move to a future significant instant, such as the nearest future point in time at which a new operation begins, in which case the graphical representation will change accordingly.

[0137] In some embodiments, the scrubbing timeline 1400 may be capable of displaying a set of approximately contiguous instances of a parameterized scenario in time. In this case, a user may be able to scrub to any instant between the start and end of a simulation and view the corresponding pre-simulation graphical representation of the scenario at that instant. In such a case, selection of the play button 1401 may enable playback of the dynamic visualization at a frame rate such that the user perceives the continuous progression of the interaction scenario (i.e., video playback).

[0138] The scrubbing handle 1407 itself may be a selectable feature of the scrubbing timeline 1400. Selecting and dragging the scrubbing handle 1407 to a new location on the scrubbing timeline 1400 may cause the graphical representation to change to indicate the relative location of the simulation entities at the new moment in time. Alternatively, selecting a particular location along the scrubbing timeline 1400 may cause the scrubbing handle 1407 to move along the scrubbing timeline to the point where the selection was made.

[0139] The scrubbing timeline 1400 may also include visual indicators, such as colored or shaded regions, that indicate various phases of a parameterized scenario. For example, a particular visual indication may be assigned to a region of the scrubbing timeline 1400 to indicate a set of temporal instances in which a maneuver activation condition for a particular vehicle is not met. A second visual indication may then indicate a second region. For example, this region may represent a period in which a maneuver occurred or where all assigned maneuvers have been performed. For example, the example scrubbing timeline 1400 for FIG. 1A includes an unshaded pre-actuation region 1403, representing a period in which a scenario activation condition is not met. Also shown is a shaded maneuver region 1409, which indicates a period in which a maneuver assigned to actor vehicles TV1 and TV2 is in progress. The example scrubbing timeline 1400 further includes an unshaded post-actuation region 1413, indicating a period in which a maneuver assigned to actor vehicles TV1 and TV2 is completed.

[0140] As shown in Figure 14b, scrubbing timeline 1400 may further include symbolic indicators such as 1405 and 1411 that represent boundaries between scenario phases. For example, exemplary scrubbing timeline 1400 includes a first boundary indicator 1405 that represents the moment a manipulation is activated. Similarly, a second boundary point 1411 represents the boundary point between a manipulation phase 1409 and a post-manipulation phase 1413. Note that the symbols used to indicate the boundary points in Figures 14a, 14b, and 14c may not be the same in all embodiments.

[0141] Figures 14a, 14b, and 14c illustrate the progression of time for a single scenario. In Figure 14a, the scrubbing handle 1407 is positioned at the first boundary point 1405 between the pre-interaction phase 1403 and the interaction phase 1409 of the scenario. As a result, the actor vehicle TV1 is shown at the position where this transition occurs (point X2). In Figure 14b, the actor vehicle TV1 has performed its first maneuver (cut-in) and reached point X3. At this moment, the actor vehicle TV1 will begin performing its second maneuver (deceleration maneuver). Because time has passed since the maneuver was initiated at point X2, i.e., the corresponding first boundary point 1405, the scrubbing handle 1407 has moved to correspond to the point where the second maneuver begins. Note that in Figure 14b, the scrubbing handle 1407 is within the maneuvering phase 1409, as indicated by the shading. Figure 14c illustrates the moment the maneuver is completed. The subject vehicle TV1 has reached point X4, and the scrubbing handle has advanced to second boundary point 1411, where the operation has ended.

[0142] The scenario visualization is a real-time rendered depiction of actors (in this case, vehicles) on the specific road segment selected for the scenario. The ego-vehicle EV is shown in black, while other vehicles are represented by labels (TV1, TV2, etc.). The visual overlay can be toggled on demand and shows the start and end points of the interaction, the vehicle's position and trajectory, and its distance from other actors. Selection of different vehicle nodes in the corresponding node editing user interface, such as Figure 9b, controls which vehicles or operating actors are shown in the visual overlay.

[0143] The timeline controller allows the user to play back scenario interactions in real time (play button), jump from one interaction point to the next (skip forward / backward buttons), or scrub forward or backward in time using the scrubbing handle 1407. A circled "+" indicates the first interaction point in the timeline, and a circled "x" indicates the final interaction point. This encompasses all actors in the scenario; that is, a circled "+" indicates the point where the first operation begins for any actor in the simulation, and a circled "x" indicates the point where the last operation ends for any actor in the simulation.

[0144] FIG. 14d shows an alternative visual representation on the user interface that allows the user to select a specific interaction point (time instance). In FIG. 14d, a map view, indicated by 1415, is presented to the user on the user interface display. The location of each interaction (or action) is visually represented as an interaction point in the map view. The first interaction point is represented by location 1417, and the second interaction point is represented by location 1419. The map view shows the road layout (scene topology) in which these locations 1417 and 1419 are represented. Note that in this embodiment, the location of the interaction point is determined by the location of the ego-vehicle in that particular interaction. The action location may also be defined by one of the other actors in the scenario's interactions. To select a particular time instance, the user may engage with the illustrated location. For example, the user may click on the location using a cursor or any other display interaction mechanism. Once the user selects a time instance, the user interface presents the selected interaction as described above. Interaction points may be highlighted along the map where they are expected to occur based on a timeline. The map view mode allows for a panoramic view of the map space used for the unfolding scenario and indicates where interactions will occur at various times. When moving from the map view 1415 to the normal view, clicking on one of the interaction points provides a perspective of the roads (scene topology) presented within a specified radius of the selected point. For example, as shown in Figure 14d, the circle indicates the scene topology presented to the user in the scenario view.

[0145] When playing back the timeline, the visualization of the subjects will show their movements as specified by their scenario actions. In the example given in FIG. 14a, subject TV1 first interacts with the ego EV at a point 5 m ahead and 1.5 m lateral to the ego EV, designated point X2. This triggers the first action (designated by the circled "1"), which causes TV1 to perform a lane change action from lane 1 to lane 2, given the speed and acceleration constraints in the scenario. When this action is completed, the subject will proceed to the next action. The second action, designated by the circled "2" in FIG. 14b, will be triggered when TV1 is 30 m ahead of the ego EV, which is the second interaction point. TV1 will then perform its designated deceleration action to achieve a specific speed. When this speed is reached, the second action is completed, as shown in FIG. 14c. Because no other actions are assigned to this subject, no further operations will be performed.

[0146] These example images show a second subject (TV2) in the scenario, which has been assigned the behavior of following lane 2 and maintaining a steady speed. Because the visualization perspective is a bird's-eye top view of the road and is self-tracking, the subjects' movements are only seen relative to each other, and TV2's movements are not visible in the scenario visualization.

[0147] 15a is a high-level schematic diagram of the process by which the system recognizes all instances of parameterized static layer 7201a of scenario 7201 on map 7205. Parameterized scenario 7201 may also include data regarding dynamic layer entities and their interactions, but is shown with data subgroups 7201a and 1501 relating to the static layer and static layer distance requirements, respectively, defined in scenario 7201. As an example, static layer parameters 7201a and scenario execution distance 1501, when combined, may define a 100m section of a two-lane road ending in a four-lane "division" "T-junction."

[0148] The identification process 1505 represents the system's analysis of one or more maps stored in the map database. The system can identify instances on one or more maps that satisfy the parameterized static layer parameters 7201a and the scenario execution distance 1501. The map 7205 with suitable instances of the parameterized road segments may then be provided to the user for simulation.

[0149] The system may search for suitable road segments by comparing the criteria of the parameterized static layer against existing data on road segments in each map, where the system will distinguish between a subset of suitable road segments 1503 and a subset of unsuitable other road segments 1507.

[0150] Figure 15b shows an example map 7205 with multiple different types of road segments. As a result of the user parameterizing static layer 7201a and scenario run distance 1501 as part of scenario 7201, the system has identified all road segments in map 7205 that are good instances of the parameterized road layout. The good instances 1503 identified by the system are highlighted in blue in Figure 15b.

Claims

1. 1. A computer-implemented method for generating scenarios to be executed in a simulation environment for testing the behavior of an autonomous vehicle, comprising: Rendering an interactive visualization for editing on a display of a computing device for a scenario model comprising one or more interactions between an ego-vehicle object and one or more dynamic opponent objects, each interaction being defined as a set of time constraints and / or relationship constraints between the ego-vehicle object and at least one of the opponent objects, the scenario model comprising a scene topology, the interactive visualization including scene objects in which the ego-vehicle object and the at least one opponent object are displayed in the scene topology, and the scenario is associated with a timeline extending in a traveling direction of the ego-vehicle object relative to the scene topology; Rendering on the display, in response to user input, timing controls for selecting moments along the timeline; generating on the display an interactive visualization of the scene topology and scene objects of the scenario displayed at the selected moment; A computer-implemented method comprising:

2. The method of claim 1 , comprising the step of, in response to selecting the moment, rendering a dynamic visualization of the scene according to the scenario model from the selected moment onwards.

3. 3. The method of claim 1, comprising displaying the interactive visualization at an initial moment prior to selection of the moment, and in response to selection of the selected moment, rendering on the display a new interactive visualization of the scene at the selected moment without rendering views of the scene at moments between the initial moment and the selected moment.

4. The method of claim 2 or 3, wherein rendering the dynamic visualization of the scene occurs automatically in response to the selection of the moment.

5. displaying to a user in an editing user interface of the computing device the set of time and / or relationship constraints that define one or more of the interactions presented in the scenario, and accepting user input to edit one or more of the set of time and / or relationship constraints for each one or more of the interactions; regenerating and rendering on the display a new interactive visualization of the scenario comprising the one or more edited interactions; The method of any one of claims 1 to 4, comprising:

6. The method of claim 3 , wherein the selected instant is later in the timeline than the initial instant.

7. The method of claim 3 , wherein the selected instant is earlier in the timeline than the initial instant.

8. The method of claim 1 , comprising rendering a visual representation of the timeline associated with the timing control on the display.

9. 9. The method of claim 8, comprising defining a start constraint that triggers the interaction in the scenario, and rendering on the display a first visual representation of a set of moments prior to the start constraint and a second visual representation of a set of moments during the interaction.

10. 10. The method of claim 1, further comprising presenting to the user on the display a playback control which, when selected by the user, causes a dynamic visualization of the scenario to be played back from the currently selected moment.

11. 10. A method according to any preceding claim, comprising presenting to the user on the display a playback control which, when selected by the user, causes a dynamic visualization of the scenario to be played back from the start of the scenario.

12. 10. A method according to any preceding claim, comprising presenting to the user on the display a map view in which at least one selectable location is rendered in a map corresponding to a selectable moment in time.

13. 1. A computer system for generating scenarios to be executed in a simulation environment for testing the behavior of an autonomous vehicle, comprising: a user interface configured to display an interactive visualization for editing of a scenario model comprising one or more interactions between an ego-vehicle object and one or more dynamic opponent objects, each interaction being defined as a set of time constraints and / or relationship constraints between the ego-vehicle object and at least one of the opponent objects, the scenario model comprising a scene topology, the interactive visualization including scene objects in which the ego-vehicle object and the at least one opponent object are displayed in the scene topology, and the scenario is associated with a timeline extending in a traveling direction of the ego-vehicle object relative to the scene topology; rendering timing controls in the user interface for selecting a moment along the timeline in response to user input from a user engaging with the user interface; generating in the user interface an interactive visualization of the scene topology and scene objects of the scenario displayed at the selected moment; a processor configured to: A computer system comprising:

14. The computer system of claim 13 , wherein the processor is configured to generate the interactive visualization from stored parameters of the scenario model.

15. A non-transitory computer readable medium having stored thereon computer readable instructions that, when executed by one or more processors, perform the method of any of claims 1 to 12.

Images