Mobile robot support tool

Abstract

A computer-implemented method of generating a static layer and / or a dynamic layer for a test scene for performance testing of a robotic system in a simulation environment, the method comprising: inputting to a geometric constraint solver a scene framework defining geometric elements and geometric constraints on the geometric elements, the geometric element may be modified based on the set of geometric variables, and the geometric constraint is characterized by one or more constraint parameters such that the geometric constraint solver determines one or more functions of the constraint parameters to return values of the geometric variables that satisfy the geometric constraint; receiving a value of the constraint parameter; calculating a value of a geometric variable that satisfies the geometric constraint using the received value of the constraint parameter and one or more functions; and configuring a static layer of the test scene and / or a dynamic layer of the test scene based on the received value of the constraint parameter, the calculated value of the geometric variable and the scene framework.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to support tools for autonomous vehicles and other mobile robots. BACKGROUND

[0002] The field of autonomous vehicles (AVs) has seen significant and rapid development. An autonomous vehicle (AV) is a vehicle equipped with sensors and control systems that enable it to operate without human control of its behavior. Autonomous vehicles are equipped with sensors that enable them to perceive their physical environment, such sensors include, for example, cameras, radar, and lidar. Autonomous vehicles are equipped with appropriately programmed computers that are able to process data received from the sensors and make safe and predictable decisions based on the context perceived by the sensors.

[0003] Autonomous vehicles can be fully autonomous (in that they are designed to operate without human oversight or intervention, at least in certain situations) or semi-autonomous. Semi-autonomous systems require varying degrees of human oversight and intervention. Advanced Driver Assist Systems (ADAS) and certain levels of Autonomous Driving Systems (ADS) can be classified as semi-autonomous. A “Level 5” vehicle is one that can operate fully autonomously in all situations, as it is always guaranteed to reach a certain minimum level of safety. Such a vehicle does not require manual controls (steering wheel, pedals, etc.) at all. In contrast, Level 3 and Level 4 vehicles can operate fully autonomously, but only in certain specific situations (e.g., within a geofenced area). Level 3 vehicles must be equipped with autonomous handling of any situation that requires an immediate response (such as emergency braking, etc.); however, changes in situations can trigger “transitional demands” that require the driver to take control of the vehicle within a limited timeframe. Level 4 vehicles have similar limitations; however, Level 4 vehicles must also be able to autonomously implement a “minimum risk maneuver” (MRM), i.e., take some appropriate action to bring the vehicle to a safe condition (e.g., slow down and stop), if the driver does not react within the prescribed timeframe. Level 2 vehicles require the driver to be ready to intervene at all times, and the driver is responsible for intervening if the autonomous system is not responding properly at any time. For Level 2 automation, the responsibility for determining when intervention is required is on the driver; for Levels 3 and 4, this responsibility shifts to the autonomous system of the vehicle, which must alert the driver when intervention is required.

[0004] Physical world testing will still be an important factor in testing the ability of autonomous vehicles to make safe and predictable decisions. However, testing in the physical world is both expensive and time consuming. Moreover, as AV systems improve in safety, the number of accidents per mile driven is decreasing. To gain widespread acceptance of autonomous vehicles, it is estimated that autonomous vehicles should not cause more than 1 serious accident per 10Λ9 miles driven (in comparison, an average human driver causes 1 such accident per 10Λ6 miles driven). It is simply not feasible to identify and mitigate problems based on “miles driven” in the real world for an autonomous vehicle that is operating orders of magnitude below the minimum performance level ultimately required.

[0005] Therefore, there is an increasing reliance on testing using simulated environments. If testing in simulated environments is to be increased, it is desirable that these environments reflect real world scenarios as closely as possible. Autonomous vehicles need to be able to operate in a wide variety of situations that human drivers can operate in. Such situations can be highly unpredictable. There is an increasing focus on the creation of simulated environments that can provide such testing in a way that one can have confidence that the results of the testing represent the potential real-world behavior of autonomous vehicles.

[0006] Scenarios must be described in sufficient detail and accuracy in a form that facilitates high quality simulation, otherwise the results of the testing will not adequately reflect real-world AV performance. A typical driving scenario includes a static road layout and various dynamic agents (other vehicles, pedestrians, cyclists, animals, etc.) that the autonomous vehicle (ego vehicle) needs to navigate. The ego vehicle can need to predict the motion of other agents and plan safely in complex road networks. In an offline context, the scenario description can be required as input to a simulator in order to perform simulation-based testing of the autonomous vehicle stack before deployment onto a real-world vehicle.

[0007] For example, ASAM (Association for Standardization of Automation and Measuring Systems) OpenSCENARIO (R) defines a file format for describing dynamic driving scenario content. OpenSCENARIO can be used together with the ASAM OpenDRIVE (R) schema for describing static road networks. The stated purpose of OpenDRIVE is "to provide a road network description that can be fed into a simulation to develop and validate ADAS and Autonomous Driving (AD) functionality"; see ASAM OpenDRIVE V1.7.0 (and accompanying user guide), published on August 3, 2021 [available at https: / / www.asam.net / standards / detail / opendrive / ].

[0008] Other mobile robots, for example for transporting goods in and outside industrial zones, are under development. Such mobile robots will have no human on board, and belong to a class of mobile robots called unmanned autonomous vehicles (UAVs). Autonomous aerial mobile robots (drones) are also under development. SUMMARY

[0009] "Relevance" (ensuring that scenarios are actually useful for testing) is a core issue for scalable simulation-based robotic system testing. When testing the performance of a robotic system in a large number of scenarios, it is important to guarantee relevance. It is simple to generate and run many simulation scenarios at scale. However, it is more challenging to ensure that these scenarios remain relevant. One class of irrelevant scenarios is those that are not realistic at all. Testing the performance of a mobile robot in scenarios that do not adequately reflect the real world will not yield informative results. As another example, a parameterized scenario can be designed to test the reaction of an ego agent to a certain type of event (such as another agent cutting in), but such an event does not actually occur in all parameterized scenarios. Test results obtained in irrelevant scenarios provide limited insight, and worse, can distort results in a way that gives a misleading portrayal of real-world performance. Worst of all, a batch of simulation test results for irrelevant scenarios can indicate that an AV system is performing at a given safety level, when in reality that safety level cannot be achieved in the real world.

[0010] For the large-scale simulation type required in AV testing, it is not feasible to manually check the significance of each scenario. Recall that a “safe” AV vehicle should cause no more than 1 serious accident per 10^9 miles driven. Verifying this level of performance in simulation requires running several orders of magnitude more simulations than 10^9.

[0011] Herein, there is provided a scenario design tool that allows test scenarios to be generated in a flexible manner, and hard-codes significance constraints at the design stage.

[0012] A first aspect of the present disclosure provides a computer-implemented method of generating a static layer and / or a dynamic layer for a test scenario for performance testing of a robotic system in a simulation environment, the method comprising: inputting, to a geometric constraint solver, a scenario framework defining geometric elements and geometric constraints on the geometric elements, the geometric elements being modifiable based on a set of geometric variables, and the geometric constraints being characterised by one or more constraint parameters, thereby causing the geometric constraint solver to determine one or more functions of the constraint parameters to return values of the geometric variables that satisfy the geometric constraints; receiving values of the constraint parameters; using the received values of the constraint parameters and the one or more functions to calculate values of the geometric variables that satisfy the geometric constraints; and configuring a static layer of the test scenario and / or a dynamic layer of the test scenario based on the received values of the constraint parameters, the calculated values of the geometric variables, and the scenario framework.

[0013] In embodiments, the method can comprise: receiving second values of the constraint parameter arguments; using the received second values of the constraint parameters and the one or more functions to calculate second values of the geometric variables that satisfy the geometric constraints; and configuring a second static layer of the test scenario and / or a second dynamic layer of the test scenario based on the received second values of the constraint parameters, the calculated second values of the geometric variables, and the scenario framework.

[0014] In this way, the same solution (function) can be used to generate multiple static / dynamic layers that satisfy the constraint conditions with different values of the constraint parameters (new values of the geometric variables are obtained by simply “plugging in” new values of the constraint parameters), without needing to re-solve the constraint. This allows multiple significant variants of a scenario to be generated (all satisfying the designer’s intent, as expressed in the constraints) with higher computational efficiency for testing purposes, and the solution to the constraint problem only needs to be determined once.

[0015] The method can comprise receiving initial values of the geometric variables. The received values of the constraint parameters and the received initial values of the geometric variables can not satisfy the geometric constraints, and the initial values of the geometric variables can be provided to the geometric constraint solver for use in determining the functions of the constraint parameters.

[0016] The geometric constraints can be represented in the form of a constraint graph, and the geometric solver can extract functions from the constraint graph using a plurality of predetermined derivation rules.

[0017] The geometric elements can comprise road elements, and the geometric constraints can comprise constraints on the road elements. The received values and the computed values can be used to generate a static layer comprising a road layout satisfying the geometric constraints.

[0018] The scenario framework can be generated from scenario creation input received at a graphical user interface.

[0019] The scenario framework can be generated from a selected region of the map, and at least some of the geometric elements can correspond to respective map elements within the selected region of the map.

[0020] The method can comprise storing associations between at least some of the geometric elements and respective map elements, and the static layer can be generated from the map according to the values of the geometric variables and the stored associations.

[0021] The geometric constraints can comprise inter-element geometric constraints between respective subsets of the geometric elements.

[0022] The method can comprise executing a test scenario comprising at least one of the static layer and the dynamic layer, the ego agent of the test scenario being controlled by the robot planner under test; and generating a test result set by a test oracle for evaluating performance of the robot planner in the test scenario.

[0023] The robot planner can be tested in conjunction with one or more other components (e.g. one or more of a perception system, a prediction system, and a controller).

[0024] The method can comprise using the test result set to identify and mitigate problems in the robot planner or other components tested in conjunction with the robot planner.

[0025] Other aspects herein provide a computer system comprising one or more computers configured to implement the method of the first aspect or any embodiment thereof, and computer-readable instructions stored in a transitory or non-transitory medium, the computer-readable instructions being configured to implement the method when executed in the computer system. BRIEF DESCRIPTION OF DRAWINGS

[0026] The illustrative embodiments will now be described, by way of example only, with reference to the following schematic drawings, in which:

[0027] Figure 1A A schematic functional block diagram of an autonomous vehicle stack is shown;

[0028] Figure 1BA schematic overview of the autonomous vehicle testing paradigm is shown;

[0029] Figure 1C A schematic block diagram of the scene extraction pipeline is shown;

[0030] Figure 2 A schematic block diagram of the testing pipeline is shown;

[0031] Figure 3 A schematic block diagram of the visualization component for presenting a graphical user interface displaying test results is shown;

[0032] Figure 4 Views available within the graphical user interface are shown;

[0033] Figure 5 An example road network is shown;

[0034] Figure 6 A portion of the road network annotated with OpenDRIVE elements for describing the road network is shown;

[0035] Figure 7 A lane graph of a static road layout is shown;

[0036] Figure 8 A schematic block diagram of the scene design tool is shown;

[0037] Figure 9 How the scene framework can be used to generate or modify road layouts within a map is shown;

[0038] Figure 10 An example of a geometric constraint problem with multiple solutions is shown. DETAILED DESCRIPTION

[0039] This paper describes a scene design tool that can construct a dimensioned geometric framework (or more generally, a static layer for testing scenes) for a map. The framework can be generated “from scratch” or alternatively, existing map fragments can be used as templates. The map framework geometry is then added to a constraint solver with the desired dimensions and constraints (applied during the design phase) and solved in default positions. The scene framework has variables that can be modified, and constraints introduce dependencies between these variables. Constraints are characterized by parameters, which are treated as independent variables (called “sizes” or “constraint parameters”), and geometric variables are treated as dependent variables (“unknowns”). In the geometric context, the dependent variable may be referred to as the “geometry”. The selection of independent variables is instructed to the constraint solver, whose task is to solve for the dependent variables. The solver primarily uses a predefined set of derivation rules to analytically derive solutions, meaning that each dependent variable is determined as a function of one or more independent variables (constraint parameters / sizes). Once completed, the frame can be continuously manipulated using its relevant dimensions (such as distance, length, angle, etc.) and deformed in a way that maintains the invariants required by the designer encapsulated within the constraints.

[0040] Constraint problems consist of a set of variables Defined by a set of constraints on the variables, which are determined by the size. To characterize, size It is a quantity that is considered as the independent variable. In the context of... In the selection process, the scenario designer instructs the user which variables they should retain control over. For a given... The goal of this selection is to solve for the dependent variable. (Dependent variable / unknown quantity). Regarding To solve "analytically" means to solve for each dependent variable Find a function that makes The constraints are satisfied. The solution is a set of functions. , usually for multiple The value is valid (it may be valid for all possible values). (Value or only valid for a subset of possible values). Once for a given... Once you find the right approach, you can effectively solve different problems by "plugging in" the substitution process. The value is used multiple times to generate variants of the frame (all variants satisfy the constraints).

[0041] Constraint problems may be "under-defined," or they may be well-defined but have multiple solution branches. As described further below, under-defined problems and well-defined problems with multiple solutions can be handled differently, and a tool's response to such problems may depend on the context. Analytical solutions typically provide a set of... The value is valid. However, in some cases, The value is valid. However, in some cases, The initial choice of value can guide the solver on how to solve an under-defined problem or a well-defined problem with multiple solution branches (e.g., favoring solution branches that have a size value The value is valid. However, in some cases, The value is valid. However, in some cases,

[0042] A binding between the dimension map framework and the static layer is created, and when the user manipulates the map framework, the static layer (typically much richer in content than the framework) dynamically reacts to the changes. In this way, the tool can generate a continuous set of map variants, which can then be used to form the basis of an autonomous system test suite. Preferably, the transformations are chosen such that the map topology (or static layer topology) is invariant under these transformations, making it relatively simple to execute the same dynamic layer on each map variant without needing to materially change the dynamic layer.

[0043] A scenario "test suite" refers to a set of specific scenarios (whose geometry is defined) that are related to some extent (e.g., corresponding to a "test suite of all passing scenarios" or some defined subset of passing scenarios, or a subset of scenarios with a particular road topology). In the current context, a scenario test suite can be defined by a scenario framework, with variants (specific scenarios) resulting from different parameterizations of the framework (all variants satisfying constraints imposed by the design phase). Note that the term "scenario" can be used herein to refer to either a specific scenario or a broader test suite, the meaning to be clear from the context.

[0044] Stationary parked vehicles and occluding objects that can be considered part of the map can be included in the map framework, so the approach is quite powerful.

[0045] The design tool can also be applied to impose some degree of constraint on the dynamic agents in the scenario, as described in further detail below.

[0046] Consider a test scenario in which a self-agent is required to navigate a real or modeled physical context. The self-agent is a real or simulated mobile robot that moves under the control of the stack being tested. The physical context includes static and / or dynamic elements that the stack being tested needs to respond to effectively. For example, the mobile robot can be a fully or semi-autonomous vehicle (a "self-vehicle") under the control of the stack. The physical context can include a static road layout and a given set of environmental conditions (e.g., weather, time of day, lighting conditions, humidity, pollution / particulate levels, etc.), which can remain constant or vary as the scenario progresses. The dynamic scenario also includes one or more other agents ("external" agents, e.g., other vehicles, pedestrians, cyclists, animals, etc.).

[0047] In the offline simulation context, a scenario description is provided as input to an offline simulator in order to expose the stack under test to the simulation scenario and test its performance in the simulation scenario. This allows performance issues (such as safety issues, etc.) to be identified and mitigated prior to real-world deployment. In the online context, a perception system can be used to generate a scenario description that can be used as a basis for advanced functions such as motion prediction and planning, which can involve some form of online simulation to simulate possible futures and plan accordingly.

[0048] The design tool is described below in the context of offline simulation to generate a suite of scenario tests for scalable performance testing.

[0049] The scenario description can be encoded using a scenario description language (SDL), or any other form that any of the components that need it can use. As briefly discussed, the ASAM OpenDRIVE (R) standard defines a storage format for static descriptions of road networks, and OpenSCENARIO (R) can be used to add dynamic content. Other forms of scenario description can be used, including custom languages and formats, and the present technology is not limited to any particular SDL, storage format, schema, or standard.

[0050] A “scenario run” or “scenario instance” refers to a specific event in which an agent navigates a physical context, optionally in the presence of one or more other agents. A single scenario description can result in multiple simulation runs, yielding different results, especially since these results depend on decisions made by the stack under test (which can change the outcome of a given specific scenario when changes are made to the stack to improve performance). In this context, the terms “run” and “instance” can be used interchangeably.

[0051] Figure 1A A highly schematic block diagram of an AV runtime stack 100 is shown in context. The stack 100 can be fully autonomous or semi-autonomous. For example, the stack 100 can operate as an autonomous driving system (ADS) or an advanced driver assistance system (ADAS).

[0052] The runtime stack 100 is shown as including a perception system 102, a prediction system 104, a planning system (planner) 106, and a control system (controller) 108.

[0053] In a real-world context, the perception system 102 receives sensor outputs from the AV’s on-board sensor system 110 and uses those sensor outputs to detect external agents and measure their physical states, such as their locations, velocities, accelerations, etc. The on-board sensor system 110 can take different forms, but generally includes various sensors such as image capture devices (cameras / optical sensors), lidar and / or radar units, satellite positioning sensors (global positioning system (GPS), etc.), motion / inertial sensors (accelerometers, gyroscopes, etc.), etc. Thus, the on-board sensor system 110 provides rich sensor data from which detailed information about the surrounding environment, as well as the state of the AV and any external participants (vehicles, pedestrians, cyclists, etc.) within that environment can be extracted. The sensor outputs typically include sensor data in multiple sensor modalities, such as stereo images from one or more stereo optical sensors, lidar, radar, etc. The sensor data in multiple sensor modalities can be combined using filters, fusion components, etc.

[0054] The perception system 102 generally includes multiple perception components that cooperate to interpret the sensor outputs, providing perception outputs to the prediction system 104.

[0055] In a simulation context, depending on the nature of the test, it can or can not be necessary to model the on-board sensor system 100. For example, when testing the planner 106 (or just the planner 106 and the controller 108) on a “perfect” representation of the scene (i.e., directly on the simulator ground truth), there is no need for simulated sensor data, and thus no need for complex sensor modeling. A stand-in model of the perception system 102 (or portions thereof) can also be used to test planner performance (or planner and controller performance) in the presence of real perception errors, without the need to use a sensor model.

[0056] The prediction system 104 uses the perception outputs from the perception system 102 to predict the future behavior of external actors (agents) such as other vehicles in the vicinity of the AV.

[0057] The predictions computed by the prediction system 104 are provided to the planner 106, which uses these predictions to make autonomous driving decisions for execution by the AV in a given driving scenario. The inputs received by the planner 106 will generally indicate a drivable region, and will also capture the predicted motions of any external agents (obstacles from the AV’s perspective) within the drivable region. The drivable region can be determined using perception outputs from the perception system 102 in conjunction with map information such as a high definition (HD) map.

[0058] A core function of the planner 106 is to plan a trajectory for the AV (ego trajectory) taking into account predicted agent motion. This can be referred to as trajectory planning. The planned trajectory is to achieve an intended goal in the scene. For example, the goal can be to enter a roundabout and exit at an intended exit; to overtake a vehicle in front; or to stay on the current lane at a target speed (lane following). The goal can be determined, for example, by the autonomous route planner 120 (also referred to as goal generator 120).

[0059] The controller 108 implements decisions made by the planner 106 by providing appropriate control signals to the AV’s on-board actuator system 112. In particular, the planner 106 plans a trajectory for the AV, and the controller 108 generates control signals to implement the planned trajectory. Typically, the planner 106 will plan into the future such that the planned trajectory can only be partially implemented at the control level before the planner 106 plans a new trajectory. The actuator system 112 includes “primary” vehicle systems such as braking, acceleration, and steering systems, as well as secondary systems (such as signals, wipers, headlights, etc.).

[0060] Figure 1A Examples of the architecture 100 consider a relatively “modular” architecture with separable perception, prediction, planning, and control systems 102-108. The sub-stacks themselves can also be modular, for example with separable planning modules within the planning system 106. For example, the planning system 106 can include multiple trajectory planning modules that can be applied to different physical contexts (e.g., simple lane driving versus complex intersections or roundabouts). This is relevant to simulation testing for the reasons described above, as it allows components (such as the planning system 106 or individual planning modules therein) to be tested individually or in different combinations. For the avoidance of doubt, for a modular stack architecture, the term stack can refer not only to the complete stack, but also to any individual subsystem or module therein.

[0061] There can be significant differences in the degree of integration or separability of various stack functions between different stack implementations— in some stacks, certain aspects can be tightly coupled and indistinguishable. For example, in other stacks, planning and control can be integrated together (e.g., these stacks can plan directly in terms of control signals), while other stacks (such as the architecture 100 shown) can explicitly distinguish between the two (e.g., plan in terms of trajectories, and determine how to best execute the planned trajectory at the control signal level through a separate control optimization). Likewise, in some stacks, prediction and planning can be more tightly coupled. In the extreme case, in so-called “end-to-end” driving, perception, prediction, planning, and control can be inseparable in nature. Unless otherwise indicated, the perception, prediction, planning, and control terminology used herein does not imply any particular coupling or modularity of these aspects. Figure 1A

[0062] ​A "complete" stack typically involves processing and interpretation of low-level sensor data (perception), feeding into major high-level functions such as prediction and planning, and control logic that generates appropriate control signals to implement planning-level decisions (e.g., control braking, steering, acceleration, etc.). For autonomous vehicles, Level 3 stacks include some logic that implements the transition demand, and Level 4 stacks also include some logic that implements the minimum risk operation. The stack can also implement ancillary control functions such as signals, headlamps, windshield wipers, etc.

[0063] While the following description refers to the stack 100 in the context of testing, the testing can apply to individual components / parts of the stack, such as the perception stack 102, the prediction stack 104, the planning stack 106, or the control stack (individually or in various combinations), or individual components thereof. The stack (or component) can refer to software only, i.e., one or more computer programs that can be executed on one or more general-purpose computer processors. However, such terminology can also include hardware. In simulation, the software of the stack can be tested on "generic" non-vehicle computer systems before being uploaded to the on-board computer systems of the physical vehicle. However, in "hardware-in-the-loop" testing, the testing can extend to the underlying hardware of the vehicle itself. For example, the stack software can be run on the on-board computer systems (or copies thereof) coupled with a simulator for testing. In this context, the stack under testing extends to the underlying computer hardware of the vehicle. As another example, certain functions of the stack 100 (e.g., perception functions) can be implemented in dedicated hardware. In the context of simulation, hardware-in-the-loop testing can involve feeding synthetic sensor data to the dedicated hardware perception components.

[0064] Within the stack 100, the scene description 116 can be used as a basis for planning and prediction. The scene description 116 is generated using the perception system 102, as well as a high-definition (HD) map 114. By localizing the ego vehicle 114 on the map, the information extracted in the perception system 102 (including dynamic agent information) can be combined with pre-existing environmental information contained in the HD map 114. The scene description 116, in turn, is used as a basis for motion prediction in the prediction system 104, and the resulting motion predictions 118 are used in conjunction with the scene description 116 as a basis for planning in the planning system 106.

[0065] Figure 1B A highly schematic overview of the autonomous vehicle testing paradigm is shown. By running multiple instances of scenarios in the simulator 202, and evaluating the performance of the stack 100 (and / or individual sub-stacks therein) in the test oracle 252, the ADS / ADAS stack 100 (e.g., the perception stack 102, the prediction stack 104, the planning stack 106, or the control stack) can be tested in a simulated environment. Figure 1AThe output of the test oracle 252 provides information to the experts 122 (team or individuals) allowing them to identify problems in the stack 100 and modify the stack 100 to mitigate those problems (S124). The results also help the experts 122 select further test scenarios (S126), and the process continues, with the stack 100 being repeatedly modified, tested, and evaluated in simulation. The improved stack 100 is eventually incorporated (S125) in a real-world AV 101, which is equipped with the sensor system 110 and the effector system 112. The improved stack 100 typically comprises program instructions (software) that are executed in one or more computer processors of an on-board computer system (not shown) of the vehicle 101. At step S125, the software of the improved stack is uploaded to the AV 101. Step S125 can also involve modifications to the underlying vehicle hardware. On the AV 101, the improved stack 100 receives sensor data from the sensor system 110 and outputs control signals to the effector system 112. Real-world testing (S128) can be used in conjunction with simulation-based testing. For example, after acceptable levels of performance are reached through simulation testing and stack refinement processes, appropriate real-world scenarios can be selected (S130), and the performance of the AV 101 in these real-world scenarios can be captured in the test oracle 252 and evaluated similarly.

[0066] Figure 1C A highly schematic block diagram of the test pipeline applied to the scenario description 148 is shown. The scenario generator 146 generates the scenario description 148 in a form that can be used by the simulator 202, allowing multiple simulation runs to be derived from it (e.g., using different configurations of the AV stack 100). A ground truth 150 is provided for each simulation run.

[0067] The ground truth 150 for each run includes the trajectories of the ego agent and any other dynamic agents in the scenario. A trajectory is a history of the position and motion of an agent in the course of the scenario. There are many ways to represent a trajectory. Trajectory data typically includes spatial and motion data for the agents in the environment.

[0068] Further details of an example test pipeline incorporating the test oracle 252 will now be described.

[0069] Figure 2A schematic block diagram of a test pipeline is shown, denoted by reference numeral 200. The test pipeline 200 is shown to comprise a simulator 202 and a test oracle 252. For the purpose of testing the entire or a part of the AV runtime stack 100, the simulator 202 runs a simulated scenario, and the test oracle 252 evaluates the performance of the stack (or sub-stack) on the simulated scenario. As discussed, it is possible to only test a sub-stack of the runtime stack, but for simplicity, the following description always relates to the (complete) AV stack 100. However, the description is equally applicable to a sub-stack instead of the complete stack 100. The term“slice” is used herein as a collection or subset of stack components that are selected for testing.

[0070] The idea of testing based on simulation is to run a simulated driving scenario, which the ego agent has to navigate under the control of the stack 100 that is being tested. Typically, the scenario comprises a static drivable area (e.g. a specific static road layout) that the ego agent has to navigate, typically in the presence of one or more other dynamic agents (like other vehicles, bicycles, pedestrians, etc.). To this end, the simulator 202 provides simulated inputs 203 to the stack 100 that is being tested.

[0071] The slice of the stack determines the form of the simulated inputs 203. As an example, Figure 2 The prediction system 104, the planning system 106 and the control system 108 within the AV stack 100 that is being tested are shown. For testing Figure 1A the complete AV stack, the perception system 102 can also be applied during testing. In this case, the simulated inputs 203 will comprise synthetic sensor data that is generated using appropriate sensor models, and processed within the perception system 102 in the same way as real sensor data. This requires generating synthetic sensor inputs that are realistic enough (such as photo-realistic image data and / or equally realistic simulated lidar / radar data, etc.). The resulting outputs of the perception system 102 will in turn be fed to the higher-level prediction system 104 and planning system 106.

[0072] By contrast, a so-called“planning-level” simulation would essentially bypass the perception system 102. The simulator 202 would provide much simpler, higher-level inputs 203 directly to the prediction system 104. In some contexts, it is even possible to bypass the prediction system 104 in order to test the planner 106 according to predictions that are obtained directly from the simulated scenario (i.e.“perfect” predictions).

[0073] Between these extremes, there is a range of different levels of input slicing, e.g. testing only a subset of the perception system 102, such as“subsequent” (higher-level) perception components (e.g. components such as filters or fusion components that operate on outputs from lower-level perception components such as object detectors, bounding box detectors, motion detectors, etc.), etc.

[0074] As an alternative to generating high-fidelity synthetic sensor data to feed to the perception system 102, the perception system 102 can be modeled in whole or in part using one or more “surrogate” perception models. Surrogate models model the perception system 102 (or certain components thereof), but operate more efficiently on low-fidelity inputs derived from simulated ground truth without the need for sensor models. Surrogate models allow simulation inputs 203 to be generated with realistic errors (comparable to those produced by the perception system 102 in similar real-world situations) without using sensor models.

[0075] Regardless of their form, the simulation inputs 203 are used (directly or indirectly) as the basis for planner 106 decisions. The controller 108 in turn implements the planner’s decisions by outputting control signals 109. In a real-world context, these control signals would drive the AV’s physical actuator system 112. In simulation, a ego vehicle dynamics model 204 is used to translate the resulting control signals 109 into realistic motion of the ego agent in the simulation, thereby simulating the physical response of the autonomous vehicle to the control signals 109.

[0076] Alternatively, a simpler form of simulation assumes that the ego agent precisely follows each planned trajectory between steps of the plan. This approach bypasses the control system 108 (to the extent it can be separated from the plan) and eliminates the need for an ego vehicle dynamics model 204. This can be sufficient to test certain aspects of the plan.

[0077] In cases where external agents exhibit autonomous behavior / decision making within the simulator 202, some form of agent decision logic 210 is implemented to execute those decisions and determine the agent’s behavior in the scenario. The agent decision logic 210 can be comparable in complexity to the ego stack 100 itself, or it can have more limited decision capabilities. The goal is to provide sufficiently realistic external agent behavior within the simulator 202 to enable effective testing of the ego stack 100’s decision capabilities. In some contexts, this can not require any agent decision making logic 210 at all (open loop simulation), while in other contexts, relatively limited agent logic 210 (such as basic adaptive cruise control (ACC)) can provide useful testing. If appropriate, one or more agent dynamics models 206 can be used to provide more realistic agent behavior.

[0078] The scenario is run according to the scenario description 201, which typically has both static and dynamic elements. The scenario run is orchestrated by a test orchestration component 260.

[0079] Static elements are encoded in a static layer 201a of the scene description 201. Static elements typically include road layout. For example, the static layer 201a can be encoded in OpenDrive format, or in a similar road network SDL that describes both the geometry and topology of the road layout.

[0080] Dynamic elements are encoded in a dynamic layer 201b, and typically include one or more external agents in the scene, such as other vehicles, pedestrians, bicycles, etc.

[0081] The extent of dynamic information provided to the simulator 202 for each external agent can vary. For example, a scene can be described with separable static and dynamic layers. A given static layer (e.g. defining road layout) can be used in conjunction with different dynamic layers to provide different scene instances. For each external agent, the dynamic layer can include one or both of a spatial path to be followed by the agent, and motion data and behavior data associated with that path. In a simple open-loop simulation, the external participant simply follows the spatial path and motion data defined in the dynamic layer, and the external participant is non-reactive, i.e. does not react to the ego agent in the simulation. Such an open-loop simulation can be implemented without any agent decision logic 210. However, in a closed-loop simulation, the dynamic layer instead defines at least one behavior (e.g. an ACC behavior) to be followed along the static path. In this case, the agent decision logic 210 implements the behavior reactively in the simulation, i.e. in reaction to the ego agent and / or other external agents. The motion data can still be associated with the static path, but in this case it is less prescriptive, e.g. can be a target along the path. For example, for an ACC behavior, a target speed can be set along the path that the agent will seek to match, but the agent decision logic 210 can be allowed to reduce the speed of the external agent below the target at any point along the path, to maintain a target separation from a vehicle ahead.

[0082] The output of the simulator 202 for a given simulation includes an ego trajectory 212a for the ego agent, and one or more agent trajectories 212b for one or more external agents (trajectories 212). Each trajectory 212a, 212b is a complete history of the behavior of the agent within the simulation, with both spatial and motion components. For example, each trajectory 212a, 212b can take the form of a spatial path with motion data associated with points along the path, such as speed, acceleration, jerk (rate of change of acceleration), snap (rate of change of jerk), etc.

[0083] Additional information is also provided to supplement the trajectories 212 and provide context for them. This additional information is referred to as "context" data 214. The context data 214 is related to the physical context of the scenario and can have static components (such as road layout, etc.) and dynamic components (such as weather conditions that vary to some extent over the course of the simulation, etc.).

[0084] The test oracle 252 receives the trajectories 212 and the context data 214 and scores these outputs according to a set of performance evaluation rules 254. The performance evaluation rules 254 are shown to provide an overview of the inputs to the test oracle 252.

[0085] The rules 254 are categorical in nature (e.g., pass / fail type rules). Certain performance evaluation rules are also associated with numerical performance metrics that are used to "score" the trajectories (e.g., to indicate a degree of success or failure, or to indicate other quantities that are helpful to interpret or otherwise correlate with the categorical results). The evaluation of the rules 254 is time-based - a given rule can have different outcomes at different points in the scenario. The scoring is also time-based: for each performance evaluation metric, the test oracle 252 tracks how the value of the metric ("robustness" score) changes over time as the simulation progresses. The test oracle 252 provides an output 256 that includes a time series of categorical (e.g., pass / fail) results for each rule 256a, as well as a score time graph for each performance metric 256b, as described in further detail later. The results 256a and scores 256b provide information to the expert 122 that can be used to identify and mitigate performance issues within the test stack 100. The test oracle 252 also provides overall (aggregated) results for the scenario (e.g., overall pass / overall fail). The output 256 of the test oracle 252 is stored in the test database 258 in association with information about the scenario to which the output 256 pertains.

[0086] Figure 3 A schematic block diagram of the visualization component 320 is shown. The visualization component 320 is shown as having an input connected to the test database 258 for presenting the output 256 of the test oracle 252 on a graphical user interface (GUI) 300. The GUI is presented on a display system 322.

[0087] Figure 4An example view of the GUI 300 is shown. This view relates to a particular scenario containing multiple agents, and is shown to include a scene visualization 301 and a set of driving performance assessment results 302. In this example, the test forecaster output 256 relates to multiple external agents, and the results are organized according to the agents. For each agent, at some point in the scene, there is a time series of results for each rule that applies to that agent. Some form of visual indicator (such as a coding) is used to distinguish pass / fail periods for a particular rule.

[0088] Figure 5 An example road network 500 encoded in the static layer 201a is schematically depicted. The following description assumes that the road network 500 is described using the OpenDRIVE schema or a similar format that adopts certain definitions and conventions from OpenDRIVE. However, it will be appreciated that these principles extend more generally to other formats, and that the described techniques are not limited to any particular data format or schema.

[0089] The road network 500 is shown to include a first road 502, a second road 504, a third road 506, and a fourth road 508 (roads 1 to 4) with <Road> elements having road identifiers (IDs) 1, 2, 3, and 4, respectively. The road network 500 also includes a first intersection 510, a second intersection 512, and a third intersection 514 (intersections 1 to 3) with <Intersection> elements having intersection IDs 1, 2, and 3, respectively. <road>) elements to describe. Roads 502-508 pass through intersections denoted by <intersection> ( <junction>The intersections 510 are interconnected by elements described below. Each road 502-508 is defined by a single road reference line, represented by a thick solid arrow, and contains a single center lane with zero width. The center lane is not depicted separately, and for simplicity it is assumed that the center lane of each road 502-508 follows the road reference line (although as mentioned above, a non-zero offset can be defined between the road reference line and the center lane). The road reference lines of the first road 502 and the second road 504 are denoted by reference numerals 503 and 505, respectively. The road reference line is directional and can be more precisely described as the longitudinal axis or "s-axis" of the road, along which the s-coordinate extends, and the t-coordinate is orthogonal thereto. As shown by the reference lines 503, 505, the positive t-direction is defined to extend to the left of the s-axis. The lanes are numbered relative to the center lane; the center lane is always numbered 0, and the side lanes to the left of the center lane (+t) are assigned increasing positive lane numbers (1, 2, 3,...), and the lanes to the right of the center lane (-t) are assigned decreasing negative lane numbers (-1, -2, -3,...).

[0090] Each road has at least one side lane. In this example, the first road 502 has only positive-numbered side lanes to the left of the center lane, while the second road 504 has two positive-numbered side lanes to the left of the center lane and one negative-numbered side lane to the right of the center lane. Roads can be divided into "lane segments" to accommodate road segments with different numbers of lanes (see below).

[0091] In the following description, "lane n" refers to the lane with lane number "n", and "road m" refers to the road with road ID "m". Other road elements, such as intersections, are referred to in the same way.

[0092] Adjacent lane segments can be referred to as "lane segment n" and "lane segment n+1" (the lane segment numbers n, n+1 do not form part of the OpenDRIVE schema, but can still be derived from the static layer 201a).

[0093] A left-hand traffic (LHT) road system is described in this example. However, the schema can be used for LHT or right-hand traffic (RHT) networks. Each road is defined by a single road reference line, represented by a thick solid arrow, and contains a single center lane with zero width. The center lane is not depicted separately, and for simplicity it is assumed that the center lane of each road follows the road reference line (although as mentioned above, a non-zero offset can be defined between the road reference line and the center lane). The road reference lines of the first road 502 and the second road 504 are denoted by reference numerals 503 and 505, respectively. The road reference line is directional and can be more precisely described as the longitudinal axis or "s-axis" of the road, along which the s-coordinate extends, and the t-coordinate is orthogonal thereto. As shown by the reference lines 503, 505, the positive t-direction is defined to extend to the left of the s-axis. The lanes are numbered relative to the center lane; the center lane is always numbered 0, and the side lanes to the left of the center lane (+t) are assigned increasing positive lane numbers (1, 2, 3,...), and the lanes to the right of the center lane (-t) are assigned decreasing negative lane numbers (-1, -2, -3,...). <road>The "rule" attribute of an element indicates whether the road is LHT (vehicles drive on the left) or RHT (vehicles drive on the right).

[0094] A global Cartesian coordinate system is defined, with the x-direction pointing east and the y-axis extending north (OpenDRIVE calls this the inertial coordinate system).

[0095] The lane number only indicates the relative direction of traffic flow within a road: for any given road, the traffic flow direction is the same for positively numbered single-side lanes, and opposite for negatively numbered single-side lanes. Bi-directional lanes support bi-directional traffic flow, regardless of the road traffic rules. However, the lane number alone is not sufficient to infer the direction of traffic flow, as the direction of the s-axis can be chosen (more or less) arbitrarily and does not indicate the direction of travel. For example, in Figure 5 , the s-axis 505 of the second road 504 extends from east towards the intersection, while the s-axis of the fourth road 508 extends from west towards the intersection 510 in the opposite direction. Thus, along the second road 504, positive lane numbers indicate the direction of traffic flow from east to west, while along the fourth road 508, the east-west traffic flow is indicated by negative lane numbers. For LHT roads, the lanes to the left of the road reference line (+t) carry traffic in the direction of the road reference line (+s), while the lanes to the right of the road reference line (-t) carry traffic in the opposite direction (-s). In RHT roads, the lanes to the left of the road centerline (+t) carry traffic in the opposite direction of the road reference line (-s), while the lanes to the right (-t) carry traffic in the direction of the road reference line (+s).

[0096] It is also possible to specify the direction of traffic flow by including a <direction> element within the <lane> element <lane>The type (@type) attribute set to "two-way" defines a two-way side lane, allowing traffic to flow in both directions. Two-way lanes will be described in more detail below.

[0097] Therefore, to determine the absolute direction of traffic flow, the number of lanes is not enough; the direction of the road reference line (s-axis) must also be considered, as well as the @type attribute.

[0098] Lanes are not necessarily drivable. For example, lane 1 of lane segment 2 of road 2 is not drivable. The outer bounds of a road can also be defined by non-drivable lanes, such as a pedestrian / bike lane type.

[0099] The link element is used to explicitly define links between roads and lanes. If there are only two roads, the link can be defined directly with a link element between the roads, provided the link is unambiguous. More complex connections require the use of an intersection element.

[0100] <LINK> ( <link> The element can contain <successor> ( <successor>) element and <predecessor> ( <predecessor>one or both of the elements. A predecessor / successor of a road can be another road or an intersection. A predecessor / successor of a lane is another lane. The "predecessor" and "successor" relationship is defined relative to the s-axis of the road in question (the t-axis of a road extends from its predecessor, if any, to its successor, if any) and does not indicate the direction of travel. The s-axis of a road extends from its predecessor, if any, to its successor, if any.

[0101] The predecessor / successor relationship can be "asymmetric". For example, if road n is the predecessor of road m, this does not mean that road m is the successor of road n; road m can also be the predecessor of road n if the s-axis of road n is oriented in the opposite direction of the s-axis of road m. Another example is that if road m is part of an intersection, then road m cannot be the predecessor or successor of road n, as the intersection would be the predecessor / successor of road n (see more examples below).

[0102] Roads with different numbers of lanes are considered using lane segments. In the OpenDRIVE schema, a lane is a "lane" in the sense of a single lane of a multi-lane road. A lane segment is a "lane" in the sense of a single lane of a multi-lane road, but it is a segment of a lane, i.e. it has a start and an end. <road><lane> of elements <lanes>as described in the element. <lanes>Elements can be used with <lane segment> ( <lanesection>) elements are split into multiple segments (lane segments). Each individual lane is represented by <lanesection>In the element <lane>Element definitions. Each lane segment has a fixed number of lanes, numbered according to the convention described above. For example, the first road 502 is shown divided into two lane segments (lane segment 1 and lane segment 2 of road 1): the number of lanes increases from two to three as the intersection 510 is approached. As one enters lane segment 2 from lane segment 1 of the first road 502, a new rightmost lane is created, which starts with zero width and gradually increases in width. In lane segment 1, the leftmost and rightmost lanes (center lane left) are numbered 2 and 1, respectively, while in lane segment 2, the leftmost lane and the new rightmost lane are numbered 3 and 1, respectively; the center lane is now numbered 2.

[0103] Use of link elements between lanes in adjacent lane segments <link> The lane elements are described in more detail. In lane segment 2, lane 2 of lane segment 1 is the predecessor of lane 3 of lane segment 2, and lane 1 of lane segment 1 is the predecessor of lane 2 of lane segment 2. That is, the lane element describing lane 3 of lane segment 2 will contain a link element indicating lane 2 as its predecessor, and it implies that the link refers to the immediately preceding lane segment (lane segment 1), etc.

[0104] Similarly, in lane 1, lane 3 of lane segment 2 is the successor of lane 2 of lane segment 1, and lane 2 of lane segment 2 is the successor of lane 1 in lane segment 1. That is, the lane element describing lane 2 of lane segment 1 will indicate lane 3 as its successor, and it implies that the link refers to the next lane segment (lane segment 2), etc.

[0105] Since lane 1 of lane segment 2 initially has zero width, it does not link to any lane of lane segment 1. However, in different cases, a lane can have multiple successors or predecessors, for example, if the lane immediately splits into two lanes, each with non-zero width at the lane segment boundary.

[0106] In Figure 5 Within the intersection element 510, additional roads are defined, whose reference lines are represented by thick solid arrows. In this example, fifth through ninth roads 512, 514, 516, 518, 520 (roads 5 through 9) are depicted within the intersection 510. Although Figure 5 Although the side lanes within the intersection 510 are not depicted in the

[0107] In Figure 5 In the intersection 510, the first road 502, the second road 504, and the fourth road 508 are successors, as their s-axes extend toward the intersection 510. Thus, the <road>The element will contain <link> The element, wherein <successor>The element indicates an intersection 510. The intersection 510 is the predecessor of the third road 506 because its s-axis extends away from the intersection, thus describing the third road 506 <road>The element will contain <link> The element and the indication of the intersection 510 <predecessor>Element.

[0108] Within the intersection element 510, the connecting roads are represented by <road>and <connect> ( <connection>Element Description.

[0109] A fifth road 512 (Road 5) is shown connecting the first road 502 and the third road 506. The fifth road 512 is formed by <junction>within element 510 <road>Element definition, where the first road 502 is the predecessor and the third road 506 is the successor (by describing the fifth road 506 <road>in the element <predecessor>and <successor>Similarly, the sixth road 514 has the fourth road 508 as its successor and the second road 504 as its predecessor. The seventh road 516 connects the second road 504 and the third road 506, with the second road 504 as its predecessor and the third road 506 as its successor. The eighth road 518 connects the second road 504 and the first road 502, with the second road 504 as the successor road and the first road 502 as the predecessor road. Finally, the ninth road 520 connects the first road 502 and the fourth road 508, with the fourth road 508 as its successor and the first road 502 as its predecessor. Again, the predecessor / successor relationship is not a direct indicator of the direction of travel.

[0110] <connection>The elements are used to indicate the direction of travel of traffic flows joining the junction. The connection element indicates the approach road and the connecting road (but does not explicitly define the exit road) from which the traffic flow enters the junction from the approach road. For example, the first <connection>elements, indicating that the road with ID = 1 (the first road 502) is an incoming road, and the road with ID = 5 (the fifth road 512) is a connecting road; the connecting element indicates that traffic flow can enter the intersection 510 from the first road 502 to the fifth road 512. The second <connection>The elements will indicate that the first road 502 is an inbound road, and the eighth road 518 is a connector road, and so on. In this example, the seventh connector road 516 is a two-way road, carrying traffic flow from the second road 504 to the third road 506 and vice versa. Therefore, two <connection>The elements, one indicating that the second road is an inlet, and the seventh road 516 is a connection, and the other indicating that the third road 506 is an inlet, and the seventh road 516 is a connection. The OpenDRIVE specification strongly recommends against using double-line connected roads, although it is possible to implement double-line connected roads using the schema. The seventh connected road 516 violates this recommendation, but does not violate the specification (and, in practice, it is more likely that two single-line connected roads would be defined). The fourth road 508 is not an inlet road to the intersection 510 (even though its s-axis extends toward the intersection 510) because it is a single-line road that only carries traffic flow away from the intersection 510.

[0111] When lane changes are allowed within the intersection 510, a connected road with multiple lanes is used; if lane changes are not allowed, a multiple single-lane road is used.

[0112] A "virtual" connection can also be used without any connected roads <connection>Elements are described to virtual connections are limited to "virtual" intersections, no need to split the main road.

[0113] Links between lanes are described by describing the lanes <lane>The link elements in the element are used to describe the predecessor and successor lanes in a manner similar to the roads. In order for a first lane of a first road to be a predecessor or successor of a second lane of a second road, the second road must be a predecessor or successor of the first road. Since the intersection 510 is a successor of the first road 502, the second road 504, and the fourth road 508, these roads 502, 504, 508 do not have a direct successor road; therefore, none of their lanes can have a lane successor. Likewise, since the intersection 510 is a predecessor of the third road 506, the third road 506 does not have a direct predecessor road; therefore, none of its lanes can have a predecessor. However, within the intersection 510, each of the connecting roads 512-520 has both a direct successor road and a direct predecessor road; for example, the fifth road 512 has the first road 502 as its direct successor road and the third road 506 as its direct predecessor road. Therefore, the lanes of the connecting roads 512-520 can connect to the lanes of their predecessor and successor roads. For example, the fifth road 512 typically contains two positive-numbered side lanes 1 and 2. The side lanes of road 5 are not shown in Figure 5 but are shown in Figure 6 The lane elements describing lane 2 and lane 1 of road 5 will in turn contain link elements indicating that the lane numbers "3" and "2" are their respective predecessors (implying that these are the lanes of road 5's successor, i.e., road 1), and that the lane numbers "2" and "1" are their respective successors (implying road 5's successor, i.e., road 3). <connection>The element also includes any lane.

[0114] Figure 6 It shows Figure 5 This is part of the road network 500, annotated with portions of OpenSCENARIO code that define certain elements of the road network.

[0115] Starting from any lane of any of income roads 502, 506, or 506, within the intersection <connection>The elements describe all possible routes through the intersection 510 (at the road level). For a given lane on a given road, obtaining a list of all routes through the intersection will mean locating all roads that will identify the road of interest as an incoming road through its road ID <connection>elements, and <connection>The elements also have lanes linked to the relevant lane. In order to obtain further lane links, it is necessary to locate its connecting road <road>elements to identify the outgoing road (for the fifth road 512, the outgoing road will be the third road 506) and to determine the lane links between the connecting road and the outgoing road.

[0116] The first code portion 602 contains a connection element (Connection ID = 0) that indicates that the first road 502 (road with ID 1) is the incoming road and that the fifth road 512 (road with ID 5) is the connecting road. The first and second <lane links> (Lane Link ID = 1 and 2) indicate that the lane links between the first road 502 and the fifth road 512 are the first lane 504 and the second lane 508, respectively. <lanelink>) elements connect lane 3 of road 1 (which enters road) to lane 2 of road 5, and connect lane 2 of road 1 to lane 1 of road 5. The second part 604 of the code contains the description of road 5 within intersection 510 <road>Element (Intersection ID = 0). The road element in <link> Element contains <successor>Elements and <predecessor>both elements. The road link in the predecessor element indicates that road 1 is the predecessor of road 5, which reflects the corresponding <connection>Lane link in element (connection 0). The successor element indicates the third road 506 (road with ID 3) as successor of road 5. Within the road element of road 5 it is also shown that <lane>element, the <lane>The element describes the lane 2 of road 5 (other lane elements are omitted for brevity); the lane element contains link elements that in turn indicate that lane 2 is successor to and predecessor of lane 3. To meaningfully interpret the lane links, it is necessary to consider the road link information; road 1 is predecessor of road 5, so lane 3 of road 1 is predecessor of lane 2 of road 5; road 3 is successor of road 5, so lane 2 of road 3 is successor of lane 2 of road 5. The third code portion 606 contains a road element describing road 3. The intersection 510 (intersection ID = 0) is indicated as predecessor of road 3, and a lane element describing lane 2 of road 3 is shown (other lane elements are omitted for brevity).

[0117] Figure 7 A portion of a directed side-lane graph 700 representing the road topology of the road network 500 is shown. For brevity, the directed side-lane graph 700 is referred to as lane graph 700. The lane graph 700 describes the lane interconnections from road 1 to intersection 510 and can be derived from the underlying OpenDrive document taking into account the considerations described above. For example, a first node 702 and a second node 704 correspond to lane 1 of lane segment 1 of road 1 and lane 2 of lane segment 2 of road 1, respectively. A forward edge 706 indicates the topological relationship between those lanes, i.e. the latter is a “forward” lane of the former (i.e. a vehicle can drive from the former to the latter without having to perform a lane change maneuver). The directed edge from node 702 to node 704 has an edge type (in this case “forward”) describing the connection between the respective lanes.

[0118] The lane graph 700 can be constructed by interpreting the code of the static layer 201a. Note that the edges of the lane graph represent the driving direction. To determine the forward edges, it is necessary to consider the lane links, but also the driving direction.

[0119] The edges are also used to provide left and right relationships. For example, a third node 708 and a fourth node 710 are depicted, representing lane 3 and lane 1 of lane segment 2 of road 1, respectively. A right edge 711 from the second node 704 to the fourth node 710 represents the possibility to move from lane 2 to lane 1 of this lane segment by a right lane change maneuver. A left edge 709 from the second node 704 to the third node 708 represents the possibility to move from lane 2 to lane 3 by a left lane change maneuver. The left edge 709 and the right edge 711 have respective types (“left” and “right”), again describing the nature of the lane connection. This information is not provided in the underlying description of the road network, but can be derived from the structure of the road. <link> The elements are provided in the element, but can be derived from the structure of the road.

[0120] The fifth node 714 represents lane 1 in the only road segment of road 5, which is the forward lane of lane 2 of road segment 2 of road 1. Thus, the forward edge 712 points from the second node 704, which represents the predecessor, to the fifth node 714, which represents the successor.

[0121] The dynamic layer 201b can be defined and executed for the lane graph 700. For example, a particular agent can be associated with a particular node at certain times (such as the beginning of the scenario, or a defined "key frame" or synchronization point, etc.). In the simulation, the topological description of the scenario dynamics is translated into a concrete geometric realization. Thus, changes can be made to the road network 500, and if these changes do not change the relevant road topology of the lane graph 700 (typically some subgraph or subgraphs in the lane graph 700), the same dynamic layer 201b can be run directly on the modified road network without modification.

[0122] The dynamic layer 201b can also contain geometric information such as the relative position of an agent at a particular point in time (e.g., at the beginning of the scenario, or at some "synchronization" point during the scenario).

[0123] Constraint-based scenario design

[0124] Figure 8 A highly schematic block diagram of a scenario design system 800 is shown, as well as a series of illustrative processing steps performed in the scenario design system 800.

[0125] The scenario design system allows for easy generation of multiple test scenarios, referred to as scenario "frameworks" (or constraint schemes), from a custom set of geometric elements 802 and constraints 804 imposed on the geometric elements 802. The constraints 804 can include individual geometric constraints on individual geometric elements and inter-element (e.g., pairwise) constraints between subsets of geometric elements (e.g., pairs of elements). The geometric elements 802 are controlled by geometric variables X, which are configurable in the sense that they can be assigned different values, also referred to as framework variables or geometric shapes.

[0126] The geometric elements 802 can be selected from a predefined set of geometric primitives 808, such as straight lines, points, arcs, spirals, polynomials, etc., which can be configured by geometric variables such as position, angle, radius of curvature, polynomial coefficients, etc. The geometric elements 802 selected for the framework 801 are used to represent scenario elements such as reference lines, lane boundaries, road markings, buildings, etc. static layer elements, and / or dynamic agents such as vehicles, pedestrians, cyclists, animals, etc.

[0127] The frame variables Y are at least partly defined by the selected geometric elements 802. Their values, indicated by reference 812, can be modified according to a processing flow described below. The frame variables 802 are independent at this stage, and they can initially be assigned any values. However, only certain combinations of values will satisfy the constraints 804 (meaning that they are treated as dependent variables at the solving stage - see below). This is achieved using a geometric constraint solver 810.

[0128] When creating the frame 801, a set of initial values is assigned to the variables Y that do not need to satisfy the constraints 804 (step 1).

[0129] As indicated above, at step 1, the scenario designer additionally selects a set of dimensions (indicated as X), which are the quantities that are treated as independent variables by the constraint solver 810. The geometric variables Y are treated as dependent variables by the constraint solver 810. In selecting the dimensions X, the designer hands over control of those properties of the scenario to the end user.

[0130] Note that the dimensions X are not necessarily directly related to the variables that support the description of the unknown geometric shape, such as point x (point.x) and point y (point.y) (x and y coordinates of "point", respectively). Rather, the dimensions can be associated with constraints that require values, such as distances, angles, lengths (e.g. measured along a curve), offsets (e.g. measured from a curve), etc. In this case, the dimensions correspond to the parameters that typically appear on the right-hand side of a mathematical equation that represents a constraint scheme, such as (p.x - q.x)A2+ (p.y - q.y)A2+ (p.z - q.z)A2= dA2. This equation represents a constraint on the distance between p and q (defined on the left-hand side), which is characterised by the distance dimension d (note that at this point d is not assigned a specific value; rather, this is the mechanism by which d is introduced as an independent variable). Here, p and q are geometric variables (dependent variables), and d is a constraint parameter (independent variable). It is also possible to constrain variables directly, such as by specifying p.x = a (fixing the x coordinate of p to the value a) or a < p.x < b (fixing the x coordinate within a range of values). The constraint solver 810 will then attempt to solve for the dependent variables (e.g. p, q) given the independent variables (e.g. d, a, b).

[0131] Any of the dimensions X of the constraint scheme are independent variables, and are not subject to constraints of the scheme itself.

[0132] The frame 801 is then provided as a first input, along with a selection of the dimensions X and an indication of the initial values of the dimensions X and the geometric shape Y, to the geometric constraint solver 810 (step 2a).

[0133] As discussed, the constraint solver 810 is primarily analytical. Using a predetermined set of deduction rules, it attempts to express the dependent variables in terms of the independent variables in a set of functions determining (step 2b) a solution (and thus being "reusable") that resolves the functions in terms of (known) dimensions representing (unknown) geometry to satisfy the constraints. In this case, the solution is represented by Y = F(X), where the solver aims to compute F (the mapping between dimensions and unknowns) as a set of mathematical expressions. Those expressions can be embodied in memory as a computational graph 830 (see below). Figure 8

[0134] In this example, the designer specifies initial values for dimensions X and unknowns Y in step 1, but those initial values do not necessarily satisfy the constraints 804.

[0135] Once a solution is found in step 2b, it can be used in step 3 to assign new values to Y that do satisfy the constraints 804, along with the initial values of X. In step 3, new values are assigned to each unknown by evaluating the initial values of X.

[0136] Note that the solution Y = F(X) determined as step 2b is not necessarily unique; rather, the constraint problem can have multiple solutions (i.e., multiple mappings F between dimensions X and unknowns Y that satisfy the constraints 804).

[0137] If the constraint problem has two (or more) "point-like" solutions, each of those solutions is well-defined, and the problem is said to have two (or more) well-defined solution branches.

[0138] If there are some (non-zero) unconstrained degrees of freedom remaining after satisfying all constraints and dimensions, the constraint problem is said to be "under-defined," which means that the solution is spatial in nature, i.e., there is a continuum of solutions (one or more solution branches) on a single solution branch (under-defined problems can have one or more solution branches). In some cases, under-defined problems can trigger an error output. In other cases, the tool can handle under-defined problems by automatically selecting a solution (in which case it can or can not flag that other possible solutions exist).

[0139] Figure 10 A simple example of a well-defined constraint problem with two solution branches is depicted, where a point has a position (an unknown in Y) that is constrained to be a distance (constant) from two other points at positions (dimensions in X). The solution to this constraint problem typically has exactly two solutions, respectively and with and .​ the center and radius of and are defined. As will be appreciated, this is a simple example chosen for illustration (e.g., in more complex examples, the distances themselves can be variables, or the positions of other points can be unknown).

[0140] In some cases, the solver 810 can be adapted to infer the designer’s intent, e.g., from the initial selection of size ) values and / or some values. According to the example of Figure 10 , assume that the designer has selected initial values for the unknown quantities and size and . Given those values, the first solution returns values that are closer to the designer’s initial values than the second solution . Thus, at step 2b, the solver 810 can automatically select the first solution . In some implementations, this is flagged to the designer, e.g., by a notification located at the GUI 816.

[0141] Note that this solution is independent of the initial selection of values and will be valid for other values. Also, there is no requirement that the initial values satisfy the constraints. The initial selection of values or values simply guides the solver on how to solve an under-defined problem or a well-defined problem with multiple solution branches (e.g., favoring solutions that are valid for the initial selection of size values or solutions that are closer to the initial selection of geometry values .

[0142] As another example, a given problem can have multiple solutions, but only one (or some) of those solutions can be valid for the designer’s initial selection of X values. In that case, the solver can select the solution that is valid for the designer’s initial selection of X values.

[0143] At the other extreme, the geometry may be completely unconstrained, which typically indicates an error in the design phase (either in the selection of constraints or in the selection of sizes .

[0144] In more detail, the constraint solver 810 analyzes the scenario in a parsing phase to produce a mapping F in the form of a computed directed acyclic graph (DAG) 830 that is feasible for any set of dimension values within the non-empty range of initial dimension (X) values. The parsing phase is followed by a computation phase in which the computed DAG 830 is evaluated for a given X value selection. If the dimension (X) value is subsequently modified and the solution is still feasible for the modified X value, only the computation part of the solving process needs to be executed afterwards; the computed DAG 830 itself does not change.

[0145] Sometimes, the solver 810 can not be able to derive a solution in closed form. In this case, it can resort to numerical methods to try to find Y values that satisfy the constraints for a given X value selection. This is typically much less efficient, as unlike a solution in closed form, a numerical solver has to be re-run every time the dimension values change.

[0146] The constraints 804 provide a definition of "significance" that is hard-coded into the scenario in the design phase. All scenarios generated based on the scenario framework 801 are guaranteed to be significant in the sense of satisfying the constraints 804 imposed by the design phase.

[0147] Note that the constraints 804 are expressed in an intuitive, user-friendly way, namely in terms of primitive geometric relationships between elements (such as "parallel", "coincident", "distance", etc.). The scenario designer does not need to manually derive or input mathematical expressions that define the unknowns in terms of dimensions. Instead, the constraint solver 810 automatically derives those expressions by applying a series of derivations.

[0148] The scenario framework 801 is a "lightweight" description of a scenario (or part of a scenario) that is designed around some aspect (or aspects) of the scenario that is intended to be changed in the simulation test. The goal is to define a minimal number of geometric elements 802 and constraints 804 that capture that variable aspect, while reasonably limiting the extent to which it can be modified to ensure that all variants of the scenario satisfy the designer's definition of significance. A combination of variable values 812 that satisfy the constraints 804 of the framework 801 can be called "significant".

[0149] In step 3, after assigning new values to the unknowns Y that satisfy the constraints 804, the framework 801 can be used together with the initial value of X and the computed values of Y (step 4) to generate a complete scenario description 201 in an appropriate SDL (such as OpenDRIVE or OpenSCENARIO, or some variant thereof). To this end, the framework 801 and the updated values 812 are provided as input to the scenario generator 146.

[0150] The primitives used in the constraint solver 810 can or can not correspond to the primitives used in the underlying SDL (e.g. OpenDRIVE). Generally speaking, the constraint solver provides a wide range of geometric primitives, e.g. from simple points, lines and circles, to non-uniform rational basis splines (NURBS) and / or custom parametric curves. Providing the primitives used in the constraint solver primitives back into the set used by the desired SDL (within acceptable tolerances), makes step 4 feasible.

[0151] In this example, the complete scenario description has a static layer 201a and a dynamic layer 201b, the static layer 201a being encoded in a specification (document) conforming to the OpenDRIVE schema or some variant of OpenDRIVE (or other structured scenario description format), the dynamic layer 201b being encoded using OpenSCENARIO or some variant thereof. As mentioned previously, different formats / schemas can be used as an alternative to OpenDRIVE / OpenSCENARIO.

[0152] The static layer 201a is a complete detailed description of the road network, e.g. at the level of detail of Figure 5 or Figure 6 or higher.

[0153] To produce further variants, one or more of the values of the dimension X can subsequently be modified in step 5. In step 5, the X values can be modified arbitrarily (as an independent degree of freedom). Assuming the previously determined computation graph is still feasible, the existing computation DAG is used in step 6 to determine updated Y values (this is simply a case of re-evaluating the existing DAG according to the new X values).

[0154] The result of steps 5 and 6 is a new set of X and Y values that satisfy the constraints 804 of the framework 801.

[0155] The new variables obtained in steps 5 and 6 can in turn be used by the scenario generator 146 (step 7) to generate a second scenario from the framework (in SDL), which is also guaranteed to be significant in the above sense.

[0156] Steps 5-7 can be repeated as many times as desired to generate any desired number of scenarios from the framework 801, all of which satisfy the constraints 804. Continuing with the road curvature angle, continuous (or incremental) changes can be made to the road curvature angle and the values of the remaining variables recalculated each time the existing computation DAG is modified with the new road curvature angle.

[0157] The size (X) value can also be changed, which means that the earlier solution (computation graph) is no longer feasible. However, there can be a different solution that is feasible for the new size value. In this case, the constraint solver 810 can attempt to find a new solution (at a second resolution phase) in the form of a second computation DAG (not shown) Figure 8 that is feasible for the new X value.

[0158] Geometric constraint solvers have been developed in other technical fields, such as early-stage "sketch" tools in computer-aided design (CAD). Existing solvers can be used in the present context. One example of a suitable geometric constraint solver is the Siemens (R) licensed D-Cubed 2D Dimensional Constraint Manager (2D DCM). This is just one example, and various existing solver tools can be deployed in the present context. These tools can take a problem definition (dimensions and constraints) and initialization, and return a solution in the form of a reusable computation graph.

[0159] One class of solvers receives a first input that encodes the geometric elements 802 and geometric constraints 804 in the form of a constraint graph. Certain nodes of the graph represent geometric elements, other nodes represent constraints, and edges between nodes define how the constraints apply to the geometric elements (see Figure 9 and accompanying description for more details). Such solvers are typically based on a (possibly large) set of deduction rules that allow individual subgraphs within the constraint graph 804 to be solved. For example, specific deduction rules can be provided for the case where a point is defined in terms of a distance to two other points (see Figure 10 ). As solvable subgraphs are identified in the constraint graph, a computation graph is built until the entire constraint graph has been solved for the selected dimensions, or until the solver determines that the problem is under-defined.

[0160] To facilitate steps 1-7, a framework generator 814 in the form of an application programming interface (framework API) is provided. The framework API 814 is shown as including a framework builder 814a, a framework binder 814b, and a framework modifier 814c. Each of these components 814a, 814b, 814c is implemented as a set of one or more API functions that can be called by another system component built on top of the framework API 814, such as a graphical user interface (design GUI) 816.

[0161] The design GUI 816 provides a set of graphical tools that can be used to select geometric elements 802 from the geometric primitives 808, and define constraints 804 on the selected elements 802.

[0162] The frame binder 814b is responsible for storing the associations (bindings) between the geometric elements 802 of the frame 810 and corresponding elements of the scene description 201 (in the static layer 201a and / or the dynamic layer 201b).

[0163] In one use case, the frame 801 is initially generated based on selected regions in an existing scene description. The existing scene description is saved in the scene database 818. As each geometric element 802 is added to the frame 801, the association between this element and the corresponding element (or elements) of the existing scene description is recorded. As elements 802 are added and bindings created, the initial values of the geometric variables 804 can also be derived from the existing scene description. Then, when values are subsequently modified in step 2 / 6, the bindings between the frame 801 and the scene description 201 can be used to modify the corresponding elements of the existing scene description accordingly, in a way that always guarantees that the constraints 801 are met. Here, the aim is to construct the frame 801 with the minimum number of geometric elements needed to define the aspect (or aspects) to be changed, and to constrain the control in a way that maintains significance.

[0164] Alternatively, a new scene description can be generated "from scratch" based on the frame 801 and initial variable values, in which case the bindings can be created from scratch when generating the scene description 201. Thereafter, the bindings can be used to more easily implement changes in variable values.

[0165] The frame 801 can also be initially created from an existing scene description, in which case the geometric elements 802 can be bound to the existing scene description once the frame 801 has been created.

[0166] The system is very flexible, allowing users to add any elements they choose to the frame 801 and constrain them according to their specific testing needs.

[0167] One application of the scene design system 800 is to generate a static layer 201a in which the geometric elements 802 correspond to road elements. In this case, the tool can be used to generate or modify a map contained in the static layer 201a according to the constraints of the designer.

[0168] The full scene description 201 is typically much more detailed than the skeleton 801. For example, to impose and constrain a certain degree of control over the road angle, it can be sufficient to include in the skeleton only the applicable road reference line (to fully define and constrain the road curvature relative to the road reference line). In the full scene description 801, there can be many additional elements defined relative to the road reference line, such as drivable / non-drivable side lines, and thus not required to be included in the skeleton 801 to exhibit the required control. These elements can be omitted from the skeleton 801, as changes to the road reference line will automatically propagate to all other elements of the static layer 201a defined relative to the road reference line.

[0169] Figure 9 It is shown how to define a curved road reference line from a first straight line LI and a second straight line L2, an arc Al, and six points PI,... P6 used to define the straight and arc segments. A single dimension a is considered in this example. This will involve geometric elements, constraints, and variables such as: a. Constrain PI and P2 to coincide with the first line LI (the first line segment defined by PI, P2, and L2 can in turn be tied to the corresponding first straight line segment of the road reference line in the full static layer 201a); b. Constrain P3 and P4 to coincide with the second line L2 (similarly define a second line segment, possibly at a different angle); c. Define the dimension a in X as the angle between LI and L2 (the road curvature angle, the property that the user wishes to manipulate in this example); d. Constrain P5 and P6 to coincide with the arc Al, thereby defining a curve segment (which in turn can be tied to the corresponding curve segment of the road reference line in the full static layer 201a); e. Constrain P2 to coincide with P5 and P3 to coincide with P6 (guarantee a continuous road reference line between PI and P4); f. Constrain the slope of Al at P5 to match the slope of LI and the slope of Al at P6 to match the slope of LI (guarantee a continuous slope road reference line between PI and P4); g. Set distance constraints between PI and P2 and between P3 and P4, respectively, to limit the size of the straight segments, etc.

[0170] Additional / alternative constraints can be imposed to match the road segment described by the skeleton 201 to an existing map (e.g., fix the positions of PI and P4 so that they connect correctly to the wider road network).

[0171] A portion of the constraint graph 900 is shown, which is a graph-based representation of the frame 801. The constraint graph 900 encapsulates the geometric elements 802, the constraints 804 and their variables in a collection of nodes and edges. As shown, the graph 900 includes first nodes corresponding to the geometric elements P1, …, P6, L1, L2, A2 and second nodes defining their variables and constraints. The coincidence constraints are represented by coincidence nodes (coi), whereby the coincidence relationship between two geometric elements is encoded as a coincidence node connected to the respective element nodes in the graph. Distance and angle variables and any constraints attached to these variables can be similarly represented with additional nodes and connections in the constraint graph 900. For example, a gradient node (Gr) is shown connecting the nodes representing L1, P5 and A1 to define the constraint that the gradient of A1 at P5 must equal the gradient of L1. A distance node (Dist) is shown connecting the P1 and P2 nodes to define a distance variable between those points P1, P2, which can be constrained as desired.

[0172] The constraint graph 900 is provided to the solver 810 together with an indication that a is an independent dimension and a set of initial values 902 for the variables to guide the solver 810. The solver 810 constructs a computation graph 930 for computing the dependent variables in a manner that satisfies the constraint graph 900, which in turn can be provided to the scene generator 146 to generate a complete map description that satisfies the constraints. This is just the case where the computation graph 930 is evaluated for a selected value of the road curvature angle a to compute the dependent variables and use the established bindings.

[0173] The curvature angle a is defined by an angular node (Ang) of the constraint graph 900 connecting the nodes representing L1 and L2. Note that defining the variable a as an independent dimension is a choice of the scene designer. The design tool is flexible enough to accommodate any dimension definition the designer wants.

[0174] The same constraint graph 900 can be used but with a different choice of dimensions. For example, a can become a dependent variable, while some other variable, such as the curvature of A1, can be chosen as an independent dimension. This would be a different constraint problem to solve for the new dependent variable in the same way.

[0175] While Figure 9 Only static layer frames have been considered, but the same techniques can also be used to configure the dynamic layer 201b. In this case, the agents at a given point in time can be treated as static elements whose geometry at that point in time can be constrained in exactly the same way. Then, by repeatedly solving the constraints 804 in the problem described above, different variants of the dynamic layer 201 can be realized. One example is a set of parked vehicles (corresponding to certain geometric elements of the frame 801) whose starting positions are constrained in the frame 801.

[0176] For static agents, the positions are always fixed throughout the scenario. However, the same technique can be used to determine the starting positions of dynamic agents at the beginning of the scenario or at specified "keyframes" within the scenario. The latter means that the agent positions are determined at some point in the scenario. With open-loop simulation (non-self agents do not react to self agents), this is easily achieved by interpolating the relevant agent variables (position, heading, velocity, acceleration, etc.) back to the beginning of the scenario (or an earlier keyframe).

[0177] Constrained keyframes can be constructed from the skeleton and subject to a given set of constraints on the agent properties (e.g., position, heading, velocity, acceleration, etc.) provided to the constraint solver 810. This addresses the problem of dynamic significance to some extent; for example, a constrained keyframe can be constructed to ensure that two non-self agents interact in some specified way throughout all variants of the scenario.

[0178] With closed-loop simulation (non-self agents can react to self agents), the agent positions determined by the geometric constraint solver 810 at a given keyframe can be used as target positions / states for the planning of non-self agents (in this case, there is no strong guarantee of significance, as the non-self agents can not be able to achieve the target, which can depend on the behavior of the self agents themselves. However, it does provide a weaker guarantee that the non-self agents plan their actions in a significant way).

[0179] The dynamic layer 201b is preferably defined at the road topology level, which generally means associating agents with specific lanes or other road elements, the interconnections of which are later encoded in the static layer 201a. For example, in one implementation, the dynamic layer 201b is defined on top of a lane graph 700 of the static layer 201a, as shown in Figure 7 In this case, agents and behaviors can be associated with individual nodes of the lane graph 700 to define the scenario dynamics at the road topology level. The dynamic layer 201b can be implemented directly on different variants of the static layer 201a in the case that modifications of the scenario do not change the lane graph 700 (or a subset of the lane graph 700 that defines the agent dynamics). "Agent topology" refers to the definition of agent dynamics at the road topology level.

[0180] In this case, the dynamic layer 201b can include a series of geometric shapes. For example, the starting positions of a group of agents can be defined according to their starting lanes. Variants of the dynamic layer 210b can then be implemented with different agent starting positions within their respective lanes, or with different shapes / sizes of agents (or, more generally, different agent geometries within the same agent topology).

[0181] By pairing different dynamic layers with the same static layer 201a, different scenarios can be created. This essentially means placing different agents and behaviors on the same road layout.

[0182] Furthermore, two dynamic layers defined on the same road topology (or a subset of the road topology) are easily interchangeable with a static layer having that topology. In the current context, this means that the static layer 201a can be modified to adjust quantities such as road curvature, lane width, intersection angles, etc., and within the scope of the modification not changing the road topology (or a subset of the road topology defining the dynamic layers 201b), the same dynamic layers 201a can be executed on the modified static layer.

[0183] References herein to components, functions, modules, etc. represent functional components of a computer system that can be implemented in various ways at the hardware level. The computer system includes execution hardware that can be configured to perform the method / algorithm steps disclosed herein and / or configured to implement models trained using the present technology. The term execution hardware includes any form / combination of hardware configured to perform the relevant method / algorithm steps. The execution hardware can take the form of one or more processors, which can be programmable or non-programmable, or can use a combination of programmable and non-programmable hardware. Examples of suitable programmable processors include general-purpose processors based on instruction set architectures, such as central processing units (CPUs), graphics processing units (GPUs) / accelerator processors, etc. Such general-purpose processors typically execute computer-readable instructions held in a memory coupled with or internal to the processor, and perform the relevant steps in accordance with those instructions. Other forms of programmable processors include field programmable gate arrays (FPGAs), which have circuit configurations that can be programmed through circuit description code. Examples of non-programmable processors include application specific integrated circuits (ASICs). The code, instructions, etc. can be stored on a transitory or non-transitory medium (examples of the latter include solid state, magnetic, and optical storage devices, etc.) as appropriate. Figure 1A The subsystems 102-subsystem 108 can be implemented in programmable or dedicated processors in a vehicle or non-vehicle computer system in the context of testing, etc., or in a combination of both. Figure 2 Figure 4 and Figure 5 Various components of the system 100, such as the simulator 202 and the test oracle 252, can similarly be implemented in programmable and / or dedicated hardware.​< / lane> < / lane> < / connection> < / predecessor> < / successor> < / road> < / lanelink> < / road> < / connection> < / connection> < / connection> < / connection> < / lane> < / connection> < / connection> < / connection> < / connection> < / connection> < / successor> < / predecessor> < / road> < / road> < / junction> < / connection> ​< / road> < / predecessor> < / road> < / successor> < / road> < / lane> < / lanesection> < / lanesection> < / lanes> < / lanes> < / road> < / predecessor> < / successor> < / lane> < / road> < / junction> < / road>

Claims

1. A computer-implemented method for generating static and / or dynamic layers for a test scenario used to perform performance testing on a robot system in a simulation environment, the method comprising: A scene framework defining geometric elements and geometric constraints on those geometric elements is input to the geometric constraint solver. The geometric elements can be modified based on a set of geometric variables, and the geometric constraints are characterized by one or more constraint parameters. This allows the geometric constraint solver to determine one or more functions of the constraint parameters to return the values ​​of the geometric variables that satisfy the geometric constraints. Receive the value of the constraint parameter; The values ​​of the geometric variables that satisfy the geometric constraints are calculated using the received constraint parameters and the one or more functions. as well as Based on the received constraint parameter values, the calculated values ​​of the geometric variables, and the scene framework, configure the static layer and / or the dynamic layer of the test scene.

2. The method of claim 1, further comprising receiving initial values ​​for the geometric variables, wherein, The received values ​​of constraint parameters and the received initial values ​​of geometric variables do not satisfy the geometric constraints, wherein the received initial values ​​of geometric variables are provided to the geometric constraint solver to determine a function of the constraint parameters.

3. The method according to claim 1 or 2, wherein, The geometric constraints are represented in the form of a constraint graph, wherein the geometric solver uses multiple predetermined derivation rules to extract the function from the constraint graph.

4. The method according to claim 1, 2 or 3, wherein, The geometric elements include road elements, and the geometric constraints include constraints on the road elements, wherein the received and calculated values ​​are used to generate the static layer, the static layer including a road layout that satisfies the geometric constraints.

5. The method according to any one of the preceding claims, wherein, The scene framework is generated based on scene creation input received at the graphical user interface.

6. The method according to any one of the preceding claims, wherein, The scene frame is generated based on a selected area of ​​the map, wherein at least some of the geometric elements correspond to corresponding map elements within the selected area of ​​the map.

7. The method of claim 6, further comprising storing associations between at least some geometric elements and corresponding map elements, wherein, The static layer is generated from the map based on the values ​​of the geometric variables and the stored associations.

8. The method according to any one of the preceding claims, wherein, The geometric constraints include inter-element geometric constraints between corresponding subsets of the geometric elements.

9. The method according to any one of the preceding claims, comprising: Execute a test scenario comprising at least one of the static layer and the dynamic layer, wherein the self-agent of the test scenario is controlled by the robot planner being tested; and The test oracle generates a set of test results to evaluate the performance of the robot planner in the test scenario.

10. The method according to claim 9, wherein, The robot planner is tested in conjunction with one or more other components.

11. The method according to claim 10, wherein, The one or more other components include one or more of a sensing system, a prediction system, and a controller.

12. The method according to any one of claims 9 to 11, comprising using the test result set to identify and mitigate problems in the robot planner or other components tested in conjunction with the robot planner.

13. The method according to any one of the preceding claims, comprising: Receive the second value of the constraint parameter independent variable; The second value of the geometric variable that satisfies the geometric constraint is calculated using the second value of the received constraint parameter and the one or more functions; as well as The second static layer and / or the second dynamic layer of the test scenario are configured based on the second value of the received constraint parameters, the second value of the calculated geometric variables, and the scenario framework.

14. A computer system, comprising: One or more computers configured to implement the method of any of the preceding claims.

15. Computer-readable instructions stored in a transient or non-transient medium, which, when executed in a computer system, are configured to implement the method of any one of claims 1 to 13.

Images