Performance Testing of a Trajectory Planner for a Mobile Robot

Abstract

A computer-implemented method for evaluating performance of a trajectory planner for a mobile robot in a real or simulated scenario includes receiving a scenario ground truth for the scenario, the scenario ground truth being generated using the trajectory planner to control an ego-agent of the scenario in response to at least one scenario element of the scenario. One or more performance evaluation rules for the scenario and at least one activation condition for each performance evaluation rule are received. A test oracle processes the scenario ground truth to determine whether an activation condition for each performance evaluation rule is satisfied across multiple time steps of the scenario. Each performance evaluation rule is evaluated by the test oracle to provide at least one test result only if the activation condition is satisfied.

Description

[Technical field]

[0001] The present disclosure relates to a method for evaluating the performance of a trajectory planner in real or simulated scenarios, as well as computer programs and systems for implementing the same. Such a planner is capable of autonomously planning the self-trajectory of a fully / semi-autonomous vehicle or other form of mobile robot. Application examples include ADS (Autonomous Driving System) and ADAS (Advanced Driver Assist System) performance testing. [Background technology]

[0002] There has been great and rapid development in the field of autonomous vehicles. An autonomous vehicle (AV) is a vehicle equipped with sensors and control systems that enable it to operate without human control of its behavior. An autonomous vehicle is equipped with sensors that enable it to perceive its physical environment, such sensors include, for example, cameras, radar, and lidar. An autonomous vehicle is equipped with a suitably programmed computer that is capable of processing data received from the sensors and making safe and predictable decisions based on the context perceived by the sensors. An autonomous vehicle may be fully autonomous (in that it is designed to operate without human supervision or intervention, at least in certain circumstances) or semi-autonomous. Semi-autonomous systems require various levels of human supervision and intervention, such systems include advanced driver assistance systems and level 3 autonomous driving systems. There are various aspects to testing the behavior of sensors and control systems on board a particular autonomous vehicle or type of autonomous vehicle.

[0003] A "Level 5" vehicle is one that can operate fully autonomously in any situation while always being guaranteed to meet a minimum safety level. Such a vehicle does not require any manual controls (steering wheel, pedals, etc.).

[0004] In contrast, Level 3 and Level 4 vehicles can operate fully autonomously, but only within certain defined circumstances (e.g., within a geofenced area). Level 3 vehicles must be equipped to autonomously handle any situation that requires an immediate response (e.g., emergency braking), but a change in circumstances may trigger a "transition request" that requires the driver to take control of the vehicle within a limited time frame. Level 4 vehicles have similar limitations, but if the driver fails to respond within the required time frame, Level 4 vehicles must also be capable of autonomously performing a "minimum risk maneuver" (MRM), i.e., taking appropriate measures to bring the vehicle to a safe state (e.g., slowing down and stopping the vehicle). Level 2 vehicles require the driver to be ready to intervene at any time, and it is the driver's responsibility to intervene at any time if the autonomous system is no longer able to respond appropriately. At level 2 automation, it is the driver's responsibility to decide when their intervention is required, at levels 3 and 4 this responsibility passes to the vehicle's autonomous system and it is the vehicle that must alert the driver when intervention is required.

[0005] As the level of autonomy increases and more responsibilities are transferred from humans to machines, safety becomes an increasingly challenging challenge. In autonomous driving, the importance of guaranteed safety is recognized. Guaranteed safety does not necessarily imply zero accidents, but rather guarantees that a minimum level of safety is met in defined circumstances. It is generally believed that for autonomous driving to be feasible, this minimum level of safety must significantly exceed that of a human driver.

[0006] According to Shalev-Shwartz et al., "On a Formal Model of Safe and Scalable Self-driving Cars" (2017), arXiv:1708.06374 (RSS paper), which is incorporated herein by reference in its entirety, human drivers average 100 vehicles per hour. -6 Based on the assumption that autonomous driving systems would need to reduce this by at least three orders of magnitude, the RSS paper estimates that -9 They conclude that a minimum safety level of the order of 1000 accidents needs to be guaranteed, and therefore point out that a purely data-driven approach would require vast amounts of driving data to be collected every time a change is made to the AV system's software or hardware.

[0007] The RSS paper provides a model-based approach to assured safety. A rule-based Responsibility-Sensitive Safety (RSS) model is built by formalizing a small number of "common sense" driving rules: "1. Don't hit someone from behind. 2. Do not interrupt without reason. 3. The right of way is given, not taken. 4. Be careful in areas with poor visibility. 5. If you can avoid an accident without causing another one, you should do so.”

[0008] The RSS model is first proven to be safe, in the sense that no accidents will occur if all agents always follow the rules of the RSS model. The aim is to reduce by several orders of magnitude the amount of driving data that needs to be collected to demonstrate a desired level of safety.

[0009] A safety model (e.g., RSS) can be used as a basis for evaluating the quality of trajectories achieved by an ego-agent in real or simulated scenarios under the control of an autonomous system (the stack). The stack is tested by exposing it to various scenarios and evaluating the resulting ego-trajectories for compliance with the rules of the safety model (rule-based testing). The rule-based testing approach can also be applied to other aspects of performance, such as comfort or progress towards a defined goal. Summary of the Invention [Means for solving the problem]

[0010] According to a first aspect of the present specification, there is provided a computer-implemented method for evaluating performance of a trajectory planner of a mobile robot in a real or simulated scenario, the computer-implemented method comprising: receiving a scenario ground truth for the scenario, the scenario ground truth being generated using the trajectory planner to control an ego-agent of the scenario in response to at least one scenario element of the scenario; receiving one or more performance evaluation rules for the scenario and at least one activation condition for each performance evaluation rule; and processing the scenario ground truth by a test oracle to determine whether the activation condition for each performance evaluation rule is satisfied across multiple time steps of the scenario. Each performance evaluation rule is evaluated by the test oracle to provide at least one test result only if its activation condition is satisfied.

[0011] In the context of pass / fail rules, this provides a third "not applicable" that a rule can evaluate at a given time step. Specifically, when evaluating large amounts of scenario data (typically generated in simulation or a combination of simulations in testing), evaluating a potentially complex rule across many time steps and many scenarios can require very large amounts of computational resources. By "deactivating" a rule based on simpler activation conditions (which are less expensive to evaluate than the rule itself), significant resource savings can be realized in a way that does not harm the final result. This may actually improve the quality of the results, since a "not applicable" (inactive) result is often more informative, since it distinguishes between situations where the rule is applicable and therefore passes / fails, and situations where the rule was never applicable in the first place. For example, in an intersection scenario, rules may be defined that relate to various distance thresholds to multiple other agents on the road that the agent wishes to merge into, and are activated only when the agent crosses the road boundary. If this rule were instead always active, not only would this be costly to evaluate when the agent is waiting at an intersection, but the results during that period would be less informative (e.g., compared to a situation where no distinction is made between "pass" and "no-pass").

[0012] In an embodiment, the scenario ground truth may be processed to determine whether an activation condition of each performance evaluation rule is satisfied for each scenario element of the set of multiple scenario elements across multiple time steps of the scenario. Each performance evaluation rule may be evaluated only if its activation condition is satisfied for at least one of the scenario elements and only between the own agent and the scenario elements for which the activation condition is satisfied.

[0013] In an embodiment, each performance evaluation rule may be coded as a second logical predicate in a portion of the rule creation code, and its activation condition is coded therein as a first logical predicate, and at each time step, the test oracle evaluates the first logical predicate for each scenario element, and evaluates the second logical predicate only between the agent and any scenario elements that satisfy the first logical predicate.

[0014] A plurality of performance evaluation rules having different respective activation conditions may be received and selectively evaluated by the test oracle according to their different respective activation conditions.

[0015] Each performance evaluation rule may relate to driving performance.

[0016] The method may include rendering on a graphical user interface (GUI) results for each of a plurality of time steps in the time series, where the results for each time step visually indicate one of at least three categories, including a first category for when an activation condition is not met, a second category for when an activation condition is met and a rule is passed, and a third category for when an activation condition is met and a rule is failed.

[0017] For example, the results may be rendered as one of at least three different colors corresponding to at least three categories.

[0018] An activation condition of a first one of the performance evaluation rules may depend on an activation condition of at least a second one of the performance evaluation rules.

[0019] For example, if a second performance evaluation rule (eg, relating to safety) is active, a first performance evaluation rule (eg, relating to comfort) may be deactivated.

[0020] A scenario element may comprise one or more other agents.

[0021] At least one of the performance evaluation rules may be selectively evaluated pairwise between the own agent and one scenario element of a set of scenario elements in the scenario, and its activation condition may be evaluated independently for each scenario element to determine whether the performance evaluation rule should be evaluated between the own agent and other agents at each time step.

[0022] The set of scenario elements may be a set of other agents.

[0023] The activation conditions may be evaluated for each scenario element to compute at each time step an iterable comprising identifiers of any scenario elements for which the activation conditions are satisfied, and the performance evaluation rules may be evaluated by looping over the iterable at each time step.

[0024] Performance evaluation rules may be defined as a computational graph that is applied to one or more signals extracted from the scenario ground truth, and an iterable is passed through the computational graph to evaluate the rule between the agent itself and any scenario elements that satisfy the activation condition.

[0025] A further aspect herein provides a computer-implemented method for evaluating performance of a trajectory planner for a mobile robot in a real or simulated scenario, the method including receiving a scenario ground truth for the scenario, the scenario ground truth being generated using the trajectory planner to control an own agent of the scenario in response to one or more scenario elements of the scenario; receiving one or more performance evaluation rules for the scenario and at least one activation condition for each performance evaluation rule; and processing the scenario ground truth by a test oracle to determine whether an activation condition for each performance evaluation rule is satisfied for each scenario element across multiple time steps of the scenario, wherein each performance evaluation rule is evaluated by the test oracle only if its activation condition is satisfied for at least one of the scenario elements to provide at least one test result between the own agent and the scenario elements for which the activation condition is satisfied.

[0026] A further aspect provides a computer system comprising one or more computers configured to implement the method of the first aspect or any embodiment thereof, and executable program instructions for programming the computer system to implement the same.

[0027] For a better understanding of the present disclosure and to show how embodiments thereof may be put into effect, reference is made, by way of example only, to the following figures: [Brief description of the drawings]

[0028] [Figure 1A] FIG. 1 is a schematic functional block diagram of an autonomous vehicle stack. [Figure 1B] A schematic diagram of the autonomous vehicle testing paradigm. [Figure 1C]FIG. 1 is a schematic block diagram of a scenario extraction pipeline. [Diagram 2] A schematic block diagram of the test pipeline. [Figure 2A] FIG. 13 shows further details of a possible implementation of the test pipeline. [Figure 3A] FIG. 1 shows an example of a rule tree that is evaluated within a test oracle. [Figure 3B] FIG. 13 illustrates an example output of a node in a rule tree. [Figure 4A] FIG. 1 shows an example of a rule tree that is evaluated within a test oracle. [Figure 4B] Figure 2 shows a second example of a rule tree evaluated on a set of scenario ground truth data. [Figure 4C] FIG. 13 illustrates how rules can be selectively applied within a test oracle. [Diagram 5] FIG. 2 is a schematic block diagram of a visualization component for rendering a graphical user interface. [Figure 5A-5C] FIG. 1 illustrates the various views available within the graphical user interface. [Figure 6A] FIG. 1 illustrates a first instance of an interruption scenario. [Figure 6B] FIG. 13 illustrates an example oracle output for the first scenario instance. [Figure 6C] FIG. 1 illustrates a second instance of the interruption scenario. [Figure 6D] FIG. 13 illustrates an example oracle output for a second scenario instance. [Figure 7] FIG. 13 shows an example of rule creation code in a domain-specific language for defining rules to be applied by a test oracle. [Figure 8] FIG. 13 illustrates a further example of a GUI view for rendering the output of a custom rule tree. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] The described embodiments provide a test pipeline to facilitate rule-based testing of a mobile robot stack in real or simulated scenarios. The behavior of agents (actors) in real or simulated scenarios is evaluated by a test oracle based on defined performance evaluation rules. Such rules may evaluate various aspects of safety. For example, a safety rule set may be defined to assess the performance of the stack against a particular safety standard, regulation, or safety model (such as RSS), or a bespoke rule set may be defined to test any aspect of performance. The test pipeline is not limited to safety in its use and can be used to test any aspect of performance, such as comfort or progress towards a defined goal. A rule editor allows performance evaluation rules to be defined or modified and passed to the test oracle.

[0030] Typically, a "full" stack includes everything from processing and interpretation of lower-level sensor data (perception) to inputs to primary higher-level functions such as prediction and planning, as well as control logic to generate appropriate control signals to implement planning-level decisions (e.g., to control braking, steering, acceleration, etc.). For autonomous vehicles, a level 3 stack includes logic to implement transition requests, and a level 4 stack additionally includes logic to implement minimum-risk maneuvers. Stacks may also implement secondary control functions, e.g., signals, headlights, windscreen wipers, etc.

[0031] The term "stack" may also refer to individual subsystems (sub-stacks) of the full stack, such as the perception, prediction, planning, or control stacks, which may be tested individually or in any desired combination. A stack may refer purely to software, i.e., one or more computer programs capable of running on one or more general-purpose computer processors.

[0032] A scenario, whether real or simulated, requires an ego-agent to navigate within a real or modeled physical context. The ego-agent is a real or simulated mobile robot moving under the control of the stack under test. The physical context includes static and / or dynamic elements that the stack under test is required to respond to effectively. For example, the mobile robot may be a fully or semi-autonomous vehicle (ego-vehicle) under the control of the stack. The physical context may comprise a static road layout and a set of given environmental conditions (e.g., weather, time of day, lighting conditions, humidity, pollution / particle levels, etc.) that can be maintained or changed as the scenario progresses. An interactive scenario additionally includes one or more other agents ("external" agents, e.g., other vehicles, pedestrians, cyclists, animals, etc.).

[0033] The following example considers application to testing of autonomous vehicles, however the principles apply to other forms of mobile robots as well.

[0034] Scenarios may be expressed or defined at various levels of abstraction. More abstract scenarios accommodate a greater degree of variation. For example, a "cut-in scenario" or a "lane change scenario" are examples of highly abstracted scenarios characterized by targeted operations or behaviors that accommodate many variations (e.g., different agent starting positions and speeds, road layouts, environmental conditions, etc.). A "scenario run" refers to a concrete occurrence in which an agent progresses through a physical context, optionally in the presence of one or more other agents. For example, multiple runs of a cut-in or lane change scenario can be conducted (in the real world and / or in a simulator) with different agent parameters (e.g., starting positions, speeds, etc.), different road layouts, different environmental conditions, and / or different stack configurations. The terms "run" and "instance" are used interchangeably in this context.

[0035] In the following example, the performance of the stack is assessed, at least in part, by evaluating the own-agent's behavior within a test oracle against a given set of performance evaluation rules over the course of one or more runs. The rules are applied to the "ground truth" of the scenario run (or each scenario run), which generally means simply a suitable representation of the scenario run (including the own-agent's behavior) that is deemed reliable for testing purposes. Ground truth is specific to simulation: the simulator computes a sequence of scenario states that is, by definition, a perfect reliable representation of the simulated scenario run. In real-world scenario runs, there is no "perfect" representation of the scenario run in the same sense, but a suitably informative ground truth can nevertheless be obtained in a number of ways, for example based on manual annotation of on-board sensor data, automated / semi-automated annotation of such data (e.g., using offline / non-real-time processing), and / or the use of external sources of information (e.g., external sensors, maps, etc.).

[0036] A scenario ground truth typically includes the “traces” of the self-agent and, if applicable, any other (salient) agents. A trajectory is a history of an agent’s position and motion over the course of a scenario. There are many ways in which a trajectory can be represented. Trajectory data typically includes spatial and motion data of an agent in an environment. The term is used in relation to both real scenarios (which have real-world trajectories) and simulated scenarios (which have simulated trajectories). A trajectory is typically a record of the actual trajectory achieved by an agent in a scenario. In terms of terminology, “trajectory” and “trajectory” may include the same or similar types of information (e.g., a sequence of spatial and motion states over time). The term trajectory is commonly used in planning contexts (which may refer to a future / predicted trajectory), while the term trajectory is commonly used in relation to past behavior in testing / evaluation contexts.

[0037] In a simulation context, a "scenario description" is provided as input to the simulator. For example, the scenario description may be coded using a scenario description language (SDL) or in any other format that can be used by the simulator. The scenario description is typically a more abstract representation of the scenario and can give rise to multiple simulated runs. Depending on the implementation, the scenario description may have one or more configurable parameters that can be modified to increase the degree of possible variations. The degree of abstraction and parameterization is a design choice. For example, the scenario description may code a fixed layout using parameterized environmental conditions (e.g., weather, lighting, etc.). However, further abstraction is possible, for example, using configurable road parameters (e.g., road curvature, lane configuration, etc.). The input to the simulator comprises the scenario description together with a set of selected parameter values ​​(if applicable). The latter may be referred to as a parameterization of the scenario. The configurable parameters define a parameter space (also called a scenario space), and the parameterizations correspond to points in the parameter space. In this context, a "scenario instance" may refer to the instantiation of a scenario in a simulator based on the scenario description and (if applicable) the selected parameterization.

[0038] For brevity, the term "scenario" may be used to refer to a scenario run as well as to a scenario in the more abstract sense. The meaning of the term scenario will be clear from the context in which it is used.

[0039] Trajectory planning is a key functionality in the context of the present invention, and the terms "trajectory planner", "trajectory planning system", and "trajectory planning stack" may be used interchangeably herein to refer to one or more components capable of planning a trajectory of a mobile robot into the future. Trajectory planning decisions ultimately determine the actual trajectory realized by the own agent (however, in some test contexts this may be influenced by other factors, such as, for example, the implementation of those decisions in the control stack, and the resulting real or modeled dynamic response of the own agent to the control signals).

[0040] A trajectory planner may be tested alone or in combination with one or more other systems (e.g., perception, prediction, and / or control). Within the full stack, planning generally refers to higher-level autonomous decision-making capabilities (e.g., trajectory planning), while control generally refers to lower-level generation of control signals to implement those autonomous decisions. However, in the context of performance testing, the term control is also used in a broader sense. To avoid any doubt, when it is stated that a trajectory planner controls an agent in a simulation, it does not necessarily imply that a control system (in the narrower sense) is tested in combination with the trajectory planner.

[0041] Example AV Stack To provide context related to the described embodiments, further details of an example configuration of an AV stack will now be described.

[0042] 1A shows a highly schematic block diagram of an AV runtime stack 100. The runtime stack 100 is shown to comprise a perception (sub)system 102, a prediction (sub)system 104, a planning (sub)system (planner) 106, and a control (sub)system (controller) 108. As noted above, the term (sub)stack is sometimes used to describe the aforementioned components 102-108.

[0043] In a real-world context, the perception system 102 receives sensor outputs from the AV's on-board sensor system 110 and uses those sensor outputs to detect external agents and measure their physical state, e.g., their position, velocity, acceleration, etc. The on-board sensor system 110 can take various forms, but typically comprises a variety of sensors, such as image capture devices (cameras / optical sensors), lidar and / or radar units, satellite positioning sensors (GPS, etc.), motion / inertial sensors (accelerometers, gyroscopes, etc.). Thus, the on-board sensor system 110 provides a wealth of sensor data from which detailed information can be extracted about the surrounding environment, as well as the state of the AV and any external actors (vehicles, pedestrians, cyclists, etc.) within that environment. Typically, the sensor outputs comprise sensor data of multiple sensor modalities, such as stereo images from one or more stereo optical sensors, lidar, radar, etc. The sensor data of multiple sensor modalities may be combined using filters, fusion components, etc.

[0044] The perception system 102 typically comprises multiple perception components that work together to interpret sensory outputs and thereby provide a sensory output to the predictive system 104 .

[0045] In a simulation context, depending on the nature of the test, and in particular where the stack 100 is "sliced" for testing (see below), the in-vehicle sensor system 100 may or may not need to be modeled. High-level slicing does not require simulated sensor data, so complex sensor modeling is not required.

[0046] The perception output from the perception system 102 is used by a prediction system 104 to predict the future behavior of external actors (agents), such as other vehicles in the vicinity of the AV.

[0047] The predictions calculated by the prediction system 104 are provided to the planner 106, which uses the predictions to make autonomous driving decisions to be performed by the AV in a given driving scenario. The inputs received by the planner 106 typically indicate the drivable area and also capture the predicted movement of external agents (obstacles from the AV's perspective) within the drivable area. The drivable area can be determined using perception output from the perception system 102 in combination with map information, such as an HD (high definition) map.

[0048] The core function of the planner 106 is to plan a trajectory (self-trajectory) for the AV, taking into account the predicted agent movements. This is sometimes called trajectory planning. The trajectory is planned to accomplish a desired goal in the scenario. The goal can be, for example, entering a roundabout and exiting at a desired exit, overtaking a vehicle in front, or staying in the current lane at a target speed (lane following). The goal may be determined, for example, by an autonomous route planner (not shown).

[0049] The controller 108 executes the decisions made by the planner 106 by providing appropriate control signals to the AV's on-board actor systems 112. Specifically, the planner 106 plans a trajectory for the AV, and the controller 108 generates the control signals to execute the planned trajectory. Typically, the planner 106 plans ahead so that the planned trajectory can only be partially executed at the control level, after which a new trajectory is planned by the planner 106. The actor systems 112 include "primary" vehicle systems such as braking, acceleration, and steering systems, as well as secondary systems (e.g., signals, wipers, headlights, etc.).

[0050] It should be noted that there may be differences between the planned trajectory at a given time and the actual trajectory followed by the own agent. The planning system typically operates over a sequence of planning steps, updating the planned trajectory at each planning step to take into account changes in the scenario (or, more precisely, deviations from predicted changes) since the previous planning step. The planning system 106 may reason forward so that the planned trajectory at each planning step extends beyond the next planning step. Thus, individual planned trajectories may not be fully realized (when the planning system 106 is tested alone in a simulation, the own agent may only follow the planned trajectory exactly up to the next planning step, but as mentioned above, in other real and simulated contexts the planned trajectory may not be followed exactly up to the next planning step, because the behavior of the own agent may be influenced by other factors such as the operation of the control system 108 and the real or modeled dynamics of the own vehicle). In many testing contexts, what ultimately matters is the actual trajectory of the own agent, specifically whether the actual trajectory is safe, as well as other factors such as comfort and progress). However, the rule-based testing approach herein can also be applied to planned trajectories (even if those planned trajectories are not fully or accurately realized by the own agent). For example, even if the agent's actual trajectory is deemed safe according to a given safety rule, the momentary planned trajectory may not have been safe, and the fact that the planner 106 was considering an unsafe course of action may become apparent, even if it did not lead to unsafe agent behavior in the scenario. The momentary planned trajectory constitutes one form of internal state that can be usefully evaluated in addition to the actual agent behavior in the simulation. Other forms of internal stack state can be evaluated as well.

[0051] The example of FIG. 1A contemplates a relatively “modular” architecture with separable perception, prediction, planning and control systems 102-108. The sub-stacks themselves may also be modular, for example with separable planning modules within the planning system 106. For example, the planning system 106 may comprise multiple trajectory planning modules that can be applied to different physical contexts (e.g., simple lane driving vs. complex intersections or roundabouts). This is relevant to simulation testing for the reasons discussed above, as it allows components (e.g., the planning system 106 or its individual planning modules) to be tested individually or in different combinations. To avoid any doubt, in a modular stack architecture, the term stack may refer not only to the full stack but also to its individual sub-systems or modules.

[0052] The degree to which various stack functions are integrated or separable may vary significantly between different stack implementations, and in some stacks, certain aspects may be so tightly coupled that they are indistinguishable. For example, in other stacks, planning and control may be integrated (e.g., such stacks may perform planning directly in terms of control signals), while other stacks (e.g., those shown in FIG. 1A) may be designed in a way that makes a clear distinction between the two (e.g., performing planning in terms of trajectories and performing independent control optimization to determine how best to execute the planned trajectory at the control signal level). Similarly, in some stacks, prediction and planning may be more tightly coupled. In extreme cases, in so-called "end-to-end" operation, perception, prediction, planning, and control may be essentially inseparable. Unless otherwise specified, the terms perception, prediction, planning, and control as used herein do not imply any particular coupling or modularization of these aspects.

[0053] It will be appreciated that the term "stack" encompasses software, but can also encompass hardware. In a simulation, the software of the stack may be tested on a "generic" off-board computer system before ultimately being uploaded to the on-board computer system of a physical vehicle. However, in "hardware-in-the-loop" testing, testing may extend to the underlying hardware of the vehicle itself. For example, the stack software may be run on an on-board computer system (or a replica thereof) that is coupled to a simulator for testing purposes. In this context, the stack under test extends to the underlying computer hardware of the vehicle. As another example, a particular function of the stack 100 (e.g., a perception function) may be implemented in dedicated hardware. In a simulation context, hardware-in-the-loop testing may include feeding synthetic sensor data to a perception component of dedicated hardware.

[0054] FIG. 1B shows a very schematic overview of a testing paradigm for autonomous vehicles. An ADS / ADAS stack 100, for example of the kind shown in FIG. 1A, is iteratively tested and evaluated in simulation by running multiple scenario instances in a simulator 202 and evaluating the performance of the stack 100 (and / or its individual sub-stacks) with a test oracle 252. The output of the test oracle 252 is useful information for an expert 122 (a team or individual), enabling the expert 122 to identify problems in the stack 100 and modify the stack 100 to mitigate those problems (S124). The results also help the expert 122 select further scenarios for testing (S126), and the process continues by iteratively modifying, testing, and evaluating the performance of the stack 100 in simulation. The improved stack 100 is finally integrated into a real-world AV 101 equipped with sensor systems 110 and actor systems 112 (S125). The improved stack 100 typically includes program instructions (software) that execute on one or more computer processors of the vehicle 101's on-board computer system (not shown). The improved stack software is uploaded to the AV 101 in step S125. Step 125 may also include modifications to the underlying vehicle hardware. Once on board the AV 101, the improved stack 100 receives sensor data from the sensor system 110 and outputs control signals to the actor system 112. Real-world testing (S128) can be used in combination with simulation-based testing. For example, once an acceptable level of performance has been reached through the process of simulation testing and stack refinement, appropriate real-world scenarios may be selected (S130) and the performance of the AV 101 in those real-world scenarios may be captured and similarly evaluated in the test oracle 252.

[0055] Scenarios can be obtained for simulation purposes in a variety of ways, including manual coding. The system can also extract scenarios from real-world runs for simulation purposes, allowing real-world situations and variations thereof to be recreated within the simulator 202.

[0056] FIG. 1C shows a highly schematic block diagram of a scenario extraction pipeline. Real-world run data 140 is passed to a “ground truthing” pipeline 142 for the purpose of generating a scenario ground truth. The run data 140 may comprise, for example, sensor data and / or perception output captured / generated on one or more vehicles (which may be autonomous, human-driven, or a combination thereof) and / or data captured from other sources such as external sensors (e.g., CCTV). The run data is processed within the ground truthing pipeline 142 to generate appropriate ground truth 144 (trajectory and context data) for the real-world run. As discussed, the ground truthing process may be based on manual annotation of the “raw” run data 140, or the process may be fully automated (e.g., using offline perception methods), or a combination of manual and automated ground truthing may be used. For example, 3D bounding boxes may be placed around vehicles and / or other agents captured in the run data 140 to determine the spatial and motion states of their trajectories. A scenario extraction component 146 receives the scenario ground truth 144 and processes it to extract a more abstracted scenario description 148 that can be used for simulation purposes. The scenario description 148 is used by the simulator 202 to allow multiple simulated runs to be conducted. A simulated run is a variation of the original real-world run, with the degree of possible variation depending on the degree of abstraction. A ground truth 150 is provided for each simulated run.

[0057] Test Pipeline Next, further details of the test pipeline and test oracle 252 are described. The following example focuses on simulation-based testing. However, as noted above, the test oracle 252 can be similarly applied to evaluate stack performance in real-world scenarios, and the relevant discussion below applies to real-world scenarios as well. The following discussion refers to the stack 100 of FIG. 1A as an example. However, as noted above, the test pipeline 200 is highly flexible and can be applied to any stack or sub-stack operating at any autonomy level.

[0058] FIG. 2 shows a schematic block diagram of a test pipeline, denoted by reference numeral 200. The test pipeline 200 is shown to include a simulator 202 and a test oracle 252. The simulator 202 runs simulated scenarios for the purpose of testing all or part of the AV runtime stack 100, and the test oracle 252 evaluates the performance of the stack (or sub-stack) in the simulated scenario. As discussed, only a sub-stack of the runtime stack may be tested, but for simplicity, the following description refers to the (full) AV stack 100 throughout. However, the description applies equally to a sub-stack instead of the full stack 100. The term "slicing" is used herein for the selection of a set or subset of stack components for testing.

[0059] As mentioned before, the idea of ​​simulation-based testing is to run simulated driving scenarios in which the own agent has to navigate under the control of the stack 100 under test. Typically, a scenario comprises a static drivable area (e.g. a certain static road layout) in which the own agent is asked to navigate, typically in the presence of one or more other dynamic agents (e.g. other vehicles, cyclists, pedestrians, etc.). For this purpose, simulated inputs 203 are provided from a simulator 202 to the stack 100 under test.

[0060] The slicing of the stack dictates the form of the simulated inputs 203. As an example, FIG. 2 shows prediction, planning and control systems 104, 106 and 108 in the AV stack 100 under test. To test the full AV stack of FIG. 1A, the perception system 102 can also be applied under test. In this case, the simulated inputs 203 comprise synthetic sensor data, generated using appropriate sensor models, and processed in the perception system 102 in the same way as real sensor data. This requires the generation of sufficiently realistic synthetic sensor inputs (e.g., photorealistic image data and / or similarly realistic simulated lidar / radar data). The resulting output of the perception system 102 is then fed to the higher-level prediction and planning systems 104, 106.

[0061] In contrast, so-called "planning-level" simulation essentially bypasses the perception system 102. Instead, the simulator 202 provides simpler, higher-level inputs 203 directly to the prediction system 104. In some contexts, it may even be appropriate to bypass the prediction system 104 as well, in order to test the planner 106 based on predictions derived directly from simulated scenarios (i.e., "perfect" predictions).

[0062] In between these extremes there is room for many different levels of input slicing, for example testing only a subset of the perception system 102, e.g., “late” (higher level) perception components, such as filters or fusion components that operate on output from lower level perception components (e.g., object detectors, bounding box detectors, motion detectors, etc.).

[0063] Whatever form it takes, the simulated inputs 203 are used (directly or indirectly) as the basis for decision-making by the planner 108. The controller 108 then implements the planner's decisions by outputting control signals 109. In a real-world context, these control signals drive the AV's physical actor-system 112. The simulation uses an ego-vehicle dynamics model 204 to simulate the autonomous vehicle's physical responses to the control signals 109 by translating the resulting control signals 109 into realistic movement of the ego-agent within the simulation.

[0064] Alternatively, a simpler form of simulation assumes that the ego-agent follows each planned trajectory exactly between planning steps. This approach bypasses the control system 108 (to the extent that it is separable from the plan) and removes the need for an ego-vehicle dynamics model 204. This may be sufficient to test certain aspects of the plan.

[0065] To the extent that the external agent exhibits autonomous behavior / decision making within the simulator 202, some form of agent decision logic 210 is implemented to make those decisions and determine the agent's behavior within the scenario. The agent decision logic 210 may be of comparable complexity to the self-stack 100 itself, or may have a more limited decision-making capability. The aim is to provide sufficiently realistic external agent behavior within the simulator 202 to allow useful testing of the decision-making capabilities of the self-stack 100. In some contexts this does not require agent decision logic 210 at all (open-loop simulation), in other contexts useful testing can be provided using relatively limited agent logic 210 such as basic adaptive cruise control (ACC). Where appropriate, one or more agent dynamics models 206 may be used to provide more realistic agent behavior.

[0066] A scenario is run according to the scenario's scenario description 201a and (if applicable) selected parameterization 201b. A scenario typically has both static and dynamic elements, which may be "hard coded" in the scenario description 201a or may be configurable and thus determined by the scenario description 201a in combination with the selected parameterization 201b. In a driving scenario, the static elements typically include a static road layout.

[0067] Dynamic elements typically include one or more external agents in the scenario, e.g. other vehicles, pedestrians, cyclists, etc.

[0068] The extent of dynamic information provided to the simulator 202 for each external agent can vary. For example, a scenario may be described by separable static and dynamic layers. A given static layer (e.g., defining a road layout) can be used in combination with various dynamic layers to provide different scenario instances. The dynamic layer may comprise, for each external agent, the spatial path followed by that agent along with one or both of the motion and behavior data associated with that path. In a simple open-loop simulation, the external actor simply follows the spatial path and motion data defined in the dynamic layer, which is non-reactive, i.e., does not react to the own agent in the simulation. Such an open-loop simulation can be implemented without the agent decision logic 210. However, in a closed-loop simulation, the dynamic layer instead defines at least one behavior (e.g., ACC behavior) that is followed along the static path. In this case, the agent decision logic 210 implements that behavior in a reactive manner in the simulation, i.e., reactively to the own agent and / or other external agents. The motion data may still be associated with the static path, but in this case it is less prescriptive and may, for example, serve as targets along the path. For example, an ACC behavior may set a target speed along the path that the agent is attempting to match, but the agent decision logic 210 may be allowed to slow the external agent's speed below the target at any point along the path in order to maintain the target distance from the vehicle ahead.

[0069] As will be appreciated, a scenario can be described in many ways with any degree of configurability for simulation purposes. For example, the number and type of agents, as well as their movement information, may be configurable as part of the scenario parameterization 201b.

[0070] The output of the simulator 202 for a given simulation includes an self-trajectory 212a for an own agent and one or more agent trajectories 212b (trajectories 212) for one or more foreign agents. Each trajectory 212a, 212b is a complete history of the agent's behavior within the simulation having both spatial and motion components. For example, each trajectory 212a, 212b may take the form of a spatial path with motion data associated with points along the path, such as velocity, acceleration, jerk (rate of change of acceleration), snap (rate of change of jerk), etc.

[0071] Additional information is also provided to supplement and provide context to the trajectory 212. Such additional information is referred to as "context" data 214. The context data 214 relates to the physical context of the scenario and can have both static components (e.g., road layout) and dynamic components (e.g., weather conditions to the extent that they change over the course of the simulation). The context data 214 may be to some extent "pass-through" in that it is not affected by the outcome of the simulation, since it is directly defined by the selection of the scenario description 201a or parameterization 201b. For example, the context data 214 may include a static road layout that is directly brought about by the scenario description 201a or parameterization 201b. Typically, however, the context data 214 will include at least some elements that are derived within the simulator 202. This may include, for example, simulated environment data, such as weather data, and the simulator 202 is free to change the weather conditions as the simulation progresses. In that case, the weather data may be time-dependent, and that time-dependency is reflected in the context data 214.

[0072] A test oracle 252 receives the trajectories 212 and the context data 214 and scores their outputs against a set of performance evaluation rules 254. The performance evaluation rules 254 are shown provided as inputs to the test oracle 252.

[0073] The rules 254 are typically categorical (e.g., pass / fail type rules). A particular performance evaluation rule is also associated with a numerical performance metric (e.g., indicating the degree of achievement or failure, or other quantity that helps to explain or is otherwise related to the categorical outcome) that is used to "score" the trajectory. The evaluation of the rules 254 is time-based, and a given rule may have different outcomes at different points in the scenario. Scoring is also time-based, and for each performance evaluation metric, the test oracle 252 tracks how the value (score) of that metric changes over time as the simulation progresses. The test oracle 252 provides an output 256 comprising a time sequence 256a of the categorical (e.g., pass / fail) results for each rule and a score-time plot 256b for each performance metric, as described in more detail below. The results and scores 256a, 256b are useful information for the expert 122 and can be used to identify and mitigate performance issues in the tested stack 100. The test oracle 252 also provides an overall (aggregate) outcome of the scenario (e.g., overall pass / fail). The output 256 of the test oracle 252 is stored in a test database 258 in association with information about the scenario to which the output 256 pertains. For example, the output 256 may be stored in association with the scenario description 210a (or an identifier thereof) and the selected parameterization 201b. As well as time-dependent outcomes and scores, an overall score may also be assigned to the scenario and stored as part of the output 256. For example, an aggregate score for each rule (e.g., overall pass / fail) and / or an aggregate outcome across all rules 254 (e.g., pass / fail).

[0074] Figure 2A shows another option for slicing, using reference numbers 100 and 100S to represent the full stack and sub-stack, respectively. It is the sub-stack 100S that is the subject of testing in the test pipeline 200 of Figure 2.

[0075] Several "late" perceptual components 102B form part of the sub-stack 100S being tested and are applied to the simulated perceptual input 203 during testing. The late perceptual components 102B may include filtering or other fusion components that fuse the perceptual inputs from multiple early perceptual components.

[0076] In the full stack 100, the later perception component 102B receives actual perceptual input 213 from the early perception component 102A. For example, the early perception component 102A may comprise one or more 2D or 3D bounding box detectors, in which case the simulated perceptual input provided to the later perception component may comprise simulated 2D or 3D bounding box detection results derived by ray tracing in simulation. The early perception component 102A generally comprises components that operate directly on the sensor data. In the slicing of FIG. 2A, the simulated perceptual input 203 corresponds formally to the actual perceptual input 213 typically provided by the early perception component 102A. However, the early perception component 102A is not applied as part of the test, but instead is used to train one or more perceptual error models 208, which can be used to introduce realistic errors in a statistically rigorous manner into the simulated perceptual input 203 provided to the later perception component 102B of the sub-stack 100 under test.

[0077] Such a perceptual error model may be referred to as a Perception Statistical Performance Model (PSPM), or synonymously as "PRISM". Further details of the principles of PSPM, and suitable techniques for constructing and training PSPMs, can be found in International Patent Publications Nos. 2021037763, 2021037760, 2021037765, 2021037761, and 2021037766, each of which is incorporated herein by reference in its entirety. The idea behind PSPM is to efficiently introduce realistic errors into the simulated perceptual inputs provided to the sub-stack 100S (i.e., reflecting the kind of errors that would be expected if the early perceptual component 102A were applied in the real world). In a simulation context, "perfect" ground truth perceptual inputs 203G are provided by the simulator, but these are used to derive more realistic perceptual inputs 203 with realistic errors introduced by the perceptual error model 208.

[0078] As explained in the above-cited references, a PSPM can rely on one or more variables that represent physical conditions ("confounders"), allowing different levels of error to be introduced that reflect different possible real-world conditions. Thus, simulator 202 can simulate different physical conditions (e.g., different weather conditions) simply by changing the values ​​of the weather confounders to vary how perceived error is introduced.

[0079] The later perceptual components 102b in the sub-stack 100S process the simulated perceptual input 203 in exactly the same way as they process real-world perceptual input 213 in the full-stack 100, and their output drives prediction, planning, and control.

[0080] Alternatively, PRISM can be used to model the entire perceptual system 102, including the late perceptual components 208, in which case PSPM is used to generate realistic perceptual outputs that are passed directly as input to the predictive system 104.

[0081] Depending on the implementation, there may or may not be a deterministic relationship between a given scenario parameterization 201b and the outcome of a simulation for a given configuration of the stack 100 (i.e., the same parameterization may or may not always lead to the same outcome for the same stack 100). Non-determinism may arise in various ways. For example, if the simulation is based on PRISM, PRISM may model a distribution of possible sensory outputs for each given time step of the scenario, from which realistic sensory outputs are probabilistically sampled. This may lead to non-deterministic behavior in the simulator 202, such that different sensory outputs are sampled and therefore different outcomes may be obtained for the same stack 100 and scenario parameterization. Alternatively or additionally, the simulator 202 may be non-deterministic in nature, e.g., weather, lighting, or other environmental conditions may be randomized / stochastic to some extent within the simulator 202. As will be appreciated, this is a design choice, and in other implementations, the various environmental conditions may instead be fully specified in the scenario parameterization 201b. In a non-deterministic simulation, multiple scenario instances can be run for each parameterization. For a particular parameterization 201b choice, an aggregate pass / fail outcome can be assigned, for example as a count or percentage of pass / fail outcomes.

[0082] The test orchestration component 260 is responsible for selecting a scenario for simulation purposes. For example, the test orchestration component 260 may automatically select a scenario description 201a and an appropriate parameterization 201b based on the test oracle output 256 from a previous scenario.

[0083] Test oracle rules: Performance evaluation rules 254 are constructed as computational graphs (rule trees) that are applied within a test oracle. Unless otherwise noted, the term "rule tree" herein refers to the computational graph that is configured to implement a given rule. Each rule is constructed as a rule tree, and a set of multiple rules may be referred to as a "forest" of multiple rule trees.

[0084] FIG. 3A illustrates an example of a rule tree 300 constructed from a combination of extractor nodes (leaf objects) 302 and assessor nodes (non-leaf objects) 304. Each extractor node 302 extracts a time-varying numeric (e.g., floating point) signal (score) from a set of scenario data 310. The scenario data 310 is a form of scenario ground truth in the sense described above and may be referred to as such. The scenario data 310 has been obtained by deploying a trajectory planner (e.g., planner 106 of FIG. 1A) in a real or simulated scenario and is shown to comprise self and agent trajectories 212 and context data 214. In the simulation context of FIG. 2 or FIG. 2A, the scenario ground truth 310 is provided as an output of the simulator 202.

[0085] Each assessor node 304 is shown to have at least one child object (node), which is either one of the extractor nodes 302 or another one of the assessor nodes 304. Each assessor node receives the outputs from its child nodes and applies an assessor function to those outputs. The output of the assessor function is a time series of categorical results. The following example considers simple binary pass / fail results, but the technique can be easily extended to non-binary results. Each assessor function evaluates the output of its child nodes against predefined atomic rules. Such rules can be flexibly combined depending on the desired safety model.

[0086] In addition, each assessor node 304 derives a time-varying numerical signal from the outputs of its child nodes, which is related to a categorical outcome by a threshold condition (see below).

[0087] The top-level root node 304a is an assessor node that is not a child node of any other node. The top-level node 304a outputs a sequence of final results, and its descendants (i.e., nodes that are direct or indirect children of the top-level node 304a) provide underlying signals and intermediate results.

[0088] 3B visually illustrates an example time series of derived signal 312 and corresponding results 314 calculated by assessor node 304. Results 314 are correlated to derived signal 312 in that a pass result is returned if (and only if) the derived signal exceeds a fail threshold 316. As will be appreciated, this is just one example of a threshold condition relating a time sequence of results to a corresponding signal.

[0089] Signals extracted directly from the scenario ground truth 310 by the extractor nodes 302 may be referred to as "raw" signals to distinguish them from "derived" signals computed by the assessor nodes 304. The results and the raw / derived signals may be discretized in time.

[0090] FIG. 4A shows an example of a rule tree implemented within the test platform 200.

[0091] A rule editor 400 is provided for constructing rules that are implemented in the test oracle 252. The rule editor 400 receives rule creation input from a user, who may or may not be an end user of the system. In this example, the rule creation input is coded in a domain specific language (DSL) and defines at least one rule graph 408 that is implemented within the test oracle 252. In the following example, the rules are logical rules, with true and false representing pass and fail respectively (as will be appreciated, this is purely a design choice).

[0092] The following examples consider rules that are formulated using combinations of atomic logical predicates. Examples of basic atomic predicates include elementary logic gates (OR, AND, etc.) and logical functions, such as "greater than", (Gt(a,b)), which returns true if a is greater than b, and false otherwise.

[0093] The Gt function is for implementing the safe lateral distance rule between the self agent and other agents in the scenario (with agent identifier "other_agent_id"). Two extractor nodes (latd, latsd) apply the LateralDistance and LateralSafeDistance extractor functions, respectively. These functions operate directly on the scenario ground truth 310 to extract a time-varying lateral distance signal (measuring the lateral distance between the self agent and the identified other agent) and a time-varying safe lateral distance signal for the self agent and the identified other agent, respectively. The safe lateral distance signal may depend on various factors such as the speed of the self agent and the speed of the other agent (captured in the trajectory 212) as well as the environmental conditions (e.g., weather, lighting, road type, etc.) captured in the context data 214.

[0094] The assessor node (is_latd_safe) is the parent of the latd and latsd extractor nodes and is mapped to the Gt atomic predicate. Thus, when the rule tree 408 is implemented, the is_latd_safe assessor node applies the Gt function to the outputs of the latd and latsd extractor nodes to calculate a true / false result for each time step of the scenario, returning true for each time step where the latd signal exceeds the latsd signal and false otherwise. In this way, a "safe lateral distance" rule is constructed from the atomic extractor functions and predicates, and if the lateral distance reaches or falls below the safe lateral distance threshold, the ego fails the safe lateral distance rule. As will be appreciated, this is a very simple example of a rule tree. Rules of arbitrary complexity can be constructed following the same principles.

[0095] The test oracle 252 applies the rule tree 408 to the scenario ground truth 310 and provides the results via a user interface (UI) 418.

[0096] FIG. 4B shows an example of a rule tree including a lateral distance branch corresponding to FIG. 4A. Additionally, the rule tree includes a longitudinal distance branch and a top-level OR predicate (safe distance node, is_d_safe) to implement a safe distance metric. Similar to the lateral distance branch, the longitudinal distance branch extracts the longitudinal distance and longitudinal distance threshold signals (extractor nodes lond and lonsd, respectively) from the scenario data, and the longitudinal safety assessor node (is_lond_safe) returns true if the longitudinal distance is above the safe longitudinal distance threshold. The top-level OR node returns true if one or both of the lateral and longitudinal distances are safe (below the appropriate threshold) and false if neither is safe. In this context, it is sufficient that only one of the distances is above the safety threshold (e.g., if two vehicles are traveling in adjacent lanes, when they are next to each other, the longitudinal separation is zero or near zero, but the situation is not dangerous if the vehicles have sufficient lateral separation).

[0097] The numerical output of the top node can be, for example, a time-varying robustness score.

[0098] Different rule trees can be constructed, for example, to implement different rules for a given safety model, to implement different safety models, or to selectively apply rules to different scenarios (for a given safety model, not all rules necessarily apply to all scenarios, and with this approach different rules or combinations of rules can be applied to different scenarios).Within this framework, rules can also be constructed to assess comfort (e.g., based on instantaneous acceleration and / or jerk along the trajectory), progress (e.g., based on the time it takes to reach a defined goal), etc.

[0099] The above examples consider simple logical predicates that are evaluated on a result or signal at a single time, e.g., OR, AND, Gt. In practice, however, it may be desirable to formulate certain rules in terms of temporal logic.

[0100] Hekmatnejad et al., "Encoding and Monitoring Responsibility Sensitive Safety Rules for Automated Vehicles in Signal Temporal Logic" (2019), MEMOCODE '19: Proceedings of the 17th ACM-IEEE International Conference on Formal Methods and Models for System Design, incorporated herein by reference in its entirety, discloses a signal temporal logic (STL) encoding of RSS safety rules. Temporal logic provides a formal framework for constructing predicates that are conditional on time. This means that the result computed by the assessor at a given time can depend on the results and / or signal values ​​at other times.

[0101] For example, a requirement in a safety model may be that an agent responds to a particular event within a set time window. Such rules can be coded in a similar way, using temporal logic predicates in a rule tree.

[0102] In the above example, the performance of the stack 100 is evaluated at each time step of the scenario. From this, an overall test result (e.g., pass / fail) can be derived, for example, a particular rule (e.g., a safety-critical rule) may result in an overall fail if the rule is failed at any time step in the scenario (i.e., the rule must be passed at all time steps to obtain an overall pass of the scenario). For other types of rules, the overall pass / fail criteria may be "softer" (e.g., for a particular rule, a fail may be triggered only if the rule has failed for a certain number of consecutive time steps), and such criteria may be context dependent.

[0103] 4C illustrates generally the hierarchy of rule evaluation implemented within a test oracle 252. A set of rules 254 is received for implementation in the test oracle 252.

[0104] Certain rules apply only to the own agent (one example is the comfort rule, which assesses whether a maximum acceleration or jerk threshold is being exceeded by the own trajectory at any given time).

[0105] Other rules relate to interactions between the ego agent and other agents (e.g., the "no collision" rule or the safe distance rule discussed above). Each such rule is evaluated in a pairwise manner between the ego agent and each other agent. As another example, a "pedestrian emergency braking" rule may be activated only if a pedestrian walks in front of the ego vehicle, and only with respect to that pedestrian agent.

[0106] Not all rules are necessarily applicable to all scenarios, and some rules may only be applicable to some of the scenarios. Rule activation logic 422 within test oracle 252 determines whether and when each of rules 254 is applicable to the scenario in question, and if so, selectively activates the rule when applicable. Thus, a rule may remain active throughout the entire scenario, may never be activated in a given scenario, or may only be activated in some parts of the scenario. Furthermore, rules may be evaluated for different numbers of agents at different points in the scenario. Selectively activating rules in this manner can greatly improve the efficiency of test oracle 252.

[0107] The activation or deactivation of a given rule may depend on the activation / deactivation of one or more other rules: for example, the "optimal comfort" rule may be considered non-applicable when the pedestrian emergency braking rule is activated (since pedestrian safety is the primary concern), and the former may be deactivated whenever the latter is active.

[0108] The rule evaluation logic 424 evaluates each active rule for the period of time that it remains active. Each interactive rule is evaluated in a pairwise manner between the self agent and the other agents to which it applies.

[0109] There may also be some degree of interdependence in the application of rules: for example, another way to deal with the relationship between a comfort rule and an emergency braking rule would be to increase the jerk / acceleration threshold of the comfort rule whenever the emergency braking rule is activated for at least one other agent.

[0110] Although pass / fail outcomes are considered, rules may be non-binary. For example, two categories of failures may be introduced: "acceptable" and "unacceptable". Considering again the relationship between a comfort rule and an emergency braking rule, an acceptable failure of a comfort rule may occur when that rule was failed but the emergency braking rule was active. Thus, interdependencies between rules can be addressed in various ways.

[0111] The activation criteria for rules 254 may be specified in the rule creation code provided to the rule editor 400, as may the nature of the rule interdependencies and the mechanisms for implementing those interdependencies.

[0112] Graphical User Interface 5 shows a schematic block diagram of a visualization component 520. The visualization component is shown having an input connected to a test database 258 for rendering the output 256 of the test oracle 252 on a graphical user interface (GUI) 500. The GUI is rendered on a display system 522.

[0113] FIG. 5A shows an example view of the GUI 500 for a particular scenario involving multiple agents. In this example, the test oracle output 526 pertains to multiple external agents, and the results are organized by agent. For each agent, a time series of results is available for each rule that applies to that agent at some point in the scenario. In the example shown, the summary view for "Agent01" has been selected, and the "top" results calculated for each applicable rule are displayed. There is a top result calculated at the root node of each rule tree. Color coding is used to distinguish between periods when the rule is inactive ("not applicable") for that agent, active and passing, and active and failing.

[0114] For each time series of results a first selectable element 534a is provided, which allows results at a lower level of the rule tree, i.e. results calculated further down the rule tree, to be accessed.

[0115] FIG. 5B shows a first expanded view of the results of “Rule 02”, where the results of the lower level nodes are also visualized. For example, for the “Safe Distance” rule in FIG. 4B, the results of the “is_latd_safe” and “is_lond_safe” nodes may be visualized (labeled “C1” and “C2” in FIG. 5B). In the first expanded view of Rule 02, it can be seen that the success / failure of Rule 02 is defined by a logical OR relationship between the results C1 and C2, and Rule 02 fails only if both C1 and C2 fail (as in the case of the “Safe Distance” rule above).

[0116] For each result time series a second selectable element 534b is provided which allows an associated numerical performance score to be accessed.

[0117] FIG. 5C shows a second expanded view, where the results for Rule 02 and the results for "C1" have been expanded to reveal the associated scores for the time periods that these rules were active for Agent 01. The scores are displayed as a visual score-time plot, similarly color-coded to represent pass / fail.

[0118] Example scenario: 6A illustrates a first instance of a cut-in scenario in simulator 202 that ends with a collision event between ego vehicle 602 and another vehicle 604. The cut-in scenario is characterized as a multi-lane driving scenario, in which ego vehicle 602 is traveling along a first lane 612 (ego lane) and the other vehicle 604 is traveling initially along a second adjacent lane 604. At some point in this scenario, the other vehicle 604 moves from the adjacent lane 614 into ego lane 612, ahead of ego vehicle 602 (the cut-in distance). In this scenario, ego vehicle 602 is unable to avoid a collision with the other vehicle 604. The first scenario instance ends in response to this collision event.

[0119] FIG. 6B shows an example of a first oracle output 256a obtained from the ground truth 310a of the first scenario instance. The "no collision" rule is evaluated between the ego-vehicle 602 and the other vehicle 604 for the duration of the scenario. A collision event results in a failure of this rule at the end of the scenario. In addition, the "safe distance" rule of FIG. 4B is evaluated. When the other vehicle 604 approaches the ego-vehicle 602 laterally, there comes a point (t1) when both the safe lateral distance threshold and the safe longitudinal distance threshold are violated, which results in a failure of the safe distance rule that persists until a collision event at time t2.

[0120] 6C illustrates a second instance of the cut-in scenario, in which the cut-in event does not result in a collision and the ego vehicle 602 is able to reach a safe distance behind the other vehicle 604 after the cut-in event.

[0121] 6D shows an example of a second oracle output 256b obtained from the ground truth 310b of the second scenario instance. In this case, the "no collision" rule is passed throughout. The safe distance rule is violated at time t3 when the lateral distance between the ego-vehicle 602 and the other vehicle 604 becomes unsafe. However, at time t4, the ego-vehicle 602 manages to reach a safe distance behind the other vehicle 604. Thus, the safe distance rule is only failed between time t3 and time t4.

[0122] Rules Editor - Domain Specific Language (DSL) FIG. 7 shows an example of rule creation input to the test oracle 400 coded with a particular DSL choice.

[0123] 7, a custom rule graph can be constructed within the test platform 200. The test oracle 252 is configured to provide a set of modular "building blocks" in the form of predefined extractor functions 702 and predefined assessor functions 704.

[0124] The rule editor 400 receives rule creation input from a user. The rule creation input is coded in a DSL, and an exemplary section of the rule creation code 706 is shown. The rule creation code 706 defines a custom rule graph 408 corresponding to FIG. 4A. The choice of rule graph is purely exemplary; an advantage of the DSL is that a desired rule graph can be constructed in a bespoke manner by the user. The rule editor 400 interprets the rule creation code 706 and causes the custom rule graph 408 to be implemented in the test oracle 252.

[0125] Within code 706, an extractor node creation input is shown and labeled 711. Extractor node creation input 711 is shown to include an identifier 712 of one of the predefined extractor functions 702.

[0126] A create assessor node input 713 is also shown, and is shown to comprise an identifier 714 of one of the predefined assessor functions 704. Here, the input 713 indicates that an assessor node is to be created having two child nodes with node identifiers 715a, 715b (which happen to be extractor nodes in this example, but could in general be assessor nodes, extractor nodes, or a combination of both).

[0127] The nodes of a custom rule graph are objects in the object-oriented programming (OOP) sense. A node factory class (Nodes()) is provided in the test oracle 252. To implement a custom rule graph 408, a node factory class 710 is instantiated and the node creation function (add_node) of the resulting factory object 710 (node-factory) is called with details of the node to be created.

[0128] According to code 706, the Gt function is used to implement the safe lateral distance rule between the own agent and other agents in the scenario (with agent identifier "other_agent_id"). Two extractor nodes (latd, latsd) are defined in code 406 and are mapped to predefined LateralDistance and LateralSafeDistance extractor functions, respectively. These functions operate directly on the scenario ground truth 310 to extract a time-varying lateral distance signal (measuring the lateral distance between the own agent and the identified other agent) and a time-varying safe lateral distance signal for the own agent and the identified other agent, respectively. The safe lateral distance signal may depend on various factors such as the speed of the own agent and the speed of the other agent (captured in the trajectory 212), as well as environmental conditions (e.g., weather, lighting, road type, etc.) captured in the context data 214. This is largely invisible to the end user, who simply selects the desired extractor function (however, depending on the implementation, one or more configurable parameters of the function may be exposed to the end user).

[0129] An assessor node (is_latd_safe) is defined in code 706 as the parent of the latd and latsd extractor nodes, and is mapped to the Gt atomic predicate. Thus, when the rule tree 408 is implemented, the is_latd_safe assessor node applies the Gt function to the outputs of the latd and latsd extractor nodes to calculate a true / false result for each time step of the scenario, returning true for each time step where the latd signal exceeds the latsd signal, and false otherwise. In this way, a "safe lateral distance" rule is constructed from the atomic extractor function and predicate, and if the lateral distance reaches or falls below the safe lateral distance threshold, the own agent fails the safe lateral distance rule. As will be appreciated, this is a very simple example of a custom rule. Rules of arbitrary complexity can be constructed following the same principles. The test oracle 252 applies the custom rule tree 408 to the scenario ground truth 310 and provides the results in the form of an output graph 717, i.e., the test oracle 252 does not just provide the top-level output, but also the output computed at each node of the custom rule graph 408. In the "safe lateral distance example," the time series of results computed by the is_latd_safe node is provided, but the underlying signals latd and latsd are also provided in the output graph 717, allowing the end user to easily investigate the cause of the failure of a particular rule at any level in the graph. In this example, the output graph 717 is a visual representation of the custom rule graph 408 displayed via a user interface (UI) 418, where each node of the custom rule graph is complemented by a visualization of its output, as shown in Figures 5A-C.

[0130] FIG. 8 shows a further exemplary view of the GUI 500 for rendering a custom rule tree. Multiple output graphs are available through the GUI and are displayed in association with a visualization 501 of the scenario ground truth to which the output graph pertains. Each output graph is a visual representation of a particular rule graph, supplemented by a visualization of the outputs of each node of that rule graph. Each output graph is initially displayed in a collapsed form, where only the root node of each computation graph is displayed. First and second visual elements 802, 804 represent the root nodes of the first and second computation graphs, respectively. The first output graph is drawn in a collapsed form, where only the time series of the binary pass / fail results of the root node is visualized (as a simple colored horizontal bar in the first visual element 802). However, the first visual element 802 is selectable to expand the visualization to lower level nodes and their outputs. The second output graph is drawn in an expanded form, where it is accessed by selecting the second visual element 804. Visual elements 806, 808 represent lower level assessor nodes in the applicable rule graph, and their results are visualized as well. Visual elements 810, 812 represent extractor nodes in the graph. Each node visualization is also selectable to render an expanded view of that node. The expanded view provides a visualization of the time-varying numerical signal calculated or extracted at that node. A second visual element 804 is shown expanded, displaying a visualization of its derived signal instead of its resulting binary sequence. The derived signal is color-coded based on a fail threshold (signal dropping below zero represents a fail of the applicable rule in this example). The extractor node visualizations 810, 812 are similarly expandable to render a visualization of their raw signal. The view in FIG. 8 renders the output of the rule graph once it has been evaluated on a given set of scenario ground truths. Additionally, an initial visualization may be rendered prior to its evaluation for the benefit of the user creating the rule graph.The initial visualization may be updated in response to changes in the rule creation code 406.

[0131] Although not shown in FIG. 7, the node creation inputs 711, 713 may additionally set values ​​for one or more configurable parameters (e.g., thresholds, time intervals, etc.) of the associated assessor or extractor function.

[0132] In certain embodiments, improved computational efficiency can be achieved through selective evaluation of rule graphs. For example, in the graph of FIG. 7, if is_latd_safe returns true at a time step or time interval, the output of the topmost is_d_safe node can be calculated without evaluating the forward or backward distance branches for that time step / interval. Such increased efficiency is based on a "top-down" evaluation of the graph, i.e., starting from the top of the tree, only computing branches down to the extractor node as necessary to obtain the topmost output.

[0133] The assessor or extractor functions may have one or more configurable parameters. For example, the latsd and lonsd nodes may have a configurable parameter that specifies how the threshold distance is extracted from the scenario ground truth 310, for example as a configurable function of ego velocity.

[0134] Further efficiency gains are obtained by caching and reusing results whenever possible.

[0135] For example, when a user modifies the graph or some parameter, only the outputs of the affected nodes (possibly only to the extent needed to compute the top-level result - see above) may be recalculated.

[0136] While the above examples contemplate outputs in the form of time-varying signals and / or time series of categories (e.g., pass / fail or true / false results), alternatively or additionally, other types of outputs can be passed between nodes. For example, time-varying iterables (i.e., objects that can be iterated over in a for loop) may be passed between nodes.

[0137] Variables may be assigned at runtime and / or passed through the tree and bound. The combination of runtime variables and iterables provides loop control and runtime (scenario-relevant) parameterization while the tree itself remains "static".

[0138] A for loop can define scenario-specific conditions under which the rule should be applied (e.g. "for the agent ahead" or "for each traffic light at this intersection"). To implement such a loop, variables are needed (e.g. to implement a "for each nearby agent" loop based on the "other_agent" variable), but they can also be used to define (remember) variables in the current context, which can then be accessed (loaded) by other blocks (nodes) further down in the tree.

[0139] Time periods may be calculated only as needed (again in a top-down manner), and the results may be cached and merged for new required time periods.

[0140] For example, one rule (rule graph) may require that the acceleration of the vehicle ahead be calculated to check against the adaptive cruise control following distance, while another rule (rule tree) may require the acceleration of all vehicles around the own agent ("nearby" agents).

[0141] If the relevant time periods overlap, one tree may be able to reuse the other's acceleration data (e.g., if the duration over which "other_vehicle" is considered "ahead" is a subset of the duration over which it is considered "nearby").

[0142] Referring to FIG. 4C, the rule activation logic 422 may be implemented based on looping over the iterables in the manner described above as the scenario run progresses. The DSL can be extended to implement looping over any predicate at any given time step. In this case, the first logical predicate defines the activation condition applicable for each agent. For example, the first predicate may define the notion of a "nearby" agent in terms of a distance threshold condition (e.g., satisfied by an agent that is within a threshold distance from the own agent), or the notion of an agent "ahead" as a set of appropriate conditions on the agent's location (e.g., satisfied by a single agent if that agent is (i) in front of the own agent, (ii) in the same lane as the own agent, and (iii) closer to the own agent than any other agent that satisfies conditions (i) and (ii)). The first logical predicate that defines the activation condition can be coded in the DSL in the same way as the rule itself. The rule tree can then be defined as above with the second logical predicate. This extends the DSL framework to incorporate looping over any predicate. Rules and activation conditions coded in the DSL using loops of the form "for any agent that satisfies predicate 1, evaluate predicate 2"; at each step of a scenario run, a set of agents (if any) that satisfy predicate 1 is constructed, and predicate 2 is evaluated only for members of that set. "Predicate 1" defines the activation condition of the rule per agent, and "Predicate 2" defines the rule tree itself. A time-varying iterable is constructed to keep track of which agents satisfy predicate 1 at any point over the duration of the scenario run, and can be passed down the rule tree as necessary to facilitate efficient rule evaluation.

[0143] Each rule and its activation conditions may be defined, for example, in first order logic.

[0144] Below is provided a section of code that uses the alternative syntax to define a custom rule graph (ALKS_01) as a temporal logic predicate.

[0145]

number

[0146] In the above example, LongitudinalDistance() and VelocityAlongRoadLateralAxis() are predefined extractor functions, while functions such as "and", Eventually(), Next(), and Always() are atomic assessor functions. The function AgentIsOnSameLane() is an assessor function that is applied directly to the scenario to determine if a given agent is in the same lane as the self agent.

[0147] Here, NearbyAgents() is a time-varying iterable that identifies other agents that meet a distance threshold to our agent. This is an example of a rule activation condition that applies between our agent and each other agent based on their distance from our agent.

[0148] While the above examples consider testing AV stacks, the techniques can be applied to test components of other forms of mobile robots. For example, other mobile robots are being developed to carry cargo in industrial areas both in and out of the country. Such mobile robots do not have people on board and belong to a class of mobile robots called UAVs (unmanned autonomous vehicles). Autonomous aerial mobile robots (drones) are also being developed.

[0149] The computer system comprises execution hardware that may be configured to execute the method / algorithm steps disclosed herein and / or to implement a model trained using the present technology. The term execution hardware encompasses any form / combination of hardware configured to execute the relevant method / algorithm steps. The execution hardware may take the form of one or more processors, which may be programmable or non-programmable, or a combination of programmable and non-programmable hardware may be used. Examples of suitable programmable processors include general purpose processors based on instruction set architectures such as CPUs, GPUs / accelerator processors, etc. Such general purpose processors typically execute computer readable instructions held in a memory coupled to or embedded in the processor, and perform the relevant steps in accordance with those instructions. Other forms of programmable processors include field programmable gate arrays (FPGAs) that have circuit configurations programmable through circuit description code. Examples of non-programmable processors include application specific integrated circuits (ASICs). Code, instructions, etc. may be stored in a transitory or non-transitory medium as appropriate (examples of the latter include solid state, magnetic and optical storage devices, etc.). The runtime stack subsystems 102-108 of Figure 1 may be implemented in a vehicle or in off-vehicle computing systems in a testing context, or on programmable or dedicated processors, or a combination of both. Various components of Figure 2, such as simulator 202 and test oracle 252, may likewise be implemented in programmable and / or dedicated hardware.

Claims

1. 1. A computer-implemented method for evaluating performance of a trajectory planner for a mobile robot in a real or simulated scenario, comprising: receiving a scenario ground truth for the scenario, the scenario ground truth being generated using the trajectory planner for controlling an own agent of the scenario in response to at least one scenario element of the scenario; receiving one or more performance evaluation rules for the scenario and at least one activation condition for each performance evaluation rule; processing the scenario ground truth by a test oracle to determine whether the activation condition of each performance evaluation rule is satisfied across multiple time steps of the scenario, each performance evaluation rule being evaluated by the test oracle to provide at least one test result only if the activation condition is satisfied; A computer-implemented method in which the scenario ground truth is processed to determine whether the activation condition of each performance evaluation rule is satisfied for each scenario element of a set of multiple scenario elements across multiple time steps of the scenario, and each performance evaluation rule is evaluated only if its activation condition is satisfied for at least one of the scenario elements and only between the own agent and the scenario element for which the activation condition is satisfied.

2. 2. The method of claim 1, wherein each performance evaluation rule is coded as a second logical predicate in a portion of rule creation code and its activation condition is coded as a first logical predicate in said portion of rule creation code, and at each time step, the test oracle evaluates the first logical predicate for each scenario element and evaluates the second logical predicate only between the own agent and any scenario element that satisfies the first logical predicate.

3. 3. The method of claim 1 or 2, wherein a plurality of performance evaluation rules having different respective activation conditions are received and selectively evaluated by the test oracle according to their different respective activation conditions.

4. The method of any one of claims 1 to 3, wherein each performance evaluation rule relates to driving performance.

5. Rendering the results for each of the plurality of time steps in the time series on a graphical user interface (GUI), the results for each time step comprising: a first category in which the activation condition is not met; and a second category where the activation condition is met and the rule is passed; and a third category if the activation condition is met and the rule is failed.

6. The method of claim 5 , wherein the results are rendered as one of at least three different colors corresponding to the at least three categories.

7. The method according to any one of claims 1 to 6, wherein the activation condition of a first one of the performance evaluation rules depends on the activation condition of at least a second one of the performance evaluation rules.

8. The method of claim 7 , wherein if the second performance evaluation rule is active, the first performance evaluation rule is deactivated.

9. The method of claim 8 , wherein the second performance evaluation rule relates to safety and the first performance evaluation rule relates to comfort.

10. A method according to any preceding claim, wherein the scenario element comprises one or more other agents.

11. The method of claim 10 , wherein the set of scenario elements is a set of other agents.

12. A method according to claim 10 or 11 when dependent on claim 2, wherein the activation conditions are evaluated for each scenario element and the performance evaluation rules are evaluated by looping over the iterable at each time step to compute at each time step an iterable containing identifiers of any scenario elements for which the activation conditions are satisfied.

13. 13. The method of claim 12, wherein the performance evaluation rules are defined as a computation graph applied to one or more signals extracted from the scenario ground truth, and the iterable is passed through the computation graph to evaluate the rules between the self agent and any scenario elements that satisfy the activation conditions.

14. A computer system comprising one or more computers configured to implement the method according to any one of claims 1 to 13.

15. Executable program instructions for programming a computer system to implement a method according to any one of claims 1 to 13.

Images