Support Tools for Autonomous Vehicle Testing

Abstract

1. A computer system for testing a real-time perception system for a sensor-equipped vehicle, the computer system comprising: an input configured to receive data of a real-world driving run performed by the sensor-equipped vehicle comprising a time series of sensor data and a time series of at least one associated run-time recognition output extracted by the perception system from the time series of sensor data; a rendering component configured to render a GUI comprising a recognition error timeline having a visual indication of recognition errors occurring at each time step of the run; a ground truthing pipeline configured to process at least one of the sensor data and the run-time recognition output by applying a non-real-time and / or non-causal recognition algorithm to the at least one of the sensor data and the run-time recognition output to extract a ground truth recognition output; and a recognition oracle configured to compare the run-time recognition output to the ground truth recognition output and identify recognition errors that occurred to generate the recognition error timeline.

Description

[Technical Field]

[0001] This disclosure relates to tools and methods for evaluating the performance of autonomous vehicle systems and trajectory planners in real or simulated scenarios, and computer programs and systems for implementing them. Example applications include performance testing of ADS (Autonomous Driving Systems) and ADAS (Advanced Driver Assist Systems). [Background technology]

[0002] There has been significant and rapid development in the field of autonomous vehicles. An autonomous vehicle (AV) is a vehicle equipped with sensors and control systems that enable it to operate without human control of its behavior. An autonomous vehicle is equipped with sensors that enable it to perceive its physical environment, including, for example, cameras, radar, and lidar. An autonomous vehicle is equipped with a suitably programmed computer that can process data received from the sensors and make safe and predictable decisions based on the context perceived by the sensors. An autonomous vehicle can be fully autonomous (in that it is designed to operate without human supervision or intervention, at least in certain situations) or semi-autonomous. Semi-autonomous systems require various levels of human supervision and intervention, including advanced driver assistance systems and level 3 autonomous driving systems. There are various aspects to testing the behavior of the sensors and control systems onboard a particular autonomous vehicle or type of autonomous vehicle.

[0003] A "Level 5" vehicle is one that can operate fully autonomously in any situation, while always being guaranteed to meet a minimum safety level. Such a vehicle does not require any manual controls (steering wheel, pedals, etc.).

[0004] In contrast, Level 3 and Level 4 vehicles can operate fully autonomously, but only within certain defined circumstances (e.g., within a geofenced area). Level 3 vehicles must be equipped to autonomously handle any situation requiring immediate response (e.g., emergency braking), but a changing situation may trigger a "transition request" requiring the driver to take control of the vehicle within a limited timeframe. Level 4 vehicles have similar restrictions, but if the driver fails to respond within the required timeframe, Level 4 vehicles must also be capable of autonomously performing a "minimum risk maneuver" (MRM), i.e., taking appropriate measures to bring the vehicle to a safe state (e.g., slowing down and parking the vehicle). Level 2 vehicles require the driver to be ready to intervene at any time, and it is the driver's responsibility to intervene whenever the autonomous system is unable to respond appropriately. At level 2 automation, it is the driver's responsibility to decide when intervention is required, at levels 3 and 4 this responsibility shifts to the vehicle's autonomous system, and it is the vehicle that must alert the driver when intervention is required.

[0005] As levels of autonomy increase and more responsibilities are transferred from humans to machines, safety becomes an increasingly challenging challenge. In autonomous driving, the importance of guaranteed safety is recognized. Guaranteed safety does not necessarily imply zero accidents, but rather guarantees that a minimum level of safety is met in defined situations. It is generally believed that for autonomous driving to be feasible, this minimum level of safety must significantly exceed that of a human driver.

[0006] According to Shalev-Shwartz et al., "On a Formal Model of Safe and Scalable Self-driving Cars" (2017), arXiv:1708.06374 (RSS paper), which is incorporated herein by reference in its entirety, human drivers drive at a rate of 10 -6 Based on the assumption that autonomous driving systems would need to reduce this by at least three orders of magnitude, the RSS paper estimates that this would result in an average of 10 fatal accidents per hour. -9 They conclude that a minimum safety level of the order of 1000 times the number of severe accidents must be guaranteed, and therefore point out that a purely data-driven approach would require vast amounts of driving data to be collected every time a change is made to the AV system's software or hardware.

[0007] The RSS paper provides a model-based approach to guaranteed safety. A rule-based Responsibility-Sensitive Safety (RSS) model is constructed by formalizing a small number of "common sense" driving rules: "1. Don't hit someone from behind. 2. Do not interrupt unnecessarily. 3. The right of way is given, not taken. 4. Be careful in places with poor visibility. 5. If you can avoid an accident without causing another one, you should do so. The RSS model is proven safe in the sense that if all agents always follow the rules of the RSS model, no accidents will occur. The aim is to reduce by several orders of magnitude the amount of driving data that needs to be collected to demonstrate the required level of safety.

[0008] A safety model (e.g., RSS) can be used as a basis for assessing the quality of trajectories planned or realized by an autonomous agent in real or simulated scenarios under the control of an autonomous system (stack). The stack is tested by exposing it to various scenarios and evaluating the resulting trajectories for compliance with the rules of the safety model (rule-based testing). A rule-based testing approach can also be applied to other aspects of performance, such as comfort or progress toward a defined goal. Summary of the Invention [Problem to be solved by the invention]

[0009] Techniques are described that enable experts to assess both the perception errors and driving performance of an AV system. Evaluating the perception output of an AV's perception system by comparing it to ground truth perception outputs allows experts to assess the impact of perception challenges on the overall performance of a given AV system. A UI is described herein that presents perception errors and driving performance in a single visualization, providing a correlation between perception and driving performance and helping experts identify sources of perception errors that may affect overall driving performance. [Means for solving the problem]

[0010] A first aspect herein is a computer system for testing a real-time recognition system, the real-time recognition system being for deployment on a vehicle equipped with sensors, the computer system comprising: at least one input configured to receive data of at least one real-world driving run performed by a sensor-equipped vehicle, the data comprising: (i) a time series of sensor data captured by the sensor-equipped vehicle; and (ii) a time series of at least one associated driving-time perception output extracted from the sensor data time series by a real-time perception system under test; a rendering component configured to generate rendering data for rendering a graphical user interface (GUI), the graphical user interface comprising, for each of a plurality of time steps of at least one real-world driving run, a recognition error timeline having a visual indication of recognition errors occurring at the time step; a ground truthing pipeline configured to process at least one of (i) the time series of sensor data and (ii) the time series of drive-time recognition outputs by applying at least one non-real-time and / or non-causal recognition algorithm to the at least one to extract at least one time series of ground truth recognition outputs ("pseudo-ground truth") for comparison with the drive-time recognition outputs; a recognition oracle configured to identify recognition errors occurring in one or more time intervals to generate a recognition error timeline by comparing a time series of run-time recognition outputs with a time series of ground truth recognition outputs; This invention relates to a computer system comprising:

[0011] In an embodiment, a recognition error may be identified by calculating a numerical error value between a time series of running recognition outputs and a time series of ground truth recognition outputs and comparing the numerical error value to at least one recognition error threshold.

[0012] For example, a numerical error value may be identified as a recognition error only if the numerical error value exceeds an error threshold.

[0013] The error threshold may be fixed or variable, for example, different recognition error thresholds may be applied to different actors / agents or different types thereof (e.g., different thresholds for vehicles vs. pedestrians).

[0014] The error threshold may be adjustable or otherwise configurable, for example, via a GUI or via rule definition instructions (e.g., coded in a Domain-Specific Language (DSL)) provided to the recognition oracle. A rule editor may be provided to code the rule definition instructions in a DSL in the form of a recognition error specification. The latter approach provides what is referred to herein as a "recognition error framework."

[0015] The error threshold may also be varied depending on one or more scene variables (drive variables) of the driving drive, e.g., variables of the object to which the error threshold applies. For example, for a given object (e.g., an agent or a static object), the recognition error threshold of that object may increase with the distance between the object and the own agent (based on the fact that smaller recognition errors are more significant for nearby objects). The same effect can be achieved using a fixed threshold, except that the numerical error value is weighted depending on the scene variable (e.g., weighted by the inverse of the distance). In this specification, unless otherwise indicated, reference to a "variable threshold" encompasses the latter implementation.

[0016] The (weighted) numerical recognition error may be normalized, i.e., converted to a predetermined scale, e.g., the range [-1, 1] with the fail threshold set to zero, optionally with a fixed error threshold. The normalized recognition error is sometimes called a recognition "robustness" score.

[0017] The weighting criteria / variable thresholds may be configurable, for example, via a GUI or DSL.

[0018] In addition to the identified recognition errors, the (normalized) error values ​​may be made accessible via the GUI.

[0019] More complex rules can be applied to identify recognition errors based on one or more error thresholds, for example by mapping multiple recognition error values ​​or combinations thereof.

[0020] "Recognition error" can be a binary indicator of recognition error (error / no error), or a non-binary categorical indicator (e.g., a "traffic light" style classification of red, green, and blue).

[0021] The recognition error can be, for example, the number of recognition errors aggregated across multiple objects and / or sensors and / or sensor modalities.

[0022] For example, recognition error rules may be defined hierarchically. For example, multiple sensors and / or sensor modalities (e.g., lidar, radar, camera, etc.) and / or multiple objects may be used to extract aggregate recognition errors aggregated across multiple modalities / objects. In this case, multiple recognition error timelines may be derived, for example, by applying predetermined rules to "lower-level" timelines (e.g., those relating to specific objects, sensors, and / or sensor modalities) to populate a "top-level" aggregate timeline. The top-level timeline may be expandable to view the lower-level timelines. Recognition errors may be aggregated over a time window to provide a "zoomed-out" view of the driving trip.

[0023] The recognition oracle may be configured to filter at least one time interval of the trip, which time interval is omitted from the recognition error timeline, and the filtering may be performed based on one or more filtering criteria applied to the recognition errors (e.g., to filter time intervals in which no recognition errors occurred) and / or one or more tags / labels associated with the real-world driving trip (e.g., to include only intervals in which a particular type of scene element is present, such as a vulnerable road user). For example, the tags may include ontology tags related to dynamic and / or static scene elements or conditions (e.g., actors, weather, lighting, etc.). Such filtering may also be referred to as "slicing" the timeline.

[0024] A timeline may aggregate multiple driving trips, and slicing is a useful tool in this context as a way to reduce the scope of "uninteresting" information displayed in the timeline.

[0025] The tags may be accessible via a GUI.

[0026] A schematic representation of the driving drive may be displayed on the GUI. A static representation may display a static snapshot of the driving drive at the current time step, with the current time step selectable via a command to the GUI. As the current time step is changed, a visual indicator may change to mark the current time step on the perception error timeline. Along with the schematic representation, (raw) data of at least one real-world driving drive may also be displayed. For example, a schematic top-down view may be displayed overlaid with at least one 3D point cloud (e.g., lidar, radar, or mono / stereo depth point cloud, or any combination / aggregation thereof) of the real-world driving drive. Alternatively or additionally, at least one captured image from one real-world driving drive for the current time step may be displayed (changing the current time step causes the GUI to update accordingly with the corresponding image).

[0027] A schematic representation of the driving journey may be rendered using a time series of drive-time recognition outputs, which may include, for example, a time series of ground truth bounding boxes (position, pose, size) for each of a plurality of detected objects and each object's identified object type, which are used to render a visual icon of that object on a known road layout (e.g., derived from a map) of the driving journey.

[0028] The time series of run-time recognition output may also be displayed via a GUI for visual comparison with the ground truth recognition output. For example, the time series of run-time recognition output may be overlaid on a schematic representation derived from the latter. For example, the run-time recognition output may include multiple time series of detected real-time bounding boxes, and a subset of the run-time bounding boxes associated with the current time step may be overlaid on a snapshot of the current time step.

[0029] The perception ground truth may be in the form of a trajectory (trace) for each agent (self and / or other agents), where a trajectory is a time sequence of spatial and motion states (e.g., bounding boxes and detected velocity or other motion vectors).

[0030] The extracted trajectory may be used to visualize the run in a GUI.

[0031] The GUI may provide the option to dynamically "replay" the scenario, with a video indicator moving along a recognition error timeline as the scenario progresses.

[0032] A second driving performance timeline may also be displayed on the GUI, conveying the results of a driving performance assessment applied to the same ground truth recognition output (e.g., trajectory). For example, a test oracle may be provided for this purpose.

[0033] Driving data may include two or more of multiple sensor modalities, for example, lidar, radar, and imagery (e.g., depth data from stereo or mono imaging).

[0034] In some embodiments, one sensor modality (or combination of sensor modalities) may be used to provide ground truth for other sensor modalities (or combinations of sensor modalities). For example, more accurate lidar may be used to derive a pseudo-ground truth that is used as a baseline for detection results or other recognition outputs derived from radar or imagery (mono or stereo) data.

[0035] A relatively small amount of manually labeled ground truth may be used within the system, for example as a pseudo-ground truth or a baseline for validating or measuring the accuracy of the driving recognition output.

[0036] While the above considered recognition errors derived from pseudo-ground truth, in other aspects of the invention, the above GUI can be used to render recognition errors derived in other ways, including those derived from real-world data without the use of pseudo-ground truth, and recognition errors from simulated driving runs generated in a simulator. In the case of simulated runs, the above discussion applies equally to ground truth provided directly by the simulator (without the need for a ground truthing pipeline) and to scene variables from the simulated run.

[0037] A second aspect herein is a computer system for assessing performance of an autonomous vehicle, comprising: at least one input configured to receive performance data for at least one autonomous driving journey, the performance data including a time series of at least one perception error and a time series of at least one driving performance outcome; a rendering component configured to generate rendering data for rendering a graphical user interface, the graphical user interface being for visualizing the performance data; (i) the recognition error timeline, and (ii) driving assessment timeline; a rendering component comprising: Equipped with The timelines are temporally aligned and divided into a plurality of time steps of at least one driving trip, and for each time step, the recognition error timeline includes a visual indication of whether a recognition error occurred at that time step, and the driving assessment timeline includes a visual indication of driving performance at that time step. Providing computer systems.

[0038] The driving assessment timeline and the recognition error timeline may be parallel to each other.

[0039] The above tools visually link driving performance to perception errors to help experts determine when ADS / ADAS performance is poor or unacceptable. For example, by focusing on an area of ​​the driving performance timeline where a critical driving rule failure occurred, experts can view the perception error timeline for the same moment to see whether perception errors may have contributed to that rule failure.

[0040] In an embodiment, driving performance may be assessed with respect to one or more predefined driving rules.

[0041] The driving performance timeline may aggregate driving performance across multiple individual driving rules and may be expandable to view the driving performance timeline for each individual driving rule.

[0042] The driving performance (or each driving performance) may be expandable to view a computational graph representation of the rules (as described below).

[0043] The driving journey may be a real-world journey, with driving rules being applied to the real-world trajectory.

[0044] In some cases, a ground truthing pipeline may be used to extract (pseudo) ground truth trajectories / recognition outputs, which are used to determine recognition errors and assess performance with respect to driving rules (similar to the first aspect above).

[0045] Alternatively, recognition errors may be identified without the use of pseudo-ground truth. For example, such errors may be identified from "flickering" objects (which appear / disappear when run-time object detectors fail) or "jumping" objects (which appear to jump within the scene in a kinematically infeasible way; e.g., a run-time detector may "swap" two nearby objects at some point in the run).

[0046] The performance data may include a time series of at least one numerical recognition score indicative of a recognition area of ​​interest, and the graphical user interface may comprise at least a corresponding timeline of the numerical recognition scores, and for each time step, the numerical recognition score timeline includes a visual indication of the numerical recognition score associated with that time step.

[0047] The time series of numerical recognition scores may be a time series of hardness scores that indicate a measure of difficulty for the recognition system at each time step.

[0048] The performance data may include a time series of at least one user-defined score, and the graphical user interface may comprise at least one corresponding custom timeline, where for each time step, the custom timeline includes a visual indication of the user-defined score assessed at that time step.

[0049] Alternatively, the run may be a simulated run and the recognition error may be simulated.

[0050] For example, one or more perception error (or perception performance) models may be used to sample perception errors, or more generally, to translate ground truth simulator states into more realistic perception errors that are provided to higher-level components in the stack under test during simulation.

[0051] As another example, synthetic sensor data may be generated in a simulation and processed by the stack's perception systems in the same way as real sensor data, in which case simulated perception errors can be derived in the same way as real-world perception errors (although in this case, no pipeline ground truth is required, as perception errors can be identified by comparison with ground truth inherent in the simulator).

[0052] Filters / slicing may also be applied to the timeline to, for example, present only periods around failures for a particular rule / rule combination. Thus, based on the rules applied to the driving performance timeline, the perception error timeline can be filtered / sliced, and vice versa.

[0053] The graphical user interface may include a progress bar aligned with the timeline, the progress bar having one or more markers indicating time intervals, each interval comprising one or more time steps of a driving journey, and a subset of the markers may be labeled with a numerical time indicator.

[0054] The graphical user interface may include a scrubber bar that spans the timeline and indicates a selected time step of the driving trip. In response to a user clicking a point on the timeline to select a new time step of the driving trip, the scrubber bar may move along the timeline so that the scrubber bar spans the timeline at the selected point.

[0055] The graphical user interface may include a zoom input that can be used to increase or decrease the number of time steps of a driving trip included in the timeline. The timeline may be configured such that using the zoom input to increase or decrease the number of time steps in the timeline causes a visual indicator of each time step to shrink or grow, respectively, so that the timeline maintains a constant length.

[0056] The progress bar may be configured such that when the zoom input is used to decrease the number of time steps in the timeline below a threshold, the markers adjust to indicate shorter time intervals, and when the zoom input is used to increase the number of time steps in the timeline above a threshold, the markers adjust to indicate longer time intervals.

[0057] Using the zoom input to adjust the number of time steps in the driving trip may adjust the timeline to include only time steps within a defined range from a reference point on the timeline. The reference point may be the start point of the driving trip. Alternatively, the reference point may be the currently selected time step of the driving trip. The currently selected point may be indicated by a scrubber bar.

[0058] The zoom input may include a zoom slider bar, which may be used to adjust the number of time steps in the timeline by moving an indicator along the slider bar. The indicator may be moved by clicking the slider and dragging it along the bar or by clicking a point on the slider to which the indicator should be moved. The zoom input may include a pinch gesture on a touchscreen, which adjusts the number of time steps in the timeline based on changes in the distance between two fingers touching the screen. Alternatively, the zoom input may include a mouse wheel, which adjusts the number of time steps in the timeline in response to the user rotating the wheel back and forth.

[0059] The timeline may be scrollable, with multiple time steps displayed on the timeline adjusted to shift forward or backward in time in response to user scrolling actions.

[0060] A portion of a driving trip may be selected by clicking a first point on the progress bar indicating the start time of the portion and dragging along the progress bar to a second point defining the end time of the portion. Driving data corresponding to the selected portion may be extracted and stored in a database.

[0061] The first aspect above refers to testing a real-time recognition system by comparing run-time recognition outputs with a set of derived (pseudo) ground truth recognition outputs. In other aspects, any of the above features of the embodiments may be applied more generally to evaluate any sequence of recognition outputs by comparison with a corresponding sequence of ground truth recognition outputs. In this context, ground truth may be any baseline that is deemed accurate for purposes of evaluating recognition outputs by comparison with the baseline.

[0062] A third aspect of the present specification is at least one input configured to receive data relating to at least one driving run, the data including (i) a time series of first recognition outputs and (ii) a time series of second ground truth recognition outputs, the time series of ground truth recognition outputs and the time series of drive-time recognition outputs relating to at least one time interval; a rendering component configured to generate rendering data for rendering a graphical user interface (GUI), the graphical user interface comprising, for each of a plurality of time steps of at least one driving journey, a recognition error timeline having a visual indication of recognition errors occurring at that time step; a recognition oracle configured to identify recognition errors occurring in one or more time intervals to generate a recognition error timeline by comparing a time series of recognition outputs with a time series of ground truth recognition outputs; This invention relates to a computer system comprising:

[0063] Note that the term "recognition output" is used broadly in this context and includes recognition data obtained from human annotations as well as the output of a vehicle's recognition stack.

[0064] The computer system may further include a ground truthing pipeline. The ground truthing pipeline may be configured to apply at least one non-real-time and / or acausal recognition algorithm to data from at least one driving run to process the data, the data including a time series of sensor data from the driving run and a time series of associated drive-time recognition outputs extracted from the time series of sensor data by the recognition system, to generate a first time series of recognition outputs. The ground truth recognition outputs may be generated by manual annotation of the at least one driving run. In this embodiment, the recognition outputs generated by the recognition system are "pseudo" ground truth recognition outputs, which may be compared with manually annotated ground truth recognition outputs received for the same driving run to identify recognition errors in the pseudo ground truth recognition outputs. This comparison may be used as a method for evaluating the suitability of the pseudo ground truth recognition outputs obtained from the ground truthing pipeline to be used as ground truth for comparison with a set of other recognition outputs to be evaluated. This comparison may be based on only a subset of manually annotated driving data, allowing pseudo-GT to be used to assess recognition output for larger data sets where human annotation is not available.

[0065] Alternatively, the recognition system may comprise a real-time recognition system for deployment in a sensor-equipped vehicle, and the recognition output may include a time series of run-time recognition outputs extracted by the real-time recognition system from the time series of sensor data for a given driving trip. The ground truth recognition output may be generated by processing at least one of the time series of sensor data or the time series of run-time recognition outputs by applying at least one non-real-time and / or non-causal recognition algorithm to at least one of the time series of sensor data or the time series of run-time recognition outputs by a ground truthing pipeline. Alternatively, the ground truth recognition output may be generated by manual annotation of the driving trip.

[0066] The driving journey may be a real-world driving journey.

[0067] Alternatively, the driving run may be a simulated driving run, where the sensor data is generated by a simulator and the driving recognition output may be obtained by applying a real-time recognition system to the simulated sensor data. Ground truth recognition output may be obtained directly from the simulator for comparison with the driving recognition output.

[0068] A further aspect herein is a computer-implemented method for testing a real-time recognition system, the real-time recognition system being for deployment on a sensor-equipped vehicle, the method comprising: receiving at input data of at least one real-world driving journey performed by a sensor-equipped vehicle, the data comprising: (i) a time series of sensor data captured by the sensor-equipped vehicle; and (ii) a time series of at least one associated driving-time perception output extracted from the time series of sensor data by a real-time perception system under test; generating, by a rendering component, rendering data for rendering a graphical user interface (GUI) comprising a perception error timeline, the perception error timeline having, for each of a plurality of time steps of at least one real-world driving run, a visual indication of perception errors occurring at the time step; processing at least one of (i) the sensor data time series and (ii) the drive-time recognition output time series in a ground truthing pipeline by applying at least one non-real-time and / or non-causal recognition algorithm to the at least one to extract at least one ground truth recognition output time series for comparison with the drive-time recognition output; identifying recognition errors that occurred in one or more time intervals to generate a recognition error timeline by comparing the time series of run-time recognition outputs with the time series of ground truth recognition outputs; A computer-implemented method is provided, comprising:

[0069] A further aspect provides executable program instructions for programming a computer system to implement any of the methods described herein.

[0070] For a better understanding of the present disclosure and to show how embodiments thereof may be put into effect, reference will now be made, by way of example only, to the following figures: [Brief explanation of the drawings]

[0071] [Figure 1] Figure 10 shows a set of use cases for the recognition error specification. [Figure 2A] FIG. 1 is a schematic functional block diagram of an autonomous vehicle stack. [Figure 2B] Schematic representation of the autonomous vehicle testing paradigm. [Figure 2C] FIG. 1 is a schematic block diagram of a scenario extraction pipeline. [Figure 3] FIG. 1 illustrates the user interface for reviewing manually tagged driving trips. [Figure 4A] Figure 1 shows an overview of the ground truthing pipeline. [Figure 4B] A set of noisy and refined bounding boxes. [Figure 5A] Figure 1 illustrates a set of detection result refinement techniques used in the ground truthing pipeline. [Figure 5B] Figure 1 illustrates a set of offline detection techniques used in the ground truthing pipeline. [Figure 6A] A schematic block diagram of the test pipeline. [Figure 6B] Figure 10 shows further details of a possible implementation of the test pipeline. [Figure 7A] FIG. 10 shows an example of a rule tree that is evaluated within a test oracle. [Figure 7B] FIG. 10 illustrates an example output of a node in a rule tree. [Figure 8A] FIG. 10 shows an example of a rule tree that is evaluated within a test oracle. [Figure 8B] Figure 1 shows a second example of a rule tree evaluated on a set of scenario ground truth data. [Figure 8C] Figure 1 illustrates how rules can be selectively applied within a test oracle. [Figure 9A] FIG. 1 is a schematic block diagram of a visualization component for rendering a graphical user interface. [Figure 9B-9D] FIG. 1 illustrates the various views available within the graphical user interface. [Figure 10A] FIG. 1 illustrates a first instance of an interruption scenario. [Figure 10B]FIG. 10 illustrates an example oracle output for the first scenario instance. [Figure 10C] FIG. 10 illustrates a second instance of the interruption scenario. [Figure 10D] FIG. 10 illustrates an example oracle output for the second scenario instance. [Figure 11] FIG. 1 illustrates an exemplary architecture for evaluating recognition errors. [Figure 12A] FIG. 1 illustrates an exemplary graphical user interface of a triage tool. [Figure 12B] 1 is a schematic representation of a driving scenario including sensor data displayed in a graphical user interface. [Figure 12C] FIG. 10 illustrates an exemplary user interface with zoom functionality and a timeline scrubber. [Figure 12D] Figure 10 illustrates the selection of a scenario subsection in the user interface. [Figure 13] A focused view of the graphical user interface showing the recognition rules. [Figure 14] FIG. 1 illustrates an exemplary rule definition within the recognition error framework. [Figure 15] FIG. 10 shows an exemplary graph of numerical scores for recognition errors with defined error thresholds. [Figure 16] FIG. 1 illustrates how a single perception error specification can be applied to real and simulated driving scenarios. [Figure 17] FIG. 10 illustrates how a defined recognition error specification can be used in testing the recognition stack and the planning stack. [Figures 18A-18B] Figure 1 shows the filtering tools applied to identify scenario related errors. [Figure 19A]FIG. 10 illustrates how the error threshold can be adjusted via the graphical user interface. [Figure 19B] FIG. 1 illustrates the selection and analysis of a "slice" of a driving scenario. DETAILED DESCRIPTION OF THE INVENTION

[0072] FIG. 11 shows an exemplary architecture in which a “Perception Oracle” 1108 receives perception error data from multiple sources (real and / or simulated) and uses that data to populate a “Perception Triage” graphical user interface (GUI) 500.

[0073] The test oracle 252 assesses driving performance, and certain implementations of GUI 500 allow for driving performance assessment coupled with cognitive information on each timeline.

[0074] The specific recognition errors may be derived from ground truth trajectories of real or simulated drives, and those same ground truth trajectories are used by the test oracle to assess driving performance.

[0075] The test oracle 252 and the recognition oracle 1108 mirror each other insofar as each applies configurable rule-based logic to populate the timeline on the GUI 500. The former applies a hierarchical rule tree to (pseudo) ground truth trajectories to assess driving performance over a drive (or multiple drives), while the latter applies similar logic to identify notable recognition errors. The rendering component 1120 generates rendering data for rendering the GUI on a display.

[0076] Our co-pending international patent applications PCT / EP2022 / 053406 and PCT / EP2022 / 053413, which are incorporated herein by reference, describe a domain-specific language (DSL) for encoding rules in a test oracle. An extension of the DSL for encoding rules for identifying significant recognition errors in a recognition oracle is described below.

[0077] The described embodiments provide a test pipeline to facilitate rule-based testing of mobile robot stacks in real or simulated scenarios, which incorporates additional functionality to identify and communicate the presence of perception errors in a flexible manner.

[0078] Typically, a "full" stack includes everything from processing and interpreting lower-level sensor data (perception) to inputs to primary higher-level functions such as prediction and planning, as well as control logic for generating appropriate control signals to implement planning-level decisions (e.g., to control braking, steering, acceleration, etc.). For autonomous vehicles, a level 3 stack includes logic for implementing transition requests, and a level 4 stack additionally includes logic for implementing minimum-risk maneuvers. Stacks may also implement secondary control functions, e.g., signals, headlights, windscreen wipers, etc.

[0079] The term "stack" may also refer to individual subsystems (sub-stacks) of a full stack, such as a perception, prediction, planning, or control stack, which may be tested individually or in any desired combination. A stack may also refer purely to software, i.e., one or more computer programs capable of running on one or more general-purpose computer processors.

[0080] The testing framework described below provides a pipeline for generating scenario ground truth from real-world data, which may then be used as the basis for recognition testing by comparing the generated ground truth with the recognition output of the recognition stack under test, as well as assessing driving behavior against driving rules.

[0081] The behavior of agents (actors) in real or simulated scenarios is evaluated by a test oracle based on defined performance evaluation rules. Such rules may evaluate various aspects of safety. For example, a safety rule set may be defined to assess the performance of the stack against specific safety standards, regulations, or safety models (such as RSS), or a custom rule set may be defined to test any aspect of performance. The test pipeline is not limited to safety and can be used to test any aspect of performance, such as comfort or progress toward a defined goal. The rule editor allows performance evaluation rules to be defined or modified and passed to the test oracle.

[0082] Similarly, a vehicle's perception can be evaluated by a "recognition oracle" based on defined recognition rules. These may be defined within a recognition error specification that provides a standard format for defining recognition errors.

[0083] Figure 1 illustrates a set of possible use cases for the perception error framework. Defining rules in the perception error framework allows areas of interest in real-world driving scenarios to be highlighted to the user (1602), for example, by flagging these areas in a scenario replay presented in the user interface. This allows the user to review apparent errors in the perception stack and identify possible reasons for the errors, such as occlusions in the original sensor data. Evaluating such perception errors also allows a "contract" to be defined between the perception and planning components of the AV stack (1604), where perception performance requirements can be specified and a stack that meets these perception performance requirements promises to be safely planned. The integrated framework may be used to evaluate real perception errors from real-world driving scenarios as well as simulated errors (1606), either directly simulated using a perception error model or computed by applying the perception stack to simulated sensor data, such as a photorealistic simulation of camera images.

[0084] The ground truth determined by the pipeline can itself be evaluated within the same recognition error specification by comparing it to a "true" ground truth determined by manually reviewing and annotating scenarios according to defined rules (1608). Finally, the results of applying the recognition error testing framework can be used to guide a testing strategy for testing both the recognition and prediction subsystems of the stack (1610).

[0085] A scenario, whether real or simulated, requires an ego-agent to navigate within a real or modeled physical context. The ego-agent is a real or simulated mobile robot moving under the control of the stack under test. The physical context includes static and / or dynamic elements to which the stack under test is required to respond effectively. For example, the mobile robot may be a fully or semi-autonomous vehicle (ego-vehicle) under the control of the stack. The physical context may include a static road layout and a given set of environmental conditions (e.g., weather, time of day, lighting conditions, humidity, pollution / particle levels, etc.) that can be maintained or changed as the scenario progresses. An interactive scenario additionally includes one or more other agents ("external" agents, e.g., other vehicles, pedestrians, cyclists, animals, etc.).

[0086] The following example considers application to testing autonomous vehicles, however the principles apply to other forms of mobile robots as well.

[0087] Scenarios may be expressed or defined at various levels of abstraction. More abstract scenarios accommodate greater degrees of variation. For example, a "cut-in scenario" or a "lane-change scenario" are examples of highly abstract scenarios characterized by interesting operations or behaviors that accommodate many variations (e.g., various agent starting positions and speeds, road layouts, environmental conditions, etc.). A "scenario run" refers to a specific occurrence in which an agent navigates through a physical context, optionally in the presence of one or more other agents. For example, multiple runs of a cut-in or lane-change scenario can be conducted (in the real world and / or in a simulator) with different agent parameters (e.g., starting positions, speeds, etc.), different road layouts, different environmental conditions, and / or different stack configurations. The terms "run" and "instance" are used interchangeably in this context.

[0088] In the following example, stack performance is assessed, at least in part, by evaluating the own agent's behavior within a test oracle against a given set of performance evaluation rules over one or more runs. The rules are applied to the "ground truth" of a scenario run (or each scenario run), which generally simply means an adequate representation of the scenario run (including the own agent's behavior) that is deemed reliable for testing purposes. Ground truth is specific to simulation: the simulator computes a sequence of scenario states that, by definition, is a perfect, reliable representation of the simulated scenario run. While a "perfect" representation of a scenario run in the same sense does not exist in real-world scenario runs, a suitably informative ground truth can nevertheless be obtained in a number of ways, for example, based on manual annotation of on-board sensor data, automated / semi-automated annotation of such data (e.g., using offline / non-real-time processing), and / or the use of external sources of information (e.g., external sensors, maps, etc.).

[0089] The scenario ground truth typically includes the "trajectories" of the agent and, if applicable, any other (salient) agents. A trajectory is the history of an agent's position and movement across a scenario. There are many ways in which a trajectory can be represented. Trajectory data typically includes spatial and movement data of an agent within an environment. The term is used in conjunction with both real-world scenarios (with real-world trajectories) and simulated scenarios (with simulated trajectories). A trajectory is typically a record of the actual path achieved by an agent within a scenario. In terms of terminology, "trajectory" and "trajectory" may include the same or similar types of information (e.g., a sequence of spatial and movement states over time). The term trajectory is commonly used in planning contexts (and may refer to a future / predicted trajectory), while the term trajectory is commonly used in conjunction with past behavior in testing / evaluation contexts.

[0090] In a simulation context, a "scenario description" is provided as input to the simulator. For example, the scenario description may be coded using a scenario description language (SDL) or in any other format that can be used by the simulator. The scenario description is typically a more abstract representation of the scenario and can result in multiple simulated runs. Depending on the implementation, the scenario description may have one or more configurable parameters that can be changed to increase the degree of possible variations. The degree of abstraction and parameterization is a design choice. For example, the scenario description may code a fixed layout using parameterized environmental conditions (e.g., weather, lighting, etc.). However, further abstraction is possible using, for example, configurable road parameters (e.g., road curvature, lane configuration, etc.). The input to the simulator includes the scenario description along with a set of chosen parameter values ​​(if applicable). The latter is sometimes called the parameterization of the scenario. The configurable parameters define a parameter space (also called a scenario space), and the parameterization corresponds to a point in the parameter space. In this context, a "scenario instance" may refer to the instantiation of a scenario in a simulator based on the scenario description and (if applicable) the chosen parameterization.

[0091] For the sake of brevity, the term "scenario" may be used to refer to a scenario run as well as a scenario in the more abstract sense. The meaning of the term scenario will be clear from the context in which it is used.

[0092] Trajectory planning is a key feature in the context of this invention, and the terms "trajectory planner," "trajectory planning system," and "trajectory planning stack" may be used interchangeably herein to refer to one or more components capable of planning a mobile robot's trajectory into the future. Trajectory planning decisions ultimately determine the actual trajectory realized by the own agent (however, in some test contexts, this may be influenced by other factors, such as, for example, the implementation of those decisions in the control stack and the resulting real or modeled dynamic response of the own agent to the control signals).

[0093] A trajectory planner may be tested alone or in combination with one or more other systems (e.g., perception, prediction, and / or control). Within the full stack, planning generally refers to higher-level autonomous decision-making capabilities (e.g., trajectory planning), while control generally refers to lower-level generation of control signals to implement those autonomous decisions. However, in the context of performance testing, the term control is also used in a broader sense. To avoid doubt, when it is stated that a trajectory planner controls an agent in a simulation, it does not necessarily imply that a control system (in the narrower sense) is tested in combination with the trajectory planner.

[0094] An example AV stack: To provide context related to the described embodiments, further details of an exemplary form of an AV stack will now be described.

[0095] 2A shows a highly schematic block diagram of an AV runtime stack 100. Runtime stack 100 is shown to include a perception (sub)system 102, a prediction (sub)system 104, a planning (sub)system (planner) 106, and a control (sub)system (controller) 108. As noted above, the term (sub)stack is sometimes used to describe the aforementioned components 102-108.

[0096] In a real-world context, the perception system 102 receives sensor outputs from the AV's onboard sensor system 110 and uses those sensor outputs to detect external agents and measure their physical state, e.g., their position, velocity, acceleration, etc. The onboard sensor system 110 can take various forms but generally comprises a variety of sensors, such as image capture devices (cameras / optical sensors), lidar and / or radar units, satellite positioning sensors (e.g., GPS), and motion / inertial sensors (e.g., accelerometers, gyroscopes, etc.). The onboard sensor system 110 therefore provides rich sensor data from which detailed information can be extracted about the surrounding environment, as well as the state of the AV and any external actors (e.g., vehicles, pedestrians, bicyclists, etc.) within that environment. Typically, the sensor output includes sensor data from multiple sensor modalities, such as stereo imagery from one or more stereo optical sensors, lidar, radar, etc. The sensor data from multiple sensor modalities may be combined using filters, fusion components, etc.

[0097] The recognition system 102 typically comprises multiple recognition components that work together to interpret sensor outputs and thereby provide recognition outputs to the prediction system 104 .

[0098] In a simulation context, depending on the nature of the test, and in particular where the stack 100 is "sliced" for testing (see below), the in-vehicle sensor system 100 may or may not need to be modeled. At higher levels of slicing, simulated sensor data is not required, so complex sensor modeling is not necessary.

[0099] The perception output from perception system 102 is used by prediction system 104 to predict the future behavior of external actors (agents), such as other vehicles in the vicinity of the AV.

[0100] The predictions calculated by prediction system 104 are provided to planner 106, which uses the predictions to make autonomous driving decisions to be performed by the AV in a given driving scenario. The inputs received by planner 106 typically describe the drivable area and also capture the predicted movement of external agents (obstacles from the AV's perspective) within the drivable area. The drivable area can be determined using recognition output from perception system 102 in combination with map information, such as an HD (high-definition) map.

[0101] The core function of the planner 106 is to plan a trajectory (self-trajectory) for the AV, taking into account the predicted agent movements. This is sometimes called trajectory planning. The trajectory is planned to accomplish a desired goal within the scenario. The goal can be, for example, entering a roundabout and exiting at a desired exit, overtaking a preceding vehicle, or staying in the current lane at a target speed (lane following). The goal may be determined, for example, by an autonomous route planner (not shown).

[0102] The controller 108 executes the decisions made by the planner 106 by providing appropriate control signals to the AV's onboard actor systems 112. Specifically, the planner 106 plans a trajectory for the AV, and the controller 108 generates control signals to implement the planned trajectory. Typically, the planner 106 plans far into the future, allowing the planned trajectory to be only partially implemented at the control level, after which a new trajectory is planned by the planner 106. The actor systems 112 include "primary" vehicle systems such as braking, acceleration, and steering systems, as well as secondary systems (e.g., signals, wipers, headlights, etc.).

[0103] Note that there may be differences between the planned trajectory at a given moment and the actual trajectory followed by the ego-agent. A planning system typically operates over a sequence of planning steps, updating the planned trajectory at each planning step to account for changes in the scenario (or, more precisely, deviations from predicted changes) since the previous planning step. The planning system 106 may reason forward so that the planned trajectory at each planning step deviates from the next planning step. Thus, individual planned trajectories may not be fully realized. (When the planning system 106 is tested alone in a simulation, the ego-agent may simply follow the planned trajectory exactly up to the next planning step, but as noted above, in other real and simulated contexts, the planned trajectory may not be followed exactly up to the next planning step because the ego-agent's behavior may be influenced by other factors, such as the operation of the control system 108 and the real or modeled dynamics of the ego-vehicle.) In many testing contexts, what ultimately matters is the ego-agent's actual trajectory—specifically, whether the actual trajectory is safe, as well as other factors such as comfort and progress. However, the rule-based testing approach herein can also be applied to planned trajectories (even if those planned trajectories are not fully or accurately realized by the agent itself). For example, even if an agent's actual trajectory is deemed safe according to given safety rules, the instantaneous planned trajectory may not have been safe, revealing the fact that the planner 106 was considering an unsafe course of action, even if it did not lead to unsafe agent behavior within the scenario. The instantaneous planned trajectory constitutes one form of internal state that can be usefully evaluated in addition to the actual agent's behavior in the simulation. Other forms of internal stack state can be evaluated as well.

[0104] The example of FIG. 2A contemplates a relatively “modular” architecture with separable perception, prediction, planning, and control systems 102-108. The sub-stacks themselves may also be modular, with separable planning modules, for example, within planning system 106. For example, planning system 106 may comprise multiple trajectory planning modules that can be applied to different physical contexts (e.g., simple lane driving versus complex intersections or roundabouts). This is relevant to simulation testing for the reasons discussed above, as it allows components (e.g., planning system 106 or its individual planning modules) to be tested individually or in different combinations. To avoid confusion, in a modular stack architecture, the term stack may refer not only to the full stack but also to its individual sub-systems or modules.

[0105] The degree to which various stack functions are integrated or separable may vary significantly between different stack implementations, and in some stacks, certain aspects may be so tightly coupled that they are indistinguishable. For example, in other stacks, planning and control may be integrated (e.g., such stacks may perform planning directly in terms of control signals), while other stacks (e.g., those shown in FIG. 2A) may be designed in a way that makes a clear distinction between the two (e.g., performing planning in terms of trajectories and performing independent control optimization to determine how best to execute the planned trajectory at the control signal level). Similarly, in some stacks, prediction and planning may be more tightly coupled. In extreme cases, in so-called "end-to-end" operation, perception, prediction, planning, and control may be essentially inseparable. Unless otherwise specified, the terms perception, prediction, planning, and control as used herein do not imply any particular coupling or modularization of these aspects.

[0106] It will be understood that the term "stack" encompasses software, but can also encompass hardware. In simulation, the software of the stack may be tested on a "general-purpose," off-board computer system before ultimately being uploaded to the on-board computer system of a physical vehicle. However, in "hardware-in-the-loop" testing, testing may extend to the underlying hardware of the vehicle itself. For example, the stack software may run on an on-board computer system (or a replica thereof) coupled to a simulator for testing purposes. In this context, the stack under test extends to the underlying computer hardware of the vehicle. As another example, certain functions of the stack 100 (e.g., perception functions) may be implemented in dedicated hardware. In a simulation context, hardware-in-the-loop testing may include feeding synthetic sensor data to the perception components of dedicated hardware.

[0107] Exemplary testing paradigms: FIG. 2B shows a highly schematic overview of an autonomous vehicle testing paradigm. An ADS / ADAS stack 100, such as the type shown in FIG. 2A, is repeatedly tested and evaluated in simulation by running multiple scenario instances in a simulator 202 and evaluating the performance of the stack 100 (and / or its individual sub-stacks) with a test oracle 252. The output of the test oracle 252 is useful to an expert 122 (a team or individual), allowing the expert 122 to identify issues in the stack 100 and modify it to mitigate those issues (S124). The results also help the expert 122 select further scenarios for testing (S126), and the process continues by iteratively modifying, testing, and evaluating the performance of the stack 100 in simulation. The improved stack 100 is ultimately integrated into a real-world AV 101 equipped with a sensor system 110 and actor system 112 (S125). The improved stack 100 typically includes program instructions (software) that execute on one or more computer processors of the vehicle 101's onboard computer system (not shown). The improved stack software is uploaded to the AV 101 in step S125. Step S125 may also include modifications to the underlying vehicle hardware. Once installed on the AV 101, the improved stack 100 receives sensor data from the sensor system 110 and outputs control signals to the actor system 112. Real-world testing (S128) can be used in conjunction with simulation-based testing. For example, once an acceptable level of performance has been reached through the process of simulation testing and stack refinement, appropriate real-world scenarios may be selected (S130), and the AV 101's performance in those real-world scenarios may be captured and similarly evaluated in a test oracle 252.

[0108] Scenarios can be obtained for simulation purposes in a variety of ways, including manual coding. The system can also extract scenarios from real-world drives for simulation purposes, allowing real-world situations and variations thereof to be recreated within simulator 202.

[0109] FIG. 2C shows a highly schematic block diagram of the scenario extraction pipeline. Real-world driving data 140 is passed to a “ground truthing” pipeline 142 for the purpose of generating a scenario ground truth. Driving data 140 may include, for example, sensor data and / or recognition output captured / generated on one or more vehicles (which may be autonomous, human-driven, or a combination thereof) and / or data captured from other sources, such as external sensors (e.g., CCTV). The driving data is processed within the ground truthing pipeline 142 to generate appropriate ground truth 144 (trajectory and context data) for the real-world driving. As discussed, the ground truthing process can be based on manual annotation of the “raw” driving data 140, or the process can be fully automated (e.g., using offline recognition methods), or a combination of manual and automated ground truthing can be used. For example, 3D bounding boxes may be placed around vehicles and / or other agents captured in the drive data 140 to determine the spatial and motion states of their trajectories. A scenario extraction component 146 receives the scenario ground truth 144 and processes it to extract a more abstracted scenario description 148 that can be used for simulation purposes. The scenario description 148 is used by the simulator 202 to enable multiple simulated runs to be conducted. A simulated run is a variation of the original real-world run, with the degree of variation possible depending on the degree of abstraction. A ground truth 150 is provided for each simulated run.

[0110] The real-world scenario ground truth 144 and the simulated ground truth 150 may be processed by a cognitive triage tool 152 to evaluate the cognitive stack and / or by a test oracle 252 to assess the stack based on the ground truth 144 or the simulated ground truth 150.

[0111] In the current non-in-vehicle context, trajectories do not need to be extracted in real time (or, more precisely, trajectories do not need to be extracted to support real-time planning); rather, trajectories are extracted “offline.” Examples of offline perception algorithms include non-real-time and non-causal perception algorithms. Offline techniques are in contrast to “online” techniques that can be executable implemented within AV stack 100 to facilitate real-time planning / decision-making.

[0112] For example, it is possible to use non-real-time processing that cannot be performed online due to hardware or other practical constraints of the AV's on-board computer system. To extract a trajectory, for example, one or more non-real-time perception algorithms can be applied to the real-world driving data 140. The non-real-time perception algorithms may be algorithms that would be unable to operate in real time due to the computational or memory resources they require.

[0113] In this context, it is also possible to use "acausal" perception algorithms. Acausal algorithms may or may not be able to operate in real time at runtime, but in either case, they require future knowledge and therefore cannot be implemented in an online context. For example, perception algorithms that detect the state of an agent at a particular moment (e.g., position, pose, speed, etc.) based on subsequent data require future knowledge and therefore cannot support real-time planning within the stack 100 in an online context (unless they are constrained to operate with a short look-ahead window). For example, backward-pass filtering is an acausal algorithm that requires future knowledge, although it may be able to operate in real time.

[0114] The term "recognition" generally refers to techniques for recognizing structures in real-world data 140, such as 2D or 3D bounding box detection, position detection, pose detection, motion detection, etc. For example, trajectories may be extracted as time series of bounding boxes or other spatial states in 3D or 2D space (e.g., in a bird's-eye view reference frame) along with associated motion information (e.g., speed, acceleration, jerk, etc.).

[0115] Ground Truth Pipeline A problem when testing the real-world performance of an autonomous vehicle stack is that autonomous vehicles generate vast amounts of data. This data can later be used to analyze or evaluate the AV's performance in the real world. However, a potential challenge is locating relevant data within this footage and identifying events of interest that occurred during driving. One option is to manually parse the data and identify events of interest through human annotation. However, this can be costly.

[0116] FIG. 3 illustrates an example of manually tagging real-world driving data during a drive. An AV is equipped with sensors, such as cameras. Footage is collected by the cameras along the drive, as shown in example image 1202. In an example drive on a highway with a human driver, if the driver notices something of interest, the driver can provide a flag to the AV, which can then tag that frame in the data collected by the sensors. The image shows a visualization of the drive on a map 1200, with bubbles indicating points along the drive where the driver tagged something. In this example, each tagged point corresponds to a frame of the camera image, which is used to filter the data analyzed after the drive so that only tagged frames are later examined.

[0117] As shown in map 1200, there are large gaps between tagged frames in the driving path, and any data collected in these gaps is not tagged and therefore will not be used. By filtering the data using manual annotations by the vehicle's driver, subsequent analysis of the driving data is limited to only those events that a human driver or test engineer felt were significant enough to flag, or that were present enough time to flag. However, useful insights about vehicle performance may exist at other points in the remaining data, and it would be useful to determine automatic methods for more fully processing and evaluating driving performance. Furthermore, identifying more issues than manual tagging for the same amount of data provides the AV system with more opportunities to make improvements for the same amount of collected data.

[0118] A possible solution is to create a unified analytics pipeline that uses the same metrics to assess both scenario simulations and real-world driving. The first step is to extract driving trajectories from actually collected data. For example, the approximate location of the ego vehicle and the approximate locations of other agents can be estimated based on on-board detection results. However, on-board detection results are not perfect because of limited computing resources and because on-board detection works in real time, meaning that the data informing a given detection result is only what the sensor has observed up to that point in time. This means that detection results may be noisy and inaccurate.

[0119] Figure 4A shows how data is processed and refined within a data ingestion pipeline to determine a pseudo-ground truth 144 for a given set of real-world data. Note that "true" ground truth cannot be extracted from real-world data; the ground truth pipeline described herein provides an estimate of ground truth sufficient for evaluation. This pseudo-ground truth 144 is sometimes referred to herein simply as "ground truth."

[0120] The data ingestion pipeline (or "ingestion" tool) ingests recognition data 140 from a given stack, and optionally from any other data sources 1300, such as manual annotations, and refines the data to extract a pseudo-ground truth 144 of the real-world driving scenario captured in the data. As shown, sensor data and detection results from the vehicle are ingested, optionally along with additional inputs such as offline detection results or manual annotations. These are processed to apply offline detectors 1302 to the raw sensor data and / or refine 1304 detection results received from the vehicle's on-board recognition stack. The refined detection results are then output as the pseudo-ground truth 144 for the scenario. This may then be used as the basis for various use cases, including evaluating the ground truth against driving rules with a test oracle (described below), comparing vehicle detection results to the pseudo-ground truth to determine recognition errors, and extracting scenarios for simulation. Other metrics may be calculated for the input data, including a recognition "difficulty" score 1306, which can be applied, for example, to the detection results or the entire camera image, and indicates the difficulty the recognition stack will have in correctly handling the given data.

[0121] Figure 4B shows an example set of bounding boxes before and after refinement. In the example in Figure 4B, the top image shows a set of "unrefined" noisy 3D bounding boxes that define the vehicle's position and orientation at each time step; these bounding boxes represent the ground truth with noise added. While the illustrated example corresponds to bounding boxes with noise added, the same effect can be achieved when refining vehicle detection results from real-world driving stacks. As shown in Figure 4B, the bounding boxes contain noise, and both the position and orientation of the detected bounding boxes fluctuate over time due to recognition errors.

[0122] The refinement pipeline can use various methods to remove this noise. The bottom trajectory in Figure 4B shows a denoised pseudo-ground truth trajectory 144 of a vehicle. As shown, the vehicle's orientation and its position are consistent from frame to frame, forming a smooth driving trajectory. The multiple possible methods used by the pipeline to perform this smoothing will not be described in detail. However, the pipeline not only benefits from greater computing power than online detectors, allowing more accurate detectors to be used, but also benefits from using past and future detection results to smooth the trajectory. Because the real-world detection results collected from the vehicle operate in real time, they are based solely on past data. For example, if an object is partially occluded at time t but becomes fully visible by the car's sensors at time t+n, the offline refinement pipeline can use the detection results at time t+n to inform earlier detection results based on the partially occluded data, resulting in a more complete detection result overall.

[0123] Various types of offline detectors or detection refinement methods can be used. Figure 5A shows a table of possible detection refinement techniques, and Figure 5B shows a table of possible offline detectors that can be applied to sensor data to obtain improved detection results.

[0124] Various techniques are used to refine the detection results. One example is semantic keypoint detection applied to camera images. After refinement, the result is a stable detection with a well-sized cuboid that smoothly tracks the car, as shown in Figure 4B.

[0125] Reference is made to International Patent Publication No. 2021 / 013792, which is incorporated herein by reference. The above-cited document discloses a class of offline annotation methods that can be implemented within a ground truthing pipeline 400 to extract a pseudo-ground truth trajectory for each agent of interest. Trajectories are extracted by applying automated annotation techniques to annotate real-world driving data 140 with a sequence of refined 3D bounding boxes (in this case, the agent trajectory consists of the refined 3D boxes).

[0126] The method roughly works as follows: real-world driving data 140 consists of a sequence of frames, each frame consisting of a set of 3D structural points (e.g., a point cloud). Each agent of interest (self and / or other agents) is tracked as an object across multiple frames (this agent is a "common structural component" in the terminology of the above cited paper).

[0127] A "frame" in this context refers to any captured 3D structural representation, i.e., one that contains captured points (3D structural points) that define a structure in 3D space, providing an essentially static "snapshot" of the 3D structure captured in that frame (i.e., a static 3D scene). Although a frame may be said to correspond to a single moment in time, this does not necessarily imply that the frame or the underlying sensor data from which the frame is derived must be captured instantaneously; for example, lidar measurements may be captured in a lidar sweep by a moving vehicle over a short interval (e.g., about 100 ms) and then "untwisted" to account for the vehicle's motion to form a single point cloud. In this case, the single point cloud may still be said to correspond to a single moment in time.

[0128] Real-world driving data may consist of multiple frame sequences, e.g., two or more separate sequences of lidar, radar, and depth frames (a depth frame in this context refers to a 3D point cloud derived by depth imaging, such as stereo or monocular depth imaging). Frames may also consist of a fused point cloud, which is computed by fusing multiple point clouds from different sensors and / or different sensor modalities.

[0129] The method starts with an initial set of 3D bounding box estimates (coarse size / pose estimates) for each agent of interest and uses these estimates to build a 3D model of that agent from the frames themselves. Here, pose refers to the 6D pose (3D position and orientation in 3D space). The following example considers 3D model extraction specifically from lidar, but the discussion applies equally to other sensor modalities. Multiple modalities of sensor data can be used, e.g., coarse 3D boxes can be provided by one or a second sensor modality or multiple modalities (e.g., radar or depth imaging). For example, an initial coarse estimate can be computed by applying a 3D bounding box detector to the point cloud of a second modality (or multiple modalities). A coarse estimate can also be determined from the same sensor modality (lidar in this case), and subsequent processing techniques are used to refine this estimate. As another example, a real-time 3D box from a test recognition system 102 can be used as an initial rough estimate (e.g., calculated on-board the vehicle while driving in the real world). In the latter approach, this method can be described as a form of detection result refinement.

[0130] To create an aggregated 3D object model for each agent, points belonging to that object are aggregated across multiple frames by taking a subset of points contained in a coarse 3D bounding box for each frame (or the coarse 3D bounding box may be slightly enlarged to provide additional "room" for object point extraction). Roughly speaking, aggregation works by first transforming a subset of points from each frame into the agent's frame of reference. Because the agent's pose in each frame is only roughly known, the transformation to the agent's frame of reference is not precisely known at this point. The transformation is first estimated from the coarse 3D bounding box. For example, the transformation can be implemented efficiently by transforming the subset of points to align with the axes of the coarse 3D bounding box in each frame. Although the subsets of points from different frames mostly belong to the same object, they may be misaligned in the agent's frame of reference due to errors in the initial pose estimate. To correct for the misalignment, a registration method is used to align the two subsets of points. Such methods work roughly by transforming (rotating / translating) one subset of object points to align it with another using some form of matching algorithm (e.g., Iterative Closest Point). The matching uses the knowledge that most of the two subsets of points belong to the same object. This process can then be repeated across subsequent frames to build a dense 3D model of the object. Having built a dense 3D model in this way, noise points (those that do not belong to the object) can be filtered even more easily, as they are isolated from the actual object points.Next, by applying a 3D object detector to the dense, filtered 3D object model, a tightly fitting 3D bounding box of more accurate size can be determined for the agent in question (this assumes a rigid agent, where the size and shape of the 3D bounding box do not change from frame to frame; the only variables each frame are its position and orientation). Finally, the dense 3D model is matched with corresponding object points in each frame to pinpoint the location of a more accurate 3D bounding box in each frame, providing a refined 3D bounding box estimate (which forms part of the pseudo-ground truth) for each frame. This process can be repeated iteratively, whereby an initial 3D model is extracted, a pose is refined, the 3D object model is updated based on the refined pose, and so on.

[0131] The refined 3D bounding box serves as a pseudo-ground truth position state in determining the degree of recognition error in position-based recognition outputs (e.g., running boxes, pose estimates, etc.).

[0132] To incorporate kinematic information, a 3D bounding box may be jointly optimized with a 3D kinematic model. The kinematic model can provide the kinematic state (e.g., speed / velocity, acceleration, etc.) of the agent in question, which may be used as a pseudo-ground truth for motion detection results during running (e.g., speed / velocity, acceleration estimates, etc., calculated by the test subject's perception system 102). The kinematic model can facilitate a realistic (kinematically feasible) 3D box across frames. For example, the joint optimization can be formulated based on a cost function that penalizes discrepancies between the aggregate 3D model and points in each frame, while simultaneously penalizing kinematically infeasible changes in the agent pose between frames.

[0133] The motion model also allows the 3D box to be accurately located in frames containing object missed detections (i.e., when a coarse estimate is not available, which may occur if the coarse estimate is an on-vehicle detection result and the perception system under test 102 fails in a given frame) by interpolating the pose of the 3D agent between adjacent frames based on the motion model. This allows object missed detections to be identified within the perception triage tool 152.

[0134] The 3D model can be in the form of a set of points, or a surface model (e.g., a distance field) may be fitted to the points. International Patent Publication No. 2021 / 013791, incorporated herein by reference, discloses further details of a 3D object modeling technique in which the 3D surface of a 3D object model is encoded as a (signed) distance field fitted to the extracted points.

[0135] The application of these refinement techniques is that they can be used to obtain a pseudo-ground truth 144 of the agents of a scene, including the ego-vehicle and external agents, and the refined detection results can be treated as real trajectories taken by the agents in the scene. This may be used to assess how accurate the on-board recognition of a vehicle was by comparing the car detection results to the pseudo-ground truth. The pseudo-ground truth can also be used to see how the system under test (i.e., the ego-vehicle stack) drove in violation of highway rules.

[0136] The pseudo-ground truth detection results 144 can also be used to semantically tag and query the collected data. For example, a user can enter a query such as "Find all events with cut-ins," where a cut-in is when an agent moves in front of the ego vehicle in the ego vehicle's lane. Because the pseudo-ground truth contains the trajectories of all agents in the scene, including their positions and orientations at any given time, it is possible to identify cut-ins by searching the agent trajectories for instances where the agent moved in front of another vehicle in a lane. More complex queries can be constructed. For example, a user might enter the query "Find all cut-ins where the agent had a speed of at least x." Because the agent's movement is defined by the pseudo-ground truth trajectories extracted from the data, it is straightforward to search the refined detection results for instances of cut-ins where the agent exceeded a given speed. When these queries are selected and executed, they reduce the time required to manually analyze the data. This means that there is no need to rely on the driver to identify areas of interest in real time, but instead areas of interest can be automatically detected within the collected data, from which interesting scenarios can be extracted for further analysis. This allows more data to be used, and in some cases allows scenarios to be identified that may be overlooked by a human driver.

[0137] Test Pipeline: Next, further details of the test pipeline and test oracle 252 are described. The following example focuses on simulation-based testing. However, as noted above, the test oracle 252 can similarly be applied to evaluate stack performance in real-world scenarios, and the relevant discussion below applies equally to real-world scenarios. Specifically, the test pipeline described below may be used with the extracted ground truth 144 obtained from real-world data, as described in Figures 1-5. The application of the described test pipeline with a cognitive evaluation pipeline in a real-world data analysis tool is described in more detail below. The following discussion refers to the stack 100 of Figure 2A as an example. However, as noted above, the test pipeline 200 is highly flexible and can be applied to any stack or sub-stack operating at any autonomy level.

[0138] FIG. 6A shows a schematic block diagram of a test pipeline, represented by reference numeral 200. Test pipeline 200 is shown to include simulator 202 and test oracle 252. Simulator 202 runs simulated scenarios to test all or a portion of AV run-time stack 100, and test oracle 252 evaluates the performance of the stack (or sub-stack) in the simulated scenario. As discussed, only a sub-stack of the run-time stack may be tested, but for simplicity, the following description will refer to the (full) AV stack 100 throughout. However, this description equally applies to a sub-stack instead of the full stack 100. The term "slicing" is used herein to refer to the selection of a set or subset of stack components for testing.

[0139] As mentioned above, the idea of ​​simulation-based testing is to drive through simulated driving scenarios in which the own agent has to navigate under the control of the stack 100 under test. Typically, the scenario comprises a static drivable area (e.g., a particular static road layout) through which the own agent is required to navigate, typically in the presence of one or more other dynamic agents (e.g., other vehicles, cyclists, pedestrians, etc.). For this purpose, simulated inputs 203 are provided from a simulator 202 to the stack 100 under test.

[0140] The slicing of the stack dictates the form of the simulated input 203. As an example, FIG. 6A shows the prediction, planning, and control systems 104, 106, and 108 in the AV stack 100 under test. To test the full AV stack of FIG. 2A, the perception system 102 can also be applied during testing. In this case, the simulated input 203 includes synthetic sensor data generated using appropriate sensor models and processed within the perception system 102 similarly to real sensor data. This requires the generation of sufficiently realistic synthetic sensor inputs (e.g., photorealistic image data and / or similarly realistic simulated lidar / radar data). The resulting output of the perception system 102 is then fed to the higher-level prediction and planning systems 104, 106.

[0141] In contrast, so-called "planning-level" simulation essentially bypasses the cognitive system 102. Instead, the simulator 202 provides simpler, higher-level inputs 203 directly to the predictive system 104. In some contexts, it may even be appropriate to bypass the predictive system 104 as well, in order to test the planner 106 based on predictions derived directly from simulated scenarios (i.e., "perfect" predictions).

[0142] Between these extremes there is room for many different levels of input slicing, for example testing only a subset of the recognition system 102, e.g., "late" (higher-level) recognition components, e.g., components such as filters or fusion components that operate on the output from lower-level recognition components (e.g., object detectors, bounding box detectors, motion detectors, etc.).

[0143] Whatever form it takes, the simulated inputs 203 are used (directly or indirectly) as the basis for decision-making by the planner 108. The controller 108 then implements the planner's decisions by outputting control signals 109. In a real-world context, these control signals drive the AV's physical actor-system 112. The simulation uses an ego-vehicle dynamics model 204 to simulate the autonomous vehicle's physical response to the control signals 109 by translating the resulting control signals 109 into realistic movement of the ego-agent within the simulation.

[0144] Alternatively, a simpler form of simulation assumes that the ego-agent follows each planned trajectory exactly between planning steps. This approach bypasses the control system 108 (to the extent that it is separable from the plan) and removes the need for an ego-vehicle dynamics model 204. This may be sufficient to test certain aspects of the plan.

[0145] To the extent that an external agent exhibits autonomous behavior / decision-making within the simulator 202, some form of agent decision-making logic 210 is implemented to make those decisions and determine the agent's behavior within the scenario. The agent decision-making logic 210 may be of comparable complexity to the self-stack 100 itself, or may have more limited decision-making capabilities. The aim is to provide sufficiently realistic external agent behavior within the simulator 202 to allow useful testing of the self-stack 100's decision-making capabilities. In some contexts, this does not require agent decision-making logic 210 at all (open-loop simulation), while in other contexts, useful testing can be provided using relatively limited agent logic 210, such as basic adaptive cruise control (ACC). Where appropriate, one or more agent dynamics models 206 may be used to provide more realistic agent behavior.

[0146] A scenario is run according to the scenario's scenario description 201a and (if applicable) selected parameterization 201b. A scenario typically has both static and dynamic elements, which may be "hard coded" in the scenario description 201a or may be configurable and thus determined by the scenario description 201a in combination with the selected parameterization 201b. In a driving scenario, the static elements typically include the static road layout.

[0147] Dynamic elements typically include one or more external agents in the scenario, such as other vehicles, pedestrians, cyclists, etc.

[0148] The range of dynamic information provided to simulator 202 for each external agent can vary. For example, a scenario may be described by separable static and dynamic layers. A given static layer (e.g., defining a road layout) can be used in combination with various dynamic layers to provide different scenario instances. The dynamic layer may include, for each external agent, a spatial path followed by that agent, along with motion and / or behavior data associated with that path. In a simple open-loop simulation, the external actor simply follows the spatial path and motion data defined in the dynamic layer, which is non-reactive, i.e., does not react to the external agent within the simulation. Such an open-loop simulation can be implemented without agent decision logic 210. However, in a closed-loop simulation, the dynamic layer instead defines at least one behavior (e.g., ACC behavior) followed along the static path. In this case, agent decision logic 210 implements that behavior in a reactive manner within the simulation, i.e., reactively to the external agent and / or other external agents. Motion data may still be associated with the static path, but in this case it may be less prescriptive and, for example, may serve as a target along the path. For example, an ACC behavior may set a target speed along the route that the agent is attempting to match, but the agent decision logic 210 may allow the external agent to slow down below the target at any point along the route in order to maintain the target distance from the vehicle ahead.

[0149] As will be appreciated, a scenario can be described in many ways with any degree of configurability for simulation purposes. For example, the number and type of agents, as well as their movement information, may be configurable as part of the scenario parameterization 201b.

[0150] The output of simulator 202 for a given simulation includes a self-trajectory 212a for the own agent and one or more agent trajectories 212b (trajectories 212) for one or more foreign agents. Each trajectory 212a, 212b is a complete history of the agent's behavior within the simulation, having both spatial and kinematic components. For example, each trajectory 212a, 212b may take the form of a spatial path with kinematic data associated with points along the path, such as speed, acceleration, jerk (rate of change of acceleration), snap (rate of change of jerk), etc.

[0151] Additional information is also provided to supplement and provide context for the trajectory 212. Such additional information is referred to as “context” data 214. The context data 214 relates to the physical context of the scenario and can have both static components (e.g., road layout) and dynamic components (e.g., weather conditions, to the extent that they change throughout the simulation). Because the context data 214 is directly defined by the selection of the scenario description 201a or parameterization 201b, it may be somewhat “pass-through” in that it is not affected by the outcome of the simulation. For example, the context data 214 may include a static road layout directly driven by the scenario description 201a or parameterization 201b. However, typically, the context data 214 includes at least some elements derived within the simulator 202. This may include, for example, simulated environmental data, such as weather data, which the simulator 202 is free to change as the simulation progresses. In this case, the weather data may be time-dependent, and that time-dependency is reflected in the context data 214.

[0152] Test oracle 252 receives trajectories 212 and context data 214 and scores their outputs against a set of performance evaluation rules 254. Performance evaluation rules 254 are shown provided as inputs to test oracle 252.

[0153] Rules 254 are typically categorical (e.g., pass / fail type rules). A particular performance evaluation rule is also associated with a numeric performance metric (e.g., indicating a degree of achievement or failure, or other quantity that helps explain or is otherwise related to the categorical outcome) that is used to “score” the trajectory. Rule 254 evaluation is time-based; a given rule may have different results at different points in a scenario. Scoring is also time-based; for each performance evaluation metric, the test oracle 252 tracks how the value (score) of that metric changes over time as the simulation progresses. The test oracle 252 provides output 256, including a time sequence 256a of each rule’s categorical (e.g., pass / fail) results and a score-time plot 256b of each performance metric, as described in more detail below. The results and scores 256a, 256b are useful to the expert 122 and can be used to identify and mitigate performance issues within the tested stack 100. The test oracle 252 also provides an overall (aggregate) outcome for the scenario (e.g., overall pass / fail). The output 256 of the test oracle 252 is stored in a test database 258 in association with information about the scenario to which the output 256 pertains. For example, the output 256 may be stored in association with the scenario description 210a (or its identifier) ​​and the selected parameterization 201b. As with time-dependent outcomes and scores, an overall score may also be assigned to the scenario and stored as part of the output 256. For example, an aggregate score for each rule (e.g., overall pass / fail) and / or an aggregate outcome across all rules 254 (e.g., pass / fail).

[0154] Figure 6B shows another slicing option, using reference numerals 100 and 100S to represent the full stack and sub-stack, respectively. It is sub-stack 100S that is the subject of testing in test pipeline 200 of Figure 6A.

[0155] Several "late" recognition components 102B form part of the sub-stack 100S being tested and, during testing, are applied to simulated recognition input 203. Late recognition components 102B may include filtering or other fusion components that fuse recognition inputs from multiple early recognition components.

[0156] In the full stack 100, the late recognition component 102B receives actual recognition input 213 from the early recognition component 102A. For example, the early recognition component 102A may include one or more 2D or 3D bounding box detectors, in which case the simulated recognition input provided to the late recognition component may include simulated 2D or 3D bounding box detection results derived by ray tracing in a simulation. The early recognition component 102A generally includes components that operate directly on sensor data. In the slicing of FIG. 6B, the simulated recognition input 203 typically corresponds formally to the actual recognition input 213 provided by the early recognition component 102A. However, the early recognition component 102A is not applied as part of testing; instead, it is used to train one or more recognition error models 208, which can be used to introduce realistic errors in a statistically rigorous manner into the simulated recognition input 203 provided to the late recognition component 102B of the sub-stack 100 under test.

[0157] Such a recognition error model may be referred to as a Perception Statistical Performance Model (PSPM), or synonymously as "PRISM." Further details of the principles of PSPM, and suitable techniques for constructing and training a PSPM, can be found in International Patent Publications 2021037763, 2021037760, 2021037765, 2021037761, and 2021037766, each of which is incorporated herein by reference in its entirety. The idea behind a PSPM is to efficiently introduce realistic errors into the simulated recognition input provided to the sub-stack 100S (i.e., reflecting the kind of errors expected if the early recognition component 102A were applied in the real world). In a simulation context, "perfect" ground truth recognition inputs 203G are provided by the simulator, which are used to derive more realistic recognition inputs 203 with realistic errors introduced by the recognition error model 208.

[0158] As explained in the above-cited references, a PSPM can rely on one or more variables representing physical conditions ("confounders"), allowing different levels of error to be introduced to reflect different possible real-world conditions. Thus, simulator 202 can simulate different physical conditions (e.g., different weather conditions) simply by changing the values ​​of the meteorological confounders to vary how perception error is introduced.

[0159] The later recognition components 102b in the sub-stack 100S process the simulated recognition input 203 in exactly the same way as they process real-world recognition input 213 in the full stack 100, and their output drives prediction, planning, and control.

[0160] Alternatively, PRISM can be used to model the entire recognition system 102, including the late recognition component 102B, in which case PSPM is used to generate realistic recognition outputs that are passed directly as input to the prediction system 104.

[0161] Depending on the implementation, there may or may not be a deterministic relationship between a given scenario parameterization 201b and the outcome of a simulation for a given configuration of the stack 100 (i.e., the same parameterization may or may not always lead to the same outcome for the same stack 100). Nondeterminism can arise in various ways. For example, if the simulation is based on PRISM, PRISM may model a distribution of possible recognition outputs for each given time step of the scenario, from which realistic recognition outputs are stochastically sampled. This leads to nondeterministic behavior within the simulator 202, such that different recognition outputs are sampled and therefore different outcomes may be obtained for the same stack 100 and scenario parameterization. Alternatively or additionally, the simulator 202 may be inherently nondeterministic; for example, weather, lighting, or other environmental conditions may be randomized / stochastic to some extent within the simulator 202. It will be appreciated that this is a design choice, and in other implementations, various environmental conditions may instead be fully specified in the scenario parameterization 201b. In a non-deterministic simulation, multiple scenario instances can be run for each parameterization. For a particular parameterization 201b selection, an aggregate pass / fail outcome can be assigned, for example, as a count or percentage of pass / fail outcomes.

[0162] The test orchestration component 260 is responsible for selecting a scenario for simulation purposes. For example, the test orchestration component 260 may automatically select a scenario description 201 a and appropriate parameterization 201 b based on the test oracle output 256 from a previous scenario.

[0163] Test Oracle Rules: Performance evaluation rules 254 are constructed as computational graphs (rule trees) that are applied within a test oracle. Unless otherwise specified, the term "rule tree" herein refers to the computational graph configured to implement a given rule. Each rule is constructed as a rule tree, and a set of multiple rules may be referred to as a "forest" of multiple rule trees.

[0164] FIG. 7A shows an example of a rule tree 300 constructed from a combination of extractor nodes (leaf objects) 302 and assessor nodes (non-leaf objects) 304. Each extractor node 302 extracts a time-varying numeric (e.g., floating-point) signal (score) from a set of scenario data 310. The scenario data 310 is a form of scenario ground truth in the sense described above and may be referred to as such. The scenario data 310 was obtained by deploying a trajectory planner (e.g., planner 106 of FIG. 2A) in a real or simulated scenario and is shown to include self and agent trajectories 212 and context data 214. In the simulation context of FIG. 6 or FIG. 6A, the scenario ground truth 310 is provided as an output of the simulator 202.

[0165] Each assessor node 304 is shown to have at least one child object (node), which is either one of the extractor nodes 302 or another of the assessor nodes 304. Each assessor node receives outputs from its child nodes and applies an assessor function to those outputs. The output of the assessor function is a time series of categorical results. The following example considers simple binary pass / fail results, but the technique can be easily extended to non-binary results. Each assessor function evaluates the output of its child nodes against predefined atomic rules. Such rules can be flexibly combined depending on the desired safety model.

[0166] In addition, each assessor node 304 derives a time-varying numerical signal from the output of its child nodes, which is related to a categorical outcome by a threshold condition (see below).

[0167] The top-level root node 304a is an assessor node that is not a child node of any other node. The top-level node 304a outputs a sequence of final results, and its descendants (i.e., nodes that are direct or indirect children of the top-level node 304a) provide underlying signals and intermediate results.

[0168] 7B visually illustrates an example time series of derived signal 312 and corresponding results 314 calculated by assessor node 304. Results 314 correlate with derived signal 312 in that a pass result is returned if (and only if) the derived signal exceeds a fail threshold 316. As will be appreciated, this is just one example of a threshold condition relating a time sequence of results to a corresponding signal.

[0169] The signals extracted directly from the scenario ground truth 310 by the extractor nodes 302 may be referred to as "raw" signals to distinguish them from the "derived" signals calculated by the assessor nodes 304. The resulting raw / derived signals may be discretized in time.

[0170] FIG. 8A shows an example of a rule tree implemented within test platform 200.

[0171] A rule editor 400 is provided for constructing rules that are implemented in the test oracle 252. The rule editor 400 receives rule creation input from a user (who may or may not be an end user of the system). In this example, the rule creation input is coded in a domain-specific language (DSL) and defines at least one rule graph 408 that is implemented within the test oracle 252. In the example below, the rules are logical rules, with true and false representing pass and fail, respectively (it will be understood that this is purely a design choice).

[0172] The following examples consider rules formulated using combinations of atomic logical predicates. Examples of basic atomic predicates include elementary logic gates (OR, AND, etc.) and logical functions, such as "greater than", (Gt(a,b)), which returns true if a is greater than b, and false otherwise.

[0173] The Gt function implements the safe lateral distance rule between the self agent and other agents in the scenario (with the agent identifier "other_agent_id"). Two extractor nodes (latd, latsd) apply the LateralDistance and LateralSafeDistance extractor functions, respectively. These functions operate directly on the scenario ground truth 310 to extract a time-varying lateral distance signal (measuring the lateral distance between the self agent and the identified other agents) and a time-varying safe lateral distance signal for the self agent and the identified other agents, respectively. The safe lateral distance signal can depend on various factors, such as the speed of the self agent and the speed of the other agents (captured in the trajectory 212) and the environmental conditions (e.g., weather, lighting, road type, etc.) captured in the context data 214.

[0174] The assessor node (is_latd_safe) is the parent of the latd and latsd extractor nodes and is mapped to the Gt atomic predicate. Thus, when the rule tree 408 is implemented, the is_latd_safe assessor node applies the Gt function to the outputs of the latd and latsd extractor nodes to calculate a true / false result for each time step of the scenario, returning true for each time step in which the latd signal exceeds the latsd signal, and false otherwise. In this way, a "safe lateral distance" rule is constructed from the atomic extractor function and predicate, and if the lateral distance is at or below the safe lateral distance threshold, the ego fails the safe lateral distance rule. As can be seen, this is a very simple example of a rule tree. Rules of arbitrary complexity can be constructed following the same principles.

[0175] The test oracle 252 applies the rule tree 408 to the scenario ground truth 310 and provides the results via a user interface (UI) 418.

[0176] Figure 8B shows an example of a rule tree including a lateral distance branch corresponding to Figure 8A. Additionally, the rule tree includes a longitudinal distance branch and a top-level OR predicate (safe distance node, is_d_safe) to implement a safe distance metric. Similar to the lateral distance branch, the longitudinal distance branch extracts longitudinal distance and longitudinal distance threshold signals (extractor nodes lond and lonsd, respectively) from the scenario data. The longitudinal safety assessor node (is_lond_safe) returns true if the longitudinal distance exceeds the safe longitudinal distance threshold. The top-level OR node returns true if either or both of the lateral and longitudinal distances are safe (below the appropriate threshold) and false if neither is safe. In this context, it is sufficient for only one of the distances to exceed the safety threshold (e.g., if two vehicles are traveling in adjacent lanes, and the longitudinal distance is zero or near zero when they are next to each other, but the situation is not dangerous if the vehicles have sufficient lateral separation).

[0177] The numerical output of the top node can be, for example, a time-varying robustness score.

[0178] Different rule trees can be constructed to, for example, implement different rules for a given safety model, implement different safety models, or selectively apply rules to different scenarios (for a given safety model, not all rules necessarily apply to all scenarios; with this approach, different rules or combinations of rules can be applied to different scenarios).Within this framework, rules can also be constructed to assess comfort (e.g., based on instantaneous acceleration and / or jerk along the trajectory), progress (e.g., based on the time it takes to reach a defined goal), etc.

[0179] The above examples consider simple logical predicates that are evaluated with a result or signal at a single point in time, e.g., OR, AND, Gt, etc. However, in practice it may be desirable to formulate certain rules in terms of temporal logic.

[0180] Hekmatnejad et al., "Encoding and Monitoring Responsibility Sensitive Safety Rules for Automated Vehicles in Signal Temporal Logic" (2019), MEMOCODE '19: Proceedings of the 17th ACM-IEEE International Conference on Formal Methods and Models for System Design (incorporated herein by reference in its entirety), discloses a signal temporal logic (STL) encoding of RSS safety rules. Temporal logic provides a formal framework for constructing predicates that are conditional over time. This means that the result computed by an assessor at a given moment can depend on the results and / or signal values ​​at other moments.

[0181] For example, a requirement of a safety model may be that an agent responds to a particular event within a set timeframe. Such rules can be coded in a similar way, using temporal logic predicates in a rule tree.

[0182] In the above example, the performance of the stack 100 is evaluated at each time step of the scenario. From this, an overall test result (e.g., pass / fail) can be derived; for example, a particular rule (e.g., a safety-critical rule) may result in an overall fail if the rule fails at any time step in the scenario (i.e., the rule must pass at all time steps to obtain an overall pass for the scenario). For other types of rules, the overall pass / fail criteria may be "softer" (e.g., for a particular rule, a fail may be triggered only if the rule fails for a certain number of consecutive time steps), and such criteria may be context-dependent.

[0183] 8C illustrates a schematic of the hierarchy of rule evaluation implemented within test oracle 252. A set of rules 254 is received for implementation in test oracle 252.

[0184] Certain rules apply only to the own agent (one example is the comfort rule, which assesses whether a maximum acceleration or jerk threshold is exceeded by the own trajectory at any given moment).

[0185] Other rules relate to interactions between the ego-agent and other agents (e.g., the "no collision" rule or the safe distance rule discussed above). Each such rule is evaluated pairwise between the ego-agent and each other agent. As another example, a "pedestrian emergency braking" rule may be activated only if a pedestrian walks in front of the ego-vehicle, and only with respect to that pedestrian agent.

[0186] Not all rules necessarily apply to all scenarios, and some rules may only apply to some of the scenarios. Rule activation logic 422 within test oracle 252 determines whether and when each rule 254 applies to the scenario in question, and, if so, selectively activates the rule when applicable. Thus, a rule may remain active throughout the entire scenario, may never be activated in a given scenario, or may only be activated during part of the scenario. Furthermore, a rule may be evaluated for different numbers of agents at different points in the scenario. Selectively activating rules in this manner can significantly improve the efficiency of test oracle 252.

[0187] The activation or deactivation of a given rule may depend on the activation / deactivation of one or more other rules: for example, the "optimal comfort" rule may be considered non-applicable if the pedestrian emergency braking rule is activated (since pedestrian safety is the primary concern), and the former may be deactivated whenever the latter is active.

[0188] Rule evaluation logic 424 evaluates each active rule for the period it remains active. Each interactive rule is evaluated pairwise between the agent and the other agent to which it applies.

[0189] There may also be some degree of interdependence in the application of rules: for example, another way to deal with the relationship between a comfort rule and an emergency braking rule would be to increase the jerk / acceleration threshold of the comfort rule whenever the emergency braking rule is activated for at least one other agent.

[0190] Although a pass / fail outcome is considered, rules may also be non-binary. For example, two categories of failure may be introduced: "acceptable" and "unacceptable." Again, considering the relationship between a comfort rule and an emergency braking rule, an acceptable failure of a comfort rule may occur when that rule was failed but the emergency braking rule was active. Thus, interdependencies between rules can be addressed in various ways.

[0191] The activation criteria for rules 254 can be specified in the rule creation code provided to the rule editor 400, as can the interdependent nature of rules and the mechanisms for implementing those interdependencies.

[0192] Graphical User Interface: 9A shows a schematic block diagram of the visualization component 520. The visualization component is shown having an input connected to the test database 258 for rendering the output 256 of the test oracle 252 on a graphical user interface (GUI) 500. The GUI is rendered on a display system 522.

[0193] Figure 9B shows an example view of GUI 500 for a particular scenario involving multiple agents. In this example, the test oracle output 526 pertains to multiple external agents, and the results are organized by agent. For each agent, a time series of results is available for each rule that applies to that agent at a given point in the scenario. In the illustrated example, the summary view for "Agent01" is selected, displaying the "top" results calculated for each applicable rule. There is a top result calculated at the root node of each rule tree. Color coding is used to distinguish between periods when the rule is inactive, active and passing, and active and failing for that agent.

[0194] For each time series of results, a first selectable element 534a is provided, which allows results at a lower level in the rule tree, i.e. results calculated further down the rule tree, to be accessed.

[0195] FIG. 9C shows a first expanded view of the results of "Rule 02," where the results of lower-level nodes are also visualized. For example, for the "Safe Distance" rule in FIG. 4B, the results of the "is_latd_safe" and "is_lond_safe" nodes may be visualized (labeled "C1" and "C2" in FIG. 9C). In the first expanded view of Rule 02, it can be seen that the success / failure of Rule 02 is defined by a logical OR relationship between results C1 and C2, and Rule 02 fails only if both C1 and C2 fail (as with the "Safe Distance" rule above).

[0196] For each result time series, a second selectable element 534b is provided which allows the associated numerical performance score to be accessed.

[0197] Figure 9D shows a second expanded view, where the results for Rule 02 and the results for "C1" have been expanded to reveal the associated scores for the time periods these rules were active for Agent 01. The scores are displayed as a visual score-time plot, similarly color-coded to represent pass / fail.

[0198] Example scenario: 10A illustrates a first instance of a cut-in scenario in simulator 202, terminating in a collision event between ego vehicle 602 and another vehicle 604. The cut-in scenario is characterized as a multi-lane driving scenario in which ego vehicle 602 is traveling along a first lane 612 (ego lane) and the other vehicle 604 is initially traveling along a second, adjacent lane 604. At some point in this scenario, the other vehicle 604 moves from the adjacent lane 614 into ego lane 612, ahead of ego vehicle 602 (the cut-in distance). In this scenario, ego vehicle 602 is unable to avoid a collision with the other vehicle 604. The first scenario instance terminates in response to the collision event.

[0199] FIG. 10B shows an example of a first oracle output 256a obtained from the ground truth 310a for the first scenario instance. The "no collision" rule is evaluated between the ego-vehicle 602 and the other vehicle 604 for the duration of the scenario. A collision event results in a failure of this rule at the end of the scenario. In addition, the "safe distance" rule of FIG. 4B is evaluated. When the other vehicle 604 approaches the ego-vehicle 602 laterally, there comes a point (t1) when both the safe lateral distance threshold and the safe longitudinal distance threshold are violated, which results in a failure of the safe distance rule that persists until a collision event at time t2.

[0200] 10C illustrates a second instance of the cut-in scenario, in which the cut-in event does not result in a collision and the ego vehicle 602 is able to reach a safe distance behind the other vehicle 604 after the cut-in event.

[0201] 10D shows an example of a second oracle output 256b obtained from the ground truth 310b for the second scenario instance. In this case, the "no collision" rule is passed throughout. The safe distance rule is violated at time t3, when the lateral distance between the ego-vehicle 602 and the other vehicle 604 is no longer safe. However, at time t4, the ego-vehicle 602 manages to reach a safe distance behind the other vehicle 604. Therefore, the safe distance rule is failed only between time t3 and time t4.

[0202] Recognition Error Framework As described above, both the recognition errors and the driving rules can be assessed based on the extracted pseudo-ground truth 144 determined by the ground truthing pipeline 144 and presented in the GUI 500.

[0203] 11 illustrates an architecture for evaluating perception errors. A triage tool 152 with a perception oracle 1108 is used to extract and evaluate perception errors for both real and simulated driving scenarios, outputting results that are rendered in a GUI 500 alongside results from the test oracle 252. Note that while the triage tool 152 is referred to herein as a perception triage tool, it may be used more generally to extract and present to a user driving data, including perception data and driving performance data, useful for testing and improving an autonomous vehicle stack.

[0204] For real sensor data 140 from a driving run, the output of the online recognition stack 102 is passed to a triage tool 152 which determines a numerical "real-world" recognition error 1102 based on the extracted ground truth 144 obtained by running both the real sensor data 140 and the online recognition output through the ground truthing pipeline 400.

[0205] Similarly, for a simulated driving run in which sensor data is simulated from scratch and a perception stack is applied to the simulated sensor data, the triage tool 152 calculates a simulated perception error 1104 based on a comparison of the detection results from the perception stack to the simulation ground truth. However, for a simulation, the ground truth can be obtained directly from the simulator 202.

[0206] When the simulator 202 directly models the recognition error to simulate the output of the recognition stack, the difference between the simulated detection result and the simulation ground truth, i.e., the simulated recognition error 1110, is known and is passed directly to the recognition oracle 1108.

[0207] The recognition oracle 1108 receives a set of recognition rule definitions 1106, which may be defined via a user interface or written in a domain-specific language, as described in more detail below. The recognition rule definitions 1106 may apply thresholds or rules that define recognition errors and their limits. The recognition oracle applies the defined rules to real or simulated recognition errors obtained for a driving scenario and identifies where the recognition errors violate the defined rules. These results are passed to the rendering component 1120, which renders visual indicators of the evaluated recognition rules for display in the graphical user interface 500. For clarity, the input to the test oracle is not shown in FIG. 11 ; however, it should be noted that the test oracle 252 also relies on ground truth scenarios obtained from either the ground truthing pipeline 400 or the simulator 202.

[0208] Next, further details of the framework for evaluating the perception errors of the real-world driving stack against the extracted ground truth are described. As mentioned above, both the perception errors and the driving rule analysis with the test oracle 252 can be incorporated into a real-world driving analysis tool, which is described in more detail below.

[0209] Not all errors have the same importance. For example, a 10 cm translation error in an agent 10 meters away from the ego vehicle is much more important than the same translation error in an agent 100 meters away. A simple solution to this challenge would be to scale errors based on distance from the ego vehicle. However, the relative importance of different perception errors, or the sensitivity of the ego vehicle's driving performance to different errors, depends on the use case of a given stack. For example, if designing a cruise control system for driving on straight roads, it should be sensitive to translation errors but not particularly sensitive to orientation errors. However, an AV dealing with the entrance to a roundabout should be very sensitive to orientation errors, since it uses the detected orientation of the agent as an indication of whether the agent is about to exit the roundabout and therefore whether it is safe to enter the roundabout. Therefore, it is desirable to make the system's sensitivity to different perception errors configurable for each use case.

[0210] A domain-specific language is used to define the recognition error. This can be used to create recognition rules 1402 (see Figure 14) by defining, for example, the tolerance limit for translation error. The rules implement a configurable set of safe error levels for different distances from the ego vehicle, defined in table 1400. For example, if a vehicle is less than 10 meters away, its position error (i.e., the distance between the car detection and the refined pseudo-ground truth detection) can be defined to be 10 cm or less. If an agent is 100 meters away, the allowable error may be defined to be a maximum of 50 cm. Using a lookup table, rules can be defined to suit any given use case. Based on these principles, more complex rules can be built. Rules can be defined such that the errors of other agents are completely ignored based on their position relative to the ego vehicle, for example, an agent in an oncoming lane when the ego vehicle is separated from oncoming traffic by a median. Traffic behind the ego vehicle beyond the defined cutoff distance may also be ignored based on the rule definition.

[0211] A set of rules can then be applied collectively to a given driving scenario by defining a recognition error specification 1600 that includes all applicable rules. Typical recognition rules that can be included in the specification 1600 define thresholds for longitudinal and lateral translation error (measuring the average error of a detection result relative to the ground truth in the longitudinal and lateral directions, respectively), orientation error (defining the minimum angle a detection result must be rotated to match the corresponding ground truth), and size error (the error in each dimension of the detected bounding box, or the intersection over union ratio of the aligned ground truth and detected boxes to obtain a volumetric difference). Further rules may be based on vehicle dynamics, including agent speed and acceleration errors, and classification errors, such as those defining penalty values ​​for misclassifying a car as a pedestrian or truck. Rules may also include false positives or negatives, as well as detection delays.

[0212] Based on the defined recognition rules, a robustness score can be constructed. Effectively, this can be used to say that if the detection results are within the specified thresholds of the rules, the system should be able to drive safely; if not (e.g., if there is too much noise), something bad may happen that the vehicle may not be able to handle, and this needs to be formally captured. Complex combinations of rules can be included, for example, to evaluate detection results over time and to incorporate complex weather dependencies.

[0213] These rules can be used to associate errors with the replay of the scenario in the UI. As shown in Figure 14, different recognition rules are displayed in different colors on the rule's timeline, corresponding to different results when applying a given rule definition in the DSL. This is the main use case of DSLs (i.e., visualization for triage tools): users write rules in the DSL and the rules are displayed on the UI timeline.

[0214] DSL can also be used to define a contract between the system's perception and planning stacks based on a robustness score calculated for a defined rule. Figure 15 shows an example graph of robustness scores for a given error definition, such as translation error. If the robustness score exceeds a defined threshold of 1500, this indicates that the perception error is within expected performance and the overall system should deliver safe driving. If the robustness score falls below the threshold, as shown in Figure 15, the error is "out of contract" because the planner 106 cannot be expected to drive safely with that level of perception error. This contract essentially becomes a requirement specification for the perception system. It can be used to assign responsibility to either perception or planning. If an error is identified as within contract when the car is misbehaving, this points to a planner issue rather than a perception problem. Conversely, if perception is out of contract, poor behavior is attributed to perception errors.

[0215] By annotating whether recognition errors are considered in-contract or out-of-contract, contract information can be displayed in the UI 500. This uses a mechanism that retrieves the contract specification from the DSL and automatically flags out-of-contract errors on the front end.

[0216] Figure 16 illustrates a third use case, integrating perception errors across different modalities (i.e., real-world and simulated). The above description relates to real-world driving, where a real car drives and collects data, and offline refinement and triage tools 152 calculate perception errors and determine whether these errors are in-contract or out-of-contract. However, the same perception error specification 1600, which specifies perception error rules for evaluating errors, can be applied to simulated driving runs. The simulation can be by generating simulated sensor data that is processed by the perception stack, as described above with reference to Figure 11, or by directly simulating detection results from ground truth using a perception error model.

[0217] In the first case, detection results based on simulated sensor data 1112 have errors 1104, and a DSL can be used to define whether these errors are within or outside the contract. This can be done by simulation based on a recognition error model 208 (i.e., adding noise to the object list), and the injected errors 1110 can be calculated and verified to check that the simulator 202 is modeling what is expected to be modeled. This can also be used to avoid stack failures purely due to recognition errors by intentionally injecting errors within the contract rather than injecting errors outside the contract. In one use case, errors within the contract but near the edge of the contract may be injected in simulation, and it can be verified that the planning system operates correctly given the expected recognition performance. This decouples the development of recognition and planning because they can be tested independently against this contract; once the recognition meets the contract and the planner performs within the limits of the contract, these systems should work together satisfactorily.

[0218] Depending on where the recognition model is sliced, e.g., when doing fusion, we may know very little about what will come out of the simulator, so assessing it in terms of in-contract and out-of-contract errors is useful for analyzing simulated scenarios.

[0219] Another use of DSL is to assess the accuracy of the pseudo ground truth itself. While it is impossible to refine imperfect detection results to obtain perfect ground truth, there is likely an acceptable accuracy that the refinement pipeline must reach in order to be used reliably. DSL rules can be used to assess the pseudo ground truth at this point in time to determine how close it is to the "true" GT currently and how close it needs to be in the future. This may involve the same contract used to check online recognition errors calculated against the pseudo ground truth, but with stricter limits on accuracy applied to ensure there is sufficient confidence that the pseudo ground truth is "correct" enough for online detection results to be assessed. The acceptable accuracy of the pseudo ground truth can be defined as the error within the contract when measured against the "true" ground truth. It is acceptable to produce some error even after refinement, as long as it is within a certain threshold. If different systems have different use cases, each system will apply a different set of DSL rules.

[0220] The "true" ground truth against which refined detection results are assessed is obtained by selecting a real-world dataset, manually annotating it, and evaluating the pseudo-GT against this manual GT according to defined DSL rules to determine whether acceptable accuracy is achieved. Each time the refinement pipeline is updated, the accuracy assessment of the refined detection results can be re-run to check for regressions in the pipeline.

[0221] Another use of DSL is that once a contract is defined between recognition 102 and planning 106, it becomes possible to split the types of tests that need to be done at the recognition layer. This is shown in Figure 17. For example, the recognition layer can be fed a set of sensor readings that all contain errors that must be within the contract, and DSL rules can be applied to check whether that is the case. Similarly for the planning layer, first a ground truth test 1702 can be applied, and if that passes, an within-contract test 1704 is applied, so the system is fed a list of objects with within-contract errors to see if the planner will behave safely.

[0222] In one exemplary testing scheme, the planner may be considered "given" and simulation may be used to generate recognition errors to find limits on acceptable recognition accuracy for the planner to perform as intended. These limits may then be used to semi-automatically create a contract for the recognition system. A set of recognition systems may be tested against this contract to find one that meets it, or the contract may be used as a guide when developing recognition systems.

[0223] Real-World Driving Analysis Tools The above-described testing framework, i.e., test oracle 252 and cognitive triage tool 152, may be combined in a real-world driving analysis tool, in which both cognitive and driving evaluations are applied to the cognitive ground truth extracted from the ground truth pipeline 400, as shown in FIG. 2C.

[0224] FIG. 12A shows an exemplary user interface for analyzing driving scenarios extracted from real-world data. In the example of FIG. 12A, a schematic bird's-eye view representation 1204 of a scene based on point cloud data (e.g., derived from lidar, radar, or stereo or mono depth imaging) is shown, with the corresponding camera frame 1224 shown in the inset. Road layout information may be obtained from high-definition map data. The camera frame 1224 may be annotated with detection results. The UI may also show sensor data collected during driving, such as lidar, radar, or camera data. This is shown in FIG. 12B. The scene visualization 1204 is also overlaid with annotations based on the derived pseudo-ground truth as well as detection results from the in-vehicle perception component. In the illustrated example, three vehicles are present, each annotated with a box. The solid box 1220 shows the simulated ground truth for the agent in the scene, and the outline 1222 shows the raw detection results from the ego vehicle's perception stack 102. A visualization menu 1218 is shown, in which the user can select whether to display sensor data, online, and offline detection results. These may be toggled on and off as needed. Showing the real sensor data side-by-side with both the vehicle detection results and the ground truth detection results allows the user to identify or confirm specific errors in the vehicle detection results. The UI 500 allows playback of the selected footage, and a timeline view is shown, allowing the user to select any time point 1216 in the footage to show bird's-eye view snapshots and camera frames corresponding to the selected time point.

[0225] As described above, the recognition stack 102 can be assessed by comparing detection results to the refined pseudo-ground truth 144. Recognition is assessed against defined recognition rules 1106, which can depend on the use case of a particular AV stack. These rules specify different value ranges for discrepancies between the position, orientation, or scale of the vehicle detection results and those of the pseudo-ground truth detection results. The rules can be defined in a domain-specific language (as described above with reference to FIG. 14). As shown in FIG. 12A, different recognition rule results are plotted along a "top-level" recognition timeline 1206 of a driving scenario that aggregates the recognition rule results, with periods on the timeline flagged when any recognition rule is broken. This can be expanded to present a set of individual recognition rule timelines 1210 for each defined rule.

[0226] The recognition error timeline may be "zoomed out" to present a longer period of the driving trip. A zoomed-out view may not be able to show the recognition errors with the same granularity as when zoomed in. In this case, the timeline may show an aggregation of the recognition errors over the time window to provide a combined set of recognition errors for the zoomed-out view.

[0227] A second driving assessment timeline 1208 shows how the simulated ground truth data was assessed against the driving rules. The aggregated driving rules are displayed in the top-level timeline 1208, which can be expanded into a set of individual timelines 1212 displaying performance against each defined driving rule. Each rule timeline can be further expanded, as shown, to display a plot 1228 of the numerical performance score over time for a given rule. This corresponds to selectable element 534b described above with reference to FIG. 9C. In this case, the simulated ground truth detection results are considered the actual driving behavior of the agent in the scene. The behavior of the ego vehicle can be evaluated against the defined driving rules, for example, based on the Digital Highway Code, to determine whether the vehicle behaved safely for a given scenario.

[0228] In summary, both recognition rule evaluation and driving assessment are based on using the offline recognition methods described above to refine detection results from real-world driving. For driving assessment, the refined pseudo-ground truth 144 is used to assess the ego-vehicle's behavior against the driving rules. This can also be used to generate simulated scenarios for testing, as shown in Figure 2C. For recognition rule evaluation, a recognition triage tool 152 compares recorded vehicle detection results with the offline refined detection results to quickly identify and triage possible recognition failures.

[0229] Driving notes may also be displayed in a driving notes timeline view 1214, which may display notable events flagged during the drive. For example, driving notes may include when the vehicle brakes or turns, or when the human driver disengages the AV stack.

[0230] An additional timeline may be displayed that displays user-defined metrics to help users debug and triage potential issues. User-defined metrics may be defined to identify errors or stack flaws as well as to triage errors as they occur. Users may define custom metrics depending on the goals of a given AV stack. An example user-defined metric may flag when messages arrive out of order or message delays for acknowledged messages. This is useful for triage, as it may be used to determine whether planning occurred due to a planner error or because messages arrived late or out of order.

[0231] Figure 12B shows an example of a UI visualization 1204 where sensor data is displayed and a camera frame 1224 is displayed in the inset. Typically, sensor data from a single snapshot in time is presented. However, each frame may present sensor data aggregated over multiple time steps to obtain a static scene map when high-resolution map data is not available. As shown on the left, there are several visualization options 1218 for displaying or hiding data such as camera, radar, or lidar data collected during a real-world scenario, or online detection results from the ego-vehicle's own perception. In this example, the online detection results from the vehicle are shown as contours 1222 overlaid on a solid box 1220 representing the refined detection results of the ground truth. An orientation error can be seen between the ground truth and the vehicle's detection results.

[0232] The refinement process performed by the ground truthing pipeline 400 is used to generate pseudo ground truth 144 as a basis for multiple tools. The presented UI displays results from the perception triage tool 152, which allows assessing the driving ability of the ADAS in a single driving example using a test oracle 252, detecting defects, extracting scenarios that reproduce issues (see Figure 2C), and sending identified issues to developers to improve the stack.

[0233] Figure 12C shows an exemplary user interface configured to allow a user to zoom in on a subsection of a scenario. Figure 12C shows a snapshot of the scenario, including a schematic representation 1204 and a camera frame 1224 presented in an inset, as described above with respect to Figure 12A. The set of recognition error timelines 1206, 1210, as well as the expandable driving assessment timeline 1208 and drive notes timeline 1214, described above, are also shown in Figure 12C.

[0234] In the example shown in Figure 12C, the current snapshot of the driving scenario is indicated by a scrubber bar 1230 that spans all timeline views simultaneously. This may be used instead of the current point in the scenario indication 1216 on a single playback bar. The user can click on the scrubber bar 1230 to select and move it to any point in the driving scenario. For example, the user may be interested in a particular error, such as a point within a section colored red or otherwise indicated as containing an error on the position error timeline. The indication is determined based on the position error observed at that time between the "ground truth" and the detection results for the period corresponding to the indicated section. The user can click on the scrubber bar and drag it to the point of interest in the position error timeline. Alternatively, the user can click on a point on any of the timelines spanned by the scrubber to position the scrubber at that point. This updates the schematic view 1204 and inset 1224 to present a top-down schematic view and camera frame corresponding to the selected time point, respectively. The user can then inspect the schematic view and any available camera or other sensor data to verify position errors and identify possible reasons for recognition errors.

[0235] A "ruler" bar 1232 is presented above the recognition timeline 1206 and below the schematic view. It contains a series of "notches" that indicate time intervals of the driving scenario. For example, if the timeline view displays a 10-second time interval, notches indicating 1-second intervals are presented. Some time points are also labeled with a numerical indicator, e.g., "0 seconds," "10 seconds," etc.

[0236] A zoom slider 1234 is provided at the bottom of the user interface. The user can drag the indicator along the zoom slider to change the portion of the driving scenario presented in the timeline. Alternatively, the indicator's position may be adjusted by clicking the desired point on the slider bar to which the indicator should move. A percentage is presented to indicate the currently selected level of zoom. For example, if the entire driving scenario is one minute long, timelines 1206, 1208, and 1214 would present the recognition errors, driving assessments, and driving notes, respectively, over one minute of driving, with the zoom slider indicating 100% and the button in its leftmost position. If the user slides the button until the zoom slider indicates 200%, the timeline adjusts to present only the results corresponding to a 30-second snippet of the scenario.

[0237] The zoom may be configured to adjust the visible portion of the timeline depending on the position of the scrubber bar. For example, if a one-minute scenario has the zoom set to 200%, the zoomed-in timeline will present a 30-second snippet centered around the selected time point where the scrubber is located, i.e., 15 seconds of timeline will be presented before and after the point indicated by the scrubber. Alternatively, the zoom may be applied relative to a reference point, such as the start of the scenario. In this case, the zoomed-in snippet presented in the timeline after zooming will always start at the start of the scenario. The granularity of the notches and numeric labels on the ruler bar 1232 may adjust depending on the degree to which the timeline is zoomed in or out. For example, if a scenario is zoomed in from 30 seconds to present a three-second snippet, before zooming, the numeric labels may be displayed at 10-second intervals and the notches at 1-second intervals, and after zooming, the numeric labels may be displayed at 1-second intervals and the notches at 100-ms intervals. The visualization of time steps in the timelines 1206, 1208, 1214 is "stretched" to accommodate the zoomed-in snippet. A higher level of detail may be displayed in the timeline in a zoomed-in view because shorter snippets in time can be represented by a larger area in the timeline display in the UI. Thus, errors spanning a very short time within a longer scenario may only be visible in the timeline view when zoomed in.

[0238] Other zoom inputs may be used to adjust the timeline to show shorter or longer snippets of the scenario. For example, if the user interface is implemented on a touchscreen device, the user may apply a pinch gesture to apply a zoom to the timeline. In other examples, the user may scroll back and forth on a mouse scroll wheel to change the zoom level.

[0239] When the timeline is zoomed in to present only a subset of the driving scenario, different parts of the scenario can be inspected by the user in the timeline view, as the timeline can be scrolled in time to shift the displayed portion in time. The user can scroll by clicking and dragging a scroll bar (not shown) at the bottom of the timeline view, or by using, for example, the touchpad of the associated device on which the UI is running.

[0240] The user can also select a snippet of the scenario to be exported, for example, for further analysis or as the basis for a simulation. Figure 12D shows how a section of a driving scenario can be selected by the user. The user can click the relevant point on the ruler bar 1232 with the cursor. This can be done at any zoom level. This sets the initial boundary of the user selection. The user then drags the cursor along the timeline to extend the selection to the chosen point. If zoomed in, continuing to drag until the end of the displayed snippet of the scenario scrolls the timeline forward, allowing the selection to be further extended. The user can stop dragging at any point; the point where the user stops becomes the ending boundary of the user selection. Bar 1230 at the bottom of the user interface displays the temporal length of the selected snippet; this value is updated as the user drags the cursor to extend or shorten the selection. The selected snippet 1238 is presented as a shaded section on the ruler bar. Several buttons 1236 are presented that offer user actions, such as "Extract Trajectory Scenario" to extract data corresponding to the selection, which may be stored in a database of extracted scenarios. This may be used for further analysis or as a basis for simulating similar scenarios. After making a selection, the user can zoom in or out, and the selection 1238 on the ruler bar 1232 will also expand or contract, along with the ruler and the timelines for recognition, driving assessment, and drive notes.

[0241] The pseudo-ground truth data can also be used with a data exploration tool to search for data in the database. This tool can be used when a new version of the AV stack is deployed. For a new version of software, a car can be driven for a period of time (e.g., a week) to collect data. Within this data, a user may be interested in testing how the car behaves in specific conditions, so they may provide queries such as, for example, "show me driving at night" or "show me when it was raining." The data exploration tool pulls out relevant footage, which can then be investigated using a triage tool. This data exploration tool serves as a kind of entry point for further analysis.

[0242] For example, once a new software version is implemented and the AV has been in operation for a while and has collected a certain amount of data, further assessment tools may be used to aggregate the data and understand the overall performance of the vehicle. The vehicle may have a newly developed set of features, such as using indicators and entering and exiting roundabouts, and an overall performance assessment may be sought of how well the vehicle behaves with respect to these features.

[0243] Finally, the re-simulation tool can be used to run an open-loop simulation with the sensor data run through the new stack to check for regression issues.

[0244] Figure 13 shows an example user interface of the perception triage tool 152, highlighting the scenario visualization 1204 and the perception error timeline 1206, 1210. As shown on the left, there are several visualization options 1218 for displaying or hiding data, such as online detection results from camera, radar, or lidar data collected during a real-world scenario, or from the vehicle's own perception. In this case, the visualization is limited to refined detection results only—that is, agents detected offline and whose refinement results are indicated by solid boxes. Each solid box has an associated online detection result (not shown) that shows how the agent was perceived at that time snapshot before error correction / refinement. As mentioned above, there is a certain amount of error between the ground truth 144 and the original detection results. Various errors can be defined, including errors in the scale, position, and orientation of the agent in the scene, as well as "ghost" detections and missed detections due to false positives.

[0245] As noted above, not all errors have the same importance. The recognition rule DSL allows for the definition of rules tailored to the desired use case. For example, if designing a cruise control system for driving on a straight road, it should be sensitive to translation error but not particularly sensitive to orientation error. However, an AV dealing with the entrance to a roundabout should be very sensitive to orientation error because it uses the detected orientation of the agent as an indication of whether the agent is about to exit the roundabout and therefore whether it is safe to enter the roundabout. The recognition error framework allows for the definition of separate tables and rules that indicate the relative importance of a given translation or orientation error for that use case. For illustrative purposes, the box shown around the ego-vehicle in Figure 13 indicates an area of ​​interest that recognition rules may be defined to target. Rule evaluation results may be displayed in the recognition error timeline 1210 of the user interface. A visual indicator of the rule may be displayed in the schematic representation 1204, for example, by flagging the area where a particular rule is defined; this is not shown in Figure 13.

[0246] In addition to displaying the results of a single snapshot of a driving trip, queries and filtering can also be applied to filter the data according to the cognitive assessment results, providing more context for the user performing the analysis.

[0247] 18A and 18B show an example of a graphical user interface 500 for filtering and displaying recognition results of real-world driving trips. For a given trip, a recognition error timeline 1206 is displayed with aggregated rule evaluations for all recognition errors, as described above. A second set of timelines 1226 may be presented, showing driving scene features such as weather conditions, road features, other vehicles, and vulnerable road users. These may be defined within the same framework used to define the recognition error rules. Note that recognition rules may be defined so that different thresholds apply depending on different driving conditions. FIG. 18A also shows a filtering function 1800 that allows a user to select a query to be applied to the evaluation. In this example, the user query is to find a "slice" of the driving trip in which vulnerable road users (VRUs) are present.

[0248] This query is processed and used to filter the frames of the driving scenario representation to those tagged with vulnerable road users. Figure 18B shows an updated view of the perception timeline after the filter has been applied. As shown, a subset of the original timeline is presented, and in this subset, vulnerable road users are always present, as shown in the "VRU" timeline.

[0249] FIG. 19A illustrates other features that may be used to perform analysis within the graphical user interface 500. A set of error threshold sliders 1900 are shown that the user can adjust. The range of errors may be informed by the recognition error limits defined in the recognition rules DSL. The user may adjust the threshold for a given error by sliding a marker to the desired new threshold for that error. For example, a user may set a reject threshold for a translation error of 31 m. This threshold can then be fed back into the translation error defined in the recognition rule specification written in the aforementioned recognition rules DSL to adjust the rule definition to take the new threshold into account. The new rule evaluation results are passed to the front end, and any currently occurring rule failures with respect to the new threshold are shown in the expanded timeline view 1210 for a given error. As shown in FIG. 19A, lowering the threshold for unacceptable error values ​​causes more errors to be flagged in the timeline.

[0250] FIG. 19B illustrates how aggregate analysis can be applied to a selected slice of a driving scenario, allowing a user to select and inspect the most relevant frames based on calculated recognition errors. As previously described, a user can use filtering functionality 1800 to filter the scenario to present only frames in which vulnerable road users are present. Within matching frames, a user can further “slice” the scenario into specific snippets using selection tool 1902, which can be dragged along timeline 1206 and zoomed in to cover a period of interest. For a selected snippet, aggregate data may be displayed to the user in display 1904. Various attributes of the recognition errors captured within the selected snippet may be selected and graphed against each other. In the illustrated example, the error type and error magnitude are graphed, allowing a user to visualize the most significant errors of each type for a selected portion of the scenario. A user may select any point on the graph to display a camera image 1906 of the corresponding frame in which that error occurred, along with other scene variables such as occlusion, allowing the user to inspect the frame for factors that may have caused the error.

[0251] The ground truthing pipeline 400 may be used with additional tools to query, aggregate, and analyze vehicle performance, including the cognitive triage tool 152 and test oracle 252, as well as the data exploration and aggregation assessment tools described above. The graphical user interface 500 may display results from these tools in addition to the snapshot views described above.

[0252] While the above example considers testing AV stacks, the techniques can be applied to test components of other forms of mobile robots. For example, other mobile robots are being developed to carry cargo in industrial areas both inside and outside the country. Such mobile robots do not have a human on board and belong to a class of mobile robots called UAVs (unmanned autonomous vehicles). Autonomous aerial mobile robots (drones) are also being developed.

[0253] References herein to components, functions, modules, etc. refer to functional components of a computer system, which can be implemented at the hardware level in various ways. The computer system includes execution hardware that may be configured to perform the method / algorithm steps disclosed herein and / or to implement models trained using the present technology. The term execution hardware encompasses any form or combination of hardware configured to perform the relevant method / algorithm steps. The execution hardware may take the form of one or more processors, which may be programmable or non-programmable, or a combination of programmable and non-programmable hardware may be used. Examples of suitable programmable processors include general-purpose processors based on instruction set architectures, such as CPUs, GPUs, and accelerator processors. Such general-purpose processors typically execute computer-readable instructions stored in memory coupled to or embedded in the processor and perform the associated steps in accordance with those instructions. Other forms of programmable processors include field-programmable gate arrays (FPGAs), which have circuit configurations that are programmable through circuit-describing code. Examples of non-programmable processors include application-specific integrated circuits (ASICs). Code, instructions, etc. may be stored on transitory or non-transitory media (examples of the latter include solid-state, magnetic, and optical storage devices, etc.) as appropriate. Subsystems 102-108 of the run-time stack of FIG. 2A may be implemented in a vehicle with programmable or dedicated processors, or a combination of both, or in an off-vehicle computer system in a testing or other context. Various components of figures including FIG. 11 and FIG. 6, such as simulator 202 and test oracle 252, may likewise be implemented in programmable and / or dedicated hardware.

Claims

1. 1. A computer system for testing a real-time recognition system, the real-time recognition system being for deployment in a sensor-equipped vehicle, the computer system comprising: at least one input configured to receive data of at least one real-world driving journey performed by a sensor-equipped vehicle, said data comprising: (i) a time series of sensor data captured by the sensor-equipped vehicle; and (ii) a time series of at least one associated driving-time perception output extracted from the sensor data time series by the real-time perception system under test; a rendering component configured to generate rendering data for rendering a graphical user interface (GUI), the graphical user interface comprising, for each of a plurality of time steps of the at least one real-world driving run, a recognition error timeline having a visual indication of recognition errors occurring at the time step; and a ground truthing pipeline configured to process at least one of (i) the time series of sensor data and (ii) the time series of on-the-road recognition outputs by applying at least one non-real-time and / or non-causal recognition algorithm to the at least one to extract at least one time series of ground truth recognition outputs for comparison with the on-the-road recognition outputs; a recognition oracle configured to compare a time series of the running recognition outputs with a time series of the ground truth recognition outputs to identify recognition errors occurring in one or more time intervals to generate the recognition error timeline; Equipped with The recognition oracle calculates a numerical error value between the time series of running recognition outputs and the time series of ground truth recognition outputs, and identifies recognition errors by comparing the numerical error value to at least one recognition error threshold.

2. 2. The computer system of claim 1, wherein a numerical error value is identified as a recognition error if the numerical error value exceeds one or more of the at least one recognition error thresholds.

3. 3. The computer system of claim 1, wherein the at least one recognition error threshold is fixed.

4. 3. The computer system of claim 1, wherein the at least one recognition error threshold is variable depending on one or more scene variables.

5. 10. The computer system of claim 1, wherein the error threshold is adjustable via a graphical user interface.

6. 2. The computer system of claim 1, wherein the error threshold is adjustable via rule definition instructions provided to the recognition oracle.

7. The computer system of claim 6 , wherein a rule editor is used to define a recognition error specification comprising the rule definition instructions coded in a domain-specific language.

8. 5. The computer system of claim 4, wherein the one or more scene variables comprise a distance between a recognized object and an own agent, and the variable threshold increases with the distance between the object and the own agent.

9. 4. The computer system of claim 3, wherein the numerical error value is weighted according to one or more scene variables.

10. 2. The computer system of claim 1, wherein the numerical error value is normalized to a predetermined range.

11. 2. The computer system of claim 1, wherein in addition to the identified recognition error, the numerical error value is made accessible via the GUI.

12. The recognition error is Binary indicators, non-binary categorical indicators, 10. The computer system of claim 1, comprising at least one of:

13. The computer system of claim 1 , wherein the recognition error comprises an aggregated error calculated over multiple objects and / or sensors and / or sensor modalities and / or time windows.

14. 14. The computer system of claim 13, wherein a plurality of lower-level recognition error timelines are defined, and a top-level aggregate recognition error timeline is input by applying predetermined rules to the lower-level recognition error timelines.

15. 15. The computer system of claim 14, wherein the top-level timeline is expandable to view the lower-level timelines.

16. 2. The computer system of claim 1, wherein the recognition oracle is configured to filter at least one time interval of the driving run, the time interval being omitted from the recognition error timeline, and the filtering is performed based on one or more filtering criteria, i.e., one or more tags / labels associated with the recognition error and / or the real-world driving run.

17. 17. The computer system of claim 16, wherein the tag is accessible via the GUI.

18. 2. The computer system of claim 1, wherein a schematic representation of the driving journey is displayed on the GUI, the schematic representation displaying a static snapshot of the driving journey at a current time step, the current time step being selectable via a command to the GUI.

19. 20. The computer system of claim 18, wherein a visual indicator is changed to mark the current time step on the recognition error timeline.

20. 20. The computer system of claim 18 or 19, wherein sensor data for the at least one driving trip is displayed on the GUI.

21. The sensor data is Lidar point cloud, radar point cloud, Mono / stereo depth image, or 2D camera image 10. The computer system of claim 1, comprising at least one of:

22. 20. The computer system of claim 18, wherein the time series of ground truth recognition outputs is used to render the schematic representation in the GUI.

23. 23. The computer system of claim 22, wherein a time series of the driving recognition output is displayed via the GUI for visual comparison with the ground truth recognition output.

24. 24. The computer system of claim 23, wherein a time series of the driving recognition output is overlaid on the schematic representation.

25. The computer system of claim 1 , wherein the time series of ground truth perception outputs is in the form of a trajectory for each agent, the trajectory comprising a time sequence of spatial and motion states of the agent.

26. 10. The computer system of claim 1, further comprising a test oracle for applying a driving performance assessment to the ground truth perception output, the results of the driving performance assessment being conveyed on a second performance timeline also displayed on the GUI.

27. 10. The computer system of claim 1, wherein the sensor data comprises data from two or more sensor modalities, at least one sensor modality being used to provide ground truth for at least one other sensor modality.

28. The computer system of claim 1 , wherein manually labeled ground truth data is used in the system to measure the accuracy of the ground truth recognition output.

29. 10. The computer system of claim 1, wherein manually labeled ground truth data is used in the system to measure the accuracy of the driving recognition output.

30. 10. The computer system of claim 1, wherein recognition errors derived from simulated driving runs and / or recognition errors derived from real-world driving data are rendered in the GUI without ground truth recognition output.

31. 1. A computer-implemented method for testing a real-time perception system, the real-time perception system being for deployment on a sensor-equipped vehicle, the method comprising: receiving at an input data of at least one real-world driving journey performed by a sensor-equipped vehicle, said data comprising: (i) a time series of sensor data captured by said sensor-equipped vehicle; and (ii) a time series of at least one associated driving-time perception output extracted from said sensor data time series by said real-time perception system under test; generating, by a rendering component, rendering data for rendering a graphical user interface (GUI) comprising a recognition error timeline, the recognition error timeline having, for each of a plurality of time steps of the at least one real-world driving run, a visual indication of recognition errors that occurred at the time step; processing at least one of (i) the sensor data time series and (ii) the on-the-road recognition output time series in a ground truthing pipeline by applying at least one non-real-time and / or non-causal recognition algorithm to the at least one to extract at least one ground truth recognition output time series for comparison with the on-the-road recognition output; comparing the time series of run-time recognition outputs with the time series of ground truth recognition outputs in a recognition oracle to identify recognition errors that occurred in one or more time intervals to generate the recognition error timeline; Equipped with The recognition oracle calculates a numerical error value between the time series of running recognition outputs and the time series of ground truth recognition outputs, and identifies recognition errors by comparing the numerical error value to at least one recognition error threshold.

32. 32. A computer program product comprising executable program instructions for programming a computer system to implement the method of claim 31.

Images