Identifying significant test run involving mobile robot trajectory planner

Abstract

The present disclosure provides systems and methods for identifying significant test operations involving autonomous vehicle systems. A processor receives a plurality of sets of operational data, each set of data representing a driving scene. For each set of data, an output set is generated that includes a time-indexed event generated in response to the detected behavior of the at least one challenger agent and a series of decision indicators indicating whether a driving action of the self-agent is to be allowed. A data retrieval component is coupled to the result database and retrieves an output group based on the time indexed event and the series of decision indicators. The processor generates a series of decision indicators by generating a planned trajectory for the self-agent and determining whether a predefined driving action of the self-agent is to be allowed.

Description

Technical Field

[0001] The present disclosure relates to methods for identifying significant test runs involving trajectory planners in real or simulated scenarios, and computer programs and systems for implementing the methods. Such planners are capable of automatically planning self-trajectories for fully / semi-autonomous vehicles or other forms of mobile robots. Example applications include autonomous driving system (ADS) and advanced driver assistance system (ADAS) performance testing. Background Art

[0002] There has been significant and rapid development in the field of autonomous vehicles. An autonomous vehicle (AV) is a vehicle equipped with sensors and control systems that enable it to operate without human control over its behavior. An AV is equipped with sensors that enable it to perceive its physical environment, such sensors include, for example, cameras, radars, and lidars. An AV is equipped with appropriately programmed computers that are able to process the data received from the sensors and make safe and predictable decisions based on the context perceived by the sensors. An autonomous vehicle can be fully autonomous (in that it is designed to operate without human supervision or intervention, at least in certain circumstances) or semi-autonomous. Semi-autonomous systems require varying levels of human supervision and intervention. Advanced driver assistance systems (ADAS) and certain levels of automated driving systems (ADS) can be classified as semi-autonomous. There are different aspects of testing the behavior of the sensors and control systems on a specific autonomous vehicle or a type of autonomous vehicle.

[0003] A "Level 5" vehicle is one that can operate fully autonomously in all situations, as it always guarantees a certain minimum level of safety. Such a vehicle requires no manual controls (steering wheel, pedals, etc.) at all.

[0004] In contrast, Level 3 and Level 4 vehicles can operate fully autonomously, but only within certain defined circumstances (e.g., within geofenced areas). Level 3 vehicles must be able to automatically handle any situation that requires an immediate response (e.g., emergency braking); however, a change in circumstances may trigger a “transition demand,” requiring the driver to take control of the vehicle within some limited time frame. Level 4 vehicles have similar limitations; however, if the driver does not respond within the required time frame, Level 4 vehicles must also be able to automatically implement a “minimum risk maneuver” (MRM), which is to take some appropriate action to bring the vehicle to a safe condition (e.g., slowing and stopping the vehicle). Level 2 vehicles require the driver to be ready to intervene at all times, and it is the driver’s responsibility to intervene if the automated system fails to respond appropriately at any time. For Level 2 automation, it is the driver’s responsibility to determine when their intervention is required; for Levels 3 and 4, this responsibility shifts to the vehicle’s automated systems, which must alert the driver when intervention is required.

[0005] As the level of automation increases and more responsibility is transferred from humans to machines, safety is an increasing challenge. In autonomous driving, the importance of ensuring safety has been recognized. Ensuring safety does not necessarily mean zero accidents, but rather ensuring that a certain minimum safety level is achieved under specified circumstances. It is generally believed that this minimum safety level must be significantly higher than that of human drivers for autonomous driving to be feasible.

[0006] According to Shalev-Shwartz et al., “On a Formal Model of Safe and Scalable Self-driving Cars” (2017), arXiv:1708.06374 (RSS paper), which is incorporated herein by reference in its entirety, human driving is estimated to cause 10 -6 Assuming that autonomous driving systems will need to be reduced by at least three orders of magnitude, the RSS paper concludes that 10 -9 The minimum safety level to prevent a serious accident, and points out that a purely data-driven approach would therefore require the collection of large amounts of driving data every time a change is made to the AV system's software or hardware.

[0007] The RSS paper provides a model-based approach to safety. The rule-based Responsibility-Sensitive Safety (RSS) model is constructed by formalizing a small number of "common sense" driving rules:

[0008] "1. Don't hit people from behind.

[0009] 2. Don't cut in rashly.

[0010] 3. The right of way is given, not taken.

[0011] 4. Be careful in areas with limited visibility.

[0012] 5. If you can avoid one accident without causing another, you must do it. "

[0013] The RSS model is provably safe, in the sense that if all agents obeyed the rules of the RSS model at all times, no accidents would occur. The goal is to reduce by orders of magnitude the amount of driving data that needs to be collected to demonstrate the required level of safety.

[0014] A safety model (such as RSS) can be used as a basis for evaluating the quality of trajectories planned or achieved by a self-agent (ego agent) in real or simulated scenarios under the control of an autonomous system (stack). The stack is tested by exposing it to different scenarios and evaluating whether the resulting ego trajectories comply with the rules of the safety model (rule-based testing). Rule-based testing methods can also be applied to other aspects of performance, such as comfort or progress towards a set goal.

[0015] Even with rule-based testing, it is still necessary to expose the stack to a wide range of scenarios and cover a large amount of test mileage. Given the amount of test mileage typically required, relying solely on real scenarios is usually not practical, so most testing is performed in the form of simulated scenarios.

[0016] Whether real or simulated, scenarios (or each "run" through a scenario) will vary in terms of their salience with respect to the test stack. In this context, salience can refer to the usefulness of a scenario or run in terms of testing the stack. For example, some scenarios will involve the self-agent driving in relatively routine situations, such as on quiet roads or along routes with relatively simple road layouts, and thus will not be particularly salient. Other scenarios will present more challenging situations, such as with more congested traffic, unusual road layouts, and unexpected behavior by the challenger agent, and thus will be more salient.

[0017] It is desirable to identify salient scenarios, for example, so that they can be reused in future tests in preference to less salient scenarios, thereby saving computational resources running scenarios that provide little significance to the self-agent's performance estimate. However, in a simulated environment, it may not be clear at the outset whether a particular scenario's run is salient. For real scenarios, it is impossible to know in advance how a scenario will unfold.

[0018] Once a run is complete, manually examining the data generated by a run (hereinafter referred to as "run data") to determine its significance is not feasible given the large number of simulation runs and the associated amount of run data performed during testing. This is complicated by the fact that the significant portion of a run may be only a small portion of the total run.

[0019] The present disclosure aims to solve the difficulties discussed above, as well as any other difficulties that are apparent to those skilled in the art from the description herein. Summary of the invention

[0020] According to a first aspect of the present disclosure, there is provided a computer system for identifying significant test runs involving an autonomous vehicle system in a real or simulated scenario, the computer system comprising:

[0021] A processor configured to:

[0022] receiving a plurality of sets of operating data, each set of operating data representing a driving scenario having a self-agent and at least one challenger agent, wherein the self-driving vehicle system controls the self-agent,

[0023] For each set of running data, an output set is generated, which includes:

[0024] at least one time-indexed event generated in response to detected behavior of at least one challenger agent,

[0025] a series of decision indicators at a plurality of time steps of the operational data, the decision indicator at each time step indicating whether a predefined driving action of the ego-agent is to be permitted, each driving indicator being generated independently of the autonomous vehicle system controlling the ego-agent, and

[0026] a data retrieval component coupled to the results database and configured to retrieve a set of one or more output groups based on at least one time-indexed event and the series of decision indicators;

[0027] The processor is configured to generate the decision indicator series in the following manner:

[0028] Generate a planning trajectory based on the ego agent’s position from time step to a future time step that is independent of the ego agent’s behavior between time step and the future time step, and

[0029] Whether a predefined driving maneuver of the ego-agent will be permitted is determined by estimating the planned trajectory based on at least one predefined maneuver estimation rule.

[0030] The system can process the operational data to identify and store occurrences of two related situations: a specific behavior of the challenger agent (e.g., cutting in front of the ego agent), and whether the ego agent is allowed to perform a driving action (e.g., lane change) according to the reference driving model. The presence of these two related situations (e.g., the challenger agent has cut in front where a lane change is not allowed) can indicate a significant scenario. Therefore, such instances can be retrieved from the large amount of operational data.

[0031] The processor may be further configured to generate at least one performance result based on predefined driving performance rules applied to the self-agent and include the at least one performance result in the output set. Thus, the processor may form part of a test oracle.

[0032] The system may include a visualization component configured to generate rendering data for causing a display system to display a query graphical user interface (GUI). The query GUI may include an input configured to receive a user selection of at least one time-indexed event and a decision indicator. The data retrieval component may be configured to retrieve a set of one or more output groups based on the user selection of the at least one time-indexed event and the decision indicator.

[0033] The system may include a visualization component configured to generate rendering data to cause a display system to display a result GUI including a set of one or more output groups.

[0034] The system may include a visualization component configured to generate rendering data for causing a display system to display an output group GUI. The output group GUI may be configured to display an output group selected from a set of output groups. The output group GUI may include: a first time series display element that shows time indexed events in the selected output group on a first timeline. The output group GUI may include: a second time series display element that shows the permission of the driving action represented by the decision indicator in the selected output group on a second timeline. The output group GUI may include a third time series display element that shows at least one performance result on a third timeline.

[0035] The data retrieval component may be configured to retrieve a set of one or more output groups, wherein a time-indexed event occurs within a predetermined time range of a decision indicator having a predetermined value. The time-indexed event may occur simultaneously with the decision indicator having the predetermined value.

[0036] The at least one time-indexed event may include, for example, a cut-in event, a cut-out event, or a braking event. Each time-indexed event may be associated with a plurality of parameters.

[0037] A cut-in event may occur when another agent in front of the ego agent, such as a challenger agent, moves into the lane of the ego agent and slows down. This may constitute a hazard for the ego to avoid. The parameters of a cut-in event may include one or more of the following: the longitudinal distance from the ego agent to the challenger agent when the cut-in begins, the challenger agent's speed before the cut-in, and the challenger agent's lateral speed during the lane change.

[0038] A cut-out event may occur when a first challenger agent in front of the ego agent and in the ego agent's lane moves into an adjacent lane and a second challenger agent is shown moving at a low speed or stationary in front of it. The second challenger agent may pose a hazard for the ego agent to avoid. Parameters of the cut-out event may include one or more of the following: longitudinal distance from the first challenger agent to the second challenger agent at the start of the cut-out; lateral offset of the first challenger agent from the center of the lane; lateral offset of the second challenger agent from the center of the lane; lateral offset of the first challenger agent from the center of the lane after the lane change; speed of the second challenger agent after the cut-out; lateral speed of the first challenger agent during the lane change: and speed of the first challenger agent before the cut-out.

[0039] A braking event occurs when another agent (e.g., a challenger agent) in front of the ego agent and in the lane of the ego agent suddenly brakes. The other agent may pose a hazard for the ego agent to avoid. The parameters of the braking event may include one or more of the following: the challenger agent's speed after deceleration, the challenger agent's maximum deceleration rate, and the challenger agent's maximum jerk rate.

[0040] The predefined driving maneuvers may include, for example, one of a lane change maneuver, a highway merge maneuver, entering a roundabout, an unprotected left turn, or an unprotected right turn.

[0041] The data retrieval component may be configured to identify a portion of a set of operational data including a time-indexed event and a decision indicator having a predetermined value.

[0042] The processor may be configured to generate at least one time-indexed event by applying a plurality of conditions to behavior of the challenger agent in the operational data. The conditions may include constraints applied to one or more parameters of the event.

[0043] The predefined maneuver estimation rules may form part of an open-loop rule-based planner or a basic driver model.

[0044] The planned trajectory may include a single future state at a future time step. The processor may be configured to estimate the planned trajectory by estimating the single future state based on at least one predefined action estimation rule.

[0045] According to a second aspect of the present disclosure, there is provided a computer-implemented method for identifying notable test runs in a real or simulated scenario, comprising:

[0046] receiving a plurality of sets of operating data, each set of operating data being generated in a driving scenario having a self-agent and at least one challenger agent, the self-driving vehicle system controlling the self-agent,

[0047] generating at least one time-indexed event for each set of operational data in response to detected behavior of at least one challenger agent;

[0048] For each set of operating data, a series of decision indicators are generated over multiple time steps of the operating data, the decision indicator at each time step indicating whether a predefined driving action of the ego agent will be permitted, and each driving indicator is generated independently of the autonomous vehicle system controlling the ego agent by:

[0049] Generate a planning trajectory based on the ego agent’s position at time step to a future time step that is independent of the ego agent’s behavior between time step and the future time step, and

[0050] determining whether a predefined driving behavior of the ego-agent will be allowed by estimating the planned trajectory based on at least one predefined driving behavior estimation rule; and

[0051] Store the output groups in the results database.

[0052] The method may further comprise the optional features as defined in the first aspect herein.

[0053] In other implementations, the output set may include at least one performance result as an alternative to the series of decision indicators.

[0054] In one such aspect, a computer system for identifying notable test runs involving an autonomous vehicle system in a real or simulated scenario comprises:

[0055] A processor configured to:

[0056] receiving a plurality of sets of operating data, each set of operating data representing a driving scenario having a self-agent and at least one challenger agent, wherein the self-driving vehicle system controls the self-agent,

[0057] For each set of running data, an output set is generated, which includes:

[0058] at least one time-indexed event generated in response to detected behavior of at least one challenger agent, and

[0059] at least one performance result based on a predefined driving performance rule applied to the ego-agent,

[0060] A data retrieval component is coupled to the results database and configured to retrieve a set of one or more output groups based on at least one time-indexed event and at least one performance result.

[0061] Embodiments of the first and second aspects may also be implemented in the context of with or without performance indicators.

[0062] The present disclosure also provides a non-transitory computer-readable storage medium storing instructions, which, when executed by a computer, perform any of the methods discussed herein. The present disclosure also provides a computer program comprising instructions, which, when executed by a computer, perform any of the methods discussed herein.

[0063] BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to better understand the present disclosure and to show how embodiments of the present disclosure may be implemented, reference is made to the following drawings by way of example only, in which:

[0065] Figure 1A shows a schematic functional block diagram of an autonomous vehicle stack;

[0066] Figure 1B A schematic overview of the autonomous vehicle testing paradigm is shown;

[0067] Figure 1C shows a schematic block diagram of the scene extraction pipeline;

[0068] Figure 2 shows a schematic block diagram of the test pipeline;

[0069] Figure 2A Further details of a possible implementation of the test pipeline are shown;

[0070] Figure 3A shows an example of evaluating a rule tree in a test oracle;

[0071] Figure 3B shows sample output for a rule tree node;

[0072] Figure 4A An example of a rule tree to be evaluated in a test oracle is shown;

[0073] Figure 4B A second example of evaluating a rule tree on a set of scene ground truth data is shown;

[0074] Figure 4C Shows how to selectively apply rules in a test oracle;

[0075] Figure 5A schematic block diagram of a visualization component for presenting a graphical user interface is shown;

[0076] Figure 5A , Figure 5B and Figure 5C shows the different views available in the graphical user interface;

[0077] Fig. 6A A first instance of a cut-in scenario is shown;

[0078] Figure 6B An example oracle output for the first scenario instance is shown;

[0079] Figure 6C A second instance of a cut-through scene is shown;

[0080] Fig.6D An example oracle output is shown for the second scenario instance;

[0081] Figure 7 A schematic block diagram of an example computer system for identifying significant test runs is shown;

[0082] Figure 8 Shows that Figure 7 a predefined driving maneuver estimated by a decision indicator generator of the system;

[0083] Fig. 9 A sample query GUI is shown;

[0084] Fig.10 A sample results GUI is shown;

[0085] Fig.11 An example graphical user interface is displayed showing a selected run with events and decision indicators; and

[0086] Fig.12 is a schematic flow chart of an example method of identifying significant test runs. DETAILED DESCRIPTION

[0087] The described embodiments relate to a test pipeline to facilitate rule-based testing of a mobile robot stack in real or simulated scenarios. The behavior of agents (actors) in real or simulated scenarios is evaluated by a test oracle based on defined performance evaluation rules. These rules can evaluate different aspects of safety. For example, a safety rule group can be defined to estimate the performance of the stack for a specific safety standard, regulation, or safety model (such as RSS), or a customized rule group can be defined to test any aspect of performance. The application of the test pipeline is not limited to safety and can be used to test any aspect of performance, such as comfort or progress towards a set goal. The rule editor allows performance evaluation rules to be defined or modified and passed to the test oracle.

[0088] The "full" stack typically involves everything from processing and interpreting low-level sensor data (perception), to inputting control logic for major high-level functions such as prediction and planning, and generating appropriate control signals to implement planning-level decisions (e.g., to control braking, steering, acceleration, etc.). For autonomous vehicles, a level 3 stack includes some logic to implement transition requirements, and a level 4 stack also includes some logic to implement minimal risk strategies. The stack may also implement auxiliary control functions such as signaling, headlights, windshield wipers, etc.

[0089] The term "stack" may also refer to individual subsystems (substacks) of the entire stack, such as the perception, prediction, planning or control stacks, which may be tested individually or in any desired combination. A stack may refer purely to software, i.e., one or more computer programs that can be executed on one or more general-purpose computer processors.

[0090] Whether real or simulated, scenarios require ego agents to navigate a real or modeled physical context. The ego agent is a real or simulated mobile robot that moves under the control of the stack under test. The physical context includes static and / or dynamic elements to which the stack under test needs to respond effectively. For example, the mobile robot can be a fully or semi-autonomous driving vehicle (ego vehicle) under the control of the stack. The physical context can include a static road layout and a given set of environmental conditions (e.g., weather, time of day, lighting conditions, humidity, pollution / particle levels, etc.), which can be maintained or changed as the scenario progresses. Interactive scenarios also include one or more other agents ("external" agents, such as other vehicles, pedestrians, cyclists, animals, etc.).

[0091] The following examples consider the application of autonomous vehicle testing. However, the principles are equally applicable to other forms of mobile robots.

[0092] Scenarios can be represented or defined at different levels of abstraction. More abstract scenarios accommodate greater degrees of variation. For example, a "cut-in scenario" or "lane change scenario" are examples of highly abstract scenarios that feature a strategy or behavior of interest that accommodates many variations (e.g., different agent starting positions and speeds, road layouts, environmental conditions, etc.). A "scenario run" refers to a specific occurrence of an agent navigating a physical context, optionally in the presence of one or more other agents. For example, multiple runs of a cut-in or lane change scenario (in the real world and / or in a simulator) can be performed with different agent parameters (e.g., starting positions, speeds, etc.), different road layouts, different environmental conditions, and / or different stack configurations, etc. The terms "run" and "instance" are used interchangeably in this context.

[0093] In the following example, the performance of the stack is estimated, at least in part, by evaluating the behavior of a self-agent in a test oracle against a given set of performance evaluation rules over the course of one or more runs. The rules are applied to the "ground truth" of a scenario (or each scenario) run, where, in general, "ground truth" simply means an appropriate representation of the scenario run (including the behavior of the self-agent) that is considered authoritative for testing purposes. Ground truth is inherent to the simulation; the simulator computes a sequence of scenario states that is, by definition, a perfect, authoritative representation of the simulated scenario run. In real-world scenario runs, a "perfect" representation of the scenario run does not exist in the same sense; however, ground truth of appropriate information can be obtained in a variety of ways, such as based on manual annotation of on-board sensor data, automatic / semi-automatic annotation of such data (e.g., using offline / non-real-time processing), and / or using external information sources (e.g., external sensors, maps, etc.).

[0094] The scene ground truth typically consists of a "trace" of the ego agent and any other applicable (salient) agents. A trace is the history of the agent's position and motion over the course of the scene. There are many ways to represent a trace. Trace data typically includes spatial and motion data for agents in the environment. The term is used for both real scenes (with real-world traces) and simulated scenes (with simulated traces). The trace typically records the actual trajectories achieved by the agents in the scene. For terminology, "trace" and "trajectory" can contain the same or similar types of information (e.g. a series of spatial and motion states over time). In the context of planning, the term trajectory is often preferred (which can refer to future / predicted trajectories), while in the context of testing / evaluation, the term trace is often preferred for past behavior.

[0095] In the simulation context, a "scenario description" is provided as input to the simulator. For example, the scenario description can be encoded using a scenario description language (SDL), or in any other form that can be used by the simulator. A scenario description is typically a more abstract representation of a scenario that can generate multiple simulation runs. Depending on the implementation, the scenario description can have one or more configurable parameters that can be changed to increase the degree of possible variation. The degree of abstraction and parameterization is a design choice. For example, a scenario description can encode a fixed layout with parameterized environmental conditions (such as weather, lighting, etc.). However, further abstraction is possible, for example, using configurable road parameters (such as road curvature, lane configuration, etc.). The input to the simulator includes a scenario description and a set of selected parameter values ​​(if applicable). The latter can be referred to as the parameterization of the scenario. The configurable parameters define a parameter space (also called a scenario space), and the parameterization corresponds to a point in the parameter space. In this context, a "scenario instance" can refer to an instantiation of a scenario in a simulator based on a scenario description and (if applicable) the selected parameterization.

[0096] For simplicity, the word scenario is also used to refer to a scenario run, as well as a scenario in a more abstract sense. The meaning of the term scenario will become clear from the context in which it is used.

[0097] Trajectory planning is an important function in the present context, and the terms "trajectory planner", "trajectory planning system", and "trajectory planning stack" are used interchangeably in this article to refer to one or more components that can plan future trajectories for a mobile robot. Trajectory planning decisions ultimately determine the actual trajectory achieved by the self-agent (although in some test contexts this may be influenced by other factors, such as the implementation of these decisions in the control stack, and the true or modeled dynamic response of the self-agent to the final control signals).

[0098] The trajectory planner can be tested alone or in combination with one or more other systems (e.g., perception, prediction, and / or control). In a full stack, planning typically refers to higher-level autonomous decision-making capabilities (such as trajectory planning), while control typically refers to lower-level control signal generation used to execute those autonomous decisions. However, in the context of performance testing, the term "control" is also used in a broader sense. For the avoidance of doubt, when a trajectory planner is said to control the self-agent in simulation, this does not necessarily mean that the control system (in the narrow sense) is tested along with the trajectory planner.

[0099] Example AV stack:

[0100] To provide relevant context for the described embodiments, further details of an example form of an AV stack will now be described.

[0101] Figure 1A A highly schematic block diagram of an AV runtime stack 100 is shown. The runtime stack 100 is shown to include a perception (sub)system 102, a prediction (sub)system 104, a planning (sub)system (planner) 106, and a control (sub)system (controller) 108. As mentioned, the term (sub)stack may also be used to describe the aforementioned components 102-108.

[0102] In the real-world context, the perception system 102 receives sensor outputs from the AV's onboard sensor system 110 and uses these sensor outputs to detect external agents and measure their physical states, such as their position, velocity, acceleration, etc. The onboard sensor system 110 can take different forms, but typically includes various sensors, such as image capture devices (cameras / optical sensors), lidar and / or radar units, satellite positioning sensors (GPS (Global Position System, Global Positioning System)), motion / inertial sensors (accelerometers, gyroscopes, etc.), etc. Therefore, the onboard sensor system 110 provides rich sensor data from which detailed information about the surrounding environment and the state of the AV and any external actors in the environment (vehicles, pedestrians, cyclists, etc.) can be extracted. The sensor outputs typically include sensor data of multiple sensor modalities, such as stereo images from one or more stereo optical sensors, lidar, radar, etc. Sensor data of multiple sensor modalities can be combined using filters, fusion components, etc.

[0103] The perception system 102 typically includes multiple perception components that cooperate to interpret sensor output to provide perception output to the prediction system 104 .

[0104] In a simulation context, depending on the nature of the test, and in particular on where the stack 100 is "sliced" for testing purposes (see below), it may or may not be necessary to model the onboard sensor system 100. For higher level slices, simulated sensor data is not required, and therefore complex sensor modeling is not required.

[0105] The prediction system 104 uses the perception output from the perception system 102 to predict the future behavior of external actors (agents), such as other vehicles in the vicinity of the AV.

[0106] The predictions calculated by the prediction system 104 are provided to the planner 106, which uses the predictions to make autonomous driving decisions to be performed by the AV in a given driving scenario. The input received by the planner 106 will generally indicate a drivable area, and will also capture the predicted movements of any external agents (from the AV's perspective, obstacles) within the drivable area. The drivable area can be determined using the perception output from the perception system 102 in combination with map information (e.g., a high definition (HD) map).

[0107] The core function of the planner 106 is to plan the trajectory of the AV (ego trajectory) taking into account the predicted agent motion. This can be referred to as trajectory planning. The trajectory is planned in order to achieve a desired goal in the scenario. For example, the goal can be: entering a roundabout and leaving the roundabout at a desired exit; overtaking the vehicle in front; or staying in the current lane at a target speed (lane following). For example, this goal can be determined by an autonomous route planner (not shown).

[0108] The controller 108 executes the decisions made by the planner 106 by providing appropriate control signals to the AV's onboard participant systems 112. In particular, the planner 106 plans a trajectory for the AV, and the controller 108 generates control signals to implement the planned trajectory. Typically, the planner 106 will plan into the future, so that the planned trajectory may only be partially implemented at the control level before the planner 106 plans a new trajectory. The participant systems 112 include "primary" vehicle systems, such as braking, acceleration, and steering systems, as well as auxiliary systems (e.g., signaling, wipers, headlights, etc.).

[0109] Note that there may be a distinction between the planned trajectory at a given moment and the actual trajectory followed by the ego-agent. Planning systems typically operate on a series of planning steps, updating the planned trajectory at each planning step to account for any changes in the scene since the previous planning step (or, more precisely, any changes away from predicted changes). The planning system 106 may reason into the future so that the planned trajectory for each planning step extends beyond the next planning step. Therefore, any individual planned trajectory may not be fully realized (if the planning system 106 is tested in isolation, in simulation, the ego-agent may simply follow the planned trajectory exactly until the next planning step; however, as described above, in other real and simulated contexts, the planned trajectory may not be followed exactly to the next planning step because the behavior of the ego-agent may be affected by other factors, such as the operation of the control system 108 and the real or modeled dynamics of the ego-vehicle). In many testing contexts, the actual trajectory of the ego-agent is what matters; in particular, whether the actual trajectory is safe, as well as other factors such as comfort, progress, etc. However, the rule-based testing methods herein can also be applied to planned trajectories (even if those planned trajectories are not fully or accurately realized by the ego-agent). For example, even if the agent's actual trajectory is considered safe according to a given set of safety rules, it may be that the momentarily planned trajectory is unsafe; the fact that the planner 106 is considering an unsafe course of action may reveal that, even if it does not result in unsafe agent behavior in the scene. In addition to the actual agent behavior in the simulation, the momentarily planned trajectory constitutes one form of internal state that can be effectively evaluated. Other forms of internal stack state can be similarly evaluated.

[0110] Figure 1A The example considers a relatively "modular" architecture, with separable perception, prediction, planning, and control systems 102-108. The sub-stacks themselves may also be modular, such as having separable planning modules within the planning system 106. For example, the planning system 106 may include multiple trajectory planning modules that may be applied in different physical contexts (e.g., simple lane driving versus complex intersections or roundabouts). For the reasons discussed above, this is relevant to simulation testing as it allows components (e.g., the planning system 106 or its individual planning modules) to be tested individually or in different combinations. For the avoidance of doubt, with respect to a modular stack architecture, the term stack may refer not only to the entire stack, but also to any individual subsystem or module thereof.

[0111] The degree to which the various stack functionalities are integrated or separated may vary widely between different stack implementations - in some stacks, certain aspects may be so tightly coupled that they are indistinguishable. For example, in other stacks, planning and control may be integrated (e.g., such a stack may perform planning directly based on control signals), while in other stacks (e.g., Figure 1AThe stack shown in ) can be constructed in a way that draws a clear distinction between the two (e.g., with planning based on trajectory, and with separate control optimization to determine how to best execute the planned trajectory at the control signal level). Similarly, in some stacks, prediction and planning may be more tightly coupled. In the extreme case, in so-called "end-to-end" driving, perception, prediction, planning, and control may be essentially inseparable. Unless otherwise stated, the terminology of perception, prediction planning, and control used in this document does not imply any particular coupling or modularity of these aspects.

[0112] It should be understood that the term "stack" includes software, but may also include hardware. In simulation, the software of the stack may be tested on a "generic" off-board computer system before ultimately being uploaded to the on-board computer system of a physical vehicle. However, in "hardware-in-the-loop" testing, testing may be expanded to the underlying hardware of the vehicle itself. For example, for testing purposes, the stack software may be run on an on-board computer system (or a replica thereof) coupled to a simulator. In this context, the stack under test is expanded to the underlying computer hardware of the vehicle. As another example, certain functions of the stack 110 (e.g., perception functions) may be implemented in dedicated hardware. In a simulation context, hardware-in-the-loop testing may involve feeding synthetic sensor data to a dedicated hardware perception component.

[0113] Test example:

[0114] Figure 1B A highly schematic overview of an autonomous vehicle testing paradigm is shown. By running multiple scenario instances in the simulator 202 and evaluating the performance of the stack 100 (and / or its various sub-stacks) in the test oracle 252, the ADS / ADAS stack 100 (e.g. Figure 1AThe improved stack 100 is then subjected to repeated testing and evaluation in simulation. The output of the test oracle 252 provides information to the experts 122 (team or individual), allowing them to identify problems in the stack 100 and modify the stack 100 to mitigate these problems (S124). The results also help the experts 122 select further test scenarios (S126), and the process continues, repeatedly modifying, testing, and evaluating the performance of the stack 100 in simulation. The improved stack 100 is ultimately incorporated (S125) into a real-world AV 101 equipped with a sensor system 110 and a participant system 112. The improved stack 100 typically includes program instructions (software) executed in one or more computer processors of an onboard computer system (not shown) of the vehicle 101. At step S125, the software of the improved stack is uploaded to the AV 101. Step S125 may also involve modifications to the underlying vehicle hardware. On the AV 101, the improved stack 100 receives sensor data from the sensor system 110 and outputs control signals to the participant system 112. Real-world testing (S128) can be used in conjunction with simulation-based testing. For example, once acceptable performance levels have been achieved through the process of simulation testing and stack refinement, appropriate real-world scenarios can be selected (S130), and the performance of the AV 101 in these real-world scenarios can be captured and similarly evaluated in the test oracle 252.

[0115] The scenarios for simulation purposes can be obtained in various ways, including manual coding. The system can also extract scenarios for simulation purposes from real-world operations, allowing real-world situations and their variations to be recreated in the simulator 202.

[0116] Figure 1CA highly schematic block diagram of a scene extraction pipeline is shown. Data 140 of real-world operations is passed to a "ground truth" pipeline 142 for generating scene ground truth. Operation data 140 may include, for example, sensor data and / or sensory output captured / generated on one or more vehicles (which may be autonomous, human-driven, or a combination thereof), and / or data captured from other sources such as external sensors (CCTV (closed circuit television, closed-circuit television) etc.). Operation data is processed within the ground truth pipeline 142 to generate appropriate ground truth 144 (traces and background data) for real-world operations. As discussed, the ground truth process can be based on manual annotation of the "raw" operation data 140, or the process can be fully automated (e.g., using an offline perception method), or a combination of manual and automatic ground truth can be used. For example, a 3D bounding box can be placed around the vehicles and / or other agents captured in the operation data 140 to determine the spatial and motion state of their traces. A scene extraction component 146 receives the scene ground truth 144 and processes the scene ground truth 144 to extract a more abstract scene description 148 that can be used for simulation purposes. The scene description 148 is used by the simulator 202 to allow multiple simulation runs to be performed. A simulation run is a variation of the original real-world run, and the degree of variation possible is determined by the level of abstraction. A ground truth 150 is provided for each simulation run.

[0117] In current off-board content, there is no requirement to extract traces in real time (or, more accurately, in a manner that supports real-time planning); instead, traces are extracted "offline." Examples of offline perception algorithms include non-real time and non-causal perception algorithms. Offline techniques are contrasted with "online" techniques that can be feasibly implemented within the AV stack 100 to facilitate real-time planning / decision making.

[0118] For example, non-real-time processing can be used that cannot be performed online due to hardware or other practical constraints of the AV onboard computer system. For example, one or more non-real-time perception algorithms can be applied to the real-world operating data 140 to extract traces. A non-real-time perception algorithm can be an algorithm that cannot be run in real time due to the computational or storage resources required.

[0119] In this context, "non-causal" perception algorithms may also be used. A non-causal algorithm may or may not run in real time when executed, but in any case cannot be implemented in an online context because it requires knowledge of the future. For example, a perception algorithm that detects the state of an agent (e.g., position, posture, velocity, etc.) at a particular moment in time based on subsequent data cannot support real-time planning within the stack 100 in an online context because it requires knowledge of the future (unless it is constrained to operate with a short look ahead window). For example, filtering using a backwards pass is a non-causal algorithm that can sometimes run in real time, but requires knowledge of the future.

[0120] The term "perception" generally refers to techniques for perceiving structure in real-world data 140, such as 2D or 3D bounding box detection, position detection, pose detection, and motion detection, etc. For example, a track may be extracted as a time series of bounding boxes or other spatial states in 3D space or 2D space (e.g., in a bird's-eye reference frame) with associated motion information (e.g., velocity, acceleration, jerk, etc.). In the context of image processing, such techniques are often categorized as "computer vision," but the term perception encompasses a wider range of sensor modalities.

[0121] Test pipeline:

[0122] The details of the test pipeline and the test oracle 252 will now be described in further detail. The following examples focus on simulation-based testing. However, as described above, the test oracle 252 can also be applied to evaluate stack performance in real scenarios, and the following related descriptions are also applicable to real scenarios. The following descriptions refer to the examples by way of example. Figure 1A However, as described above, the test pipeline 200 is highly flexible and can be applied to any stack or sub-stack running at any level of automation.

[0123] Figure 2 A schematic block diagram of a test pipeline, denoted by reference numeral 200, is shown. The test pipeline 200 is shown to include a simulator 202 and a test oracle 252. The simulator 202 runs simulation scenarios for testing all or part of the AV runtime stack 100, and the test oracle 252 evaluates the performance of the stack (or sub-stack) on the simulation scenarios. As discussed, it is possible to test a sub-stack of the runtime stack, but for simplicity, the following description refers to the (full) AV stack 100 throughout. However, the description is equally applicable to a sub-stack instead of the full stack 100. The term "slice" is used herein to select a set of stack components or a subset of stack components for testing.

[0124] As mentioned before, the idea of ​​simulation-based testing is to run a simulated driving scenario that the ego agent must navigate under the control of the stack 100 being tested. Typically, the scenario includes a static drivable area (e.g., a specific static road layout) that the ego agent needs to navigate, typically with one or more other dynamic agents (e.g., other vehicles, bicycles, pedestrians, etc.). To this end, simulated input 203 is provided from a simulator 202 to the stack 100 under test.

[0125] The slices of the stack dictate the form of the simulation input 203. For example, Figure 2 The prediction, planning and control systems 104, 106 and 108 within the AV stack 100 being tested are shown. Figure 1A The perception system 102 may also be applied during testing of a full AV stack. In this case, the simulated input 203 will include synthetic sensor data, which is generated using appropriate sensor models and processed in the same manner as real sensor data within the perception system 102. This requires generating sufficiently realistic synthetic sensor inputs (e.g., photorealistic image data and / or equally realistic simulated lidar / radar data, etc.). The resulting output of the perception system 102 will in turn be fed into the higher-level prediction and planning systems 104, 106.

[0126] In contrast, so-called "planning-level" simulations essentially bypass perception system 102. Simulator 202 will instead provide simpler, higher-level inputs 203 directly to prediction system 104. In some contexts, it may even be appropriate to bypass prediction system 104 in order to test planner 106 based on predictions obtained directly from simulated scenarios (i.e., "perfect" predictions).

[0127] In between these extremes, there is a range of different levels of input slicing, such as testing only a subset of the perception system 102, such as "later" (higher level) perception components, such as components (such as filters or fusion components) that operate on outputs from lower level perception components (e.g., object detectors, bounding box detectors, motion detectors, etc.).

[0128] Regardless of the form they take, the simulated inputs 203 are used (directly or indirectly) as the basis for the decisions made by the planner 108. The controller 108, in turn, implements the planner's decisions by outputting control signals 109. In the real-world context, these control signals will drive the AV's physical actor systems 112. In the simulation, the ego vehicle dynamics model 204 is used to convert the resulting control signals 109 into real movements of the ego agent in the simulation, thereby simulating the physical response of the autonomous vehicle to the control signals 109.

[0129] Alternatively, a simpler form of simulation assumes that the ego agent follows each planned trajectory exactly between planning steps. This approach bypasses the control system 108 (to the extent that it can be separated from planning) and eliminates the need for the ego vehicle dynamics model 204. This may be sufficient to test certain aspects of planning.

[0130] To the extent that the external agent exhibits automated behavior / decisions within the simulator 202, some form of agent decision logic 210 is implemented to execute those decisions and determine the agent behavior within the scenario. The agent decision logic 210 can be comparable in complexity to the self stack 100 itself, or it can have more limited decision-making capabilities. The goal is to provide sufficiently realistic external agent behavior within the simulator 202 so that the decision-making capabilities of the self stack 100 can be effectively tested. In some contexts, this does not require any agent decision logic 210 at all (open-loop simulation), and in other contexts, relatively limited agent logic 210 (such as basic adaptive cruise control (ACC)) can be used to provide useful testing. If appropriate, one or more agent dynamic models 206 can be used to provide more realistic agent behavior.

[0131] The scenario is run according to the scenario description 201a and (if applicable) the selected parameterization 201b of the scenario. A scenario typically has static and dynamic elements, which may be "hard-coded" in the scenario description 201a or configurable and thus determined by the scenario description 201a in combination with the selected parameterization 201b. In a driving scenario, the static elements typically include a static road layout.

[0132] Dynamic elements typically include one or more external agents in the scene, such as other vehicles, pedestrians, and bicycles.

[0133] The degree of dynamic information provided to the simulator 202 for each external agent can vary. For example, a scenario can be described by a separable static layer and a dynamic layer. A given static layer (e.g., defining a road layout) can be used in combination with different dynamic layers to provide different scenario instances. For each external agent, the dynamic layer can include a spatial path for the agent to follow, and one or both of motion data and behavior data associated with the path. In a simple open-loop simulation, the external actor simply follows the spatial path and motion data defined in the dynamic layer, which is non-reactive, i.e., does not react to the self-agent in the simulation. Such an open-loop simulation can be implemented without any agent decision logic 210. However, in a closed-loop simulation, the dynamic layer instead defines at least one behavior (e.g., ACC behavior) to be followed along the static path. In this case, the agent decision logic 210 implements the behavior in the simulation in a reactive manner, i.e., reacts to the self-agent and / or other external agents. The motion data may still be associated with the static path, but in this case is less standardized, for example, it may be used as a target along the path. For example, for ACC behavior, a target speed may be set along the path that the agent will seek to match, but the agent decision logic 210 may be allowed to reduce the speed of the external agent below the target at any point along the path to maintain the target spacing from the vehicle ahead.

[0134] It will be appreciated that a scenario with any degree of configurability may be described in a variety of ways for simulation purposes. For example, the number and type of agents and their motion information may be configured as part of the scenario parameterization 201b.

[0135] The output of the simulator 202 for a given simulation includes an ego trace 212a for the ego agent and one or more agent traces 212b for one or more external agents (traces 212). Each trace 212a, 212b is a complete history of the agent's behavior in the simulation, which contains spatial and motion components. For example, each trace 212a, 212b can take the form of a spatial path with motion data associated with points along the path, such as velocity, acceleration, jerk (rate of change of acceleration), snap (rate of change of jerk), etc.

[0136] Additional information is also provided to supplement and provide context for the trace 212. This additional information is referred to as "background" data 214. The background data 214 relates to the physical context of the scene and may have both static components (e.g., road layout) and dynamic components (e.g., weather conditions, the extent to which they change during the simulation). To some extent, the background data 214 may be "transitive" in that it is directly defined by the choice of the scene description 201a or parameterization 201b and is therefore not affected by the simulation results. For example, the background data 214 may include a static road layout directly from the scene description 201a or parameterization 201b. However, typically, the background data 214 will include at least some elements derived within the simulator 202. This may, for example, include simulated environmental data, such as weather data, where the simulator 202 may be free to change weather conditions as the simulation progresses. In this case, the weather data may be time-dependent, and this time dependence is reflected in the background data 214.

[0137] The test oracle 252 receives the trace 212 and the background data 214 and scores these outputs according to a set of performance evaluation rules 254. The performance evaluation rules 254 are shown as being provided to the test oracle 252 as input.

[0138] The rules 254 are categorical in nature (e.g., pass / fail type rules). Certain performance evaluation rules are also associated with a numerical performance metric used to "score" the trajectory (e.g., indicating the degree of success or failure or some other quantity that helps explain or otherwise relates to the classification results). The evaluation of the rules 254 is time-based—a given rule may have different effects at different points in the scenario. Scoring is also time-based: for each performance evaluation metric, the test oracle 252 tracks how the value (score) of the metric changes over time as the simulation progresses. The test oracle 252 provides an output 256, which includes: a time series 256a of the classification (e.g., pass / fail) results for each rule, and a score-time graph 256b for each performance metric, as described in further detail below. The results and scores 256a, 256b provide information to the expert 122 and can be used to identify and mitigate performance issues within the test stack 100. The test oracle 252 also provides an overall (aggregate) result for the scenario (e.g., overall pass / fail). The output 256 of the test oracle 252 is stored in the test database 258 in association with information about the scenario to which the output 256 belongs. For example, the output 256 can be stored in association with the scenario description 210a (or its identifier) ​​and the selected parameterization 201b. In addition to the time-related results and scores, an overall score can also be assigned to the scenario and stored as part of the output 256. For example, an aggregate score for each rule (e.g., overall pass / fail) and / or an aggregate result of all rules 254 (e.g., pass / fail).

[0139] Figure 2A Another option for slicing is shown, and reference numerals 100 and 100S are used to represent the full stack and the sub-stack, respectively. The sub-stack 100S will be Figure 2 The test pipeline 200 is subjected to testing.

[0140] A number of "later" perceptual components 102B form part of the sub-stack 100S to be tested, and during testing are applied to simulated perceptual inputs 203. The later perceptual components 102B may include, for example, filtering or other fusion components that fuse perceptual inputs from multiple earlier perceptual components.

[0141] In the full stack 100, the later perception component 102B receives actual perception input 213 from the earlier perception component 102A. For example, the earlier perception component 102A may include one or more 2D or 3D bounding box detectors, in which case the simulated perception input provided to the later perception component may include simulated 2D or 3D bounding box detections derived via ray tracing in the simulation. The earlier perception component 102A typically includes components that operate directly on sensor data. Figure 2A, the simulated perceptual input 203 will correspond in form to the actual perceptual input 213 typically provided by the earlier perceptual component 102A. However, the earlier perceptual component 102A is not applied as part of the test, but is used to train one or more perceptual error models 208 that can be used to introduce realistic errors into the simulated perceptual input 203 in a statistically rigorous manner, which is fed to the later perceptual component 102B of the test sub-stack 100.

[0142] Such perceptual error models may be referred to as Perception Statistical Performance Models (PSPM), or synonyms "PRISM". Further details of the principles of PSPM and suitable techniques for building and training them can be found in International Patent Publication Nos. WO2021037763, WO2021037760, WO2021037765, WO2021037761, and WO2021037766, the entire contents of each of which are incorporated herein by reference. The idea behind PSPM is to effectively introduce realistic errors into the simulated perceptual inputs provided to the sub-stack 100S (i.e., reflecting the expected error types that are to be applied to the earlier perceptual components 102A in the real world). In the simulated context, "perfect" ground truth perceptual inputs 203G are provided by the simulator, but these are used to derive more realistic (ablated) perceptual inputs 203 with real errors introduced by the perceptual error model 208. Perceptual error model 208 acts as a "proxy model" (acting as a proxy for perceptual system 102, or a portion of perceptual system 102A, but operating on lower fidelity inputs).

[0143] As described in the above references, the PSPM may depend on one or more variables representing physical conditions ("disturbance factors"), thereby allowing different levels of error to be introduced to reflect different possible real-world conditions. Thus, simulator 202 may simulate different physical conditions (e.g., different weather conditions) by simply changing the value of the weather disturbance factor, which will in turn change how the perception error is introduced.

[0144] The later perception components 102b within the sub-stack 100S process the simulated perception inputs 203 in exactly the same way as they process the real-world perception inputs 213 within the full stack 100, and their outputs in turn drive prediction, planning, and control.

[0145] Alternatively, PRISM may be used to model the entire perception system 102 , including the late perception component 208 , in which case PSPM is used to generate realistic perception output that is passed directly as input to the prediction system 104 .

[0146] Depending on the implementation, there may or may not be a deterministic relationship between a given scene parameterization 201b and the simulation results for a given stack 100 configuration (i.e., the same parameterization may or may not always result in the same results for the same stack 100). Non-determinism can occur in various ways. For example, when the simulation is based on PRISM, PRISM can model the distribution of possible perceptual outputs at each given time step of the scene and probabilistically sample the real perceptual outputs from them. This leads to non-deterministic behavior within the simulator 202, whereby different results can be obtained for the same stack 100 and scene parameterization because different perceptual outputs are sampled. Alternatively or additionally, the simulator 202 may be inherently non-deterministic, for example, weather, lighting or other environmental conditions may be somewhat random / probabilistic within the simulator 202. It will be appreciated that this is a design choice: in other implementations, the varying environmental conditions may be fully specified in the parameterization 201b of the scene. For non-deterministic simulations, multiple scene instances may be run for each parameterization. An aggregated pass / fail result may be assigned to a particular selection of parameterization 201b, for example as a count or percentage of pass or fail results.

[0147] Test orchestration component 260 is responsible for selecting scenarios for simulation purposes. For example, test orchestration component 260 can automatically select scenario description 201a and appropriate parameterization 201b, which can be based on test oracle output 256 from previous scenarios and / or other criteria.

[0148] Test oracle rules:

[0149] The performance evaluation rules 254 are constructed as a computational graph (rule tree) to be applied in the test oracle. Unless otherwise specified, the term "rule tree" herein refers to a computational graph configured to implement a given rule. Each rule is constructed as a rule tree, and a group of multiple rules can be referred to as a "forest" of multiple rule trees.

[0150] Figure 3A An example of a rule tree 300 constructed from a combination of extractor nodes (leaf objects) 302 and estimator nodes (non-leaf objects) 304 is shown. Each extractor node 302 extracts a time-varying numerical (e.g., floating point) signal (score) from a set of scene data 310. The scene data 310 is a form of scene ground truth in the sense described above and may be referred to as such. The scene data 310 has been generated by deploying a trajectory planner (such as a Figure 1A 106) and is displayed as including ego and agent traces 212 and background data 214. Figure 2 or Figure 2AIn the context of simulation, scene ground truth 310 is provided as an output of simulator 202. When applied to real-world data, scene ground truth 310 may be provided by ground truth pipeline 142.

[0151] Each estimator node 304 is shown as having at least one child object (node), where each child object is one of the extractor nodes 302 or another of the estimator nodes 304. Each estimator node receives outputs from its child nodes and applies an estimator function to these outputs. The output of the estimator function is a time series of classification results. The following example considers a simple binary pass / fail result, but the technology can be easily expanded to non-binary results. Each estimator function estimates the output of its child nodes according to predefined atomic rules. These rules can be flexibly combined according to the desired security model.

[0152] Furthermore, each estimator node 304 derives a time-varying digital signal (robustness score) from the output of its child nodes, which is related to the classification result via a threshold condition (see below).

[0153] The top-level root node 304a is an estimator node that is not a child of any other node. The top-level node 304a outputs a final series of results, and its descendants (ie, direct or indirect child nodes of the top-level node 304a) provide underlying signals and intermediate results.

[0154] Figure 3B Visually depicted is an example of a derived signal 312 and a corresponding result time series 314 calculated by the estimator node 304. The result 314 is related to the derived signal 312 in that a pass result is returned when (and only when) the derived signal exceeds a fail threshold 316. It will be appreciated that this is merely one example of a threshold condition that relates a time series of results to a corresponding signal.

[0155] The signal extracted directly from the scene ground truth 310 by the extractor node 302 may be referred to as a "raw" signal to distinguish it from the "derived" signal computed by the estimator node 304. The resulting and raw / derived signals may be discretized in time.

[0156] Figure 4A An example of a rule tree implemented within the test platform 200 is shown.

[0157] A rule editor 400 is provided for constructing rules to be implemented with the test oracle 252. The rule editor 400 receives rule creation input from a user (who may or may not be an end user of the system). In this example, the rule creation input is encoded in a domain specific language (DSL) and defines at least one rule graph 408 to be implemented in the test oracle 252. In the example below, the rules are logical rules, with TRUE and FALSE representing pass and fail, respectively (it will be appreciated that this is purely a design choice).

[0158] The following example considers rules formulated using a combination of atomic logical predicates. Examples of basic atomic predicates include basic logic gates (OR, AND, etc.) and logical functions such as "greater than" (Gt(a, b)) (returns TRUE when a is greater than b, otherwise returns false).

[0159] The Gt function is to implement the safe lateral distance rule between the ego agent and another agent (with agent identifier "other_agent_id") in the scene. Two extractor nodes (latd, latsd) apply the Lateral Distance (latd) and Lateral Safe Distance (latsd) extractor functions, respectively. These functions operate directly on the scene ground truth 310 to extract the time-varying lateral distance signal (measuring the lateral distance between the ego agent and the identified other agent) and the time-varying safe lateral distance signal of the ego agent and the identified other agent, respectively. The safe lateral distance signal can depend on various factors, such as the speed of the ego agent and the speed of the other agent (captured in the trace 212), as well as the environmental conditions captured in the background data 214 (e.g., weather, lighting, road type, etc.).

[0160] The estimator node (is_latd_safe) is the parent node of the latd and latsd extractor nodes and maps to the Gt atomic predicate. Therefore, when implementing the rule tree 408, the is_latd_safe estimator node applies the Gt function to the output of the latd and latsd extractor nodes to calculate a true / false result for each time step of the scene, returning true for each time step where the latd signal exceeds the latsd signal, and false otherwise. In this way, a "safe lateral distance" rule is constructed based on the atomic extractor function and predicate; when the lateral distance reaches or falls below the safe lateral distance threshold, the self-agent does not comply with the safe lateral distance rule. It will be appreciated that this is a very simple example of a rule tree. Arbitrarily complex rules can be constructed based on the same principles.

[0161] The test oracle 252 applies the rule tree 408 to the scene ground truth 310 and provides the results through a user interface (UI) 418.

[0162] Figure 4B shows an example rule tree containing Figure 4A corresponding lateral distance branch. In addition, the rule tree includes a longitudinal distance branch and a top-level OR predicate (safe distance node, is_d_safe) to implement the safe distance metric. Similar to the lateral distance branch, the longitudinal distance branch extracts the longitudinal distance (lond) and the longitudinal distance threshold signal (lonsd) from the scene data (extractor nodes lond and lonsd, respectively), and the longitudinal safety estimator node (is_lond_safe) returns TRUE when the longitudinal distance is above the safe longitudinal distance threshold. The top-level OR node returns TRUE when one or both of the lateral and longitudinal distances are safe (below the applicable threshold), and returns FALSE if both are unsafe. In this context, it is sufficient for one of the distances to exceed the safety threshold (for example, if two vehicles are driving in adjacent lanes, their longitudinal separation is zero or close to zero when they are side by side; but if these vehicles have sufficient lateral separation, this situation is not unsafe).

[0163] For example, the numerical output of the top-level node may be a time-varying robustness score.

[0164] Different rule trees can be constructed, for example, to implement different rules for a given safety model, to implement different safety models, or to selectively apply rules to different scenarios (not every rule in a given safety model must apply to every scenario; using this approach, different rules or combinations of rules can be applied to different scenarios). Within this framework, rules can also be constructed for evaluating comfort (e.g., based on instantaneous acceleration and / or jerkiness along a trajectory), progress (e.g., based on the time taken to reach a specified goal), etc.

[0165] The above examples considered simple logical predicates that evaluate results or signals at a single moment in time, such as OR, AND, Gt, etc. However, in practice it may be desirable to formulate certain rules in terms of temporal logic.

[0166] Hekmatnejad et al., "Encoding and Monitoring Responsibility Sensitive Safety Rules for Automated Vehicles in Signal Temporal Logic" (2019), MEMOCODE'19: Proceedings of the 17th ACM-IEEE International Conference on Formal Methods and Models for System Design (incorporated herein by reference in its entirety), discloses a signal temporal logic (STL) encoding of RSS safety rules. Temporal logic provides a formal framework to construct predicates that conform to time. This means that the result computed by an estimator at a given moment can depend on the result and / or signal value at another moment.

[0167] For example, a requirement of a security model might be that the self-agent responds to a certain event within a set time frame. Such rules can be encoded in a similar way, using temporal logic predicates in a rule tree.

[0168] In the above example, the performance of the stack 100 is evaluated at each time step of the scenario. An overall test result (e.g., pass / fail) can be derived from this - for example, if a rule (e.g., a safety-critical rule) fails at any time step in the scenario, then the rule may cause an overall failure (that is, the rule must pass at every time step to obtain an overall pass for the scenario). For other types of rules, the overall pass / fail criteria may be "softer" (e.g., if a rule fails for a certain number of consecutive time steps, then the rule may only be triggered to fail), and such criteria may be context-dependent.

[0169] Figure 4C A hierarchy of rule evaluation implemented in a test oracle 252 is schematically depicted. A set of rules 254 is received for implementation in a test oracle 252.

[0170] Some rules apply only to the ego agent (an example is a comfort rule that estimates whether the ego trajectory exceeds some maximum acceleration or jerk threshold at any given moment).

[0171] Other rules involve the interaction of the ego agent with other agents (e.g., the "no collision" rule or the safe distance rule considered above). Each such rule is evaluated in a pairwise manner between the ego agent and each other agent. As another example, a "pedestrian emergency braking" rule might be activated only when a pedestrian walks in front of the ego vehicle, and only for that pedestrian agent.

[0172] Not every rule has to apply to every scenario, and some rules may only apply to a portion of a scenario. Rule activation logic 422 within the test oracle 422 determines whether and when each rule 254 applies to the scenario in question, and selectively activates the rules when they apply. Thus, a rule may remain activated throughout a scenario, may never be activated in a given scenario, or may only be activated in certain scenarios. Furthermore, rules may be evaluated for different numbers of agents at different points in a scenario. Selectively activating rules in this way can significantly improve the efficiency of the test oracle 252.

[0173] The activation or deactivation of a given rule may be dependent on the activation / deactivation of one or more other rules. For example, when the pedestrian emergency braking rule is activated, the "optimal comfort" rule may be considered inapplicable (because the safety of pedestrians is the primary concern), and the former may be deactivated when the latter is activated.

[0174] The rule evaluation logic 424 performs evaluations on each activated rule for any period of time that it remains active.Each interaction rule is evaluated in a pairwise fashion between the ego agent and any other agent to which it applies.

[0175] There may also be a degree of interdependence in the application of these rules. For example, another way to resolve the relationship between the comfort rule and the emergency braking rule is to increase the jerk / acceleration threshold of the comfort rule whenever the emergency braking rule is activated for at least one other agent.

[0176] While pass / fail outcomes have been considered, rules may be non-binary. For example, two categories of failures may be introduced - "acceptable" and "unacceptable". Again, considering the relationship between the comfort rule and the emergency braking rule, an acceptable failure of the comfort rule may occur when the rule fails but the emergency braking rule is effective. Therefore, interdependencies between rules can be handled in different ways.

[0177] The activation criteria for the rules 254 may be specified in the rule creation code provided to the rule editor 400, as well as the nature of any rule interdependencies and the mechanisms used to implement those interdependencies.

[0178] Graphical User Interface:

[0179] Figure 5 A schematic block diagram of a visualization component 520 is shown. The visualization component is shown as having an input connected to the test database 258 for presenting the output 256 of the test oracle 252 on a graphical user interface (GUI) 500. The GUI is presented on a display system 522.

[0180] Figure 5AAn example view of the GUI 500 is shown. The view relates to a particular scenario involving multiple agents. In this example, the test oracle output 526 relates to multiple external agents, and the results are organized by agent. For each agent, there is a time series of results for each rule that applies to that agent at a point in time in the scenario. In the example described, the summary view has been selected for "Agent 01" so that the "top level" results of the calculations are displayed for each applicable rule. The root node of each rule tree has the top level calculation results. Color coding is used to distinguish between time periods when the rules were inactive, active and passed, and active and failed for that agent.

[0181] A first optional element 534a is provided for the result of each time series. This allows access to lower level results of the rule tree, ie results calculated at lower positions in the rule tree.

[0182] Figure 5B The first expanded view of the results for "Rule 02" is shown, where the results for lower level nodes are also visualized. Figure 4B The results of the "safe distance" rule, the "is_latd_safe node" and the "is_lond_safe node" can be visualized (in Figure 5B In the first expanded view of rule 02, you can see that the success / failure on rule 02 is defined by the logical OR relationship between results C1 and C2; rule 02 fails only when both C1 and C2 fail (like the "safe distance" rule above).

[0183] A second selectable element 534b is provided for each time series of results, which allows access to the associated numerical performance score.

[0184] Figure 5C A second expanded view is shown where the results for Rule 02 and the "C1" results have been expanded to reveal the relative scores of those rules for the time period that Agent 01 was activated. The scores are displayed as a visual score-versus-time graph that is similarly color coded to indicate pass / fail.

[0185] Example scenario:

[0186] Fig. 6AA first instance of a cut-in scenario in the simulator 202 is depicted, which terminates in a collision event between an ego vehicle 602 and another vehicle 604. The cut-in scenario is characterized as a multi-lane driving scenario in which the ego vehicle 602 moves along a first lane 612 (the ego lane) and the other vehicle 604 initially moves along a second adjacent lane 604. At some point in the scenario, the other vehicle 604 moves from the adjacent lane 614 into the ego lane 612 in front of the ego vehicle 602 (the cut-in distance). In this case, the ego vehicle 602 cannot avoid colliding with the other vehicle 604. The first scenario instance terminates in response to the collision event.

[0187] Figure 6B An example of a first oracle output 256a obtained from the ground truth 310a of a first scenario instance is depicted. The "no collision" rule is evaluated for the duration of the scenario between the ego vehicle 602 and the other vehicle 604. At the end of the scenario, a collision event causes this rule to fail. Additionally, the evaluation Figure 4B When the other vehicle 604 moves laterally close to the ego vehicle 602, there is a point in time (t1) where both the safe lateral distance and safe longitudinal distance thresholds are violated, causing the safe distance rule to fail until a collision event at time t2.

[0188] Figure 6C A second example of a cut-in scenario is described. In the second example, the cut-in event does not result in a collision, and the ego vehicle 602 is able to reach a safe distance behind the other vehicle 604 after the cut-in event.

[0189] Fig.6D An example of a second oracle output 256b obtained from the ground truth 310b of the second scenario instance is depicted. In this case, the "no collision" rule applies throughout. The safe distance rule is violated at time t3 when the lateral distance between the ego vehicle 602 and the other vehicle 604 becomes unsafe. However, at time t4, the ego vehicle 602 manages to reach a safe distance behind the other vehicle 604. Therefore, the safe distance rule fails only between time t3 and time t4.

[0190] Identify significant test runs:

[0191] Figure 7 A computer system 700 for identifying significant test runs is schematically illustrated. The system 700 includes a test oracle 710, a results database 720, and a data retrieval component 730.

[0192] The test oracle 710 and the result database 720 correspond to the test oracle 252 and the result database 258 described above with reference to Figures 1 to 6, respectively, and add further additional functions described below.

[0193] The test oracle 710 is configured to receive multiple sets of operational data, each set of operational data relating to the operation of a driving scenario. The driving scenario involves a self-agent controlled by an autonomous vehicle system, such as the AV stack 100 described herein. The scenario also includes challenger agents or "external" agents, such as other vehicles, pedestrians, cyclists, animals, etc. In some cases, the scenario will involve multiple challenger agents of the same or different types.

[0194] As discussed above, the operational data, generally indicated by reference numeral 701, may include traces 212 of the ego agent and the challenger agent. The operational data 701 may also include the additional contextual data 214 discussed above. Figure 7 The test oracle 710 is shown to receive operation data from the simulator 202, but in other examples, the operation data can belong to a real scene. In such an example, the operation data 701 can be similar to the ground truth 144 of the real operation discussed above.

[0195] The test oracle 710 includes a challenger behavior detector 711. The challenger behavior detector 711 generates at least one time index event according to the operation data 701 in response to the behavior or action of the challenger agent in the operation data 701.

[0196] An event may be detected by applying multiple conditions to the behavior of the challenger agent and / or the ego agent to determine that the event has occurred.

[0197] An example event may be a "cut-in" event, where the challenger agent changes lanes to Fig. 6A and 6C The described approach cuts in front of the self-agent. In such an example, the conditions may include the following:

[0198] The challenger agent is in a lane adjacent to the ego agent (i.e., near side or offside);

[0199] Self-agent does not change lanes

[0200] The challenger agent crosses the line between the lane it occupies and the ego agent's lane within a given spacing distance in front of the ego agent

[0201] The challenger agent occupies the ego lane within a given spacing distance

[0202] If all of these conditions are met, the behavior detector 711 will determine that a cut-in event has occurred. If not all of the conditions are met (e.g., because the challenger agent is far enough ahead and outside the spacing distance), the behavior detector 711 will determine that a cut-in event has not occurred.

[0203] Events can be associated with multiple parameters. For example, a cut-in event can be associated with the longitudinal distance (i.e., the gap) from the ego agent to the challenger agent at the start of the cut-in, the speed of the challenger agent before the cut-in, and the lateral speed of the challenger agent during the lane change. The conditions can then take the form of constraints imposed on the parameters. For example, if the gap distance, challenger agent speed, and lateral speed are within a predetermined range, then it is determined that the event has occurred.

[0204] The event is time-indexed in the sense that challenger behavior detector 711 is configured to determine one or more time steps in operational data 701 during which the event occurred. For example, challenger behavior detector 711 may output a true value corresponding to each time step associated with operational data 701 when the event occurred, and a false value corresponding to each time step associated with operational data 701 when the event is no longer occurring. However, in other examples, challenger behavior detector 711 may instead output other information that allows identification of the time index at which the event occurred, such as an indication of the time index that forms the start and end of the event.

[0205] Other example events include cut-out events and braking events.

[0206] A cut-out event occurs when a first challenger agent that is in front of the ego agent and in the ego agent's lane moves into an adjacent lane and displays a second challenger agent that is slow or stationary in front of it. The second challenger agent may pose a hazard that the ego wants to avoid. Example parameters associated with a cut-out event may include one or more of the following: longitudinal distance from the first challenger agent to the second challenger agent at the start of the cut-out; lateral offset of the first challenger agent from the center of the lane; lateral offset of the second challenger agent from the center of the lane; lateral offset of the first challenger from the center of the lane after the lane change; speed of the second challenger agent after the cut-out; lateral speed of the first challenger during the lane change: and speed of the first challenger agent before the cut-out.

[0207] A braking event occurs when another agent (e.g., a challenger agent) brakes suddenly in front of the ego agent and in the lane of the ego agent. The other agent may pose a hazard for the ego agent to avoid. Example parameters associated with a braking event may include one or more of the following: the challenger agent's speed after deceleration, the challenger agent's maximum deceleration rate, and the challenger agent's maximum jerk rate.

[0208] It will be appreciated that a variety of different events may be directly specified using the above-described types of conditions and parameters. Challenger behavior detector 711 is capable of detecting a number of different defined events from operational data 701 .

[0209] The test oracle 710 also includes a decision indicator generator 712, also referred to herein as a decision component. The decision indicator generator 712 generates a series of decision indicators over multiple time steps of the run data. At each time step, the decision indicator indicates whether a specific driving action that can be performed by the self-agent is permissible if the self-agent chooses to perform it. In order to make this estimate, a "basic driver model" is used to plan a (hypothetical) trajectory from the self's current state to a short time in the future (e.g., one or a few seconds) to perform the action in question (e.g., lane change). This hypothetical trajectory is then estimated according to one or more driving rules (similar to estimating the actual behavior of the self in the test oracle). In the example below, the calculation is simplified by reducing this hypothetical trajectory to a single future state at a single point in time (the basic driver model calculates a single future state of the self-agent a few seconds in the future, assuming that it has performed the action in question, and then estimates the single state against the applicable driving rules). In other implementations, this can be expanded to consider multiple future states.

[0210] Note that a rule-based planner does not require a "full" planner logic. In the example above, the rule-based planner is implemented as a simplified base driver model, using a second event type that models simplified driver behavior. For example, when considering the safety of a lane change, the simplified rule-based planner can look for a traffic gap in the lane next to the ego. Based on the predefined assumption that if the ego starts to perform a lane change at the current time step, completes it within "x" seconds, maintains its current speed, then determine whether the lane change strategy is safe, e.g., whether it will impact any oncoming vehicles in that lane. Using this as a simplified planner decision, the base driver model can be configured to find traffic gaps that the hypothetical planner may have decided to enter or not enter based on its perception. This can then be compared to the ground truth to determine (for example) that because the ego missed the relevant agent from its perceived worldview, it may have believed that the lane change was safe, when in fact the ground truth estimate shows that the change was unsafe.

[0211] The estimation performed by the decision indicator generator 712 is independent of the stack under test 100, which is controlled by the ego-agent during runtime. That is, the estimation performed by the decision indicator generator 712 is not fed back to the ego-agent and is independent of the ego-agent's performance under test. The only reliance on the ego-agent stems from the fact that the hypothetical trajectory is planned based on the ego's current state. This hypothetical trajectory is not a prediction, as the base driver model does not attempt to predict what the ego-agent might do next; rather, the estimation is whether it is safe to take a given action at a given point in time. For example, the ego-agent may remain in a single lane for a given time interval, and may generate a yes / no decision indicator at each time step within that time interval, indicating whether a lane change is safe according to the base driver model.

[0212] Given a current "snapshot" of the driving scene at a certain point in time (i.e., a specific time step of the operational data 701, also referred to as a driving scene or frame) and an action to be evaluated, the decision component classifies the action as belonging to a decision class in a set of predefined decision classes. In the following example, the decision component is rule-based and is made by evaluating one or more decision rules (e.g., driving safety rules - see below).

[0213] The decision classes can be binary, e.g., there are only two possible decision classes indicating whether an action is allowed or not (referred to as “pass” and “fail”, respectively; note that for decision indicators, the “pass / fail” estimate is related to the base driver model and not the ego-agent: a “fail” on the decision indicator indicates that the trajectory generated by the base driver model does not comply with at least one applicable rule, and does not mean that the actual behavior of the ego-agent violates any rule). This binary classification constitutes a decision on whether an action can be performed at the current point in time. The determination of whether an action is allowed or not typically takes safety into account, and may also take other aspects of driving performance (comfort, progress, etc.) into account. For example, in a lane driving scenario, given a snapshot of the scenario, the decision component can decide whether a lane change strategy is safe / allowed.

[0214] In the following example, decision component 712 is applied to a series of snapshots of an evolving scenario (i.e., operational data 701) over a series of time steps. In this context, a decision point is a time step at which the decision class changes relative to the previous time step. For example, a lane change strategy may be classified as unsafe in one time step and may be classified as safe in the next time step, representing a lane change decision point.

[0215] The method applied by decision component 712 does not require the use of stack 100 to determine or synthesize trajectories. In the following example, actions are classified for a given snapshot at time step n by "rolling forward" from the current snapshot to some appropriate later time using some motion model and estimating the result.

[0216] Simple motion models can be used for this purpose. For example, in a lane change scenario, the end position of the ego agent after a predetermined policy interval (e.g., 4 seconds) can be determined, which places the ego agent in the center of the target lane. The other agents can be assumed to move at a constant speed over the time interval of, e.g., 4 seconds, allowing their end positions to be determined based on their speeds given in the snapshot at time n. The decision rule or group of rules can then be evaluated at the end of the window, e.g., whether the ego agent is at a safe distance from all other agents at the end of the window of, e.g., 4 seconds. In this paper, the decision class is decided based on a single evaluation, e.g., 4 seconds after the current snapshot (this does not pick out edge cases where unsafe things happened during the policy but the final state is safe, but this is very rare in practice).

[0217] A snapshot at time step n refers to a view of the scene at or until time n. In the example below, the snapshot includes only current state data (e.g., the agent state at time n). However, a snapshot can include historical data (e.g., the state before time n).

[0218] The decision making component 712 may be referred to as a "rule-based" planner or a "base driver model."

[0219] As described above, safety models have been used in autonomous driving to precisely define the concept of "safety". Examples of known safety models include RSS and the Automated Lane Keeping Specification (ALKS) being developed by the UK government. Within a test pipeline, safety models can be used as the basis for rule-based driving performance testing based on well-defined numerical quantities such as lateral or longitudinal distances (e.g., between two agents or relative to a road reference point or line, etc.). Rule-based testing can be expanded to other aspects of driving performance, such as progress and comfort. As described above, the test oracle 710 applies some predefined set of driving rules to a given scenario run (which can be real or simulated).

[0220] These driving rules are redesigned to provide a rule-based planner 712 that runs in a "shadow mode". Shadow mode sometimes means an automated system that runs in the background on a human-driven vehicle. However, in this article, the term is used in a broader sense to mean any open-loop decision (i.e., a decision that is based on some scenario but does not affect the behavior of the self-agent in that scenario). The purpose of the rule-based planner is not to estimate the performance of the self-agent in a scenario run. Instead, the scenario run is simply used to provide a static "snapshot" of the scenario. Given a snapshot and a desired goal, the rule-based planner 712 provides an estimate of whether the goal is allowed (e.g., safe or more generally allowed by the driving rule set) given the snapshot.

[0221] The series of decision indicators derived by a rule-based planner is not necessarily the optimal course of action (although it is guaranteed to be safe by definition) nor does it necessarily reflect the actual decisions taken by the self-agent during the course of a scenario (e.g., a scenario can be used to evaluate the safety of a lane-changing maneuver. However, the lane-changing maneuver may or may not be executed in the scenario itself).

[0222] Figure 8 An estimation of a predefined driving maneuver in the form of an offside lane change maneuver is shown, involving the ego agent 801 and the challenger agent 802. As shown, the decision indicator generator 712 generates a planned trajectory 803 associated with the driving maneuver. To generate the planned trajectory 803, the driving maneuver may be associated with a duration for performing the maneuver, such as 4 seconds in the case of a lane change. The decision indicator 712 may assume that the ego agent 801 and the challenger agent 802 are traveling at a speed (Ve0 and Vo1, respectively) that remains constant throughout the maneuver. The decision indicator generator 712 is then configured to estimate the planned trajectory 803 to determine whether it corresponds to an allowed maneuver, or whether it is not allowed because it violates one of the predefined rules of the rule-based planner. In the example shown, the offside lane change is allowed because it causes the ego agent 801 to move to a position behind the challenger agent 802 with sufficient spacing.

[0223] Figure 8 The example of illustrates the estimation of an offside lane change. Other example predefined driving actions that may be estimated by the decision indicator generator include an on-side lane change, entering a roundabout, merging with freeway traffic, and an unprotected left or right turn.

[0224] As described above, the rules of the rule-based planner used to estimate the planned trajectory 803 (hereinafter referred to as "action estimation rules") can correspond to the performance evaluation rules 254 used by the oracle 252 / 700 to estimate the operation data 701. That is, the same set of rules can be used for two different purposes: estimating the actual action of the self-agent during operation, and estimating whether a predefined driving action is allowed at a given time step in the operation. However, the performance evaluation rules 254 do not have to correspond to the action estimation rules. Instead, different rules or partially overlapping rule sets can be provided for these two different purposes.

[0225] Decision indicator generator 712 may perform an estimation over a series of time steps of operating data 701 to generate a series of decision indicators indicating whether a predefined driving action is allowed. For example, decision indicator generator 712 may perform an estimation at each time step of operating data 701. In other examples, estimation may be performed on only a portion of operating data 701. For example, a decision related to lane change may be applied only to a portion of operating data 701 involving a road having multiple lanes.

[0226] The time-indexed events generated by the challenger behavior detector 711 and the decision indicators generated by the decision indicator generator 712 for a particular run form a set of outputs for that run, which the test oracle 700 configures to be stored in the results database 720. The set of outputs may also include the output 256 discussed above.

[0227] The stored outputs may then be retrieved by the data retrieval component 730. The data retrieval component 730 includes a suitable query interface for querying the database 720, and may also retrieve the outputs stored in the database 720 by querying the time-indexed events and decision indicators. In other words, the data retrieval component 730 is capable of searching the outputs stored in the database 720 to retrieve results that satisfy the filter. For example, the data retrieval component 730 is capable of retrieving outputs that include a specific time-indexed event (e.g., cut-in) and a specific decision indicator (e.g., lane change unsafe).

[0228] In some examples, the data retrieval component 730 may retrieve results where the time index of the event and the time index of the decision indicator are the same or within a range of each other. That is, in one example, the data retrieval component 730 retrieves results where the event (e.g., cut-in) occurs while a particular decision indicator (e.g., lane change unsafe) is indexed. In another example, the data retrieval component 730 retrieves results where the event (e.g., cut-in) occurs at a time index within a predefined time index range of a particular decision indicator (e.g., lane change unsafe), such as within 10 seconds or 30 seconds of it. In these cases where the results are limited based on the relative time indexes of the event and decision indicator, the data retrieval component 730 may additionally identify portions of the operating data 701 where the event and decision indicator exist.

[0229] As mentioned above Figure 5 6 , the output of the test oracle 252 may be visualized by a visualization component 520 and displayed on a GUI 500 on a suitable display system 522. The output of the test oracle 700 may be similarly visualized and displayed on the GUI 500. Thus, the data retrieval component 730 may be coupled to the visualization component 520, or incorporated into the visualization component 520.

[0230] Fig. 9 An example query GUI 510 is shown through which a user can query based on time-indexed events and decision indicators. The query screen includes event and decision input boxes 511 that allow the user to specify events and / or decision indicators. This can be in the form of a label input box, although other suitable input boxes can be provided. In this figure, the user has entered input 511A corresponding to the presence of a cut-in event, and input 511B corresponding to a safe lane change value of false. The GUI 510 also includes an input box 512 for adding additional search criteria such as weather or time of day, an input box 513 for limiting the date range of the operating data, and a search button 514 for executing the query.

[0231] Fig.10 An example result GUI 550 is shown that displays the results of a query entered via the query GUI 510. The result GUI 550 includes a list 551 of runs retrieved based on the query, each of which can be selected and viewed in detail. Associated with each run in the run list 551 is a button 551a that allows retrieval of a run portion including events and / or decision indicators.

[0232] like Fig.11As shown, GUI 550 may allow selection of a run and thus viewing of the run using GUI 500A. GUI 500A corresponds to GUI 500 discussed above, with the addition of another time series 536 showing occurrences of events, and a time series 537 showing allowance / disallowance of driving actions represented by decision indicators.

[0233] Although the functionality of the challenger behavior detector 711, the decision indicator generator 712, and / or the data retrieval component 720 has been described as forming part of the test oracle 700, in other examples, these functions may be performed independently of the test oracle 700. In other words, the processing of the operational data 701 to generate time-indexed events and decision indicators may be performed as part of a system or method that does not involve evaluating the operational data 701 to estimate the behavior of the self-agent associated with the performance evaluation rules 254. For example, the operational data 701 may be processed as part of a pipeline for identifying suitable scenarios that may form training data for an AV stack. In this case, the performance results of the self-agent tested in the operational data 701 may be irrelevant.

[0234] Furthermore, the functions of challenger behavior detector 711 and decision indicator generator 712 may be performed at different times, and the evaluation of performance evaluation rules 254 may be performed at different times. For example, the generation of time indexed events and decision indicators may be performed in separate offline or batch processes.

[0235] Fig.12 A schematic flow chart of a computer-implemented method for identifying significant test runs is shown. The method includes a step S1201 of receiving multiple sets of run data, each set of run data being generated in a driving scenario having an ego agent and at least one challenger agent, wherein an autonomous vehicle system controls the ego agent. The method includes a step S1202 of generating at least one time-indexed event for each set of run data, the event being generated in response to a detected behavior of at least one challenger agent. The method includes a step S1203 of generating, for each set of run data, a series of decision indicators at multiple time steps of the run data, the decision indicator at each time step indicating whether a predefined driving action of the ego agent will be allowed, each driving indicator being generated independently of the autonomous vehicle system controlling the ego agent. The decision indicators are generated by generating a planning trajectory based on the position of the ego agent from a time step to a future time step, and determining whether a predefined driving behavior of the ego agent is allowed by estimating the planning trajectory based on at least one predefined driving behavior estimation rule, the planning trajectory being independent of the behavior of the ego agent between the time step and the future time step. The method also includes a step S1204 of storing an output set in a results database. The method may include the Figure 7-Figure 11Further steps discussed.

[0236] While the above examples consider AV stack testing, the technique can be used to test components of other forms of mobile robots. Other mobile robots are being developed, for example, to deliver goods within and outside industrial areas. Such mobile robots do not have a human on board and belong to a class of mobile robots called unmanned autonomous vehicles (UAVs). Autonomous aerial mobile robots (drones) are also being developed.

[0237] References to components, functions, modules, etc. herein represent functional components of a computer system that can be implemented at the hardware level in various ways. The computer system includes execution hardware that can be configured to perform the method / algorithm steps disclosed herein and / or implement a model trained using the present technology. The term execution hardware includes hardware configured to perform any form / combination of related method / algorithm steps. The execution hardware can take the form of one or more processors, which can be programmable or non-programmable, or a combination of programmable and non-programmable hardware can be used. Examples of suitable programmable processors include general-purpose processors based on instruction group architectures, such as CPUs, GPU / accelerator processors, etc. Such general-purpose processors typically execute computer-readable instructions stored in a memory coupled to the processor or inside the processor, and perform related steps according to these instructions. Other forms of programmable processors include field programmable gate arrays (FPGAs), which have circuit configurations programmable by circuit description codes. Examples of non-programmable processors include application specific integrated circuits (ASICs). Codes, instructions, etc. can be appropriately stored on transient or non-transient media (examples of the latter include solid-state, magnetic, and optical storage devices, etc.). Figure 1A The subsystems 102 - 108 of the runtime stack may be implemented in a programmable or dedicated processor, or a combination of both, in an onboard or offboard computer system in the context of testing or the like. Figure 2 Various components of, such as simulator 202 and test oracle 252, may similarly be implemented in programmable and / or dedicated hardware.

Claims

1. A computer system for identifying notable test runs involving an autonomous vehicle system in a real or simulated scenario, the computer system comprising: A processor configured to: receiving a plurality of sets of operating data, each set of operating data representing a driving scenario having an ego agent and at least one challenger agent, wherein the autonomous vehicle system controls the ego agent, For each set of running data, an output set is generated, the output set comprising: at least one time-indexed event generated in response to detected behavior of the at least one challenger agent, a series of decision indicators at a plurality of time steps of the operating data, the decision indicator at each time step indicating whether a predefined driving action of the ego-agent is to be permitted, each driving indicator being generated independently of control of the ego-agent by the autonomous vehicle system, and a data retrieval component coupled to the results database, the data retrieval component configured to retrieve a set of one or more output groups based on the at least one time-indexed event and the series of decision indicators; The processor is configured to generate the decision indicator series in the following manner: generating a planning trajectory based on the position of the ego-agent from a time step to a future time step, the planning trajectory being independent of the behavior of the ego-agent between the time step and the future time step, and Whether a predefined driving maneuver of the ego-agent will be permitted is determined by estimating the planned trajectory based on at least one predefined maneuver estimation rule.

2. The computer system of claim 1, wherein: The processor is further configured to generate at least one performance result based on predefined driving performance rules applied to the self-agent and include the at least one performance result in the output set.

3. The computer system according to claim 1 or 2, comprising: a visualization component configured to generate rendering data to cause a display system to display a query graphical user interface GUI, wherein the query GUI includes an input configured to receive a user selection of the at least one time-indexed event and the decision indicator; and Wherein, the data retrieval component is configured to retrieve the set of one or more output groups based on a user selection of the at least one time-indexed event and the decision indicator.

4. A computer system according to any preceding claim, comprising a visualization component configured to generate rendering data for causing a display system to display a result GUI, the result GUI comprising the set of one or more output groups.

5. A computer system according to any preceding claim, comprising a visualization component, the visualization component being configured to generate rendering data for causing a display system to display an output group GUI, the output group GUI being configured to display an output group selected from the set of output groups, the output group GUI comprising: a first time series display element showing time-indexed events in the selected output group on a first timeline, and A second time series display element showing on a second timeline the allowance of the driving action represented by the decision indicator in the selected output group.

6. A computer system according to claim 5 as appended to claim 2, wherein: The output group GUI also includes a third time series display element that shows at least one performance result on a third timeline.

7. A computer system according to any preceding claim, wherein: The data retrieval component is configured to retrieve a set of one or more output groups, wherein the time-indexed event occurs within a predetermined time range of a decision indicator having a predetermined value.

8. The computer system according to claim 7, wherein: The time-indexed event occurs simultaneously with the decision indicator having the predetermined value.

9. A computer system according to any preceding claim, wherein: The at least one time-indexed event includes one of a cut-in event, a cut-out event, and a braking event.

10. A computer system according to any preceding claim, wherein: The predefined driving maneuver includes one of a lane change maneuver, a highway merge maneuver, entering a roundabout, an unprotected left turn, or an unprotected right turn.

11. A computer system according to any preceding claim, wherein: The data retrieval component is configured to identify a portion of a set of operational data, the operational data comprising the time-indexed event and a decision indicator having a predetermined value.

12. A computer system according to any preceding claim, wherein: The processor is configured to generate the at least one time-indexed event by applying a plurality of conditions to behavior of the challenger agent in the operational data.

13. A computer system according to any preceding claim, wherein: The predefined action estimation rules form part of an open-loop rule based planner.

14. A computer system according to any preceding claim, wherein: The planned trajectory includes a single future state for the future time step, and the processor is configured to estimate the planned trajectory by estimating the single future state based on the at least one predefined action estimation rule.

15. A computer-implemented method for identifying notable test runs in a real or simulated scenario, comprising: receiving a plurality of sets of operating data, each set of operating data being generated in a driving scenario having an ego agent and at least one challenger agent, wherein the autonomous vehicle system controls the ego agent, generating at least one time-indexed event for each set of operational data in response to detected behavior of the at least one challenger agent; For each set of operating data, generating a series of decision indicators at a plurality of time steps of the operating data, the decision indicator at each time step indicating whether a predefined driving action of the ego-agent is permitted, each driving indicator being generated independently of the autonomous vehicle system controlling the ego-agent by: generating a planning trajectory based on the position of the ego-agent from a time step to a future time step, the planning trajectory being independent of the behavior of the ego-agent between the time step and the future time step, and determining whether a predefined driving behavior of the ego-agent will be permitted by estimating the planned trajectory based on at least one predefined driving behavior estimation rule; and Store the output groups in the results database.

16. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer, implement the method of claim 15.

Images