Safety Analysis Framework

Abstract

Techniques for determining safety metrics associated with a vehicle controller are discussed herein. To determine whether a complex system (which may be untestable) can operate safely, various operational regimes (scenarios) can be identified based on operational data and adjusted in association with scenario parameters. To verify the safe operation of such a system, scenarios can be identified for testing. Error metrics of subsystems of the system can be quantified. Error metrics can be introduced into the scenarios in addition to probabilistic errors of other systems / subsystems. Scenario parameters can also be perturbed. Any number of such perturbations can be instantiated in a simulation to test, for example, a vehicle controller. Safety metrics associated with the vehicle controller can be determined based on the simulation, as well as any causes of failure.

Description

[Technical field]

[0001] The present invention relates to a safety analysis framework.

[0002] [Related Applications] This patent application claims priority to U.S. utility patent application entitled "SAFETY ANALYSIS FRAMEWORK" having serial number 16 / 586,838, filed on September 27, 2019, and to U.S. utility patent application entitled "ERROR MODELING FRAMEWORK" having serial number 16 / 586,853, filed on September 27, 2019. Application serial numbers 16 / 586,838 and 16 / 586,853 are incorporated by reference herein in their entireties. [Background technology]

[0003] An autonomous vehicle can use an autonomous vehicle controller to guide the autonomous vehicle through an environment. For example, the autonomous vehicle controller can use planning methods, apparatus, and systems to determine a driving path and guide the autonomous vehicle through an environment that includes dynamic objects (e.g., vehicles, pedestrians, animals, etc.) and static objects (e.g., buildings, signs, stationary vehicles, etc.). However, it is important to verify the safety of the controller to ensure the safety of the occupants. [Brief description of the drawings]

[0004] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears, and the use of the same reference number in different drawings indicates similar or identical components or features.

[0005] [Figure 1] FIG. 1 illustrates generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. [Diagram 2]FIG. 2 illustrates a computing device that generates scenario data based at least in part on log data generated by a vehicle, the scenario data indicative of one or more variations of a scenario. [Diagram 3] FIG. 3 illustrates generating error model data based at least in part on vehicle data and ground truth data. [Figure 4] FIG. 4 illustrates the use of error model data to perturb a simulation by providing at least one of an error or uncertainty associated with the simulated environment. [Diagram 5] FIG. 5 illustrates a computing device that generates perception error model data based at least in part on log data and ground truth data generated by a vehicle. [Figure 6] FIG. 6 illustrates a computing device for generating simulation data based at least in part on parameterized scenario data and generating safety metric data based at least in part on the simulation data. [Figure 7] FIG. 7 shows a block diagram of an example system for implementing the techniques described herein. [Figure 8] FIG. 8 illustrates a flow diagram of an example process for determining a safety metric associated with a vehicle controller, according to an example of the disclosure. [Figure 9] FIG. 9 illustrates a flow diagram of an example process for determining a statistical model associated with a subsystem of an autonomous vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] The techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined using simulation, for example, in conjunction with determining other performance metrics. Simulation can be used to validate software (e.g., vehicle controller) running on a vehicle (e.g., autonomous vehicle) and gather safety metrics to ensure that the software is capable of safely controlling such a vehicle in various scenarios. In additional or alternative examples, simulation can be used to learn about the constraints of an autonomous vehicle using an autonomous controller. For example, simulation can be used to understand the operating space of an autonomous vehicle (e.g., the envelope in which an autonomous controller effectively controls the autonomous vehicle) taking into account road conditions, ambient noise, failed components, etc. Simulation can also be useful to generate feedback to improve the operation and design of an autonomous vehicle. For example, in some examples, simulation can be useful to determine the amount of redundancy needed in an autonomous controller, or how to modify the behavior of an autonomous controller based on what is learned through simulation. Furthermore, in additional or alternative examples, simulation can be useful to inform the hardware design of an autonomous vehicle, such as optimizing the placement of sensors on the autonomous vehicle.

[0007] When creating a simulation environment to perform testing and validation, it is possible to specifically enumerate the environment with various specific examples. Each instantiation of such an environment is unique and definable. Manually enumerating all possible scenarios may require excessive time, and if all possible scenarios are not constructed, various scenarios may not be tested. Scenario parameters can be used to parameterize characteristics and / or attributes of objects in a scenario to provide scenario variations.

[0008] For example, a vehicle or multiple vehicles can traverse an environment and generate log data associated with the environment. The log data can include sensor data captured by one or more sensors in the vehicle, sensory data indicative of objects identified by one or more systems onboard the vehicle (or generated at a post-processing stage), predictive data indicative of the object's intent (whether generated during recording or thereafter), and / or status data indicative of diagnostic information, trajectory information, and other information generated by the vehicle. The vehicles can transmit the log data over a network to a database that stores the log data and / or to a computing device that analyzes the log data.

[0009] The computing device can determine various scenarios, frequencies of the various scenarios, and regions of the environment associated with the various scenarios based on the log data. In some examples, the computing device can group similar scenarios represented in the log data. For example, the scenarios can be grouped, for example, using k-means clustering and / or by evaluating weighted distances (e.g., Euclidean) between parameters of the environment (e.g., daytime, nighttime, precipitation, vehicle position / speed, object position / speed, road segments, etc.). As described above, clustering similar or similar scenarios can reduce the amount of computational resources required to simulate an autonomous controller in an environment by simulating the autonomous controller in its own scenario, rather than simulating an autonomous vehicle in nearly similar scenarios that would result in redundant simulation data / results. As can be appreciated, an autonomous controller can be expected (and / or may obviously perform) similarly in similar scenarios.

[0010] For example, the computing device can determine a rate at which pedestrians appear at a crosswalk based on the number of pedestrians represented in the log data. In some examples, the computing device can determine a probability of detecting a pedestrian at a crosswalk based on the rate and a time period for operating the autonomous vehicle. Based on the log data, the computing device can determine a scenario and identify scenario parameters based on the scenario that can be used in the simulation.

[0011] In some examples, simulations can be used to test and validate the response of an autonomous vehicle controller to faulty (and / or faulty) sensors in the vehicle and / or faulty (and / or faulty) processing of sensor data. In such examples, a computing device can be configured to introduce inconsistencies in scenario parameters of an object. For example, an error model can indicate errors and / or error percentages associated with the scenario parameters. A scenario can incorporate the errors and / or error percentages into a simulated scenario to simulate the response of an autonomous vehicle controller. Such errors can be represented by lookup tables determined based on statistical aggregation using, without limitation, ground truth data, functions (e.g., errors based on input parameters), or any other model that maps parameters to specific errors. In at least some examples, such error models can map specific errors with probabilities / frequencies of occurrence.

[0012] By way of example, and not by way of limitation, the error model may indicate that a scenario parameter, such as a speed associated with an object in a simulated environment, is associated with an error percentage. For example, an object may traverse a simulated scenario at a speed of 10 meters per second, and the error percentage may be 20%, resulting in a range of speeds from 8 meters per second to 12 meters per second. In some examples, the range of speeds may be associated with a probability distribution indicating that some parts of the range are more likely to occur than other parts of the range. (For example, 8 meters per second and 12 meters per second are associated with a 15% probability, 9 meters per second and 11 meters per second are associated with a 30 percent probability, and 10 meters per second is associated with a 10% probability.)

[0013] Based on the error model and / or the scenario, a parameterized scenario can be generated. The parameterized scenario can provide a set of scenario variations. Thus, instantiating the autonomous vehicle controller in the parameterized scenario and simulating the parameterized scenario can efficiently cover a wide range of scenario variations without requiring manual enumeration of the variations. Additionally, based at least in part on executing the parameterized scenario, simulation data can indicate how the autonomous vehicle controller responded (or would respond) to the parameterized scenario, and based at least in part on the simulation data, a successful or unsuccessful outcome can be determined.

[0014] Aggregating the simulation data related to the parameterization scenarios can provide safety metrics associated with the parameterization scenarios. For example, the simulation data can indicate success and / or failure rates of the autonomous vehicle controller and the parameterization scenarios. In some examples, meeting or exceeding the success rate can indicate successful validation of the autonomous vehicle controller, which can then be downloaded (or otherwise transferred) by the vehicle for further vehicle control and operation.

[0015] For example, the parameterized scenario can be associated with an outcome. The simulation data can indicate that the autonomous vehicle controller responded consistently or inconsistently to the outcome. By way of example, and not by way of limitation, the parameterized scenario can represent a simulated environment including a vehicle controlled by the autonomous vehicle controller crossing at a speed and performing a stopping action in front of an object ahead of the vehicle. The speed can be associated with a scenario parameter indicating a range of speeds for the vehicle. The parameterized scenario can be simulated based at least in part on the range of speeds and can generate simulation data indicating a distance between the vehicle and the object when the vehicle completes the stopping action. The parameterized scenario can be associated with an outcome indicating that the distance between the vehicle and the object meets or exceeds a distance threshold. Based on the simulation data and the scenario parameters, a success rate can indicate the number of times that the distance between the vehicle and the object meets or exceeds a distance threshold when the vehicle completes the stopping action compared to the total number of times the vehicle completes the stopping action.

[0016] The techniques described herein provide various computational efficiencies. For example, by using the techniques described herein, a computing device requires fewer computational resources and multiple simulated scenarios can be generated faster than is available via conventional techniques. Conventional techniques are not scalable. For example, generating a set of unique simulated environments as many as are required to train, test, and / or validate a system (e.g., one or more components of an AI stack) on an autonomous vehicle (e.g., before such autonomous vehicle is deployed to a corresponding new real environment) can take a very long time, thereby limiting the ability to train, test, and / or validate such a system (e.g., one or more components of an AI stack) on an autonomous vehicle before entering a real scenario and / or environment. The techniques described herein are unconventional in that they leverage sensor data collected from a real environment and supplement that data with additional data to more efficiently generate a simulated environment that is substantially more accurate than is available via conventional techniques (e.g., for a corresponding real environment). Furthermore, the techniques described herein, including various aspects of the scenarios, enable the generation of many scalable simulated scenarios in less time and with fewer computational resources than are available with conventional techniques.

[0017] Furthermore, the techniques described herein are directed to improving safety. That is, the simulated environments resulting from the generative techniques described herein can be used to test, train, and validate systems onboard an autonomous vehicle to ensure that such systems can safely operate the autonomous vehicle when deployed in a real environment. That is, the simulated environments resulting from the generative techniques described herein can be used to test, train, and validate planner and / or predictive systems of an autonomous vehicle controller that can be used by an autonomous vehicle to navigate the autonomous vehicle along trajectories in a real environment. Thus, such training, testing, and validation enabled by the techniques described herein can provide an opportunity to ensure that an autonomous vehicle can safely operate in a real environment. Thus, the techniques described herein improve safety and impact navigation.

[0018] FIG. 1 illustrates an example 100 of generating vehicle performance data associated with a vehicle controller based on parameterized scenarios. Input data 102 can be used to generate the scenarios. The input data 102 can include vehicle data 104 and / or additional situational data 106. The vehicle data 104 can include log data captured by the vehicle traversing through an environment. As described above, the log data can be used to identify scenarios for simulating the autonomous controller. For illustrative purposes, the vehicle can be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle that can perform all safety-critical functions for the entire journey without expecting a driver (or passenger) to be in control of the vehicle at all times. In such an example, the vehicle can be unmanned, as it can be configured to control all functions from start to stop, including all parking functions. This is merely an example, and the systems and methods described herein can be incorporated into any land, air, or water vehicle, including vehicles that require manual control at all times by a driver, to those that are partially or fully autonomously controlled.

[0019] The vehicle may include a computing device that includes a perception engine and / or planner and perform operations such as detecting, identifying, segmenting, classifying, and / or tracking objects from sensor data collected from the environment. For example, objects such as pedestrians, bicycles / bicyclists, motorcycles / motorcyclers, buses, trams, trucks, animals, etc. may be present in the environment.

[0020] As the vehicle traverses through the environment, the sensors can capture sensor data associated with the environment. For example, some of the sensor data can be associated with objects (e.g., vehicles, cyclists, and / or pedestrians). In some examples, the sensor data can be associated with other objects, including, but not limited to, buildings, road surfaces, signs, barriers, etc. Thus, in some examples, the sensor data can be associated with dynamic objects and / or static objects. Dynamic objects, as described above, can be objects associated with movement (e.g., vehicles, motorcycles, cyclists, pedestrians, animals, etc.) or capable of moving within the environment (e.g., parked vehicles, standing pedestrians, etc.). Static objects, as described above, can be objects associated with the environment, such as, for example, buildings / structures, road surfaces, road markers, signs, barriers, trees, sidewalks, etc. In some examples, the vehicle computing device can determine information about objects in the environment, such as bounding boxes, classifications, and segmentation information.

[0021] The vehicle computing device can use the sensor data to generate a trajectory for the vehicle. In some examples, the vehicle computing device can also determine pose data associated with the position of the vehicle. For example, the vehicle computing device can use the sensor data to determine position data, coordinate data, and / or orientation data for the vehicle within the environment. In some examples, the pose data can include xyz coordinates and / or pitch, roll, and yaw data associated with the vehicle.

[0022] The vehicle computing device can generate vehicle data 104, which can include the data described above. For example, the vehicle data 104 can include sensor data, perception data, planning data, vehicle status data, speed data, intent data, and / or other data generated by the vehicle computing device. In some examples, the sensor data can include data captured by sensors such as time-of-flight sensors, position sensors (GPS, compass, etc.), inertial sensors (inertial measurement unit (IMU), accelerometer, magnetometer, gyroscope, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, intensity, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, etc. The sensor data can be data captured by such sensors, such as time-of-flight data, position data, lidar data, radar data, sonar data, image data, audio data, etc. Such log data may further include intermediate outputs by any one or more systems or subsystems of the vehicle, including, but not limited to, messages indicating detection of an object, object tracks, predictions of future object positions, multiple trajectories generated in response to such detection, control signals passed to one or more systems or subsystems used to execute commands, etc. In some examples, vehicle data 104 may include time data associated with other data generated by the vehicle computing device.

[0023] In some examples, input data 102 can be used to generate scenarios. Input data 102 can include vehicle data 104 and / or additional situational data 106. By way of example, and without limitation, additional situational data 106 can include data such as incident reports from third party sources. Third party sources can include law enforcement agencies, motor vehicle departments, and / or safety management that can publish and / or store reports of activities and / or incidents. For example, the reports can include a type of activity (e.g., traffic hazards such as debris on the road, localized flooding, etc.), a location, and / or a description of the activity. By way of example, and without limitation, the reports can describe a driver being struck by a fallen tree branch in the roadway while crossing at a speed of 15 meters per second while operating a vehicle. The reports can be used to generate similar scenarios that can be used in simulations.

[0024] In some examples, the additional situation data 106 can include captured sensor data (e.g., image data). By way of example and not by way of limitation, a driver of a vehicle can use a camera to capture image data while the driver is operating the vehicle. In some examples, the image data can capture an activity, such as an incident. By way of example and not by way of limitation, a driver can use a dashboard camera (e.g., a camera mounted on the interior dashboard of the vehicle) to capture image data while the driver is operating the vehicle. As the driver operates the vehicle, an animal may run across the roadway and the driver may immediately brake to slow the vehicle. The dashboard camera can capture image data of an animal running across the roadway and the vehicle slows down. The image data can be used to generate a scenario of an animal running across the roadway. As described above, a probability associated with the scenario can be determined and a scenario for simulation can be identified based on the probability of meeting or exceeding a probability threshold. By way of example and not by way of limitation, if the likelihood of encountering a scenario is less than 0.001%, a probability threshold of 0.001% can be used and then the high probability scenarios can be prioritized for simulation to determine a safety metric associated with the high probability scenario.

[0025] The input data 102, e.g., vehicle data 104 and / or additional situation data 106, can be used by a scenario editor component 108 to generate an initial scenario 110. For example, the input data 102 can be input to the scenario editor component 108, which can generate a synthetic environment that represents at least a portion of the input data 102 within the synthetic environment. Examples of generated scenarios, such as vehicle-generated data that can be included in the initial scenario 110 and vehicle data 104, can be found, for example, in U.S. patent application Ser. No. 16 / 392,094, filed April 23, 2019, entitled "Scenario Editor and Simulator," which is incorporated by reference in its entirety.

[0026] The scenario editor component 108 can be configured to scan the input data 102 to identify one or more scenarios represented in the input data 102. By way of example, and not by way of limitation, the scenario editor component 108 can determine that a portion of the input data 102 represents a pedestrian crossing a street without a right-of-way (e.g., no crosswalk, an intersection without pedestrian signs, etc.). The scenario editor component 108 can identify this as a scenario (e.g., jaywalking parameters) and label (and / or classify) the scenario, for example, as a jaywalking scenario. For example, the scenario editor component 108 can generate an initial scenario 110 using rules that define actions. By way of example, and not by way of limitation, a rule can define that a pedestrian crossing a road in an area not associated with a crosswalk is a jaywalker. In some examples, the scenario editor component 108 can receive label data from a user of the scenario editor component 108 to associate a portion of the input data 102 with the labels and generate the initial scenario 110.

[0027] In some examples, the scenario editor component 108 can scan other portions of the input data 102, identify similar scenarios, and label the similar scenarios with the same jaywalking label. In some examples, the scenario editor component 108 can identify scenarios that do not correspond to (or are excluded from) existing labels and generate new labels for these scenarios. In some examples, the scenario editor component 108 can generate a library of scenarios and store the library of scenarios in a database within the scenario editor component 108. By way of example, and not limitation, the library of scenarios can include a pedestrian crossing scenario, a merge scenario, a lane change scenario, etc.

[0028] In at least some examples, such initial scenarios 110 may be specified manually: for example, one or more users may specify particular scenarios to be tested to ensure that the vehicle can operate safely when executing such scenarios despite never (or rarely) having encountered the scenarios before.

[0029] The parameters component 112 can determine scenario parameters associated with the initial scenario 110 identified by the scenario editor component 108. By way of example, and not limitation, the parameters component 112 can analyze a jaywalking scenario and determine scenario parameters associated with the jaywalking scenario including pedestrian position, pedestrian pose, pedestrian size, pedestrian speed, pedestrian footsteps, distance between the vehicle and the pedestrian, vehicle speed, road width, etc.

[0030] In some examples, the parameter component 112 can determine a range or set of values ​​associated with a scenario parameter. For example, the parameter component 112 can determine a classification associated with an object (e.g., a pedestrian) represented in the initial scenario 110 and other objects of the same classification in the input data 102. The parameter component 112 can then determine a range of values ​​associated with the scenario parameter as represented by the initial scenario 110. By way of example, and not limitation, the scenario parameter can indicate that a pedestrian can have a speed ranging from 0.5 to 1.5 meters per second.

[0031] In some examples, the parameter component 112 can determine probabilities associated with the scenario parameters. By way of example, and not limitation, the parameter component 112 can associate a probability distribution, such as a Gaussian distribution (also referred to as a normal distribution), with the scenario parameters. In some examples, the parameter component 112 can determine the probabilities associated with the scenario parameters based on the input data 102. As described above, the parameter component 112 can determine a classification associated with an object represented in the input data 102 and determine other objects of the same classification in the input data 102 and / or other log data. The parameter component 112 can then determine a probability distribution for the scenario parameters associated with the object as represented by the input data 102 and / or other log data.

[0032] As an example, but not by way of limitation, the parameters component 112 may determine that 30% of pedestrians walk at a speed less than 0.3 meters per second, 30% of pedestrians walk at a speed greater than 1.2 meters per second, and 40% of pedestrians walk at a speed between 0.3 and 1.2 meters per second. The parameters component 112 may use the distribution as a probability that a pedestrian in a jaywalking scenario will walk at a particular speed. As an additional example, but not by way of limitation, the parameters component 112 may determine that a jaywalking scenario probability is 1%, which may indicate that a vehicle traversing an environment will encounter a jaywalker 5% of the time while traversing the environment. In some examples, the scenario probabilities may be used to include scenarios at a rate associated with the scenario probabilities during a simulation of the autonomous vehicle controller.

[0033] In some examples, the parameter component 112 can receive the complementary data 114 incorporated into the distribution. By way of example, and not by way of limitation, the parameter component 112 can determine a scenario parameter indicating that a pedestrian may have a range of distances to a vehicle of 30 to 60 meters while the vehicle is moving at a speed of 15 meters per second, or alternatively expressed as a time to impact of 2 to 4 seconds. The complementary data 114 (e.g., regulations or guidelines) can indicate that the vehicle must address the scenario with a time to impact of 1.5 seconds, which can be a lower limit (also referred to as a parameter threshold). The parameter component 112 can incorporate the complementary data 114 and determine the scenario parameter to have a time to impact of 1.5 to 4 seconds (although any time range can be specified). In some examples, the parameter component 112 can use the probability distributions described above to determine (using interpolation and / or extrapolation techniques) a probability associated with the complementary data 114.

[0034] The error model component 116 can determine an error model that can indicate an error associated with the scenario parameters. For example, the perception error model can generate a perception error associated with a perception parameter of the simulated object, the prediction error model can generate a prediction error associated with a prediction parameter of the simulated object, etc. In some examples, as the vehicle traverses the environment, the sensors can generate erroneous sensor data and / or the vehicle's computing device can misprocess the sensor data that can result in a perception error. Testing and simulating the perception error can help indicate the vehicle's operating margins associated with potential perception errors. For example, the scenario parameters, such as the perception parameters, can indicate the size of an object or a range of positions of an object in the environment. The error model component 116 can use the perception error model to indicate a potential error associated with the size of an object that can result in the vehicle's perception data indicating that the object is larger or smaller than the actual object in the environment.

[0035] The error model component 116 can determine the perception error model, for example, by comparing the input data 102 to ground truth data. In some examples, the ground truth data can be determined from manually labeled and / or other validated machine learning components. For example, the input data 102 can include sensor data and / or perception data generated by a vehicle. The error model component 116 can compare the input data 102 to the ground truth data, which can indicate actual parameters of objects in the environment. By comparing the input data 102 to the ground truth data, the error model component 116 can determine a perception error. As an example, but not by way of limitation, the input data 102 can indicate that a pedestrian is 1.8 meters tall, while the ground truth data indicates that the pedestrian is 1.75 meters tall, and thus the perception error model can indicate a perception error of about 3% (e.g., [(1.8-1.75) / 1.8]*100).

[0036] In some examples, the error model component 116 can determine a classification associated with an object represented in the input data 102 and determine other objects of the same classification in the input data 102. The error model component 116 can then determine a probability distribution associated with a range of error for the object (also referred to as an error distribution) and associate the probability distribution with the object in the initial scenario 110. By way of example and not by way of limitation, the error model component 116 can determine that an object with a pedestrian classification has a perception error of 4% to 6% and an object with a vehicle classification has a perception error of 3% to 5%. In some examples, the error model component 116 can determine a probability distribution indicating, for example, that objects larger than a threshold size are more likely or less likely to have an error (e.g., in the classification).

[0037] In some examples, the parameters component 112 can use the region data 118 to determine a set of regions of an environment that are compatible with the scenario and the scenario parameters. By way of example, and not by way of limitation, a scenario can indicate that a vehicle has a speed of 15 meters per second, a pedestrian has a speed of 1.5 meters per second, and a distance between the vehicle and the pedestrian is 30 meters. The parameters component 112 can determine the regions based on the region data 118 of an environment that fits the scenario. By way of example, and not by way of limitation, the parameters component 112 can exclude school zones because the speed of the vehicles in the scenario may exceed the speed limit associated with the school zone, and thus the scenario is not valid in the school zone. However, such vehicle speeds (e.g., 15 m / s) may be reasonable on a county road adjacent to farmland, and thus such regions may be considered within the set of regions.

[0038] In some examples, the parameter component 112 can store region data 118 including segments of drivable regions of the environment. Techniques for identifying drivable surface segments and similar segments, as well as segment classifications and / or stereotypes, can be found, for example, in U.S. patent application Ser. No. 16 / 370,696, entitled "Extension of Autonomous Driving Functionality to New Regions," filed Mar. 19, 2019, and U.S. patent application Ser. No. 16 / 376,842, entitled "Simulating Autonomous Driving Using Map Data and Driving Data," filed Apr. 5, 2019, which are incorporated herein by reference in their entireties.

[0039] For example, the area data 118 of the environment can be parsed into segments and similar segments can be identified. In some examples, the segments can include junction segments, e.g., intersections, merges, etc., or connecting road segments, e.g., the length and / or width of the road between the junctions. By way of example and not limitation, all two-lane road segments having speed limits within a 10 mph range can be associated with the same stereotype. In some examples, data can be associated with each of the individual segments. For example, a junction segment can include a junction type, e.g., a merge, "T", roundabout, etc., the number of roads that meet at the junction, the relative positions of those roads, e.g., the angle between the roads that meet at the junction, information about the traffic control signal at the junction, and / or other features. Data associated with a connecting road segment can include the number of lanes, the width of those lanes, the direction of crossing in each of the lanes, the identification of parking lanes, the speed limit of the road segment, and / or other features.

[0040] In some examples, segments of a drivable surface can be grouped according to a segment classification or segment stereotype. By way of example and not limitation, some or all junction segments that fit some range of a metric or attribute can be grouped together (e.g., using k-means, evaluating a weighted distance (e.g., Euclidean) between segment parameters, or otherwise clustering such segments based on segment parameters).

[0041] Scenarios that include the same or similar stereotypes can be used to verify the functionality of an autonomous vehicle controller. For example, an autonomous vehicle may be expected to do (and / or may obviously do) the same things in the same or similar stereotypes. In some examples, the use of stereotypes can reduce the number of comparisons to be made. For example, by identifying similar regions, the number of simulation scenarios that can provide useful information is reduced. The techniques described herein can reduce computational complexity, memory requirements, and processing time by optimizing over certain scenarios that provide useful information for validation and testing.

[0042] The parameterized scenario component 120 can use the data determined by the parameter component 112 (e.g., the initial scenario 110, the scenario parameters, the set of regions, and / or the error model data) to generate the parameterized scenario 122. For example, the initial scenario 110 can represent scenarios such as a lane change scenario, a right turn scenario, a left turn scenario, an emergency stop scenario, etc. The scenario parameters can represent a speed associated with a vehicle controlled by the autonomous vehicle controller, a pose of the vehicle, a distance between the vehicle and an object, etc. In some examples, the scenario parameters can represent an object, a position associated with the object, a speed associated with the object, etc. Additionally, an error model (e.g., a perception error model, a prediction error model, etc.) can indicate an error associated with the scenario parameter and provide a range of values ​​and / or probabilities associated with the scenario parameter. By way of example, and without limitation, a scenario parameter such as a speed of the vehicle can be associated with a range of speeds, such as 8 to 12 meters per second. As discussed above, the range of speeds can be associated with a probability distribution indicating the probability of a speed within the range of speeds occurring.

[0043] The set of regions can represent portions of an environment that can be used to place objects within the simulated environment. For example, the initial scenario 110 can represent a scenario that includes a two-way multi-lane driving surface associated with a speed limit of 35 miles per hour. Based on the initial scenario 110, the set of regions can exclude regions that would not include a two-way multi-lane driving surface associated with a speed limit of 35 miles per hour, such as parking lots. The parameterized scenarios 122 can be used to cover variations provided by the scenario parameters, error models, region data 118, etc.

[0044] By way of example, and not by way of limitation, the scenario parameters may include a vehicle traversing an environment at a speed of 10, 11, or 12 meters per second (or any speed) while approaching a junction. The set of regions may include an uncontrolled junction, a junction with a four-way stop, and a junction with a traffic light. In addition, the perception error model may indicate a perception error of 1.34%, as may be provided by a perception metric determined for the perception system being tested. Thus, the parameterization scenario 122 may allow for a total of nine different scenarios by varying the scenario parameters and regions (e.g., three speeds * three regions = nine scenarios / permutations). In addition, when the simulation component 124 simulates the parameterization scenario, the simulation component 124 may use the perception error model to introduce a perception error associated with the perception data determined by the vehicle. As can be appreciated, this is merely an example, and the parameterization scenario 122 may include more or fewer permutations, as well as different types, regions, and / or perception errors of the scenario parameters.

[0045] The simulation component 124 can execute the parameterized scenarios 122 as a set of simulation instructions and generate simulation data 126. For example, the simulation component 124 can instantiate a vehicle controller in the simulated scenario. In some examples, the simulation component 124 can execute multiple simulated scenarios simultaneously and / or in parallel. In addition, the simulation component 124 can determine the outcome of the parameterized scenarios 122. For example, the simulation component 124 can execute variations of the parameterized scenarios 122 for use in a simulation for testing and validation. The simulation component 124 can generate simulation data 126 indicative of how the autonomous vehicle controller performed (e.g., responded) and can compare the simulation data 126 to predetermined outcomes and / or determine if any predetermined rules / assertions were violated / triggered.

[0046] In some examples, the variations for the simulation can be selected based on a generalized interval of the scenario parameters. By way of example and not limitation, the scenario parameters can be associated with the speed of the vehicle. In addition, the scenario parameters can be associated with a range of values ​​for the vehicle. The variations for the simulation can be selected based on a generalized interval to increase coverage of the range of values ​​(e.g., selecting the 25th percentile, 50th percentile, 75th percentile, etc.). In some examples, the variations can be selected randomly and / or the variations can be selected randomly within a standard deviation of the range of values.

[0047] In some examples, the predefined rules / assertions can be based on the parameterized scenario 122 (e.g., a traffic rule regarding a pedestrian crossing can be enabled based on a pedestrian crossing scenario, or a traffic rule regarding crossing lane markers can be disabled for a stopped vehicle scenario). In some examples, the simulation component 124 can dynamically enable and disable the rules / assertions as the simulation progresses. For example, as the simulated object approaches a school zone, a rule / assertion related to the school zone can be enabled or disabled as the simulated object leaves the school zone. In some examples, the rules / assertions can include, for example, a comfort metric related to how fast the object can accelerate given the simulated scenario. In at least some examples, the rules can include, for example, following the rules of the road, leaving a safety buffer between objects, etc.

[0048] Based at least in part on determining that the autonomous vehicle controller performed consistent with a predetermined outcome (i.e., the autonomous vehicle controller did everything it was supposed to do) and / or determining that no rules were broken or assertions were triggered, simulation component 124 can determine that the autonomous vehicle controller was successful. Based at least in part on determining that the autonomous vehicle controller's performance was inconsistent with a predetermined outcome (i.e., the autonomous vehicle controller did something it was not supposed to do) and / or determining that a rule was broken or assertions were triggered, simulation component 124 can determine that the autonomous vehicle controller failed. Thus, based at least in part on executing parameterized scenarios 122, simulation data 126 can indicate how the autonomous vehicle controller responds to each variation of parameterized scenarios 122, as described above, and a successful or unsuccessful outcome can be determined based at least in part on simulation data 126.

[0049] The analysis component 128 can be configured to determine a degree of success or failure. By way of example, and not by way of limitation, a rule can indicate that a vehicle controlled by an autonomous vehicle controller must stop within a threshold distance of an object. The simulation data 126 can indicate that in a first variation of the parameterized scenario 122, the simulated vehicle stopped more than 5 meters from the threshold distance. In a second variation of the parameterized scenario 122, the simulation data 126 can indicate that the simulated vehicle stopped more than 10 meters from the threshold distance. The analysis component 128 can indicate that the simulated vehicle performed better in the second variation compared to the simulated vehicle in the first variation. For example, the analysis component 128 can determine an ordered list (e.g., ordered according to a relative success scale) that includes the simulated vehicle and associated variations of the parameterized scenario 122. Such variations can also be used to determine limitations for various components of the simulated system.

[0050] The analysis component 128 can determine additional variations of the parameterized scenario 122 based on the simulation data 126. For example, the simulation data 126 output by the simulation component 124 can indicate variations of the parameterized scenario 122 associated with success or failure (which may be expressed as a continuous likelihood). The analysis component 128 can determine additional variations based on variations associated with failure. By way of example, and not by way of limitation, a variation of the parameterized scenario 122 associated with failure can represent a vehicle crossing the driving surface at a speed of 15 meters per second and an animal crossing the driving surface at a distance of 20 meters ahead of the vehicle. The analysis component 128 can determine additional variations of the scenario to determine additional simulation data 126 for analysis. By way of example, and not by way of limitation, the analysis component 128 can determine additional variations including a vehicle crossing at 10 meters per second, 12.5 meters per second, 17.5 meters per second, 20 meters per second, and so on. Additionally, the analysis component 128 can determine additional variations including animals crossing the driving surface at distances such as 15 meters, 17.5 meters, 22.5 meters, and 25 meters. The additional variations can be input to the simulation component 124 to generate additional simulation data. Such additional variations can be determined, for example, based on perturbations to scenario parameters of the scenarios executed in the simulation.

[0051] In some examples, the analysis component 128 can determine additional variations of the scenario by disabling the scenario parameters. For example, the parameterized scenario can include a first scenario parameter associated with the speed of the object and a second scenario parameter associated with the position of the object. The parameterized scenario 122 can include a first range of values ​​associated with the speed and a second range of values ​​associated with the position. In some examples, after simulating the parameterized scenario 122, the simulation data 126 can indicate that some variations of the parameterized scenario 122 resulted in successful outcomes and some variations resulted in unsuccessful outcomes. The analysis component 128 can then determine to disable the first scenario parameter (e.g., set a fixed value associated with the first scenario parameter) and vary the parameterized scenario 122 based on the second scenario parameter. By disabling one of the scenario parameters, the analysis component 128 can determine whether the scenario parameter and / or the value of the scenario parameter is associated with a successful outcome or a failed outcome. Such parameters can be disabled based on a likelihood of failure, randomly, or otherwise. By way of example, and not limitation, simulation data 126 as a result of invalidating all of the scenario parameters may indicate a problem with the planning component of an autonomous vehicle.

[0052] In some examples, the analysis component 128 can be used to perform sensitivity analysis. For example, the analysis component 128 can disable scenario parameters and determine, based on the simulation data 126 generated by the simulation component 124, how disabling the scenario parameters affects the simulation data 126 (e.g., increases the success rate, decreases the success rate, minimally affects the success rate). In some examples, the analysis component 128 can disable scenario parameters individually and determine how disabling each scenario parameter affects the simulation data 126. The analysis component 128 can collect statistical data indicative of how individual scenario parameters affect the simulation data 126 over the course of many simulations. In some examples, the analysis component 128 can be configured to disable a set of scenario parameters (e.g., disable a nighttime environment parameter and disable a wet conditions environment parameter). As described above, the analysis component can collect statistical data indicative of how a set of scenario parameters affects the simulation data 126. The statistical data can be used to determine scenario parameters that increase or decrease the likelihood of resulting in a successful simulation, and can be used to identify subsystems of the autonomous vehicle associated with scenario parameters that increase or decrease the success rate of the simulation data 126.

[0053] In some examples, the analysis component 128 can adjust the extent to which the scenario parameters are adjusted. By way of example, and not limitation, the scenario parameters can indicate wet environmental conditions (e.g., rainy conditions). The scenario parameters can be adjusted over a range (e.g., a quarter inch of rain, an inch of rain, etc.). The analysis component 128 can adjust the magnitude of the scenario parameters and perform a sensitivity analysis based on the magnitude of the scenario parameters to determine thresholds associated with the scenario parameters that can result in a successful or unsuccessful outcome of the simulation. In some examples, the thresholds can be determined using a binary search algorithm, a particle filter algorithm, and / or a Monte Carlo method, although other suitable algorithms are contemplated.

[0054] Vehicle performance component 130 can determine vehicle performance data 132 based on simulation data 126 (and / or additional simulation data based on additional variations from analysis component 128) and the type of failure. In some examples, vehicle performance data 132 can indicate how the vehicle performs in an environment. By way of example, and not by way of limitation, vehicle performance data 132 can indicate that a vehicle traversing at a speed of 15 meters per second has a stopping distance of 15 meters. In some examples, vehicle performance data can indicate a safety metric. By way of example, and not by way of limitation, vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such an indication can be binary (presence or absence of failure), coarse (level of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing the likelihood of failure), although any other indication is contemplated. For example, for event type 1 and cause type 1, data 134(1) can indicate a safety rating, as can data 134(2) through 134(4). In some examples, cause type 1 and cause type 2 may indicate a fault, such as a vehicle fault or an object (e.g., a cyclist) fault. The vehicle performance data 132 may then indicate a safety metric associated with the parameterization scenario 122. In some examples, the vehicle performance component 130 may use a target metric and compare the vehicle performance data 132 to the target metric to determine whether the safety metric meets or exceeds the target metric. In some examples, the target metric may be based on a standard and / or regulation associated with the autonomous vehicle.

[0055] In some examples, the vehicle performance data 132 can be input to a filter component 134 to determine filtered data 136 based on the vehicle performance data 132. For example, the filter component 134 can be used to determine filtered data 136 that identifies areas that do not meet a coverage threshold. By way of example, and not limitation, the initial scenario 110 can indicate a speed limit of 35 miles per hour and a two-way multi-lane driving surface associated with the area data 118. Based on the initial scenario 110 and the area data 118, the parameters component 112 can identify five areas of the environment that can be used to simulate the initial scenario 110. After simulating the parameterized scenario 122, the vehicle performance data 132 can indicate that simulation data 126 is associated with three of the five areas. For example, the simulation component 124 can simulate the scenario and generate simulation data 126 based on running the scenario in three of the five areas identified by the parameters component 112. The filter component 134 can determine filtered data 136 that indicates that the remaining two of the five regions do not meet a coverage threshold (e.g., a minimum number of simulations associated with the region) based on one or more filters.

[0056] In some examples, the filter component 134 can determine filtered data 136 indicative of an occurrence of an event based on the vehicle performance data 132. By way of example, and not limitation, the simulation data 126 can include an occurrence of an event such as an emergency stop, a flat tire, an animal crossing the driving surface, etc. The filter component 134 can determine filtered data 136 indicative of an occurrence of an emergency stop based on one or more filters. Additionally, the filtered data 136 can include a portion of the simulation data 126 associated with the occurrence of an emergency stop, such as, for example, a stopping distance associated with the emergency stop.

[0057] FIG. 2 illustrates an example 200 of a vehicle 202, similar to the vehicle generating vehicle data 104 described with reference to FIG. 1, that may generate and transmit the vehicle data 104 to a computing device 204. As described above, the scenario editor component 108 may be configured to scan the input data 102 (e.g., the vehicle data 104 and / or the additional situation data 106) and identify one or more scenarios represented in the input data 102. As a non-limiting example, such scenarios may be determined, for example, based on clustering (e.g., using k-means, etc.) a parameterization of the log data. In some examples, the scenario editor component 108 may use scenario definition data 206 to identify one or more scenarios represented in the input data 102. For example, the scenario definition data 206 may identify features associated with a type of scenario. As an example, but not by way of limitation, the scenario definition data 206 may identify a jaywalking scenario that includes features such as a pedestrian crossing a portion of a driving surface that is not associated with a crosswalk. The scenario editor component 108 can scan the input data 102 to identify portions of the input data that include characteristics of a pedestrian crossing a portion of the driving surface that is not associated with a crosswalk to determine a jaywalking scenario. In at least some examples, such scenarios may be further manually input and / or derived from third party data (e.g., police reports, publicly available video clips, etc.).

[0058] In addition, as described above, the parameters component 112 can determine scenario parameters, which can indicate values ​​or ranges of values ​​associated with parameters of objects in a scenario. The parameters component 112 can generate scenario data 208, as depicted in FIG.

[0059] The scenario data 208 may represent a base scenario including a vehicle 210 traversing along a driving surface, and an object 212 (which may be a different vehicle) traversing near the vehicle 210 in the same direction as the vehicle 210. The vehicle 210 and the object 212 may approach a junction with a pedestrian crossing. The parameter component 112 may determine scenario parameters indicating a distance between the vehicle 210 and the object 212 as having a range of distances. Thus, scenario S1 may represent a first scenario of a set of scenarios having a first distance between the vehicle 210 and the object 212, scenario S2 may represent a second scenario of a set of scenarios having a second distance between the vehicle 210 and the object 212, and so on. N may represent the Nth scenario of the set of scenarios having the Nth distance between the vehicle 210 and the object 212. Examples of additional types of parameters (also referred to as attributes) may be found, for example, in U.S. patent application Ser. No. 16 / 363,541, entitled "Pedestrian Prediction Based on Attributes," filed Mar. 25, 2019, and incorporated by reference herein in its entirety.

[0060] For example, the scenario parameters may include a velocity of the object 212, an acceleration of the object 212, an x-position of the object 212 (e.g., a position relative to a global position, a local position, and / or any other frame of reference), a y-position of the object 212 (e.g., a size, a pose, a local position, a global position, and / or a position relative to any other frame of reference), a bounding box associated with the object 212 (e.g., extent (length, width, and / or height), yaw, pitch, roll, etc.), lighting conditions (e.g., brake lights, blinker lights, hazard lights, headlights, reverse lights, etc.), a wheel position of the object 212, ... These may include, but are not limited to, direction of the lane, map elements (e.g., distance between the object 212 and a traffic light, a stop sign, a speed bump, an intersection, a yield sign, etc.), classification of the object 212 (e.g., vehicle, car, truck, bicycle, motorcycle, pedestrian, animal, etc.), object characteristics (e.g., whether the object is changing lanes, whether the object 212 is a double parked vehicle, etc.), proximity to one or more objects (in any coordinate frame), lane type (e.g., lane direction, parking lane, etc.), road markings (e.g., indicating whether passing or lane changing is allowed, etc.), object density, etc.

[0061] As described above, parameter component 112 can determine ranges of values ​​associated with scenario parameters as represented by vehicle data 104 and / or other input data. Thus, each of the example scenario parameters identified above, as well as other scenario parameters, can be associated with a set or range of values ​​that can be used to generate a set of scenarios, where the scenarios of the set of scenarios differ by one or more values ​​of the scenario parameters.

[0062] 3 illustrates an example 300 of generating error model data based at least in part on vehicle data and ground truth data. As illustrated in FIG. 3, a vehicle 202 can generate vehicle data 104 and transmit the vehicle data 104 to an error model component 116. As described above, the error model component 116 can determine an error model that can indicate errors associated with scenario parameters. For example, the vehicle data 104 can be data associated with a subsystem of the vehicle 202, such as a perception system, a planning system, a tracking system (also referred to as a tracker system), a prediction system, etc. By way of example, and not limitation, the vehicle data 104 can be associated with a perception system, and the vehicle data 104 can include bounding boxes associated with objects detected by the vehicle 202 in the environment.

[0063] The error model component 116 can receive ground truth data 302, which can be determined from manually labeled and / or other validated machine learning components. By way of example, and not limitation, the ground truth data 302 can include validated bounding boxes associated with objects in the environment. By comparing the bounding boxes of the vehicle data 104 to the bounding boxes of the ground truth data 302, the error model component 116 can determine errors associated with subsystems of the vehicle 202. In some examples, the vehicle data 104 can include one or more characteristics (also referred to as parameters) associated with the detected entity and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity can include, but are not limited to, x-position (global location), y-position (global location), z-position (global location), orientation, entity type (e.g., classification, etc.), velocity of the entity, range (size) of the entity, etc. The characteristics associated with the environment can include, but are not limited to, the presence of another entity in the environment, the state of another entity in the environment, time of day, day of the week, season, weather conditions, dark / light indication, etc. Therefore, the error can be related to other characteristics (eg, environmental parameters).

[0064] The error model component 116 can process the plurality of vehicle data 104 and the plurality of ground truth data 302 to determine the error model data 304. The error model data 304 can include errors calculated by the error model component 116, which can be represented as errors 306(1)-(3). In addition, the error model component 116 can determine probabilities associated with the errors 306(1)-(3), which can be associated with environmental parameters to present the error models 310(1)-(3). By way of example and not limitation, the vehicle data 104 can include a bounding box associated with an object at a distance of 50 meters from the vehicle 202 in an environment including rainfall. The ground truth data 302 can provide a verified bounding box associated with the object. The error model component 116 can determine the error model data 304 determining that an error is associated with a perception system of the vehicle 202. A distance of 50 meters and rainfall can be used as environmental parameters to determine which of error models 310(1)-(3) to use. Once an error model is identified, error models 310(1)-(3) can provide errors 306(1)-(3) based on probabilities 308(1)-(3), where errors 306(1)-(3) associated with higher probabilities 308(1)-(3) are more likely to be selected than errors 306(1)-(3) associated with lower probabilities 308(1)-(3).

[0065] 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or uncertainty associated with a simulated environment. As described above, the error model data can include an error model that associates errors 306 with probabilities 308, and the error models can be associated with environmental parameters. The simulation component 124 can inject errors using the error model data 304, which can result in perturbed parameterized scenarios for perturbing the simulation. Based on the injected errors, the simulation data can indicate how the autonomous controller responds to the injected errors.

[0066] For example, the simulation component 124 can perturb the simulation by successively injecting errors into the simulation. By way of example, and not by way of limitation, example 402 shows a bounding box 404 associated with an object at time t0. The bounding box 404 can represent a detection of the object by a vehicle, including errors such as errors in the size of the bounding box and / or the location of the bounding box. At time t1, the simulation component 124 can use a bounding box 406 representing the object and including different errors. For example, at each simulation time (e.g., t0, t1, or t2), the simulation component 124 can use a different error 306 based on a probability 308 associated with the error 306. At time t2, the simulation component 124 can use a bounding box 408 representing the object and including different errors.

[0067] In some examples, the simulation component 124 can perturb the simulation by injecting uncertainty associated with a bounding box that represents an object in the environment. By way of example, and not limitation, example 410 shows a bounding box 412 associated with an object at time t0. The bounding box can include a 5% uncertainty, which can represent uncertainty in the size and / or position of the object by an amount of 5%. Additionally, the uncertainty can persist with the object through times t1 and t2, rather than injecting different errors at different simulation times, as depicted in example 402.

[0068] FIG. 5 illustrates an example 500 of a vehicle 202 generating vehicle data 104 and transmitting the vehicle data 104 to a computing device 204. As described above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As described above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle 202 and / or perception data generated by a perception system of the vehicle 202. The perception error model can be determined by comparing the vehicle data 104 to ground truth data 302. The ground truth data 302 can be manually labeled, can be associated with an environment, and can represent known outcomes. Thus, deviations from the ground truth data 302 in the vehicle data 104 can be identified as errors in the sensor system and / or perception system of the vehicle 202. By way of example, and not by way of limitation, the perception system may identify an object as a cyclist, where the ground truth data 302 indicates that the object is a pedestrian. As another example, without limitation, a sensor system may generate sensor data that represents an object as having a width of 2 meters, where the ground truth data 302 indicates that the object has a width of 1.75 meters.

[0069] As described above, the error model component 116 can determine a classification associated with an object represented in the vehicle data 104 and determine other objects of the same classification in the vehicle data 104 and / or other log data. The error model component 116 can then determine a probability distribution associated with a range of errors associated with the object. Based on the comparison and range of errors, the error model component 116 can determine the perceptual error model data 502.

[0070] As shown in Figure 5, environment 504 can include objects 506(1)-(3), which are represented as bounding boxes generated by a perception system. Perception error model data 502 can indicate scenario parameters as 508(1)-(3) and errors associated with the scenario parameters as 510(1)-(3). As shown in Figure 5, the errors associated with object 508(1) can be visualized in environment 504 as a larger bounding box 512 that indicates uncertainty about the dimensions of object 508(1).

[0071] FIG. 6 illustrates an example 600 of a computing device 204 that generates simulation data 126 and determines vehicle performance data 132. The parameterization scenario component 120 can determine a parameterization scenario based on a scenario parameter, a set of regions, and a perception error model. For example, the scenario parameter can indicate an object, a position associated with the object, a speed associated with the object, etc., in the parameterization scenario. In addition, the scenario parameter can indicate a range indicating a range of values ​​and / or probabilities associated with the scenario parameter. The set of regions can indicate a portion of the environment that can be used to place an object in the simulated environment. Furthermore, the perception error model can indicate an error associated with the scenario parameter. These can be combined to create a parameterization scenario that can cover the variations provided by the scenario parameter, the set of regions, and / or the perception error model, as detailed herein.

[0072] The parameterized scenarios can be used by the simulation component 124 to simulate variations of the parameterized scenarios. For example, the simulation component 124 can execute variations of the parameterized scenarios for use in a simulation for testing and validation. The simulation component 124 can generate simulation data 126 that indicates how the autonomous vehicle controller performed (e.g., responded) and can compare the simulation data 126 to predetermined results and / or determine if any predetermined rules / assertions were violated / triggered.

[0073] As illustrated in FIG. 6, the simulation data 126 can indicate several simulations (e.g., Simulation 1, Simulation 2, etc.) and results of the simulations (e.g., Outcome 1, Outcome 2). For example, as described above, the results can indicate pass or fail based on rules / assertions that were broken / triggered. In addition, the simulation data 126 can indicate the probability of encountering a scenario. By way of example, and not by way of limitation, the simulation component 124 can simulate a scenario involving a jaywalking pedestrian. The input data can indicate that the vehicle encounters a jaywalking pedestrian at a rate of one minute per hour of driving. This can be used to determine the probability of encountering a particular simulation associated with a variation of the parameterized scenario. In some examples, the simulation component 124 can identify variations of the parameterized scenario that have a low probability and run simulations corresponding to those variations. This can enable testing and validation of the autonomous vehicle controller in more unique situations.

[0074] Additionally, the simulation component 124 can identify parameterized scenario variations for additional simulations based on the results. By way of example, and not limitation, the results of a simulation can be a fault where the scenario parameters are associated with a vehicle speed of 15 meters per second. The simulation component 124 can identify speeds close to the 15 meters per second speed to determine thresholds that the simulation will pass, which can further aid in the development of a safer vehicle controller.

[0075] Based on the simulation data 126, vehicle performance component 130 can generate vehicle performance data 132. As described above, for example, for event type 1 and cause type 1, data 134(1) can indicate a safety rating, as can data 134(2) through 134(4). In some examples, the event type can indicate a cost meeting or exceeding a cost threshold, although other event types are contemplated. For example, the cost can include, but is not limited to, a reference cost, an obstacle cost, a lateral cost, a longitudinal cost, etc.

[0076] The reference cost may include a cost associated with a difference between a point on the reference trajectory (also referred to as a reference point) and a corresponding point on the target trajectory (also referred to as a point or a target point), whereby the difference represents one or more differences in yaw, lateral offset, velocity, acceleration, curvature, curvature rate, etc. In some examples, reducing the weight associated with the reference cost may reduce the penalty associated with a target trajectory that is located a distance away from the reference trajectory, which may provide a smoother transition leading to safer and / or more comfortable vehicle operation.

[0077] In some examples, the obstacle cost may include a cost associated with a distance between a point on a reference trajectory or a target trajectory and a point associated with an obstacle in the environment. As an example, the point associated with an obstacle may correspond to a point on a boundary of a drivable area or may correspond to a point associated with an obstacle in the environment. In some examples, the obstacle in the environment may include, but is not limited to, a static object (e.g., a building, a curb, a sidewalk, a lane marking, a sign, a traffic light, a tree, etc.) or a dynamic object (e.g., a vehicle, a bicyclist, a pedestrian, an animal, etc.). In some examples, the dynamic object may also be referred to as an agent. In some examples, the static object or the dynamic object may be generally referred to as an object or an obstacle.

[0078] In some examples, the lateral cost may refer to a cost associated with a steering input to the vehicle, such as a maximum steering input relative to the speed of the vehicle. In some examples, the longitudinal cost may refer to a cost associated with the speed and / or acceleration of the vehicle (e.g., maximum braking and / or acceleration). Such costs may be used to ensure that the vehicle is operating within feasible and / or comfort limits for passengers operating on the ferry.

[0079] In some examples, cause type 1 and cause type 2 may indicate a fault, such as a fault in the vehicle or a fault in an object (e.g., a cyclist). The vehicle performance component 130 may determine the fault using predefined rules / assertions. By way of example, and without limitation, a rule may indicate that if the vehicle is struck by an object behind the vehicle, then the fault may be associated with the object. In some examples, additional rules may be used, such as indicating that the vehicle must traverse the environment in a forward direction when struck by an object behind. In some examples, the cause types (e.g., cause type 1 and / or cause type 2) may be associated with components of the autonomous vehicle controller. By way of non-limiting example, such causes may include failures of a perception system, a prediction system, a planner system, network latency, torque / acceleration faults, and / or any other component or subcomponent of the vehicle.

[0080] As described above, the analysis component can determine to disable a scenario parameter (e.g., set a fixed value associated with the scenario parameter) and vary other scenario parameters based on the simulation data 126. By isolating the scenario parameters, the analysis component can determine the scenario parameters associated with a successful or failed outcome. The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario. In addition, the analysis component can perform a sensitivity analysis to determine the cause of failure. For example, the analysis component can individually disable the scenario parameters to isolate one or more scenario parameters, determine how disabling the scenario parameters affects the response of the autonomous vehicle, capture statistical data associated with disabling one or more scenario parameters, and capture the outcome as a successful or failed outcome. The statistical data can indicate how a set of scenario parameters affects the outcome and can be used to determine scenario parameters that increase or decrease the likelihood of resulting in a successful simulation and can be used to identify subsystems of the autonomous vehicle associated with the scenario parameters as increasing or decreasing the success rate of the simulation data.

[0081] 7 shows a block diagram of an example system 700 for implementing the techniques described herein. In at least one example, the system 700 can include a vehicle 202. In the illustrated example 700, the vehicle 202 is an autonomous vehicle, although the vehicle 202 can be any other type of vehicle (e.g., a driver-controlled vehicle that can provide an indication of whether it is safe to perform various operations).

[0082] Vehicle 202 may include a computing device 702, one or more sensor systems 704, one or more emitters 706, one or more communication connections 708 (also referred to as communication devices and / or modems), at least one direct connection 710 (e.g., for physically coupling with vehicle 202 to exchange data and / or provide power), and one or more drive systems 712. The one or more sensor systems 704 may be configured to capture sensor data associated with an environment.

[0083] The sensor system 704 may include time-of-flight sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement unit (IMU), accelerometer, magnetometer, gyroscope, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, intensity, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, etc. The sensor system 704 may include multiple instances of each of these or other types of sensors. For example, the time-of-flight sensors may include individual time-of-flight sensors positioned at the corners, front, rear, sides, and / or top of the vehicle 202. As another example, the camera sensors may include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 202. The sensor system 704 may provide input to the computing device 702.

[0084] The vehicle 202 may also include one or more emitters 706 for emitting light and / or sound. The one or more emitters 706 in this example include interior audio and visual emitters for communicating with passengers of the vehicle 202. By way of illustration, and not by way of limitation, the interior emitters may include speakers, lights, symbols, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The one or more emitters 706 in this example also include exterior emitters. By way of example, and not by way of limitation, the exterior emitters in this example include lights for signaling a direction of travel or other indicators of the operation of the vehicle (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with one or more pedestrians or other nearby vehicles, which may include acoustic beam steering technology.

[0085] Vehicle 202 may also include one or more communications connections 708 that enable communication between vehicle 202 and one or more other local or remote computing devices (e.g., remote teleoperated computing devices) or remote services. For example, communications connections 708 may facilitate communication with other local computing devices on vehicle 202 and / or drive system 712. Communications connections 708 may also enable vehicle 202 to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.).

[0086] The communications connections 708 may include physical and / or logical interfaces for connecting the computing device 702 to another computing device or to one or more external networks 714 (e.g., the Internet). For example, the communications connections 708 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4GLTE, 5G, etc.), satellite communications, dedicated short-range communications (DSRC), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices. In at least some examples, the communications connections 708 may comprise one or more modems, as described in detail above.

[0087] In at least one example, the vehicle 202 can include one or more drive systems 712. In some examples, the vehicle 202 can have a single drive system 712. In at least one example, if the vehicle 202 has multiple drive systems 712, individual drive systems 712 can be located at both ends of the vehicle 202 (e.g., the front and the rear, etc.). In at least one example, the drive system 712 can include one or more sensor systems 704 for detecting conditions surrounding the drive system 712 and / or the vehicle 202. By way of example and not limitation, the sensor systems 704 can include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive system, cameras or other imaging sensors, ultrasonic sensors, lidar sensors, radar sensors, etc. for acoustically detecting objects surrounding the drive system. Some sensors, such as wheel encoders, can be specific to the drive system 712. In some cases, the sensor systems 704 on the drive system 712 may overlap or complement corresponding systems (e.g., sensor system 704) of the vehicle 202.

[0088] The drive system 712 can include many vehicle systems, including a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for braking force distribution to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the vehicle's exterior environment), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). In addition, the drive system 712 can include a drive system controller that can receive and pre-process data from the sensor system 704 and control the operation of various vehicle systems. In some examples, the drive system controller can include one or more processors and a memory communicatively coupled to the one or more processors. The memory can store one or more modules for performing various functions of the drive system 712. Additionally, drive system 712 also includes one or more communications connections that enable each drive system to communicate with one or more other local or remote computing devices.

[0089] Computing device 702 may include one or more processors 516 and memory 518 communicatively coupled to the one or more processors 716. In the illustrated example, memory 718 of computing device 702 stores a localization component 720, a perception component 722, a prediction component 724, a planning component 726, and one or more system controllers 728. While shown as residing in memory 718 for illustrative purposes, it is contemplated that localization component 720, perception component 722, prediction component 724, planning component 726, and one or more system controllers 728 may additionally or alternatively be accessible to computing device 702 (e.g., stored in a different component of vehicle 202) and / or accessible to vehicle 202 (e.g., stored remotely).

[0090] In the memory 718 of the computing device 702, the localization component 720 can include functionality for receiving data from the sensor system 704 to determine the position of the vehicle 202. For example, the localization component 720 can include and / or request / receive a three-dimensional map of the environment and can continually determine the position of the autonomous vehicle within the map. In some examples, the localization component 720 can use SLAM (Simultaneous Localization and Mapping) or CLAMS (Calibration, Simultaneous Localization and Mapping) to receive time-of-flight data, image data, lidar data, radar data, sonar data, IMU data, GPS data, wheel encoder data, or any combination thereof, etc. to precisely determine the position of the autonomous vehicle. In some examples, the localization component 720 can provide data to various components of the vehicle 202 to determine an initial position of the autonomous vehicle for generating a trajectory, as discussed herein.

[0091] The perception component 722 may include functionality for performing object detection, segmentation, and / or classification. In some examples, the perception component 722 may provide processed sensor data indicative of the presence of an entity proximate to the vehicle 202 and / or the classification of the entity as an entity type (e.g., automobile, pedestrian, bicyclist, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional and / or alternative examples, the perception component 722 may provide processed sensor data indicative of one or more characteristics (also referred to as parameters) associated with the detected entity and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity may include, but are not limited to, x-position (global location), y-position (global location), z-position (global location), direction, entity type (e.g., classification, etc.), velocity of the entity, range (size) of the entity, etc. The characteristics associated with the environment may include, but are not limited to, the presence of another entity in the environment, the state of another entity in the environment, time of day, day of the week, season, weather conditions, geographic location, dark / light indication, etc.

[0092] The perception component 722 may include functionality for storing the perception data generated by the perception component 722. In some examples, the perception component 722 may determine a track corresponding to an object classified as an object type. For illustrative purposes only, the perception component 722 using the sensor system 704 may capture one or more images of an environment. The sensor system 704 may capture an image of the environment including an object, such as a pedestrian. The pedestrian may be at a first location at time T and at a second location at time T+t (e.g., movement during a span of time t after time T). In other words, the pedestrian may move from a first location to a second location during this time period. Such movement may be logged, for example, as stored perception data associated with the object.

[0093] The stored sensory data may include fused sensory data captured by the vehicle in some examples. The fused sensory data may include a fusion or other combination of sensor data from sensor systems 704, such as image sensors, lidar sensors, radar sensors, time-of-flight sensors, sonar sensors, global positioning system sensors, interior sensors, and / or any combination thereof. The stored sensory data may additionally or alternatively include classification data including a semantic classification of an object (e.g., pedestrian, vehicle, building, road surface, etc.) represented in the sensor data. The stored sensory data may additionally or alternatively include track data (a collection of historical positions, orientations, sensor features, etc. associated with an object over time) corresponding to the movement of an object classified as a dynamic object through the environment. The track data may include multiple tracks of multiple different objects over time. This track data may be mined to identify images of a particular type of object (e.g., pedestrian, animal, etc.) when the object is stationary (e.g., stationary) or moving (e.g., walking, running, etc.). In this example, the computing device determines a track corresponding to a pedestrian.

[0094] The prediction component 724 can generate one or more probability maps that represent predicted probabilities of possible locations of one or more objects in the environment. For example, the prediction component 724 can generate one or more probability maps for vehicles, pedestrians, animals, etc., that are within a threshold distance from the vehicle 202. In some examples, the prediction component 724 can measure the tracks of the objects and generate discretized predicted probability maps, heat maps, probability distributions, discretized probability distributions, and / or trajectories of the objects based on the observed and predicted behavior. In some examples, the one or more probability maps can represent the intent of one or more objects in the environment.

[0095] The planning component 726 can determine a path for the vehicle 202 to follow to traverse through an environment. For example, the planning component 726 can determine various routes and paths as well as various levels of detail. In some examples, the planning component 726 can determine a route for traveling from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this description, the route can be a series of waypoints for traveling between the two locations. As non-limiting examples, the waypoints include roads, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 726 can generate instructions for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 726 can determine how to guide the autonomous vehicle from a first waypoint in the series of waypoints to a second waypoint in the series of waypoints. In some examples, the instructions can be a path or a portion of a path. In some examples, multiple paths can be generated substantially simultaneously (i.e., within technical tolerances) according to a receding horizon technique. The single path of multiple paths in the receding data horizon that has the highest confidence level can be selected for maneuvering the vehicle.

[0096] In other examples, the planning component 726 can alternatively or additionally use data from the perception component 722 to determine a path for the vehicle 202 to follow to traverse through the environment. For example, the planning component 726 can receive data from the perception component 722 regarding objects associated with the environment. Using this data, the planning component 726 can determine a path to travel from a first location (e.g., a current location) to a second location (e.g., a target location) to avoid the objects in the environment. In at least some examples, such a planning component 726 can determine that there is no such collision-free path and then provide a path that leads the vehicle 202 to a safe stop that avoids all collisions and / or otherwise mitigates damage.

[0097] In at least one example, computing device 702 can include one or more system controllers 728, which can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 202. These system controllers 728 can communicate with and / or control corresponding systems of drive system 712 and / or other components of vehicle 202, which can be configured to operate according to the path provided from planning component 726.

[0098] Vehicle 202 can be connected to computing device 204 via network 514 and can include one or more processors 730 and memory 732 communicatively coupled to the one or more processors 730. In at least one example, the one or more processors 730 can be similar to processor 716, and memory 732 can be similar to memory 718. In the illustrated example, memory 732 of computing device 204 stores scenario editor component 108, parameter component 112, error model component 116, parameterized scenario component 120, simulation component 124, analysis component 128, and vehicle performance component 130. While depicted as residing in memory 732 for illustrative purposes, it is contemplated that the scenario editor component 108, the parameter component 112, the error model component 116, the parameterized scenario component 120, the simulation component 124, the analysis component 128, and the vehicle performance component 130 may additionally or alternatively be accessible to the computing device 204 (e.g., stored in a different component of the computing device 204) and / or accessible to the computing device 204 (e.g., stored remotely). The scenario editor component 108, the parameter component 112, the error model component 116, the parameterized scenario component 120, the simulation component 124, the analysis component 128, and the vehicle performance component 130 may be substantially similar to the scenario editor component 108, the parameter component 112, the error model component 116, the parameterized scenario component 120, the simulation component 124, the analysis component 128, and the vehicle performance component 130 of FIG.

[0099] The processor 716 of the computing device 702 and the processor 730 of the computing device 204 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processors 716 and 730 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it into registers or other electronic data that can be stored in memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0100] Memory 718 of computing device 702 and memory 732 of computing device 204 are examples of non-transitory computer-readable media. Memory 718 and 732 can store an operating system and one or more software applications, instructions, programs, and / or data that implement the methods and functions attributed to the various systems described herein. In various implementations, memory 718 and 732 can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, of which those illustrated in the accompanying drawings are merely examples relevant to the description herein.

[0101] In some examples, some or all aspects of the components described herein may include any model, algorithm, and / or machine learning algorithm. For example, in some examples, the components in memories 718 and 732 may be implemented as neural networks.

[0102] 8 illustrates an example process 800 for determining a safety metric associated with a vehicle controller. Some or all of the process 800 may be performed by one or more components of Figures 1-7 as described herein. For example, some or all of the process 800 may be performed by computing device 204 and / or computing device 702.

[0103] At operation 802 of the example process 800, the process 800 may include receiving log data associated with operation of the autonomous vehicle within an environment. In some examples, the log data may be generated by the vehicle capturing at least sensor data of the environment.

[0104] At operation 804 of the example process 800, the process 800 can include determining a set of scenarios based on the log data (or other data), where the scenarios of the set of scenarios include scenario parameters associated with aspects of the environment. In some examples, the computing device can group similar scenarios represented in the log data. For example, the scenarios can be grouped using, for example, k-means clustering and / or evaluating a weighted distance (e.g., Euclidean) between parameters of the environment. Additionally, the scenario parameters can represent environmental parameters, such as nighttime environmental parameters or wet condition environmental parameters. In some examples, the scenario parameters can be associated with a vehicle or object (e.g., pose, speed, etc.).

[0105] At operation 806 of the example process 800, the process 800 can include determining an error model associated with a subsystem of the autonomous vehicle. The error model component can compare the vehicle data (e.g., log data) to ground truth data to determine a difference between the vehicle data and the ground truth data. In some examples, the vehicle data can represent estimates associated with an object in the environment, such as an estimated position, an estimated orientation, an estimated range, etc., and the ground truth data can represent an actual position, an actual orientation, or an actual range of the object. Based on the difference, the error model component can determine an error associated with the subsystem of the vehicle (e.g., a perception system, a tracking system, a prediction system, etc.).

[0106] In operation 808 of the example process 800, the process 800 can include determining parameterized scenarios based on the scenario parameters and the error model. These can be combined to create parameterized scenarios that can cover the variations provided by the scenario parameters and / or the error model. In some examples, the scenario parameters can be randomly selected and combined to create the parameterized scenarios. In some examples, the scenario parameters can be combined based on the probability of them occurring together. As an example, but not by way of limitation, the log data can indicate that 5% of driving experiences include pedestrian encounters, and the parameterized scenario component can include pedestrians as a scenario parameter in 5% of the parameterized scenarios generated by the parameterized scenario component. In some examples, the parameterized scenario component can validate the parameterized scenarios to eliminate unlikely or unlikely combinations of scenario parameters. As an example, but not by way of limitation, the vehicle is not located in a lake and pedestrians are not walking at a speed of 30 meters per second. As non-limiting examples, such parameterized scenarios may include ranges of distances, speeds, lighting conditions, weather conditions, etc. of vehicles and jaywalkers on particular defined roads with various Gaussian (or other) distributions of errors in perception models, predictive models, etc. based at least in part on scenario parameters.

[0107] At operation 810 of the example process 800, the process 800 may include perturbing the parameterized scenario by modifying at least one of the parameterized scenario, the scenario parameters, or the components of the simulated vehicle based at least in part on the error. In some examples, an uncertainty may be associated with the scenario parameters. By way of example and not limitation, an object's position may be associated with a 5% uncertainty and an autonomous controller may be caused to traverse the environment while accounting for the 5% uncertainty. In some examples, the simulator may determine from the error model the error to incorporate into the simulation as the simulator runs the simulation.

[0108] At operation 812 of the example process 800, the process 800 can include instantiating a simulated vehicle in the perturbed parameterized scenario. The simulator can use the simulated vehicle, which can be associated with an autonomous controller, to cause the autonomous controller to traverse the simulated environment. Instantiating the autonomous vehicle controller with the parameterized scenario and simulating the parameterized scenario can efficiently cover a wide range of scenario variations without requiring manual enumeration of the variations. Additionally, based at least in part on executing the parameterized scenario, simulation data can indicate how the autonomous vehicle controller responded to the parameterized scenario, and a successful or unsuccessful outcome can be determined based at least in part on the simulation data.

[0109] At operation 814 of the example process 800, the process may include receiving simulation data indicating how a simulated vehicle responds to the perturbed parameterization scenario. After the simulation, the results may indicate a pass (e.g., a successful outcome), a fail, and / or a degree of success or failure associated with the vehicle controller.

[0110] In operation 816 of the example process 800, the process may include determining a safety metric associated with the vehicle controller based on the simulation data. For example, each simulation may result in a success or failure outcome. Additionally, as described above, the vehicle performance component may determine vehicle performance data based on the simulation data that may indicate how the vehicle performs in the environment. Based on the sensitivity analysis, the vehicle performance data may indicate scenarios where the simulation outcome was not successful, causes of the simulation failure, and / or bounds of scenario parameters that indicate values ​​of the scenario parameters where the simulation outcome was successful. Thus, the safety metric may indicate pass / fail rates of the vehicle controller in various simulated scenarios.

[0111] 9 illustrates a flow diagram of an example process for determining a statistical model associated with a subsystem of an autonomous vehicle. Some or all of the process 900 may be performed by one or more components of FIGS. 1-7 as described herein. For example, some or all of the process 900 may be performed by computing device 204 and / or computing device 702.

[0112] At operation 902 of the example process 900, the process 900 may include receiving vehicle data (or other data) associated with a subsystem of an autonomous vehicle. The vehicle data may include log data captured by the vehicle traversing through an environment. In some examples, the vehicle data may include control data (e.g., data used to control systems such as steering, braking, etc.) and / or sensor data (e.g., lidar data, radar data, etc.).

[0113] At operation 904 of the example process 900, the process 900 may include determining output data associated with a subsystem based on the vehicle data. By way of example, and without limitation, the subsystem may be a perception system and the output data may be a bounding box associated with an object in the environment.

[0114] At operation 906 of the example process 900, the process 900 can include receiving ground truth data associated with the subsystem. In some examples, the ground truth data can be determined from manual labeling and / or other validated machine learning components. By way of example, and without limitation, the ground truth data can include validated bounding boxes associated with objects in the environment.

[0115] At operation 908 of the example process 900, the process 900 may include determining a difference between a first portion of the output data and a second portion of the ground truth data, the difference representing an error associated with the subsystem. As described above, the output data may include a bounding box associated with an object in the environment detected by the vehicle's perception system, and the ground truth data may include a verified bounding box associated with the object. The difference between the two bounding boxes may indicate an error associated with the vehicle's perception system. By way of example, and not by way of limitation, the bounding box of the output data may be larger than the verified bounding box, indicating that the perception system has detected that the object is larger than in the environment.

[0116] In operation 910 of the example process 900, the process 900 can include determining a statistical model associated with the subsystem based on the difference. For example,

[0117] [Example of invention content] A: A system including one or more processors; and one or more computer-readable media having computer-executable instructions stored thereon that, when executed, cause the system to perform operations including receiving log data associated with operating an autonomous vehicle in an environment; determining a set of scenarios based at least in part on the log data, where scenarios of the set of scenarios include scenario parameters associated with aspects of the environment; determining error models associated with subsystems of the autonomous vehicle; determining parameterized scenarios based at least in part on the scenario parameters and the error models; perturbing the parameterized scenarios by adding errors to at least one of the scenario parameters or simulated vehicle components instantiated in the perturbed parameterized scenarios, where the simulated vehicle is controlled by a vehicle controller; instantiating the simulated vehicle in the perturbed parameterized scenarios; receiving simulation data indicative of how the simulated vehicle responds to the perturbed parameterized scenarios; and determining a safety metric associated with the vehicle controller based at least in part on the simulation data.

[0118] B: The system of paragraph A, wherein determining the set of scenarios includes clustering the log data to determine a first set of clusters, each cluster of the first set of clusters being associated with an individual scenario; determining a probability associated with each cluster based at least in part on the first set of clusters; and determining a second set of clusters based at least in part on a probability threshold and the first set of clusters.

[0119] C: The system of paragraph A, wherein determining the error model includes receiving ground truth data associated with the environment, determining an error based at least in part on comparing the ground truth data to the log data, and determining an error distribution based at least in part on the error, wherein the error model includes the error distribution.

[0120] D: The system of paragraph A, wherein the parameterization scenario is a first parameterization scenario, the perturbed parameterization scenario is a first perturbed parameterization scenario, and the simulation data is first simulation data, and the operations further include determining a second parameterization scenario including at least one of a first subset of the scenario parameters or a second subset of the error models based on the first simulation data, perturbing the second parameterization scenario as a second perturbed parameterization scenario, instantiating the simulated vehicle in the second perturbed parameterization scenario, receiving second simulation data, and updating the safety metric based at least in part on the second simulation data.

[0121] E: A method comprising: determining a scenario including scenario parameters describing a portion of an environment; receiving an error model associated with a subsystem of a vehicle; determining a parameterized scenario based at least in part on the scenario, the scenario parameters, and the error model; perturbing the parameterized scenario as a perturbed parameterized scenario; receiving simulation data indicative of how the subsystem of the vehicle responds to the perturbed parameterized scenario; and determining a safety metric associated with the subsystem of the vehicle based at least in part on the simulation data.

[0122] F: The method of paragraph E, wherein the scenario parameters are associated with at least one of object size, object velocity, object pose, object density, vehicle speed, and vehicle trajectory.

[0123] G: The method of paragraph E, wherein determining the scenario includes receiving log data associated with an autonomous vehicle; clustering the log data to determine a first set of clusters, each cluster of the first set of clusters being associated with the scenario; determining a probability associated with each cluster based at least in part on the first set of clusters; and determining that the probability meets or exceeds a probability threshold.

[0124] H: The method of paragraph E, wherein the error model is determined based at least in part on receiving ground truth data associated with the environment, determining an error based at least in part on comparing the ground truth data to log data associated with a vehicle, and determining an error distribution based at least in part on the error, and the error model includes the error distribution.

[0125] I: The method of paragraph E, wherein the parameterized scenario is a first parameterized scenario, the perturbed parameterized scenario is a first perturbed parameterized scenario, and the simulation data is first simulation data, the method further including: determining a second parameterized scenario including at least one of a first subset of the scenario parameters or a second subset of the error model based on the first simulation data; perturbing the second parameterized scenario; receiving second simulation data; and updating the safety metric based at least in part on the second simulation data.

[0126] J: The method of paragraph I, further comprising: invalidating at least a first portion of one of the scenario parameters or the error model; and associating the second simulation data with at least a second portion of one of the scenario parameters or the error model that has not been invalidated.

[0127] K: The method of paragraph E, wherein the safety metric indicates a probability that the safety metric meets or exceeds a cost threshold.

[0128] L: The method of paragraph E, wherein the portion is a first portion, the method further including receiving map data, a second portion of the map data being associated with the first portion of the environment, and determining that the second portion of the map data is associated with a scenario associated with a probability of meeting or exceeding a threshold probability associated with the scenario parameter.

[0129] M: A non-transitory computer-readable medium storing instructions executable by a processor that, when executed, cause a processor to perform operations including: determining a scenario including scenario parameters describing a portion of an environment; receiving or determining one or more of error models associated with a subsystem of a vehicle; determining a parameterized scenario based at least in part on the scenario, the scenario parameters, and the error models; perturbing the parameterized scenario as a perturbed parameterized scenario; receiving simulation data indicative of how the subsystem of the vehicle responds to the perturbed parameterized scenario; and determining a safety metric associated with the subsystem of the vehicle based at least in part on the simulation data.

[0130] N: The non-transitory computer-readable medium of paragraph M, wherein the scenario parameters are associated with at least one of object size, object velocity, object pose, object density, vehicle speed, and vehicle trajectory.

[0131] O: The non-transitory computer-readable medium of paragraph M, wherein determining the scenario includes receiving log data associated with an autonomous vehicle; clustering the log data to determine a first set of clusters, each cluster of the first set of clusters being associated with the scenario; determining a probability associated with each cluster based at least in part on the first set of clusters; and determining that the probability meets or exceeds a probability threshold.

[0132] P: The non-transitory computer-readable medium of paragraph M, wherein the error model is determined at least in part based on receiving ground truth data associated with the environment, determining an error based at least in part on comparing the ground truth data to log data associated with the vehicle, and determining an error distribution based at least in part on the error, and the error model includes the error distribution.

[0133] Q: The non-transitory computer-readable medium of paragraph M, wherein the parameterized scenario is a first parameterized scenario, the perturbed parameterized scenario is a first perturbed parameterized scenario, and the simulation data is first simulation data, and the operations further include determining a second parameterized scenario including at least one of a first subset of the scenario parameters or a second subset of the error model based on the first simulation data, perturbing the second parameterized scenario, receiving second simulation data, and updating the safety metric based at least in part on the second simulation data.

[0134] R: The non-transitory computer-readable medium of paragraph Q, wherein the operation further includes: invalidating at least a first portion of one of the scenario parameters or the error model; and associating the second simulation data with at least a second portion of one of the scenario parameters or the error model that has not been invalidated.

[0135] S: The non-transitory computer-readable medium of paragraph M, wherein the safety metric indicates a probability of meeting or exceeding a cost threshold.

[0136] T: The non-transitory computer-readable medium of paragraph M, wherein the error model is associated with one or more of a perception system of the vehicle, a prediction system of the vehicle, or a planner system of the vehicle.

[0137] U: A system including one or more processors; and one or more computer-readable media having computer-executable instructions stored thereon that, when executed, cause the system to perform operations including: receiving vehicle data; inputting at least a first portion of the vehicle data to a subsystem of an autonomous vehicle, the subsystem being associated with at least one of a perception system, a planning system, a tracking system, or a prediction system; determining an environmental parameter based at least in part on a second portion of the vehicle data; receiving an estimate from the subsystem; receiving ground truth data associated with the subsystem; determining a difference between the estimate and the ground truth data, the difference representing an error associated with the subsystem; and determining a statistical model associated with the subsystem indicative of a probability of the error based at least in part on the difference, the probability being associated with the environmental parameter.

[0138] V: The system of paragraph U, wherein the vehicle data includes sensor data from sensors on the autonomous vehicle, the environmental parameters include one or more of a speed of the autonomous vehicle or weather conditions, the subsystem is a perception subsystem, the estimates are one or more of an estimated position, an estimated orientation, or an estimated range of an object represented in the vehicle data, and the ground truth data represents an actual position, actual orientation, or actual range of the object.

[0139] W: The system described in paragraph U, wherein determining the statistical model includes determining a first frequency associated with the environmental parameter and a second frequency associated with the difference based at least in part on the vehicle data, and determining the probability based at least in part on the first frequency and the second frequency.

[0140] X: The system described in paragraph U, wherein the operations further include determining a simulated environmental parameter based at least in part on simulated vehicle data, determining that the simulated environmental parameter corresponds to the environmental parameter, determining a simulated estimate based at least in part on the simulated vehicle data and the subsystem, and perturbing the simulated estimate by modifying a portion of a corresponding simulated scenario based at least in part on the error based at least in part on the probability.

[0141] Y: A method comprising: receiving data associated with a vehicle; determining environmental parameters based at least in part on a first portion of the data; determining output data associated with a system of the vehicle based at least in part on a second portion of the data; receiving ground truth data associated with the system and the data; determining a difference between the output data and the ground truth data, the difference representing an error associated with the system; and determining a statistical model associated with the system indicative of a probability of the error based at least in part on the difference, the probability being associated with the environmental parameters.

[0142] Z: The method of paragraph Y, wherein determining the statistical model includes determining a frequency associated with the error based at least in part on the data.

[0143] AA: The method of paragraph Y, wherein the environmental parameters include one or more of the speed of the vehicle, weather conditions, the geographic location of the vehicle, or time of day.

[0144] AB: The method of paragraph Y, further comprising: generating a simulation; determining that simulated environmental parameters of the simulation correspond to the environmental parameters; inputting simulated data to the system; receiving simulated outputs from the system; and perturbing the simulation based at least in part on the probabilities and the errors.

[0145] AC: The method of paragraph Y, wherein the system is a perception system, the output data includes a first bounding box associated with an object, and the ground truth data includes a second bounding box associated with the object, and determining the difference includes determining the difference between at least one of a first extent of the first bounding box and a second extent of the second bounding box, or a first pose of the first bounding box and a second pose of the second bounding box.

[0146] AD: The method described in paragraph Y, wherein the system is a tracker system, the output data includes planned trajectory data for the vehicle, and the ground truth data includes a measured trajectory of the vehicle, and determining the difference includes determining the difference between the planned trajectory data and the measured trajectory.

[0147] AE: The method of paragraph Y, wherein the system is associated with a prediction system, the data includes a predicted trajectory of an object in an environment, the ground truth data includes an observed trajectory of the object, and determining the difference includes determining the difference between the predicted trajectory and the observed trajectory.

[0148] AF: The method of paragraph Y, wherein the data is first data, the environmental parameter is a first environmental parameter, the difference is a first difference, the error is a first error, and the probability is a first probability, the method further including: receiving second data associated with the system of the vehicle; determining a second environmental parameter based at least in part on the second data; determining a second difference between a third portion of the output data and a fourth portion of the ground truth data, the second difference representing a second error associated with the system; and updating the statistical model associated with the system, the statistical model indicating a second probability of the second error, the second probability being associated with the second environmental parameter.

[0149] AG: A non-transitory computer readable medium storing instructions executable by a processor that, when executed, cause a processor to perform operations including receiving data; determining environmental parameters based at least in part on the data; determining output data based at least in part on the data and a system of a vehicle; receiving ground truth data associated with the system and the data; determining a difference between a first portion of the output data and a second portion of the ground truth data, the difference representing an error associated with the system; determining a statistical model associated with the system indicative of a probability of the error based at least in part on the difference; and associating the probability with the environmental parameters.

[0150] AH: The non-transitory computer-readable medium of paragraph AG, wherein determining the statistical model includes determining a frequency associated with the difference based at least in part on the data.

[0151] AI: The non-transitory computer-readable medium of paragraph AG, wherein the environmental parameters include one or more of the vehicle's speed, weather conditions, or time of day.

[0152] AJ: The non-transitory computer readable medium of paragraph AG, further comprising: generating a simulation in which the operation includes a simulated vehicle; receiving simulated data; determining that simulated environmental parameters correspond to the environmental parameters; inputting at least a portion of the simulated data into the system; receiving simulated output data from the system; and modifying the simulated output data based at least in part on requirements and probabilities and errors.

[0153] AK: The non-transitory computer-readable medium of paragraph AG, wherein the system is a perception system, the data includes a first bounding box associated with an object, and the ground truth data includes a second bounding box associated with the object, and determining the difference includes determining the difference between at least one of a first extent of the first bounding box and a second extent of the second bounding box, or a first pose of the first bounding box and a second pose of the second bounding box.

[0154] AL: The non-transitory computer-readable medium of paragraph AG, wherein the system is a tracker system, the data includes planned trajectory data for the vehicle, and the ground truth data includes a measured trajectory of the vehicle, and determining the difference includes determining the difference between the planned trajectory data and the measured trajectory.

[0155] AM: The non-transitory computer-readable medium of paragraph AG, wherein the system is associated with a prediction system, the data includes a predicted trajectory of an object in an environment, the ground truth data includes an observed trajectory of the object, and determining the difference includes determining the difference between the predicted trajectory and the observed trajectory.

[0156] AN: The non-transitory computer-readable medium of paragraph AG, further comprising: the data is first data, the environmental parameter is a first environmental parameter, the difference is a first difference, the error is a first error, and the probability is a first probability, and the operations further include receiving second data associated with the system of the vehicle; determining a second environmental parameter based at least in part on the second data; determining a second difference between a third portion of the output data and a fourth portion of the ground truth data, the second difference representing a second error associated with the system; and updating the statistical model associated with the system, the statistical model indicating a second probability of the second error, the second probability being associated with the second environmental parameter.

[0157] Although the above example subject matter has been described with respect to one particular implementation, it should be understood in the context of this document that the example subject matter may also be implemented via a method, device, system, computer-readable medium, and / or another implementation. Furthermore, any of the examples A-AN may be implemented alone or in combination with any other one or more of the examples A-AN.

[0158] [Conclusion] Although one or more examples of the techniques described herein have been described, various modifications, Additions, permutations, and their equivalents are included within the scope of the technology described herein.

[0159] In the illustrative description, reference is made to the accompanying drawings, which form a part hereof, showing by way of example specific examples of the claimed subject matter. It is to be understood that other examples can be used and modifications or substitutions, such as structural changes, can be made. Such examples, modifications or substitutions do not necessarily depart from the intended scope of the claimed subject matter. Although the steps herein may be presented in a particular order, in some cases the order can be changed, such as providing certain inputs at different times or in a different order, without changing the functionality of the described systems and methods. The disclosed procedures can also be performed in different orders. Furthermore, the various calculations herein need not be performed in the order disclosed, and other examples using alternative orders of calculations can be easily implemented. In addition to reordering, calculations can also be decomposed into sub-calculations that produce the same results.

Claims

1. determining a scenario including scenario parameters describing a portion of an environment, said scenario being a situation of a simulated environment created by simulating a real environment on a computing device, or a process of events including classifications, characteristics or attributes of objects placed in said simulated environment, said scenario parameters being ranges or sets of values ​​associated with said classifications, characteristics or attributes of said objects; receiving an error model associated with a subsystem of an autonomous vehicle; determining a parameterized scenario based at least in part on the scenario, the scenario parameters, and the error model, respectively; perturbing said parameterized scenario as a perturbed parameterized scenario; receiving simulation data indicative of how the subsystems of the autonomous vehicle respond to the perturbation parameterization scenarios; determining a safety metric associated with the subsystem of the autonomous vehicle based at least in part on the simulation data; A method comprising:

2. 2. The method of claim 1, wherein the scenario parameters are associated with at least one of: a size of the object, a velocity of the object, a pose of the object, a density of the object, a speed of the autonomous vehicle, and a trajectory of the autonomous vehicle.

3. The step of determining a scenario includes: receiving log data associated with the autonomous vehicle; clustering the log data to determine a first set of clusters, each cluster in the first set of clusters being associated with a scenario; determining probabilities associated with each of the clusters based at least in part on the first set of clusters; determining whether the probability meets or exceeds a probability threshold to identify the scenario; The method of claim 1 or 2, comprising:

4. The error model is receiving ground truth data associated with the environment; determining an error based at least in part on comparing the ground truth data to the log data associated with the autonomous vehicle; determining an error distribution based at least in part on the error; Based at least in part on the error model includes the error distribution, The method according to claim 3.

5. the parameterization scenario is a first parameterization scenario, the perturbation parameterization scenario is a first perturbation parameterization scenario, and the simulation data is first simulation data; determining a second parameterized scenario including at least one of a first subset of the scenario parameters or a second subset of the error models based on the first simulation data, the first subset being a particular group selected from a set of ranges or values ​​associated with the scenario parameters and the second subset being a particular group selected from a plurality of error models associated with the subsystems of the autonomous vehicle; perturbing the second parameterized scenario; receiving second simulation data; updating the safety metric based at least in part on the second simulation data; The method of claim 1 , further comprising:

6. invalidating a first portion of at least one of the scenario parameters or the error model; associating the second simulation data with a second portion of at least one of the scenario parameters or the error model that are not invalidated; The method of claim 5 further comprising:

7. The method of claim 1 , wherein the safety metric indicates a probability of meeting or exceeding a cost threshold.

8. the portion of the environment is a first portion, receiving map data, a second portion of the map data being associated with the first portion of the environment; determining that the second portion of the map data is associated with a scenario associated with a probability of meeting or exceeding a threshold probability associated with the scenario parameter; The method of claim 1 , further comprising:

9. A computer program comprising coded instructions which, when executed on a computer, implements the method of any one of claims 1 to 8.

10. one or more processors; one or more non-transitory computer-readable media having instructions stored thereon; A system comprising: The instructions, when executed, cause the one or more processors to: determining a scenario including scenario parameters describing a portion of an environment, said scenario being a state of a simulated environment created by simulating a real environment on a computing device, or a process of events including classifications, characteristics, or attributes of objects placed in said simulated environment, said scenario parameters being ranges or sets of values ​​associated with said classifications, characteristics, or attributes of said objects; Receiving or determining one or more of the error models associated with the subsystems of the autonomous vehicle; determining a parameterized scenario based at least in part on the scenario, the scenario parameters, and the error model, respectively; perturbing said parameterized scenario as a perturbed parameterized scenario; receiving simulation data indicative of how the subsystems of the autonomous vehicle respond to the perturbation parameterization scenarios; determining a safety metric associated with the subsystem of the autonomous vehicle based at least in part on the simulation data; and perform an operation that includes system.

11. the scenario parameters are associated with at least one of a size of the object, a velocity of the object, a pose of the object, a density of the object, a speed of the autonomous vehicle, and a trajectory of the autonomous vehicle; the safety metric indicates a probability of meeting or exceeding a cost threshold; or the error model is associated with one or more of a perception system, a prediction system, or a planning system, each of which belongs to the autonomous vehicle; The system of claim 10, wherein the at least one of

12. Determining the scenario includes: receiving log data associated with the autonomous vehicle; clustering the log data to determine a first set of clusters, each cluster in the first set of clusters being associated with the scenario; and determining a probability associated with each of the clusters based at least in part on the first set of clusters; determining whether the probability meets or exceeds a probability threshold to identify the scenario; The system according to claim 10 or 11, comprising:

13. The error model is receiving ground truth data associated with the environment; determining an error based at least in part on comparing the ground truth data to the log data associated with the autonomous vehicle; and determining an error distribution based at least in part on the errors; and Based at least in part on the error model includes the error distribution, The system of claim 12.

14. the parameterization scenario is a first parameterization scenario, the perturbation parameterization scenario is a first perturbation parameterization scenario, the simulation data is first simulation data, and the operation is determining a second parameterized scenario including at least one of a first subset of the scenario parameters or a second subset of the error models based on the first simulation data, the first subset being a particular group selected from a set of ranges or values ​​associated with the scenario parameters and the second subset being a particular group selected from a plurality of error models associated with the subsystems of the autonomous vehicle; perturbing the second parameterized scenario; receiving second simulation data; updating the safety metric based at least in part on the second simulation data; and The system of claim 10 , further comprising:

15. The operation is invalidating a first portion of at least one of the scenario parameters or the error model; Associating the second simulation data with a second portion of at least one of the scenario parameters or the error model that are not invalidated; The system of claim 14 further comprising: