Perceptual Error Model

Abstract

Techniques are described for determining an error model based on vehicle data and ground truth data. To determine whether a complex system (which may not be testable) can safely operate, various operational regimes (scenarios) can be identified based on operational data. To provide for safe operation of such a system, an error model can be determined that can provide probabilities associated with perceptual data, and the vehicle can determine a trajectory based on the probabilities of errors associated with the perceptual data.

Description

[Technical field]

[0001] This patent application claims priority to U.S. Utility Patent Application No. 16 / 708,019, filed December 9, 2019. Application No. 16 / 708,019 is incorporated by reference herein in its entirety. [Background technology]

[0002] An autonomous vehicle may use an autonomous vehicle controller to guide the autonomous vehicle through an environment. For example, the autonomous vehicle controller may use planning methods, apparatus, and systems to determine a driving path and guide the autonomous vehicle through an environment that includes dynamic objects (e.g., vehicles, pedestrians, animals, etc.) and static objects (e.g., buildings, signs, stationary vehicles, etc.). However, to ensure the safety of the occupants, it is important to verify the safety of the controller. [Brief description of the drawings]

[0003] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the drawing in which that reference number first appears. Use of the same reference number in different drawings indicates similar or identical components or features. [Figure 1] FIG. 1 illustrates generating vehicle performance data associated with a vehicle controller based on parameterized scenarios. [Diagram 2] FIG. 2 illustrates a computing device that generates scenario data based at least in part on log data generated by a vehicle, the scenario data depicting one or more variations of a scenario. [Diagram 3] FIG. 3 illustrates generating error model data based at least in part on vehicle data and ground truth data. [Figure 4]FIG. 4 illustrates the use of error model data to perturb a simulation by providing at least one of the errors or uncertainties associated with the simulated environment. [Diagram 5] FIG. 5 illustrates a computing device that generates perceptual error model data based at least in part on logged data generated by a vehicle and ground truth data. [Figure 6] FIG. 6 illustrates a computing device that generates simulation data based at least in part on parameterized scenario data and generates safety metric data based at least in part on the simulation data. [Figure 7] FIG. 7 illustrates a block diagram of an example system for implementing the techniques described herein. [Figure 8] FIG. 8 illustrates a flow diagram of an example process for determining a safety metric associated with a vehicle controller, according to an example of the disclosure. [Figure 9] FIG. 9 illustrates a flow diagram of an example process for determining a statistical model associated with a subsystem of an autonomous vehicle. [Figure 10] FIG. 10 shows several regions of the environment associated with error probabilities. [Figure 11] FIG. 11 shows vehicle data and the environment represented by the vehicle data, as well as the differences between the vehicle data and the environment. [Figure 12] FIG. 12 illustrates a flow diagram of an exemplary process for determining an error model and determining perturbed simulation data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] The techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulation in conjunction with other performance metric determinations. Simulation may be used to validate software (e.g., vehicle controller) running on a vehicle (e.g., autonomous vehicle) and gather safety metrics to ensure that the software can safely control such a vehicle in various scenarios. In additional or alternative examples, simulation may be used to learn about the constraints of an autonomous vehicle using an autonomous controller. For example, simulation may be used to understand the operating space of the autonomous vehicle (e.g., the envelope within which the autonomous controller effectively controls the autonomous vehicle) taking into account surface conditions, ambient noise, failed components, etc. Simulation may also be useful to generate feedback to improve the operation and design of the autonomous vehicle. For example, in some examples, simulation may be useful to determine the amount of redundancy needed in the autonomous controller or how to modify the behavior of the autonomous controller based on what is learned through the simulation. Furthermore, in additional or alternative examples, simulation may be useful to inform the hardware design of the autonomous vehicle, such as optimizing the placement of sensors on the autonomous vehicle.

[0005] When generating a simulation environment for testing or validation, the environment can be specifically enumerated with various instantiations. Each instantiation of such an environment is unique and definable. Manual enumeration of all possible scenarios can require excessive time, and if all possible scenarios are not constructed, various scenarios may not be tested. Scenario parameters can be used to parameterize characteristics and / or attributes of objects in a scenario to provide scenario variations.

[0006] For example, one or more vehicles can traverse an environment and generate log data associated with the environment. The log data can include sensor data captured by one or more sensors of the vehicle, sensory data indicative of objects identified by one or more systems onboard the vehicle (or generated during a post-processing stage), predictive data (whether generated during recording or subsequently) indicative of the object's intent, and / or status data indicative of diagnostic, trajectory, and other information generated by the vehicle. The vehicles can transmit the log data over a network to a database that stores the log data and / or a computing device that analyzes the log data.

[0007] The computing device can determine various scenarios, frequencies of the various scenarios, and regions of the environment associated with the various scenarios based on the log data. In some examples, the computing device can group similar scenarios represented in the log data. For example, the scenarios can be grouped together using, for example, K-means clustering and / or evaluating weighted distances (e.g., Euclidean) between parameters of the environment (e.g., daytime, nighttime, precipitation, vehicle position / velocity, object position / velocity, road segments, etc.). As described above, clustering similar or homogenous scenarios can reduce the amount of computational resources required to simulate an autonomous controller in an environment by simulating the autonomous controller in its own scenario, rather than simulating an autonomous vehicle in nearly similar scenarios that would result in redundant simulation data / results. As can be appreciated, an autonomous controller can be expected (and / or apparently may) perform similarly in similar scenarios.

[0008] For example, the computing device may determine a rate at which pedestrians appear at a crosswalk based on a number of pedestrians represented in the log data. In some examples, the computing device may determine a probability of detecting a pedestrian at a crosswalk based on the rate and a duration of operating the autonomous vehicle. Based on the log data, the computing device may determine a scenario and identify scenario parameters based on the scenario that may be used in the simulation.

[0009] In some cases, simulations can be used to test and validate the response of an autonomous vehicle controller to faulty (and / or defective) sensors of the vehicle and / or faulty (and / or defective) processing of sensor data. In such examples, the computing device can be configured to introduce inconsistencies in scenario parameters of the object. For example, an error model can indicate an error and / or an error percentage associated with the scenario parameters. The scenario can incorporate the error and / or error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors can be represented by lookup tables determined based on statistical aggregation using, but not limited to, ground truth data, functions (e.g., errors based on input parameters), or any other model that maps parameters to specific errors. In at least some examples, such error models can map specific errors with a probability / frequency of occurrence.

[0010] By way of example, and not limitation, the error model may indicate that a scenario parameter, such as a speed associated with an object in a simulated environment, is associated with an error percentage. For example, an object may move in a simulated scenario at a speed of 10 meters per second, and the error percentage may be 20%, resulting in a speed range between 8 meters per second and 12 meters per second. In some examples, the range of speeds may be associated with a probability distribution indicating that some parts of the range have a higher probability of occurring than other parts of the range (e.g., 8 meters per second and 12 meters per second associated with a 15% probability, 9 meters per second and 11 meters per second associated with a 30% probability, and 10 meters per second associated with a 10% probability).

[0011] Based on the error model and / or the scenarios, parameterized scenarios can be generated. The parameterized scenarios can provide a set of scenario variations. Thus, instantiating the autonomous vehicle controller with the parameterized scenarios and simulating the parameterized scenarios can efficiently cover a wide range of scenario variations without requiring manual enumeration of the variations. Additionally, based at least in part on executing the parameterized scenarios, simulation data can indicate how the autonomous vehicle controller responded (or would respond) to the parameterized scenarios, and a successful or unsuccessful outcome can be determined based at least in part on the simulation data.

[0012] Aggregating the simulation data related to the parameterization scenarios can provide safety metrics associated with the parameterization scenarios. For example, the simulation data can indicate a success rate and / or a failure rate of the autonomous vehicle controller and the parameterization scenario. In some examples, meeting or exceeding a success rate can indicate successful validation of the autonomous vehicle controller, which can then be downloaded by (or otherwise transferred to) the vehicle for further vehicle control and operation.

[0013] For example, the parameterized scenario can be associated with an outcome. The simulation data can indicate that the autonomous vehicle controller responded consistently or inconsistently to the outcome. By way of example, and not by way of limitation, the parameterized scenario can represent a simulated environment including a vehicle controlled by the autonomous vehicle controller traveling at a speed and performing a stopping action in front of an object ahead of the vehicle. The speed can be associated with a scenario parameter indicating a range of speeds for the vehicle. The parameterized scenario can be simulated based at least in part on the range of speeds and can generate simulation data indicating a distance between the vehicle and the object when the vehicle completes the stopping action. The parameterized scenario can be associated with an outcome indicating that the distance between the vehicle and the object meets or exceeds a distance threshold. Based on the simulation data and the scenario parameters, a success rate can indicate the number of times that the distance between the vehicle and the object meets or exceeds a distance threshold when the vehicle completes the stopping action compared to the total number of times the vehicle completes the stopping action.

[0014] The techniques described herein provide various computational efficiencies. For example, by using the techniques described herein, a computing device requires fewer computational resources and multiple simulated scenarios can be generated faster than those available via conventional techniques. Conventional techniques are not scalable. For example, generating a set of unique simulated environments, as many as are required for training, testing, and / or validating a system (e.g., one or more components of an AI stack) on an autonomous vehicle (e.g., before such an autonomous vehicle is deployed to a corresponding new real environment), can take a very long time, thereby limiting the ability to train, test, and / or validate such a system (e.g., one or more components of an AI stack) on an autonomous vehicle before entering a real scenario and / or environment. The techniques described herein differ from conventional techniques in that they leverage sensor data collected from a real environment and supplement that data with additional data to efficiently generate a simulated environment that is substantially more accurate than that available through conventional techniques (e.g., for a corresponding real environment). Moreover, various aspects of the scenario techniques described herein enable the generation of many scalable simulated scenarios in less time and with fewer computational resources than are available with conventional techniques.

[0015] Moreover, the techniques described herein are directed to improved safety. That is, the simulated environments resulting from the generative techniques described herein can be used in test, training, and certification systems onboard autonomous vehicles, which can ensure that the autonomous vehicle can operate safely when deployed in a real environment. That is, the simulated environments resulting from the generative techniques described herein can be used to test, train, and certify the planner and / or predictive systems of an autonomous vehicle controller, which can be used by the autonomous vehicle to navigate the autonomous vehicle along a trajectory in a real environment. Thus, such training, testing, and certification enabled by the techniques described herein can provide an opportunity to ensure that an autonomous vehicle can operate safely in a real-world environment. And the techniques described herein improve safety and impact navigation.

[0016] FIG. 1 illustrates an example 100 of generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. Input data 102 can be used to generate the scenario. The input data 102 can include vehicle data 104 and / or additional situation data 106. The vehicle data 104 can include log data captured by the vehicle traveling through an environment. As described above, the log data can be used to identify scenarios for simulating the autonomous controller. For illustrative purposes, the vehicle can be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout its journey, and where a driver (or passenger) is not expected to control the vehicle at any time. In such an example, the vehicle can be configured to control all functions from start to stop, including all parking functions, and thus can be unmanned. This is by way of example only, and the systems and methods described herein can be incorporated into any ground, air, or water vehicle, including vehicles ranging from those that must be manually controlled at all times by a driver to those that are partially or fully autonomously controlled.

[0017] The vehicle may include a computing device that includes a perception engine and / or planner and may perform operations such as detecting, identifying, segmenting, classifying, and / or tracking objects from sensor data collected from the environment. For example, objects such as pedestrians, bicycles / bicyclists, motorcycles / motorcyclers, buses, trams, trucks, animals, etc. may be present in the environment.

[0018] As the vehicle traverses through the environment, the sensors can capture sensor data associated with the environment. For example, some of the sensor data can be associated with objects (e.g., vehicles, bicyclists, and / or pedestrians). In some examples, the sensor data can be associated with other objects, including, but not limited to, buildings, road surfaces, signs, barriers, etc. Thus, in some examples, the sensor data can be associated with dynamic objects and / or static objects. Dynamic objects, as described above, can be objects that are associated with or allow for movement within the environment (e.g., vehicles, motorcycles, bicyclists, pedestrians, animals, etc.). Static objects, as described above, can be objects associated with the environment, such as, for example, buildings / structures, road surfaces, road markers, signs, barriers, trees, walkways, etc. In some examples, the vehicle computing device can determine information about objects in the environment, such as bounding boxes, classification, segmentation information, etc.

[0019] The vehicle computing device can use the sensor data to generate a trajectory for the vehicle. In some examples, the vehicle computing device can also determine pose data associated with the position of the vehicle. For example, the vehicle computing device can use the sensor data to determine position data, coordinate data, and / or orientation data for the vehicle within the environment. In some examples, the pose data can include xyz coordinates and / or pitch, roll, and yaw data associated with the vehicle.

[0020] The vehicle computing device can generate vehicle data 104, which can include the data described above. For example, the vehicle data 104 can include sensor data, perception data, planning data, vehicle state data, speed data, intent data, and / or other data generated by the vehicle computing device. In some examples, the sensor data can include data captured by sensors such as time-of-flight sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, intensity, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, etc. The sensor data can be data captured by such sensors, such as time-of-flight data, position data, lidar data, radar data, sonar data, image data, audio data, etc. Such log data may further include intermediate outputs by any one or more systems or subsystems of the vehicle, including, but not limited to, messages indicative of object detection, object tracks, predictions of future object positions, multiple trajectories generated in response to such detection, control signals passed to one or more systems or subsystems used to execute commands, etc. In some examples, vehicle data 104 may include time data associated with other data generated by the vehicle computing device.

[0021] In some examples, input data 102 may be used to generate scenarios. Input data 102 may include vehicle data 104 and / or additional situational data 106. By way of example, and without limitation, additional situational data 106 may include data such as incident reports from third party sources. Third party sources may include law enforcement agencies, motor vehicle departments, and / or safety management that may publish and / or store reports of activities and / or incidents. For example, the reports may include a type of activity (e.g., traffic hazards such as debris on the road, localized flooding, etc.), a location, and / or a description of the activity. By way of example, and without limitation, the reports may describe a driver operating a vehicle that was struck by a fallen tree branch in the roadway while traveling at a speed of 15 meters per second. The reports may be used to generate similar scenarios that may be used in simulations.

[0022] In some examples, the additional situation data 106 may include captured sensor data (e.g., image data). By way of example, and not by way of limitation, a driver of a vehicle may use a camera to capture image data while the driver is operating the vehicle. In some examples, the image data may capture an activity, such as an incident. By way of example, and not by way of limitation, a driver may use a dashboard camera (e.g., a camera mounted on the interior dashboard of the vehicle) to capture image data while the driver is operating the vehicle. As the driver operates the vehicle, an animal may run across the roadway, and the driver may immediately apply the brakes to slow the vehicle. The dashboard camera may capture image data of an animal running across the roadway and the vehicle slowing down. The image data may be used to generate a scenario of an animal running across the roadway. As described above, probabilities associated with the scenarios may be determined to identify scenarios for the simulation based on probabilities that meet or exceed a probability threshold. As an example, and not by way of limitation, if the likelihood of encountering a scenario is less than 0.001%, a probability threshold of 0.001% may be used, and then scenarios with higher probabilities may be prioritized for simulation and safety metrics associated with the scenarios with higher probabilities may be determined.

[0023] The input data 102, e.g., vehicle data 104 and / or additional situation data 106, can be used by a scenario editor component 108 to generate an initial scenario 110. For example, the input data 102 can be input to the scenario editor component 108, which can generate a synthetic environment that represents at least a portion of the input data 102 in the synthetic environment. Examples of generating scenarios, such as the initial scenario 110 and vehicle-generated data that can be included in the vehicle data 104, can be found, for example, in U.S. patent application Ser. No. 16 / 392,094, filed Apr. 23, 2019, entitled “Scenario Editor and Simulator,” which is incorporated by reference in its entirety.

[0024] The scenario editor component 108 may be configured to scan the input data 102 to identify one or more scenarios represented in the input data 102. By way of example, and not by way of limitation, the scenario editor component 108 may determine that a portion of the input data 102 represents a pedestrian crossing a street without a right-of-way (e.g., a street without a crosswalk, an intersection without pedestrian signs, etc.). The scenario editor component 108 may identify this as a scenario (e.g., a jaywalking parameter) and label (and / or classify) the scenario, for example, as a jaywalking scenario. For example, the scenario editor component 108 may generate an initial scenario 110 using rules that define actions. By way of example, and not by way of limitation, a rule may define that a pedestrian crossing a road in an area not associated with a crosswalk is a jaywalker. In some examples, the scenario editor component 108 may receive label data from a user of the scenario editor component 108 to associate a portion of the input data 102 with the label and generate the initial scenario 110.

[0025] In some examples, the scenario editor component 108 can scan other portions of the input data 102, identify similar scenarios, and label the similar scenarios with the same jaywalking label. In some examples, the scenario editor component 108 can identify scenarios that do not correspond to (or are excluded from) existing labels and generate new labels for these scenarios. In some examples, the scenario editor component 108 can generate a library of scenarios and store the library of scenarios in a database within the scenario editor component 108. By way of example, and not limitation, the library of scenarios can include a pedestrian crossing scenario, a merge scenario, a lane change scenario, etc.

[0026] In at least some examples, such initial scenarios 110 may be manually specified. For example, one or more users may specify particular scenarios to be tested to ensure that the vehicle can operate safely when executing such scenarios, despite never having encountered (or only rarely encountering) the scenarios before.

[0027] The parameters component 112 can determine scenario parameters associated with the initial scenario 110 identified by the scenario editor component 108. By way of example, and not limitation, the parameters component 112 can analyze a jaywalking scenario and determine scenario parameters associated with the jaywalking scenario, including pedestrian position, pedestrian pose, pedestrian size, pedestrian speed, pedestrian footsteps, distance between the vehicle and the pedestrian, vehicle speed, road width, etc.

[0028] In some examples, the parameter component 112 can determine a range or set of values ​​associated with a scenario parameter. For example, the parameter component 112 can determine a classification associated with an object (e.g., a pedestrian) represented in the initial scenario 110 and determine other objects of the same classification in the input data 102. The parameter component 112 can then determine a range of values ​​associated with the scenario parameter as represented by the initial scenario 110. By way of example, and not by way of limitation, the scenario parameter can indicate that a pedestrian can have a speed in the range of 0.5 to 1.5 meters per second.

[0029] In some examples, the parameter component 112 can determine probabilities associated with the scenario parameters. By way of example, and not limitation, the parameter component 112 can associate a probability distribution, such as a Gaussian distribution (also referred to as a normal distribution), with the scenario parameters. In some examples, the parameter component 112 can determine the probabilities associated with the scenario parameters based on the input data 102. As described above, the parameter component 112 can determine a classification associated with an object represented in the input data 102 and determine other objects of the same classification in the input data 102 and / or other log data. The parameter component 112 can then determine a probability distribution for the scenario parameters associated with the object as represented by the input data 102 and / or other log data.

[0030] As an example, but not by way of limitation, the parameters component 112 may determine that 30% of pedestrians walk at a speed less than 0.3 meters per second, 30% of pedestrians walk at a speed greater than 1.2 meters per second, and 40% of pedestrians walk at a speed between 0.3 and 1.2 meters per second. The parameters component 112 may use the distribution as a probability that a pedestrian in a jaywalking scenario will walk at a particular speed. As an additional example, but not by way of limitation, the parameters component 112 may determine that a jaywalking scenario probability is 1%, which may indicate that a vehicle traversing an environment will encounter a jaywalker 5% of the time while traversing the environment. In some examples, during a simulation of the autonomous vehicle controller, the scenario probabilities may be used to include scenarios at a rate associated with the scenario probabilities.

[0031] In some examples, the parameter component 112 can receive the complementary data 114 incorporated into a distribution. By way of example, and not by way of limitation, the parameter component 112 can determine a scenario parameter indicating that a pedestrian may have a range of distances to a vehicle of 30-60 meters while the vehicle is moving at a speed of 15 meters per second, or alternatively expressed as a time to impact of 2-4 seconds. The complementary data 114 (e.g., regulations or guidelines) can indicate that the vehicle must address the scenario with a time to impact of 1.5 seconds, which can be a lower limit (also referred to as a parameter threshold). The parameter component 112 can incorporate the complementary data 114 and determine the scenario parameter to have a time to impact of 1.5-4 seconds (although any time range can be specified). In some examples, the parameter component 112 can use the probability distributions described above to determine (using interpolation and / or extrapolation techniques) a probability associated with the complementary data 114.

[0032] The error model component 116 can determine an error model that can indicate an error associated with the scenario parameters. For example, the perceptual error model can generate a perceptual error associated with a perceptual parameter of the simulated object, the predictive error model can generate a predictive error associated with a predictive parameter of the simulated object, etc. In some examples, as the vehicle traverses the environment, the sensors can generate erroneous sensor data and / or the vehicle's computing device can misprocess the sensor data that can result in a perceptual error. Testing and simulating the perceptual error can help indicate the vehicle's operating margins associated with potential perceptual errors. For example, the scenario parameters, such as the perceptual parameters, can indicate a range of object sizes or positions of objects in the environment. The error model component 116 can use the perceptual error model to indicate potential errors associated with object sizes that can result in the vehicle's perceptual data indicating that the object is larger or smaller than the actual object in the environment.

[0033] The error model component 116 can determine the perceptual error model, for example, by comparing the input data 102 to ground truth data. In some examples, the ground truth data can be determined from manually labeled and / or other validated machine learning components. For example, the input data 102 can include sensor data and / or sensory data generated by a vehicle. The error model component 116 can compare the input data 102 to the ground truth data, which can indicate actual parameters of objects in the environment. By comparing the input data 102 to the ground truth data, the error model component 116 can determine a perceptual error. As an example, but not by way of limitation, the input data 102 can indicate that a pedestrian is 1.8 meters tall, while the ground truth data indicates that the pedestrian is 1.75 meters tall, and thus the perceptual error model can indicate a perceptual error of approximately 3% (e.g., [(1.8-1.75) / 1.8]*100).

[0034] In some examples, the error model component 116 can determine a classification associated with an object represented in the input data 102 and determine other objects of the same classification in the input data 102. The error model component 116 can then determine a probability distribution associated with a range of error for the object (also referred to as an error distribution) and associate the probability distribution with the object in the initial scenario 110. By way of example and not by way of limitation, the error model component 116 can determine that an object with a pedestrian classification has a perceived error of 4% to 6% and an object with a vehicle classification has a perceived error of 3% to 5%. In some examples, the error model component 116 can determine a probability distribution indicating, for example, that objects larger than a threshold size are more likely or less likely to have an error (e.g., in the classification).

[0035] In some examples, the parameters component 112 can use the region data 118 to determine a set of regions of an environment that are compatible with the scenario and the scenario parameters. By way of example, and not by way of limitation, a scenario can indicate that a vehicle has a speed of 15 meters per second, a pedestrian has a speed of 1.5 meters per second, and a distance between the vehicle and the pedestrian is 30 meters. The parameters component 112 can determine the regions based on the region data 118 of an environment that fits the scenario. By way of example, and not by way of limitation, the parameters component 112 can exclude a school zone because the speed of a vehicle in the scenario may exceed a speed limit associated with the school zone, and thus the scenario is not valid in the school zone. However, such a vehicle speed (e.g., 15 m / s) may be reasonable on a county road adjacent to farmland, and thus such a region may be considered within the set of regions.

[0036] In some examples, the parameter component 112 can store region data 118 including segments of drivable regions of the environment. Techniques for identifying drivable surface segments and similar segments, and segment classifications and / or stereotypes, can be found, for example, in U.S. patent application Ser. No. 16 / 370,696, filed Mar. 19, 2019, entitled “Extension of Autonomous Driving Functionality to New Regions,” and U.S. patent application Ser. No. 16 / 376,842, filed Apr. 5, 2019, entitled “Simulating Autonomous Driving Using Map Data and Driving Data,” both of which are incorporated herein by reference in their entireties.

[0037] For example, the area data 118 of the environment can be parsed into segments and similar segments can be identified. In some examples, the segments can include junction segments, e.g., intersections, merges, etc., or connecting road segments, e.g., the length and / or width of the road between the junctions. By way of example and not limitation, all two-lane road segments having speed limits within a 10 mph range can be associated with the same stereotype. In some examples, data can be associated with each of the individual segments. For example, a junction segment can include a junction type, e.g., a merge, "T", roundabout, etc., the number of roads that meet at the junction, the relative positions of those roads, e.g., the angle between the roads that meet at the junction, information about the traffic control signal at the junction, and / or other features. Data associated with a connecting road segment can include the number of lanes, the width of those lanes, the direction of travel in each lane, identification of parking lanes, the speed limit of the road segment, and / or other features.

[0038] In some examples, segments of a drivable surface can be grouped according to a segment classification or segment stereotype. By way of example and not limitation, some or all junction segments that fit a certain range of metrics or attributes can be grouped together (e.g., using k-means, evaluating a weighted distance (e.g., Euclidean distance) between segment parameters, or otherwise clustering such segments based on segment parameters).

[0039] Scenarios that include the same or similar stereotypes can be used to verify the functionality of an autonomous vehicle controller. For example, an autonomous vehicle may be expected to do (and / or may obviously do) the same things in the same or similar stereotypes. In some instances, the use of stereotypes can reduce the number of comparisons made. For example, by identifying similar areas, the number of simulation scenarios that can provide useful information is reduced. The techniques described herein can reduce computational complexity, memory requirements, and processing time by optimizing over specific scenarios that provide useful information for validation and testing.

[0040] The parameterized scenario component 120 can generate a parameterized scenario 122 using data determined by the parameter component 112 (e.g., the initial scenario 110, the scenario parameters, the set of regions, and / or the error model data). For example, the initial scenario 110 can indicate a scenario such as a lane change scenario, a right turn scenario, a left turn scenario, an emergency stop scenario, etc. The scenario parameters can indicate a speed associated with a vehicle controlled by the autonomous vehicle controller, a pose of the vehicle, a distance between the vehicle and an object, etc. In some examples, the scenario parameters can indicate an object, a position associated with the object, a speed associated with the object, etc. Additionally, an error model (e.g., a perception error model, a prediction error model, etc.) can indicate an error associated with the scenario parameter and provide a range of values ​​and / or probabilities associated with the scenario parameter. By way of example, and not by way of limitation, a scenario parameter such as a speed of the vehicle can be associated with a range of speeds, such as 8-12 meters per second. As discussed above, the range of speeds can be associated with a probability distribution indicating the probability of a speed within the range of speeds occurring.

[0041] The set of regions may represent portions of an environment that may be used to place objects in the simulated environment. For example, the initial scenario 110 may represent a scenario that includes a two-way, multi-lane driving surface associated with a speed limit of 35 miles per hour. Based on the initial scenario 110, the set of regions may exclude regions that do not include a two-way, multi-lane driving surface associated with a speed limit of 35 miles per hour, such as parking lots. The parameterized scenarios 122 may be used to cover variations provided by the scenario parameters, error models, region data 118, etc.

[0042] By way of example, and not by way of limitation, the scenario parameters may include a vehicle traversing an environment at a speed of 10 meters, 11 meters, or 12 meters per second (or any speed) while approaching a junction. The set of regions may include an uncontrolled junction, a junction with a four-way stop, and a junction with a traffic light. In addition, the perceptual error model may indicate a perceptual error of 1.34%, as may be provided by the perceptual metrics determined for the perception system being tested. Thus, the parameterized scenarios 122 may allow for a total of nine different scenarios by varying the scenario parameters and regions (e.g., three speeds * three regions = nine scenarios / permutations). In addition, when the simulation component 124 simulates the parameterized scenarios, the simulation component 124 may use the perceptual error model to introduce a perceptual error associated with the perception data determined by the vehicle. As can be appreciated, this is merely an example, and the parameterized scenarios 122 may include more or fewer permutations and different types of scenario parameters, regions, and / or perceptual errors.

[0043] The simulation component 124 can execute the parameterized scenarios 122 as a set of simulation instructions and generate simulation data 126. For example, the simulation component 124 can instantiate a vehicle controller in the simulated scenario. In some examples, the simulation component 124 can execute multiple simulated scenarios simultaneously and / or in parallel. In addition, the simulation component 124 can determine the outcome of the parameterized scenarios 122. For example, the simulation component 124 can execute variations of the parameterized scenarios 122 for use in a simulation for testing and validation. The simulation component 124 can generate simulation data 126 indicative of how the autonomous vehicle controller performed (e.g., responded) and compare the simulation data 126 to predetermined outcomes and / or determine whether any predetermined rules / assertions were violated / triggered.

[0044] In some examples, the variations for the simulation can be selected based on a generalized interval of the scenario parameters. By way of example and not limitation, the scenario parameters can be associated with the speed of the vehicle. In addition, the scenario parameters can be associated with a range of values ​​for the vehicle. The variations for the simulation can be selected based on a generalized interval to increase coverage of the range of values ​​(e.g., selecting the 25th percentile, 50th percentile, 75th percentile, etc.). In some examples, the variations can be selected randomly and / or the variations can be selected randomly within a standard deviation of the range of values.

[0045] In some examples, the predefined rules / assertions can be based on the parameterized scenario 122 (e.g., a traffic rule regarding a pedestrian crossing can be enabled based on a pedestrian crossing scenario, or a traffic rule regarding crossing lane markers can be disabled for a stopped vehicle scenario). In some examples, the simulation component 124 can dynamically enable and disable rules / assertions as the simulation progresses. For example, a rule / assertion related to the school zone can be enabled as the simulated object approaches a school zone, and a rule / assertion related to the school zone can be disabled as the simulated object leaves the school zone. In some examples, the rules / assertions can include, for example, comfort metrics related to how fast the object can accelerate given the simulated scenario. In at least some examples, the rules can include, for example, following the rules of the road, leaving safety buffers between objects, etc.

[0046] Based at least in part on determining that the autonomous vehicle controller performed consistent with a predetermined outcome (i.e., the autonomous vehicle controller did everything it was supposed to do) and / or determining that no rules were broken or assertions were triggered, simulation component 124 may determine that the autonomous vehicle controller was successful. Based at least in part on determining that the autonomous vehicle controller's performance was inconsistent with a predetermined outcome (i.e., the autonomous vehicle controller did something it was not supposed to do) and / or determining that a rule was broken or assertions were triggered, simulation component 124 may determine that the autonomous vehicle controller failed. Thus, based at least in part on executing parameterized scenarios 122, simulation data 126 may indicate how the autonomous vehicle controller responded to each variation of parameterized scenarios 122, as described above, and a successful or unsuccessful outcome may be determined based at least in part on simulation data 126.

[0047] The analysis component 128 may be configured to determine a degree of success or failure. By way of example, and not by way of limitation, a rule may indicate that a vehicle controlled by an autonomous vehicle controller must stop within a threshold distance of an object. The simulation data 126 may indicate that in a first variation of the parameterized scenario 122, the simulated vehicle stopped more than 5 meters from the threshold distance. In a second variation of the parameterized scenario 122, the simulation data 126 may indicate that the simulated vehicle stopped more than 10 meters from the threshold distance. The analysis component 128 may indicate that the simulated vehicle performed more successfully in the second variation compared to the simulated vehicle in the first variation. For example, the analysis component 128 may determine an ordered list (e.g., ordered according to a relative success scale) including the simulated vehicle and associated variations of the parameterized scenario 122. Such variations may also be used to determine limitations for various components of the simulated system.

[0048] The analysis component 128 can determine additional variations of the parameterized scenario 122 based on the simulation data 126. For example, the simulation data 126 output by the simulation component 124 can indicate variations of the parameterized scenario 122 associated with success or failure (which may be expressed as a continuous likelihood). The analysis component 128 can determine additional variations based on variations associated with failure. By way of example, and not by way of limitation, a variation of the parameterized scenario 122 associated with failure can represent a vehicle traveling across a driving surface at a speed of 15 meters per second and an animal crossing the driving surface at a distance of 20 meters ahead of the vehicle. The analysis component 128 can determine additional variations of the scenario to determine additional simulation data 126 for analysis. By way of example, and not by way of limitation, the analysis component 128 can determine additional variations including a vehicle traveling at 10 meters per second, 12.5 meters per second, 17.5 meters per second, 20 meters per second, and so on. Additionally, the analysis component 128 may determine additional variations including animals crossing the driving surface at distances such as 15 meters, 17.5 meters, 22.5 meters, and 25 meters. The additional variations may be input to the simulation component 124 to generate additional simulation data. Such additional variations may be determined, for example, based on perturbations to scenario parameters of scenarios executed in the simulation.

[0049] In some examples, the analysis component 128 can determine additional variations of the scenario by disabling the scenario parameters. For example, the parameterized scenario can include a first scenario parameter associated with the speed of the object and a second scenario parameter associated with the position of the object. The parameterized scenario 122 can include a first range of values ​​associated with the speed and a second range of values ​​associated with the position. In some examples, after simulating the parameterized scenario 122, the simulation data 126 can indicate that some variations of the parameterized scenario 122 resulted in successful outcomes and some variations resulted in unsuccessful outcomes. The analysis component 128 can then determine to disable the first scenario parameter (e.g., set a fixed value associated with the first scenario parameter) and vary the parameterized scenario 122 based on the second scenario parameter. By disabling one of the scenario parameters, the analysis component 128 can determine whether the scenario parameter and / or the value of the scenario parameter is associated with a successful outcome or a failed outcome. Such parameters can be disabled randomly, based on a likelihood of failure, or otherwise. By way of example, and not limitation, simulation data 126 as a result of invalidating all of the scenario parameters may indicate an issue with the planning component of an autonomous vehicle.

[0050] In some examples, the analysis component 128 can be used to perform sensitivity analysis. For example, the analysis component 128 can disable scenario parameters and determine, based on the simulation data 126 generated by the simulation component 124, how disabling the scenario parameters affects the simulation data 126 (e.g., increases the success rate, decreases the success rate, minimally affects the success rate). In some examples, the analysis component 128 can disable scenario parameters individually to determine how disabling each scenario parameter affects the simulation data 126. The analysis component 128 can collect statistical data indicative of how individual scenario parameters affect the simulation data 126 over the course of many simulations. In some examples, the analysis component 128 can be configured to disable a set of scenario parameters (e.g., disable a nighttime environment parameter and disable a wet conditions environment parameter). As described above, the analysis component 128 can collect statistical data indicative of how a set of scenario parameters affects the simulation data 126. The statistical data can be used to determine scenario parameters that increase or decrease the likelihood of resulting in a successful simulation, and can be used to identify autonomous vehicle subsystems associated with scenario parameters that increase or decrease the success rate of the simulation data 126.

[0051] In some examples, the analysis component 128 can adjust the extent to which the scenario parameters are adjusted. By way of example, and not limitation, the scenario parameters can indicate wet environmental conditions (e.g., rainy conditions). The scenario parameters can be adjusted in ranges (e.g., a quarter inch of rain, an inch of rain, etc.). The analysis component 128 can adjust the magnitude of the scenario parameters and perform a sensitivity analysis based on the magnitude of the scenario parameters to determine thresholds associated with the scenario parameters that can result in a successful or unsuccessful outcome of the simulation. In some examples, a binary search algorithm, a particle filter algorithm, and / or a Monte Carlo method can be used to determine the thresholds, although other suitable algorithms are contemplated.

[0052] Vehicle performance component 130 can determine vehicle performance data 132 based on simulation data 126 (and / or additional simulation data based on additional variations from analysis component 128) and the type of failure. In some examples, vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example, and not by way of limitation, vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some examples, vehicle performance data can indicate safety metrics. By way of example, and not by way of limitation, vehicle performance data 132 can indicate an event (e.g., failure) and a cause of the event. In at least some examples, such indications can be binary (presence or absence of failure), coarse (level of failure, e.g., “critical”, “non-critical” and “pass”), or continuous (e.g., representing the likelihood of failure), although any other indications are contemplated. For example, for event type 1 and cause type 1, data 134(1) can indicate a safety rating, as can data 134(2)-134(4). In some examples, cause type 1 and cause type 2 may indicate a fault, such as a vehicle fault or an object (e.g., a cyclist) fault. The vehicle performance data 132 may indicate a safety metric associated with the parameterization scenario 122. In some examples, the vehicle performance component 130 may use a target metric and compare the vehicle performance data 132 to the target metric to determine whether the safety metric meets or exceeds the target metric. In some examples, the target metric may be based on a standard and / or regulation associated with the autonomous vehicle.

[0053] In some examples, the vehicle performance data 132 may be input to a filter component 136 to determine filtered data 138 based on the vehicle performance data 132. For example, the filter component 136 may be used to determine filtered data 138 that identifies areas that do not meet a coverage threshold. By way of example, and not limitation, the initial scenario 110 may indicate a speed limit of 35 miles per hour and a two-way multi-lane driving surface associated with the area data 118. Based on the initial scenario 110 and the area data 118, the parameters component 112 may identify five areas of the environment that can be used to simulate the initial scenario 110. After simulating the parameterized scenario 122, the vehicle performance data 132 may indicate that the simulation data 126 is associated with three of the five areas. For example, the simulation component 124 may simulate the scenario and generate the simulation data 126 based on running the scenario in three of the five areas identified by the parameters component 112. The filter component 136 can generate filtered data 138 that indicates that the remaining two of the five regions do not meet a coverage threshold (e.g., a minimum number of simulations associated with the region) based on one or more filters.

[0054] In some examples, the filter component 136 can determine filtered data 138 indicative of an occurrence of an event based on the vehicle performance data 132. By way of example, and not limitation, the simulation data 126 can include an occurrence of an event such as an emergency stop, a flat tire, an animal crossing a driving surface, etc. The filter component 136 can determine filtered data 138 indicative of an occurrence of an emergency stop based on one or more filters. In addition, the filtered data 138 can include a portion of the simulation data 126 associated with the occurrence of an emergency stop, such as, for example, a stopping distance associated with the emergency stop.

[0055] FIG. 2 illustrates an example 200 of a vehicle 202, similar to the vehicle generating vehicle data 104 described with reference to FIG. 1, that may generate and transmit the vehicle data 104 to a computing device 204. As described above, the scenario editor component 108 may be configured to scan the input data 102 (e.g., the vehicle data 104 and / or the additional situation data 106) and identify one or more scenarios represented in the input data 102. As a non-limiting example, such scenarios may be determined, for example, based on clustering (e.g., using k-means, etc.) a parameterization of the log data. In some examples, the scenario editor component 108 may use scenario definition data 206 to identify one or more scenarios represented in the input data 102. For example, the scenario definition data 206 may identify features associated with a type of scenario. As an example, but not by way of limitation, the scenario definition data 206 may identify a jaywalking scenario that includes features such as a pedestrian crossing a portion of a driving surface that is not associated with a pedestrian crossing. The scenario editor component 108 can scan the input data 102 to identify portions of the input data that include characteristics of a pedestrian crossing a portion of the driving surface that is not associated with a crosswalk to determine a jaywalking scenario. In at least some examples, such scenarios can be further manually input and / or derived from third party data (e.g., police reports, publicly available video clips, etc.).

[0056] In addition, as described above, the parameters component 112 can determine scenario parameters, which can indicate values ​​or ranges of values ​​associated with parameters of objects in a scenario. The parameters component 112 can generate scenario data 208, as depicted in FIG.

[0057] The scenario data 208 may represent a base scenario including a vehicle 210 traversing along a driving surface and an object 212 (which may be a different vehicle) traversing near the vehicle 210 in the same direction as the vehicle 210. The vehicle 210 and the object 212 may approach a junction with a pedestrian crossing. The parameter component 112 may determine scenario parameters indicating a distance between the vehicle 210 and the object 212 as having a range of distances. Thus, scenario S1 may represent a first scenario of a set of scenarios having a first distance between the vehicle 210 and the object 212, scenario S2 may represent a second scenario of a set of scenarios having a second distance between the vehicle 210 and the object 212, and so on. N may represent the Nth scenario of the set of scenarios having the Nth distance between the vehicle 210 and the object 212. Examples of additional types of parameters (also referred to as attributes) may be found, for example, in U.S. patent application Ser. No. 16 / 363,541, filed Mar. 25, 2019, entitled “Pedestrian Prediction Based on Attributes,” which is incorporated by reference in its entirety.

[0058] For example, the scenario parameters may include a velocity of the object 212, an acceleration of the object 212, an x-position of the object 212 (e.g., a position relative to a global position, a local position, and / or any other frame of reference), a y-position of the object 212 (e.g., a size, a pose, a local position, a global position, and / or a position relative to any other frame of reference), a bounding box associated with the object 212 (e.g., extent (length, width, and / or height), yaw, pitch, roll, etc.), lighting conditions (e.g., brake lights, blinker lights, hazard lights, headlights, reverse lights, etc.), wheels of the object 212, These may include, but are not limited to, direction of the object 212, map elements (e.g., distance between the object 212 and a traffic light, a stop sign, a speed bump, an intersection, a yield sign, etc.), classification of the object 212 (e.g., vehicle, car, truck, bicycle, motorcycle, pedestrian, animal, etc.), object characteristics (e.g., whether the object is changing lanes, whether the object 212 is a double parked vehicle, etc.), proximity to one or more objects (in any coordinate frame), lane type (e.g., lane direction, parking lane, etc.), road markings (e.g., indicating whether passing or lane changing is allowed, etc.), object density, etc.

[0059] As described above, parameter component 112 can determine ranges of values ​​associated with scenario parameters, as represented by vehicle data 104 and / or other input data. Thus, each of the example scenario parameters identified above, as well as other scenario parameters, can be associated with a set of values ​​or ranges of values ​​that can be used to generate a set of scenarios, where the scenarios in the set differ by values ​​of one or more of the scenario parameters.

[0060] 3 illustrates an example 300 of generating error model data based at least in part on vehicle data and ground truth data. As illustrated in FIG. 3, a vehicle 202 can generate vehicle data 104 and transmit the vehicle data 104 to an error model component 116. As described above, the error model component 116 can determine an error model that can indicate errors associated with scenario parameters. For example, the vehicle data 104 can be data associated with a subsystem of the vehicle 202, such as a perception system, a planning system, a tracking system (also referred to as a tracker system), a predictive system, etc. By way of example, and not limitation, the vehicle data 104 can be associated with a perception system, and the vehicle data 104 can include bounding boxes associated with objects detected by the vehicle 202 in the environment.

[0061] The error model component 116 can receive ground truth data 302, which can be determined from manually labeled and / or other validated machine learning components. By way of example, and not limitation, the ground truth data 302 can include validated bounding boxes associated with objects in the environment. By comparing the bounding boxes of the vehicle data 104 to the bounding boxes of the ground truth data 302, the error model component 116 can determine errors associated with subsystems of the vehicle 202. In some examples, the vehicle data 104 can include one or more characteristics (also referred to as parameters) associated with the detected entity and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity can include, but are not limited to, x-position (global location), y-position (global location), z-position (global location), orientation, entity type (e.g., classification, etc.), velocity of the entity, range (size) of the entity, etc. The characteristics associated with the environment can include, but are not limited to, the presence of another entity in the environment, the state of another entity in the environment, time of day, day of the week, season, weather conditions, dark / light indication, etc. Therefore, the error may be related to other characteristics (eg, environmental parameters).

[0062] The error model component 116 can process the plurality of vehicle data 104 and the plurality of ground truth data 302 to determine the error model data 304. The error model data 304 can include errors calculated by the error model component 116, which can be represented as errors 306(1)-(3). In addition, the error model component 116 can determine probabilities associated with the errors 306(1)-(3), which can be represented as probabilities 308(1)-(3), that can be associated with environmental parameters to present the error models 310(1)-(3). By way of example, and not by way of limitation, the vehicle data 104 can include a bounding box associated with an object at a distance of 50 meters from the vehicle 202 in an environment including rainfall. The ground truth data 302 can provide a verified bounding box associated with the object. The error model component 116 can determine the error model data 304 that determines that the errors are associated with a perception system of the vehicle 202. A distance of 50 meters and rainfall may be used as environmental parameters to determine which of error models 310(1)-(3) to use. Once the error models are identified, error models 310(1)-(3) may provide errors 306(1)-(3) based on probabilities 308(1)-(3), where errors 306(1)-(3) associated with higher probabilities 308(1)-(3) are more likely to be selected than errors 306(1)-(3) associated with lower probabilities 308(1)-(3).

[0063] 4 illustrates an example 400 of perturbing a simulation using error model data by providing at least one of an error or uncertainty associated with a simulated environment. As described above, the error model data can include an error model that associates an error 306 with a probability 308, and the error model can be associated with an environmental parameter. The simulation component 124 can inject an error using the error model data 304, which can result in a perturbed parameterization scenario for perturbing the simulation. Based on the injected error, the simulation data can indicate how the autonomous controller responds to the injected error.

[0064] For example, the simulation component 124 can perturb the simulation by continuously injecting errors into the simulation. By way of example, and not by way of limitation, example 402 shows a bounding box 404 associated with an object at time t0. The bounding box 404 can represent a detection of the object by a vehicle that includes errors, such as errors in the size of the bounding box and / or errors in the position of the bounding box. At time t1, the simulation component 124 can use a bounding box 406 representing the object and including a different error. For example, at each simulation time (e.g., t0, t1, or t2), the simulation component 124 can use a different error 306 based on a probability 308 associated with the error 306. At time t2, the simulation component 124 can use a bounding box 408 representing the object and including a different error.

[0065] In some examples, the simulation component 124 can perturb the simulation by injecting uncertainty associated with a bounding box that represents an object in the environment. By way of example, and not limitation, example 410 shows a bounding box 412 associated with an object at time t0. The bounding box can include a 5% uncertainty, which can represent uncertainty in the size and / or position of the object by an amount of 5%. Additionally, the uncertainty can persist with the object through times t1 and t2, rather than injecting different errors at different simulation times, as depicted in example 402.

[0066] FIG. 5 illustrates an example 500 of a vehicle 202 generating vehicle data 104 and transmitting the vehicle data 104 to a computing device 204. As described above, the error model component 116 can determine a perceptual error model that can indicate an error associated with a scenario parameter. As described above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle 202 and / or perceptual data generated by a perception system of the vehicle 202. The perceptual error model can be determined by comparing the vehicle data 104 to ground truth data 302. The ground truth data 302 can be manually labeled, can be associated with an environment, and can represent known outcomes. Thus, deviations in the vehicle data 104 from the ground truth data 302 can be identified as errors in the sensor system and / or perception system of the vehicle 202. By way of example, and not by way of limitation, the perception system may identify an object as a bicyclist, where the ground truth data 302 indicates that the object is a pedestrian. As another example, without limitation, a sensor system may generate sensor data that represents an object as having a width of 2 meters, where the ground truth data 302 indicates that the object has a width of 1.75 meters.

[0067] As described above, the error model component 116 can determine a classification associated with an object represented in the vehicle data 104 and determine other objects of the same classification in the vehicle data 104 and / or other logged data. The error model component 116 can then determine a probability distribution associated with a range of errors associated with the object. Based on the comparison and range of errors, the error model component 116 can determine the perceptual error model data 502.

[0068] As shown in Figure 5, environment 504 may include objects 506(1)-(3) that are represented as bounding boxes generated by a perception system. Perceptual error model data 502 may indicate scenario parameters as 508(1)-(3) and errors associated with the scenario parameters as 510(1)-(3). As shown in Figure 5, the errors associated with object 508(1) may be visualized in environment 504 as a larger bounding box 512 that indicates uncertainty about the dimensions of object 508(1).

[0069] FIG. 6 illustrates an example 600 of a computing device 204 that generates simulation data 126 and determines vehicle performance data 132. The parameterization scenario component 120 can determine a parameterization scenario based on scenario parameters, a set of regions, and a perceptual error model. For example, the scenario parameters can indicate an object, a position associated with the object, a speed associated with the object, etc., in the parameterization scenario. Also, the scenario parameters can indicate a range indicating a range of values ​​and / or probabilities associated with the scenario parameters. The set of regions can indicate a portion of the environment that can be used to place the object in the simulated environment. Furthermore, the perceptual error model can indicate an error associated with the scenario parameters. These can be combined to create a parameterization scenario that can cover the variations provided by the scenario parameters, the set of regions, and / or the perceptual error model, as detailed herein.

[0070] The parameterized scenarios can be used by the simulation component 124 to simulate variations of the parameterized scenarios. For example, the simulation component 124 can execute variations of the parameterized scenarios for use in a simulation for testing and validation. The simulation component 124 can generate simulation data 126 indicative of how the autonomous vehicle controller performed (e.g., responded), compare the simulation data 126 to predetermined results, and / or determine if any predetermined rules / assertions were violated / triggered.

[0071] As illustrated in FIG. 6, the simulation data 126 can indicate several simulations (e.g., Simulation 1, Simulation 2, etc.) and results of the simulations (e.g., Outcome 1, Outcome 2). For example, as described above, the results can indicate pass or fail based on rules / assertions that were broken / triggered. The simulation data 126 can also indicate the probability of encountering a scenario. By way of example, and not by way of limitation, the simulation component 124 can simulate a scenario that includes a jaywalking pedestrian. The input data can indicate that the vehicle encounters a jaywalking pedestrian at a rate of one minute per hour of driving. This can be used to determine the probability of encountering a particular simulation associated with a variation of the parameterized scenario. In some examples, the simulation component 124 can identify variations of the parameterized scenario that have a low probability and run simulations that correspond to those variations. This can enable testing and validation of the autonomous vehicle controller in more unique situations.

[0072] Also, the simulation component 124 can identify parameterized scenario variations for additional simulations based on the results. By way of example, and not by way of limitation, the results of a simulation can be a fault where the scenario parameters are associated with a vehicle speed of 15 meters per second. The simulation component 124 can identify speeds that are close to the 15 meters per second speed to determine a threshold for the simulation to pass, which can further aid in the development of a safer vehicle controller.

[0073] Based on the simulation data 126, vehicle performance component 130 may generate vehicle performance data 132. As described above, for example, for event type 1 and cause type 1, data 134(1) may indicate a safety rating, as may data 134(2)-134(4). In some examples, the event type may indicate that other event types are contemplated, but that a cost meets or exceeds a cost threshold. For example, the costs may include, but are not limited to, a reference cost, an obstacle cost, a lateral cost, a longitudinal cost, etc.

[0074] The reference cost may include a cost associated with a difference between a point on the reference trajectory (also referred to as a reference point) and a corresponding point on the target trajectory (also referred to as a point or a target point), whereby the difference represents a difference in one or more of yaw, lateral offset, velocity, acceleration, curvature, curvature rate, etc. In some examples, reducing the weight associated with the reference cost may reduce the penalty associated with a target trajectory that is located a distance away from the reference trajectory, which may provide a smoother transition leading to safer and / or more comfortable vehicle operation.

[0075] In some examples, the obstacle cost may include a cost associated with a distance between a point on the reference trajectory or the target trajectory and a point associated with an obstacle in the environment. As an example, the point associated with the obstacle may correspond to a point on a boundary of the drivable area or may correspond to a point associated with an obstacle in the environment. In some examples, the obstacle in the environment may include, but is not limited to, static objects (e.g., buildings, curbs, sidewalks, lane markings, signs, traffic lights, trees, etc.) or dynamic objects (e.g., vehicles, bicyclists, pedestrians, animals, etc.). In some examples, the dynamic objects may also be referred to as agents. In some examples, the static or dynamic objects may be generally referred to as objects or obstacles.

[0076] In some examples, the lateral cost may refer to a cost associated with a steering input to the vehicle, such as a maximum steering input relative to the speed of the vehicle. In some examples, the longitudinal cost may refer to a cost associated with the speed and / or acceleration of the vehicle (e.g., maximum braking and / or acceleration). Such costs may be used to ensure that the vehicle is operating within feasible and / or comfort limits for passengers being carried on the ferry.

[0077] In some examples, cause type 1 and cause type 2 may indicate a fault, such as a fault in the vehicle or a fault in an object (e.g., a bicyclist). The vehicle performance component 130 may determine the fault using predefined rules / assertions. By way of example, and without limitation, a rule may indicate that if the vehicle is impacted by an object behind the vehicle, then the fault may be associated with the object. In some examples, additional rules may be used, such as indicating that the vehicle must traverse the environment in a forward direction when impacted by an object behind. In some examples, the cause types (e.g., cause type 1 and / or cause type 2) may be associated with components of the autonomous vehicle controller. By way of non-limiting example, such causes may include failures of a perception system, a prediction system, a planner system, network latency, torque / acceleration faults, and / or any other component or subcomponent of the vehicle.

[0078] As described above, the analysis component can determine to disable scenario parameters (e.g., set fixed values ​​associated with the scenario parameters) and vary other scenario parameters based on the simulation data 126. By isolating the scenario parameters, the analysis component can determine the scenario parameters associated with a successful or failed outcome. The vehicle performance data 132 can indicate safety metrics associated with the parameterized scenario. The analysis component can also perform sensitivity analysis to determine the cause of failure. For example, the analysis component can individually disable the scenario parameters to isolate one or more scenario parameters, determine how disabling the scenario parameters affects the response of the autonomous vehicle, capture statistical data associated with disabling one or more scenario parameters, and capture the outcome as a successful or failed outcome. The statistical data can indicate how a set of scenario parameters affects the outcome and can be used to determine scenario parameters that increase or decrease the likelihood of resulting in a successful simulation, and can be used to identify subsystems of the autonomous vehicle associated with the scenario parameters as increasing or decreasing the success rate of the simulation data.

[0079] 7 shows a block diagram of an example system 700 for implementing the techniques described herein. In at least one example, the system 700 can include a vehicle 202. In the illustrated example 700, the vehicle 202 is an autonomous vehicle, although the vehicle 202 can be any other type of vehicle (e.g., a driver-controlled vehicle that can provide an indication of whether it is safe to perform various operations).

[0080] Vehicle 202 may include a computing device 702, one or more sensor systems 704, one or more emitters 706, one or more communication connections 708 (also referred to as communication devices and / or modems), at least one direct connection 710 (e.g., for physically coupling with vehicle 202 to exchange data and / or provide power), and one or more drive systems 712. The one or more sensor systems 704 may be configured to capture sensor data associated with an environment.

[0081] The sensor system 704 may include time-of-flight sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement unit (IMU), accelerometer, magnetometer, gyroscope, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, intensity, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, etc. The sensor system 704 may include multiple instances of each of these or other types of sensors. For example, the time-of-flight sensors may include individual time-of-flight sensors positioned at the corners, front, rear, sides, and / or top of the vehicle 202. As another example, the camera sensors may include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 202. The sensor system 704 may provide input to the computing device 702.

[0082] The vehicle 202 may also include one or more emitters 706 for emitting light and / or sound. The one or more emitters 706 in this example include interior audio and visual emitters for communicating with passengers of the vehicle 202. By way of example, and not by way of limitation, the interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The one or more emitters 706 in this example also include exterior emitters. By way of example, and not by way of limitation, the exterior emitters in this example include lights for signaling a direction of travel or other indicators of the vehicle's operation (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with one or more pedestrians or other nearby vehicles, which may include acoustic beam steering technology.

[0083] Vehicle 202 may also include one or more communications connections 708 that enable communication between vehicle 202 and one or more other local or remote computing devices (e.g., remote teleoperated computing devices) or remote services. For example, communications connections 708 may facilitate communication with other local computing devices on vehicle 202 and / or drive system 712. Communications connections 708 may also enable vehicle 202 to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.).

[0084] The communications connections 708 may include physical and / or logical interfaces for connecting the computing device 702 to another computing device or to one or more external networks 714 (e.g., the Internet). For example, the communications connections 708 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), satellite communications, dedicated short-range communications (DSRC), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices. In at least some examples, the communications connections 708 may comprise one or more modems, as detailed above.

[0085] In at least one example, the vehicle 202 can include one or more drive systems 712. In some examples, the vehicle 202 can have a single drive system 712. In at least one example, when the vehicle 202 has multiple drive systems 712, the individual drive systems 712 can be located at opposite ends of the vehicle 202 (e.g., the front and the rear, etc.). In at least one example, the drive system 712 can include one or more sensor systems 704 for detecting conditions surrounding the drive system 712 and / or the vehicle 202. By way of example and not limitation, the sensor systems 704 can include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive system, cameras or other imaging sensors, ultrasonic sensors, lidar sensors, radar sensors, etc. for acoustically detecting objects surrounding the drive system. Certain sensors, such as wheel encoders, can be specific to the drive system 712. In some cases, the sensor system 704 on the drive system 712 may overlap or supplement a corresponding system (e.g., the sensor system 704) of the vehicle 202.

[0086] The drive system 712 can include many vehicle systems, including a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for braking force distribution to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as headlights / taillights that illuminate the perimeter of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). The drive system 712 can also include a drive system controller that can receive and pre-process data from the sensor system 704 and control the operation of various vehicle systems. In some examples, the drive system controller can include one or more processors and a memory communicatively coupled to the one or more processors. The memory can store one or more modules for performing various functions of the drive system 712. In addition, drive system 712 also includes one or more communication connections that enable each drive system to communicate with one or more other local or remote computing devices.

[0087] Computing device 702 may include one or more processors 516 and memory 518 communicatively coupled to the one or more processors 716. In the illustrated example, memory 718 of computing device 702 stores a localization component 720, a perception component 722, a prediction component 724, a planning component 726, and one or more system controllers 728. While shown as residing in memory 718 for illustrative purposes, it is contemplated that localization component 720, perception component 722, prediction component 724, planning component 726, and one or more system controllers 728 may additionally or alternatively be accessible to computing device 702 (e.g., stored in a different component of vehicle 202) and / or accessible to vehicle 202 (e.g., stored remotely).

[0088] In the memory 718 of the computing device 702, the localization component 720 can include functionality for receiving data from the sensor system 704 to determine a position of the vehicle 202. For example, the localization component 720 can include and / or request / receive a three-dimensional map of the environment and can continually determine a position of the autonomous vehicle within the map. In some examples, the localization component 720 can use simultaneous localization and mapping (SLAM) or calibration, localization and mapping, simultaneously (CLAMS) to receive time-of-flight data, image data, lidar data, radar data, sonar data, IMU data, GPS data, wheel encoder data, or any combination thereof, etc. to precisely determine a position of the autonomous vehicle. In some examples, the localization component 720 can provide data to various components of the vehicle 202 to determine an initial position of the autonomous vehicle for generating a trajectory, as discussed herein.

[0089] The perception component 722 may include functionality for performing object detection, segmentation, and / or classification. In some examples, the perception component 722 may provide processed sensor data indicative of the presence of an entity proximate to the vehicle 202 and / or the classification of the entity as an entity type (e.g., automobile, pedestrian, bicyclist, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional and / or alternative examples, the perception component 722 may provide processed sensor data indicative of one or more characteristics (also referred to as parameters) associated with the detected entity and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity may include, but are not limited to, x-position (global location), y-position (global location), z-position (global location), direction, entity type (e.g., classification, etc.), velocity of the entity, range (size) of the entity, etc. The characteristics associated with the environment may include, but are not limited to, the presence of another entity in the environment, the state of another entity in the environment, time of day, day of the week, season, weather conditions, geographic location, dark / light indication, etc.

[0090] The perception component 722 may include functionality for storing the perception data generated by the perception component 722. In some examples, the perception component 722 may determine a track corresponding to an object classified as an object type. For purposes of illustration only, the perception component 722 using the sensor system 704 may capture one or more images of an environment. The sensor system 704 may capture an image of an environment including an object, such as a pedestrian. The pedestrian may be at a first location at time T and at a second location at time T+t (e.g., movement during a span of time t after time T). In other words, the pedestrian may move from a first location to a second location during this time period. Such movement may be stored in a log, for example, as stored perception data associated with the object.

[0091] The stored perception data may include fused perception data captured by the vehicle in some examples. The fused perception data may include a fusion or other combination of sensor data from a sensor system 704, such as image sensors, lidar sensors, radar sensors, time-of-flight sensors, sonar sensors, global positioning system sensors, interior sensors, and / or any combination thereof. The stored perception data may additionally or alternatively include classification data including a semantic classification of an object (e.g., a pedestrian, a vehicle, a building, a road surface, etc.) represented in the sensor data. The stored perception data may additionally or alternatively include track data (a collection of historical positions, orientations, sensor features, etc. associated with an object over time) corresponding to the movement of an object classified as a dynamic object through the environment. The track data may include multiple tracks of multiple different objects over time. This track data may be mined to identify images of a particular type of object (e.g., a pedestrian, an animal, etc.) when the object is stationary (e.g., stationary) or moving (e.g., walking, running, etc.). In this example, the computing device determines a track corresponding to a pedestrian.

[0092] The prediction component 724 can generate one or more probability maps that represent predicted probabilities of possible locations of one or more objects in the environment. For example, the prediction component 724 can generate one or more probability maps for vehicles, pedestrians, animals, etc., that are within a threshold distance from the vehicle 202. In some examples, the prediction component 724 can measure the tracks of the objects and generate discretized predicted probability maps, heat maps, probability distributions, discretized probability distributions, and / or trajectories of the objects based on the observed and predicted behavior. In some examples, the one or more probability maps can represent the intent of one or more objects in the environment.

[0093] The planning component 726 can determine a path for the vehicle 202 to follow to traverse through an environment. For example, the planning component 726 can determine various routes and paths and various levels of detail. In some examples, the planning component 726 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this description, a route may be a series of waypoints for travel between the two locations. As non-limiting examples, the waypoints include roads, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 726 can generate instructions for guiding the autonomous vehicle along at least a portion of a route from the first location to the second location. In at least one example, the planning component 726 can determine how to guide the autonomous vehicle from a first waypoint in the series of waypoints to a second waypoint in the series of waypoints. In some examples, the instructions may be a path or a portion of a path. In some examples, multiple paths may be generated substantially simultaneously (i.e., within technical tolerances) in accordance with a receding horizon technique. A single path among multiple paths in the receding data horizon that has the highest confidence level can be selected for maneuvering the vehicle.

[0094] In other examples, the planning component 726 can alternatively or additionally use data from the perception component 722 to determine a path for the vehicle 202 to follow to traverse through the environment. For example, the planning component 726 can receive data from the perception component 722 regarding objects associated with the environment. Using this data, the planning component 726 can determine a path to travel from a first location (e.g., a current location) to a second location (e.g., a target location) to avoid the objects in the environment. In at least some examples, such a planning component 726 can determine that there is no such collision-free path and then provide a path that leads the vehicle 202 to a safe stop that avoids all collisions and / or otherwise mitigates damage.

[0095] In at least one example, computing device 702 can include one or more system controllers 728, which can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 202. These system controllers 728 can communicate with and / or control corresponding systems of drive system 712 and / or other components of vehicle 202, which can be configured to operate according to the path provided from planning component 726.

[0096] Vehicle 202 can be connected to computing device 204 via network 514 and can include one or more processors 730 and memory 732 communicatively coupled to the one or more processors 730. In at least one example, one or more processors 730 can be similar to processor 716 and memory 732 can be similar to memory 718. In the illustrated example, memory 732 of computing device 204 stores scenario editor component 108, parameter component 112, error model component 116, parameterization scenario component 120, simulation component 124, analysis component 128, and vehicle performance component 130. While depicted as residing in memory 732 for illustrative purposes, it is contemplated that the scenario editor component 108, the parameter component 112, the error model component 116, the parameterized scenario component 120, the simulation component 124, the analysis component 128, and the vehicle performance component 130 may additionally or alternatively be accessible to the computing device 204 (e.g., stored in a different component of the computing device 204) and / or be accessible to the computing device 204 (e.g., stored remotely). The scenario editor component 108, the parameter component 112, the error model component 116, the parameterized scenario component 120, the simulation component 124, the analysis component 128, and the vehicle performance component 130 may be substantially similar to the scenario editor component 108, the parameter component 112, the error model component 116, the parameterized scenario component 120, the simulation component 124, the analysis component 128, and the vehicle performance component 130 of FIG.

[0097] The processor 716 of the computing device 702 and the processor 730 of the computing device 204 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processors 716 and 730 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it into registers or other electronic data that can be stored in memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement the encoded instructions.

[0098] Memory 718 of computing device 702 and memory 732 of computing device 204 are examples of non-transitory computer-readable media. Memory 718 and 732 can store an operating system and one or more software applications, instructions, programs, and / or data that implement the methods and functions attributed to the various systems described herein. In various implementations, memory 718 and 732 can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, of which those shown in the accompanying drawings are merely examples relevant to the description herein.

[0099] In some examples, some or all aspects of the components described herein may include any model, algorithm, and / or machine learning algorithm, For example, in some examples, the components in memories 718 and 732 may be implemented as neural networks.

[0100] 8 illustrates an example process 800 for determining a safety metric associated with a vehicle controller. Some or all of the process 800 may be performed by one or more components of Figures 1-7 as described herein. For example, some or all of the process 800 may be performed by computing device 204 and / or computing device 702.

[0101] At operation 802 of the example process 800, the process 800 may include receiving log data associated with operation of an autonomous vehicle within an environment. In some examples, the log data may be generated by the vehicle capturing at least sensor data of the environment.

[0102] At operation 804 of the example process 800, the process 800 can include determining a set of scenarios based on the log data (or other data), where the scenarios of the set of scenarios can include scenario parameters associated with aspects of the environment. In some examples, the computing device can group similar scenarios represented in the log data. For example, the scenarios can be grouped together using, for example, K-means clustering and / or evaluating weighted distances (e.g., Euclidean) between parameters of the environment. Also, the scenario parameters can represent environmental parameters, such as nighttime environmental parameters or wet condition environmental parameters. In some examples, the scenario parameters can be associated with a vehicle or object (e.g., pose, speed, etc.).

[0103] At operation 806 of the example process 800, the process 800 may include determining an error model associated with a subsystem of the autonomous vehicle. The error model component may compare the vehicle data (e.g., logged data) to ground truth data to determine a difference between the vehicle data and the ground truth data. In some examples, the vehicle data may represent estimates associated with an object in the environment, such as an estimated position, an estimated orientation, an estimated range, etc., and the ground truth data may represent an actual position, an actual orientation, or an actual range of the object. Based on the difference, the error model component may determine an error associated with the subsystem of the vehicle (e.g., a perception system, a tracking system, a prediction system, etc.).

[0104] In operation 808 of the example process 800, the process 800 may include determining parameterized scenarios based on the scenario parameters and the error model. These may be combined to create parameterized scenarios that can cover the variations provided by the scenario parameters and / or the error model. In some examples, the scenario parameters may be randomly selected and combined to create the parameterized scenarios. In some examples, the scenario parameters may be combined based on the probability of them occurring together. As an example, but not by way of limitation, the log data may indicate that 5% of the driving experience includes pedestrian encounters, and the parameterized scenario component may include pedestrians as a scenario parameter in 5% of the parameterized scenarios generated by the parameterized scenario component. In some examples, the parameterized scenario component may validate the parameterized scenarios to eliminate unlikely or unlikely combinations of scenario parameters. As an example, but not by way of limitation, the vehicle is not located in a lake and pedestrians are not walking at a speed of 30 meters per second. As non-limiting examples, such parameterized scenarios may include ranges of vehicle and jaywalker distances, speeds, lighting conditions, weather conditions, etc. on particular defined roads with various Gaussian (or other) distributions of errors in perception models, predictive models, etc. based at least in part on scenario parameters.

[0105] At operation 810 of the example process 800, the process 800 can include perturbing the parameterized scenario by modifying at least one of the parameterized scenario, the scenario parameters, or the components of the simulated vehicle based at least in part on the error. In some examples, the uncertainty can be associated with the scenario parameters. As an example, but not by way of limitation, the position of an object can be associated with a 5% uncertainty and the autonomous controller can be caused to traverse the environment while accounting for the 5% uncertainty. In some examples, the simulator can determine from the error model the error to incorporate into the simulation as the simulator runs the simulation.

[0106] At operation 812 of the example process 800, the process 800 may include instantiating a simulated vehicle in the perturbed parameterized scenario. The simulator may use the simulated vehicle, which may be associated with an autonomous controller, to cause the autonomous controller to traverse the simulated environment. Instantiating the autonomous vehicle controller with the parameterized scenario and simulating the parameterized scenario may efficiently cover a wide range of scenario variations without requiring manual enumeration of the variations. Additionally, based at least in part on executing the parameterized scenario, simulation data may indicate how the autonomous vehicle controller responded to the parameterized scenario, and a successful or unsuccessful outcome may be determined based at least in part on the simulation data.

[0107] At operation 814 of the example process 800, the process may include receiving simulation data indicative of how a simulated vehicle responds to the perturbed parameterization scenarios. After the simulation, the results may indicate a pass (e.g., a successful outcome), a fail, and / or a degree of success or failure associated with the vehicle controller.

[0108] At operation 816 of the example process 800, the process may include determining a safety metric associated with the vehicle controller based on the simulation data. For example, each simulation may result in a success or failure outcome. Also, as described above, the vehicle performance component may determine vehicle performance data based on the simulation data that may indicate how the vehicle performs in the environment. Based on the sensitivity analysis, the vehicle performance data may indicate scenarios where the simulation results were not successful, causes of the simulation failure, and / or bounds of scenario parameters that indicate values ​​of the scenario parameters where the simulation results were successful. Thus, the safety metric may indicate pass / fail rates of the vehicle controller in various simulated scenarios.

[0109] 9 illustrates a flow diagram of an example process for determining a statistical model associated with a subsystem of an autonomous vehicle. Some or all of the process 900 may be performed by one or more components of FIGS. 1-7 as described herein. For example, some or all of the process 900 may be performed by computing device 204 and / or computing device 702.

[0110] At operation 902 of the example process 900, the process 900 may include receiving vehicle data (or other data) associated with a subsystem of an autonomous vehicle. The vehicle data may include log data captured by a vehicle traveling through an environment. In some examples, the vehicle data may include control data (e.g., data used to control systems such as steering, braking, etc.) and / or sensor data (e.g., lidar data, radar data, etc.).

[0111] At operation 904 of the example process 900, the process 900 may include determining output data associated with a subsystem based on the vehicle data. By way of example, and without limitation, the subsystem may be a perception system and the output data may be a bounding box associated with an object in the environment.

[0112] At operation 906 of the example process 900, the process 900 may include receiving ground truth data associated with the subsystem. In some examples, the ground truth data may be manually labeled and / or determined from other validated machine learning components. By way of example, and without limitation, the ground truth data may include validated bounding boxes associated with objects in the environment.

[0113] At operation 908 of the example process 900, the process 900 may include determining a difference between a first portion of the output data and a second portion of the ground truth data, the difference representing an error associated with the subsystem. As described above, the output data may include a bounding box associated with an object in the environment as detected by the object in the environment detected by the vehicle's perception system, and the ground truth data may include a verified bounding box associated with the object. The difference between the two bounding boxes may indicate an error associated with the vehicle's perception system. By way of example, and not by way of limitation, the bounding box of the output data may be larger than the verified bounding box, indicating that the perception system is detecting that the object is larger than the one in the environment.

[0114] In operation 910 of the example process 900, the process 900 can include determining a statistical model associated with the subsystem based on the difference.

[0115] FIG. 10 illustrates an example 1000 of multiple regions of an environment associated with error probabilities. As described above, the error model can indicate one or more parameters associated with the environment, and the error model can indicate a probability and / or error distribution associated with the one or more parameters. The error can represent a difference between data associated with the environment generated by the vehicle and the actual environment (or ground truth data representing the environment). For example, the error can indicate a difference in size, position, and / or speed of an object in the environment, or whether an object was detected in the environment. In some examples, the error can represent a difference between a classification of an object (e.g., pedestrian, bicyclist, vehicle, sign, static object, dynamic object, etc.) generated by the vehicle and an actual classification of the object. In some examples, the error can represent a difference between a predicted trajectory of an object and an actual trajectory of the object.

[0116] Example 1000 illustrates the probability associated with an error and how factors such as the distance between a sensor and an area in the environment, or the type of object in the environment, can affect such probability.

[0117] By way of example, and without limitation, region 1002 may be associated with a portion of the environment that is within line of sight of vehicle 210. For example, based at least in part on the distance between region 1002 and vehicle 210, the error model may indicate a probability 1004(1) associated with error 1006(1). For example, a relatively close distance between region 1002 and vehicle 210 may reduce the probability of an error due to increased precision, accuracy, and / or density of sensor data compared to sensor data in an area of ​​the environment that is relatively far from vehicle 210.

[0118] Additionally, environmental conditions may affect the probability of error. By way of example and not limitation, the probability 1004(1) of an error 1006(1) associated with an object perceived by vehicle 210 but not present in the environment (e.g., a false positive error) may be lower during ideal conditions (e.g., good lighting, sunny weather, etc.) compared to the probability 1004(1) associated with a false positive error during environmental conditions such as darkness, rain, snow, etc.

[0119] As another example, without limitation, region 1008 may be associated with a portion of the environment that is within the line of sight of vehicle 210 and associated with object 212. For example, based at least in part on the distance between region 1008 and vehicle 210 and / or the classification associated with object 212, the error model may indicate a probability 1004(2) associated with error 1006(2). As an example, without limitation, vehicle 210 may determine a size associated with object 212 based on the sensory data. The error model may indicate that the probability 1004(2) of error 1006(2) that the actual size of object 212 is twice that indicated by the sensory data is lower than the probability 1004(2) of error 1006(2) that the actual size of object 212 is only slightly larger than that indicated by the sensory data.

[0120] As another example, without limitation, region 1010 may be associated with a portion of the environment that is occluded or partially occluded by object 212. For example, based at least in part on the amount of region 1010 that is occluded, the error model may indicate a probability 1004(3) associated with error 1006(3). As an example, without limitation, the probability 1004(3) of error 1006(3) associated with an object that is not recognized (also referred to as being omitted) by vehicle 210 but is present in the environment (e.g., a false negative error) may be higher compared to the probability 1004(1) of error 1006(1) associated with a false negative error in region 1002 due to occlusion or partial occlusion of region 1010.

[0121] As another example, without limitation, region 1012 may be associated with a portion of the environment that is within the line of sight of vehicle 210 and associated with object 1014. For example, based at least in part on the distance between region 1012 and vehicle 210 and / or the classification associated with object 1014, the error model may indicate a probability 1004(4) associated with error 1006(4). As an example, without limitation, vehicle 210 may determine a pose associated with object 1014 based on the sensory data. The error model may indicate a probability 1004(4) of error 1006(4) that the actual pose of object 1014 differs from the pose indicated by the sensory data. In some examples, the probability 1004(4) of error 1006(4) associated with region 1012 may be higher than the probability 1004(2) of error 1006(2) due to an increased distance between region 1012 and vehicle 210 compared to the distance between region 1008 and vehicle 210. Additionally, errors 1006(4) and 1006(2) may vary based on the classification associated with objects 212 and 1014. That is, the error model may be conditioned based on classification, distance, environmental factors, and the like.

[0122] The error model indicative of the probabilities 1004(1), 1004(2), 1004(3), and 1004(4) and the errors 1006(1), 1006(2), 1006(3), and 1006(4) may be generated based at least in part on the vehicle data 104 generated by the vehicle 202. For example, the vehicle 202 may generate the vehicle data 104 associated with an environment and a time. The vehicle data 104 may include sensor data, sensory data, predictive data, etc. As described above, the error model component 116 may receive the vehicle data 104 and the ground truth data 302 to determine a difference between the vehicle data 104 and the ground truth data 302. The difference may be indicative of an error in the vehicle data 104. As described above, the error may be associated with a difference in a classification of an object, a size of an object, a position of an object, a speed of an object, and / or a trajectory of an object within an environment. In some examples, the difference may be associated with vehicle data 104 indicating that an object is present in the environment when the object is actually not present (e.g., a false positive error), and in some examples, the difference may be associated with vehicle data 104 indicating that an object is not present in the environment (e.g., is omitted) when the object is actually present (e.g., a false negative error).

[0123] In some examples, the vehicle data 104 may include data associated with multiple environments, and the error model component 116 may determine a frequency (also referred to as a frequency of occurrence) associated with one or more errors. By way of example, and not by way of limitation, based on the differences between the vehicle data 104 and the ground truth data 302 and the frequencies of the differences, the error model may indicate that the probability of the vehicle data 104 representing a curb height that is 2 cm higher than an actual curb in the environment is higher than the probability of the vehicle data 104 representing a curb height that is 1 meter higher than the actual curb, and the differences and probabilities may be represented as distributions in the error model.

[0124] In at least some examples, various environmental parameters, such as, but not limited to, vehicle speed, position, object position, speed, object classification, weather, time, etc., may be used in conjunction with the errors to cluster the measurements into various categories (e.g., using k-means, decision trees, other suitable clustering algorithms, etc.). The measurements within each cluster may be aggregated to determine an error distribution associated with the cluster. Then, when driving, the vehicle may select the cluster with the closest environmental parameters currently observed and then use the associated error model.

[0125] In some examples, a machine learning model may be used to determine the distribution fit. By way of example, and not by way of limitation, the differences between the vehicle data 104 and the ground truth data 302 may represent discrete points. The machine learning model may determine the distribution based on the discrete points by fitting one or more distributions to the discrete points. In some examples, the distribution may be a discrete distribution (e.g., logarithmic distribution, Poisson distribution, etc.), a Bernoulli distribution, a continuous distribution (e.g., generalized normal distribution, Gaussian distribution, etc.), a mixed discrete / continuous distribution, a joint distribution, etc. Of course, in some examples, a histogram (or similar) of the measurement data associated with a particular cluster may be stored in addition to / alternatively to determining the best fit of the distribution.

[0126] In some examples, a clustering algorithm may be used to cluster the vehicle data 104. By way of example, and without limitation, the vehicle data 104 may be clustered based on classifications associated with objects, the time of day, the amount of visible light detected by the vehicle 202, weather conditions (e.g., sunshine, overcast, precipitation, fog, etc.), the speed of the vehicle 202, the pose of the vehicle 202, and / or the region of the environment. Clustering algorithms may include, for example, class-specific clustering, k-means algorithm, expectation maximization algorithm, multivariate normal distribution, etc.

[0127] The clustered vehicle data 104 may be used to determine distributions associated with the vehicle data clusters of the clustered vehicle data 104. By way of example, and not limitation, class-specific clustering may include clustering objects represented in the vehicle data 104 by classification (e.g., pedestrian, vehicle, etc.), where the distributions associated with the pedestrian clusters may be used as pedestrian error models and the distributions associated with the vehicle clusters may be used as vehicle error models.

[0128] 11 illustrates an example 1100 of vehicle data, an environment represented by the vehicle data, and a difference between the vehicle data and the environment. Generally, the example 1100 illustrates a matrix of true positive error, false negative error, false positive error, and a true negative situation (no error), each of which are described below throughout this disclosure.

[0129] Example 1102 shows vehicle data 1104(1) associated with environment 1106(1). As illustrated in example 1102, vehicle data 1104(1) may include sensory data 1108 that may be associated with an object (e.g., object 1110) perceived by the vehicle in environment 1106(1). However, as depicted in environment 1106(1), sensory data 1108 may inaccurately indicate attribute values ​​associated with object 1110, such as the location, classification, size, etc. of associated object 1110. Thus, this may be treated as a true positive scenario, although other differences may occur between vehicle data 1104(1) and environment 1106(1).

[0130] Example 1112 shows vehicle data 1104(2) associated with environment 1106(2). As illustrated in example 1112, vehicle data 1104(2) may indicate that the vehicle does not perceive any objects in the environment, but that object 1114 is present in environment 1106(2). This may be treated as a false negative scenario.

[0131] Example 1116 shows vehicle data 1104(3) associated with environment 1106(3). As illustrated in example 1116, vehicle data 1104(3) may include sensory data 1118 that may be associated with objects that are perceived by the vehicle but are not present as illustrated by environment 1106(3). This may be treated as a false positive scenario.

[0132] Example 1120 shows vehicle data 1104(4) associated with environment 1106(4). In example 1120, vehicle data 1104(4) does not represent any objects, and similarly, environment 1106(4) does not contain any objects. Thus, this may be treated as a true negative scenario.

[0133] With respect to false positive error, the error model may be adjusted for distance, environmental factors, etc. False positive error may indicate, for example, the probability that the perception system determines that an object is present in an area of ​​the environment when the object is not present in that area of ​​the environment.

[0134] With respect to false negative error, the error model may be adjusted by distance, environmental factors, object attributes, etc. False negative error may indicate, for example, the probability that a perception system determines that an object is not present in a region of the environment when the object is actually present in that region of the environment. As discussed above, false negative error may be based at least in part on the classification of the true object, the size of the true object, the distance between the sensor capturing the data and the region of the environment associated with the object, the amount of occlusion of the true object, etc.

[0135] As described above, the error model can include distributions (e.g., true positive, false negative, false positive, and / or true negative scenarios) corresponding to examples 1102, 1112, 1116, and / or 1120. The autonomous vehicle controller can traverse the simulated environment, and the simulation component 124 can determine probabilities of true negatives, false positives, false negatives, and / or true positives associated with regions of the simulated environment based at least in part on the error model. The simulation component 124 can determine perturbed simulation data to perturb the simulated environment by generating false positive scenarios, false negative scenarios, and / or true positive scenarios, and can determine responses of the simulated vehicle controller (also referred to as a simulated autonomous vehicle controller) to the false positive scenarios, false negative scenarios, and / or true positive scenarios.

[0136] By way of example, and not by way of limitation, the simulation component 124 may select an error model based at least in part on the distance between the simulated vehicle controller and the simulated object and / or any additional environmental parameters as described herein. For example, when the simulated object is initially represented as being 5 meters away from the simulated vehicle controller, the simulation component 124 may determine an error model that represents the error encountered at such distance. At a second time after the first time, the simulation may represent the simulated object at a second distance (e.g., 10 meters) from the simulated vehicle controller. The simulation component 124 may select the error model based on such distance and may determine additional perturbation scenario data that represents the simulated object in the simulated environment at the second time. Thus, the simulation component 124 may use the error model to continuously (or intermittently) perturb the simulated environment as the simulation is executed.

[0137] In some examples, the error distribution of the error model may be based at least in part on the simulation time elapsed. By way of example, and not by way of limitation, a simulated vehicle controller may perceive a simulated object at a first time and perceive the simulated object at a second time. The error model may indicate that a probability of error associated with the perception of the simulated object at the first time may be higher than a probability of error associated with the perception of the simulated object at the second time. In some examples, the first time may represent a time at which the simulated vehicle controller first detects the simulated object. That is, the error model may vary based at least in part on a duration associated with the simulated object tracked by the simulated vehicle controller.

[0138] In some examples, the error model may be used during operation of an autonomous vehicle in a non-simulated environment. By way of example, and not by way of limitation, an autonomous vehicle may traverse an environment and capture sensor data of the environment. A perception system of the autonomous vehicle may generate sensory data indicative of objects in the environment, for example. The vehicle may include an error model component, and the error model component may be used to determine a probability of error associated with the sensory data. For example, the autonomous vehicle may perceive an object in the environment, such as a parking curb in a parking lot. The autonomous vehicle may receive a probability of error associated with a location of the parking curb. By way of example, and not by way of limitation, the error model may indicate that there is a 5% chance that the parking curb is within 0.1 meters laterally of a location where the autonomous vehicle perceived the parking curb. As described above, the probability can be based on characteristics such as x position (global position), y position (global position), z position (global position), direction, object type (e.g., classification), object speed, object range (size), presence of other objects in the environment, state of other objects in the environment, time of day, day of week, season, weather conditions, darkness / light indications, etc.

[0139] Based on the probability of perception error, the autonomous vehicle's planning system can traverse the environment more safely by taking into account the probability of perception error. In some examples, the planning system can use a probability threshold and determine a trajectory that avoids object locations that meet or exceed the probability threshold. By way of example and not by way of limitation, the planning system can use a 1% probability threshold and traverse around object locations that have a probability of 1% or greater. In some examples, the planning system can use a probability threshold associated with an object classification. By way of example and not by way of limitation, the planning system can have a lower probability threshold associated with a pedestrian classification compared to a probability threshold associated with debris on the driving surface.

[0140] FIG. 12 illustrates a flow diagram of an exemplary process for determining an error model and determining perturbed simulation data.

[0141] At operation 1202 of the example process 1200, the process 1200 may include receiving vehicle data indicative of a perceived state of an object. In some examples, the vehicle data may include log data captured by a vehicle traveling through an environment. In some examples, the vehicle data may include control data (e.g., data used to control systems such as steering, braking, etc.), sensor data (e.g., lidar data, radar data, etc.), and / or output from subsystems of the vehicle (e.g., perception data (e.g., classification data, bounding box, x position (global position), y position (global position), z position (global position), direction, entity type (e.g., classification), velocity of the entity, range of the entity, prediction data, etc.).

[0142] At operation 1204 of the example process 1200, the process 1200 may include receiving ground truth data associated with the object based at least in part on the vehicle data. In some examples, the ground truth data may be manually labeled and / or determined from other validated machine learning components. By way of example, and without limitation, the ground truth data may include validated bounding boxes associated with the object in the environment.

[0143] At operation 1206 of the example process 1200, the process 1200 may include determining a difference between the vehicle data and the ground truth data. If no difference exists, the process 1200 may return to operation 1202. If a difference exists, the process may proceed to operation 1208.

[0144] At operation 1208 of the example process 1200, the process 1200 may include determining an error based at least in part on the vehicle data and the ground truth data. As described above, a difference between the vehicle data and the ground truth data may represent an error in the vehicle data. The error may be associated with a probability, and a distribution of the error and the probability may be included in an error model.

[0145] At operation 1210 of the example process 1200, the process 1200 may include determining a plurality of parameters based at least in part on the vehicle data. In some examples, the vehicle data may include control data (e.g., data used to control systems such as steering, braking, etc.), sensor data (e.g., lidar data, radar data, etc.), and / or output from subsystems of the vehicle (e.g., perception data (e.g., classification data, bounding box, x position (global position), y position (global position), z position (global position), direction, entity type (e.g., classification), velocity of an entity, range of an entity, prediction data, etc.). In some examples, the vehicle data may include characteristics associated with the environment, such as, but not limited to, the presence of another entity in the environment, the state of another entity in the environment, the time of day, the day of the week, the season, weather conditions, dark / light indications, etc.

[0146] At operation 1212 of the example process 1200, the process 1200 may include clustering at least a portion of the vehicle data based at least in part on the plurality of parameters and the error. For example, the vehicle data may be clustered based on a classification of objects represented in the vehicle data, a type of day, an amount of visible light detected by the vehicle, weather conditions, a speed of the vehicle, a pose of the vehicle, and / or a region of the environment. Distributions of errors and probabilities may be associated with the clusters of the clustered vehicle data.

[0147] At operation 1214 of the example process 1200, the process 1200 may include determining an error model based at least in part on a portion of the vehicle data. As described above, a machine learning model may be used to determine a distribution fit to the distribution of errors and probabilities represented in the vehicle data. In some examples, a fit metric may be used to determine the fit of the distribution to the distribution of errors and probabilities. By way of example, and not by way of limitation, a negative log-likelihood function or a positive log-likelihood function may be used to determine the fit metric. By way of example, and not by way of limitation, a distribution for unclustered vehicle data may produce a higher negative log-likelihood value than a distribution for vehicle data clustered using multivariate expectation maximization, with a higher negative log-likelihood value indicating a worse fit of the distribution.

[0148] In some examples, operation 1214 may include determining, for an "empty" region of the environment, an error model associated with a false positive at that location. In some examples, operation 1214 may include determining, for an "occupied" region of the environment, an error model associated with a false negative at that location. For example, operation 1214 may include determining an error model based on distance, environmental factors, etc.

[0149] (Example clauses) A: A system comprising one or more processors and one or more computer readable media storing computer-executable instructions, the computer-executable instructions, when executed, comprising: receiving log data associated with operating an autonomous vehicle in an environment; determining a set of scenarios based at least in part on the log data, a scenario of the set of scenarios including scenario parameters associated with an aspect of the environment; determining an error model associated with a subsystem of the autonomous vehicle; determining a parameterized scenario based at least in part on the scenario parameters and the error model; perturbing the parameterized scenario by adding an error to at least one of the scenario parameters or a simulated vehicle component instantiated in the perturbed parameterized scenario, the simulated vehicle being controlled by a vehicle controller; instantiating the simulated vehicle in the perturbed parameterized scenario; receiving simulation data indicative of how the simulated vehicle responds to the perturbed parameterized scenario; and determining a safety metric associated with the vehicle controller based at least in part on the simulation data. A system that causes the system to perform operations including:

[0150] B: The system of paragraph A, wherein determining the set of scenarios includes clustering the log data to determine a first set of clusters, each cluster of the first set of clusters being associated with an individual scenario; determining a probability associated with each cluster based at least in part on the first set of clusters; and determining a second set of clusters based at least in part on a probability threshold and the first set of clusters.

[0151] C: The system described in paragraph A, wherein determining the error model includes receiving ground truth data associated with the environment, determining an error based at least in part on comparing the ground truth data to the log data, and determining an error distribution based at least in part on the error, wherein the error model includes the error distribution.

[0152] D: The system of paragraph A, wherein the parameterization scenario is a first parameterization scenario, the perturbed parameterization scenario is a first perturbed parameterization scenario, and the simulation data is first simulation data, and the operations further include determining a second parameterization scenario including at least one of a first subset of the scenario parameters or a second subset of the error model based on the first simulation data, perturbing the second parameterization scenario to be a second perturbed parameterization scenario, instantiating the simulated vehicle in the second perturbed parameterization scenario, receiving second simulation data, and updating a safety metric based at least in part on the second simulation data.

[0153] E: A method comprising: determining a scenario including scenario parameters describing a portion of an environment; receiving an error model associated with a subsystem of a vehicle; determining a parameterized scenario based at least in part on the scenario, the scenario parameters, and the error model; perturbing the parameterized scenario to result in a perturbed parameterized scenario; receiving simulation data indicative of how the subsystem of the vehicle responds to the perturbed parameterized scenario; and determining a safety metric associated with the subsystem of the vehicle based at least in part on the simulation data.

[0154] F: The method of paragraph E, wherein the scenario parameters are associated with at least one of object size, object velocity, object pose, object density, vehicle speed, and vehicle trajectory.

[0155] G: The method of paragraph E, wherein determining the scenario includes receiving log data associated with an autonomous vehicle; clustering the log data to determine a first set of clusters, each cluster of the first set of clusters being associated with the scenario; determining a probability associated with each cluster based at least in part on the first set of clusters; and determining that the probability meets or exceeds a probability threshold.

[0156] H: The method of paragraph E, wherein the error model is determined based at least in part on receiving ground truth data associated with the environment, determining an error based at least in part on comparing the ground truth data to log data associated with the vehicle, and determining an error distribution based at least in part on the error, and the error model includes the error distribution.

[0157] I: The method of paragraph E, wherein the parameterization scenario is a first parameterization scenario, the perturbed parameterization scenario is a first perturbed parameterization scenario, and the simulation data is first simulation data, the method further including: determining a second parameterization scenario including at least one of a first subset of the scenario parameters or a second subset of the error model based on the first simulation data; perturbing the second parameterization scenario; receiving second simulation data; and updating the safety metric based at least in part on the second simulation data.

[0158] J: The method of paragraph I, further comprising: invalidating at least a first portion of one of the scenario parameters or the error model; and associating the second simulation data with at least a second portion of one of the scenario parameters or the error model that has not been invalidated.

[0159] K: The method of paragraph E, wherein the safety metric indicates a probability of meeting or exceeding a cost threshold.

[0160] L: The method of paragraph E, wherein the portion is a first portion, the method further including receiving map data, a second portion of the map data being associated with the first portion of the environment, and determining that the second portion of the map data is associated with a scenario associated with a probability of meeting or exceeding a threshold probability associated with the scenario parameter.

[0161] M: A non-transitory computer-readable medium storing instructions executable by a processor, which, when executed, cause the processor to perform operations including one or more of: determining a scenario including scenario parameters describing a portion of an environment; receiving or determining an error model associated with a subsystem of a vehicle; determining a parameterized scenario based at least in part on the scenario, the scenario parameters, and the error model; perturbing the parameterized scenario to result in a perturbed parameterized scenario; receiving simulation data indicative of how the subsystem of the vehicle responds to the perturbed parameterized scenario; and determining a safety metric associated with the subsystem of the vehicle based at least in part on the simulation data.

[0162] N: The non-transitory computer-readable medium of paragraph M, wherein the scenario parameters are associated with at least one of object size, object velocity, object pose, object density, vehicle speed, and vehicle trajectory.

[0163] O: The non-transitory computer-readable medium of paragraph M, wherein determining the scenario includes receiving log data associated with an autonomous vehicle; clustering the log data to determine a first set of clusters, each cluster of the first set of clusters being associated with the scenario; determining a probability associated with the each cluster based at least in part on the first set of clusters; and determining that the probability meets or exceeds a probability threshold.

[0164] P: The non-transitory computer-readable medium of paragraph M, wherein the error is determined at least in part based on receiving ground truth data associated with an environment, determining an error based at least in part on comparing the ground truth data to log data associated with a vehicle, and determining an error distribution based at least in part on the error, and the error model includes the error distribution.

[0165] Q: The non-transitory computer-readable medium of paragraph M, wherein the parameterized scenario is a first parameterized scenario, the perturbed parameterized scenario is a first perturbed parameterized scenario, and the simulation data is first simulation data, and the operations further include determining a second parameterized scenario including at least one of a first subset of the scenario parameters or a second subset of the error model based on the first simulation data, perturbing the second parameterized scenario, receiving second simulation data, and updating the safety metric based at least in part on the second simulation data.

[0166] R: The non-transitory computer-readable medium of paragraph Q, wherein the operations further include: invalidating at least a first portion of one of the scenario parameters or the error model; and associating the second simulation data with at least a second portion of one of the scenario parameters or the error model that has not been invalidated.

[0167] S: The non-transitory computer-readable medium of paragraph M, wherein the safety metric indicates a probability of meeting or exceeding a cost threshold.

[0168] T: The non-transitory computer-readable medium of paragraph M, wherein the error model is associated with one or more of a perception system of the vehicle, a prediction system of the vehicle, or a planner system of the vehicle.

[0169] U: A system having one or more processors and one or more computer-readable media having computer-executable instructions stored thereon, the computer-executable instructions, when executed, cause the system to perform operations including: receiving vehicle data; inputting at least a first portion of the vehicle data to a subsystem of an autonomous vehicle, the subsystem being associated with at least one of a perception system, a planning system, a tracking system, or a prediction system; determining an environmental parameter based at least in part on a second portion of the vehicle data; receiving an estimate from the subsystem; receiving ground truth data associated with the subsystem; determining a difference between the estimate and the ground truth data, the difference representing an error associated with the subsystem; and determining a statistical model associated with the subsystem indicating a probability of the error based at least in part on the difference, the probability being associated with the environmental parameter.

[0170] V: The system of paragraph U, wherein the vehicle data includes sensor data from sensors of the autonomous vehicle, the environmental parameters include one or more of a speed of the autonomous vehicle or weather conditions, the subsystem is a perception subsystem, the estimates are one or more of an estimated position, an estimated orientation, or an estimated range of an object represented in the vehicle data, and the ground truth data represents an actual position, actual orientation, or actual range of the object.

[0171] W: The system described in paragraph U, wherein determining the statistical model includes determining a first frequency associated with the environmental parameter and a second frequency associated with the difference based at least in part on the vehicle data, and determining the probability based at least in part on the first frequency and the second frequency.

[0172] X: The system described in paragraph U, wherein the operations further include determining a simulated environmental parameter based at least in part on simulated vehicle data, determining that the simulated environmental parameter corresponds to the environmental parameter, determining a simulated estimate based at least in part on the simulated vehicle data and the subsystem, and perturbing the simulated estimate by modifying a portion of a corresponding simulated scenario based at least in part on the error based at least in part on the probability.

[0173] Y: A method comprising: receiving data associated with a vehicle; determining environmental parameters based at least in part on a first portion of the data; determining output data associated with a system of the vehicle based at least in part on a second portion of the data; receiving ground truth data associated with the system and the data; determining a difference between the output data and the ground truth data, the difference representing an error associated with the system; and determining a statistical model associated with the system indicative of a probability of the error based at least in part on the difference, the probability being associated with the environmental parameters.

[0174] Z: The method of paragraph Y, wherein determining the statistical model includes determining a frequency associated with the error based at least in part on the data.

[0175] AA: The method of paragraph Y, wherein the environmental parameters include one or more of the speed of the vehicle, weather conditions, the geographic location of the vehicle, or time of day.

[0176] AB: The method of paragraph Y, further comprising: generating a simulation; determining that simulated environmental parameters of the simulation correspond to the environmental parameters; inputting simulated data to the system; receiving simulated outputs from the system; and perturbing the simulation based at least in part on the probabilities and the errors.

[0177] AC: The method of paragraph Y, wherein the system is a perception system, the output data includes a first bounding box associated with an object, and the ground truth data includes a second bounding box associated with the object, and determining the difference includes determining the difference between at least one of a first extent of the first bounding box and a second extent of the second bounding box, or a first pose of the first bounding box and a second pose of the second bounding box.

[0178] AD: The method described in paragraph Y, wherein the system is a tracker system, the output data includes planned trajectory data for the vehicle, and the ground truth data includes a measured trajectory of the vehicle, and determining the difference includes determining the difference between the planned trajectory data and the measured trajectory.

[0179] AE: The method of paragraph Y, wherein the system is associated with a prediction system, the data includes a predicted trajectory of an object in an environment, the ground truth data includes an observed trajectory of the object, and determining the difference includes determining the difference between the predicted trajectory and the observed trajectory.

[0180] AF: The method of paragraph Y, wherein the data is first data, the environmental parameter is a first environmental parameter, the difference is a first difference, the error is a first error, and the probability is a first probability, the method further including: receiving second data associated with the system of the vehicle; determining a second environmental parameter based at least in part on the second data; determining a second difference between a third portion of the output data and a fourth portion of the ground truth data, the second difference representing a second error associated with the system; and updating the statistical model associated with the system, the statistical model indicating a second probability of the second error, the second probability being associated with the second environmental parameter.

[0181] AG: A non-transitory computer readable medium storing instructions executable by the processor which, when executed, cause the processor to perform operations including receiving data; determining environmental parameters based at least in part on the data; determining output data based at least in part on the data and a system of a vehicle; receiving ground truth data associated with the system and the data; determining a difference between a first portion of the output data and a second portion of the ground truth data, the difference representing an error associated with the system; determining a statistical model associated with the system indicative of a probability of the error based at least in part on the difference; and associating the probability with the environmental parameters.

[0182] AH: The non-transitory computer-readable medium of paragraph AG, wherein determining the statistical model includes determining a frequency associated with the difference based at least in part on the data.

[0183] AI: The non-transitory computer-readable medium of paragraph AG, wherein the environmental parameters include one or more of the vehicle's speed, weather conditions, or time of day.

[0184] AJ: The non-transitory computer readable medium of paragraph AG, wherein the operations further include generating a simulation including a simulated vehicle, receiving simulated data, determining that simulated environmental parameters correspond to the environmental parameters, inputting at least a portion of the simulated data into the system, receiving simulated output data from the system, and modifying the simulated output data based at least in part on requirements and probabilities and errors.

[0185] AK: The non-transitory computer-readable medium of paragraph AG, wherein the system is a perception system, the data includes a first bounding box associated with an object, and the ground truth data includes a second bounding box associated with the object, and determining the difference includes determining the difference between at least one of a first extent of the first bounding box and a second extent of the second bounding box, or a first pose of the first bounding box and a second pose of the second bounding box.

[0186] AL: The non-transitory computer-readable medium of paragraph AG, wherein the system is a tracker system, the data includes planned trajectory data for the vehicle, and the ground truth data includes a measured trajectory of the vehicle, and determining the difference includes determining the difference between the planned trajectory data and the measured trajectory.

[0187] AM: The non-transitory computer-readable medium of paragraph AG, wherein the system is associated with a prediction system, the data includes a predicted trajectory of an object in an environment, the ground truth data includes an observed trajectory of the object, and determining the difference includes determining the difference between the predicted trajectory and the observed trajectory.

[0188] AN: The non-transitory computer-readable medium of paragraph AG, wherein the data is first data, the environmental parameter is a first environmental parameter, the difference is a first difference, the error is a first error, and the probability is a first probability, and the operations further include receiving second data associated with the system of the vehicle, determining a second environmental parameter based at least in part on the second data, determining a second difference between a third portion of the output data and a fourth portion of the ground truth data, the second difference representing a second error associated with the system, and updating the statistical model associated with the system, the statistical model indicating a second probability of the second error, the second probability being associated with the second environmental parameter.

[0189] AO: A system having one or more processors and one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the system to perform operations including receiving vehicle data from a vehicle, the vehicle data associated with a state of an object; receiving ground truth data associated with the object; determining an error based at least in part on the vehicle data and the ground truth data; determining a plurality of parameters based at least in part on the vehicle data; clustering at least a portion of the vehicle data into a plurality of clusters and based at least in part on the plurality of parameters and the error; and determining an error model based at least in part on a portion of the vehicle data associated with a cluster of the plurality of clusters.

[0190] AP: The system of paragraph AO, wherein the vehicle data is based at least in part on sensor data from a sensor associated with the vehicle.

[0191] AQ: The system described in paragraph AO, wherein the plurality of parameters are associated with at least one of weather conditions, a first time of day, a second time of year, a distance to the object, a classification of the object, a size of the object, a speed of the object, a position of the object, or a direction of the object.

[0192] AR: The system described in paragraph AO, wherein the error is a first error, and the operations further include receiving sensory data, determining a second error associated with the sensory data based at least in part on the sensory data and the error model, and controlling the vehicle based at least in part on the sensory data and the second error.

[0193] AS: A method comprising: receiving vehicle data from a vehicle, the vehicle data associated with a state of an object; receiving ground truth data associated with the object based at least in part on the vehicle data; determining an error based at least in part on the vehicle data and the ground truth data; determining a parameter based at least in part on the vehicle data; clustering a portion of the vehicle data as a plurality of clusters and based at least in part on the parameter and the error; and determining an error model based at least in part on a portion of the vehicle data associated with a cluster of the plurality of clusters.

[0194] AT: The method of paragraph AS, wherein the state of the object comprises at least one of a size of the object, a location of the object, a direction of the object, a velocity of the object, or a position of the object.

[0195] AU: The method of paragraph AS, wherein the error model comprises an error distribution, the method comprising determining a frequency of occurrence associated with the error based at least in part on the vehicle data and the ground truth data, and determining the error distribution based at least in part on the frequency of occurrence.

[0196] AV: The method of paragraph AS, further comprising: determining classification data identifying a classification of the object; determining object data identifying object parameters of the object; and determining an error distribution associated with at least one of a first cluster of the plurality of clusters or a second cluster of the plurality of clusters, the first cluster being associated with a classification of the object and the second cluster being associated with an object parameter of the object, wherein the error model comprises the error distribution associated with at least one of a true positive error or a false positive error.

[0197] AW: The method described in paragraph AS, further comprising receiving simulation data associated with a simulated vehicle controller in a simulated environment, determining perturbed simulation data based at least in part on an error model and the simulation data, transmitting the perturbed simulation data to the simulated vehicle controller in the simulated environment, and determining a response indicative of how the simulated vehicle controller responds to the perturbed simulation data based at least in part on the perturbed simulation data.

[0198] AX: the error is a first error, and the simulation data comprises a classification associated with a simulated object represented in the simulated environment; The method of paragraph AW, wherein the perturbed simulation data indicates a second error associated with at least one of the position of the object, the orientation of the object, the range of the object, or the velocity of the object.

[0199] AY: The method of paragraph AS, further comprising determining a first error distribution associated with false negative errors at a first time, determining a second error distribution associated with true positive errors at a second time after the first time, and determining a third error distribution associated with false positive errors at a third time after the second time, wherein the error model comprises the first error distribution, the second error distribution, and the third error distribution.

[0200] AZ: The method of paragraph AS, further comprising determining a cost associated with fitting the vehicle data to the error model, and determining the error model based at least in part on the cost.

[0201] BA: One or more non-transitory computer-readable media storing instructions executable by one or more processors that, when executed, cause one or more processors to perform operations comprising receiving vehicle data from a vehicle, the vehicle data associated with a state of an object; receiving ground truth data associated with the object; determining an error based at least in part on the vehicle data and the ground truth data; determining a parameter based at least in part on the vehicle data; clustering a portion of the vehicle data as a plurality of clusters and based at least in part on the parameter; and determining an error model based at least in part on a portion of the vehicle data associated with a cluster of the plurality of clusters.

[0202] BB: The one or more non-transitory computer-readable media of paragraph BA, wherein the state of the object comprises at least one of a size of the object, a location of the object, a direction of the object, a velocity of the object, or a position of the object.

[0203] BC: The one or more non-transitory computer-readable media of paragraph BA, wherein the error model comprises an error distribution, and the operations comprise determining a frequency of occurrence associated with the error based at least in part on the vehicle data and the ground truth data, and determining the error distribution based at least in part on the frequency of occurrence.

[0204] BD: The one or more non-transitory computer-readable media of paragraph BA, wherein the operations further comprise determining classification data identifying a classification of the object; determining object data identifying object parameters of the object; and determining an error distribution associated with at least one of a first cluster of the plurality of clusters or a second cluster of the plurality of clusters, the first cluster being associated with a classification of the object and the second cluster being associated with an object parameter of the object, wherein the error model comprises the error distribution associated with at least one of a true positive error or a false positive error.

[0205] BE: The one or more non-transitory computer-readable media of paragraph BA, wherein the error model is one of a plurality of error models, and the operations further comprise receiving simulation data associated with a simulated vehicle controller in a simulated environment, determining perturbed simulation data based at least in part on the error model and the simulation data, transmitting the perturbed simulation data to the simulated vehicle controller in the simulated environment, and determining a response indicative of how the simulated vehicle controller responds to the perturbed simulation data, based at least in part on the perturbed simulation data.

[0206] BF: the error is a first error, and the simulation data comprises a classification associated with a simulated object represented in the simulated environment; The one or more non-transitory computer-readable media of paragraph BE, wherein the perturbed simulation data indicates a second error associated with at least one of the position of the object, the orientation of the object, the range of the object, or the velocity of the object.

[0207] BG: The one or more non-transitory computer-readable media of paragraph BA, further comprising: determining a first error distribution associated with false negative errors at a first time; determining a second error distribution associated with true positive errors at a second time after the first time; and determining a third error distribution associated with false positive errors at a third time after the second time, wherein the error model comprises the first error distribution, the second error distribution, and the third error distribution.

[0208] BH: The one or more non-transitory computer-readable media of paragraph BA, wherein the operations further include determining a cost associated with fitting the vehicle data to the error model and determining the error model based at least in part on the cost.

[0209] While the example clauses described above are described with respect to one particular implementation, it should be understood that in the context of this specification, the content of the example clauses may also be implemented via a method, device, system, computer-readable medium, and / or another implementation. Furthermore, any of Examples A-BH may be implemented alone or in combination with any other one or more of Examples A-BH.

[0210] (Conclusion) Although one or more examples of the technology described herein have been described, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0211] In the illustrative description, reference is made to the accompanying drawings, which form a part hereof, showing by way of example specific examples of the claimed subject matter. It is to be understood that other examples may be used and modifications or substitutions, such as structural changes, may be made. Such examples, modifications or substitutions do not necessarily depart from the intended scope of the claimed subject matter. Although the steps herein may be presented in a particular order, in some cases the order may be changed, such as providing certain inputs at different times or in a different order, without changing the functionality of the described systems and methods. The disclosed procedures may also be performed in different orders. Furthermore, the various calculations herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be readily implemented. In addition to reordering, calculations may also be decomposed into sub-calculations that produce the same results.

Claims

1. A method executed by one or more processors of a computer, the method comprising: receiving vehicle data from a vehicle, the vehicle data being associated with a state of an object; receiving ground truth data associated with the object based at least in part on the vehicle data; and determining an error based at least in part on the vehicle data and the ground truth data; and determining a parameter based at least in part on the vehicle data; and clustering a portion of the vehicle data as a plurality of clusters and based at least in part on the parameters and the error; determining an error model based at least in part on a portion of the vehicle data associated with a cluster of the plurality of clusters, the error model comprising an error distribution; and determining a frequency of occurrence associated with the error based at least in part on the vehicle data and the ground truth data; determining the error distribution based at least in part on the frequency of occurrence; and A method comprising:

2. The method of claim 1 , wherein the vehicle data is based at least in part on sensor data from a sensor associated with the vehicle.

3. The method of claim 1 or claim 2, wherein the parameter is associated with at least one of weather conditions, a first time of day, a second time period, a distance to the object, a classification of the object, a size of the object, a speed of the object, a position of the object, or a direction of the object.

4. The error is a first error, and the method includes: Receiving sensory data; determining a second error associated with the sensory data based at least in part on the sensory data and the error model; and controlling the vehicle based at least in part on the sensory data and the second error; and The method of claim 1 , further comprising:

5. The method of claim 1 , wherein the object state comprises at least one of the following: a size of the object, a location of the object, a direction of the object, a velocity of the object, or a position of the object.

6. A method executed by one or more processors of a computer, the method comprising: receiving vehicle data from a vehicle, the vehicle data being associated with a state of an object; receiving ground truth data associated with the object based at least in part on the vehicle data; and determining an error based at least in part on the vehicle data and the ground truth data; and determining a parameter based at least in part on the vehicle data; and clustering a portion of the vehicle data as a plurality of clusters and based at least in part on the parameters and the error; determining an error model based at least in part on a portion of the vehicle data associated with a cluster of the plurality of clusters; determining classification data identifying a classification of the object; determining object data identifying object parameters of the object; determining an error distribution associated with at least one of a first cluster of the plurality of clusters or a second cluster of the plurality of clusters, the first cluster being associated with a classification of the object and the second cluster being associated with an object parameter of the object; Equipped with The method, wherein the error model comprises the error distribution associated with at least one of a true positive error or a false positive error.

7. receiving simulation data associated with a simulated vehicle controller in a simulated environment; determining perturbed simulation data based at least in part on the error model and the simulation data; transmitting the perturbed simulation data to the simulated vehicle controller in the simulated environment; determining a response indicative of how the simulated vehicle controller will respond to the perturbed simulation data based at least in part on the perturbed simulation data; The method of claim 1 , further comprising:

8. the error is a first error, the simulation data comprises classifications associated with simulated objects represented in the simulated environment; the perturbed simulation data indicates a second error associated with at least one of the position of the object, the orientation of the object, the range of the object, or the velocity of the object.

8. The method according to any one of claims 1 to 7.

9. A method executed by one or more processors of a computer, the method comprising: receiving vehicle data from a vehicle, the vehicle data being associated with a state of an object; receiving ground truth data associated with the object based at least in part on the vehicle data; and determining an error based at least in part on the vehicle data and the ground truth data; and determining a parameter based at least in part on the vehicle data; and clustering a portion of the vehicle data as a plurality of clusters and based at least in part on the parameters and the error; determining an error model based at least in part on a portion of the vehicle data associated with a cluster of the plurality of clusters; determining a first error distribution associated with false negative errors at a first time; determining a second error distribution associated with true positive errors at a second time after the first time; determining a third error distribution associated with false positive errors at a third time after the second time; and Equipped with The method, wherein the error model comprises the first error distribution, the second error distribution, and the third error distribution.

10. A method executed by one or more processors of a computer, the method comprising: receiving vehicle data from a vehicle, the vehicle data being associated with a state of an object; receiving ground truth data associated with the object based at least in part on the vehicle data; and determining an error based at least in part on the vehicle data and the ground truth data; and determining a parameter based at least in part on the vehicle data; and clustering a portion of the vehicle data as a plurality of clusters and based at least in part on the parameters and the error; determining an error model based at least in part on a portion of the vehicle data associated with a cluster of the plurality of clusters; determining a cost associated with fitting the vehicle data to the error model; and determining the error model based at least in part on the cost; A method comprising:

11. A computer program product comprising coded instructions which, when executed on a computer, cause the computer to carry out the method of any one of claims 1 to 10.

12. one or more processors; one or more non-transitory computer-readable media storing instructions executable by the one or more processors; the instructions, when executed, receiving vehicle data from a vehicle, the vehicle data being associated with a state of an object; receiving ground truth data associated with the object; determining an error based at least in part on the vehicle data and the ground truth data; and determining a parameter based at least in part on the vehicle data; and clustering the portion of the vehicle data as a plurality of clusters and based at least in part on the parameters; determining an error model based at least in part on a portion of the vehicle data associated with a cluster of the plurality of clusters; causing the system to perform operations including The error model comprises an error distribution, and the operation comprises: determining a frequency of occurrence associated with the error based at least in part on the vehicle data and the ground truth data; determining the error distribution based at least in part on the frequency of occurrence; and The system further comprises:

13. one or more processors; one or more non-transitory computer-readable media storing instructions executable by the one or more processors; the instructions, when executed, receiving vehicle data from a vehicle, the vehicle data being associated with a state of an object; receiving ground truth data associated with the object; determining an error based at least in part on the vehicle data and the ground truth data; and determining a parameter based at least in part on the vehicle data; and clustering the portion of the vehicle data as a plurality of clusters and based at least in part on the parameters; determining an error model based at least in part on a portion of the vehicle data associated with a cluster of the plurality of clusters; determining classification data identifying a classification of the object; determining object data identifying object parameters of the object; determining an error distribution associated with at least one of a first cluster of the plurality of clusters or a second cluster of the plurality of clusters, the first cluster being associated with a classification of the object and the second cluster being associated with an object parameter of the object; causing the system to perform operations including The error model comprises an error distribution associated with at least one of a true positive error or a false positive error.

Images