Agent Behavior Model for Simulation Control

Abstract

Producing a realistic motion simulation that accounts for anomalous behavior of an object, such as a vehicle or pedestrian, can include generating an agent behavior model based at least in part on output of a perception component of the autonomous vehicle and determining differences between the output and log data that includes an indication of actual manipulation of the object's position. Producing a motion simulation of the object can include determining a predicted motion of the object using a prediction component and modifying the predicted motion based at least in part on the agent behavior model.

Description

[Technical Field]

[0001] This application claims the benefit of U.S. patent application Ser. No. 16 / 889,747, filed June 1, 2020, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Simulated execution of scenarios can provide a valuable method for testing pipelines of machine learning models that will be incorporated into autonomous driving systems and / or autonomous vehicles. However, because human behavior is unpredictable, accurately identifying and modeling rare and unusual behaviors can be challenging. For example, a human driver may suddenly veer off a highway onto an exit, make a turn in front of oncoming traffic, or run a red light at an intersection where such a turn is illegal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent Application No. 15 / 963,833 [Patent Document 2] U.S. Patent Application Serial No. 16 / 779,576 [Patent Document 3] U.S. Patent Application Serial No. 16 / 581,632 [Patent Document 4] U.S. Patent Application Serial No. 16 / 682,971 [Patent Document 5] U.S. Patent Application No. 16 / 703,625 [Patent Document 6] U.S. Patent Application Serial No. 16 / 517,506 [Brief explanation of the drawings]

[0004] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. The same reference number in different figures may refer to similar or identical items. [Figure 1] FIG. 10 illustrates an example scenario in which an autonomous vehicle may be controlled based at least in part on sensor data collected by one or more sensors associated with the autonomous vehicle and based at least in part on predictive data generated by a perception component. [Figure 2] FIG. 1 is a block diagram illustrating an example architecture for generating agent behavior models for use in a simulation system. [Figure 3] 1 is a pictorial flowchart of an example process for generating an error model that may be part of an agent behavior model for realistically simulating dynamic objects; [Figure 4A] 10A and 10B are diagrams showing examples of predicted operations and examples of actual paths / actual operations taken by objects that were the subject of predictions. [Figure 4B] FIG. 1 illustrates example scenarios and scenario labels that can be associated with different error models. [Figure 4C] FIG. 1 illustrates example scenarios and scenario labels that can be associated with different error models. [Figure 5A] FIG. 10 illustrates how error models associated with an object and / or scenario may vary based at least in part on the object's location in the environment / actions taken by the object. [Figure 5B] FIG. 10 illustrates how error models associated with an object and / or scenario may vary based at least in part on the object's location in the environment / actions taken by the object. [Figure 5C] FIG. 10 illustrates how error models associated with an object and / or scenario may vary based at least in part on the object's location in the environment / actions taken by the object. [Figure 6]1 is a flowchart illustrating an example process for realistically simulating the behavior of dynamic objects in a simulated environment, including anomalous behavior. Some techniques can include running multiple simulations of a scenario and causing the dynamic objects to perform anomalous behaviors with a rate of occurrence determined by an error model. DETAILED DESCRIPTION OF THE INVENTION

[0005] Techniques for accurately simulating the behavior of objects such as vehicles, pedestrians, cyclists, or animals can include generating an agent behavior model capable of determining instructions for controlling a simulated object, such as a simulated representation of a dynamic object (i.e., a maneuverable object such as a vehicle, pedestrian, cyclist, or animal) in a simulated environment. The agent behavior model may be designed to mimic how dynamic objects sometimes exhibit anomalous behavior. For example, the agent behavior model may cause the simulated vehicle to turn in front of oncoming traffic, swerve suddenly, cross lanes, perform illegal maneuvers, operate at illegal speeds, and / or similar frequencies that would occur in real-world scenarios. Simulating this anomalous behavior using the agent behavior model can improve testing of an automated vehicle's response to objects in its environment.

[0006] An autonomous vehicle may include a perception component that receives sensor data from one or more sensors associated with the autonomous vehicle. The perception component may use the sensor data to determine perception data that describes what is in the environment and predicts what the state of the environment will be in the future. For example, the perception data may describe the presence of objects in the environment, their location and classification (e.g., vehicles, pedestrians, signs, static objects, cyclists), and prediction data related to the objects that describes the predicted position, heading, and / or speed of the objects at a future time, and / or maneuvers the objects can take. The predicted maneuvers may be high-level descriptions of actions to be performed by the objects, such as maintaining their state (e.g., maintaining a stopped position, staying on track), turning left, stopping, or entering or exiting a roadway, crosswalk, sidewalk, or segregated lane (e.g., bike lane, highway), and / or the like.

[0007] The generation of the agent behavior model may include receiving perceptual data and log data associated with the scenario from the autonomous vehicle exploring through the scenario. The log data may comprise sensor data and perceptual data collected / generated while the autonomous vehicle is driving through the scenario. In some examples, the log data may further include labels added by the autonomous vehicle after transmitting the perceptual data and / or sensor data to a remote computing device. The labels may indicate ground truth about objects appearing in the sensor data.

[0008] The technique may comprise determining differences between the perception data and the log data and constructing an error model based on the determined differences associated with multiple identical or similar scenarios. For example, the technique may comprise processing operations including identifying different scenarios in the log data and / or its associated perception data. The scenario may comprise the layout of the environment (e.g., a four-way controlled intersection, a three-way uncontrolled intersection, a six-way uncontrolled intersection, a two-lane highway, one-way traffic direction), the number, type, and / or configuration of objects within the scenario. The configuration may identify the object's location, direction of travel, and / or trajectory / speed. The technique may comprise aggregating log data and perception data associated with the same scenario from log data and perception data received from one or more automated vehicles over a data collection period. The error model may be constructed progressively as new log data and / or perception data associated with the scenario is added to the aggregation (or for all aggregated data at once). In some examples, the error model may be determined for each scenario, each type of object, and / or each location / direction / state of an object within the scenario.

[0009] The technique may comprise constructing different types of error models. In at least one example, the technique may comprise determining a maneuver error model and a path error model. The maneuver error model may be generated based at least in part on a difference between a maneuver predicted by a perception component of the automated vehicle and an actual maneuver performed by the object indicated by the log data. The path error model may be generated based at least in part on a difference between a path predicted by the perception component and an actual path followed by the object (e.g., the Euclidean distance between the predicted and actual positions). The path may be a sequence of predicted positions and / or trajectories followed by the object. In either case, the error model may comprise a probability distribution associated with different maneuvers / paths. For example, the maneuver error model may indicate a probability distribution of a vehicle remaining centered at an intersection and maintaining that state in a first scenario in which the vehicle is centered at an intersection and maneuvers straight through the intersection. The maneuver error model may indicate a probability of the vehicle turning left, turning right, making an abrupt stop, or continuing its trajectory based on the scenario, the type of object, the position / direction / speed of the object in the environment, and / or the state of the vehicle (e.g., performing a turn, stopping). For example, techniques may comprise determining the number of times an object performs an operation and / or the number of times an operation performed by an object matches a predicted operation as observed across multiple instances.

[0010] The error model generated for an object type (e.g., vehicle, pedestrian) across different scenarios can be associated with the object as an agent behavior model for that object type. This agent behavior model may be recalled from memory when an object of that type is instantiated in the simulation. For example, the technique may comprise receiving a scenario, where the scenario may be defined based at least in part on map data, log data received from the autonomous vehicle, and / or based at least in part on input received at a computing device (e.g., an individual may interact with a user interface of the computing device to define the scenario). In some examples, the scenario may indicate characteristics of the environment, such as, for example, position, orientation, manipulation, and / or characteristics of one or more objects (e.g., material, height, width, depth, brightness). For example, the environment may include one or more static objects and / or one or more agents (e.g., dynamic objects) in a configuration specified by the scenario being simulated. In some examples, any of the dynamic objects instantiated in the scenario may be controlled based at least in part on the output of the agent behavior model.

[0011] In some examples, controlling the behavior of the simulated object using an agent in a simulation may comprise determining predictive data associated with the object based at least in part on simulated sensor data received by a perception component of the automated vehicle. The perception component may be a replica or virtualization of a perception component running on the automated vehicle. The perception component outputs predictive data associated with the simulated object, which may be received by an agent behavior model. In some examples, the predictive data may include at least a predicted maneuver and / or a predicted path of the simulated object. The agent behavior model may modify the predicted maneuver and / or the predicted path based at least in part on an error model associated with the simulated object.

[0012] For example, modifying the predicted maneuver and / or predicted path may include sampling an error model and modifying the predicted maneuver and / or predicted path based at least in part on a probability associated therewith. Techniques may include controlling movement of a simulated object based at least in part on the modified maneuver and / or modified path. Modifying the maneuver may result in a change in the maneuver, and modifying the path may include changing parameters of a curve defining the path or increasing variance associated with the location of discrete positions from a reference curve (e.g., the reference curve may be defined by the predicted maneuver). For example, a simulated object may be predicted to continue straight and only maneuver slightly laterally from a straight path defined by a "straight line" maneuver, but sampling the error model may result in a low probability event in which the simulated object performs a sharp right turn while maintaining the same speed and experiencing a large change in lateral displacement from the path associated with the sharp right turn. In some examples, the action model may indicate probabilities associated with actions, e.g., the action model may indicate different possible actions and different probabilities associated with the different actions that indicate the probability that each action will be performed by the object (and may be based on errors in the perception system, as described further herein). For example, the action model may indicate different actions that an object can perform and their associated probabilities, which sum to 1. Other ways of proportionalizing the probabilities are possible, such as percentages (which may or may not sum to 100% across all possible next actions).

[0013] In an additional or alternative example, the simulation may run a simulation of a scenario multiple times, and the technique may include tracking the rate of occurrence of a maneuver and / or path associated with the object. In such a case, modifying the predicted maneuver and / or predicted path may include modifying the maneuver and / or predicted path to conform to a rate of occurrence determined at least in part based on an error model associated with the object and / or scenario. For example, the maneuver error model may indicate that once every 1000 iterations of a scenario in which the vehicle traverses an intersection and maneuvers, the vehicle will make a sharp turn in front of an oncoming vehicle. The agent behavior model may enforce the rate of occurrence such that once every 1000 iterations, the predicted maneuver is modified to include a sharp turn in front of an oncoming vehicle.

[0014] The techniques described herein can be used in operational testing of an autonomous vehicle (e.g., a planning component of an autonomous vehicle), which can generate trajectories for controlling the operation of the autonomous vehicle based at least in part on simulated sensor data associated with simulated movements of an agent. Perception data generated by the perception component can also be provided to the planning component for testing the planning component.

[0015] The techniques can improve the testing and / or training of one or more components of an autonomous vehicle (e.g., a localization component, a perception component, a planning component), thereby improving the accuracy and safety and effectiveness of the autonomous vehicle's operation (or other systems that integrate perception components and / or planning components, etc.). Additionally, the techniques can reduce the amount of computing resources required to accurately simulate a scenario, effectively using the autonomous vehicle's perception components to control the simulation and, as modified by the agent behavior model, may require significantly less computing power than the perception components.

[0016] Example Scenario FIG. 1 illustrates an example scenario 100 including a vehicle 102. In some examples, the example scenario 100 may be a real-world scenario, and / or the example scenario 100 may be a representation of a real-world scenario modeled as a simulated scenario. In examples where the example scenario 100 is a simulated scenario, the example scenario 100 may be determined based at least in part on input received at a user interface of a computing device (e.g., a user of the computing device may define an environment, objects therein, and / or their characteristics), and / or the example scenario 100 may be based at least in part on log data received from one or more autonomous vehicles. The log data may be based at least in part on sensor data received at the autonomous vehicle, sensory data generated by a perception component, and / or instructions generated by a planning component. In some examples, the autonomous vehicle may store the log data and / or periodically transmit the log data to a remote computing device.

[0017] In some cases, the vehicle 102 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the U.S. National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire journey without a driver (or passenger) being expected to control the vehicle at any time. However, in other examples, the vehicle 102 may be a fully or partially autonomous vehicle having other levels or classifications. It is contemplated that the techniques described herein may be applicable to applications other than robotic control, such as autonomous vehicles. For example, the techniques described herein may be applicable to mining, manufacturing, augmented reality, and the like. Furthermore, while the vehicle 102 is depicted as a land vehicle, the vehicle 102 may also be a spacecraft, a watercraft, and the like. In some examples, the vehicle 102 may be represented in a simulation as a simulated vehicle. For simplicity, the description herein does not distinguish between simulated and real-world vehicles. Accordingly, references to a "vehicle" may refer to a simulated vehicle and / or a real-world vehicle.

[0018] According to an example in which the techniques and example scenario 100 described herein are real-world examples, a vehicle 102 may receive sensor data from sensors 104 of the vehicle 102. For example, the sensors 104 may include position sensors (e.g., global positioning system (GPS) sensors), inertial sensors (e.g., acceleration sensors, gyro sensors, etc.), magnetic field sensors (e.g., compasses), position / speed / acceleration sensors (e.g., speedometers, drive system sensors), depth position sensors (e.g., lidar sensors, radar sensors, sonar sensors, time-of-flight (ToF) cameras, depth cameras, and / or other depth-sensing sensors), image sensors (e.g., cameras), audio sensors (e.g., microphones), and / or environmental sensors (barometers, hygrometers, etc.). In some examples, the simulated sensors may correspond to at least one of the sensors 104 on the vehicle 102, and in the simulation, one or more of the sensors 104 may be simulated. In some examples, the simulated sensor position may correspond to the relative position of one of the sensors 104 with respect to the vehicle 102 .

[0019] The sensors 104 may generate sensor data, which may be received by a computing device 106 associated with the vehicle 102. However, in other examples, some or all of the sensors 104 and / or computing devices 106 may be separate and / or located remotely from the vehicle 102, and data acquisition, processing, instruction, and / or control may be communicated to or from the vehicle 102 by one or more remote computing devices via wired and / or wireless networks.

[0020] The computing device 106 may include a memory 108 that stores a localization component 110, a perception component 112, a planning component 114, and / or a simulation system 116. In general, the computing device 106 may include software and / or hardware systems for determining the attitude (e.g., position and / or orientation) of the vehicle 102 relative to one or more coordinate frames (e.g., relative to the environment, relative to the roadway, relative to an inertial direction of motion associated with the autonomous vehicle). The localization component 110 may output at least a portion of this data to the perception component 112, which may output at least a portion of the localization data and / or use the localization data as a reference for determining at least a portion of the perception data.

[0021] The perception component 112 can determine what is in the environment surrounding the vehicle 102, and the planning component 114 can determine how to operate the vehicle 102 according to information received from the localization component 110 and / or the perception component 112. The localization component 110, the perception component 112, and / or the planning component can include one or more machine learning (ML) models and / or other computer-executable instructions.

[0022] In some examples, the localization component 110 and / or the perception component 112 can receive sensor data from the sensors 104 and / or simulated data from the simulation system 116. Techniques described herein can comprise adding simulation data received from the simulation component 116 to the perception data output by the perception component 112, thereby improving online predictions of possible agent behaviors. For example, the simulation system 116 can generate a modified maneuver and / or a modified path using an agent behavior model described herein and, in at least one example, provide the modified maneuver and / or the modified path to the planning component 114 and / or the perception component 112.

[0023] In some cases, the perception component 112 may determine data related to objects in the vehicle's 102's surroundings (e.g., classifications, instance classifications, and tracks associated with detected objects), route data specifying the vehicle's destination, global map data identifying road characteristics (e.g., features detectable by various sensor aspects useful for locating the autonomous vehicle), local map data identifying detected characteristics (e.g., locations and / or dimensions of buildings, trees, fences, fire hydrants, stop signs, and other features detectable by various sensor aspects), etc. In some examples, the objects surrounding the vehicle 102 may be simulated objects in a simulated environment. The data generated by the perception component 112 may be collectively referred to as “perception data.” Once the perception component 112 generates the perception data, the perception component 112 may provide the perception data to the planning component 114.

[0024] The planning component 112 can determine the trajectory 118 based at least in part on sensory data, such as simulation data (e.g., modified maneuvers and / or paths) and / or localization data (e.g., where the vehicle 102 is in an environment related to a map and / or features detected by the localization component 110), and / or other information, for example. For example, the planning component 114 can determine a path for the vehicle 102 from a first position to a second position, and substantially simultaneously generate multiple potential trajectories for controlling the movement of the vehicle 102 according to a receding horizon technique (e.g., 1 microsecond, 0.5 seconds) based at least in part on the sensory data and / or simulated sensory data, control the vehicle to traverse a path (e.g., to avoid a detected object), and select one of the potential trajectories as a trajectory 118 that the vehicle 102 can use to generate drive control signals that can be sent to drive components of the vehicle 102. 1 shows an example of such a trajectory 118, represented as arrows indicating heading, speed, and / or acceleration, although the trajectory itself may include instructions for a PID controller, which may in turn actuate the drive systems of the vehicle 102. For example, the trajectory 118 may include instructions for a controller of the autonomous vehicle 102 to actuate the drive components of the vehicle 102 to achieve a steering angle and / or steering rate that may result in a vehicle position, vehicle velocity, and / or vehicle acceleration (or a simulated version thereof if the autonomous vehicle is simulated). The trajectory 118 may comprise a target heading, target steering angle, target steering rate, target position, target speed, and / or target acceleration that the controller tracks.

[0025] The example scenario 100 includes a pedestrian 120 walking on a sidewalk next to a roadway on which a vehicle 102 is operating. A perception component 112 receives sensor data related to the environment and can determine the location of the pedestrian 120 within the environment and a classification associated with the pedestrian 120 (i.e., "pedestrian"). The perception data generated by the perception component 112 can be stored in a log data store that can comprise the position of the pedestrian 120 at current and previous time steps. For example, FIG. 1 illustrates a current position 122 of the pedestrian 120 at time t, and the pedestrian's previous (historical) positions, i.e., at time t -1 at position 124 and time t -2 1 shows a position 126 in a time step. Each time step may correspond to a time interval (e.g., 100 milliseconds, 500 milliseconds, 1 second, or other time). In at least one example, the positions 122-126 and object classifications may be associated as part of the tracking generated by the perception component 112.

[0026] In example scenario 100, perception component 112 can determine a predicted maneuver 128 and / or a path for pedestrian 120 based at least in part on position 122, historical positions 124 and / or 126, object classification, and / or other data associated with pedestrian 120. FIG. 1 illustrates the path as discrete positions, including predicted position 130 at time t1 and predicted position 132 at time t2. Current and past positions are indicated by filled circles, and predicted positions are indicated by unfilled circles. Note that while the path is illustrated as comprising discrete positions, prediction component 112 can determine the path as a line or curve, and / or as a variation or parameter from a standard reference line or curve associated with the maneuver. For example, the path can define the curvature parameters of a turn maneuver or a variation from a curve defined by the maneuver.

[0027] In some examples, the predictive data may additionally or alternatively be based at least in part on map data or other data. In some examples, the predictive data may comprise a top-down segmentation of the environment, as described in more detail in U.S. Patent Application Publication No. 2009 / 0109994, the entirety of which is incorporated herein, and / or a top-down prediction related to the environment, as described in more detail in U.S. Patent Application Publication No. 2009 / 0109994, the entirety of which is incorporated herein. For example, in the illustrated example, the predicted position of the pedestrian map is based at least in part on the position of the pedestrian 120 relative to a crosswalk in the environment. The map data is stored by the memory 108. In some examples, the predictive data and / or other sensory data may be stored as part of logged data. The logged data may include sensor data and / or sensory data associated with future times t1 and t2, respectively, collected and generated by the vehicle 102 as the vehicle 102 continues to operate. Note that at the next time step, the vehicle 102 generates a new prediction. In other words, the predicted maneuver and position are based on the predicted time t -2 , and the current time may correspond to time t--2.

[0028] In some examples, the scenario label 134 may be associated with sensor data and / or sensory data stored as part of the log data. In some examples, the scenario label 134 may be added by a different computing device (e.g., a remote computing device after the log data is transmitted by the vehicle 102). For example, the scenario label may characterize the number, type, or configuration of objects in the environment and / or the layout of the environment. In some examples, the scenario label may be a data structure. The configuration may define the position, direction of travel, and / or speed of objects in the environment, and the layout of the environment may identify general characteristics of the environment (e.g., a four-way light-controlled intersection, a four-way light-controlled intersection with an uncontrolled left-turn lane, an uncontrolled three-way intersection with a one-way road). In some examples, the scenario label may be generated as part of a log data processing operation. The processing operation may comprise determining sensor data associated with sensory data that identifies the relative positions and / or types of objects. For example, the processing operation may identify the number and / or type of objects in the following areas and associate this data with the respective sensor data: The resulting metadata annotations may be clustered and a scenario label may be associated with each different cluster, which may be based on the number and / or type of objects and / or regions. Regions may include, for example: In front of vehicle 102, to the side of the vehicle 102, - rear of vehicle 102, - Within the lane from vehicle 102's lane, In the next lane away from vehicle 102, Within or near various intersection / junction configurations (e.g., 4-way controlled intersections, within T-junctions, 4-way controlled, 2-controlled, 2-uncontrolled intersections), Next to the road, etc. In some examples, the scenario labels may not be human-intelligible (the scenario labels may simply be cluster identifiers), while in other examples, human-intelligible terms may be attached to the clusters (e.g., two objects on the sides of the vehicle, one object in the front of the vehicle), learned insertions, etc.

[0029] The vehicle 102 may transmit at least a portion of the sensor data, perception data, and / or scenario labels 134 (if generated on-board the vehicle by the perception component 112) to a remote computing device (not shown in FIG. 1) as part of the log data.

[0030] System example FIG. 2 shows a block diagram of an example system 200 that implements the techniques described herein. In some cases, the example system 200 may include a vehicle 202 and may represent the vehicle 102 of FIG. 1. In some cases, the vehicle 202 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle that can perform all safety-critical functions for the entire trip, with no driver (or passenger) expected to control the vehicle at any time. However, in other examples, the vehicle 202 may be a fully or partially autonomous vehicle having other levels or classifications. Additionally, in some cases, the techniques described herein may also be used in non-autonomous vehicles.

[0031] Vehicle 202 may include vehicle computing device 204, sensors 206, emitters 208, network interfaces 210, and / or drive components 212. Vehicle computing device 204 may represent computing device 106, and sensors 206 may represent sensors 104. System 200 may additionally or alternatively include computing device 214.

[0032] In some cases, sensors 206 may represent sensors 104 and may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., global positioning system (GPS), compasses, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), image sensors (e.g., red-green-blue (RGB), infrared (IR), intensity, depth, time-of-flight cameras, etc.), microphones, wheel encoders, environmental sensors (thermometers, hygrometers, light sensors, pressure sensors, etc.), and others. Sensors 206 may include multiple instances of each of these or other types of sensors. For example, radar sensors may include individual radar sensors located at the corners, front, rear, sides, and / or top of vehicle 202. As another example, cameras may include multiple cameras positioned at various locations on the exterior and / or interior of vehicle 202. Sensors 206 may provide input to vehicle computing device 204 and / or computing device 214. In some examples, the simulated sensors described herein may correspond to a particular type of sensor and / or any sensor. The location associated with the simulated sensor may correspond to the position and / or origin (e.g., focal point) of the sensor's field of view relative to the vehicle 202 and / or the direction of motion of the vehicle 202, as described herein.

[0033] Vehicle 202 may also include emitters 208 for emitting light and / or sound, as described above. Emitters 208 in this example may include internal audio and visual emitters for communicating with passengers of vehicle 202. By way of non-limiting example, internal emitters may include speakers, lights, signs, display screens, touchscreens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (seatbelt tensioners, seat positioners, headrest positioners, etc.), and others. Emitters 208 in this example may also include external emitters. By way of non-limiting example, external emitters in this example may include lights to signal direction of travel or other directional indicators of vehicle movement (e.g., turn signal lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) may be used to audibly communicate with pedestrians and other nearby vehicles, one or more of which may include acoustic beam steering technology.

[0034] Vehicle 202 may include a network interface 210 that enables communication between vehicle 202 and one or more other local or remote computing devices. For example, network interface 210 may facilitate communication with other local computing devices on vehicle 202 and / or drive component 212. Network interface 210 may also additionally or alternatively enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Network interface 210 may also additionally or alternatively enable vehicle 202 to communicate with computing device 214. In some examples, computing device 214 may comprise one or more nodes of a distributed computing system (e.g., a cloud computing architecture).

[0035] Network interface 210 may include physical and / or logical interfaces for connecting vehicle computing device 204 to another computing device or network, such as network 216. For example, network interface 210 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 200.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices. In some cases, vehicle computing device 204 and / or sensors 206 may transmit sensor data to computing device 214 over network 216 at a particular frequency, after a period of time, in near real time, etc.

[0036] In some cases, the vehicle 202 may include one or more drive components 212. In some cases, the vehicle 202 may have a single drive component 212. In some cases, the drive component 212 may include one or more sensors for detecting the surroundings of the drive component 212 and / or the vehicle 202. By way of non-limiting example, the sensors of the drive component 212 may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive component, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the heading and acceleration of the drive component, cameras or other image sensors, ultrasonic sensors for acoustically detecting objects in the surroundings of the drive component, lidar sensors, radar sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive component 212. In some cases, the sensors on the drive component 212 may overlap or supplement corresponding systems (e.g., sensors 206) of the vehicle 202.

[0037] The drive components 212 may include a high-voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use in other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, an electronic stability control system for distributing braking force to mitigate losses in propulsion, an HVAC system, lights (e.g., head / tail lights for illuminating the exterior surroundings of the vehicle), and one or more other systems (e.g., a cooling system, a safety system, an on-board charging system, other electrical components such as DC / DC converters, high-voltage junctions, high-voltage cables, a charging system, a charge port, etc.). Additionally, the drive components 212 may include a drive component controller to receive and preprocess data from sensors and control the operation of various vehicle systems. In some cases, the drive component controller may include one or more processors and memory communicatively coupled to the one or more processors. The memory may store one or more components for performing various functions of the drive components 212. Additionally, drive components 212 may also include one or more communication connections that enable the respective drive components to communicate with one or more other local or remote computing devices.

[0038] Vehicle computing device 204 may include a processor 218 and a memory 220 communicatively coupled to the one or more processors 218. Memory 220 may represent memory 108. Computing device 214 may also include a processor 222 and / or a memory 224, where processor 218 and / or 222 may be any suitable processor capable of executing instructions, processing data, and performing the operations described herein. By way of non-limiting example, processor 218 and / or 222 may comprise portions of one or more central processing units (CPUs), graphics processing units (GPUs), integrated circuits (e.g., application specific integrated circuits (ASICs)), gate arrays (e.g., field configurable circuit arrays (FPGAs)), and / or any other device or device that processes electronic data and converts it into other electronic data that can be stored in registers and / or memory.

[0039] Memory 220 and / or 224 may be examples of non-transitory computer-readable media. Memory 220 and / or 224 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functionality attributed to the various systems described herein. In various implementations, memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, and those shown in the accompanying figures are merely examples relevant to the description herein.

[0040] In some cases, memory 220 and / or memory 224 can store localization component 226, perception component 228, planning component 230, log data 232, map 234, and / or system controller 236, any of which may be hardware such as a GPU, CPU, and / or other processing unit. Localization component 226 can represent localization component 110, perception component 228 can represent perception component 112, planning component 230 can represent planning component 114, and simulation system 238 can represent simulation system 116.

[0041] In at least one example, the localization component 226 may include hardware and / or software for receiving data from the sensors 206 to determine the position, velocity, and / or orientation (e.g., one or X-, Y-, Z-position, roll, pitch, or yaw) of the vehicle 202. For example, the localization component 226 may include and / or request / receive a map 234 of the environment and may continuously determine the position, velocity, and / or orientation of the autonomous vehicle within the map 234. In some cases, the localization component 226 may receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, etc., utilizing SLAM (simultaneous localization and mapping), CLAMS (calibration, localization, and mapping, simultaneously), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to accurately determine the position, attitude, and / or velocity of the autonomous vehicle. In some cases, localization component 226 may provide data to various components of vehicle 202 to determine an initial position of the autonomous vehicle for generating a trajectory and / or generating map data, as described herein. In some examples, localization component 226 may provide perception component 228 with the position and / or heading of vehicle 202 relative to the environment and / or sensor data associated therewith.

[0042] In some cases, perception component 228 may comprise a primary perception system and / or a predictive system implemented in hardware and / or software. Perception component 228 may detect objects in the environment surrounding vehicle 202 (e.g., identify that an object is present), classify objects (e.g., determine an object type associated with a detected object), segment sensor data and / or other representations of the environment (e.g., identify portions of sensor data and / or representations of the environment as associated with a detected object and / or object type), determine characteristics associated with the object (e.g., tracking that identifies current, predicted, and / or previous position, heading, speed, and / or acceleration associated with the object), etc. Data determined by perception component 228 is referred to as perception data.

[0043] The planning component 230 may receive the position and / or orientation of the vehicle 202 from the localization component 226 and / or sensory data from the perception component 228, and may determine instructions for controlling the operation of the vehicle 202 based at least in part on either of this data. In some examples, determining the instructions may comprise determining the instructions based at least in part on a format associated with an associated system (e.g., first instructions for controlling the movement of the autonomous vehicle may be formatted in a first format of messages and / or signals (e.g., analog, digital, pneumatic, kinematic) that the system controller 236 and / or drive component 212 can analyze / execute, and second instructions to the emitter 208 may be formatted according to a second format associated therewith). In some examples where the planning component 230 comprises hardware / software-in-the-loop in a simulation (e.g., for testing and / or training the planning component 230), the planning component 230 may generate instructions that can be used to control the simulated vehicle. The planning component 230 may, in some examples, be part of a simulation system 238.

[0044] The log data 232 may comprise sensor data, perception data, and / or scenario labels collected / determined by the vehicle 202 (e.g., by the perception component 228), as well as any other messages generated and / or transmitted by the vehicle 202 during operation, including, but not limited to, control messages, error messages, etc. In some examples, the vehicle 202 may transmit the log data 232 to the computing device 214. The computing device 214 may identify one or more scenarios based at least in part on the log data, which may also comprise defining the scenario. For example, the computing device 214 may determine the environmental layout, number, type, and configuration of objects in the environment and / or associate this definition with one or more portions of the log data related to the scenario. In some examples, the log data 232 may include (historical) perception data 240 generated by the vehicle 102 during operation of the vehicle. In additional or alternative examples, the perception data 240 may include perception data generated during a simulation.

[0045] In some examples, the computing device 214 may determine the agent behavior model 242 based at least in part on the log data 232 and the sensory data 240 in accordance with techniques described herein.

[0046] Simulation system 238 may operate on vehicle 202 and / or computing device 214 (although FIG. 2 depicts simulation system 238 as operating on computing device 214). If simulation system 238 is operating on a vehicle, simulation system 238 may provide alternative predictions regarding possible maneuvers and / or paths an object may take. These alternative predictions may be provided as inputs to planning component 230. Simulation system 238 may run in parallel with, and / or perception component 228 may be part of, and / or provide perception data to, simulation system 238.

[0047] The simulation system 238 may determine a simulation of the environment and / or the vehicle 202. For example, the simulation may comprise a representation of the position, orientation, movement, and / or quality of portions of the environment and / or the vehicle 202. The environment may comprise agents such as another vehicle, pedestrians, vegetation, buildings, and signs. The simulation may be used to test the operation of various components of the vehicle 202. In some examples, the simulation may be used to test the perception component 228 and / or the planning component 230 of the autonomous vehicle based at least in part on providing data from the simulation to the perception component 228 and / or the planning component 230.

[0048] The simulation system 238 can receive the scenario data 244 and / or the log data 232. The simulation system 332 can receive the scenario data 244 and / or the log data 232 to determine a simulation 246, which may be a two-dimensional or three-dimensional representation of the scenario. For example, the three-dimensional representation can comprise position, orientation, geometric data (e.g., polygonal representations, digital wire-mesh representations) and / or manipulation data associated with one or more objects of the environment, and / or can include material, signal, and / or signal data, although in other examples this data may be missing. While in some examples the scenario data 244 can be generated procedurally as part of combinatorial iterations through a scenario, received in response to interaction with a user interface of the computing device 214 (e.g., in response to user input), etc., in at least one example the scenario data 244 can be defined at least in part by the scenario associated with the log data (and / or the agent behavior model 242). In some examples, the scenario data 244 can alter characteristics of a scenario generated based on the log data 232.

[0049] The scenario data 244 may comprise a two-dimensional representation of the environment associated with the scenario, the objects contained therein, and their associated features, all of which may be part of the scenario associated with the log data. For example, the scenario data 244 may identify the location of an object, the area occupied by the object, the speed and / or acceleration associated with the object, whether the object is static or dynamic, the type of object associated with the object (e.g., classifications such as "pedestrian," "bicyclist," "vehicle," "large vehicle," "traffic light," "traffic sign," "building," "roadway," "crosswalk, sidewalk," etc.), and / or other kinematic properties associated with the object and / or object type (e.g., coefficient of friction, elasticity, malleability). With respect to the environment itself, the scenario data may identify the topology of the environment, weather conditions associated with the environment, lighting conditions (sunny, cloudy, night), the location of light sources, etc. In some examples, the topology associated with the scenario data 244, the location and dimensions of fixed objects (e.g., buildings, trees, signs), etc. may be generated based at least in part on the map 234. In some examples, scenario data 244 may be used to instantiate three-dimensional representations of objects, and / or a simulated environment may be instantiated based at least in part on map data (e.g., which may define the topology of the environment, including the location and / or dimensions of fixtures such as signs, plants, and / or buildings) and / or scenario data. In some examples, agent behavior models 242 may be associated with a scenario, or vice versa.

[0050] In some examples, the simulated sensor may determine sensor data 248 based at least in part on the simulation 246. For example, U.S. Patent Application Publication No. 2019 / 0129999, filed September 24, 2019, and incorporated herein, describes this in more detail. In additional or alternative examples, the simulation 246 itself may include simulated sensor data.

[0051] The perception component 228 (e.g., a copy thereof, which may comprise software and / or hardware, which may include hardware-in-the-loop simulation) may receive simulated sensor data 248 generated based on the simulation 246 and may output prediction data 250, which may comprise a predicted manipulation and / or a predicted path of a simulated object. The perception component 228 may provide the prediction data 250 to the planning component 230 and / or the agent behavior model 242, which may be part of or invoked by the simulation system 238 when an object of a type associated with the agent behavior model 242 is instantiated based at least in part on the agent behavior model 242.

[0052] The agent behavior model 242 can modify the prediction data 250 according to techniques described herein to generate an agent maneuver and / or path 252. The agent maneuver and / or path 252 can be used to control the movement of a simulated object in the simulation 246. In other words, the agent maneuver and / or path 252 can be part of the simulation 246, the simulation system 238 can include an agent control component for controlling the movement of the simulated dynamic object, and the agent control component can receive the agent maneuver and / or path 252 from the agent behavior model 242. The agent maneuver and / or path 252 can be used to test the perception component 228 and / or the planning component 230.

[0053] The testing and / or learning of planning component 230 can be based at least in part on agent operations / paths 252. For example, simulation system 238 can provide a safe and fast way to test how planning component 230 reacts to various scenarios that a vehicle may or may not encounter while driving, and determine whether planning component 230 generates safe and / or efficient control instructions to explore the scenarios.

[0054] In some examples, the simulation system 238 can additionally or alternatively store a rule set and determine whether the planning component 230 passed or failed a scenario based at least in part on the rule set. The rule set can be associated with the scenario data 244. In some examples, the simulation system 238 can record a version of the planning component 230 in association with a scenario identifier and / or an indication of whether the planning component 230 passed or failed. In additional or alternative examples, the simulation system 238 can determine a non-binary indicator (e.g., a score in addition to or instead of a pass / fail indicator) related to the performance of the planning component 230. The non-binary indication can be based at least in part on a set of weights associated with the rule set. In at least one example, the rule set can specify a maximum number of collisions per n iterations of the scenario, where n is a positive integer. In some examples, the rule set may be part of or replaced by an event detection system (Patent Document 4, filed November 13, 2019, the entire contents of which are incorporated herein) and / or a collision monitoring system (Patent Document 5, filed December 4, 2019).

[0055] Memory 220 and / or 224 may additionally or alternatively store a mapping system, a planning system, a ride management system, etc. While perception component 228 and / or planning component 230 are shown as stored in memory 220 and / or 224, perception component 228 and / or planning component 230 may include processor-executable instructions, machine learning models (e.g., neural networks), and / or hardware.

[0056] The localization component 226, the perception component 228, the planning component 230, the agent behavior model 242, the simulation system 238, and / or other components of the system 200 may comprise one or more ML models. For example, the localization component 226, the perception component 228, the planning component 230, and / or the simulation system 238 may each comprise a different ML model pipeline. In some examples, the ML model may comprise a neural network. An exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to generate an output. Each layer of the neural network may comprise another neural network, or may comprise any number of layers (whether convolutional or not). As can be understood in the context of this disclosure, a neural network may utilize machine learning, which may refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0057] Although described in the context of neural networks, any type of machine learning can be used consistently with this disclosure. For example, machine learning algorithms include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic nets, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomizer 3 (ID3), chi-squared), and others. Dynamic Interaction Detection (CHAID), Decision Stamps, Conditional Decision Trees), Bayesian algorithms (e.g., Naive Bayes, Gaussian Naive Bayes, Multinomial Naive Bayes, Average One-Dependent Estimator (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), clustering algorithms (e.g., k-means, k-medians, Expectation Maximization (EM), Hierarchical Clustering), association rule learning algorithms (e.g., Perceptron, Backpropagation, Hopfield Networks, Radial Basis Function Networks (RBFN)), deep learning algorithms (e.g., Deep Boltzmann Machines (DBM), Deep Belief Networks (DBN), Convolutional Neural Networks) (CNN), stacked autoencoders), dimensionality reduction techniques (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), Adaboost, stacked generalization (blending), gradient boosting machines (GBM), gradient boosted regression trees (GBRT), random forests), support vector machines (SVMs), supervised learning, unsupervised learning, semi-supervised learning, etc. Additional example architectures include neural networks such as ResNet50, ResNet-l0l, VGG, DenseNet, PointNet, etc.In some examples, the ML models described herein may include PointPillars, SECOND, top-down feature layers, and / or VoxelNet (see, for example, U.S. Patent Application Publication No. 2009 / 0129994, the entire contents of which are incorporated herein by reference). Architecture latency optimizations include MobilenetV2, Shufflenet, Channelnet, and Peelenet. The ML models may, in some examples, comprise a residual block such as Pixor.

[0058] Memory 220 may additionally or alternatively store one or more system controllers 236, which may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 202. These system controllers 236 may communicate with and / or control corresponding systems of drive component 212 and / or other components of vehicle 202. For example, planning component 230 may be able to generate instructions based at least in part on sensory data and / or simulated sensory data generated by perception component 228 and send the instructions to system controller 236, which may control operation of vehicle 202 based at least in part on the instructions.

[0059] 2 is shown as a distributed system, it should be noted that in alternative examples, components of vehicle 202 may be associated with computing device 214 and / or components of computing device 214 may be associated with vehicle 202. That is, vehicle 202 may perform one or more of the functions associated with computing device 214, and vice versa.

[0060] Process Example FIG. 3 is a flowchart pictorially illustrating an exemplary process 300 for generating one or more error models that may be part of an agent behavior model for realistic simulation of dynamic objects. In some examples, the technique may include determining an agent behavior model for each type of object and / or for each scenario. For example, the technique may comprise determining a first agent behavior model for the object type “vehicle,” a second agent behavior model for the object type “pedestrian,” etc. In additional or alternative examples, the vehicle behavior model may comprise one or more first error models associated with a first scenario, one or more second error models associated with a second scenario, etc. In some examples, the exemplary process 300 may be performed by components of the computing device 214 and / or the vehicle 202. It should be noted that the operations described below describe associated prediction data calculated at a single time step. The operations may be repeated for sensory data determined at additional time steps.

[0061] At operation 302, the example process 300 may comprise receiving predicted data associated with an object according to any of the techniques described herein. For example, a prediction component of the autonomous vehicle may receive sensor data from sensors associated with the autonomous vehicle and determine the predicted data via a perception pipeline. The predicted data may comprise a first predicted manipulation 304 and / or a first predicted path 306, and / or a second predicted manipulation 308 and / or a second predicted path (not labeled due to space constraints) of the object in the environment. For example, FIG. 3 continues to show an example representation of predicted data generated by the perception component in association with sensor data collected in the example scenario 100. It should be noted that the perception component may predict manipulation at each time step (e.g.,

[0062]

number

[0063] Go straight,

[0064]

number

[0065] from

[0066]

number

[0067] Turn right at

[0068]

number

[0069] from

[0070]

number

[0071] (Go straight ahead). It should also be noted that the maneuvers and paths may be predicted at the same or different time intervals. For example, paths may be predicted at a finer (shorter) time interval than the maneuvers (e.g., predicted every 0.2 seconds, while maneuvers may be predicted every 1 second, although other time intervals are considered). In some examples, the perception component may determine a probability associated with the predicted maneuvers. While the manner in which the probabilities are presented may vary (e.g., a number between 0 and 1, a percentage, logit), in the illustrated example, predicted maneuver 304 is associated with a probability of 0.65, and predicted maneuver 308 is associated with a probability of 0.35. In some examples, as in the illustrated example, the probabilities associated with predicted maneuvers related to an object may sum to 1.

[0072] In some examples, the perception component can additionally or alternatively determine scenario data as the vehicle operates. For example, the perception component can generate a scenario data structure comprising at least a portion of the perception data. The data structure can indicate the layout of the environment and / or the number, type, or configuration of objects within the environment. The predicted data and / or the scenario data structure can be associated with and / or appended to the log data.

[0073] While Figure 3 depicts the predicted path 306 as a discrete location within the environment, it is understood that the predicted path 306 may additionally or alternatively be a line or curve. For example, the perception component may be trained to output a motion primitive associated with the predicted maneuver 304 (see U.S. Patent Application Publication No. 2019 / 0139999, filed July 19, 2019, the entirety of which is incorporated herein), such as a general rightward curve, and the predicted path 306 may comprise one or more parameters associated with the motion primitive that modify the length, curvature, termination, etc. In any event, the path itself may define discrete locations, lines, curves, parameters associated with the curve / line, and / or variations from a standard curve / line (e.g., the distance or variance of distance from the curve / line that an object may be at a particular location along the curve / line).

[0074] At operation 310, the example process 300 may comprise receiving log data associated with a scenario according to any of the techniques described herein. In some examples, the log data may include an indication of an actual path 312 and / or an actual maneuver 314 taken by the object. In the illustrated example, a pedestrian turns back onto a roadway, which is reflected in both the illustrated actual path 312 and actual maneuver 314. In some examples, the actual maneuver 314 may be determined from the log data. For example, a perception component operating on or off the vehicle may determine a maneuver performed by the object based on sensor data acquired by the vehicle, and the actual maneuver may be associated with the sensor data as log data. In additional or alternative examples, the actual maneuver may be labeled based on input received at a computing device (e.g., from a human, from a machine learning model).

[0075] At operation 316, the example process 300 may comprise determining a difference between the predicted data and the logged data according to any of the techniques described herein. For predicted maneuvers, this may comprise determining whether the predicted and actual maneuvers differ at the same time step and generating an indication of whether the maneuvers are the same or different. A record may be kept of the number of times the actual maneuvers differed or were the same as the predicted maneuvers (relating to the same or similar scenarios, as identified by the clusters into which the sensor data are grouped, as described above). In some examples, the accuracy of the prediction may be indicated as a percentage, although other methods are also possible. For maneuvers that differ, the display may additionally or alternatively indicate the actual maneuvers. For path data, operation 316 may comprise determining the Euclidean (or any other coordinate space) distance between the predicted path and the actual path associated with the same time step, whether those points are discrete locations or locations along a line or curve. In some examples, the path difference may additionally or alternatively comprise differences in direction and / or speed, and the predicted and / or actual path may additionally or alternatively comprise such data. Figure 3 shows the Euclidean distance line between the predicted position and the actual position corresponding to the same time step.

[0076] At operation 318, the example process 300 may comprise determining an error model associated with the object type based at least in part on the difference determined in operation 314, according to any of the techniques described herein. Operation 318 may comprise determining a probability distribution associated with the object type and / or scenario based at least in part on the difference calculated over multiple time steps, such as for the same scenario, one or more time steps for multiple instances of the same scenario, the same location within the environment for multiple instances of the same scenario, etc. For example, the error model may indicate the percentage of time that a predicted trajectory associated with an object of a particular object classification was correct and / or when a particular trajectory was predicted, such as making a left turn, a right turn, or a straight turn. Further explaining, a particular error model may be associated with a particular predicted trajectory of an object classification, such as pedestrian, staying on track, making a left turn, etc., and the observed probability that the predicted trajectory is correct. The error model may additionally or alternatively be calculated for each position and / or time step. In other words, the error model may be different for each position or time step within a scenario. In some examples, the error model may additionally or alternatively be associated with a state of the autonomous vehicle. In other words, an error model may be determined for each vehicle state, predicted trajectory, and / or object classification associated with the object whose trajectory is predicted.

[0077] In some examples, operation 318 may comprise determining an error model associated with the predicted maneuver (i.e., maneuver error model 320), which indicates the probability that an object of a particular object type in a particular scenario will adhere to the predicted maneuver. The error model may additionally or alternatively indicate probabilities associated with different maneuvers. For example, maneuver error model 320 may comprise a first probability and / or probability distribution that the vehicle will engage in a first maneuver, a second probability and / or probability distribution that the vehicle will engage in a second maneuver, etc. In additional or alternative examples, the error model may indicate probabilities associated with maneuver changes (e.g., changing from performing a left turn to stopping, changing from a left turn to a right turn, changing from a left turn to going straight).

[0078] In some examples, operation 318 may comprise determining an error model (i.e., path error model 322) associated with the predicted path that indicates the probability that an object of a particular object type in a particular scenario will adhere to the predicted path. The path error model 322 may indicate the probability that an object of the object type will deviate from the path by a distance. The path error model 322 may comprise multiple such probabilities and distances. For example, the path error model 322 may indicate a first probability that a pedestrian will be located a first distance from the predicted location, a second probability that a pedestrian will be located a first distance from the predicted location, etc.

[0079] Determining the error models may additionally or alternatively comprise clustering the error models based at least in part on the scenario data associated therewith to determine error models associated with similar scenarios. Error models associated with the same cluster may be associated and / or merged into an error model representing the cluster.

[0080] In additional or alternative examples, exemplary process 300 may occur as part of a simulation (e.g., when a vehicle replays log data from the same or a different vehicle), but in some examples, exemplary process 300 may be accomplished as a pre-calculated step prior to exemplary process 600 on multiple logs received from one or more vehicles, or at least logs associated with one or more instances of a scenario.

[0081] Predicted operation examples, predicted path examples, scenario examples, and scenario examples FIG. 4A illustrates an example of a predicted maneuver 400, a predicted path 402, and an example of an actual path 402 taken by an object that was the subject of the prediction. In the illustrated example, the predicted path 402 is a continuous curve instead of discrete positions, although the discrete positions may be defined by a curve at different time steps. In some examples, the general shape of the predicted path 402 (i.e., a standard curve / line) may be based at least in part on the predicted maneuver 400, and the predicted path 402 may comprise one or more parameters that modify its general shape (e.g., an increase / decrease in the length of the curve, which may also modify the speed / acceleration as the path is defined over time, curvature, etc.). FIG. 4A also illustrates log data as black shaded circles, indicating the actual positions the object was located at different time steps. In the illustrated example, if the object, a vehicle, performed a right turn, the predicted maneuver was correct. It should also be noted that while only one predicted maneuver is shown, the perception component may determine a predicted maneuver associated with each time step. In the illustrated example, the forecast maneuver for the entire forecast horizon shown in FIG. 4A can be assumed to be a "left turn."

[0082] However, the actual path of the object varies from the predicted path 402. This is indicated in part by the distance between the actual position 406 and the portion of the predicted path 402 associated with the same time step as the actual position 406 (i.e., path variation 404). An error model generated for vehicle maneuvering in this scenario can be based at least in part on the correct maneuver prediction, and an error model generated for vehicle path in this scenario can be based at least in part on the path variation 404.

[0083] 4B and 4C illustrate different example scenarios, logged data representations 408 and 410 (e.g., top-down representations of sensor data included in logged data collected by vehicle 202 during operation), and associated simplified scenario data structures 412 and 414, respectively. For example, FIG. 4B illustrates logged data representation 408 of a scenario in which vehicle 202 encounters a four-way junction with two vehicles in the intersection and one pedestrian on the sidewalk at the location shown in logged data representation 408. In some examples, the perception component may determine a scenario data structure 412 that represents a simplified form of logged data representation 408, an overly simplified version of which is represented by scenario data structure 412, which is labeled "2 vehicles, 1 pedestrian at four-way junction." For example, the scenario data structure 412 may comprise a representation of the environment layout (e.g., roadway direction of travel, presence and location of reserved lanes, presence and location of sign / junction controls, presence, number, and location of lanes / junctions), the presence, location, and / or static objects in the environment, and the number, classification, location, direction, and / or speed of dynamic objects in the environment. In some examples, error models may be clustered based at least in part on their associated respective scenario data structure. For example, multidimensional scaling (MDS), principal component analysis (PCA), etc. may be applied to the scenario data and error models associated with the same cluster according to a clustering algorithm, and fused, and / or representative error models may be selected from among error models associated with the same cluster. In practice, this may allow a representative error model to be merged and / or determined from among error models associated with scenario data showing pedestrians on both sides of a vehicle, a scenario with multiple vehicles at an intersection, etc. For example, the error models generated in connection with FIGS. 4B and 4C may be fused according to some clustering technique.

[0084] FIG. 4C shows similar logged data representation 410 and associated (simplified) scenario data structure 414 that may be generated by the perception components of vehicle 202.

[0085] It should be noted that, in at least one example, more or fewer error models may be determined, but at least two sets of error models may be generated in association with the scenarios shown in FIGS. 4B and 4C. For example, because the scenario includes two different object types, a maneuver error model / path error model pair may be generated for the object type "vehicle" and another for the object type "pedestrian." More or fewer error models may be created in association with a single object type (e.g., when maneuver and path error models are fused, or when other error models are added for other predictive data, such as object deformation to disambiguate the object). In an additional or alternative example, at least four sets of error models may be generated (one set of error modes per vehicle type and per scenario, four sets between the two types of objects present in the two scenarios).

[0086] In additional or alternative examples, there may be a set of error models for each type of object, scenario, and / or location in the environment or time step within the scenario. For example, FIGS. 5A-5C illustrate how error models associated with an object and / or scenario may vary based at least in part on the object's location in the environment / action taken by the object. FIGS. 5A-5C illustrate a maneuver error model that may be part of the agent behavior model associated with the object type "vehicle" and the illustrated scenario. Note that the width of the error model 500 in FIG. 5A associated with a first time / location (which may be a future time / location) is wider than the probability distribution represented by the error models 502 and 504 at the next time. This width may indicate that given the position, heading, direction, and / or maneuver of the object 506 at the first time / location, the object 506 is likely to perform a maneuver that the prediction component predicts is unlikely. For example, such a maneuver may be some kind of anomalous behavior, such as swerving in front of the vehicle 202.

[0087] However, in Figure 5B, after object 506 begins to execute a turn in front of vehicle 202, the object 506 is more likely to complete the turn, as the updated prediction data can predict, and the associated maneuver error model includes a narrower probability distribution. This narrowing tendency is increased in the error model of Figure 5C.

[0088] It should be noted that although Figure 3 and Figures 5A-5C show the error model as a two-dimensional probability distribution, the error model can include a three-dimensional or higher-dimensional probability distribution.

[0089] Simulation Process Example FIG. 6 is a flowchart of an example process 600 for realistically simulating the motion of dynamic objects, including anomalous behavior, in a simulated environment. The example process 600 can include two subprocesses: an example subprocess 602 for controlling a simulated autonomous vehicle and an example subprocess 604 for controlling a simulated dynamic object. In some examples, the autonomous vehicle's perception and / or planning components (or copies thereof, which may include the use of hardware-in-the-loop simulation) can accomplish example subprocess 602, while a simulation system and / or agent behavioral model can accomplish example subprocess 604. It should be noted that operations that can be performed as part of subprocess 602 are shown on the left side of FIG. 6, and operations that can be performed as part of subprocess 604 are shown on the right side of FIG. 6. It should be noted that the vertical spacing of the blocks does not imply a relative timing of the operations. In additional or alternative examples, the simulation system and / or agent behavioral model can accomplish all or part of example subprocesses 602 and 604. In additional or alternative examples, the example process 600 does not include at least the sub-processes divided according to the diagram. In some examples, the vehicle computing device 204 and / or the computing device 214 may each or individually accomplish all or part of the example process 600.

[0090] In some examples, the example process 600 may be executed by a simulation system during real-time operation of an autonomous vehicle, with the simulated vehicle and environment described below corresponding to a real-world autonomous vehicle and the real-world environment surrounding the autonomous vehicle. In other words, the example process 600 may be executed for a real-world, real-time scenario. In such examples, the simulation system may output alternative predictions (e.g., alternative predicted maneuvers / paths). In additional or alternative examples, the example process 600 may be executed by a simulation system in a fully or partially synthetic scenario, where the scenario is generated fully based at least in part on scenario instructions or in part using sensor data, respectively. The synthetic scenario may be executed in real time or synthetic time. For example, a partially synthetic scenario may generate scenario data at least in part based on real-time sensor data or based on log data.

[0091] Some techniques may include running multiple simulations of a scenario and causing the dynamic objects to perform anomalous behaviors at a rate dictated by the agent behavior model. In additional or alternative examples, the dynamic objects may perform anomalous behaviors based at least in part on sampling the probability distributions of the agent behavior model.

[0092] At operation 606, example process 600 (and / or example subprocess 604) may comprise receiving scenario data according to any of the techniques described herein. For example, a simple simulation component may generate the scenario data. In addition to or in place of the scenario data, the scenario data may include sensor data, scenario instructions, and / or a scenario data structure. In some examples, the scenario data structure may include positions, orientations, and / or characteristics of static objects, dynamic objects, and / or the simulated vehicle in the environment, which may correspond to real-time behavior of the autonomous vehicle and / or may correspond to simulated behavior of the autonomous vehicle. In some examples, a planning component of the autonomous vehicle may generate instructions for controlling the simulated vehicle in a simulation-agnostic manner (e.g., the instructions generated by the planning component may be the same as instructions generated for an analogous real-world situation). The operation of the planning component may be tested in this manner. In some examples, the component may translate the instructions generated by the planning component into instructions for controlling a simulated vehicle in a simulated environment (e.g., updating a simulated state associated with the simulated vehicle, which may comprise a position, direction, velocity, acceleration, heading, and / or the like associated with the simulated vehicle).

[0093] The scenario data may additionally or alternatively comprise an indication of an object type associated with one or more objects and / or properties associated with one or more objects (e.g., position, velocity, acceleration, heading, material type, kinematic coefficients). Any of the data described herein may be part of a scenario data structure associated with a portion of the log data. The scenario data structure may comprise two-dimensional images, publish-subscribe messages, etc.

[0094] At operation 608, example process 600 (and / or subprocess 604) may comprise instantiating a simulated environment according to any of the techniques described herein based at least in part on the scenario data. Operation 608 may comprise procedurally generating the simulated environment based at least in part on a set of template models associated with object types. For example, the set of template models may comprise three different passenger vehicle models, four different pedestrian models, etc. Several different models may be present. In some examples, the template models may comprise three-dimensional models of the object's surface without texture, although in additional or alternative examples, the models may comprise texture.

[0095] The template model may include a polygonal mesh, a triangular mesh, etc. In some examples, models associated with dynamic objects may have a higher polygon count than models associated with static objects. In some examples, the simulated environment may comprise a surface model and lack lighting and / or texture. In additional or alternative examples, the simulated environment may comprise lighting and / or texture, although the techniques described herein function without lighting and / or texture. The simulated environment may comprise a model of a simulated vehicle. In some examples, instantiating environment portions of the simulated environment may be based at least in part on log data, scenario data, and / or map data, and objects in the environment may be based at least in part on instructions received as part of the scenario data (e.g., instructions generated at least in part based on user selections and / or interactions with a user interface, procedurally generated instructions).

[0096] At operation 610, example process 600 (and / or subprocess 602) may comprise receiving simulated sensor data according to any of the techniques described herein. For example, a perception component may receive the simulated sensor data based at least in part on a simulation generated by a simulation system (e.g., based at least in part on operations 616, 618, and / or 620).

[0097] At operation 612, example process 600 (and / or subprocess 602) may comprise predicting a manipulation and / or path of an object based at least in part on the simulated sensor data, according to any of the techniques described herein. For example, a perception component may determine a predicted manipulation and / or a predicted path of the (simulated) object based at least in part on the simulated sensor data.

[0098] At operation 614, example process 600 (and / or subprocess 602) may comprise determining a trajectory for controlling the simulated (autonomous) vehicle based at least in part on the perceptual data, according to any of the techniques described herein. For example, the perceptual data may be generated by a perception component at operation 612, and the perceptual data may comprise predictive data determined at operation 612.

[0099] At operation 616, example process 600 (and / or subprocess 604) may include modifying the predicted operation and / or path of the object based at least in part on an agent behavioral model associated with the object type and / or scenario, according to any of the techniques described herein. Operation 616 may comprise operation 618 and / or operation 620. In some examples, operation 616 may comprise obtaining an agent behavioral model associated with the object type and / or scenario and modifying the prediction data using the agent behavioral model. In examples where the agent behavioral model comprises multiple dimensions of data (e.g., different multi-dimensional error models associated with different scenarios, position, operation), operation 616 may comprise selecting an error model of the agent behavioral model that corresponds to the scenario and / or predicted operation / path.

[0100] At operation 618, example process 600 (and / or subprocess 604) may comprise sampling an error model to determine a variation from a predicted maneuver and / or a predicted path, according to any of the techniques described herein. Operation 618 may comprise randomly sampling an error model that is part of an agent behavior model associated with an object type and / or scenario (and / or location / time in the environment / scenario). The sampled portion of the probability distribution may indicate a variation from the predicted maneuver / location. Sampling the error model may comprise performing a weighted sampling of potential maneuvers associated with the state of the object based at least in part on the predicted maneuver and the error model (e.g., based on a prediction that the object will continue straight, the error model associated with such prediction and the state of the object and / or autonomous vehicle may indicate a 55% probability of continuing straight, an 18% probability of turning right, a 15% probability of stopping, and a 12% probability of turning left, and the random selection may be based at least in part on these probabilities).

[0101] For example, depending on the shape of the probability distribution, samples resulting in low-probability maneuvers can be associated with sharp deviations from the predicted maneuver / position and can also deviate from standard behavior. Different samples associated with a maneuver error model may result in maintaining the predicted maneuver, while other probabilities at or near the tails of the distribution may alter the maneuver to a different one (e.g., from a straight run to a hard stop or a change in direction). Similarly, samples from a path error model resulting in high probabilities may not alter the predicted path or may only slightly change the simulated object's position laterally. Meanwhile, behavior at or near the tail may involve broad lateral deviations from the predicted path.

[0102] In an additional or alternative example, operation 618 may comprise determining various predicted maneuvers based at least in part on the error model and selecting one of the various predicted maneuvers from among the various predicted maneuvers (by a set of weighted probabilities associated with the various predicted maneuvers based at least in part on the error model). For example, the perception component may predict that the dynamic object will perform a right turn and may predict a curve the object will follow. The agent behavior model may sample the maneuver error model and receive a standard probability, thus maintaining the maneuver as a right turn, while the path error model, when sampled, may result in a probability associated with abnormal behavior. Determining a modified maneuver / path for controlling the simulated object may comprise determining instructions that cause the simulated object to continue performing a right turn, but the turn may be too tight or too loose, cause the simulated object to exit a lane, enter a special path, enter a lane, enter a sidewalk, cross a center line, etc. This example may be reversed if the sampling result is reversed. The maneuver may be significantly altered—a vehicle that was performing a turn may suddenly turn in the opposite direction, but if the path probability is normal, the vehicle may not deviate significantly from its altered maneuver. In some examples, operation 618 may comprise altering the maneuver and / or path if the sampled probability meets or exceeds a threshold probability.

[0103] Additionally or alternatively, operation 618 may comprise determining a path variance associated with the maneuver based at least in part on a path error model. In some examples, the maneuver may define a reference line or curve, and the variance may be a lateral distance from the curve (e.g., a distance perpendicular to the curve). Again, determining the variance may be based at least in part on a random weighted selection based on the probability indicated by the path error model. In some examples, the modified maneuver may be determined at a first rate (e.g., 1 Hz, 0.5 Hz, 10 Hz), while the path change may be determined at a second rate that is the same as or different from the first rate (e.g., 1 Hz, 10 Hz, 100 Hz, 200 Hz). In at least one example, the first rate may be lower than the second rate (e.g., the first rate may be 1 Hz and the second rate may be 200 Hz), such that there may be multiple variations associated with a single modified maneuver such that the vehicle drifts laterally from the modified maneuver. In other words, a maneuver may be selected every second and the lateral position associated with the maneuver may be changed more frequently than once a second.

[0104] In operation 620, example process 600 (and / or subprocess 604) may additionally or alternatively include tracking previously simulated scenarios and selecting changes to the predicted operations and / or predicted paths to simulate the occurrence of anomalous behavior modeled by the agent behavior model according to any of the techniques described herein. For example, operation 620 may comprise tracking previous changes and distributing the selected changes consistent with an error model associated with the object type and scenario, such that changing the predicted operations and / or predicted paths changes the object's behavior similar to that of real-world objects. In other words, the agent behavior model may be used to enforce the occurrence of anomalous behavior. For example, operation 616 may use random sampling (operation 618), but if n iterations of operation 618 result in non-anomalous behavior despite the agent's behavior model indicating that a particular type of anomalous behavior occurs once in n instances of the scenario, operation 620 may enforce the changes to include the anomalous behavior, or may comprise discarding the results of operation 618 and directly selecting an action to control the simulated object.

[0105] At operation 622, example process 600 (and / or subprocess 604) may comprise controlling the simulated object based at least in part on the altered manipulation and / or altered path determined at least in part based on manipulation 618 in accordance with any of the techniques described herein.

[0106] Example clauses A. A method comprising: receiving log data from an automated vehicle, the log data including sensor data from sensors associated with the automated vehicle traversing an environment, a first detection of an object associated with a first time, a second detection of the object associated with a second time and indicative of a position of the object at the second time and a maneuver performed by the object, a predicted maneuver to be performed by the object and a predicted position of the object at the second time generated at a time prior to the second time, determining a maneuver error model and a path error model based at least in part on determining a difference between the predicted maneuver and the maneuver performed by the object and a distance between the predicted position and the position of the object at the second time; instantiating a simulated representation of the object in a simulated environment; determining a predicted movement of the simulated representation of the object; and controlling movement of the simulated representation of the object based at least in part on modifying the predicted movement using the maneuver error model and the path error model.

[0107] B. The method of paragraph A, wherein modifying the predicted movement includes determining one or more parameters associated with the simulated environment; determining an error associated with the prediction based at least in part on the one or more parameters, the predicted movement, and the operation error model; and modifying the predicted movement based at least in part on the error.

[0108] C. The method of either paragraph A or B, wherein modifying the predicted movement includes at least one of determining a modified maneuver from among a plurality of maneuvers associated with the maneuver error model based at least in part on a first probability indicated by the maneuver error model and associated with the modified maneuver; or determining a modified path based at least in part on determining a lateral variation from the modified maneuver based at least in part on a probability indicated by the path error model.

[0109] D. The method of any of paragraphs A-C, further comprising modifying the predicted movement, including determining a modified maneuver, the modified maneuver including at least one of instructions to perform a turn, perform a lane change, reverse, park, perform a traffic maneuver, enter or exit a roadway, crosswalk, sidewalk, or isolated lane, maintain current movement, and execute a stop.

[0110] E. The method of any of paragraphs A-D, wherein determining the maneuver error model includes determining a path error model including determining, based at least in part on the log data, a first number of instances in which the predicted maneuver is similar to the maneuver; determining, based at least in part on the log data, a second number of instances associated with a different maneuver performed by the object instead of the predicted maneuver; and determining, based at least in part on the log data, a third number of times the object was positioned at a distance from the trajectory associated with the predicted maneuver.

[0111] F. The method of any of paragraphs A-E, wherein the step of determining at least one of the operation error model or the path error model further includes determining one or more parameters related to the environment based at least in part on the sensor data, and determining a cluster of the plurality of error models based at least in part on the one or more parameters.

[0112] G. A system, comprising: one or more processors; and a memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations; the system comprising: receiving log data from an autonomous vehicle, the log data including: sensor data from sensors associated with the autonomous vehicle traversing an environment; a first detection of an object associated with a first time; a second detection of the object associated with a second time and indicative of a position of the object and a maneuver performed by the object at the second time; a predicted maneuver to be performed by the object and a predicted position of the object at the second time generated at a time prior to the second time; determining a maneuver error model and a path error model based at least in part on determining a difference between the predicted maneuver and the maneuver performed by the object and a distance between the predicted position and the position of the object at the second time; instantiating a simulated representation of the object in the simulated environment; determining a predicted movement of the simulated representation of the object; and controlling movement of the simulated representation of the object based at least in part on modifying the predicted movement using the maneuver error model and the path error model.

[0113] H. The system of paragraph G, wherein modifying the predicted movement includes determining one or more parameters associated with the simulated environment, determining an error associated with the prediction based at least in part on the one or more parameters, the predicted movement, and an operation error model, and modifying the predicted movement based at least in part on the error.

[0114] I. The system of either paragraph G or H, wherein modifying the predicted movement includes at least one of determining a modified maneuver from among a plurality of maneuvers associated with a maneuver error model based at least in part on a first probability indicated by the maneuver error model and associated with the modified maneuver, or determining a modified path based at least in part on determining a lateral variation from the modified maneuver based at least in part on a probability indicated by a path error model.

[0115] J. The system of any of paragraphs G through I, modifying the predicted movement including determining a modified maneuver, the modified maneuver including at least one of instructions to perform a turn, perform a lane change, reverse, park, perform a traffic maneuver, enter or exit a roadway, crosswalk, sidewalk, or isolated lane, maintain current movement, and execute a stop.

[0116] K. The system of any of paragraphs G-J, wherein determining a maneuver error model includes determining, based at least in part on the log data, a first number of instances in which the predicted maneuver is similar to the maneuver, and determining, based at least in part on the log data, a second number of instances associated with a different maneuver performed by the object instead of the predicted maneuver, wherein the path error model includes determining, based at least in part on the log data, a third number of times the object was positioned at the distance from the trajectory associated with the predicted maneuver.

[0117] L. In the system of any of paragraphs G-K, determining at least one of the operation error model or the path error model further includes determining one or more parameters related to the environment based at least in part on the sensor data, and determining a cluster of the plurality of error models based at least in part on the one or more parameters.

[0118] M. The system of any of paragraphs G-L, wherein the action further includes at least one of controlling a simulated vehicle or an autonomous vehicle based at least in part on the movement of the simulated representation.

[0119] N. A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations, including receiving log data from the autonomous vehicle including sensor data from sensors associated with the autonomous vehicle traversing an environment, a first detection of an object associated with a first time, a second detection of the object associated with a second time and indicative of a position of the object and a maneuver performed by the object at the second time, a predicted maneuver to be performed by the object and a predicted position of the object at the second time, generated at a time prior to the second time, determining a maneuver error model and a path error model based at least in part on determining a difference between the predicted maneuver and the maneuver performed by the object and a distance between the predicted position and the position of the object at the second time, instantiating a simulated representation of the object in the simulated environment, determining a predicted movement of the simulated representation of the object, and controlling movement of the simulated representation of the object based at least in part on modifying the predicted movement using the maneuver error model and the path error model.

[0120] O. The non-transitory computer-readable medium of paragraph N, wherein modifying the predicted movement includes determining one or more parameters associated with the simulated environment, determining an error associated with the prediction based at least in part on the one or more parameters, the predicted movement, and an operation error model, and modifying the predicted movement based at least in part on the error.

[0121] P. The non-transitory computer-readable medium of paragraph O, wherein modifying the predicted movement includes at least one of determining a modified maneuver from among a plurality of maneuvers associated with the maneuver error model based at least in part on a first probability indicated by the maneuver error model and associated with the modified maneuver, or determining a modified path based at least in part on determining a lateral variation from the modified maneuver based at least in part on a probability indicated by the path error model.

[0122] Q. The non-transitory computer-readable medium of any of paragraphs N through P, wherein modifying the predicted movement includes determining a modified maneuver and the modified maneuver includes at least one of instructions to perform a turn, perform a lane change, reverse, park, perform a traffic maneuver, enter or exit a roadway, crosswalk, sidewalk, or isolated lane, maintain current movement, or execute a stop.

[0123] R. The non-transitory computer-readable medium of any of paragraphs N-Q, wherein determining the maneuver error model includes determining, based at least in part on the log data, a first number of instances in which the predicted maneuver is similar to the maneuver; determining, based at least in part on the log data, a second number of instances associated with a different maneuver performed by the object instead of the predicted maneuver; and determining, based at least in part on the log data, a third number of times the object was positioned at a distance from the trajectory associated with the predicted maneuver.

[0124] S. The non-transitory computer-readable medium of any of paragraphs N through R, wherein determining at least one of the operation error model or the path error model further includes determining one or more parameters related to the environment based at least in part on the sensor data, and determining a cluster of a plurality of error models based at least in part on the one or more parameters.

[0125] T. The non-transitory computer-readable medium of any of paragraphs N-S, wherein the actions further include at least one of controlling a simulated vehicle or an autonomous vehicle based at least in part on the movement of the simulated representation. While the example clauses above are described with respect to one particular embodiment, it should be understood in the context of this specification that the content of the example clauses may also be implemented via methods, devices, systems, and / or other embodiments. Furthermore, any of Examples A-T may be implemented alone or in combination with one or more of the others of Examples A-T.

[0126] conclusion Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

[0127] The components described herein represent instructions that can be stored on any type of computer-readable medium and implemented in software and / or hardware. All of the methods and processes described above may be embodied in and fully automated via software code components and / or computer-executable instructions executed by one or more computers or processors, hardware, or some combination thereof. Alternatively, some or all of these methods may be performed in dedicated computer hardware.

[0128] At least some of the processes described herein are illustrated as logic flowcharts, with each operation representing a sequence of actions that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more non-transitory computer-readable storage media that, when executed by one or more processors, cause a computer or autonomous vehicle to perform the recited actions. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement a process.

[0129] Conditional language such as "can," "could," "may," or "might," etc., is understood within the context of presenting particular features, elements, and / or steps that include particular examples but not other examples, unless otherwise stated. Thus, such conditional language is not generally intended to imply that particular features, elements, and / or steps are somehow required for one or more examples, nor does one or more examples necessarily include logic for determining whether a particular feature, element, and / or step is included or performed in a particular example, with or without user input or prompting.

[0130] Connecting language such as "at least one of X, Y, or Z," unless otherwise stated, should be understood to represent that the item, term, etc. is either X, Y, or Z, or any combination including multiples of each element. Unless expressly stated as singular, "a" means singular as well as plural.

[0131] General descriptions, elements, or blocks in the flowcharts described herein and / or shown in the accompanying figures should generally be understood as potentially representing modules, segments, or portions of code that include one or more computer-executable instructions for implementing specific logical functions or elements. The examples described herein include alternative implementations that can be performed out of the order shown or described, including removing elements or functions, or performing operations substantially synchronously, in reverse order, with additional operations, or omitting operations, depending on the functionality involved, as will be understood by those skilled in the art. It should be noted that the term "substantially" can also indicate scope. For example, "substantially simultaneously" indicates that two activities occur within a time range of each other, and "substantially the same dimension" indicates that two elements have dimensions within each other, and / or are similar.

[0132] It should be understood that many variations and modifications can be made to the above examples, elements of which are among the other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and protected by the following claims.

Claims

1. 1. A system comprising: one or more processors; and a memory storing processor-executable instructions that, when executed by one or more of said processors, cause said system to perform operations, said operations including: receiving log data from an autonomous vehicle; sensor data from sensors associated with the autonomous vehicle traversing an environment; a first detection of the object associated with a first time and indicating a position of the object at the first time; a second detection of the object associated with a second time and indicative of a location of the object and an operation performed by the object at the second time; a predicted operation associated with the object and a predicted position of the object at the second time generated at a time prior to the second time; determining a maneuver error model and a path error model based at least in part on determining a difference in probability distribution between the predicted maneuver and the maneuver performed by the object and a distance between the predicted position and the position of the object at the second time; instantiating a simulated representation in a simulated environment, the simulated representation being at least a two-dimensional or three-dimensional representation of the object; determining a predicted movement of the simulated representation of the object; controlling a movement of the simulated representation of the object based at least in part on modifying the predicted movement using the steering error model and the path error model; and A system comprising:

2. Modifying the predicted movement may include: determining one or more parameters associated with the simulated environment; determining an error associated with the predicted movement based at least in part on the one or more parameters, the predicted movement, and the manipulation error model; modifying the predicted movement based at least in part on the error; and The system of claim 1 , comprising:

3. Modifying the predicted movement may include: determining the modified operation from among a plurality of operations associated with the operation error model based at least in part on a first probability indicated by the operation error model and associated with the modified operation; determining a modified path based at least in part on determining lateral variations from the modified maneuver based at least in part on the probabilities indicated by the path error model; and 3. The system according to claim 1, comprising at least one of:

4. modifying the predicted movement includes determining a modified maneuver; The modified operation is Implementing a turn; Implementing lane changes, To go backwards and Parking and Carrying out traffic operations; Entering or exiting a roadway, crosswalk, sidewalk, or segregated lane; Maintaining the current trend and To perform a stop 4. The system of claim 1, further comprising at least one instruction:

5. determining the operation error model determining a number of first instances in which the predicted operation is similar to the operation based at least in part on the log data; determining, based at least in part on the log data, a number of second instances associated with different operations performed by the object instead of the predicted operation; determining the path error model determining a number of third instances in which the object was positioned at the distance from a trajectory associated with the predicted maneuver based at least in part on the log data; 5. The system of claim 1, further comprising:

6. Determining at least one of the operation error model or the path error model comprises: determining one or more parameters associated with the environment based at least in part on the sensor data; determining a cluster of a plurality of error models based at least in part on the one or more parameters; The system of claim 1 , further comprising:

7. 7. The system of claim 1, wherein the action further comprises at least one of controlling a simulated vehicle or the autonomous vehicle based at least in part on the movement of the simulated representation.

8. 1. A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: receiving log data from an autonomous vehicle; sensor data from sensors associated with the autonomous vehicle traversing an environment; a first detection of the object associated with a first time and indicating a position of the object at the first time; a second detection of the object associated with a second time and indicative of a location of the object and an operation performed by the object at the second time; a predicted operation associated with the object and a predicted position of the object at the second time generated at a time prior to the second time; determining a maneuver error model and a path error model based at least in part on determining a difference in probability distribution between the predicted maneuver and the maneuver performed by the object and a distance between the predicted position and the position of the object at the second time; instantiating a simulated representation in a simulated environment, the simulated representation being at least a two-dimensional or three-dimensional representation of the object; determining a predicted movement of the simulated representation of the object; controlling a movement of the simulated representation of the object based at least in part on modifying the predicted movement using the steering error model and the path error model; and 1. A non-transitory computer-readable medium comprising:

9. Modifying the predicted movement may include: determining one or more parameters associated with the simulated environment; determining an error associated with the predicted movement based at least in part on the one or more parameters, the predicted movement, and the manipulation error model; modifying the predicted movement based at least in part on the error; and 10. The non-transitory computer-readable medium of claim 8, comprising:

10. Modifying the predicted movement may include: determining the modified operation from among a plurality of operations associated with the operation error model based at least in part on a first probability indicated by the operation error model and associated with the modified operation; determining a modified path based at least in part on determining lateral variations from the modified maneuver based at least in part on the probabilities indicated by the path error model; and 10. The non-transitory computer-readable medium of claim 9, comprising at least one of:

11. modifying the predicted movement includes determining a modified maneuver; The modified operation is Implementing a turn; Implementing lane changes, To go backwards and Parking and Carrying out traffic operations; Entering or exiting a roadway, crosswalk, sidewalk, or segregated lane; Maintaining the current trend and To perform a stop 11. The non-transitory computer-readable medium of claim 8, comprising at least one instruction of:

12. determining the operation error model determining a number of first instances in which the predicted operation is similar to the operation based at least in part on the log data; determining, based at least in part on the log data, a number of second instances associated with different operations performed by the object instead of the predicted operation; determining the path error model determining a number of third instances in which the object was positioned at the distance from a trajectory associated with the predicted maneuver based at least in part on the log data; 12. The non-transitory computer-readable medium of claim 8, comprising:

13. Determining at least one of the operation error model or the path error model comprises: determining one or more parameters associated with the environment based at least in part on the sensor data; determining a cluster of a plurality of error models based at least in part on the one or more parameters; 13. The non-transitory computer-readable medium of claim 8, further comprising:

14. 14. The non-transitory computer-readable medium of claim 8, wherein the actions further include at least one of controlling a simulated vehicle or the autonomous vehicle based at least in part on the movement of the simulated representation.

Images