Instantiating objects in a simulated environment based on log data

Abstract

Techniques are disclosed herein for instantiating objects in a simulated environment based on log data. Some of the techniques may include receiving log data representative of an environment in which a real-world vehicle was operating. Using the log data, a simulated environment may be generated for testing the simulated vehicle. The simulated environment may represent the environment in which the real-world vehicle was operating. The techniques may further include determining a position of the simulated vehicle as it traverses the simulated environment. Based at least in part on the log data, a previous position of the real-world vehicle in the environment that is closest to the position of the simulated vehicle in the simulated environment may be determined. In this manner, the simulated environment may be updated to include simulated objects that represent objects in the environment perceived by the vehicle from the previous position.

Description

[Technical field]

[0001] The present invention relates to instantiating objects in a simulated environment based on log data. [Background technology]

[0002] This application claims priority to U.S. Utility Application No. 17 / 193,826, filed March 5, 2021, the entirety of which is incorporated herein by reference.

[0003] Simulated data and simulations can be used to test and verify the functionality of vehicle systems where testing in the real world may be prohibitive (e.g., due to safety concerns, time, repeatability limitations, etc.). However, creating a simulation that both accurately reflects a real-world scenario and validates the functionality of the vehicle system is a difficult challenge. For example, if the simulation is based on data previously captured by a vehicle operating in the real world, the vehicle system tested under the simulation may react differently than the real-world vehicle that captured the data. This can make it difficult to verify that the tested vehicle system is accurate. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent Application Serial No. 15 / 644,267 [Patent Document 2] U.S. Patent Application Serial No. 15 / 693,700 [Patent Document 3] U.S. Patent Application Serial No. 16 / 198,653 [Brief description of the drawings]

[0005] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. The same reference number in different figures indicates similar or identical components or features.

[0006] [Figure 1] FIG. 1 is a pictorial flow diagram illustrating an example data flow in which a vehicle generates log data associated with an environment and transmits the log data to a computing device that determines an association between the vehicle's location and object data captured by the vehicle at different points in time. [Diagram 2] FIG. 2 shows an example environment in which, at different times, a vehicle perceives different objects in the environment and different perspectives of the different objects. [Diagram 3] FIG. 3 illustrates an example data transformation performed at least in part by a vehicle perception system. [Figure 4] FIG. 4 is a pictorial flow diagram illustrating an exemplary process in which objects are instantiated in a simulated environment based on various positions of a simulated vehicle relative to various previous positions of the vehicle in the real world. [Diagram 5] FIG. 5 is a pictorial flow diagram illustrating an example process for avoiding updating a simulated environment based on a simulated vehicle that remains stationary within the simulated environment. [Figure 6] FIG. 6 is a block diagram illustrating an example system for implementing some of the various techniques described herein. [Figure 7] FIG. 7 is a flow chart illustrating an example method for updating a simulated environment to include objects based on the position of a simulated vehicle relative to a previous position of the vehicle in the real world. [Figure 8] FIG. 8 is a flow chart illustrating another example method for updating a simulated environment to include objects based on the position of a simulated vehicle relative to a previous position of the vehicle in the real world. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Techniques for instantiating objects in a simulated environment based on logged data are disclosed herein. As discussed above, creating a simulation that accurately reflects a real-world scenario and verifies the functionality of a vehicle system is a challenging task. For example, if the simulation is based on data captured by a vehicle operating in the real world, the tested vehicle system under the simulation may react differently than the real-world vehicle from which the data was captured. This may make it difficult to verify whether the tested vehicle system is accurate. For example, the position of the simulated vehicle may deviate from the path of the vehicle that generates the logged data, but the perception is still based on the position of the vehicle that is logged. This difference in perception reduces the accuracy of the simulation and may even invalidate the results if the deviation is large.

[0008] For example, consider a simulated (e.g., test) environment for testing and / or validating an autonomous vehicle controller. The simulated environment may be generated based on log data captured by a real-world vehicle. Thus, the simulated environment may correspond to or otherwise represent a real-world environment (e.g., the Embarcadero Road in San Francisco, California). As such, a real-world environment may include both static objects that remain stationary for long periods of time (e.g., buildings, street lights, traffic signs, benches, parked vehicles, trees, barriers and fences, etc.) and dynamic objects that tend to keep moving (e.g., pedestrians, moving vehicles, bicyclists, construction equipment, etc.). Thus, the simulated environment may include both static and dynamic objects that accurately reflect a real-world environment.

[0009] However, for static objects, simulating these objects in a simulated environment can present some challenges. For example, even if they may be classified as static objects, the appearance of these objects may change over time, such as when the vehicle is moving. As another example, a real-world vehicle's sensor system may, in some instances, misdetect a static object for an instance of time over a period of time. As such, the vehicle system may not always generate a constant static bounding box output that represents the corresponding static object. Instead, the vehicle system's output that represents a static object may change, move, appear, disappear, and / or the like over a given period of time.

[0010] Furthermore, as a real-world vehicle moves through an environment, the vehicle's systems may be limited to only see / perceive those objects that are within its horizon. For example, the vehicle's systems may not perceive objects in advance before the vehicle is within range of those objects. Thus, a simulation representing a real-world vehicle in a real-world environment may present simulated object data to the simulated vehicle to mimic what the real-world vehicle perceives at a corresponding location and time in the real world. However, if the simulated vehicle deviates from the real-world vehicle (e.g., due to changes to the autonomous controller of the simulated vehicle controller), it may become difficult to present accurate object data as perceived by the simulated vehicle through the use of real-world vehicle logs.

[0011] Accordingly, the present disclosure describes techniques for improving a simulation based on a vehicle log by determining a log vehicle position that is closest to a current, simulated vehicle position, and by instantiating static objects in the simulated environment that the log vehicle would have perceived at that time and / or from that position. For example, the techniques may include receiving log data that represents an environment in which a vehicle (e.g., the log vehicle) was operating, and generating a simulated scenario for testing a vehicle controller based on the log data. In some examples, the simulated scenario may include a simulated environment that represents a portion of the environment in which the vehicle was operating. The techniques may also include inputting the simulated scenario to a vehicle controller, where the vehicle controller causes the simulated vehicle to traverse the simulated environment. While the simulation is occurring, a position of the simulated vehicle in the simulated environment may be determined. Thus, based on determining the position of the simulated vehicle, the log data may be consulted to determine a previous position of the vehicle in the environment that is closest to the position of the simulated vehicle in the simulated environment. Thus, the techniques may further include updating the simulated environment to include simulated objects that represent objects in the environment perceived by the vehicle from the previous position.

[0012] The techniques discussed herein can improve the accuracy of simulations for testing and / or validating vehicle controllers. For example, by determining the closest previous position of the logging vehicle relative to the current position of the simulated vehicle, and instantiating the static object in the simulated environment as the logging vehicle would have perceived it from the previous position, the dynamic aspects of the static object can be realistically simulated. Additionally, the techniques described herein can be used to change the appearance of the static object over time, such as when the simulated vehicle is moving (e.g., to simulate how an object's appearance changes based on a change in viewpoint in a real-world scenario). Furthermore, the techniques described herein allow for instantiating a static object in the simulated environment at the moment the logging vehicle would have perceived it in the real world. This allows for improved simulation testing by providing more accurate data and / or objects to the vehicle controller that correspond to what the real-world vehicle would have perceived, and therefore allows for more realistic scenarios where the logging vehicle does not include such information for a particular time or position of the vehicle in the simulation. Other improvements using the disclosed techniques will be readily apparent to those skilled in the art.

[0013] By way of example, but not limitation, a method according to various techniques disclosed herein may include receiving log data representative of an environment in which a vehicle was operating. In some examples, the vehicle may include an autonomous vehicle. Additionally, the log data may be associated with a period of time in which the vehicle was operating within the environment. In some examples, the log data may be generated by the vehicle's perception system based at least in part on sensor data captured by one or more vehicle sensors. In at least one example, the log data may include various log data instances representative of an environment in which the vehicle and / or another vehicle was operating during multiple periods of time. In a further example, the log data may include various images representative of an output of the real vehicle's perception system at various times. That is, the log data may include, for example, a first log data image representative of a first perception system output at a first time, a second log data image representative of a second perception system output at a second time after the first time, a third log data image representative of a third perception system output at a third time after the second time, etc. In some examples, each or various log data images may include various objects and / or various perspectives of objects perceived by the real vehicle at that time.

[0014] In some examples, the method may use the log data to generate simulated scenarios for testing the vehicle controller. The simulated scenarios may include a simulated environment representing at least a portion of an environment. The vehicle controller, in some examples, may include a controller of an autonomous vehicle. In various examples, the simulated scenarios may be input to the vehicle controller, which causes the simulated vehicle to traverse the simulated environment. In at least one example, the simulated scenarios, simulated environments, simulated objects, etc. may be input to the vehicle controller in the form of simulation instructions. That is, the simulation instructions may represent the simulated scenarios, simulated environments, simulated objects, etc.

[0015] In some examples, the method may include determining a location of the simulated vehicle within the simulated environment. For example, geolocation data may be received from the simulated vehicle based on a position of the simulated vehicle within the simulated environment, and the location of the simulated vehicle may be determined based at least in part on the geolocation data.

[0016] In examples, the method may additionally or alternatively include determining a previous position of the vehicle within the environment associated with the position of the simulated vehicle within the simulated environment. In various examples, determining the previous position may include determining a previous position of the vehicle closest to a current position of the simulated vehicle. Additionally or alternatively, based at least in part on determining the previous position, an instance of time when the vehicle was located at the previous position may be determined from the log data. For example, the log data may include an association between each previous position of the vehicles within the environment (e.g., geolocation data) and an instance of time (e.g., timestamp) associated with when the vehicles were located at their respective previous positions.

[0017] In various examples, the object data may be obtained from the log data based at least in part on determining at least one of a previous location and / or time instance. The object data may represent one or more objects in the environment that are / were perceived by the vehicle from a previous location and / or from that time instance. By way of example and not limitation, for any given time instance during a period during which the vehicle was operating in the environment, the log data may store one or more of a respective timestamp (e.g., indicative of the time instance), respective geolocation data (e.g., indicative of the location of the vehicle at that time instance), and respective object data (e.g., indicative of one or more objects perceived by the vehicle at that time instance and from that location). In some examples, the one or more objects in the environment may include static objects and / or dynamic objects. For example, the one or more objects in the environment may include buildings, structures, barriers, fences, obstacles, vehicles, pedestrians, bicyclists, benches, street lights, traffic signs, trees, etc.

[0018] In some examples, the method may further update the simulated environment to include simulated objects that represent objects in the environment. That is, the simulated objects may be instantiated within the simulated environment. Additionally, the simulated objects may be positioned within the simulated environment at locations that correspond to locations where the objects were positioned in the real-world environment. In various examples, updating the simulated environment may include updating the simulated environment to include multiple simulated objects that represent multiple objects in the environment.

[0019] In an additional or alternative example, the simulated scenario and / or the simulated environment may be updated by presenting to the vehicle controller of the simulated vehicle various logged data images representing the perception system output of the real vehicle when the real vehicle is positioned at a real location in the real environment that most closely matches the simulated location of the simulated vehicle in the simulated environment. The logged data images may include simulated representations of one or more real objects perceived by the real vehicle from their real locations in the real environment. In this manner, as the simulated vehicle moves through the simulated environment, various logged data images may be presented to the vehicle controller of the simulated vehicle based on the current simulated location of the simulated vehicle within the simulated environment. Similarly, when the simulated vehicle stops within the simulated environment, the same or similar logged data images may be repeatedly presented to the vehicle controller to simulate the vehicle remaining stopped.

[0020] In at least one example, the method may include determining that a position of the simulated vehicle changes. That is, the method may include determining that the simulated autonomous vehicle is located at a second location in the simulated environment that is different from the original location. Thus, a second previous location of the vehicle in the environment associated with the second location of the simulated vehicle may be determined. In various examples, determining the second previous location may include determining a second previous location of the vehicle that is closest to the second location of the simulated vehicle. Additionally or alternatively, based at least in part on determining the second previous location, a second instance of time at which the vehicle was located at the second previous location may be determined from the log data. In some examples, second object data may be obtained from the log data based at least in part on determining at least one of the second previous location and / or the second instance of time. Based at least in part on the object data, in some examples, the method may include updating at least one of an appearance or a position of the simulated object with respect to the simulated vehicle. Additionally or alternatively, the method may include updating the simulated environment to include another simulated object representing another object in the environment.

[0021] In some examples, the method may additionally include controlling the simulated vehicle based at least in part on the simulated object. For example, the vehicle controller being tested may perceive simulated objects in the simulated environment and alter the trajectory of the simulated vehicle based at least in part on perceiving the simulated object.

[0022] Additionally, in some examples, the user-specified objects may be simulated at various locations in the simulated environment. Thus, the method may include receiving input indicating user-specified locations at which the simulated objects are to be instantiated within the simulated environment, and updating the simulated environment to include the simulated objects at the user-specified locations. In further examples, the simulated scenarios may be generated based at least in part on a three-dimensional (3D) simulation, a previous simulation, or the like.

[0023] The techniques described herein can be implemented in many ways. Exemplary implementations are provided below with reference to the following figures. Although described in the context of an autonomous vehicle, the techniques described herein can be applied to a variety of systems that require control and / or interaction with objects in an environment and are not limited to autonomous vehicles. In another example, the systems, methods, and apparatus can be used in an aviation or nautical context. Additionally, the techniques described herein can be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination thereof.

[0024] 1 is a pictorial flow diagram illustrating an example data flow 100 in which a vehicle 102 generates log data 104 associated with an environment 106 and transmits the log data to a computing device 108 that determines an association between the location of the vehicle 102 and object data captured by the vehicle 102 at different points in time. Additionally, the computing device 108 may also generate simulation instructions 112 based at least in part on the log data 104. In some examples, the log data 104 can include three-dimensional voxel data, image data, bounding boxes, two-dimensional and / or three-dimensional contours of objects, classification data, segmentation data, occlusion data, etc.

[0025] Vehicle 102 may include a computing device having a perception engine and / or planner component to perform operations such as detecting, identifying, segmenting, classifying, and / or tracking objects from sensor data captured within environment 106. For example, a computing device associated with vehicle 102 may detect objects 114(1), 114(2), 114(3), and 114(4). Objects 114(1), 114(2), 114(3), and 114(4) are depicted as parked vehicles, trees, buildings, and street lights within environment 106, although objects 114(1), 114(2), 114(3), and 114(4) may also include bicyclists, pedestrians, animals, road signs, signs, traffic lights, mailboxes, barriers, fences, benches, and / or other objects.

[0026] The computing device associated with the vehicle 102 may include one or more processors and memory communicatively coupled to the one or more processors. The one or more processors may include, for example, one or more FPGAs, SoCs, ASICs, and / or CPUs. The vehicle 102 may traverse the environment 106 and determine and / or capture data. For example, the computing device associated with the vehicle 102 may determine vehicle status data, vehicle diagnostic data, vehicle metrics data, and / or map data. In some examples, the vehicle 102 may include one or more sensors, where the one or more sensors may include one or more time-of-flight sensors, lidar sensors, radar sensors, sonar sensors, image sensors, audio sensors, infrared sensors, geolocation sensors, wheel encoders, IMUs, etc., or any combination thereof, although other types of sensors are contemplated.

[0027] As the vehicle 102 traverses the environment 106, the sensors may capture sensor data associated with the environment 106. For example, and as discussed above, some of the sensor data may be associated with objects, such as objects 114(1), 114(2), 114(3), and 114(4). Objects 114(1), 114(2), 114(3), and 114(4) may represent static objects, although in some examples, the sensor data may also be associated with dynamic objects. As used herein, a "dynamic object" may be an object associated with movement within the environment 106 (e.g., a vehicle, a motorcycle, a bicyclist, a pedestrian, an animal, etc.) or an object capable of movement (e.g., a parked vehicle, a standing pedestrian, etc.). In contrast, a "static object" may be an object associated with the environment 106, such as, for example, a building / structure, a road surface, a road sign, a traffic light, a barrier, a tree, a sidewalk, a street light, a parked vehicle, etc. In some examples, a computing device associated with vehicle 102 may determine information about objects in the environment, such as bounding box information, classification information, segmentation information, etc.

[0028] A computing device associated with the vehicle 102 may generate log data 104. For example, the log data 104 may include sensor data, perception data, planning data, vehicle status data, speed data, intent data, and / or other data generated by the vehicle computing device. In some examples, the sensor data may include data captured by sensors such as time-of-flight sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, illuminance, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, etc. The sensor data may be data captured by such sensors such as time-of-flight data, position data, lidar data, radar data, sonar data, image data, audio data, etc. In some examples, the log data 104 may include time data associated with other data generated by the vehicle computing device.

[0029] The log data 104 may represent various states of the environment 106 along a timeline. As shown in FIG. 1, the states may include various representations of the environment (real-world environment, environment represented by captured sensor data, data based on the captured sensor data (e.g., bounding boxes, object classification, planner decisions), and / or environment represented by a simulator) at individual times or over time. For example, the log data 104 may represent a state 116 of the environment 106 at time t0118. In some examples, the state 116 may be an initial state of the environment 106 associated with a beginning (or any) portion of the log data 104. In some examples, the state 116 may be a state of the environment 106 associated with any (e.g., intermediate) portion of the log data 104. The log data 104 may represent a state 120 of the environment 106 at time t1122 after time t0118. 1, the log data 104 may further represent a state 124 of the environment 106 at a time t2 126, which is subsequent to time t0118 and time t1122. For purposes of illustration only, the state 116 may represent the environment 106 at an initial time t0118 at 0 seconds, the state 120 may represent the environment 106 at time t1122, 5 seconds later, and the state 124 may represent the environment 106 at time t2 126, 15 seconds later. Thus, in some examples, the times t0118, t1122, t2 126 may be non-contiguous (e.g., there is time elapsed between the times t0118, t1122, and t2 126) or the times t0118, t1122, and t2 126 may be continuous (e.g., there is no time elapsed between the times t0118, t1122, and t2 126).

[0030] The vehicle 102 may transmit some or all of the log data to the computing device 108 over one or more networks 128. For example, the vehicle 102 may transmit the log data 104, including time data, perception data, object data, location data, and / or map data, to the computing device 108 for storage of the log data. Such transmission may be wired, wireless, or otherwise, as detailed herein. Examples of vehicle generated data that may be included in the log data 104 are described, for example, in U.S. patent application Ser. No. 15 / 644,267, filed Jul. 7, 2017, entitled “Interactions Between Vehicle and Teleoperations System,” which describes, in part, communication signals that may include sensor data and data indicative of an occurrence of an event, as well as vehicle control data that may be derived from data received from sensors and a planner, an object data calculator, an object classifier, a collision prediction system, a kinematics calculator, a safety system actuator, and control operations of a drive system; U.S. patent application Ser. No. 15 / 693,700, filed Sep. 1, 2017, entitled “Occupant Protection System Including Expandable Curtain and / or Expandable Bladder,” which describes, in part, data representative of a vehicle trajectory, position data, object data, and vehicle control data; and U.S. patent application Ser. No. 15 / 693,700, filed Sep. 1, 2017, entitled “Executable Component Interface and No. 16 / 198,653, filed Nov. 21, 2018, entitled "A Type of Log Data Controller," each of which is incorporated by reference in its entirety for all purposes. In some examples, the computing device 108 may include a server that stores the log data 104 in a database. In some examples, the computing device 108 may operate a scenario generator to generate simulation instructions 112 that define a simulated scenario and / or a simulated environment.

[0031] Based on the log data 104, the scenario generator may identify objects represented in the log data 104 (e.g., objects 114(1), 114(2), 114(3), and 114(4)). The scenario generator may then determine portions of the log data 104 associated with the objects. For example, the scenario generator may identify object 114(3) that is a building in the environment 106 represented in the log data at state 116. The scenario generator may identify object 114(3) represented in the log data 104 at states 120 and 124. In some examples, the log data 104 may include sensory data associated with the object 114(3) that identifies, for example, the range of the object 114(3). In some examples, the sensory data may include or be based on sensor data from a sensor or combination of sensors of the vehicle 102. In some examples, the log data 104 may include location data associated with the object 114(3) that identifies the location of the object 114(3) within the environment 106.

[0032] Using the perception data, the scenario generator may determine one or more log data associations 130 that identify associations between respective positions of the vehicle 102, respective times at which the vehicle 102 was located at the respective positions, and respective object data generated by the vehicle 102 when the vehicle 102 was located at the respective positions at the respective times. For example, “Position 1” of the log data association 130 may include first geolocation data (e.g., GPS coordinates) indicating a particular position from which the vehicle 102 perceived the environment 106, as shown in state 116, “Position 2” of the log data association 130 may include second geolocation data indicating a particular position from which the vehicle 102 perceived the environment 106, as shown in state 120, and “Position 3” of the log data association 130 may include third geolocation data indicating a particular position from which the vehicle 102 perceived the environment 106, as shown in state 124. Additionally, “t0” of log data association 130 may include a first timestamp indicating a particular instance in time when vehicle 102 is / was located at location 1, “t1” of log data association 130 may include a second timestamp indicating a particular instance in time when vehicle 102 is / was located at location 2, and “t2” of log data association 130 may include a third timestamp indicating a particular instance in time when vehicle 102 is / was located at location 3. Further, the "state 116 object data" of the logged data association 130 may include object data representing one or more objects perceived by the vehicle 102 in the environment 106 from position 1 at time t0, the "state 120 object data" of the logged data association 130 may include object data representing one or more objects perceived by the vehicle 102 in the environment 106 from position 2 at time t1, and the "state 124 object data" of the logged data association 130 may include object data representing one or more objects perceived by the vehicle 102 in the environment 106 from position 3 at time t2.

[0033] In some examples, the computing device 108 may determine instantiation attributes associated with an object included in the log data 104. The instantiation attributes may indicate how the vehicle 102 detected the object and / or an initial perception of the object. For purposes of illustration only, the object 114(2) may come into detection range of the vehicle 102 (e.g., as shown in state 120) by the vehicle 102 approaching the object 114(2) and / or the object 114(2) becoming visible (e.g., the object 114(2) may be behind an obscurant). In some examples, the detection range (also referred to as a perception threshold) may be associated with the capabilities of one or more sensors of the vehicle 102. In some examples, the detection range may be associated with a confidence value determined by a perception engine of the vehicle 102.

[0034] In some examples, computing device 108 may additionally or alternatively determine an ending attribute associated with an object included in log data 104. The ending attribute may indicate the manner in which vehicle 102 may no longer detect the object and / or an updated perception of the object. For purposes of illustration only, object 114(1) may be in a position outside of detection range of vehicle 102 due to vehicle 102 moving away from object 114(1) and / or object 114(1) being occluded (e.g., as shown in states 120 and 124).

[0035] Based on the log data 104 and / or the log data associations 130, the scenario generator can generate simulation instructions 112 that describe a simulated scenario including a simulated environment. For example, the log data 104 may include map data and position data indicative of the environment 106, as well as how the vehicle 102 traverses the environment 106. Additionally, the log data 104 may include status data of the vehicle 102 as it traverses the environment 106, including, but not limited to, acceleration data, steering angle data, indicator data, and the like. The log data 104 may include time data that associates actions and observations (e.g., sensor data) of the vehicle 102 with a set of time stamps. The log data 104 may be used by the scenario generator to generate simulation data that, when executed, includes a simulated environment indicative of the environment 106 and generates simulation instructions 112 that include simulated objects in various states (e.g., states 116, 120, and 124). For example, the simulation data may include a time stamp that indicates a time t x The state of the simulated environment 132 at 134, at time t x Time t after 134 y The state 136 of the simulated environment at 138 and at time t y Time t after 138 z The state 140 of the simulated environment at 142 may be represented.

[0036] In some examples, the scenario generator may create simulation instructions 112 that represent a simulated environment that includes more or less detail from the environment 106 as depicted in Figure 1 (e.g., by filtering objects, by representing objects with bounding boxes, and / or by adding additional objects). In some examples, the scenario generator may input the simulation instructions 112 to a vehicle controller associated with a simulated vehicle to determine how the simulated vehicle would have responded under circumstances presented in the simulated scenario.

[0037] FIG. 2 illustrates an example environment 200 in which the vehicle 102 perceives different objects within the environment and different perspectives of the different objects at different times.

[0038] Vehicle 102 may traverse an environment (e.g., environment 106 described in FIG. 1 above) according to a network of roads that includes one or more road segments, such as road segments 202(1), 202(2), and 202(3). Each one of road segments 202(1), 202(2), and 202(3) may be interconnected. That is, in some examples, road segment 202(1) may be connected to road segment 202(2), road segment 202(2) may be connected to road segment 202(3), etc. Additionally or alternatively, road segments 202(1), 202(2), and 202(3) may be connected to one another via other road segments not shown.

[0039] Vehicle 102 may traverse road segments 202(1), 202(2), and 202(3) in a first direction, as shown by timeline 204. For example, vehicle 102 may be located at a first position 206(1) at time t0118, at a second position 206(2) at a time t1122 after time t0118, and at a third position 206(3) at a time t2126 after time t1122. As such, states 116, 120, and 124 may represent the environment in which vehicle 102 is operating as perceived by vehicle 102 from positions 206(1), 206(2), and 206(3), and at times t0118, t1122, and t2126, respectively. That is, states 116, 120, and 124 may represent the environment as perceived by the vehicle 102, a sensor system of the vehicle 102, a perception system of the vehicle 102, a controller of the vehicle 102, etc. For purposes of illustration only, state 116 may represent the environment 106 at an initial time t0118 at 0 seconds, state 120 may represent the environment 106 at a time t1122 5 seconds later, and state 124 may represent the environment 106 at a time t2126 15 seconds later. Thus, in some examples, times t0118, t1122, t2126 may be non-contiguous (e.g., there is time elapsed between times t0118, t1122, and t2126) or times t0118, t1122, and t2126 may be continuous (e.g., there is no time elapsed between times t0118, t1122, and t2126).

[0040] 2, states 116, 120, and 124 may include different objects perceived by vehicle 102 at different times t0, t1, and t2, and from different positions 206(1), 206(2), and 206(3). For example, at time t0, when vehicle 102 is at first position 206(1), state 116 includes parked vehicles, buildings, street lights, and the like (e.g., objects 114(1), 114(3), and 114(4), respectively, of environment 106 shown in FIG. 1).

[0041] However, as vehicle 102 moves through the environment to second location 206(2), vehicle 102 may perceive objects as shown in state 120 including trees, buildings, and street lights (e.g., objects 114(2), 114(3), and 114(4), respectively, of environment 106 shown in FIG. 1 ). Additionally, the appearance of objects as seen from the perspective of vehicle 102 may change between state 116 and state 120 based on the change in position of vehicle 102 and the disappearance of some objects (e.g., parked vehicles). Furthermore, objects such as trees may appear in state 120 based on vehicle 102 moving to second location 206(2), which may be within range of the trees. That is, the sensor system of vehicle 102 may be within range of the trees as shown in state 120 to detect the presence of the trees. Finally, at time t2126, when the vehicle 102 moves through the environment to a third location 206(3), the vehicle 102 may perceive the object as shown in state 124.

[0042] 3 illustrates an example data transformation 300 performed at least in part by a vehicle perception system 302. The vehicle perception system 302 may output different representations of an environment (e.g., environment 106) in which a vehicle (e.g., vehicle 102) was operating based on different representations of the environment perceived by the vehicle from different positions at different times, as described herein.

[0043] For example, the vehicle perception system 302 may receive sensor data captured by a sensor system of the vehicle. The sensor data may include various states that represent an environment, such as states 116, 120, and 124. Each one of the states 116, 120, and 124 may represent the environment as seen from the position of the vehicle at a particular time. In an example, time points t0 118, t1 122, and t2 126 of the timeline 204 may be associated with states 116, 120, and 124, respectively.

[0044] By way of example and not limitation, the vehicle perception system 302 may receive sensor data representing state 116 of the environment at time t0118 and output a simulated representation of the environment shown in state 132. State 132 may include one or more bounding boxes representing objects seen by the vehicle at state 116. For example, the vehicle perception system 302 may apply one or more rules, heuristics, transformations, and / or the like to the sensor data to output the simulated representation shown in state 132. Additionally or alternatively, the vehicle perception system 302 may use one or more machine learning models to generate the simulated representation of state 132. The vehicle perception system 302 may perform similar or analogous processing on the sensor data representing states 120 and 124 to generate the simulated representations shown in states 136 and 140. In an example, the log data described herein may include data associated with any one of states 116, 120, and / or 124, as well as states 132, 136, and / or 140.

[0045] In various examples, states 132, 136, and 140 are x 134, t y 138 and t z In this way, the scenario generator, when executed by the simulator, may be associated with a time t x 134, t y 138 and t z The simulation instructions may generate simulation data representing the states 132, 136, and 140 of the simulated environment associated with time t142. By way of example and not limitation, when the simulated vehicle is in the same or proximate position as the vehicle that captured the sensor data illustrated in state 116 at time t118, the simulated environment as perceived by the simulated vehicle may be x This may correspond to state 132 at 134.

[0046] 4 is a pictorial flow diagram illustrating an example process 400 in which objects are instantiated in a simulated environment based on various positions of a simulated vehicle 408 relative to various previous positions of the vehicle 102. The vehicle 102 may capture log data on which the simulated environments 410(1), 410(2), and 410(3) are based.

[0047] In operation 402, process 400 may include generating a simulated scenario including simulated environment 410(1). The simulated scenario may be generated using the log data. In an example, whether simulated environment 410(1) includes or avoids including a particular simulated object may be based at least in part on a current position of simulated vehicle 408 and a previous position 206(1) of vehicle 102. For example, because the position of simulated vehicle 408 is closest to previous position 206(1) of vehicle 102 as opposed to positions 206(2) or 206(3), simulated environment 410(1) may include the simulated object shown in FIG. 4 with stippling. In some examples, one or more simulated objects of simulated environment 410(1) may represent actual objects perceived by vehicle 102 from previous position 206(1) at time t0112. That is, simulated environment 410(1) may include one or more simulated objects representing objects that were within range of a sensor system of vehicle 102 from position 206(1) at time t0112.

[0048] In operation 404, process 400 may include updating the simulated scenario to include simulated environment 410(2) based at least in part on the location of simulated vehicle 408 moving closest to vehicle 102's previous location 206(2) at time t1116. Simulated environment 410(2) includes updated appearances of objects previously included in simulated environment 410(1) based at least in part on simulated vehicle 408 changing its location. Additionally, simulated environment 410(2) includes fewer objects than were included in simulated environment 410(1) as well as additional objects not included in simulated environment 410(1) based on the change in location of simulated vehicle 408. For example, as vehicle 102 moves from location 206(1) to location 206(2), additional objects may have come into range or otherwise become detectable by a sensor system of vehicle 102. Similarly, objects that were once detectable by vehicle 102 may have moved out of range or otherwise become undetectable. Thus, the simulated environment 410(2) may be updated to take into account these changes that the vehicle 102 would have perceived in the real world at time t1116.

[0049] In operation 406, process 400 may include updating the simulated scenario to include simulated environment 410(3) based at least in part on the location of simulated vehicle 408 moving closest to vehicle 102's previous location 206(3) at time t2120. Simulated environment 410(3) includes updated appearances of objects previously included in simulated environments 410(1) and / or 410(2) based at least in part on simulated vehicle 408's repositioned location. Additionally, simulated environment 410(3) includes fewer objects than were included in simulated environments 410(1) and / or 410(2) based on the change in location of simulated vehicle 408. For example, when vehicle 102 moves from location 206(2) to location 206(3), objects that were once detectable by vehicle 102 may have moved out of range or otherwise become undetectable. Thus, the simulated environment 410(3) may be updated to take into account these changes that the vehicle 102 would have perceived in the real world at time t2120.

[0050] It will be understood that more or fewer locations than shown in Figure 4 (e.g., locations 206(1), 206(2), and 206(3)) may be used. For example, Figure 4 includes three locations and three simulated environments, but any number of locations and / or simulated environments may be possible to create a realistic simulation.

[0051] 5 is a pictorial flow diagram illustrating an example process 500 for avoiding updating a simulated environment based on a simulated vehicle 508 that remains stationary within the simulated environment relative to a previous position 206(1) of the vehicle 102. The vehicle 102 may capture log data on which the simulated environment 510 is based.

[0052] In operation 502, process 500 may include generating a simulated scenario including simulated environment 510. The simulated scenario may be generated using the log data. In an example, whether simulated environment 510 includes or avoids including a particular simulated object may be based at least in part on a current position of simulated vehicle 508 and a previous position 206(1) of vehicle 102. For example, because the position of simulated vehicle 508 is closest to previous position 206(1) of vehicle 102 as opposed to positions 206(2) or 206(3), simulated environment 510 may include the simulated object shown in FIG. 5 with stippling. In some examples, one or more simulated objects of simulated environment 510 may represent actual objects perceived by vehicle 102 from previous position 206(1) at time t0112. That is, the simulated environment 510 may include one or more simulated objects representing objects that were within range of a sensor system of the vehicle 102 from the position 206(1) at time t0112.

[0053] In operation 504, process 500 includes avoiding updating the simulated environment 510 of the simulated scenario. Avoiding updating the simulated environment in operation 504 may be based at least in part on the position of the simulated vehicle 508 remaining stationary. That is, the simulated environment is not updated to a view from a different position because the simulated vehicle 508 is still closest to the previous position 206(1) of the vehicle 102 at time t0112. Thus, instead of updating the simulated environment 510 to match the perspective seen by the vehicle 102 from position 206(2) at time t0116, the simulator may recycle the simulated environment 510 so that the environment appears stationary to the simulated vehicle. Similarly, in operation 506, process 500 includes avoiding updating the simulated environment 510 of the simulated scenario based at least in part on the position of the simulated vehicle 508 remaining stationary.

[0054] 6 is a block diagram illustrating an example system 600 for implementing some of the various techniques described herein. In at least one example, the example system 600 includes a vehicle 602, which may be similar to the vehicle 102 described above with reference to Figures 1-5. In the illustrated example system 600, the vehicle 602 is an autonomous vehicle, although the vehicle 602 may be any other type of vehicle.

[0055] The vehicle 602 may be a driverless vehicle, such as an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions of the entire journey without the expectation that a driver (or passenger) is in control of the vehicle at all times. In such an example, the vehicle 602 may be configured to control all functions from the beginning to the completion of the journey, including all parking functions, and therefore may not include a driver and / or controls for driving the vehicle 602, such as a steering wheel, accelerator pedal, and / or brake pedal. This is by way of example only, and the systems and methods described herein may be incorporated into any land-based, air-based, or water-based vehicle, including vehicles that require manual control by a driver at all times, to vehicles that are partially or fully autonomously controlled.

[0056] Vehicle 602 may be a vehicle of any configuration, such as, for example, a van, a sport utility vehicle, a crossover vehicle, a truck, a bus, an agricultural vehicle, and / or a construction vehicle. Vehicle 602 may be powered by one or more internal combustion engines, one or more electric motors, hydrogen power, any combination thereof, and / or any other suitable power source. Vehicle 602 has four wheels, although the systems and methods described herein may be incorporated into vehicles having a fewer or greater number of wheels and / or tires. Vehicle 602 may have four-wheel steering and may operate with generally equal or similar performance characteristics in all directions, such that, for example, a first end of vehicle 602 is the front end of vehicle 602 when traveling in a first direction, and the first end is the rear end of vehicle 602 when traveling in the opposite direction. Similarly, a second end of vehicle 602 is the front end of vehicle 602 when traveling in a second direction, and the second end is the rear end of vehicle 602 when traveling in the opposite direction. These exemplary characteristics may facilitate greater maneuverability in tight spaces or crowded environments, such as parking lots and / or urban areas, for example.

[0057] Vehicle 602 may include a computing device 604, one or more sensor systems 606, one or more emitters 608, one or more communication connections 610 (also referred to as communication devices and / or modems), at least one direct connection 612 (e.g., for physically coupling with vehicle 602 to exchange data and / or provide power), and one or more drive systems 614. The one or more sensor systems 606 may be configured to capture sensor data associated with the environment.

[0058] The sensor system 606 may include time-of-flight sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement unit (IMU) accelerometers, magnetometers, gyroscopes, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, illuminance, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, etc. The sensor system 606 may include multiple instances of each of these or other types of sensors. For example, the time-of-flight sensors may include individual time-of-flight sensors located at the corners, front, rear, sides, and / or top of the vehicle 602. As another example, the camera sensors may include multiple cameras positioned at various locations about the exterior and / or interior of the vehicle 602. The sensor system 606 may provide input to the computing device 604.

[0059] The vehicle 602 may also include one or more emitters 608 for emitting light and / or sound. The one or more emitters 608 in this example include interior audio and visual emitters for communicating with occupants of the vehicle 602. By way of example and not limitation, the interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The one or more emitters 608 in this example may also include exterior emitters. By way of example and not limitation, the exterior emitters in this example include lights for indicating direction of travel or other indicators of the vehicle's actions (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters for audibly communicating with pedestrians or other nearby vehicles (e.g., speakers, speaker arrays, horns, etc.), one or more of which may be configured with acoustic beam steering technology.

[0060] Vehicle 602 may also include one or more communications connections 610 that enable communication between vehicle 602 and one or more other local or remote computing devices (e.g., remote teleoperated computing devices) or remote services. For example, communications connections 610 may facilitate communication with other local computing devices on vehicle 602 and / or drive system 614. Communications connections 610 may also enable vehicle 602 to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.).

[0061] The communications connections 610 may include physical and / or logical interfaces for connecting the computing device 604 to another computing device or to one or more external networks (e.g., the Internet). For example, the communications connections 610 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), satellite communications, short-range communications (DSRC), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices.

[0062] In at least one example, the vehicle 602 may include one or more drive systems 614. In some examples, the vehicle 602 may have a single drive system 614. In at least one example, if the vehicle 602 has multiple drive systems 614, the individual drive systems 614 may be positioned at opposite ends of the vehicle 602 (e.g., front and rear, etc.). In at least one example, the drive system 614 may include one or more sensor systems 606 for detecting conditions surrounding the drive system 614 and / or the vehicle 602. By way of example and not limitation, the sensor systems 606 may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive system, cameras or other image sensors, ultrasonic sensors for acoustically detecting objects surrounding the drive system, lidar sensors, radar sensors, etc. Some sensors, such as the wheel encoders, may be intrinsic to the drive system 614. In some examples, the sensor system 606 on the drive system 614 may overlap or complement a corresponding system (e.g., the sensor system 606) on the vehicle 602.

[0063] The drive system 614 may include many vehicle systems including a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for braking force distribution to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the vehicle's exterior environment), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, the drive system 614 may include a drive system controller that may receive and pre-process data from sensors and control the operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more modules and perform various functions of the drive system 614. Additionally, drive system 614 may also include one or more communication connections that enable the respective drive system to communicate with one or more other local or remote computing devices.

[0064] The computing device 604 may be similar to the vehicle computing device described above with reference to FIG. 1. The computing device 604 may include one or more processors 618 and a memory 620 communicatively coupled to the one or more processors 618. In the illustrated example, the memory 620 of the computing device 604 stores a localization component 622, a perception component 624, a prediction component 626, a planning component 628, and one or more system controllers 630. While depicted as residing in the memory 620 for illustrative purposes, it is contemplated that the localization component 622, the perception component 624, the prediction component 626, the planning component 628, and the one or more system controllers 630 may additionally or alternatively be accessible to the computing device 604 (e.g., stored in a different component of the vehicle 602) and / or accessible to the vehicle 602 (e.g., stored remotely).

[0065] In the memory 620 of the computing device 604, the localization component may include functionality to receive data from the sensor system 606 to determine the position of the vehicle 602. For example, the localization component 622 may include and / or request / receive a three-dimensional map of the environment and continuously determine the location of the autonomous vehicle within the map. In some examples, the localization component 622 may receive time-of-flight data, image data, lidar data, radar data, sonar data, IMU data, GPS data, wheel encoder data, or any combination thereof, etc., using SLAM (simultaneous localization and mapping) or CLAMS (calibration, localization and mapping, simultaneously) to accurately determine the location of the autonomous vehicle. In some examples, the localization component 622 may provide data to various components of the vehicle 602 to determine an initial position of the autonomous vehicle for generating a trajectory, as discussed herein. In at least one example, the location component 622 may determine the location of the simulated vehicle within the simulated environment and / or may determine the closest previous location of the vehicle that recorded the logged data on which the simulated environment is based.

[0066] The perception component 624 may include functionality to perform object detection, segmentation, and / or classification. In some examples, the perception component 624 may provide processed sensor data indicative of the presence of an entity in proximity to the vehicle 602 and / or the classification of the entity as a type of entity (e.g., car, pedestrian, bicyclist, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional and / or alternative examples, the perception component 624 may provide processed sensor data indicative of one or more characteristics associated with the detected entity and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity may include, but are not limited to, x-position (global position), y-position (global position), z-position (global position), orientation, type of entity (e.g., classification), speed of the entity, range (size) of the entity, etc. Characteristics associated with the environment may include, but are not limited to, the presence of another entity in the environment, the state of another entity in the environment, time of day, day of the week, season, weather, indication of darkness / lightness, etc. In at least one example, the perception component 624 may generate log data associated with a vehicle traversing an environment, as described herein. The log data may be used to generate simulated scenarios, including simulated environments for testing the vehicle, among other things.

[0067] The perception component 624 may include functionality to store the perception data generated by the perception component 624. In some examples, the perception component 624 may determine a trajectory corresponding to an object classified as a type of object. By way of example only, the perception component 624 may capture one or more images of the environment using the sensor system 606. The sensor system 606 may capture images of the environment including objects such as buildings, vehicles, trees, street lights, pedestrians, etc.

[0068] The stored sensory data may, in some examples, include fused sensory data captured by the vehicle. The fused sensory data may include a fusion or other combination of sensor data from sensor systems 606, such as image sensors, lidar sensors, radar sensors, time-of-flight sensors, sonar sensors, global positioning system sensors, interior sensors, and / or any combination thereof. The stored sensory data may additionally or alternatively include classification data including semantic classifications of objects (e.g., pedestrians, vehicles, buildings, road surfaces, etc.) represented in the sensor data.

[0069] The prediction component 626 may generate one or more probability maps that represent predicted probabilities of possible locations of one or more objects in the environment. For example, the prediction component 626 may generate one or more probability maps for vehicles, pedestrians, animals, etc., that are within a threshold distance from the vehicle 602. In some examples, the prediction component 626 may measure the trajectories of the objects and generate discretized predicted probability maps, heat maps, probability distributions, discretized probability distributions, and / or trajectories of the objects based on the observed and predicted behavior. In some examples, the one or more probability maps may represent the intent of one or more objects in the environment.

[0070] The planning component 628 may determine a path for the vehicle 602 to follow to traverse an environment. For example, the planning component 628 may determine various routes and paths, as well as various levels of detail. In some examples, the planning component 628 may determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this discussion, the route may be a sequence of waypoints for traveling between the two locations. As non-limiting examples, the waypoints include streets, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, the planning component 628 may generate instructions for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 628 may determine how to guide the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions may be a path, or a portion of a path. In some examples, the diverse paths may be generated substantially simultaneously (i.e., within technical tolerances) according to a receding horizon technique. One path of the diverse paths within the receding data horizon that has the highest confidence level may be selected for maneuvering the vehicle.

[0071] In other examples, the planning component 628 may alternatively or additionally use data from the perception component 624 to determine a path for the vehicle 602 to follow to traverse the environment. For example, the planning component 628 may receive data from the perception component 624 regarding objects associated with the environment. Using this data, the planning component 628 may determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location) to avoid the objects in the environment. In at least some examples, such a planning component 628 may determine that there is no such collision-free path and, as a result, provide a path that avoids all collisions and / or otherwise mitigates damage and brings the vehicle 602 to a safe stop.

[0072] In at least one example, the computing device 604 may include one or more system controllers 630 that may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 602. These system controllers 630 may communicate with and / or control systems corresponding to the drive system 614 and / or other components of the vehicle 602, which may be configured to operate according to a path provided from the planning component 628.

[0073] The vehicle 602 may connect to the computing device 632 via the network 616 and may include one or more processors 634 and a memory 636 communicatively coupled to the one or more processors 634. In at least one example, the one or more processors 634 may be similar to the processor 618 and the memory 636 may be similar to the memory 620. In the illustrated example, the memory 636 of the computing device 632 stores a log data component 638, an object component 640, a scenario component 642, and a simulation component 644. While depicted as residing in the memory 636 for illustrative purposes, it is contemplated that the log data component 638, the object component 640, the scenario component 642, and the simulation component 644 may additionally or alternatively be accessible to the computing device 632 (e.g., stored in a different component of the computing device 632) and / or may be accessible to the computing device 632 (e.g., stored remotely).

[0074] In the memory 636 of the computing device 632, the log data component 638 may determine the log data used to generate the simulated scenario. As discussed above, a database may store one or more log data. In some examples, the computing device 632 may act as a database that stores the log data. In some examples, the computing device 632 may connect to another computing device to access the log data database. The log data component 638 may scan the log data stored in the database and identify log data that includes an event of interest. As discussed above, in some examples, the log data may include an event marker that indicates that an event of interest is associated with the particular log data. In some examples, a user may select the log data of the set of log data used to generate the simulated scenario.

[0075] Additionally, the object component 640 may identify one or more objects associated with the log data. For example, the object component 640 may determine objects, such as vehicles, pedestrians, animals, bicyclists, trees, buildings, street lights, barriers, fences, etc., in the log data and represent them as simulated objects. In some examples, the object component 640 may determine objects that interact with the vehicle associated with the log data. In some examples, the object component 640 may determine attributes associated with the object. For example, the attributes may include the object's range (also referred to as the object's size, which may represent length, width, height, and / or volume), the object's classification, the object's pose, the object's trajectory, the object's waypoints, the object's instantiation attributes, and / or the object's termination attributes.

[0076] The range of an object may indicate the size and / or volume of the object (e.g., 5 meters long). The classification of an object may indicate the classification and / or type of object (e.g., vehicle, pedestrian, bicyclist, etc.). The pose of an object may indicate the xyz coordinates (e.g., position or position data) of the object within the environment and / or may include the pitch, roll, yaw associated with the object. The trajectory of an object may indicate the path followed by the object. The waypoint of an object may indicate a position within the environment indicating a route between two positions.

[0077] An instantiation attribute of an object may indicate how a vehicle detects the object. For example, an object may enter detection range of the vehicle by (1) the vehicle approaching the object, (2) the object approaching the vehicle, and / or (3) the object becoming visible (unoccluded) or occlusion of the object ceasing. An exit attribute of an object may indicate how a vehicle no longer detects the object. For example, an object may be in a position outside detection range of the vehicle by (1) the vehicle moving away from the object, (2) the object moving away from the vehicle, and / or (3) the object becoming occluded or occlusion of the object.

[0078] The scenario component 642 may generate a simulated scenario using the log data identified by the log data component 638. The simulated scenario may include a simulated environment (e.g., roads, road signs, buildings, etc.) and simulated objects (e.g., other vehicles, buildings, trees, street lights, barriers, fences, pedestrians, bicyclists, animals, etc.). In some examples, the scenario component 642 may apply simulated object models to the simulated objects using the perception data generated by the perception component 624. For example, the scenario component 642 may identify a simulated vehicle model and apply it to a simulated object associated with the vehicle. In some examples, the scenario component 642 may identify a simulated building model and apply it to a simulated object associated with the building.

[0079] By way of example and not limitation, the log data may include objects of various sizes, such as mailboxes, trees, buildings, and / or the like. The scenario component 642 may use a volume-based filter such that objects associated with a volume equal to or greater than a threshold volume of 3 cubic meters, such as buildings, are represented in the simulated scenario, and objects associated with a volume less than 3 cubic meters, such as mailboxes, are not represented in the simulated scenario. In some examples, the scenario component 642 may use a motion-based filter such that objects associated with a movement or trajectory in accordance with the log data are represented in or removed from the simulated scenario. In some examples, the filters may be applied in combination or in a mutually exclusive manner.

[0080] In some examples, the scenario component 642 may filter objects that do not meet or exceed a confidence threshold. By way of example and not limitation, the log data may indicate that an object is associated with a pedestrian classification attribute and that the confidence value associated with the classification is 5%. The scenario component 642 may have a confidence threshold of 75% and may filter objects based on confidence values ​​that do not meet or exceed the confidence threshold. In some examples, a user may provide a user-created filter that includes one or more attribute thresholds such that the scenario component 642 may filter objects that do not meet or exceed one or more attribute thresholds indicated by the user-created filter.

[0081] The simulation component 644 may execute simulated scenarios as a set of simulation instructions and generate simulation data. In some examples, the simulation component 644 may execute multiple simulated scenarios simultaneously and / or in parallel. This may allow a user to edit the simulated scenarios and execute an array of simulated scenarios with variations between each simulated scenario. The simulation component 644, in at least one example, tests a vehicle controller by simulating scenarios.

[0082] Additionally, the simulation component 644 may determine outcomes for simulated scenarios. For example, the simulation component 644 may execute the scenarios for use in simulations for testing and validation. The simulation component 644 may generate simulation data indicative of how the autonomous controllers performed (e.g., responded), compare the simulation data to predetermined outcomes, and / or determine if any predetermined rules / assertions were violated / triggered.

[0083] In some examples, the predefined rules / assertions may be based on a simulated scenario (e.g., a traffic rule regarding a crosswalk may be enabled based on a crosswalk scenario, or a traffic rule regarding crossing lane markers may be disabled for a stopped vehicle scenario). In some examples, the simulation component 644 may dynamically enable and disable rules / assertions as the simulation progresses. For example, rules / assertions related to the school zone may be enabled as the simulated object approaches a school zone and disabled as the simulated object leaves the school zone. In some examples, the rules / assertions may include comfort metrics related to, for example, how fast the object accelerates out of the simulated scenario.

[0084] Based at least in part on determining that the autonomous controller performed actions consistent with a predetermined outcome (i.e., the autonomous controller did everything it was supposed to do) and / or determining that no rules were broken or assertions were triggered, the simulation component 644 may determine that the autonomous controller was successful. Based at least in part on determining that the autonomous controller's performance was inconsistent with a predetermined outcome (i.e., the autonomous controller did something it was not supposed to do) and / or determining that a rule was broken or assertions were triggered, the simulation component 644 may determine that the autonomous controller failed. Thus, based at least in part on executing simulated scenarios, the simulation data may indicate how the autonomous controller responded to each simulated scenario, as described above, and based at least in part on the simulation data, a successful or unsuccessful outcome may be determined.

[0085] The processor 618 of the computing device 604 and the processor 634 of the computing device 632 may be any suitable processor capable of processing data and executing instructions to perform the operations described herein. By way of example and not limitation, the processors 618 and 634 may comprise one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it into other electronic data that may be stored in registers or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors to the extent that they are configured to implement coded instructions.

[0086] The memory 620 of the computing device 604 and the memory 636 of the computing device 632 are examples of non-transitory computer-readable media. The memory 620 and 636 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functions attributed to the various systems described herein. In various implementations, the memory 620 and 636 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, the ones shown in the accompanying drawings of which are merely examples relevant to the description herein.

[0087] In some examples, some or all aspects of the components discussed herein may include any models, algorithms, and / or machine learning algorithms. For example, in some examples, the components in memories 620 and 636 may be implemented as neural networks.

[0088] 7 and 8 are flow charts illustrating an example method of instantiating objects in a simulated environment based on log data. The methods illustrated in FIGS. 7 and 8 are described with reference to one or more of the vehicles and / or systems described in FIGS. 1 through 6 for convenience and ease of understanding. However, the methods illustrated in FIGS. 7 and 8 are not limited to being performed using the vehicles, systems, and / or techniques described in FIGS. 1 through 6, but may be implemented using any of the other vehicles, systems, and techniques described herein, as well as vehicles, systems, and techniques other than the vehicles, systems, and techniques described herein. Additionally, the vehicles, systems, and user interfaces described herein are not limited to performing the methods illustrated in FIGS. 7 and 8.

[0089] Methods 700 and 800 are illustrated as a collection of blocks in a logical flow graph, which represent sequences of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as limiting, and any number of the described blocks can be combined in any order and / or in parallel to implement a process. In some embodiments, one or more blocks of processing may be omitted entirely. Additionally, methods 700 and 800 may be combined in whole or in part with each other or in other ways.

[0090] FIG. 7 is a flow chart illustrating an example method 700 for updating a simulated environment to include objects based on the position of a simulated vehicle relative to a previous position of the vehicle in the real world.

[0091] The method 700 begins at operation 702 with receiving log data representative of a real environment in which a real vehicle was operating for a period of time. For example, the computing device 108 may receive the log data from one or more vehicles 102 over one or more networks 128. In some examples, the log data may be generated by a perception system of the vehicle based at least in part on sensor data captured by one or more sensors of the vehicle. In at least one example, the log data includes multiple instances of log data representative of an environment in which the vehicle and / or another vehicle was operating during multiple periods of time.

[0092] At operation 704, the method 700 includes generating a simulated scenario for testing a controller of the autonomous vehicle using the log data, the simulated scenario including a simulated environment that represents at least a portion of the real environment. For example, the computing device 108 may generate a simulated scenario that includes a simulated environment that represents a portion of the environment 106.

[0093] At operation 706, method 700 includes causing, by a controller of the autonomous vehicle, the simulated autonomous vehicle to traverse the simulated environment. In various examples, a simulated path of the simulated autonomous vehicle in / through the simulated environment may deviate from a real path of the real vehicle in / through the real environment. In at least one example, the simulated scenario, simulated environment, simulated objects, and / or the like may be input to the controller of the autonomous vehicle in the form of simulation instructions. That is, the simulation instructions may represent the simulated scenario, simulated environment, simulated objects, and / or the like.

[0094] At operation 708, the method 700 includes determining a simulated position of the simulated autonomous vehicle within the simulated environment. For example, the computing device 108 may receive geolocation data from the simulated vehicle based on a position of the simulated vehicle within the simulated environment, and the simulated position of the simulated vehicle may be determined based at least in part on the geolocation data. At operation 710, the method 700 includes determining a first time during a period of time at which a real position of the real vehicle in the real environment most closely matched the simulated position of the simulated autonomous vehicle in the simulated environment based at least in part on the log data. For example, the computing device 108 may determine a first time (e.g., t0112) at which the previous position 206(1) of the vehicle 102 was closest to the position of the simulated vehicle 508. Additionally or alternatively, the computing device 108 may determine the first time based at least in part on the log data associations 130.

[0095] At operation 712, the method includes obtaining object data from the log data representing a real object in the real environment perceived by the real vehicle at a first time. For example, the computing device 108 may obtain the object data from the log data associations 130 based at least in part on which previous location of the vehicle 102 the simulated vehicle is closest to. At operation 716, the method 700 includes generating a simulated object in the simulated environment representing the real object such that the simulated vehicle perceives the simulated object. For example, the computing device 108 may instantiate the simulated object within the simulated environment at a location where the vehicle 102 perceived the object. In some examples, whether the simulated object is generated within the simulated environment may be based at least in part on determining a classification of the real object. That is, if the real object is a dynamic object, the simulated object may not be generated. However, if the real object is a static object, the simulated object may be generated and / or instantiated within the simulated environment.

[0096] At operation 716, the method 700 includes determining whether to end the simulation. For example, if the simulation is at the end of the log data file on which the simulation is based, the computing device 108 may determine to end the simulation and proceed to operation 718. However, if the simulation is not yet at the end of the log data file or if the simulation continues, the method 700 may loop back to operation 708. In this manner, operations 708 through 716 may continue until the end of the simulation is reached, and the simulated environment continues to be updated based on the simulated position of the simulated vehicle. At operation 718, if the simulation is ended, the method 700 includes determining whether the controller of the autonomous vehicle succeeded in the simulated test. For example, the computing device 108 may determine whether the simulated vehicle of the simulation safely, effectively traversed the simulated environment, etc.

[0097] FIG. 8 is a flow chart illustrating another example method 800 for updating a simulated environment to include objects based on the position of a simulated vehicle relative to a previous position of the vehicle in the real world.

[0098] The method 800 begins at operation 802 with receiving log data representative of a real environment in which a real vehicle was operating. For example, the computing device 108 may receive the log data from one or more vehicles 102 over one or more networks 128. In some examples, the log data may be generated by a perception system of the real vehicle based at least in part on sensor data captured by one or more sensors of the real vehicle. In at least one example, the log data includes multiple instances of log data representative of an environment in which the vehicle and / or another vehicle was operating during multiple periods of time.

[0099] At operation 804, the method 800 may include using the log data to generate a simulated environment for testing the vehicle controller, where the simulated environment represents the real environment. For example, the computing device 108 may generate simulation instructions 112 including the simulated environment. At operation 806, the method 800 may include determining a simulated position of the simulated vehicle as it traverses the simulated environment. The simulated vehicle may be controlled by the vehicle controller. Additionally, in some examples, a simulated pose of the simulated vehicle in the simulated environment may deviate from a real pose of the real vehicle in the real environment (e.g., the simulated vehicle is oriented 90 degrees from north, while the real vehicle is oriented 93 degrees from north). In some examples, the computing device 108 may receive geolocation data from the simulated vehicle based on a position of the simulated vehicle within the simulated environment, and a simulated position of the simulated vehicle may be determined based at least in part on the geolocation data.

[0100] At operation 808, method 800 may include determining a real position of the real vehicle in the real environment associated with the simulated position of the simulated vehicle in the simulated environment based at least in part on the log data. For example, computing device 108 may determine the real position of the real vehicle. Additionally, computing device 108 may determine the real position of the real vehicle based at least in part on receiving geolocation data associated with the simulated position of the simulated vehicle in the simulated environment.

[0101] At operation 810, method 800 may include generating a simulated object in the simulated environment that represents the real object in the real environment as perceived by the real vehicle from the real location. For example, computing device 108 may instantiate the simulated object at a simulated location in the simulated environment that matches the real-world location of the real object in the real environment.

[0102] At operation 812, the method 800 may include determining whether to end the simulation. For example, if the simulation is at the end of the log data file on which the simulation is based, the computing device 108 may determine to end the simulation and proceed to operation 814. However, if the simulation is not yet at the end of the log data file or if the simulation continues, the method 800 may loop back to operation 806. In this manner, operations 806 through 812 may continue until the end of the simulation is reached, and the simulated environment continues to be updated based on the simulated position of the simulated vehicle. At operation 814, if the simulation is ended, the method 800 includes determining whether the simulated vehicle passed the simulated test. For example, the computing device 108 may determine whether the simulated vehicle of the simulation traversed the simulated environment safely, effectively, etc.

[0103] Example clauses A. one or more processors; and a method, when executed by the one or more processors, for receiving log data representative of a real environment in which a real vehicle was operating over a period of time; generating simulated scenarios using the log data to test a controller of an autonomous vehicle, the simulated scenarios including a simulated environment representing at least a portion of the real environment; causing a simulated autonomous vehicle, by a controller of the autonomous vehicle, to traverse the simulated environment, where a path of the simulated autonomous vehicle in the simulated environment deviates from a path of the real vehicle in the real environment; and testing the simulated environment. and one or more non-transitory computer-readable media storing instructions that cause a system to perform operations including: determining a simulated position of a simulated autonomous vehicle; determining a first time during the period at which a real position of the real vehicle in the real environment most closely matched the simulated position of the errant simulated autonomous vehicle in the simulated environment based at least in part on the logged data; obtaining object data from the logged data that represents a real object in the real environment as perceived by the real vehicle at the first time; and generating a simulated object in the simulated environment that represents the real object.

[0104] B. The system of paragraph A, further including at least one of: the simulated location includes a first simulated location, the real location includes a first real location, and the operation includes determining a second simulated location of the simulated autonomous vehicle within the simulated environment, the second simulated location being different from the first simulated location; determining a second time during the period that a second real location of the real vehicle in the real environment most matched the second simulated location of the departing simulated autonomous vehicle in the simulated environment based at least in part on the log data; and updating the simulated object to match the real object in the real environment as perceived by the real vehicle at the second time or generating in the simulated environment a second simulated object representing the second real object in the real environment as perceived by the real vehicle at the second time.

[0105] C. The system of any one of paragraphs A or B, wherein the simulated objects include a first simulated object, and the operation further includes avoiding generating in the simulated environment a second simulated object representing a second real object in the real environment as perceived by the real vehicle at a second time after the first time based at least in part on at least one of the simulated position of the simulated autonomous vehicle or the first time.

[0106] D. The system of any one of paragraphs A to C, wherein the operations further include determining whether the object data representing the real object is associated with a static object or a dynamic object, and generating the simulated object representing the real object is based at least in part on determining that the object data is associated with a static object.

[0107] E. The system of any one of paragraphs A-D, wherein a simulated pose of the simulated autonomous vehicle in the simulated environment deviates from a real pose of the real vehicle in the real environment.

[0108] F. A method comprising: receiving log data representative of a first environment in which a first vehicle was operating; generating a simulated environment for testing a vehicle controller using the log data, the simulated environment representing a real first environment; determining a position of a second vehicle as it traverses the simulated environment, the second vehicle being controlled by the vehicle controller, wherein a path of the second vehicle in the simulated environment deviates from a path of the first vehicle in the first environment; determining a position of the first vehicle in the first environment associated with the deviated position of the second vehicle in the simulated environment based at least in part on the log data; and generating a simulated object in the simulated environment representing a first object in the first environment as perceived by the first vehicle from the position.

[0109] G. The method of paragraph F, further comprising: determining that the second vehicle changes position within the simulated environment; and updating the simulated environment to include another simulated object representing a second object in the first environment as perceived by the first vehicle from a second position associated with the changed position of the second vehicle.

[0110] H. The method of any one of paragraphs F or G, wherein the log data is associated with a period during which the first vehicle was operating in the first environment, the method further comprising: determining a first time during the period during which the first vehicle was located at the location based at least in part on the log data; and obtaining object data from the log data representing the first object in the first environment as perceived by the first vehicle at the first time, wherein the simulated object is generated based at least in part on the object data.

[0111] I. The method of any one of paragraphs F to H, wherein determining the position of the first vehicle includes determining the position of the first vehicle in the first environment that most closely matches a position of the second vehicle in the simulated environment.

[0112] J. updating simulation instructions to include the simulated environment and inputting the simulation instructions to the vehicle controller such that the vehicle controller controls the second vehicle to traverse the simulated environment; The method of any one of paragraphs F through I, further comprising:

[0113] K. The method of any one of paragraphs F to J, wherein the log data includes either simulated sensor data or actual sensor data captured by a sensor of the first vehicle.

[0114] L. The method of any one of paragraphs F through K, further comprising controlling, by the vehicle controller, the second vehicle based at least in part on perceiving the simulated object.

[0115] M. The method of any one of paragraphs F through L, further comprising: receiving an input indicating a user-specified location within the simulated environment that contains another simulated object; and generating the other simulated object in the simulated environment at the user-specified location.

[0116] N. The method of any one of paragraphs F to M, wherein the log data includes multiple instances of log data representative of the first environment in which the first vehicle or a third vehicle was operating during multiple periods of time, and determining the position of the second vehicle is based at least in part on the multiple instances of log data.

[0117] O. The method of any one of paragraphs F to N, wherein determining whether the first object in the first environment is a static object or a dynamic object and generating the simulated object in the simulated environment is based at least in part on determining that the first object is a static object.

[0118] P. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause one or more processors to perform operations including receiving log data representative of a first environment in which a first vehicle was operating; generating a simulated environment using the log data for testing a vehicle controller, the simulated environment representing a real first environment; determining a position of a second vehicle as it traverses the simulated environment, the second vehicle being controlled by the vehicle controller, wherein a pose of the second vehicle in the simulated environment deviates from a pose of the first vehicle in the first environment; determining a position of the first vehicle in the first environment associated with the deviated position of the second vehicle in the simulated environment based at least in part on the log data; and generating a simulated object in the simulated environment representing a first object in the first environment as perceived by the first vehicle from the position.

[0119] Q. The one or more non-transitory computer-readable media of paragraph P, wherein the operation further comprises determining that the second vehicle changes position within the simulated environment and updating the simulated environment to include another simulated object representing a second object in the first environment as perceived by the first vehicle from a second position associated with the changed position of the second vehicle.

[0120] R. The one or more non-transitory computer-readable media of any one of paragraphs P or Q, wherein the log data is associated with a period of time during which the first vehicle was operating in the first environment, and the operation further comprises determining a first time during the period during which the first vehicle was located at the location based at least in part on the log data, and obtaining object data from the log data representing the first object in the first environment as perceived by the first vehicle at the first time.

[0121] S. One or more non-transitory computer-readable media described in any one of paragraphs P to R, wherein determining the position of the first vehicle includes determining the position of the first vehicle in the first environment that most closely matches a position of the second vehicle in the simulated environment.

[0122] T. The one or more non-transitory computer-readable media of any one of paragraphs P to S, wherein the operations further include receiving input indicating a user-specified location within the simulated environment where another simulated object is to be generated; and generating the other simulated object in the simulated environment at the user-specified location.

[0123] U. receiving multiple log data files representing a real environment in which a real vehicle was operating during a period of time, the multiple log file data representing different objects in the real environment as perceived by the real vehicle from different positions along a path traversed by the real vehicle at different instances of time during the period in which the vehicle was traversing the path; presenting a first log data file of the multiple log data files to a vehicle controller to simulate a scenario for testing the vehicle controller, the first log data file representing objects in the real environment as perceived by the real vehicle from a first position along the path in which the real vehicle was located at a first instance of time; causing a simulated vehicle, by the vehicle controller, to traverse a simulated environment representing the real environment based on the first log data file; and causing the vehicle controller to control the simulated vehicle to traverse a simulated environment representing the real environment. determining to move the real vehicle from a first position in the simulated environment to a second position in the simulated environment, the first position corresponding to the first position; determining that the second position deviates from the path traversed by the vehicle; determining a second position along the path where the real vehicle was located at a second instance of time based at least in part on determining that the second position deviates from the path, the second position most corresponding to the second position; identifying a second log data file of the various log data files, the second log data file representing the object in the real environment as perceived by the real vehicle from the second position along the path where the real vehicle was located at the second instance of time; and presenting the second log data file to the vehicle controller.

[0124] Although the above-mentioned exemplary clauses are described with respect to one particular implementation, in the context of this specification, it should be understood that the contents of the exemplary clauses can also be implemented via a method, a device, a system, a computer-readable medium, and / or another implementation. Furthermore, any of the example AUs can be implemented alone or in combination with any one or more of the other example AUs.

[0125] conclusion One or more examples of the technology described herein have been described; however, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0126] In the illustrative description, reference is made to the accompanying drawings, which form a part hereof, showing by way of example specific examples of the claimed subject matter. It is to be understood that other examples can be used and modifications or substitutions, such as structural changes, can be made. Such examples, modifications or substitutions do not necessarily depart from the intended scope of the claimed subject matter. Although the steps herein may be presented in a particular order, in some cases the order can be changed, such as providing certain inputs at different times or in a different order, without changing the functionality of the described systems and methods. The disclosed procedures can also be performed in different orders. In addition, the various calculations herein need not be performed in the order disclosed, and other examples using alternative orders of calculations can be easily implemented. In addition to reordering, calculations can also be decomposed into sub-calculations that produce the same results.

Claims

1. one or more processors; When executed by the one or more processors, receiving log data representative of a real environment in which a real vehicle was operating for a period of time; using the logged data to generate simulated scenarios for testing a controller of an autonomous vehicle, the simulated scenarios including a simulated environment that represents at least a portion of the real environment; and causing a simulated autonomous vehicle to traverse the simulated environment, by a controller of the autonomous vehicle, wherein a path of the simulated autonomous vehicle in the simulated environment deviates from a path of the real vehicle in the real environment; determining a simulated position of the simulated autonomous vehicle within the simulated environment; determining a first time during the period that a real position of the real vehicle in the real environment most closely matched the simulated position of the errant simulated autonomous vehicle in the simulated environment based at least in part on the log data; obtaining object data from the log data representative of real objects in the real environment as perceived by the real vehicle at the first time; generating simulated objects in the simulated environment that represent the real objects; One or more non-transitory computer-readable media storing instructions for causing the system to perform operations including: A system comprising:

2. The simulated location includes a first simulated location, the real location includes a first real location, and the manipulation comprises: determining a second simulated position of the simulated autonomous vehicle within the simulated environment, the second simulated position being different from the first simulated position; and determining, based at least in part on the log data, a second time during the period that a second real position of the real vehicle in the real environment most closely matched the second simulated position of the errant simulated autonomous vehicle in the simulated environment; and updating the simulated objects to correspond to the real objects in the real environment as perceived by the real vehicle at the second time; or generating a second simulated object in the simulated environment that represents a second real object in the real environment as perceived by the real vehicle at the second time; The system of claim 1 further comprising:

3. 3. The system of claim 1 or claim 2, wherein the simulated objects include a first simulated object, and the manipulation further includes avoiding generating, in the simulated environment, a second simulated object that represents a second real object in the real environment as perceived by the real vehicle at a second time after the first time based at least in part on at least one of the simulated position of the simulated autonomous vehicle or the first time.

4. The operation is determining whether the object data representing the real-world object is associated with a static object or a dynamic object; generating the simulated object representing the real object is based at least in part on determining that the object data is associated with a static object; A system according to any one of claims 1 to 3.

5. a simulated pose of the simulated autonomous vehicle in the simulated environment deviates from a real pose of the real vehicle in the real environment; A system according to any one of claims 1 to 4.

6. receiving log data representative of a first environment in which a first vehicle was operating; using the log data to generate a simulated environment for testing a vehicle controller, the simulated environment representing a real-world first environment; and determining a position of a second vehicle as it traverses the simulated environment, the second vehicle being controlled by the vehicle controller, wherein a path of the second vehicle in the simulated environment deviates from a path of the first vehicle in the first environment; determining a position of the first vehicle within the first environment associated with the position of the stray second vehicle within the simulated environment based at least in part on the log data; generating a simulated object in the simulated environment representing a first object in the first environment as perceived by the first vehicle from the location; A method for providing the above.

7. determining that the second vehicle changes position within the simulated environment; and updating the simulated environment to include another simulated object representing a second object in the first environment as perceived by the first vehicle from a second position associated with the changed position of the second vehicle; The method of claim 6 further comprising:

8. The log data is associated with a period of time during which the first vehicle was operating in the first environment, and the method further comprises: determining a first time during the period that the first vehicle was located at the location based at least in part on the log data; obtaining object data from the log data representative of the first object in the first environment as perceived by the first vehicle at the first time, the simulated object being generated based at least in part on the object data; The method of claim 6 or claim 7, further comprising:

9. 7. The method of claim 6, wherein determining the location of the first vehicle includes determining the location of the first vehicle in the first environment that most closely matches a position of the second vehicle in the simulated environment.

10. updating simulation instructions to include the simulated environment; inputting the simulation instructions to the vehicle controller such that the vehicle controller controls the second vehicle to traverse the simulated environment; The method of any one of claims 6 to 9, further comprising:

11. the logged data includes either simulated sensor data or actual sensor data captured by a sensor of the first vehicle.

11. The method according to any one of claims 6 to 10.

12. controlling, by the vehicle controller, the second vehicle based at least in part on perceiving the simulated object; The method of any one of claims 6 to 11, further comprising:

13. receiving an input indicating a user-specified location within the simulated environment that includes another simulated object; generating the other simulated object in the simulated environment at a user-specified location; and The method of any one of claims 6 to 12, further comprising:

14. the log data includes multiple instances of log data representative of the first environment in which the first vehicle or a third vehicle was operating during multiple periods of time, and determining the position of the second vehicle is based at least in part on the multiple instances of log data. The method according to any one of claims 6, 8 and 11.

15. One or more non-transitory computer readable media storing instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 6 to 14.

Images