Instantiating objects in a simulated environment based on log data

Abstract

Techniques are disclosed herein for instantiating objects in a simulated environment based on log data. Some of the techniques may include receiving log data representative of an environment in which a real-world vehicle was operating. Using the log data, a simulated environment may be generated for testing the simulated vehicle. The simulated environment may represent the environment in which the real-world vehicle was operating. The techniques may further include determining a position of the simulated vehicle as it traverses the simulated environment. Based at least in part on the log data, a previous position of the real-world vehicle in the environment that is closest to the position of the simulated vehicle in the simulated environment may be determined. In this manner, the simulated environment may be updated to include simulated objects that represent objects in the environment perceived by the vehicle from the previous position.

Description

[Technical Field]

[0001] This invention relates to the instantiation of objects in a simulated environment based on log data. [Background technology]

[0002] This application claims priority to U.S. Utility Patent Application No. 17 / 193,826, filed on 5 March 2021, which is incorporated herein by reference in its entirety.

[0003] Simulated data and simulations can be used to test and verify the functionality of vehicle systems where real-world testing may be prohibited (e.g., due to safety concerns, time constraints, or limitations on reproducibility). However, creating simulations that accurately reflect real-world scenarios and verify the functionality of vehicle systems is a challenging task. For example, if a simulation is based on data previously captured by a vehicle operating in the real world, the vehicle system tested under simulation may react differently to the real-world vehicle that captured the data. This can make it difficult to verify that the tested vehicle system is accurate. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] U.S. Patent Application No. 15 / 644,267 [Patent Document 2] U.S. Patent Application No. 15 / 693,700 [Patent Document 3] U.S. Patent Application No. 16 / 198,653 [Brief explanation of the drawing]

[0005] Detailed explanations are provided with reference to the attached drawings. In the drawings, the leftmost digit of the reference number identifies the drawing in which that reference number first appears. The same reference number in different drawings indicates similar or identical components or features.

[0006] [Figure 1] Figure 1 is a pictorial flowchart illustrating an exemplary data flow in which a vehicle generates log data associated with its environment and sends that log data to a computing device that determines the association between the vehicle's location and object data captured by the vehicle at different points in time. [Figure 2] Figure 2 shows an exemplary environment in which a vehicle perceives various objects in the environment and various viewpoints of those objects at different points in time. [Figure 3] Figure 3 shows an exemplary data transformation performed, at least partially, by the vehicle perception system. [Figure 4] Figure 4 is a pictorial flowchart illustrating an exemplary process in which objects are instantiated in a simulated environment based on various simulated vehicle positions relative to various previous positions of the real-world vehicle. [Figure 5] Figure 5 is a pictorial flowchart illustrating an exemplary process that avoids updating the simulated environment based on a simulated vehicle that remains stationary within the simulated environment. [Figure 6] Figure 6 is a block diagram showing an exemplary system for implementing some of the various technologies described herein. [Figure 7] Figure 7 is a flowchart illustrating an exemplary method for updating a simulated environment to include objects based on the simulated vehicle's position relative to the vehicle's previous position in the real world. [Figure 8] Figure 8 is a flowchart illustrating another exemplary method for updating a simulated environment to include objects based on the simulated vehicle's position relative to the vehicle's previous position in the real world. [Modes for carrying out the invention]

[0007] A technique for instantiating objects in a simulated environment based on log data is disclosed herein. As discussed above, creating a simulation that accurately reflects real-world scenarios and verifies the functionality of a vehicle system is a challenging task. For example, if a simulation is based on data captured by a vehicle operating in the real world, the tested vehicle system under simulation may react differently to the real-world vehicle from which the data was captured. This can make it difficult to verify whether the tested vehicle system is accurate. For instance, the position of a simulated vehicle may deviate from the path of the vehicle generating the log data, while perception is still based on the vehicle's position being logged. This discrepancy in perception reduces the accuracy of the simulation, and in cases of large deviations, may even invalidate the results.

[0008] For example, consider a simulated (e.g., test) environment for testing and / or verifying the controller of an autonomous vehicle. The simulated environment may be generated based on log data captured by a real-world vehicle. Thus, the simulated environment may correspond to, or otherwise represent, a real-world environment (e.g., the Embarcadero track in San Francisco, California). Therefore, a real-world environment may include both static objects that remain stationary for long periods (e.g., buildings, streetlights, traffic signs, benches, parked vehicles, trees, barriers and fences, etc.) and dynamic objects that tend to keep moving (e.g., pedestrians, moving vehicles, cyclists, construction equipment, etc.). Thus, a simulated environment may include both static and dynamic objects that accurately reflect a real-world environment.

[0009] However, simulating static objects in a simulated environment can present several challenges. For example, even if they can be classified as static objects, their appearance may change over time, such as when a vehicle is moving. As another example, the sensor systems of a real-world vehicle may, in some cases, misdetect static objects for a given time instance over a period of time. Therefore, the vehicle system may not always produce a constant static bounding box output representing the corresponding static object. Instead, the vehicle system's output representing a static object may change, move, appear, disappear, and / or so over a given period of time.

[0010] Furthermore, when a real-world vehicle moves through an environment, the vehicle's systems may be limited to seeing / perceiving only those objects within its horizon. For example, the vehicle's systems may not perceive an object in advance before the vehicle enters the range of those objects. Therefore, a simulation representing a real-world vehicle in a real-world environment may present simulated object data to the simulated vehicle to mimic what the real-world vehicle perceives at the corresponding real-world location and time. However, if the simulated vehicle deviates from the real-world vehicle (for example, by changes to the autonomous controller of the simulated vehicle controller), it may become difficult to present the accurate object data perceived by the simulated vehicle through the use of the real-world vehicle log.

[0011] Accordingly, this disclosure describes techniques for improving a simulation based on a vehicle log by determining the position of a logged vehicle closest to the current, simulated vehicle position, and by instantiating static objects in the simulated environment that the logged vehicle would have perceived at that time and / or from that position. For example, the technique may include receiving log data representing the environment in which a vehicle (e.g., a logged vehicle) was operating, and generating a simulated scenario for testing a vehicle controller based on the log data. In some examples, the simulated scenario may include a simulated environment representing a portion of the environment in which the vehicle was operating. The technique may also include inputting the simulated scenario to a vehicle controller so that the vehicle controller allows the simulated vehicle to traverse the simulated environment. While the simulation is in progress, the position of the simulated vehicle in the simulated environment may be determined. Accordingly, based on determining the position of the simulated vehicle, the log data may be examined to determine the previous position of the vehicle in the environment closest to the current position of the simulated vehicle in the simulated environment. Accordingly, the technique may further include updating the simulated environment to include simulated objects representing objects in the environment perceived by the vehicle from the previous position.

[0012] The techniques discussed herein can improve the accuracy of simulations for testing and / or verifying vehicle controllers. For example, the dynamic aspects of static objects can be realistically simulated by determining the previous position of the log vehicle closest to the current position of the simulated vehicle, and by instantiating static objects in the simulated environment that the log vehicle would have perceived from its previous position. Additionally, the techniques described herein can be used to change the appearance of static objects over time, such as when the simulated vehicle is moving (for example, to simulate how the appearance of an object changes based on a change in viewpoint in a real-world scenario). Furthermore, the techniques described herein enable the instantiation of static objects in the simulated environment at the moment that the log vehicle would have perceived them in the real world. This improves simulation testing by providing the vehicle controller with more accurate data and / or objects corresponding to what a real-world vehicle would perceive, and thus the log vehicle can reproduce more realistic scenarios for a particular time or position of the vehicle in the simulation, where the log vehicle does not have such information. Other improvements using the disclosed techniques will be readily apparent to those skilled in the art.

[0013] By way of example and not limitation, a method according to various techniques disclosed herein may include receiving log data representing an environment in which a vehicle was operating. In some examples, the vehicle may include an autonomous vehicle. Additionally, the log data may be associated with a certain period of time during which the vehicle was operating in the environment. In some examples, the log data may be generated by a vehicle's perception system based at least in part on sensor data captured by one or more vehicle sensors. In at least one example, the log data may include instances of various log data representing environments in which a vehicle and / or another vehicle was operating over a plurality of certain periods of time. In further examples, the log data may include various images representing the output of a vehicle's perception system at various times. That is, the log data may include, for example, a first log data image representing a first perception system output at a first time, a second log data image representing a second perception system output at a second time after the first time, a third log data image representing a third perception system output at a third time after the second time, and so on. In some examples, each or various log data images may include various objects perceived by the vehicle at that time and / or various viewpoints of the objects.

[0014] In some examples, the method may generate a simulated scenario for testing a vehicle controller using the log data. The simulated scenario may include a simulated environment representing at least a portion of the environment. The vehicle controller may, in some examples, include a controller of an autonomous vehicle. In various examples, the simulated scenario may be input to the vehicle controller, and the vehicle controller may drive the simulated vehicle across the simulated environment. In at least one example, the simulated scenario, the simulated environment, the simulated objects, etc. may be input to the vehicle controller in the form of simulation instructions. That is, the simulation instructions may represent the simulated scenario, the simulated environment, the simulated objects, etc.

[0015] In some examples, the method may include determining the position of a simulated vehicle within a simulated environment. For example, geolocation data may be received from the simulated vehicle based on its position within the range of the simulated environment, and the position of the simulated vehicle may be determined at least in part based on the geolocation data.

[0016] In the examples, the method may additionally or alternatively include determining the previous location of a vehicle in a simulated environment, associated with the location of the simulated vehicle in the simulated environment. In various examples, determining the previous location may include determining the previous location of the vehicle closest to the current location of the simulated vehicle. Additionally or alternatively, at least in part based on determining the previous location, instances of time in which the vehicle was located at a previous location may be determined from log data. For example, the log data may include associations between each previous location of a vehicle in the environment (e.g., geolocation data) and instances of time (e.g., timestamps) associated with when the vehicle was located at each of those previous locations.

[0017] In various examples, object data can be obtained from log data, at least in part based on determining at least one of a previous position and / or instance of time. The object data can represent one or more objects within the environment that was / were perceived by the vehicle from a previous position and / or from an instance at that time. By way of non-limiting example, for any given instance of time during a period in which the vehicle was operating within the environment, the log data can store one or more of each time stamp (e.g., indicating the instance of time), each geolocation data (e.g., indicating the position of the vehicle at that instance of time), and each object data (e.g., indicating one or more objects perceived by the vehicle at that instance of time and from that position). In some examples, one or more objects within the environment can include static objects and / or dynamic objects. For example, one or more objects within the environment can include buildings, structures, barriers, fences, obstacles, vehicles, pedestrians, bicyclists, benches, streetlights, traffic signs, trees, etc.

[0018] In some examples, the method can further update a simulated environment to include simulated objects representing objects within the environment. That is, the simulated objects can be instantiated within the scope of the simulated environment. Additionally, the simulated objects can be positioned within the simulated environment at positions corresponding to the positions where the objects were positioned in the real-world environment. In various examples, updating the simulated environment can include updating the simulated environment to include various simulated objects representing various objects within the environment.

[0019] In additional or alternative examples, a simulated scenario and / or simulated environment may be updated by presenting the vehicle controller of the simulated vehicle with various log data images representing the perceptual system output of the real vehicle when the real vehicle is positioned at a real location in the real environment that best matches the simulated location of the simulated vehicle in the simulated environment. The log data images may include simulated representations of one or more real objects perceived by the real vehicle from its real location in the real environment. Thus, as the simulated vehicle moves within the simulated environment, various log data images may be presented to the vehicle controller of the simulated vehicle based on the current simulated location of the simulated vehicle within the simulated environment. Similarly, when the simulated vehicle stops within the simulated environment, the same or similar log data images may be repeatedly presented to the vehicle controller to simulate the vehicle remaining stationary.

[0020] In at least one example, the method may include determining that the position of a simulated vehicle has changed; that is, the method may include determining that a simulated autonomous vehicle is positioned at a second position in the simulated environment that is different from its original position. Thus, a second previous position of the vehicle in the environment associated with the second position of the simulated vehicle may be determined. In various examples, determining the second previous position may include determining the second previous position of the vehicle closest to the second position of the simulated vehicle. Additionally or alternatively, at least in part on determining the second previous position, a second instance of time in which the vehicle was positioned at the second previous position may be determined from log data. In some examples, second object data may be obtained from log data at least in part on determining the second previous position and / or at least one of the second instance of time. At least in part on the object data, in some examples, the method may include updating at least one of the appearance or position of a simulated object with respect to the simulated vehicle. Additionally or alternatively, the method may include updating the simulated environment to include another simulated object representing another object in the environment.

[0021] In some examples, the method may involve controlling a simulated vehicle based at least partially on additionally simulated objects. For example, the vehicle controller under test may be able to perceive simulated objects in a simulated environment and may change the trajectory of the simulated vehicle based at least partially on perceiving the simulated objects.

[0022] Additionally, in some examples, user-specified objects may be simulated at various locations within the simulated environment. Therefore, the method may include receiving input indicating user-specified locations within the simulated environment where the simulated objects are instantiated, and updating the simulated environment to include the simulated objects at the user-specified locations. In further examples, the simulated scenario may be generated at least partially based on a three-dimensional (3D) simulation, a previous simulation, etc.

[0023] The techniques described herein can be implemented in many ways. Exemplary implementations are provided below with reference to the following figures. Although described in the context of autonomous vehicles, the techniques described herein can be applied to a variety of systems that require the control and / or interaction of objects in an environment, and are not limited to autonomous vehicles. In another example, the systems, methods, and apparatus can be used in an aviation or navigation context. Additionally, the techniques described herein can be used with real-world data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination thereof.

[0024] Figure 1 is a pictorial flowchart illustrating an exemplary data flow 100 in which a vehicle 102 generates log data 104 associated with the environment 106 and sends that log data to a computing device 108 that determines the association between the vehicle 102's position and object data captured by the vehicle 102 at different points in time. Additionally, the computing device 108 may also generate simulation instructions 112 based at least partially on the log data 104. In some examples, the log data 104 may include 3D voxel data, image data, bounding boxes, 2D and / or 3D contours of objects, classification data, segmentation data, occlusion data, etc.

[0025] Vehicle 102 may include a computing device having a perception engine and / or planner component for performing actions such as detecting, identifying, segmenting, classifying, and / or tracking objects from sensor data captured within the environment 106. For example, the computing device associated with vehicle 102 may detect objects 114(1), 114(2), 114(3), and 114(4). Objects 114(1), 114(2), 114(3), and 114(4) are depicted as parked vehicles, trees, buildings, and streetlights within the environment 106, but objects 114(1), 114(2), 114(3), and 114(4) may also include cyclists, pedestrians, animals, road signs, signs, traffic lights, mailboxes, barriers, fences, benches, and / or other objects.

[0026] The computing device associated with the vehicle 102 may include one or more processors and memory communicably coupled to one or more processors. The one or more processors may include, for example, one or more FPGAs, SoCs, ASICs, and / or CPUs. The vehicle 102 may traverse the environment 106 and determine and / or capture data. For example, the computing device associated with the vehicle 102 may determine vehicle status data, vehicle diagnostic data, vehicle metrics data, and / or map data. In some examples, the vehicle 102 may include one or more sensors, which may include one or more time-of-flight sensors, lidar sensors, radar sensors, sonar sensors, image sensors, voice sensors, infrared sensors, geolocation sensors, wheel encoders, IMUs, etc., or any combination thereof, but other types of sensors are also intended.

[0027] As vehicle 102 traverses environment 106, sensors may capture sensor data associated with environment 106. For example, and as discussed above, some of the sensor data may be associated with objects such as objects 114(1), 114(2), 114(3), and 114(4). While objects 114(1), 114(2), 114(3), and 114(4) may represent static objects, in some examples, sensor data may also be associated with dynamic objects. As used herein, “dynamic object” may be an object associated with movement within the environment 106 (e.g., vehicle, motorcycle, cyclist, pedestrian, animal, etc.) or an object that is capable of movement (e.g., parked vehicle, standing pedestrian, etc.). In contrast, “static object” may be an object associated with environment 106, such as a building / structure, road surface, road sign, signal, barrier, tree, sidewalk, streetlamp, parked vehicle, etc. In some examples, a computing device associated with vehicle 102 may determine information about objects in the environment, such as bounding box information, classification information, and segmentation information.

[0028] A computing device associated with the vehicle 102 may generate log data 104. For example, log data 104 may include sensor data, perception data, planning data, vehicle status data, speed data, intention data, and / or other data generated by the vehicle computing device. In some examples, sensor data may include data captured by sensors such as time-of-flight sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement unit (IMU), accelerometer, magnetometer, gyroscope, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, illuminance, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, and wheel encoders. Sensor data may include time-of-flight data, position data, lidar data, radar data, sonar data, image data, and audio data captured by such sensors. In some examples, log data 104 may include time data associated with other data generated by the vehicle computing device.

[0029] The log data 104 may represent various states of the environment 106 along a timeline. As shown in Figure 1, the states may include various representations of the environment (the real environment, the environment represented by captured sensor data, the environment represented by data based on the captured sensor data (e.g., bounding boxes, object classification, planner decisions), and / or the environment represented by the simulator) at individual points in time or over time. For example, the log data 104 may represent the state 116 of the environment 106 at time t0118. In some examples, state 116 may be the initial state of the environment 106 associated with the beginning (or any) portion of the log data 104. In some examples, state 116 may be the state of the environment 106 associated with any (e.g., an intermediate) portion of the log data 104. The log data 104 may represent the state 120 of the environment 106 at time t1122, after time t0118. As shown in Figure 1, the log data 104 may further represent the state 124 of the environment 106 at time t2126, following time points t0118 and t1122. For illustrative purposes only, state 116 may represent the environment 106 at the initial time point t0118 at 0 seconds, state 120 may represent the environment 106 at time point t1122 at 5 seconds later, and state 124 may represent the environment 106 at time point t2126 at 15 seconds later. Therefore, in some examples, time points t0118, t1122, and t2126 may be discontinuous (e.g., there is elapsed time between time points t0118, t1122, and t2126), or time points t0118, t1122, and t2126 may be continuous (e.g., there is no elapsed time between time points t0118, t1122, and t2126).

[0030] Vehicle 102 may transmit some or all of the log data to a computing device 108 via one or more networks 128. For example, vehicle 102 may transmit log data 104, including time data, perception data, object data, location data, and / or map data, to the computing device 108 for storage. Such transmission may be wired, wireless, or otherwise, as detailed herein. Examples of vehicle-generated data that may be included in log data 104 include, for example, U.S. Patent Application No. 15 / 644,267, filed July 7, 2017, titled "Interactions Between Vehicle and Teleoperations System" (which may include communication signals that may include sensor data and data indicating the occurrence of events, as well as vehicle control data, object data calculators, object classifiers, collision prediction systems, kinematics calculators, safety system actuators, and drive system control operations that may be derived from data received from sensors and planners), U.S. Patent Application No. 15 / 693,700, filed September 1, 2017, titled "Occupant Protection System Including Expandable Curtain and / or Expandable Bladder" (which may include data representing the vehicle's trajectory, position data, object data, and vehicle control data), and "Executable Component Interface and These are described in U.S. Patent Application No. 16 / 198,653, filed November 21, 2018, and include a type of log data, each of which is incorporated by reference in whole for any purpose. In some examples, the computing device 108 may include a server that stores log data 104 in a database. In some examples, the computing device 108 may operate a scenario generator for generating simulation instructions 112 that define simulated scenarios and / or simulated environments.

[0031] Based on the log data 104, the scenario generator may identify the objects represented in the log data 104 (e.g., objects 114(1), 114(2), 114(3), and 114(4)). The scenario generator may then determine the portion of the log data 104 associated with the objects. For example, in state 116, the scenario generator may identify object 114(3), which is a building in the environment 106 represented in the log data. The scenario generator may also identify object 114(3) represented in the log data 104 in states 120 and 124. In some examples, the log data 104 may include perceptual data associated with object 114(3), for example, identifying the extent of object 114(3). In some examples, the perceptual data may include, or be based on, sensor data from the vehicle 102's sensors or combination of sensors. In some examples, the log data 104 may include location data associated with object 114(3), identifying the location of object 114(3) in the environment 106.

[0032] Using the perception data, the scenario generator may determine one or more log data associations 130 that identify the associations between each position of the vehicle 102, each time the vehicle 102 was positioned at each position, and each object data generated by the vehicle 102 when the vehicle 102 was positioned at each position at each time. For example, "Position 1" of log data association 130 may include first geolocation data (e.g., GPS coordinates) indicating a specific location where the vehicle 102 perceived the environment 106, as shown in state 116; "Position 2" of log data association 130 may include second geolocation data indicating a specific location where the vehicle 102 perceived the environment 106, as shown in state 120; and "Position 3" of log data association 130 may include third geolocation data indicating a specific location where the vehicle 102 perceived the environment 106, as shown in state 124. Additionally, "t0" of log data association 130 may include a first timestamp indicating a specific instance of the time when vehicle 102 was located at location 1, "t1" of log data association 130 may include a second timestamp indicating a specific instance of the time when vehicle 102 was located at location 2, and "t2" of log data association 130 may include a third timestamp indicating a specific instance of the time when vehicle 102 was located at location 3. Furthermore, the "State 116 Object Data" of the log data association 130 may include object data representing one or more objects perceived by the vehicle 102 in the environment 106 from location 1 at time t0, the "State 120 Object Data" of the log data association 130 may include object data representing one or more objects perceived by the vehicle 102 in the environment 106 from location 2 at time t1, and the "State 124 Object Data" of the log data association 130 may include object data representing one or more objects perceived by the vehicle 102 in the environment 106 from location 3 at time t2.

[0033] In some examples, the computing device 108 may determine instantiation attributes associated with an object contained in the log data 104. The instantiation attributes may indicate how the vehicle 102 detected the object and / or its initial perception. For illustrative purposes only, object 114(2) may enter the detection range of the vehicle 102 (e.g., as shown in state 120) by the vehicle 102 approaching object 114(2) and / or by object 114(2) becoming visible (e.g., object 114(2) may be behind an obstruction). In some examples, the detection range (also called the perception threshold) may be associated with the capabilities of one or more sensors in the vehicle 102. In some examples, the detection range may be associated with a confidence value determined by the perception engine of the vehicle 102.

[0034] In some examples, the computing device 108 may additionally or alternatively determine termination attributes associated with objects contained in the log data 104. Termination attributes may indicate a state in which the vehicle 102 can no longer detect the object, and / or an updated perception of the object. For illustrative purposes only, object 114(1) may be in a position outside the detection range of the vehicle 102 as the vehicle 102 moves away from object 114(1) and / or object 114(1) is obscured (e.g., as shown in states 120 and 124).

[0035] Based on log data 104 and / or log data association 130, the scenario generator can generate simulation instructions 112 that describe a simulated scenario including a simulated environment. For example, log data 104 may include map data and location data representing the environment 106, as well as how the vehicle 102 traverses the environment 106. Additionally, log data 104 may include, but not limited to, status data of the vehicle 102 as it traverses the environment 106, such as acceleration data, steering angle data, and indicator data. Log data 104 may include time data associating the vehicle 102's actions and observations (e.g., sensor data) with a set of timestamps. When used and executed by the scenario generator, log data 104 can generate simulation data including a simulated environment representing the environment 106, and generate simulation instructions 112 that include simulated objects in various states (e.g., states 116, 120, and 124). For example, the simulation data may include time t x State of the simulated environment at 134, time t x Time point t after 134 y The simulated environment state at 138, and time t y Time point t after 138 z This can represent the simulated environment state 140 in 142.

[0036] In some examples, the scenario generator may create a simulation instruction 112 that represents a simulated environment, including more or less detail from an environment 106 as depicted in Figure 1 (for example, by filtering objects, representing objects with bounding boxes, and / or adding additional objects). In some examples, the scenario generator may input the simulation instruction 112 to a vehicle controller associated with a simulated vehicle to determine how the simulated vehicle responded under the circumstances presented in the simulated scenario.

[0037] Figure 2 shows an exemplary environment 200 in which the vehicle 102 perceives various objects in the environment and various viewpoints of various objects at different points in time.

[0038] Vehicle 102 may traverse the environment (e.g., environment 106 as described in Figure 1 above) according to a road network that includes one or more road segments such as road segments 202(1), 202(2), and 202(3). Each of the road segments 202(1), 202(2), and 202(3) may be interconnected. That is, in some examples, road segment 202(1) may be connected to road segment 202(2), road segment 202(2) may be connected to road segment 202(3), and so on. Additionally or alternatively, road segments 202(1), 202(2), and 202(3) may be connected to each other via other road segments not shown.

[0039] Vehicle 102 may traverse road segments 202(1), 202(2), and 202(3) in a first direction, as shown by the timeline 204. For example, vehicle 102 may be positioned at a first position 206(1) at time t0118, at a second position 206(2) at time t1122 after time t0118, and at a third position 206(3) at time t2126 after time t1122. Thus, states 116, 120, and 124 may represent the environment in which vehicle 102 operates, as perceived by vehicle 102, from positions 206(1), 206(2), and 206(3), and at time t0118, t1122, and t2126, respectively. That is, states 116, 120, and 124 may represent the environment as perceived by the vehicle 102, the vehicle 102's sensor system, the vehicle 102's perception system, the vehicle 102's controller, etc. For illustrative purposes only, state 116 may represent the environment 106 at an initial time t0118 at 0 seconds, state 120 may represent the environment 106 at a time t1122 after 5 seconds, and state 124 may represent the environment 106 at a time t2126 after 15 seconds. Therefore, in some examples, the times t0118, t1122, and t2126 may be discontinuous (e.g., there is elapsed time between times t0118, t1122, and t2126), or the times t0118, t1122, and t2126 may be continuous (e.g., there is no elapsed time between times t0118, t1122, and t2126).

[0040] As shown in Figure 2, states 116, 120, and 124 may include different objects perceived by the vehicle 102 at different time points t0118, t1122, and t2126, and from different locations 206(1), 206(2), and 206(3). For example, at time point t0118, when the vehicle 102 is at the first location 206(1), state 116 may include parked vehicles, buildings, and streetlights (for example, objects 114(1), 114(3), and 114(4) in the environment 106 shown in Figure 1, respectively).

[0041] However, as vehicle 102 moves through the environment to a second position 206(2), vehicle 102 may perceive objects such as those shown in state 120, including trees, buildings, and streetlights (for example, objects 114(2), 114(3), and 114(4) in environment 106 shown in Figure 1, respectively). Additionally, the appearance of objects as seen from the vehicle 102's viewpoint may change between state 116 and state 120 based on the change in vehicle 102's position and the disappearance of some objects (e.g., parked vehicles). Furthermore, objects such as trees may appear in state 120 based on vehicle 102 moving to a second position 206(2) which may be within the range of trees. That is, the vehicle 102's sensor system may be within the range of trees shown in state 120 in order to detect the presence of trees. Ultimately, if vehicle 102 moves through the environment to a third location 206(3) at time t2126, vehicle 102 may perceive an object as shown in state 124.

[0042] Figure 3 shows an exemplary data transformation 300 performed at least partially by the vehicle perception system 302. The vehicle perception system 302 may output different representations of the environment in which the vehicle (e.g., vehicle 102) was operating (e.g., environment 106) based on different representations of the environment perceived by the vehicle from different locations at different times.

[0043] For example, the vehicle perception system 302 may receive sensor data captured by the vehicle's sensor system. The sensor data may include various states representing the environment, such as states 116, 120, and 124. Each of states 116, 120, and 124 may represent the environment as seen from the vehicle's position at a particular time. In the example, the times t0118, t1122, and t2126 on the timeline 204 may be associated with states 116, 120, and 124, respectively.

[0044] By way of example and not limitation, the vehicle perception system 302 may receive sensor data representing the state 116 of the environment at time t0118 and output a simulated representation of the environment as shown in state 132. State 132 may include one or more bounding boxes representing objects seen by the vehicle in state 116. For example, the vehicle perception system 302 may apply one or more rules, heuristics, transformations, and / or the like to the sensor data to output the simulated representation shown in state 132. Additionally or alternatively, the vehicle perception system 302 may use one or more machine learning models to generate the simulated representation of state 132. The vehicle perception system 302 may perform similar or analogous processing on sensor data representing states 120 and 124 to generate the simulated representations shown in states 136 and 140. In the example, the log data described herein may include data associated with any one of states 116, 120, and / or 124, similar to states 132, 136, and / or 140.

[0045] In various examples, states 132, 136, and 140 may be associated with times t x 134, t y 138, and t z 142. In this way, the scenario generator may generate simulation instructions that, when executed by the simulator, generate simulation data representing the simulated environmental states 132, 136, and 140 associated with times t x 134, t y 138, and t z 142. By way of example and not limitation, when the simulated vehicle is at the same or a proximate location as the vehicle that captured the sensor data shown in state 116 at time t0118, the simulated environment perceived by the simulated vehicle may match state 132 at time t x 134.

[0046] Figure 4 is a pictorial flowchart illustrating an exemplary process 400 in which objects are instantiated in a simulated environment based on various positions of a simulated vehicle 408 relative to various previous positions of vehicle 102. Vehicle 102 may capture log data on which the simulated environments 410(1), 410(2), and 410(3) are based.

[0047] In operation 402, process 400 may include generating a simulated scenario that includes a simulated environment 410(1). The simulated scenario may be generated using log data. In an example, whether the simulated environment 410(1) includes or avoids including certain simulated objects may be based at least in part on the current position of the simulated vehicle 408 and the previous position 206(1) of the vehicle 102. For example, since the position of the simulated vehicle 408 is closest to the previous position 206(1) of the vehicle 102, as opposed to position 206(2) or 206(3), the simulated environment 410(1) may include the simulated objects shown in the stippled Figure 4. In some examples, one or more simulated objects in the simulated environment 410(1) may represent actual objects perceived by the vehicle 102 from the previous position 206(1) at time t0112. In other words, the simulated environment 410(1) may include one or more simulated objects representing objects that were within range of the vehicle 102's sensor system from position 206(1) at time t0112.

[0048] In operation 404, process 400 may include updating the simulated scenario to include the simulated environment 410(2) based at least in part on the position of the simulated vehicle 408, which moves closest to the previous position 206(2) of vehicle 102 at time t1116. The simulated environment 410(2) includes an updated appearance of objects that were previously included in the simulated environment 410(1), based at least in part on the position of the simulated vehicle 408. In addition, the simulated environment 410(2) includes not only additional objects that were not included in the simulated environment 410(1), but also fewer objects than were included in the simulated environment 410(1), based on the change in the position of the simulated vehicle 408. For example, when vehicle 102 moves from position 206(1) to position 206(2), additional objects may have moved into or become detectable by the vehicle 102's sensor system. Similarly, objects that were previously detectable by vehicle 102 may have moved out of range or become undetectable. Therefore, the simulated environment 410(2) can be updated to take into account these changes that the vehicle 102 would have perceived in the real world at time t1116.

[0049] In operation 406, process 400 may include updating the simulated scenario to include the simulated environment 410(3) based at least in part on the position of the simulated vehicle 408, which moves closest to the previous position 206(3) of vehicle 102 at time t2120. The simulated environment 410(3) includes an updated appearance of objects that were previously included in the simulated environment 410(1) and / or 410(2), based at least in part on the changed position of the simulated vehicle 408. Additionally, the simulated environment 410(3) includes fewer objects than were included in the simulated environment 410(1) and / or 410(2), based on the change in the position of the simulated vehicle 408. For example, when vehicle 102 moves from position 206(2) to position 206(3), objects that were previously detectable by vehicle 102 may have moved out of range or otherwise become undetectable. Therefore, the simulated environment 410(3) can be updated to take into account these changes that the vehicle 102 would have perceived in the real world at time t2120.

[0050] It will be understood that more or fewer locations (e.g., locations 206(1), 206(2), and 206(3)) than those shown in Figure 4 may be used. For example, Figure 4 includes three locations and three simulated environments, but any number of locations and / or simulated environments may be possible to create a realistic simulation.

[0051] Figure 5 is a pictorial flowchart illustrating an exemplary process 500 that avoids updating the simulated environment based on a simulated vehicle 508 that remains stationary in the simulated environment with respect to the vehicle 102's previous position 206(1). The vehicle 102 may capture log data on which the simulated environment 510 is based.

[0052] In operation 502, process 500 may include generating a simulated scenario that includes a simulated environment 510. The simulated scenario may be generated using log data. In an example, whether the simulated environment 510 includes or avoids including certain simulated objects may be based at least in part on the current position of the simulated vehicle 508 and the previous position 206(1) of the vehicle 102. For example, since the position of the simulated vehicle 508 is closest to the previous position 206(1) of the vehicle 102, as opposed to position 206(2) or 206(3), the simulated environment 510 may include the simulated objects shown in the stippled Figure 5. In some examples, one or more simulated objects in the simulated environment 510 may represent actual objects perceived by the vehicle 102 from its previous position 206(1) at time t0112. In other words, the simulated environment 510 may include one or more simulated objects representing objects that were within range of the vehicle 102's sensor system from position 206(1) at time t0112.

[0053] In operation 504, process 500 includes avoiding updating the simulated environment 510 of the simulated scenario. Avoiding updating the simulated environment in operation 504 may be at least in part on the stationary position of the simulated vehicle 508. That is, since the simulated vehicle 508 is still closest to the previous position 206(1) of the vehicle 102 at time t0112, the simulated environment is not updated to a view from a different position. Therefore, instead of updating the simulated environment 510 to match the viewpoint seen by the vehicle 102 from position 206(2) at time t0116, the simulator may recycle the simulated environment 510 so that the environment appears stationary to the simulated vehicle. Similarly, in operation 506, process 500 includes avoiding updating the simulated environment 510 of the simulated scenario, at least in part on the stationary position of the simulated vehicle 508.

[0054] Figure 6 is a block diagram showing an exemplary system 600 for implementing some of the various technologies described herein. In at least one example, the exemplary system 600 includes a vehicle 602, which can be similar to the vehicle 102 described above with reference to Figures 1 to 5. In the illustrated exemplary system 600, the vehicle 602 is an autonomous vehicle, but the vehicle 602 could be any other type of vehicle.

[0055] Vehicle 602 may be a driverless vehicle, such as an autonomous vehicle configured to operate in accordance with the Level 5 classification issued by the U.S. Department of Transportation's National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire journey without expecting a driver (or occupant) to constantly control the vehicle. In such an example, Vehicle 602 may be configured to control all functions from the start to the end of the journey, including all parking functions, and therefore may not include a driver and / or controls for driving Vehicle 602, such as a steering wheel, accelerator pedal, and / or brake pedal. This is merely an example, and the systems and methods described herein may be incorporated into any ground-based, air-based, or water-based vehicle, ranging from vehicles that always require manual control by a driver to vehicles that are partially or fully autonomously controlled.

[0056] Vehicle 602 may be any configuration of vehicle, such as a van, a sports utility vehicle, a crossover vehicle, a truck, a bus, an agricultural vehicle, and / or a construction vehicle. Vehicle 602 may be powered by one or more internal combustion engines, one or more electric motors, hydrogen power, any combination thereof, and / or any other suitable power source. Vehicle 602 has four wheels, however the systems and methods described herein may be incorporated into vehicles having fewer or more wheels and / or tires. Vehicle 602 may have four-wheel steering and may operate with substantially equal or similar performance characteristics in all directions, for example, such that when traveling in a first direction, the first end of vehicle 602 becomes the front end of vehicle 602, and when traveling in the opposite direction, the first end becomes the rear end of vehicle 602. Similarly, the second end of vehicle 602 may become the front end of the vehicle when traveling in a second direction, and the second end may become the rear end of vehicle 602 when traveling in the opposite direction. These exemplary characteristics can facilitate greater maneuverability, for example, in confined spaces or crowded environments such as parking lots and / or urban areas.

[0057] The vehicle 602 may include a computing device 604, one or more sensor systems 606, one or more emitters 608, one or more communication connections 610 (also referred to as communication devices and / or modems), at least one direct connection 612 (for example, for physically connecting to the vehicle 602 to exchange data and / or provide power), and one or more drive systems 614. One or more sensor systems 606 may be configured to capture sensor data associated with the environment.

[0058] The sensor system 606 may include time-of-flight sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measuring unit (IMU), accelerometer, magnetometer, gyroscope, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, illuminance, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, and the like. The sensor system 606 may include various instances of each of these or other types of sensors. For example, the time-of-flight sensors may include individual time-of-flight sensors located at the corners, front, rear, sides, and / or top of the vehicle 602. As another example, the camera sensors may include various cameras located at various positions on the exterior and / or interior of the vehicle 602. The sensor system 606 may provide input to the computing device 604.

[0059] Vehicle 602 may also include one or more emitters 608 for emitting light and / or sound. One or more emitters 608 in this example include internal audio and visual emitters for communicating with the occupants of vehicle 602. Internal emitters may include, but are not limited to, speakers, lights, signs, display screens, touchscreens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.). One or more emitters 608 in this example may also include external emitters. External emitters may include, but are not limited to, lights for indicating the direction of travel or other indicators of the vehicle's action (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters for audible communication with pedestrians or other nearby vehicles (e.g., speakers, speaker arrays, horns, etc.), one or more of which may constitute acoustic beam steering technology.

[0060] The vehicle 602 may also include one or more communication connections 610 that enable communication between the vehicle 602 and one or more other local or remote computing devices (e.g., remotely operated computing devices) or remote services. For example, a communication connection 610 may facilitate communication between the vehicle 602 and / or other local computing devices on the drive system 614. Alternatively, a communication connection 610 may enable the vehicle 602 to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.).

[0061] The communication connection 610 may include physical and / or logical interfaces for connecting the computing device 604 to another computing device or one or more external networks (e.g., the Internet). For example, the communication connection 610 may enable Wi-Fi-based communication such as via frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth®, cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), satellite communication, narrow-area communication (DSRC), or any suitable wired or wireless communication protocol that enables each computing device to interface with other computing devices.

[0062] In at least one example, the vehicle 602 may include one or more drive systems 614. In some examples, the vehicle 602 may have a single drive system 614. In at least one example, if the vehicle 602 has a variety of drive systems 614, the individual drive systems 614 may be located at opposing ends of the vehicle 602 (e.g., front and rear). In at least one example, the drive system 614 may include one or more sensor systems 606 for detecting the state of the drive system 614 and / or the surroundings of the vehicle 602. Examples, but not limited, the sensor system 606 may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the drive system's wheels, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive system, cameras or other image sensors, ultrasonic sensors, lidar sensors, radar sensors, etc. for acoustically detecting objects around the drive system. Some sensors, such as wheel encoders, may be specific to the drive system 614. In some examples, the sensor system 606 on the drive system 614 may overlap with or complement the system corresponding to the vehicle 602 (e.g., sensor system 606).

[0063] The drive system 614 may include many vehicle systems, including a high-voltage battery, a motor to propel the vehicle, an inverter to convert DC from the battery to AC for use in other vehicle systems, a steering system including a steering motor and steering rack (which may be electric), a brake system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for brake force distribution to reduce traction loss and maintain control, an HVAC system, lighting (e.g., headlights / taillights illuminating the external environment of the vehicle), and one or more other systems (e.g., a cooling system, safety systems, an on-board charging system, a DC / DC converter, a high-voltage junction, a high-voltage cable, a charging system, a charging port, and other electrical components). Additionally, the drive system 614 may include a drive system controller that can receive and preprocess data from sensors and control the operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and a memory communicatively coupled to one or more processors. The memory may store one or more modules and perform various functions of the drive system 614. Furthermore, the drive system 614 may also include one or more communication connections that enable each drive system to communicate with one or more other local or remote computing devices.

[0064] The computing device 604 may be similar to the vehicle computing device described above with reference to Figure 1. The computing device 604 may include one or more processors 618 and a memory 620 communicatively coupled to one or more processors 618. In the illustrated example, the memory 620 of the computing device 604 stores a localization component 622, a perception component 624, a prediction component 626, a planning component 628, and one or more system controllers 630. Although depicted as residing in memory 620 for illustrative purposes, the localization component 622, the perception component 624, the prediction component 626, the planning component 628, and one or more system controllers 630 may, additionally or alternatively, be accessible to the computing device 604 (e.g., stored in different components of the vehicle 602) and / or be accessible to the vehicle 602 (e.g., stored separately).

[0065] In the memory 620 of the computing device 604, the localization component may include the ability to receive data from the sensor system 606 to determine the position of the vehicle 602. For example, the localization component 622 may include and / or request / receive a three-dimensional map of the environment and continuously determine the position of the autonomous vehicle within the map. In some examples, the localization component 622 may use SLAM (simultaneous localization and mapping) or CLAMS (calibration, localization and mapping, simultaneously) to receive time-of-flight data, image data, lidar data, radar data, sonar data, IMU data, GPS data, wheel encoder data, or any combination thereof to accurately determine the position of the autonomous vehicle. In some examples, the localization component 622 may provide data to various components of the vehicle 602, as discussed herein, to determine the initial position of the autonomous vehicle for generating a trajectory. In at least one example, the positioning component 622 may determine the position of a simulated vehicle within a simulated environment and / or the nearest previous position of a vehicle that recorded log data on which the simulated environment is based.

[0066] The perceptual component 624 may include the ability to perform object detection, segmentation, and / or classification. In some examples, the perceptual component 624 may provide processed sensor data indicating the presence of entities in close proximity to the vehicle 602 and / or the classification of the entities as a type (e.g., car, pedestrian, cyclist, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional and / or alternative examples, the perceptual component 624 may provide processed sensor data indicating one or more characteristics associated with the detected entities and / or the environment in which the entities are located. In some examples, characteristics associated with the entities may include, but are not limited to, x-position (global position), y-position (global position), z-position (global position), orientation, type of entity (e.g., classification), velocity of the entity, and extent (size) of the entity. Characteristics associated with the environment may include, but are not limited to, the presence of other entities in the environment, the state of other entities in the environment, time of day, day of the week, season, weather, and indication of darkness / brightness. In at least one example, the perceptual component 624 may generate log data associated with a vehicle traversing an environment, as described herein. The log data may be used, in particular, to generate simulated scenarios, including a simulated environment for testing the vehicle.

[0067] The perceptual component 624 may include a function for storing perceptual data generated by the perceptual component 624. In some examples, the perceptual component 624 may determine trajectories corresponding to objects classified as object types. For illustrative purposes only, the perceptual component 624 may use the sensor system 606 to capture one or more images of the environment. The sensor system 606 may capture images of the environment that include objects such as buildings, vehicles, trees, streetlights, and pedestrians.

[0068] The stored perceptual data may, in some examples, include fused perceptual data captured by a vehicle. The fused perceptual data may include fused or other combinations of sensor data from sensor systems 606, such as image sensors, lidar sensors, radar sensors, time-of-flight sensors, sonar sensors, global positioning system sensors, internal sensors, and / or any combination thereof. The stored perceptual data may additionally or alternatively include classification data, which includes semantic classification of objects represented in the sensor data (e.g., pedestrians, vehicles, buildings, road surfaces, etc.).

[0069] The prediction component 626 may generate one or more probability maps representing the predicted probabilities of the possible locations of one or more objects in the environment. For example, the prediction component 626 may generate one or more probability maps for vehicles, pedestrians, animals, etc., that are within a threshold distance from vehicle 602. In some examples, the prediction component 626 may measure the trajectories of objects and, based on observed and predicted behavior, generate discretized predicted probability maps, heat maps, probability distributions, discretized probability distributions, and / or object trajectories. In some examples, one or more probability maps may represent the intentions of one or more objects in the environment.

[0070] The planning component 628 can determine the path that the vehicle 602 should follow to traverse the environment. For example, the planning component 628 can determine various routes and paths, as well as various levels of detail. In some examples, the planning component 628 can determine a route to travel from a first location (e.g., the current location) to a second location (e.g., the target location). For the purposes of this discussion, the route may be a sequence of waypoints for traveling between the two locations. In non-limiting examples, waypoints may include streets, intersections, Global Positioning System (GPS) coordinates, etc. Furthermore, the planning component 628 can generate instructions for guiding the autonomous vehicle along at least part of the route from the first location to the second location. In at least one example, the planning component 628 can determine how to guide the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions may be a route, or part of a route. In some cases, multiple paths can be generated substantially simultaneously (i.e., within technical tolerances) according to the receding horizon technique. One of the multiple paths within the receding data horizon with the highest level of confidence can be selected for vehicle operation.

[0071] In other examples, the planning component 628 may, alternatively or additionally, use data from the perceiving component 624 to determine the path that the vehicle 602 should follow to traverse the environment. For example, the planning component 628 may receive data from the perceiving component 624 regarding objects associated with the environment. Using this data, the planning component 628 may determine a route to travel from a first location (e.g., the current location) to a second location (e.g., the target location) in order to avoid objects in the environment. In at least some examples, such a planning component 628 may determine that there is no such collision-free path and, as a result, provide a path that avoids all collisions and / or otherwise mitigates damage, leading the vehicle 602 to a safe stop.

[0072] In at least one example, the computing device 604 may include one or more system controllers 630 that can be configured to control the steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 602. These system controllers 630 may communicate with and / or control systems corresponding to the drive system 614 and / or other components of the vehicle 602, which may be configured to operate according to a path provided by the planning component 628.

[0073] Vehicle 602 may be connected to a computing device 632 via a network 616 and may include one or more processors 634 and a memory 636 communicatively coupled to one or more processors 634. In at least one example, one or more processors 634 may be similar to processor 618, and memory 636 may be similar to memory 620. In the illustrated example, memory 636 of computing device 632 stores a log data component 638, an object component 640, a scenario component 642, and a simulation component 644. Although depicted as residing in memory 636 for illustrative purposes, the log data component 638, the object component 640, the scenario component 642, and the simulation component 644 may, additionally or alternatively, be accessible to computing device 632 (e.g., stored in different components of computing device 632) and / or be accessible to computing device 632 (e.g., stored separately).

[0074] In the memory 636 of computing device 632, the log data component 638 may determine the log data to be used to generate a simulated scenario. As discussed above, the database may store one or more log data. In some examples, computing device 632 may behave as a database for storing log data. In some examples, computing device 632 may connect to another computing device to access the log data database. The log data component 638 may scan the log data stored in the database and identify log data containing events of interest. As discussed above, in some examples, the log data may include event markers indicating that events of interest are associated with specific log data. In some examples, a user may select log data from a set of log data to be used to generate a simulated scenario.

[0075] Additionally, the object component 640 may identify one or more objects associated with the log data. For example, the object component 640 may determine and represent as simulated objects objects in the log data such as vehicles, pedestrians, animals, cyclists, trees, buildings, streetlights, barriers, and fences. In some examples, the object component 640 may determine objects that interact with vehicles associated with the log data. In some examples, the object component 640 may determine attributes associated with an object. For example, attributes may include the object's scope (also called the object's size, which may represent length, width, height, and / or volume), the object's classification, the object's pose, the object's trajectory, the object's waypoints, the object's instantiation attributes, and / or the object's termination attributes.

[0076] Object scope may indicate the size and / or volume of the object (e.g., a length of 5 meters). Object classification may indicate the classification and / or type of the object (e.g., vehicle, pedestrian, cyclist, etc.). Object pose may indicate the xyz coordinates of the object in the environment (e.g., position or position data) and / or include pitch, roll, and yaw associated with the object. Object trajectory may indicate the trajectory followed by the object. Object waypoint may indicate a position in the environment that shows the route between two locations.

[0077] The instantiation attribute of an object may indicate how a vehicle can detect an object. For example, an object may enter the vehicle's detection range by (1) the vehicle approaching the object, (2) the object approaching the vehicle, and / or (3) the object becoming visible (unobstructed) or the object's obstruction ending. The termination attribute of an object may indicate how a vehicle can no longer detect an object. For example, an object may be in a position outside the vehicle's detection range by (1) the vehicle moving away from the object, (2) the object moving away from the vehicle, and / or (3) the object becoming obstructed or the object being occluded.

[0078] The scenario component 642 can generate a simulated scenario using log data identified by the log data component 638. The simulated scenario may include a simulated environment (e.g., roads, road signs, buildings, etc.) and simulated objects (e.g., other vehicles, buildings, trees, streetlights, barriers, fences, pedestrians, cyclists, animals, etc.). In some examples, the scenario component 642 can apply a simulated object model to simulated objects using perceptual data generated by the perception component 624. For example, the scenario component 642 may identify a simulated vehicle model and apply it to simulated objects associated with the vehicle. In some examples, the scenario component 642 may identify a simulated building model and apply it to simulated objects associated with the building.

[0079] As an example, and not an exhaustive one, log data may include objects of various sizes, such as mailboxes, trees, buildings, and / or similar objects. Scenario component 642 may use volume-based filters so that objects associated with a volume equal to or greater than a threshold volume of 3 cubic meters, such as buildings, are represented in the simulated scenario, while objects associated with a volume less than 3 cubic meters, such as mailboxes, are not represented in the simulated scenario. In some examples, scenario component 642 may use motion-based filters so that objects associated with movement or trajectories according to the log data are represented in the simulated scenario or removed from the simulated scenario. In some examples, filters may be applied in combination or mutually exclusive.

[0080] In some examples, scenario component 642 may filter out objects that do not meet or exceed a confidence threshold. For example, but not limited to, log data may indicate that an object is associated with a pedestrian classification attribute and has a confidence value of 5% associated with the classification. Scenario component 642 may have a 75% confidence threshold and may filter out objects based on a confidence value that does not meet or exceed that threshold. In some examples, a user may provide a user-generated filter that includes one or more attribute thresholds so that scenario component 642 can filter out objects that do not meet or exceed one or more attribute thresholds indicated by the user-generated filter.

[0081] The simulation component 644 can execute simulated scenarios as a set of simulation instructions and generate simulation data. In some examples, the simulation component 644 can execute various simulated scenarios simultaneously and / or in parallel. This may allow the user to edit simulated scenarios and execute arrays of simulated scenarios with variations between each simulated scenario. In at least one example, the simulation component 644 tests a vehicle controller by simulating a scenario.

[0082] Additionally, the simulation component 644 may determine the results for a simulated scenario. For example, the simulation component 644 may run a scenario for use in a test and verification simulation. The simulation component 644 may generate simulation data showing how the autonomous controller performed (e.g., responded), compare the simulation data to a predetermined result, and / or determine whether any predetermined rule / assertion was broken or triggered.

[0083] In some examples, a given rule / assertion may be based on a simulated scenario (for example, a traffic rule regarding a crosswalk may be enabled based on a crosswalk scenario, or a traffic rule regarding crossing a lane marker may be disabled for a stationary vehicle scenario). In some examples, the simulation component 644 may dynamically enable or disable rules / assertions as the simulation progresses. For example, when a simulated object approaches a school zone, rules / assertions related to school zones may be enabled, and when the simulated object moves away from the school zone, they may be disabled. In some examples, rules / assertions may include comfort metrics related to, for example, how fast an object accelerates out of a simulated scenario.

[0084] Based at least in part on determining that the autonomous controller performed actions consistent with a predetermined outcome (i.e., the autonomous controller did everything it was supposed to do), and / or determining that no rules were broken or no assertions were triggered, the simulation component 644 may determine that the autonomous controller was successful. Based at least in part on determining that the autonomous controller's performance was inconsistent with a predetermined outcome (i.e., the autonomous controller did something it was not supposed to do), and / or determining that a rule was broken or an assertion was triggered, the simulation component 644 may determine that the autonomous controller was unsuccessful. Thus, based at least in part on running the simulated scenarios, the simulation data shows how the autonomous controller responds to each simulated scenario as described above, and based at least in part on the simulation data, a successful or unsuccessful outcome can be determined.

[0085] Processor 618 of computing device 604 and processor 634 of computing device 632 may be any suitable processor capable of processing data and executing instructions for performing the operations described herein. Processors 618 and 634 may comprise one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or part of a device that processes electronic data and converts that electronic data into other electronic data that can be stored in registers or memory. In some examples, integrated circuits (e.g., ASICs), gate arrays (e.g., FPGAs), and other hardware devices may also be considered processors insofar as they are configured to implement encoded instructions.

[0086] Memory 620 of computing device 604 and memory 636 of computing device 632 are examples of non-temporary computer-readable media. Memories 620 and 636 may store the operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functions resulting from various systems described herein. In various implementations, memories 620 and 636 may be implemented using any suitable memory technology, such as static random-access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, the ones shown in the accompanying drawings are merely examples relevant to the description herein.

[0087] In some examples, some or all aspects of the components discussed herein may include any model, algorithm, and / or machine learning algorithm. For example, in some examples, the components in memory 620 and 636 may be implemented as a neural network.

[0088] Figures 7 and 8 are flowcharts illustrating exemplary methods for instantiating objects in a simulated environment based on log data. For convenience and ease of understanding, the methods shown in Figures 7 and 8 are illustrated with reference to one or more vehicles and / or systems described in Figures 1 through 6. However, the methods shown in Figures 7 and 8 are not limited to being performed using the vehicles, systems, and / or technologies described in Figures 1 through 6, but may be performed using any other vehicles, systems, and technologies described herein, as well as vehicles, systems, and technologies other than those described herein. Furthermore, the vehicles, systems, and user interfaces described herein are not limited to performing the methods shown in Figures 7 and 8.

[0089] Methods 700 and 800 are presented as a collection of blocks of a logical flow graph, representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In a software context, a block represents a computer executable instruction stored in one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular abstract data type. The order in which the operations are described is not intended to be interpreted as limiting, and any number of the described blocks can be combined in any order and / or in parallel to implement a process. In some embodiments, one or more blocks of a process can be omitted entirely. Furthermore, methods 700 and 800 can be combined in whole or in part with each other or with other methods.

[0090] Figure 7 is a flowchart illustrating an exemplary method 700 for updating a simulated environment to include objects based on the simulated vehicle's position relative to the vehicle's previous position in the real world.

[0091] Method 700 begins with operation 702, which receives log data representing the real environment in which a real vehicle was operating over a period of time. For example, a computing device 108 may receive log data from one or more vehicles 102 through one or more networks 128. In some examples, the log data may be generated by the vehicle's perception system, at least in part, based on sensor data captured by one or more sensors of the vehicle. In at least one example, the log data includes diverse instances of log data representing the environment in which the vehicle and / or other vehicles were operating over a series of period of time.

[0092] In operation 704, method 700 includes using log data to generate a simulated scenario for testing the controller of an autonomous vehicle, wherein the simulated scenario includes a simulated environment that represents at least a portion of the real environment. For example, computing device 108 may generate a simulated scenario that includes a simulated environment that represents a portion of the environment 106.

[0093] In operation 706, method 700 includes having the autonomous vehicle's controller cause the simulated autonomous vehicle to traverse a simulated environment. In various examples, the simulated path of the simulated autonomous vehicle within / through the simulated environment may deviate from the actual path of a real vehicle within / through a real environment. In at least one example, a simulated scenario, a simulated environment, a simulated object, and / or such may be input to the autonomous vehicle's controller in the form of a simulation instruction. That is, a simulation instruction may represent a simulated scenario, a simulated environment, a simulated object, and / or such.

[0094] In operation 708, method 700 includes determining the simulated position of a simulated autonomous vehicle within a simulated environment. For example, a computing device 108 may receive geolocation data from a simulated vehicle based on the position of the simulated vehicle within the range of the simulated environment, and the simulated position of the simulated vehicle may be determined at least in part on the geolocation data. In operation 710, method 700 determines, at least in part on log data, the real position of a real vehicle in a real environment within the simulated environment deviant This includes determining a first time within a given period that best matches the simulated position of the simulated autonomous vehicle. For example, computing device 108 may determine a first time (e.g., t0112) when the previous position 206(1) of vehicle 102 was closest to the position of the simulated vehicle 508. Additionally or alternatively, computing device 108 may determine the first time based at least in part on log data association 130.

[0095] Operation 712 includes obtaining object data from log data representing real objects in a real environment perceived by a real vehicle at a first time. For example, computing device 108 may obtain object data from log data association 130 based at least in part on which vehicle 102 the simulated vehicle is closest to at a previous position. Operation 71 4 In this, method 700 includes generating simulated objects representing real-world objects in a simulated environment so that a simulated vehicle perceives the simulated objects. For example, a computing device 108 may instantiate a simulated object within the scope of the simulated environment at the location where the vehicle 102 perceives the object. In some examples, whether a simulated object is generated within a simulated environment may be based at least in part on determining the classification of the real-world object. That is, if the real-world object is a dynamic object, the simulated object cannot be generated. However, if the real-world object is a static object, the simulated object can be generated and / or instantiated within the scope of the simulated environment.

[0096] In operation 716, method 700 includes determining whether to terminate the simulation. For example, if the simulation is at the end of the log data file on which the simulation is based, computing device 108 may decide to terminate the simulation and proceed to operation 718. However, if the simulation is not yet at the end of the log data file, or if the simulation is continuing, method 700 may loop back to operation 708. In this way, operations 708 through 716 can continue until the end of the simulation is reached, and the simulated environment continues to be updated based on the simulated position of the simulated vehicle. In operation 718, if the simulation is terminated, method 700 includes determining whether the autonomous vehicle controller has succeeded in the simulated test. For example, computing device 108 may determine whether the simulated vehicle in the simulation safely and effectively traversed the simulated environment, etc.

[0097] Figure 8 is a flowchart illustrating another exemplary method 800 for updating a simulated environment to include objects based on the simulated vehicle's position relative to the vehicle's previous position in the real world.

[0098] Method 800 begins with operation 802, which includes receiving log data representing the real environment in which a real vehicle was operating. For example, a computing device 108 may receive log data from one or more vehicles 102 through one or more networks 128. In some examples, the log data may be generated by the perception system of the real vehicle, at least in part, based on sensor data captured by the sensors of one or more real vehicles. In at least one example, the log data includes diverse instances of log data representing the environment in which a vehicle and / or another vehicle was operating over a series of periodic periods.

[0099] In operation 804, method 800 may include generating a simulated environment for testing a vehicle controller using log data, wherein the simulated environment represents a real environment. For example, computing device 108 may generate a simulation instruction 112 that includes the simulated environment. In operation 806, method 800 may include determining the simulated position of a simulated vehicle as it traverses the simulated environment. The simulated vehicle may be controlled by a vehicle controller. Additionally, in some examples, the simulated pose of a simulated vehicle in the simulated environment may deviate from the real pose of a real vehicle in the real environment (for example, the simulated vehicle may be facing 90 degrees north, while the real vehicle is facing 93 degrees north). In some examples, computing device 108 may receive geolocation data from the simulated vehicle based on the position of the simulated vehicle within the range of the simulated environment, and the simulated position of the simulated vehicle may be determined at least in part on the geolocation data.

[0100] In operation 808, method 800 may include determining the real location of a real vehicle in a real environment associated with the simulated location of a simulated vehicle in a simulated environment, at least in part on log data. For example, computing device 108 may determine the real location of a real vehicle. Additionally, computing device 108 may determine the real location of a real vehicle at least in part on receiving geolocation data associated with the simulated location of a simulated vehicle in a simulated environment.

[0101] In operation 810, method 800 may include generating a simulated object in the simulated environment that represents a real object in the real environment as perceived by a real vehicle from a real location. For example, computing device 108 may instantiate the simulated object at a simulated location in the simulated environment that coincides with the real-world location of a real object in the real environment.

[0102] In operation 812, method 800 may include determining whether to terminate the simulation. For example, if the simulation is at the end of the log data file on which the simulation is based, computing device 108 may decide to terminate the simulation and proceed to operation 814. However, if the simulation is not yet at the end of the log data file, or if the simulation is continuing, method 800 may loop back to operation 806. In this way, operations 806 through 812 can continue until the end of the simulation is reached, and the simulated environment continues to be updated based on the simulated position of the simulated vehicle. In operation 814, if the simulation is terminated, method 800 includes determining whether the simulated vehicle has successfully completed the simulated test. For example, computing device 108 may determine whether the simulated vehicle in the simulation has safely and effectively traversed the simulated environment, etc.

[0103] Exemplary clause A. One or more processors, which, when executed by the one or more processors, receive log data representing the real environment in which a real vehicle was operating for a certain period of time, and use the log data to generate a simulated scenario for testing an autonomous vehicle controller, wherein the simulated scenario includes a simulated environment representing at least a portion of the real environment, and the autonomous vehicle controller causes the simulated autonomous vehicle to traverse the simulated environment, wherein the path of the simulated autonomous vehicle in the simulated environment deviates from the path of the real vehicle in the real environment, and the simulation of the simulated autonomous vehicle in the simulated environment done A system comprising one or more non-temporary computer-readable media for storing instructions causing the system to perform operations including: determining a position; determining a first time within a given period of time in which the actual position of the actual vehicle in the real environment best matches the simulated position of the deviated simulated autonomous vehicle in the simulated environment, at least in part based on the log data; obtaining object data from the log data representing real objects in the real environment perceived by the real vehicle at the first time; and generating simulated objects representing the real objects in the simulated environment.

[0104] B. The system according to paragraph A, further comprising: B. The simulated position includes a first simulated position, the real position includes a first real position, the operation is to determine a second simulated position of the simulated autonomous vehicle in the simulated environment, wherein the second simulated position is different from the first simulated position; and, at least in part based on the log data, determining a second time within a given period in which the second real position of the real vehicle in the real environment best matches the second simulated position of the deviated simulated autonomous vehicle in the simulated environment; and updating the simulated object to match the real object in the real environment as perceived by the real vehicle at the second time, or generating a second simulated object in the simulated environment that represents the second real object in the real environment as perceived by the real vehicle at the second time.

[0105] C. The system according to either paragraph A or B, wherein the simulated object includes a first simulated object, and the operation further includes avoiding generating a second simulated object in the simulated environment that represents a second real object in the real environment as perceived by the real vehicle at a second time after the first time, at least partially based on the simulated position of the simulated autonomous vehicle or at least one of the first time.

[0106] D. The system according to any one of paragraphs A to C, wherein the operation further includes determining whether the object data representing the real object is associated with a static or dynamic object, and generating the simulated object representing the real object is at least in part based on determining that the object data is associated with a static object.

[0107] E. The system according to any one of paragraphs A through D, wherein the simulated pose of the simulated autonomous vehicle in the simulated environment deviates from the actual pose of the actual vehicle in the real environment.

[0108] F. A method comprising: receiving log data representing a first environment in which a first vehicle was operating; using the log data to generate a simulated environment for testing a vehicle controller, wherein the simulated environment represents a real first environment; determining the position of a second vehicle as it traverses the simulated environment, the second vehicle being controlled by the vehicle controller, wherein the path of the second vehicle in the simulated environment deviates from the path of the first vehicle in the first environment; determining the position of the first vehicle in the first environment associated with the position of the deviated second vehicle in the simulated environment, at least in part based on the log data; and generating a simulated object in the simulated environment representing a first object in the first environment as perceived by the first vehicle from the position.

[0109] G. The method of paragraph F, further comprising deciding that the second vehicle changes its position in the simulated environment, and updating the simulated environment to include another simulated object representing a second object in the first environment perceived by the first vehicle from a second position associated with the changed position of the second vehicle.

[0110] H. The method according to either paragraph F or G, further comprising: the log data being associated with a certain period of time during which the first vehicle was operating in the first environment; the method determining a first time during the certain period of time in which the first vehicle was positioned, at least in part on the log data; and obtaining object data from the log data representing the first object in the first environment as perceived by the first vehicle at the first time, wherein the simulated object is generated at least in part on the object data.

[0111] I. The method according to any one of paragraphs F to H, wherein determining the position of the first vehicle includes determining the position of the first vehicle in the first environment that best matches the position of the second vehicle in the simulated environment.

[0112] J. Updating the simulation command to include the simulated environment, and inputting the simulation command to the vehicle controller so that the vehicle controller controls the second vehicle to traverse the simulated environment. A method that further includes the following, in any one of paragraphs F through I.

[0113] K. The method according to any one of paragraphs F to J, wherein the log data includes either simulated sensor data or actual sensor data captured by the sensors of the first vehicle.

[0114] L. The method according to any one of paragraphs F to K, further comprising controlling the second vehicle by the vehicle controller at least in part on perceiving the simulated object.

[0115] M. The method according to any one of paragraphs F to L, further comprising receiving input indicating a user-specified location containing another simulated object within the scope of the simulated environment, and generating the other simulated object in the simulated environment at the user-specified location.

[0116] N. The method according to any one of paragraphs F to M, wherein the log data includes diverse instances of log data representing the first environment in which the first vehicle or the third vehicle was operating over a set of periods, and determining the position of the second vehicle is at least partially based on the diverse instances of log data.

[0117] O. The method according to any one of paragraphs F to N, wherein determining whether the first object in the first environment is a static or dynamic object and generating the simulated object in the simulated environment is at least in part based on determining that the first object is a static object.

[0118] P. One or more non-temporary computer-readable media storing computer-executable instructions that, when executed, cause one or more processors to perform an operation including: receiving log data representing a first environment in which a first vehicle was operating; using the log data to generate a simulated environment for testing a vehicle controller, wherein the simulated environment represents a real first environment; determining the position of a second vehicle as it traverses the simulated environment, the second vehicle being controlled by the vehicle controller, wherein the pose of the second vehicle in the simulated environment deviates from the pose of the first vehicle in the first environment; determining the position of the first vehicle in the first environment associated with the deviated position of the second vehicle in the simulated environment, at least in part based on the log data; and generating a simulated object in the simulated environment representing a first object in the first environment as perceived by the first vehicle from the position.

[0119] Q. The operation further comprises determining that the second vehicle changes its position in the simulated environment, and updating the simulated environment to include another simulated object representing a second object in the first environment perceived by the first vehicle from a second position associated with the changed position of the second vehicle, according to one or more non-temporary computer-readable media as described in paragraph P.

[0120] R. The log data is associated with a certain period of time during which the first vehicle was operating in the first environment, and the operation further comprises determining a first time during the certain period of time during which the first vehicle was positioned, based at least in part on the log data, and obtaining from the log data object data representing the first object in the first environment as perceived by the first vehicle at the first time, one or more non-temporary computer-readable media according to either paragraph P or Q.

[0121] S. Determining the position of the first vehicle comprises determining the position of the first vehicle in the first environment that best matches the position of the second vehicle in the simulated environment, one or more non-temporary computer-readable media according to any one of paragraphs P to R.

[0122] T. One or more non-temporary computer-readable media as described in any one of paragraphs P to S, wherein the operation further comprises receiving an input indicating a user-specified location within the scope of the simulated environment where another simulated object is generated, and generating the other simulated object in the simulated environment at the user-specified location.

[0123] U. Receiving a variety of log data files representing the real environment in which a real vehicle was operating over a certain period of time, wherein the variety of log file data represents various objects in the real environment perceived by the real vehicle from various locations along the path traversed by the real vehicle, in various time instances during the period in which the vehicle was traversing the path; and simulating a scenario for testing the vehicle controller by presenting a first log data file of the variety of log data files to the vehicle controller, wherein the first log data file represents objects in the real environment perceived by the real vehicle from a first location along the path in which the real vehicle was located in a first time instance; and causing the vehicle controller to cause the simulated vehicle to traverse a simulated environment representing the real environment based on the first log data file; and the vehicle controller then simulating a scenario for testing the vehicle controller. A method comprising: deciding to move a vehicle from a first position in the simulated environment to a second position in the simulated environment, wherein the first position coincides with the first location; deciding that the second location deviates from the path traversed by the vehicle; determining a second location along the path in which the real vehicle was located in a second instance of time, at least based on the decision that the second location deviates from the path, wherein the second location most coincides with the second location; identifying a second log data file among the various log data files, wherein the second log data file represents the object in the real environment perceived by the real vehicle from the second location along the path in which the real vehicle was located in a second instance of time; and presenting the second log data file to the vehicle controller.

[0124] While the exemplary clauses described above illustrate one specific implementation, it should be understood that, in the context of this specification, the content of the exemplary clauses may also be implemented through methods, devices, systems, computer-readable media, and / or other implementations. Furthermore, any of the Example AUs may be implemented alone or in combination with any one or more of the other Example AUs.

[0125] conclusion While one or more examples of the techniques described herein have been described, various modifications, additions, substitutions, and equivalents thereof are included within the scope of the techniques described herein.

[0126] The illustrative descriptions refer to the accompanying drawings, which form part of this specification, illustrating specific examples of the subject matter claimed as illustrative. It should be understood that other examples may be used, and modifications or substitutions, such as structural changes, may be made. Such examples, modifications, or substitutions do not necessarily deviate from the scope of the subject matter intended to be claimed. While the steps in this specification may be presented in a particular order, in some cases the order may be changed to provide specific inputs at different times or in different orders without altering the functionality of the described systems and methods. The disclosed procedures may also be performed in different orders. In addition, the various calculations in this specification do not need to be performed in the order disclosed, and other examples using alternative orders of calculations can be readily implemented. Besides reordering, calculations may also be broken down into sub-calculations that produce the same result.

Claims

1. Receiving log data representing the actual environment in which the actual vehicle was operating for a certain period of time, Using the aforementioned log data, generate a simulated scenario for testing the autonomous vehicle's controller, wherein the simulated scenario includes a simulated environment that represents at least a portion of the real-world environment. The controller of the autonomous vehicle causes the simulated autonomous vehicle to traverse the simulated environment, wherein the simulated path of the simulated autonomous vehicle within the simulated environment may deviate from the actual path traversed by the actual vehicle within the real environment during the simulated time. In the simulated time, the simulated position of the simulated autonomous vehicle within the simulated environment is determined. Based at least in part on the log data, determine a first time within a given period in which the actual position of the actual vehicle in the real environment best matches the simulated position of the simulated autonomous vehicle that may deviate in the simulated environment, wherein the simulated time is different from the first time. From the log data, object data representing real objects in the real environment perceived by the real vehicle at the first time is obtained, Based at least partially on the object data, the simulated object representing the real object in the first time is updated in the simulated environment during the simulated time. Methods that include...

2. The simulated position includes a first simulated position, the real position includes a first real position, and the method is Determining a second simulated position of the simulated autonomous vehicle within the simulated environment, wherein the second simulated position is different from the first simulated position. Based at least in part on the log data, determine the second time within a given period in which the second real position of the real vehicle in the real environment best matches the second simulated position of the potentially deviating simulated autonomous vehicle in the simulated environment, and Updating the simulated object to match the real object in the real environment as perceived by the real vehicle at the second time, or At least one of the following: generating a second simulated object in the simulated environment that represents a second real object in the real environment as perceived by the real vehicle at the second time; The method according to claim 1, further comprising:

3. The simulated object includes a first simulated object, and the method further includes avoiding generating a second simulated object in the simulated environment that represents a second real object in the real environment as perceived by the real vehicle at a second time after the first time, at least partially based on the simulated position of the simulated autonomous vehicle or at least one of the first time. The method according to claim 1 or claim 2.

4. The aforementioned method, The process further includes determining whether the object data representing the aforementioned real-world object is associated with a static or dynamic object. Generating the simulated object representing the real object is at least in part based on determining that the object data is associated with a static object. The method according to any one of claims 1 to 3.

5. The simulated pose of the simulated autonomous vehicle in the simulated environment may deviate from the actual pose of the actual vehicle in the real environment. The method according to any one of claims 1 to 4.

6. Determining the position of the real vehicle in the real environment associated with the simulated position of the simulated autonomous vehicle that may deviate in the simulated environment, based at least in part on the log data, In the simulated environment, updating the simulated object that represents the real object in the real environment as perceived by the real vehicle from the position, The method according to any one of claims 1 to 5, further comprising:

7. The simulated autonomous vehicle decides to change its position within the simulated environment, Updating the simulated environment to include another simulated object representing a second object in the real environment as perceived by the real vehicle from a second position associated with the modified position of the simulated autonomous vehicle, The method according to any one of claims 1 to 6, further comprising:

8. Determining the position of the real vehicle includes determining the position of the real vehicle in the real environment that best matches the position of the simulated autonomous vehicle in the simulated environment. The method according to any one of claims 1 to 7.

9. Updating the simulation instructions to include the simulated environment, The simulation command is input to the autonomous vehicle's controller, and the autonomous vehicle's controller controls the simulated autonomous vehicle to traverse the simulated environment. The method according to any one of claims 1 to 8, further comprising:

10. The log data includes either simulated sensor data or actual sensor data captured by the sensors of the actual vehicle. The method according to any one of claims 1 to 9.

11. The controller of the autonomous vehicle controls the simulated autonomous vehicle, at least in part, based on perceiving the simulated object. The method according to any one of claims 1 to 10, further comprising:

12. The system receives input indicating a user-specified location containing another simulated object within the scope of the simulated environment, To generate the other simulated object at the user-specified location in the simulated environment, The method according to any one of claims 1 to 11, further comprising:

13. The log data includes a variety of log data instances representing the real environment in which the real vehicle or third vehicle was operating over a certain period of time, and determining the simulated position of the simulated autonomous vehicle is at least partially based on the variety of log data instances. The method according to any one of claims 1 to 12.

14. A system, One or more processors, One or more non-temporary computer-readable media that, when executed by the one or more processors, store instructions causing the system to perform the method according to any one of claims 1 to 13, A system that includes this.

15. One or more non-temporary computer-readable media that, when executed by a processor, store instructions causing the processor to perform the method according to any one of claims 1 to 13.

Images