Autonomous Systems Training and Testing

Abstract

The autonomous system training and testing operations may include generating a digital twin of the real-world scenario as a simulated environmental state of the simulated environment. The operations may also include running a sensor simulation model on the simulated environmental state iteratively through a plurality of time steps to obtain a simulated sensor output, obtaining at least one actuation action based on the simulated sensor output from a virtual driver of the autonomous system, updating an autonomous system state of the autonomous system based on the at least one actuation action, modeling a plurality of actors in the simulated environment according to the simulated environmental state using a plurality of actor models to obtain a plurality of actor actions, and updating the simulated environmental state according to the actor actions and the autonomous system state. The operations may further include evaluating the virtual driver after updating the simulated environmental state.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application and claims the benefit of U.S. Patent Application No. 63 / 308,042, filed February 8, 2022. U.S. Patent Application No. 63 / 308,042 is incorporated herein by reference in its entirety. Summary of the Invention [Problem to be solved by the invention]

[0002] background An autonomous system is an automated driving mode of transportation that does not require a human pilot or human driver to navigate and react to a real-world environment. Rather, an autonomous system includes a virtual driver, which is the decision-making portion of the autonomous system. Specifically, the virtual driver controls the operation of the autonomous system. The virtual driver is an artificial intelligence system that learns how to interact in the real world. As an artificial intelligence system, the virtual driver is trained and tested. However, because the virtual driver controls a mode of transportation in the real world, the training and testing of the virtual driver must be more rigorous than other artificial intelligence systems. [Means for solving the problem]

[0003] overview In one general aspect, a method may include generating a digital twin of a real-world scenario as a simulated environmental state of a simulated environment. The method may also include running a sensor simulation model on the simulated environmental state iteratively through a plurality of time steps to obtain a simulated sensor output, obtaining at least one actuation action based on the simulated sensor output from a virtual driver of the autonomous system, updating an autonomous system state of the autonomous system based on the at least one actuation action, modeling a plurality of actors in the simulated environment according to the simulated environmental state using a plurality of actor models to obtain a plurality of actor actions, and updating the simulated environmental state according to the actor actions and the autonomous system state. The method may further include evaluating the virtual driver after updating the simulated environmental state.

[0004] In one general aspect, the system may include a virtual driver of an autonomous system and a computer processor executing a simulator that causes the computer processor to perform operations. The operations may include generating a digital twin of the real-world scenario as a simulated environmental state of the simulated environment. The operations may also include running a sensor simulation model on the simulated environmental state iteratively through a plurality of time steps to obtain a simulated sensor output, obtaining at least one actuation action based on the simulated sensor output from the virtual driver of the autonomous system, updating an autonomous system state of the autonomous system based on the at least one actuation action, modeling a plurality of actors according to the simulated environmental state in the simulated environment using a plurality of actor models to obtain a plurality of actor actions, and updating the simulated environmental state according to the actor actions and the autonomous system state. The operations may further include evaluating the virtual driver after updating the simulated environmental state.

[0005] In one general aspect, a non-transitory computer-readable medium may include computer-readable program code for performing operations. The operations may include generating a digital twin of a real-world scenario as a simulated environmental state of a simulated environment. The operations may also include running a sensor simulation model on the simulated environmental state iteratively through a plurality of time steps to obtain a simulated sensor output, obtaining at least one actuation action based on the simulated sensor output from a virtual driver of the autonomous system, updating an autonomous system state of the autonomous system based on the at least one actuation action, modeling a plurality of actors according to the simulated environmental state in the simulated environment using a plurality of actor models to obtain a plurality of actor actions, and updating the simulated environmental state according to the actor actions and the autonomous system state. The operations may further include evaluating the virtual driver after updating the simulated environmental state.

[0006] Other aspects of the present invention will become apparent from the following description and the appended claims. [Brief description of the drawings]

[0007] [Figure 1] 1 illustrates a diagram of a system in accordance with one or more embodiments. [Diagram 2] 1 shows a more detailed diagram of a system according to one or more embodiments. [Diagram 3] 1 illustrates a flow diagram in accordance with one or more embodiments. [Figure 4] 1 illustrates a flow diagram in accordance with one or more embodiments. [Diagram 5] 1 illustrates a flow diagram in accordance with one or more embodiments. [Figure 6] 1 illustrates a flow diagram in accordance with one or more embodiments. [Figure 7] 1 illustrates a flow diagram in accordance with one or more embodiments. [Figure 8] 1 illustrates a flow diagram in accordance with one or more embodiments. [Figure 9] 1 illustrates a flow diagram in accordance with one or more embodiments. [Figure 10] 1 illustrates a flowchart in accordance with one or more embodiments. [Figure 11] 1 illustrates an example of a simulator in accordance with one or more embodiments. [Figure 12A] 1 illustrates an example according to one or more embodiments. [Figure 12B] 1 illustrates an example according to one or more embodiments. [Figure 13] 1 illustrates an example of various simulation scenarios in accordance with one or more embodiments. [Figure 14A] 1 illustrates an example of training in accordance with one or more embodiments. [Figure 14B] 1 illustrates an example of training in accordance with one or more embodiments. [Figure 14C] 1 illustrates an example of training in accordance with one or more embodiments. [Figure 15A] 1 shows an example of a simulator that performs a mixed reality simulation by modifying the behavior of actors from the real world. [Figure 15B] 1 shows an example of a simulator that performs a mixed reality simulation by modifying the behavior of actors from the real world. [Figure 15C] 1 shows an example of a simulator that performs a mixed reality simulation by modifying the behavior of actors from the real world. [Figure 16A] 1 illustrates a computing system in accordance with one or more embodiments of the present invention. [Figure 16B] 1 illustrates a computing system in accordance with one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Like elements in the various figures are designated with like reference numbers for consistency. Detailed Description In general, embodiments relate to training and testing of autonomous systems. An autonomous system is an automated driving mode of transportation that does not require a human pilot or human driver to navigate and react to a real-world environment. Rather, an autonomous system includes a virtual driver, which is the decision-making portion of the autonomous system. The virtual driver is an artificial intelligence system that learns how to interact in the real world. The autonomous system may be fully autonomous or semi-autonomous. In the transportation aspect, the autonomous system is housed in a housing configured to navigate the real-world environment. Examples of autonomous systems include autonomous vehicles (e.g., autonomous trucks and cars), drones, airplanes, robots, etc. The virtual driver is software that makes decisions, including moving, signaling, and stopping or maintaining a current state, that allow the autonomous system to interact with the real world.

[0009] The real-world environment is the portion of the real world that the autonomous system is designed to travel in once trained. Thus, the real-world environment may include interactions with concrete and land, people, animals, other autonomous systems, and human-driven systems, structures, and other objects as the autonomous system travels from an origin to a destination. To interact with the real-world environment, the autonomous system includes various types of sensors that are used to obtain measurements of the real-world environment, such as LiDAR sensors, among other types, and cameras that capture images from the real-world environment.

[0010] Testing and training virtual drivers for autonomous systems in real-world environments is unsafe due to the accidents that untrained virtual drivers may cause. Furthermore, training virtual drivers using only real-world log data does not capture rare events that may occur.

[0011] To test and train virtual drivers, embodiments include a simulator that models various scenarios. The modeling by the simulator includes the interaction of the autonomous system in the simulated environment and the reactions of other actors in the simulated environment. Thus, the simulator does not simply replay logged data of the real-world environment. Moreover, the simulator further adapts to the decisions of the autonomous system by building scenarios based on the autonomous system.

[0012] Referring now to the drawings, as shown in FIG. 1, a simulator (100) is configured to train and test virtual drivers of an autonomous system (102). For example, the simulator may be a unified, modular, mixed reality, closed-loop simulator for an autonomous system. The simulator (100) is a configurable simulation framework that allows for the evaluation of different autonomous components individually as well as as a complete system in a closed-loop manner. The simulator automatically reconstructs a "digital twin" of a real-world scenario, enabling accurate evaluation of the virtual driver at scale. The simulator (100) may also be configured to perform mixed reality simulations that combine real-world data with simulated data to create diverse and realistic evaluation variations and provide insight into the performance of the virtual driver. The mixed reality closed-loop simulation allows the simulator (100) to analyze the actions of the virtual driver for counterfactual "what-if" scenarios that did not occur in the real world. The simulator (100) further includes the capability to simulate and train rare but safety-critical scenarios for the entire autonomous system and closed-loop training to enable automatic and scalable refinement of autonomy.

[0013] The simulator (100) creates a simulated environment (104). The simulated environment (104) is a simulation of a real-world environment, which may or may not actually exist, through which the autonomous system is designed to navigate. Thus, the simulated environment (104) includes simulations of objects (i.e., simulated objects) and real-world backgrounds, including natural objects, structures, buildings and roads, obstacles, and other autonomous and non-autonomous objects. The simulated environment simulates environmental conditions in which the autonomous system may be deployed. Additionally, the simulated environment (104) may be configured to simulate various weather conditions that may affect inputs to the autonomous system. The simulated objects may include both stationary and non-stationary objects. The non-stationary objects are actors in the real-world environment.

[0014] The simulator (100) also includes an evaluator (110). The evaluator is configured to train and test the virtual driver (102) by creating various scenarios of the simulated environment. Each scenario is a configuration of the simulated environment, including, but not limited to, static portions, movements of simulated objects, actions of the simulated objects relative to each other, and reactions to actions taken by the autonomous system and the simulated objects. The evaluator (110) is further configured to evaluate the performance of the virtual driver using various metrics.

[0015] In one or more embodiments, the evaluator (110) is adversarial to the virtual driver of the autonomous system (102). By observing how the virtual driver (102) drives in various scenarios, the evaluator (110) learns where the virtual driver (102) may not meet the evaluation metrics, and then creates difficult scenarios based on what the evaluator has learned. The scenarios can then be used to test and / or train the virtual driver. During training, the evaluator (110) sends metric information to the virtual driver (102), which uses this information to update the virtual driver's parameters to better handle the scenarios. The more the evaluator (110) evaluates the virtual driver operating in a simulated environment, the better the evaluator will find flaws or difficulties or failures in the virtual driver. Thus, the goal of the evaluator (110) is to identify scenarios in which the virtual driver of the autonomous system (102) may perform suboptimally, and train and test on those scenarios.

[0016] As described above, the evaluator (110) evaluates the performance of the virtual driver throughout the execution of the scenario. Evaluating the performance may include applying rules. For example, the rules may be that the automated system does not collide with other actors, complies with safety and comfort criteria (e.g., passengers do not experience acceleration forces exceeding a certain acceleration force in the vehicle, the automated system does not deviate from the executed trajectory), or other rules. Each rule may be associated with metric information that associates the degree to which the rule is broken with a corresponding score. The evaluator (110) may be implemented as a data-driven neural network that learns to distinguish between good and bad driving behaviors. Various metrics of the evaluation system may be utilized to determine whether the automated system meets the requirements of the success criteria of a particular scenario. Furthermore, in addition to system-level performance, in the case of a module-based virtual driver, the evaluator may also evaluate individual modules, such as segmentation or prediction performance of actors in the scene, with respect to the ground truth recorded in the simulator.

[0017] FIG. 2 shows a more detailed diagram of a system according to one or more embodiments. As shown in FIG. 2, the system includes a simulator (100) connected to a data repository (105). The simulator is configured to operate in a number of stages selected by a stage selector (108) and in a mode selected by a mode selector (106). The stage selector (108) and the mode selector (106) may be graphical user interface or application programming interface components configured to receive a selection of a stage and a mode, respectively. The selected stages and modes define the configuration of the simulator (100). That is, the selected stages and modes define which system components communicate and the operation of the system components.

[0018] The phase may be selected using a phase selector (108). The phase (108) may be a training phase or a testing phase. In the training phase, the evaluator (110) provides metric information to the virtual driver (102), which uses the metric information to update the virtual driver (102). The evaluator (110) may further use the metric information to further train the virtual driver (102) by generating scenarios for the virtual driver. In the testing phase, the evaluator (110) does not provide metric information to the virtual driver. In the testing phase, the evaluator (110) uses the metric information to assess the virtual driver and develop scenarios for the virtual driver (102).

[0019] A mode may be selected by a mode selector (106). The mode defines the extent to which real-world data is used, whether noise is injected into the simulated data, the degree of perturbation of the real-world data, and whether the scenario is designed to be adversarial. Exemplary modes include an open-loop simulation mode, a closed-loop simulation mode, a single-module closed-loop simulation mode, a fuzzy mode, an adversarial mode, and a fuzzy mode. In an open-loop simulation mode, a virtual driver is evaluated using real-world data. In a single-module closed-loop simulation mode, a single module of the virtual driver is tested. An example of a single-module closed-loop simulation mode is a localizer closed-loop simulation mode, where a simulator evaluates how a pose estimated by a localizer drifts over time as a scenario progresses in the simulation. In a training data simulation mode, a simulator is used to generate training data. In a closed-loop evaluation mode, the virtual driver and the simulation system are run together to evaluate system performance. In an adversarial mode, the actor is modified to behave in an adversarial manner. In fuzzy mode, noise is injected into the scenario (e.g., to replicate signal processing noise and other types of noise). Other modes may exist without departing from the scope of the system.

[0020] Further, as shown in FIG. 2, the simulator (100) includes a controller (112) that includes functionality for configuring various components of the simulator (100) according to a selected mode and stage. That is, the controller (112) may modify the configuration of each component of the simulator based on the configuration parameters of the simulator (100). As described above with reference to FIG. 1, the simulator (100) includes an evaluator (110) and a simulated environment (104). The various components of the simulator (100) may also include an autonomous system model (116), a sensor simulation model (114), an asset model (117), an actor model (118), a latency model (120), a training data generator (122), and a simulated environment (104). Each of these components is described below.

[0021] The autonomous system model (116) is a detailed model of the autonomous system that the virtual driver will run on. The autonomous system model (116) includes the model, geometry, physical parameters (e.g., mass distribution, significant points), engine parameters, sensor locations and types, sensor firing patterns, information about the hardware that runs the virtual driver (e.g., processor power, amount of memory, and other hardware information), and other information about the autonomous system. Various parameters of the autonomous system model may be configurable by a user or another system.

[0022] For example, if the autonomous system is a vehicle, the modeling and kinematics may include the vehicle type (e.g., car, truck), make and model, geometry, physical parameters such as mass distribution, axle positions, engine type and performance, etc. The vehicle model may also include information about the sensors (e.g., cameras, LiDAR, etc.) on the vehicle, the relative firing synchronization patterns of the sensors, and the calibrated external (e.g., position and orientation) and internal (e.g., focal length) properties of the sensors. The vehicle model also defines the on-board computer hardware, sensor drivers, controllers, and the autonomy software release under test.

[0023] The vehicle kinematic simulation captures the virtual driver's actuation actions (e.g., steering angle, desired acceleration) and executes the actuation actions on the autonomous system in the simulated environment to update the state of the simulated environment and the autonomous system. A kinematic motion model may be used to update the state, or a kinematic dynamic motion model that considers the forces applied to the vehicle may be used to determine the state. By accessing real logged scenarios with ground truth actuation and vehicle states within the simulator at each time step, embodiments may also optimize analytical vehicle model parameters or learn parameters of a neural network that infers the new state of the autonomous system given the virtual driver output.

[0024] In one or more embodiments, the sensor simulation model (114) models active and passive sensor inputs in the simulated environment. The passive sensor inputs capture the visual appearance of the simulated environment, including stationary and non-stationary simulated objects, from the perspective of one or more cameras based on the simulated positions of the cameras in the simulated environment. Examples of passive sensor inputs include inertial measurement units (IMUs), and thermal active sensor inputs are inputs to the virtual driver of the autonomous system from active sensors such as LiDARs, RADARs, global positioning systems (GPSs), ultrasonics, etc. That is, the active sensor inputs include measurements taken by sensors, and the measurements are simulated based on the simulated environment based on the simulated positions of the sensors in the simulated environment. As an example, the active sensor measurements may be measurements that a LiDAR sensor makes to the simulated environment over time and in relation to the movement of the autonomous system.

[0025] The sensor simulation model (114) is configured to simulate sensor observations of the surrounding scene in the simulated environment (104) at each time step according to the sensor configuration on the vehicle platform. If the simulated environment directly represents the real-world environment without modification, the sensor output may be directly fed to the virtual driver. For light-based sensors, the sensor model simulates light as rays interacting with objects in the scene to generate the sensor data. Depending on the asset representation (e.g., of stationary and non-stationary objects), the embodiment may use graphic-based rendering for assets with textured meshes, neural rendering, or a combination of multiple rendering schemes. Leveraging multiple rendering schemes allows for customizable world-building with improved realism. Since assets are compositional in 3D and support a standard interface for rendering commands, different asset representations may be seamlessly composed to generate the final sensor data. Furthermore, in scenarios where one wants to replay what happened in the real world and use the same autonomous system as in the real world, the original sensor observations may be replayed at each time step.

[0026] The asset model (117) includes multiple models, each modeling an individual asset of a particular type in the real world. Assets may include inanimate objects such as construction barriers or traffic signs, parked vehicles, and backgrounds (e.g., vegetation or sky). Each of the entities in the scenario may correspond to an individual asset. Thus, an asset model, or an instance of a certain asset model, may exist for each of the entities or assets in the scenario. Assets may be composed together to form a simulated environment in three dimensions. The asset model provides all the information the simulator needs to simulate the asset. The asset model provides the information used by the simulator to represent and simulate the asset in the simulated environment. For example, the asset model may include geometry and bounding volumes, the asset's interaction with light at various wavelengths of interest (e.g., visible for cameras, infrared for LiDAR, microwave for RADAR), animation information describing deformations (e.g., rigging) and changes in lighting (e.g., turn signals), material information such as friction for different surfaces, metadata such as the asset's semantic class and key points of interest. A particular component of an asset may have different instantiations. For example, similar to a rendering engine, asset geometry may be defined in many ways: as a mesh, voxel, point cloud, analytical sign distance function, or neural network. Asset models may be created by artists, reconstructed from real-world sensor data, or optimized by algorithms to be adversarial.

[0027] Closely related and sometimes considered part of the set of asset models (117) are actor models (118). Actor models represent actors in a scenario. Actors are sentient creatures with independent decision-making processes. That is, in the real world, actors may be living beings (e.g., people or animals) that make decisions based on the environment. Actors have active movements rather than or in addition to passive movements. For each actor in a scenario, there may be an actor model or an instance of an actor model. An actor model is a model of an actor. If the actor is in a mode of transportation, the actor model includes a model of the transportation in which the actor is located. For example, actor models may represent pedestrians, children, vehicles driven by drivers, pets, bicycles, and other types of actors.

[0028] The actor model leverages the scenario specification and assets to control all actors in the scene and their actions at each time step. The actor behavior is modeled in a region of interest centered around the autonomous system. Depending on the scenario specification, the actor simulation controls the actors in the simulation to achieve the desired behavior. The actors can be controlled in various ways. One option is to leverage a heuristic actor model such as an intelligent-driver model (IDM) that attempts to maintain a certain relative distance or time-to-collision (TTC) from the leading actor, or a lane-changing actor model derived from a heuristic. Another is to directly replay actor trajectories from real-world logs, or to control actors with a data-driven traffic model. Through configurable design, embodiments may mix and match different subsets of actors controlled by different behavior models. For example, a distant actor that may not initially interact with the autonomous system and can follow a real-world logged trajectory may switch to a data-driven actor model once it comes close to the autonomous system. In another example, actors may be controlled by heuristic or data-driven actor models that still fit into high-level routes in the real world. This mixed reality simulation provides control and realism.

[0029] Additionally, actor models may be configured to be cooperative or adversarial. In cooperative mode, the actor model models actors that behave rationally in response to the conditions of the simulated environment. In adversarial mode, the actor model may model actors that behave irrationally, such as exhibiting aggressive or poor driving behavior.

[0030] The latency model (120) represents the timing latency that occurs when the autonomous system is in a real-world environment. There may be several sources of timing latency. For example, there may be a latency from when an event occurs on a sensor to detecting the sensor information from the event and sending the sensor information to the virtual driver. There may be another latency based on the difference between the computing hardware running the virtual driver in the simulated environment compared to the computing hardware of the virtual driver. Additionally, there may be another timing latency between the time the virtual driver sends an actuation signal for a change in the autonomous system (e.g., direction or speed) based on the actuation signal. The latency model (120) models various sources of timing latency.

[0031] In other words, in the real world, safety-critical decisions in the real world may involve fractions of a second, affecting response times. To ensure replication of the real-world behavior of the autonomous system, a key part of the simulation system is to simulate the exact timing and latency of the different components of the on-board system. One option to do this is to run the autonomous system on the exact same on-board computer hardware as the real-world simulation (i.e., hardware-in-the-loop (HIL)). However, using the same on-board computer is expensive and may be infeasible for simulations at scale. To allow scalable evaluation without strict requirements for exact hardware, the latency and timing of the different components of the autonomy and sensor modules are modeled while running on different computer hardware. The latency model may be a playback latency recorded from previously collected real-world data, or may have a data-driven neural network that infers the latency at each time step to match the HIL simulation setup.

[0032] The training data generator (122) is configured to generate training data (124) (described below). For example, the training data generator (122) may modify a real-world scenario to create a new scenario. The modification of a real-world scenario is referred to as mixed reality. For example, a mixed reality simulation may involve adding new actors with new behaviors, changing the behavior of one or more actors from the real world, and modifying the sensor data in the area while keeping the rest of the sensor data the same as the original log. In some cases, the training data generator (122) transforms a benign scenario into a safety-critical scenario.

[0033] The simulator is connected to a data repository (105). The data repository (105) is any type of storage unit or device configured to store data. The data repository (105) includes data collected from the real world. For example, the data collected from the real world includes real actor trajectories (126), real sensor data (128), real trajectories of the system capturing the real world (130), and real latency (132). Each of the real actor trajectories (126), real sensor data (128), real trajectories of the system capturing the real world (130), and real latency (132) is data captured by or calculated directly from one or more sensors from the real world (e.g., from real world logs). In other words, the data collected from the real world is actual events that occurred in the real world. For example, if the autonomous system is a vehicle, the real world data may be captured by the vehicle traveling in the real world using sensor equipment.

[0034] The data repository (105) also includes the ability to store training data (124). The training data (124) is data that replicates real-world data as if the scenario occurred in the real world. Thus, the training data (124) is generative data used to simulate real-world events. The training data (124) includes localization, mapping, and autonomy training data (134), behavior training data (136) that simulates the behavior of actors, and control training data (138) that simulates the control of an autonomous system.

[0035] Additionally, the data repository (105) includes the ability to store one or more scenario specifications (140). The scenario specifications (140) specify scenarios and evaluation settings for testing or training an autonomous system. For example, the scenario specifications (140) may describe the initial state of a scene, e.g., the current state of the autonomous system (e.g., full 6D pose, speed, and acceleration), map information specifying road layouts, and a scene layout specifying the initial state of all dynamic actors and objects in the scenario. The scenario specifications may also include dynamic actor information, which is an input to an actor model, describing how dynamic actors in the scenario should evolve over time. The dynamic actor information may include actor route information, desired behaviors or aggressiveness. The scenario specifications (140) may be specified by a user, programmatically generated using a domain-specification language (DSL), procedurally generated using heuristics from data-driven algorithms, or adversarial. The scenario specification (140) can also be tailored for data collected from real-world logs, such as taking place on a particular real-world map or having a subset of actors defined by their original locations and trajectories.

[0036] The interface between the virtual driver and the simulator matches the interface between the virtual driver and the real-world autonomous system. For example, the sensor simulation model (114) and the virtual driver match the virtual driver interacting with sensors in the real world. The virtual driver is the actual autonomy software running on the autonomous system. The simulated sensor data output by the sensor simulation model (114) may be in or converted to the exact message format that the virtual driver would take as input as if the virtual driver were in the real world, and the virtual driver can then be run as a black-box virtual driver built into sequentially launched components with simulated latency. The virtual driver then outputs the exact same control expressions that it uses to interface with the lower-level controller on the real autonomous system. The autonomous system model (116) then updates the state of the autonomous system in the simulated environment. Thus, the various simulation models of the simulator (100) are run asynchronously in parallel at their own frequencies to match the real-world settings.

[0037] 1 and 2 show configurations of components, other configurations may be used without departing from the scope of the invention. For example, various components may be combined to create a single component. As another example, functionality performed by a single component may be performed by two or more components.

[0038] 3-9 show examples of different modes of the simulator according to one or more embodiments. The various components shown in FIGS. 3-9 may be implemented using the components of FIGS. 1 and 2. FIG. 3 shows an example of an open-loop evaluation that may be implemented by the simulator. An open-loop evaluation is a "log replay" in which a previously recorded log collected by the same autonomous system platform in the real world (e.g., having real sensor data (304), real trajectory of the autonomous system (306), and real actor trajectory (308)) is replayed in the simulation with the exact same messages and timing, but with a different virtual driver (310) to evaluate the behavior changes. Each of the models of FIG. 2 (i.e., actor simulation, sensor simulation, latency model, vehicle model) can replay their respective messages in the simulation, and the evaluator (302) can monitor the output of the autonomy at each time step. As shown in FIG. 3, the virtual driver does not interact with the simulator or act on its outputs. Rather, the virtual driver simply generates outputs given the replayed inputs.

[0039] FIG. 4 shows an example of an open-loop evaluation with the simulator implementing a fuzz test mode (400). In other words, the fuzzy modules (402, 404, 406, 408) add perturbations to the real sensor data (410), the real trajectory of the autonomous system (412), and the real actor trajectory (414), and the real latency (416). The simulator can also implement mixed reality in this setting and perform a "fuzz test" where real-world input messages are perturbed with realistic noise. Thus, the change in the performance of the virtual driver (418) can be analyzed by the evaluator (420) to determine how well the virtual driver (418) reacts to the noise. For example, the simulator can inject noise into various modules, such as dropping sensor messages, modifying pixel colors or LiDAR point clouds, or increasing latency between modules, and verifying whether the autonomy is robust to such perturbations.

[0040] FIG. 5 shows an example of a closed-loop evaluation mode (500). Open-loop evaluation reproduces the real world but does not allow the autonomous system or other actors in the scene to deviate from the real-world trajectory. Closed-loop evaluation mode addresses what-if scenarios, e.g., "how would the virtual driver behave on a different platform where the actors behave differently, or if the scenario / map / actor behavior is completely unknown." Specifically, in closed-loop evaluation mode (500), the simulated environment state is a three-dimensional virtual environment in which the virtual driver is located. The sensor simulation model runs in the simulated environment state to generate realistic sensor inputs that are detected by sensors located in the simulated environment on the autonomous system. That is, the sensor simulation model takes the simulated environment state as input and builds the sensor inputs to the virtual driver. The latency model adds latency to the timing of the sensor inputs to reflect how the virtual driver receives and processes the sensor data in the real world. The virtual driver outputs actuation actions to control the autonomous system. The actuation actions are passed to the autonomous system model, which modifies the positioning of the autonomous system in the simulated environment based on the actuation actions. For example, the autonomous system model may reflect increasing speed, braking, turning, or other actions in the simulated environment. The actor model (504) also updates according to the simulated environment conditions and updated simulated environment conditions. That is, the actor model reacts to the actions of the virtual driver. The evaluator can then evaluate the actions of the virtual driver.

[0041] As shown, in closed-loop evaluation mode, the virtual driver (502) and actor model (504) update their behavior at each time step given the current scene, and new sensor observations reflecting the scene are simulated. In closed-loop evaluation, the simulated environment state is updated based on decisions by both the virtual driver and the actor model. The result is a more realistic system performance test, since various components of the autonomous system, including localization and sensor perception, are being tested. Thus, changes in perception processing, newly added sensor configurations, or changes in the computational platform may be evaluated for whether the changes improve the overall system performance as defined by the evaluator (506).

[0042] FIG. 6 illustrates an example of a modular level evaluation mode (600). The modular level evaluation mode may be used to individually test a single module of the virtual driver. For example, the single module may be a localization module, a motion planning, or a controller module. One or more embodiments may configure the simulator to enable certain components and disable certain components to test only certain parts of the virtual driver. For example, a sensor simulation may be run at each time step only to test localizer performance and measure drift, or realistic perception and prediction outputs may be modeled only to test the motion planner. These evaluations may be performed in either an open-loop or closed-loop setting, depending on the configuration specified for the other simulation modules.

[0043] FIG. 6 shows an example of evaluating only the self-localization system in an open-loop system. As shown, the localizer module of the virtual driver receives the sensor simulation model output along with the latency from the latency model. The localizer output is passed to an evaluator that performs the evaluation. Separately, a predefined custom trajectory, which may or may not represent the real world, is used as an input to determine an updated simulated environment state. The actor model executes based on the simulated environment state and the updated simulated environment state that is not influenced by the localizer.

[0044] FIG. 7 shows an example of a training data generation mode (700). The rendering system may be used to simulate sensor data offline to train perception models as part of training data generation. In the simulator, assets can be built from real-world data in addition to artist-derived CAD models, so the assets have more variety to test more scenario variations. Furthermore, customizable physical and neural rendering techniques allow for more realistic simulated sensor data. In addition to generating training data for perception, the simulator can also generate training data for either the simulation or autonomy modules, as well as offline calibration, mapping, and auto-labeling models. Training data generation improves cost efficiency and development time for all components of autonomy and simulation.

[0045] FIG. 8 illustrates an example of implementation in adversarial mode (800). The simulator can run scenarios defined by a simulation specification, as described above with reference to FIG. 5. An evaluator evaluates the performance of the virtual driver against various metrics. The output of the evaluator is passed to one or more adversarial actors (802). For example, in FIG. 8, the adversarial actor receives feedback that configures an adversarial model to impede the operation of a self-driving vehicle (SDV). Specifically, the output of the evaluator matches the lowest performing metric information of the virtual driver. For example, the adversarial actor may be a car that cuts in front of the virtual driver if the virtual driver misbehaves by causing a collision. By using the adversarial actor, a simulation is performed to identify and perform the virtual driver in safety-critical scenarios where the virtual driver may have difficulty. Thus, one or more embodiments address the problem of using virtual drivers in the real world, where there is a wide distribution of real-world scenarios and a long tail of rare events. Moreover, end-to-end closed-loop testing unlocks new capabilities to automatically search and find challenging scenarios for the virtual driver on its behalf, as well as to train or evaluate the virtual driver on the most critical scenarios covering real-world distributions.

[0046] For example, given a simulation specification with optimizable parameters x (e.g., actor placement, actor behavior or route, weather conditions, etc.), the simulator acts as a black-box function g and the virtual driver output runs as a function of the input sensor data as f. The evaluator ε generates a score that assesses the performance of the virtual driver in the simulated environment. A search for "adversarial" scenarios that are difficult for the virtual driver may be found using the following equation (Equation 1):

[0047]

number

[0048] In Equation 1, R sets constraints to ensure that distinct samples of x are in the set of feasible scenarios. To identify adversarial scenarios, different optimization modules such as Bayesian optimization or gradient estimation methodologies may be leveraged.

[0049] FIG. 9 shows an example of a closed-loop training mode (900). Similar to adversarial scenario identification, the evaluator can be used to learn from the simulator itself and provide feedback directly to the virtual driver to improve the performance of the virtual driver in the simulator. The simulator uses closed-loop training for any traffic scenario, including real-world dense traffic areas. Closed-loop training can ensure that the virtual driver and its various components are automatically optimized for the end goal of driving performance.

[0050] Through the machine learning process, the overall system becomes aware of the causes of failures and the autonomous system learns how to avoid such failures in the future. Similarly, the evaluator may provide feedback on areas where the virtual driver may not have performed optimally.

[0051] FIG. 10 shows a flow diagram for running the simulator in a closed-loop mode. In block 1002, a digital twin of a real-world scenario is generated as a simulated environment state. Log data from the real world is used to generate an initial virtual world. The log data defines which asset and actor models are used, as well as the initial positioning of the assets. For example, various asset types in the real world may be identified using a convolutional neural network on the log data. As another example, an offline perception system and human annotation of the log data may be used to identify asset types. Corresponding asset and actor models may then be identified based on the asset types and added to the real actor and asset positions in the real world. Thus, the assets and actor models create an initial three-dimensional virtual world.

[0052] In block 1003, a sensor simulation model is executed on the simulated environment conditions to obtain simulated sensor outputs. The sensor simulation model may use beamforming and other techniques to replicate views to the sensors of the autonomous system. Each sensor of the autonomous system has a corresponding sensor simulation model and a corresponding system. The sensor simulation model is executed based on the position of the sensor in the virtual environment to generate simulated sensor outputs. The simulated sensor outputs are in the same form as those received from the real sensor by the virtual driver.

[0053] The simulated sensor output is passed to a virtual driver. In block 1005, the virtual driver executes based on the simulated sensor output to generate an actuation action. The actuation action defines how the virtual driver controls the autonomous system. For example, for an SDV, the actuation action may be an amount of acceleration, a steering movement, a triggering of a turn signal, etc. From the actuation action, in block 1007, the autonomous system state in the simulated environment is updated. The actuation action is used as an input to the autonomous system model to determine the actual action of the autonomous system. For example, the autonomous system model may use the actuation action in addition to road and weather conditions to represent the resulting movement of the autonomous system. For example, in a wet or snowy environment, the same amount of acceleration action as in a dry environment may cause less acceleration than in a dry environment. As another example, the autonomous system model may consider the possibility of a possibly failed tire (e.g., tire slip), machine-based latency, or other imperfections in the autonomous system.

[0054] At block 1009, the actions of the actors in the simulated environment are modeled based on the simulated environment state. Concurrently with the virtual driver model, the actor model and asset model are executed in the simulated environment state to determine updates to each of the assets and actors in the simulated environment. Here, the actor actions may test the virtual driver using the previous output of the evaluator. For example, if the actor is hostile, the evaluator may indicate the lowest scoring metric for the virtual driver based on the previous actions of the virtual driver. Using the mapping of metrics to the actions of the actor model, the actor model is executed to utilize or test that particular metric.

[0055] Thus, in block 1011, the updated simulated environment state is updated according to the actor actions and the autonomous system state. The updated simulated environment includes changes in the positions of the actors and the autonomous system. Because the model runs independently of the real world, the updates may reflect deviations from the real world. Thus, the autonomous system is tested in new scenarios. In block 1013, a decision is made to continue. If the decision is made to continue, in block 1003, the testing of the autonomous system continues using the updated simulated environment state. In each iteration, during training, the evaluator provides feedback to the virtual driver. Thus, the parameters of the virtual driver are updated to improve the performance of the virtual driver in various scenarios. During testing, the evaluator can be tested using various scenarios and patterns, including edge cases that may be safety-critical. Thus, one or more embodiments improve the virtual driver and increase the safety of the virtual driver in the real world.

[0056] As shown, the virtual driver of the autonomous system acts based on the scenario and currently learned parameters of the virtual driver. The simulator captures the actions of the autonomous system and provides the reactions in the simulated environment to the virtual driver of the autonomous system. The evaluator evaluates the performance of the virtual driver and creates scenarios based on the performance. The process may continue as the autonomous system operates in the simulated environment.

[0057] The virtual driver, evaluator, and other components and models of the system may be or include one or more machine learning models trained to perform actions. Exemplary machine learning models that may be used include neural network models, decision trees, and other models. The goal is to teach the virtual driver of the autonomous system to drive intuitively, intelligently, and safely. As shown in FIG. 11, the simulator is a scalable high-fidelity closed-loop simulator as shown. Using artificial intelligence (AI), the simulated environment is an immersive and reactive environment in which tests can be designed, skills assessed, and the autonomous driving "brain" (i.e., the virtual driver) can be taught to learn to drive on its own. Using the simulated environment and simulator, the virtual driver is exposed to a vast and diverse set of experiences necessary to hone its driving skills, including both common driving scenarios and safety-critical edge cases. The goal is to reduce the need to drive test miles in the real world, resulting in a safer and more affordable solution.

[0058] The simulator may have at least four core capabilities: (1) build a digital twin of the world from data automatically, semi-automatically, and at scale; (2) conduct near real-time, high-fidelity sensor simulation that allows for testing the entire software stack in an immersive and reactive manner; (3) create scenarios to stress test virtual drivers automatically and at scale; and (4) teach the virtual drivers to learn from their mistakes and master the skill of driving without human intervention.

[0059] Simulators build digital twins of the world from data automatically and at scale. To be effective, simulators need to recreate the real world with high fidelity, in all of its variety and dynamism. The simulated environment leverages AI to reconstruct the geometry, appearance, and material properties of real-world objects and backgrounds from sensor data such as LiDAR returns and camera images. This allows simulators to automatically recreate digital twins of the world from everywhere we drive, with real-world variety, scale, and realism.

[0060] The simulator reconstructs the reality as seen and uses AI to recreate a wide range of simulated objects and backgrounds. These objects and backgrounds can be used in the simulation system to create new safety-critical scenarios to test the autonomous system. Thus, the simulator uses simulation, including LiDAR simulation, camera simulation, actor simulation, object reconstruction, and scenario generation and testing. For example, as shown in FIG. 12A, the left side of the street (1202) is a virtual world reconstruction of the same street as the right side of the street (1204). In the simulated environment, the input to the virtual driver is designed to simulate the real world. The simulation may not be an accurate depiction of real-world conditions, as the simulation may include scenario modifications. For example, as shown in FIG. 12B, a fake vehicle is added to the camera and LiDAR simulation (1208) that is used as an input to the virtual driver. Furthermore, as shown in FIG. 12B, the simulated environment is immersive for the virtual driver.

[0061] Simulators perform near real-time, high-fidelity sensor simulation, enabling testing of the entire software stack in an immersive and reactive manner. Reproducing the world accurately is a challenge. However, for a simulator to truly replace driving in the real world, the simulator should behave in the simulation in the same way as the real world. The simulator achieves this by simulating how a virtual driver observes or "sees" the virtual world through its sensors in the same way that it sees the real world. This is the only way to properly test the entire virtual driver in the simulation and teach the virtual driver how to drive.

[0062] The simulator leverages AI along with simplified, physics-based rendering to simulate realistic sensor data in near real-time. Combined with a high-quality recreated virtual world, AI algorithms learn to make the physics approximations appear more realistic while being computationally more efficient than traditional complex physics simulators.

[0063] The automatic generation of high-quality objects and virtual worlds allows the simulator to not only recreate reality as seen, but also to modify it by removing, adding or changing the behavior of "actors" (e.g., other simulated objects) in the scenario and resimulating sensors. This allows the simulator to create an infinite variety of worlds for the virtual driver to experience, unlocking the ability to realistically test the complete software stack across interesting or safety-critical edge cases and helping the virtual driver learn sophisticated driving skills.

[0064] For example, the simulator may simulate a real-world scenario. The simulator then updates the scene with a new fully simulated "actor" performing a given maneuver (e.g., lane change (left to right)) and generates simulated sensor data. This can be done on a large scale, and existing scenarios can be taken and modified to test virtual drivers. As shown in FIG. 13, multiple scenarios (1300) may be generated, with each scenario being used to test and train a virtual driver.

[0065] The simulated environment can be used to test different sensors and sensor configurations as well as vehicle platforms before the sensors and vehicles exist. The simulator can be used to rapidly design and validate new sensor configurations because it can teach a virtual driver how to use them even before they exist in the real world, ultimately enabling much more rapid development of self-driving cars and other autonomous systems.

[0066] The simulated environment automatically creates scenarios to stress-test virtual drivers. The simulator uses AI to create traffic scenarios to test and train virtual drivers, generating any kind of variation with any kind of traffic behavior, across any kind of geography, automatically and at scale.

[0067] A scenario is not simply a static scenario that is played out like a movie. Driving is an interactive experience, and the simulator replicates this interactive experience. The replication of the interactive experience is a closed-loop simulation as shown in Figure 14. Every action of the virtual driver has a reaction in the simulated environment. Specifically, the simulator moves the "actors" in the scenario forward and performs actions, simulated sensors looking at the updated simulated environment tell the virtual driver what he would observe, and the virtual driver then decides how it will react. The simulator then moves the virtual driver in the virtual world according to that decision, and the other traffic participants move accordingly. This loop continues constantly.

[0068] This type of simulation allows the virtual driver to truly experience how the scenario would play out if the virtual driver were in the real world and driving an autonomous vehicle. Thus, one or more embodiments allow for an accurate evaluation of the performance of a software stack.

[0069] The simulator uses AI to identify the weaknesses of the virtual driver and automatically creates adversarial scenarios that the driver will have difficulty dealing with. That is, the simulator with the evaluator intentionally plays against the virtual driver, identifying and exploiting their weaknesses while the virtual driver simultaneously learns its skills, i.e., scenarios versus driving skills, i.e., one AI system versus another (e.g., evaluator versus virtual driver).

[0070] Although it may seem counterintuitive, the goal is to see the virtual driver fail. The virtual driver's failure in a simulated environment reduces the likelihood of failure in the real world. Thus, the simulator teaches the virtual driver to learn from its mistakes and master the skill of driving independently.

[0071] Building a simulator that can recreate the world, simulate sensor data, and generate infinite test scenarios all serves one big, audacious objective: to teach a virtual driver to learn alone to drive safely in any vehicle, in any scenario, anywhere in the world.

[0072] For example, a virtual driver may initially have difficulty negotiating a difficult lane merge. First, the Novice virtual driver crashes into another actor, as shown in FIG. 14A. However, through closed-loop training, the virtual driver can learn over time to get better. The Intermediate virtual driver brakes to let the other vehicle through, as shown in FIG. 14B. After learning more in the simulated environment, the Advanced virtual driver realizes that the optimal maneuver is to not brake and accelerate a little smoothly, as shown in FIG. 14C, so as not to cause difficulty for the other actors. The evaluator provides feedback to the virtual driver.

[0073] The simulator exposes the virtual driver to a vast amount of diverse experience needed to hone his driving skills, including common driving scenarios and more obscure safety-critical edge cases. The simulator also delivers feedback to the virtual driver on his performance after each decision. The simulator's feedback system enables the AI ​​to learn from its mistakes independently, in an immersive and reactive manner. The virtual driver is constantly learning automatically from its actions to become a smarter driver over time.

[0074] In some cases, multiple clones of a virtual driver are run and updated simultaneously, learning to drive safely in parallel with each other in different scenarios. Each clone has the same AI network that is updated based on each scenario. For example, a virtual driver can simultaneously learn how to drive on a quiet suburban street, a five-lane highway, and in the middle of a city at rush hour.

[0075] Through the simulator, the virtual driver is dropped into a simulated environment where the virtual driver can see and behave as if the virtual driver were in the real world. The simulator then creates traffic scenarios to test the virtual driver, generating variations of the scenarios that mimic calm afternoons, rush hours, etc. The simulator reacts in real time to the virtual driver's behavior to create interesting interactions. Every action has a reaction, the virtual driver makes a choice, the traffic reacts. The virtual driver evolves through multiple scenarios, and the virtual driver's rating evolves through various scenarios. Thus, over time, the virtual driver may start as a novice driver and then learn how to drive safely.

[0076] For example, as shown in Figures 15A, 15B, and 15C, the simulator can create a mixed reality world where actor actions deviate from the real world. The left image (1502) of Figures 15A, 15B, and 15C shows a real world image captured via a real camera on an autonomous system. The right image (1504) of Figures 15A, 15B, and 15C shows a mixed reality image that deviates from a real event. As shown based on a comparison between the left and right images, a car in front of a virtual driver turns left in the real world, while in the simulated environment, it cuts in front of the virtual driver and continues to go straight. As shown, through a single data capture of the real world, the simulator can create multiple scenarios (e.g., a car turns left, a car cuts in the same lane as the virtual driver) to fully test a virtual driver.

[0077] The embodiments may be implemented on a computing system specifically designed to achieve improved technical results. When implemented in a computing system, the features and elements of the present disclosure provide significant technical advances over computing systems that do not implement the features and elements of the present disclosure. Any combination of mobile, desktop, server, router, switch, embedded device, or other type of hardware may be improved by including the features and elements described in the present disclosure. For example, as shown in FIG. 16A, a computing system (1600) may include one or more computer processors (1602), non-persistent storage (1604), persistent storage (1606), communication interfaces (1608) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functions that implement the features and elements of the present disclosure. The computer processor (1602) may be an integrated circuit for processing instructions. The computer processor may be one or more cores or micro-cores of a processor. The computer processor (1602) includes one or more processors. The one or more processors may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), combinations thereof, and the like.

[0078] The input device (1610) may include a touch screen, keyboard, mouse, microphone, touch pad, electronic pen, or any other type of input device. The input device (1610) may receive input from a user in response to data and messages presented by the output device (1612). The input may include text input, audio input, video input, etc., which may be processed and transmitted by the computing system (1600) according to the present disclosure. The communication interface (1608) may include integrated circuits for connecting the computing system (1600) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or to another device, such as another computing device.

[0079] Additionally, the output device (1612) may include a display device, a printer, an external storage device, or any other output device. One or more of the output devices may be the same as or different from the input devices. The input and output devices may be locally or remotely connected to the computer processor (1602). There are many different types of computing systems, and the aforementioned input and output devices may take other forms. The output device (1612) may display data and messages sent or received by the computing system (1600). The data and messages may include text, audio, video, etc., and may include the data and messages described above in other figures of this disclosure.

[0080] Software instructions in the form of computer readable program code for implementing the embodiments may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium, such as a CD, DVD, storage device, diskette, tape, flash memory, physical memory, or any other computer readable storage medium. In particular, the software instructions may correspond to computer readable program code configured, when executed by a processor, to implement one or more embodiments, which may include sending, receiving, presenting, and displaying data and messages as described in other figures of this disclosure.

[0081] The computing system (1600) of FIG. 16A may be connected to or may be part of a network. For example, as shown in FIG. 16B, the network (1620) may include multiple nodes (e.g., node X (1622), node Y (1624)). Each node may correspond to a computing system such as the computing system shown in FIG. 16A, or a group of nodes combined may correspond to the computing system shown in FIG. 16A. As an example, the embodiment may be implemented on a node of a distributed system connected to other nodes. As another example, the embodiment may be implemented on a distributed computing system having multiple nodes, each part may be located on a different node in the distributed computing system. Furthermore, one or more elements of the computing system (1600) described above may be located remotely and connected to other elements via a network.

[0082] Nodes in the network (1620) (e.g., node X (1622), node Y (1624)) may be configured to provide services to client devices (1626), including receiving requests and sending responses to the client devices (1626). For example, the nodes may be part of a cloud computing system. The client devices (1626) may be computing systems, such as the computing system illustrated in FIG. 16A. Additionally, the client devices (1626) may include and / or implement all or a portion of one or more embodiments.

[0083] The computing system of FIG. 16A may include functionality for presenting raw data and / or processed data, such as the results of comparisons and other processing. For example, presenting the data may be accomplished by various presentation methods. Specifically, the data may be presented by being displayed on a user interface, transmitted to different computing systems, and stored. The user interface may include a graphical user interface (GUI) that displays information on a display device. The GUI may include various GUI widgets that organize what data is displayed and how the data is presented to the user. Additionally, the GUI may present data directly to the user, e.g., data presented as actual data values ​​via text, or data rendered by the computing device into a visual representation of the data, such as by visualizing a data model.

[0084] As used herein, the term "connected to" contemplates multiple meanings. A connection may be direct or indirect (e.g., through another component or network). A connection may be wired or wireless. A connection may be a temporary, permanent, or semi-permanent communication channel between two entities.

[0085] The various illustrations of the figures may be combined and may include or be included within features described in other figures of the present application. Various elements, systems, components, and steps shown in the figures may be omitted, repeated, combined, and / or modified as shown in the figures. Thus, the scope of the disclosure should not be considered limited to the specific configurations shown in the figures.

[0086] In this application, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives of elements (i.e., any noun in this application). The use of ordinal numbers does not imply or create a particular order of elements, nor does it limit any element to only a single element, unless expressly disclosed, such as by use of "before," "after," "single," and other such terms. Rather, the use of ordinal numbers is to distinguish elements. As an example, a first element is different from a second element, a first element may encompass two or more elements, and may follow (or precede) the second element in the order of elements.

[0087] Additionally, unless otherwise stated, "or" is an inclusive or, and thus includes "and." Furthermore, items joined by "or" may include any combination of the items, any number of each item, unless otherwise stated.

[0088] In the above description, numerous specific details are described to provide a more complete understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description. Furthermore, other embodiments not expressly described above may be devised that do not depart from the scope of the claims disclosed herein. Thus, the scope should be limited only by the scope of the appended claims.

Claims

1. generating a digital twin of the real-world scenario as a simulated environmental state of the simulated environment; Iteratively through multiple time steps, running a sensor simulation model on the simulated environmental conditions to obtain simulated sensor outputs; obtaining at least one actuation action based on the simulated sensor output from a virtual driver of an autonomous system; updating an autonomous system state of the autonomous system based on the at least one actuation action; using a plurality of actor models to model a plurality of actors in the simulated environment according to the simulated environment state to obtain a plurality of actor actions; updating the simulated environment state according to the plurality of actor actions and the autonomous system state; evaluating the virtual driver after updating the simulated environmental conditions; A method comprising:

2. The method of claim 1 , further comprising: the simulated sensor outputs replicating a format of inputs to the virtual driver from multiple sensors on the autonomous system in the real world.

3. The method of claim 2 , wherein the simulated sensor output comprises a LiDAR sensor output.

4. generating an image from a viewpoint of a camera of the autonomous system; presenting the image to the virtual driver; The method of claim 2 further comprising:

5. adding a new actor to the plurality of actors to create a mixed reality scenario from the real-world scenario; modeling the behavior of the new actor in the real-world scenario using an actor model of the new actor; further comprising The method of claim 1 , wherein updating the simulated environment state takes into account the behavior of the new actor, and the virtual driver reacts to the behavior of the new actor.

6. modifying behavior of existing actors of the plurality of actors to create a mixed reality scenario from the real-world scenario; and modeling the behavior of the existing actor in the real-world scenario using actor models of the existing actors, wherein the updated autonomous system state takes into account the behavior of the existing actors; The method of claim 1 , wherein updating the simulated environment state takes into account the behavior of the existing actors, and the virtual drivers react to the behavior of the existing actors.

7. The method of claim 1 , further comprising training the virtual driver to modify the virtual driver based on evaluating the virtual driver.

8. The method of claim 1 , further comprising: performing adversarial training of the virtual driver by modifying behavior of the plurality of actors based on the evaluation of the virtual driver.

9. The method of claim 1 , further comprising using a latency model to model multiple latencies of the virtual driver running on the autonomous system in the real-world scenario.

10. 10. The method of claim 9, wherein the plurality of latencies includes a sensor latency for transmitting sensor output to the virtual driver, a computer hardware latency of the virtual driver executing on the autonomous system, and an autonomous system latency for updating the autonomous system state based on the at least one actuation action.

11. a virtual driver for an autonomous system; and a computer processor running a simulator, said simulator causing said computer processor to perform the method of any one of claims 1 to 10.

12. A computer program causing a computer system to carry out the method according to any one of claims 1 to 10.

Images