Multi-agent simulation

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, for generating a simulation of an environment with which a plurality of agents interact over a plurality of time steps, wherein the simulation includes a respective simulation state for each time step, the respective simulation state specifying the respective state of each agent at that time step. In one aspect, a method includes, for each time step: obtaining a current simulation state for the current time step; generating a plurality of candidate next simulation states for a next time step; for each candidate next simulation state, determining a discrimination score representing a likelihood that the candidate next simulation state is a true simulation state; and selecting the candidate next simulation state as the simulation state for the next time step based on the discrimination score of the candidate next simulation state.

Description

Technical Field Background Art

[0001] This specification deals with using machine learning models to simulate environments.

[0002] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate an output based on the received input and the values ​​of the model's parameters.

[0003] Some machine learning models are deep models that use multiple layers to generate outputs for their inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to its input to generate an output. Summary of the Invention

[0004] This specification describes a simulation system implemented as a computer program on one or more computers in one or more locations that can learn to control a set of agents interacting with a simulated environment so that their behavior realistically simulates the behavior of real-world agents.

[0005] According to a first aspect, there is provided a method for generating a simulation of an environment with which a group of agents interact over a sequence of time steps, performed by one or more data processing devices. The simulation includes a corresponding simulation state for each time step, the corresponding simulation state specifying the corresponding state of each agent at the time step. The method includes, for each time step, obtaining a current simulation state for the current time step. Based on the current simulation state, a set of candidate next simulation states for the next time step is generated. Generating each candidate next simulation state includes, for each agent, sampling a corresponding action from a set of possible actions that can be performed by the agent. For each agent, determining the corresponding next state of the agent at the next time step if the agent performs the corresponding sampled action at the current time step. For each candidate next simulation state, determining a discrimination score that characterizes the likelihood that the candidate next simulation state is a true simulation state. Based on the discrimination score of the candidate next simulation state, the candidate next simulation state is selected as the simulation state for the next time step.

[0006] In some implementations, the agent is a vehicle in the environment.

[0007] In some implementations, the set of possible actions that can be performed by the agent includes an action to adjust the agent's steering angle.

[0008] In some implementations, the set of possible actions that can be performed by the agent includes an action to adjust the acceleration of the agent.

[0009] In some implementations, the state of an agent at a time step includes: (i) the position of the agent at the time step, and (ii) the motion of the agent at the time step.

[0010] In some implementations, the position of the agent includes: (i) the spatial position of the agent in the environment, and (ii) the heading of the agent in the environment.

[0011] In some implementations, the motion of the agent includes: (i) the velocity of the agent, and (ii) the acceleration of the agent.

[0012] In some implementations, the method further includes, for each agent, obtaining a representation of a current state of the agent in the environment, and processing the representation of the current state of the agent in the environment using a policy neural network to generate a corresponding probability distribution over a set of possible actions for the agent. For each agent, sampling a corresponding action from the set of possible actions may include, for each agent, sampling an action from the set of possible actions according to the probability distribution over the set of possible actions for the agent.

[0013] In some implementations, determining, for each agent, the corresponding next state of the agent at the next time step if the agent performs the corresponding sampled action at the current time step includes, for each agent, processing data representing (i) the current state of the agent and (ii) the sampled action of the agent using a motion model to generate the next state of the agent at the next time step.

[0014] In some implementations, determining a discrimination score that characterizes the likelihood that the candidate next simulated state is the true simulated state includes obtaining a representation of the candidate next simulated state. The representation of the candidate next simulated state can be processed using a discriminator neural network to generate a discrimination score that characterizes the likelihood that the candidate next simulated state is the true simulated state.

[0015] In some implementations, obtaining a representation of the candidate next simulation state includes, for each agent, obtaining a representation of the agent's next state at a next time step corresponding to the candidate next simulation state. Processing the representation of the candidate next simulation state using a discriminator neural network may include, for each agent, processing the representation of the agent's next state at a next time step corresponding to the candidate next simulation state using a discriminator neural network to generate an agent-specific discrimination score representing a likelihood that the agent's next state is the true agent state. A discrimination score representing the likelihood that the candidate next simulation state is the true simulation state may be generated based on the agent-specific discrimination score.

[0016] In some implementations, generating a discrimination score that characterizes the likelihood that the candidate next simulation state is the true simulation state based on the agent-specific discrimination scores includes computing a minimum of the agent-specific discrimination scores.

[0017] In some implementations, the discriminator neural network is trained to generate a discrimination score that characterizes an environment state as real if the environment state is a real-world environment state.

[0018] In some implementations, selecting the candidate next simulated state as the simulated state for the next time step based on the discrimination scores of the candidate next simulated states includes selecting the candidate next simulated state with the highest discrimination score as the simulated state for the next time step.

[0019] According to a second aspect, a system is provided, comprising one or more computers and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the method of the first aspect.

[0020] According to a third aspect, there is provided one or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of the first aspect.

[0021] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0022] This specification describes a simulation system that can control a set of multiple simulated agents (e.g., vehicles) that interact with an environment so that their behavior realistically simulates the behavior of real-world agents (e.g., vehicles controlled by human drivers). The simulation system can be used to improve the operation of a planning system that selects actions to be performed by a partially or fully autonomous vehicle. For example, the simulation system can be used to generate a large number of simulations in which the planning system controls one vehicle and the remaining vehicles are controlled by the simulation system. These simulations can be used to identify and resolve potential "failures" in the planning system (e.g., where the planning system selects actions that cause the vehicle to operate unsafely) before the planning system is used to control the vehicle in a real-world environment. As another example, the planning system of a vehicle can use the simulation system to predict the likely future behavior of other vehicles in the environment, and then use these behavior predictions to more safely and effectively control the operation of the vehicle. As another example, the simulation system can be used to train the planning system to select actions that enable the vehicle to operate safely, efficiently, and comfortably in an environment that is simultaneously occupied by other vehicles.

[0023] Previously, to test the performance of a planning system, a simulation could be generated in which the planning system would control one vehicle, while the remaining vehicles would simply replay the recorded behavior of the real-world vehicles. Such simulations can be unrealistic because the vehicles following a predetermined trajectory cannot adjust their behavior to account for the behavior of the vehicle controlled by the planning system. In contrast, the simulation system described herein can dynamically select the action to be performed by each vehicle under its control at each time step, while taking into account the behavior of every other vehicle, including the vehicle controlled by the planning system. As a result, the simulation system described herein can generate more realistic simulations that can be used to improve the operation of the planning system.

[0024] The simulation systems described in this specification can "sample" (i.e., randomly generate) a large number of possible simulations of an environment starting from given initial conditions of the environment. Sampling different possible simulations enables the simulation system to simulate many different scenarios, for example, where a vehicle controlled by the simulation system merges behind a vehicle controlled by a planning system, and where a vehicle controlled by the simulation system merges before a vehicle controlled by the planning system. In contrast, an alternative simulation system in which vehicles replay the recorded behavior of real-world vehicles would be unable to simulate multiple scenarios starting from a given initial condition. The ability to sample different possible simulation scenarios enables the simulation systems described in this specification to be used, for example, to test a planning system under a more comprehensive set of realistic scenarios than would be possible using some alternative simulation systems.

[0025] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Illustrated is an example environment interacted with by multiple agents.

[0027] Figure 2 An example of a simulation that has been unfolded over 4 time steps is shown.

[0028] Figure 3 is a block diagram of an example simulation system.

[0029] Figure 4 An illustration of a set of occupancy maps is provided.

[0030] Figure 5 is a block diagram of an example observation embedding system.

[0031] Figure 6 is a block diagram of example onboard systems of a vehicle.

[0032] Figure 7is a flow chart of an example process for generating a simulation of an environment for multiple agents to interact with.

[0033] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0034] This specification describes a simulation system that can learn to control a set of agents (e.g., vehicles) interacting with a simulated environment so that their behavior (e.g., acceleration and steering) simulates the behavior of real-world agents (e.g., a car controlled by a human driver). The simulation system controls the agents jointly so that each agent behaves realistically, both individually and relative to every other agent in the environment. Figure 1 Take, for example, the simulated multi-agent environment 100 shown in the figure, where five agents (102-AE) jointly navigate a road. The simulation system can cause agent 102-C to stop at an intersection and avoid entering the lane occupied by agent 102-E until agent 102-E (which is also controlled by the simulation system) has passed the intersection. When agent 102-C does enter the intersection, the simulation system can control its steering and acceleration to cause it to follow the true trajectory, for example, so that it stays on the road and does not collide with any other agents.

[0035] Throughout this document, an agent's behavior is said to be "realistic" if its behavior simulates the behavior of a real-world agent (i.e., has properties similar to the behavior of a real-world agent). For example, an agent may be considered to behave realistically based on whether it collides with other objects in the environment, how the agent handles traffic lights and signs (e.g., all-way stop precedence), and how the agent handles interactions with other agents (e.g., during lane changes and merging). Similarly, an environment (or agent) is said to be realistic if the state of the environment (or agent) simulates the state of the real-world environment (or agent). In other words, a simulation is said to be realistic if it specifies simulated states that have similar properties to real-world environment states derived from recorded data, such that an observer (e.g., a discriminator neural network or a human) may have difficulty distinguishing the simulated states from real-world environment states.

[0036] The simulation system controls the agents by selecting a corresponding action to be performed by each agent at each of a plurality of time steps. In particular, for each agent, the system may generate a probability distribution over a set of possible actions using a "policy" neural network, and then use this probability distribution to select an action to be performed by the agent. Actions that can be performed by the agent may include actions that control the movement of the agent, such as steering and acceleration actions, as well as actions that affect the state of the agent in other ways (e.g., by activating or deactivating a turn signal). The actions performed by the agents at a given time step determine the simulation state (particularly the corresponding state of each agent) at the next time step. As used throughout this document, the "simulation state" at a time step refers to data that characterizes the state of each agent at that time step.

[0037] Starting from a given initial simulation state at a first time step, the simulation system generates a set of multiple "candidate" next simulation states for the next time step. Each candidate next simulation state is the result of the agent performing a corresponding action, which is sampled from the probability distribution of each agent action generated by the policy neural network at the first time step. The simulation system evaluates whether each candidate next simulation state is real by using a "discriminator" neural network to generate a corresponding "discrimination score" for the candidate next simulation state. The simulation system can use the discrimination score to filter (i.e., remove from further consideration) one or more candidate next simulation states, for example, by maintaining only the candidate next simulation state with the highest discrimination score, or the candidate next simulation state with a discrimination score that meets a threshold.

[0038] The simulation system "unfolds" the simulation over multiple time steps by generating, at each time step, a candidate next simulation state for the next time step based on the simulation state(s) at the current time step, and filtering out unrealistic candidate next simulation states based on the discrimination scores. Maintaining multiple candidate next simulation states (i.e., each of which is identified as real) enables the simulation system to unfold multiple simulations in parallel.

[0039] Figure 2 An example of a simulation 200 is shown that has been unfolded over four time steps (T=1, 2, 3, 4) starting from an initial time step (T=0). Each circle 202-AI represents a corresponding simulation state generated by the simulation system. At T=0, the simulation system generates two candidate next simulation states 202-B and 202-C from the initial simulation system 202-A. The simulation system generates discrimination scores for the candidate simulation states 202-B and 202-C and, in this example, filters the candidate simulation state 202-B. Figure 2, unfilled circles (e.g., 202-B) represent simulation states that have been filtered, while filled circles (e.g., 202-C) represent simulation states that are already maintained by the simulation system. The simulation system proceeds to generate candidate simulation states 202-D and 202-E from candidate simulation state 202-C, filter candidate simulation state 202-D, generate candidate simulation states 202-F and 202-G, filter candidate simulation state 202-G, and generate candidate simulation states 202-H and 202-1 (neither of which are filtered). In this example, the simulation system generates two possible simulations for the agent in the environment, corresponding to the sequences of simulation states: [A, C, D, F, H] and [A, C, D, F, I].

[0040] The simulation systems described herein can be used for any of a variety of purposes. In one example, the simulation system can be used to evaluate whether a vehicle's planning system (i.e., a planning system configured to make driving decisions that control the vehicle's operation) is able to safely and effectively control the vehicle, for example, during lane changes and merging maneuvers. In another example, the vehicle's planning system can use the simulation system during operation of the vehicle, for example, to predict the behavior of other agents in the vehicle's vicinity.

[0041] These and other features are described in more detail below.

[0042] Figure 3 is a block diagram of an example simulation system 300. Simulation system 300 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0043] The simulation system 300 is configured to generate one or more simulations of an environment (e.g., a road) that is being interacted with by one or more agents (e.g., vehicles) over multiple time steps. Each simulation of the environment specifies a sequence of simulation states, where each simulation state corresponds to a corresponding time step, and the simulation state for a time step represents the state of each agent at that time step. The state of an agent at a time step can specify, for example, the position and motion of the agent at that time step. The position of the agent can specify the spatial position of the agent in the environment (e.g., represented by a set of xy coordinates) and the heading (direction of travel) of the agent (e.g., represented by an angle in the range [0,2π] radians). The motion of the agent can specify the speed of the agent (e.g., in miles per hour) and the acceleration of the agent (e.g., in feet per square second). The state of the agent at a time step can also represent other aspects of the agent, such as the state of the agent's turn signal at that time step. Each simulation may specify further information characterizing the environment with which the agent is interacting, such as road map data indicating the locations of various static features of the environment (e.g., lanes), and data indicating the states of objects in the environment (e.g., traffic lights) at different points in time.

[0044] At each of the plurality of time steps, the simulation system 300 processes one or more current simulation states 302 of the current time step to generate one or more next simulation states 304 of the next time step, wherein each next simulation state extends the current simulation state. A next simulation state is said to "extend" the current simulation state if it results from each agent performing the corresponding action in the current simulation state. Figure 2 In the example illustrated in FIG, simulation states 202-B and 202-C extend simulation state 202-A. After the last time step, simulation system 300 identifies one or more simulations of the environment from the generated simulation states. Each simulation specifies a sequence of generated simulation states with the property that each simulation state in the sequence extends the previous simulation state in the sequence.

[0045] The simulation system 300 may use: the observation embedding system 500, the policy neural network 306, the motion model 308, the discriminator neural network 310, and the filtering engine 312 to process the current simulation state 302 at the current time step to generate one or more next simulation states 304 at the next time step, as will be described in more detail below.

[0046] The observation embedding system 500 generates a corresponding current observation embedding 314 for each agent. The current observation embedding 314 of an agent is an ordered set of values ​​(e.g., a vector or matrix of values) that represents the current state of the agent and the current state of the environment. For example, the current observation embedding of a given agent can represent the current state of the environment (including the current states of other agents) from the perspective of the given agent, as shown in the following example: Figure 5 The observation embedding system 500 may generate observation embeddings 314 of agents with fixed (predetermined) dimensions, for example, the observation embedding 314 of each agent may be a 256-dimensional vector.

[0047] The observation embedding system 500 can generate a current observation embedding 314 of the agent by processing the current simulation state 302 and a set of environment data 316 representing the current state of the environment. The environment data 316 may include, for example, a road map indicating the structure of the road the agent is traveling on (e.g., the location and type of lanes, road signs, crosswalks, etc.), the state of objects in the environment (e.g., traffic lights), the trajectory of the agent up to the current point in time, and the predicted future trajectory of the agent (as described in more detail below). Figure 5 The example observation embedding system 500 is described in more detail.

[0048] For each agent, the policy neural network 306 processes the agent's observation embedding 314 to generate a probability distribution 318 over a set of possible actions that can be performed by the agent. The set of possible actions can include actions that control the agent's motion and actions that otherwise affect the agent's state. In one example, the set of possible actions can be given by:

[0049] A={(A i , S j ): i=1,...,N, j=1,...,M} (1)

[0050] Each of A i represents the action of adjusting the acceleration of the agent, and each S j represents the action of adjusting the steering angle of the agent. The policy neural network can have any suitable neural network architecture that enables it to perform the function it describes. In one example, the policy neural network can include an input layer, followed by a sequence of multiple fully connected layers, and then a soft-max output layer, where each neuron of the soft-max output layer corresponds to a corresponding action in the set of possible actions.

[0051] The simulation system 300 uses the action probability distribution 318 to generate one or more candidate next simulation states 320 that extend the current simulation state 302. To generate the candidate next simulation states 320, the simulation system 300 samples the corresponding actions of each agent from the corresponding action probability distribution 318 of the agent. The simulation system 300 then uses the motion model 308 to calculate the state of the agent at the next time step if the agent performs the corresponding sampled action at the current time step. The calculated next agent states collectively specify the candidate next simulation state 320. The motion model 308, which may also be called a "dynamic model", "transition model" or "physics model", can use appropriate kinematic and geometric relationships to calculate the next state of the agent based on the current state of the agent and the actions performed by the agent.

[0052] The simulation system 300 generates a corresponding discrimination score 322 for each candidate next simulation state 320 using the discriminator neural network 310. The discrimination score 322 for the candidate next simulation state 320 represents the likelihood that the candidate next simulation state 320 is real. The discrimination score 322 for the candidate next simulation state can be represented as a numerical value, for example, a numerical value within the range [0, 1], for example, where a higher value of the discrimination score (i.e., closer to one) indicates that the candidate next simulation state is more real.

[0053] In general, the simulation system 300 can generate a discrimination score 322 for the candidate next simulation state 320 by processing the representation of the candidate next simulation state 320 using the discriminator neural network 310. In one example, the simulation system 300 can generate a representation of the candidate next simulation state by generating a corresponding observation embedding for each agent corresponding to the candidate next simulation state 320 using the observation embedding system 500. In this example, the agent observation embeddings corresponding to the candidate next simulation state 320 collectively specify the representation of the candidate next simulation state. The simulation system 300 can process each agent observation embedding using the discriminator neural network 310 to generate a corresponding agent-specific discrimination score for each agent that characterizes the likelihood that the agent's state is true. The simulation system 300 can then combine the agent-specific discrimination scores to generate an overall discrimination score for the candidate next simulation state. For example, the simulation system can generate a discrimination score D for the candidate next simulation state as:

[0054]

[0055] D i =f θ (O i ) (3)

[0056] Where N is the number of agents, D i is the agent-specific discrimination score of agent i, Oi is the observation embedding of agent i corresponding to the candidate next simulation state, and f θ (·) represents the operation performed by the discriminator neural network. Determining the discrimination score of the simulation state as the minimum of the agent-specific discrimination scores (i.e., as shown in Equation (2)) reflects the intuition that unrealistic behavior of any agent will lead to unrealistic behavior of the entire simulation state.

[0057] Optionally, the discriminator neural network 310 can process additional inputs in addition to the representation of the candidate next simulation state 320. For example, the discriminator can process the representation of the current simulation state 302 (and possibly the representation of one or more previous simulation states) in addition to the representation of the candidate next simulation state 320. Processing these additional inputs can enable the discriminator neural network to more accurately characterize whether the candidate next simulation state is real.

[0058] In general, the discriminator neural network 310 can have any suitable neural network architecture that enables it to perform the functions described herein. For example, the discriminator neural network architecture can include a sequence of fully connected layers followed by an output layer having a single neuron, i.e., where the activation of the output layer neuron specifies the discrimination score. The output layer can use, for example, a sigmoid activation function to force the discrimination score to fall within the range [0, 1].

[0059] The simulation system 300 uses a filtering engine 312 to select one or more next simulation states 304 from the candidate next simulation states 304 based on the discrimination scores 322 of the candidate next simulation states 304. In general, if the discrimination score 322 of a candidate next simulation state indicates that it is not a real state, that is, a state that is unlikely to occur in a real-world environment, the filtering engine 312 is more likely to filter (i.e., remove from further consideration) the candidate next simulation state 320. In one example, the filtering engine 312 can filter each candidate next simulation state 320 that has a discrimination score 322 that does not meet a predetermined threshold. In another example, the filtering engine 312 can filter each candidate next simulation state 320 except for a predetermined number of candidate next simulation states 304 with the highest discrimination scores.

[0060] In some cases, particularly during the training of simulation system 300 (i.e., when simulation system 300 has not yet learned to effectively select actions to be performed by the agent), many candidate next simulation states may meet the criteria for being filtered by filtering engine 312. In order to continue unfolding the simulation at the next time step, simulation system 300 must select at least one next simulation state for the next time step. Thus, simulation system 300 may continue to generate candidate next simulation states 304 (i.e., candidate next simulation states 304 that extend current simulation state 302) until filtering engine 312 has selected at least a predefined number of candidate next simulation states 304 as acceptable next simulation states for the next time step.

[0061] After identifying the simulation state 304 for the next time step, the simulation system 300 can proceed to the next time step. If the current time step is the last time step, the simulation system 300 can identify one or more simulations of the environment from the simulation states generated for each time step. If the simulation has already progressed for a predefined maximum number of time steps, the simulation system 300 can determine that the current time step is the last time step.

[0062] Typically, to select actions that cause the agent to behave realistically, simulation system 300 (particularly policy neural network 306) can take into account the agent's predicted future behavior. Simulation system 300 can take into account the agent's predicted future behavior in a purely implicit manner (e.g., implicitly learned and encoded in the parameter values ​​of policy neural network 306). However, to improve the performance of simulation system 300 and to facilitate the training of policy neural network 306, simulation system 300 can also select actions to be performed by the agent based on explicit predictions of the agent's future behavior. Explicit predictions of the agent's future behavior can be provided by one or more previously generated simulations of the environment. For example, simulation system 300 can iteratively generate simulations of the environment, where, at each iteration, simulation system 300 generates a simulation of the environment that takes into account one or more simulations generated at the previous iteration. That is, to generate a simulation at a given iteration, the simulation system can observe the predicted agent behavior represented by the simulations generated at the previous iteration.

[0063] The simulation system 300 can use simulations generated in previous iterations in any of a variety of ways when generating simulations for the current iteration. For example, for each time point, the simulation system 300 can generate a corresponding "occupancy map" that indicates whether each spatial location in the environment was occupied by an agent at that time point in any simulation of the previous iteration. More specifically, the occupancy map at a time point can be represented as a two-dimensional (2D) array of values, where each component of the array corresponds to a corresponding spatial location in the environment. The value of each component of the array indicates whether the corresponding spatial location in the environment was occupied by an agent at that time point in any simulation of the previous iteration. In each iteration, the simulation system 300 can provide the occupancy map corresponding to the previous iteration as additional input to the observation embedding system 500 as part of generating the current observation embedding 314. In the first iteration (for which there is no previous iteration), the simulation system 300 can use a default occupancy map, i.e., one with predefined default values. Figure 4 An illustration of a set of occupancy maps 400 for time points T=1, 2, 3 is provided, where components of the occupancy maps are shaded to indicate that corresponding spatial locations in the environment were occupied by an agent in one or more simulations of a previous iteration.

[0064] The simulation system 300 can jointly train the policy neural network 306 and the discriminator neural network 310 based on recorded environmental data that represents the behavior of a real-world agent interacting with a real-world environment. Portions of the recorded environmental data can be derived from sensor data generated by sensors of a vehicle navigating in the real-world environment (e.g., camera sensors, lidar sensors, radar sensors, etc.). In some cases, the recorded environmental data can be derived from sensor data of multiple vehicles navigating the same environment simultaneously. Using sensor data derived from sensors of multiple vehicles can enable the recorded sensor data to more fully and accurately represent the environment, for example, in a situation where a portion of the environment is obscured from the view of one of the vehicles but not from the view of another vehicle. The simulation system 300 processes the recorded environmental data to generate a representation of the "real-world" environmental state.

[0065] The recorded environmental data can be stored in, for example, a geographic location-specific snapshot database or a geographic location-specific trajectory database. The geographic location-specific snapshot database can group the log data for each specific geographic location in the real world so that the log data collected by different vehicles at the same geographic location at different times are in the same geographic location-specific grouping. The geographic location-specific snapshot database can be queried for a given geographic location and provide all log data generated in a given geographic location. The geographic location-specific trajectory database can similarly group the recorded trajectory data (i.e., the recorded trajectory data that characterizes the trajectory of the agent) so that the database can be queried for a given geographic location and provide all trajectories including the given geographic location. The geographic location-specific snapshot and trajectory databases are further described with reference to U.S. patent application No. 62 / 911,183, which is incorporated herein by reference. Storing the recorded data in a geographic location-specific snapshot database and a geographic location-specific trajectory database can facilitate the efficient retrieval of recorded environmental data for use in training policy neural networks and discriminator neural networks. For example, a geographic location-specific database can enable the efficient retrieval of recorded data generated by multiple vehicles in the same vicinity at the same time. As described above, this data can be combined to provide a fuller, more accurate representation of the environment, for example, in situations where a portion of the environment is occluded from the view of one vehicle but not from the view of another vehicle.

[0066] The simulation system 300 trains a discriminator neural network to classify whether a representation of an environmental state is: (i) a simulated state generated by the simulation system based on current values ​​of policy network parameters, or (ii) a real-world environmental state derived from recorded environmental data. For example, the simulation system 300 may train the discriminator neural network to generate a discrimination score value of "1" for a real-world environmental state derived from recorded environmental data, and to generate a discrimination score value of "0" for a simulated state generated by the simulation system.

[0067] Simulation system 300 trains policy neural network 306 to generate action probability distribution 318 that results in a simulation state that is classified as true by discriminator neural network 310, i.e., a state of the real-world environment. For example, simulation system 300 can use policy neural network 306 to generate one or more candidate next simulation states that extend the current simulation state, and use the discriminator neural network to process representations of the generated candidate next simulation states to generate corresponding discrimination scores. Simulation system 300 can then adjust the current values ​​of policy neural network parameters based on an objective function that depends on the discrimination scores.

[0068] The simulation system 300 can alternate between: (i) training the discriminator neural network using simulation states generated based on the latest values ​​of the policy network parameters, and (ii) training the policy neural network using discrimination scores generated based on the latest values ​​of the discriminator network parameters. Doing so encourages the policy neural network to gradually get better at selecting actions that lead to realistic simulation states, and encourages the discriminator neural network to continually adapt to recognize unrealistic aspects of simulation states generated using the policy neural network. Reference J. Ho, S. Ermon: "Generative adversarial imitation learning," 30 th An example method for jointly training a policy neural network and a discriminator neural network is described in the 30th Conference on Neural Information Processing Systems (NeurlPS), 2016. Another example method for jointly training a policy neural network and a discriminator neural network is described in J. Fu, K. Luo, S. Levine: “Learning robust reward with adversarial inverse enhancement learning,” International Conference on Learning Representations (ICLR), 2018.

[0069] In some cases, the set of possible actions that can be performed by a real-world agent is different from the set of possible actions that can be performed by a simulated agent (i.e., a simulated agent controlled by a simulation system). For example, a real-world agent can make steering angle adjustments drawn from a continuous range (e.g., the range [0, 2π] radians), while a simulated agent can be constrained to make steering angle adjustments drawn from a finite set (e.g., the range [0, 2π] radians is discretized into a finite number of bins). The difference between the corresponding sets of possible actions that can be performed by the real-world agent and the simulated agent can impair the training of a discriminator neural network. For example, a discriminator neural network can learn to distinguish between simulated and real-world environment states by implicitly identifying whether the actions performed by the agent are drawn from a continuous set or a finite set. To avoid this result, the simulation can pre-process the recorded environment data to map each action performed by the real-world agent to the closest action from the set of possible actions that can be performed by the simulated agent.

[0070] Figure 5is a block diagram of an example observation embedding system 500. The observation embedding system 500 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0071] The observation embedding system 500 is configured to process a set of environment data 502 representing an environment and a current simulation state 504 representing a current state of each of a plurality of agents interacting with the environment to generate a corresponding observation embedding 506 for each agent.

[0072] The observation embedding system 500 processes the environment data 502 using the embedding neural network 508 to generate an environment embedding 510, i.e., an alternative representation of the environment data 502. The environment data 502 may include a road map indicating the structure of the road the agent is traveling, the states of objects in the environment (e.g., traffic light states), the trajectory of the agent up to the current point in time, the current state of the agent, and the predicted future trajectory of the agent. The environment embedding 510 can be represented as a collection of point embeddings, where each point embedding is an ordered set of numerical values ​​corresponding to a corresponding spatial position in the environment. For example, the environment embedding 510 can be represented as an array of embeddings (e.g., a 256-dimensional embedding), where each embedding corresponds to a corresponding 2D or 3D spatial position in the environment (e.g., represented by (x, y) or (x, y, z) coordinates, respectively). The embedding neural network can have any suitable neural network architecture (e.g., a convolutional neural network architecture) that enables it to perform the functions it describes.

[0073] The embedded neural network 508 can be used with (refer to Figure 3 The policy neural network and the discriminator neural network of the simulated system (described above) are trained jointly, for example, by back-propagating gradients through the policy neural network and the discriminator neural network into the embedding neural network 508. The embedding neural network 508 can also be trained using an objective function that depends directly on the environment embedding 510, for example, the objective function measures how accurately the environment data 502 can be reconstructed by processing the environment embedding 510.

[0074] For each agent, the observation embedding system 500 uses a coordinate transformation engine 512 to transform the coordinates of the corresponding spatial position associated with each point embedding of the environment embedding 510 from a predefined "global" coordinate system to an "agent-centric" coordinate system. The agent-centric coordinate system is defined relative to the state of the agent. For example, the coordinate transformation engine 512 can apply a translation operation to the spatial coordinates of the point embeddings so that they are centered on the spatial position of the agent. As another example, the coordinate transformation engine 512 can apply a rotation operation to map the heading of the agent to a predefined direction in the agent-centric coordinate system. This set of point embeddings and their associated spatial positions in the agent-centric coordinate system is referred to as an "agent-centric embedding" 514. Generating the agent's observation embedding 506 based on the agent-centric embedding 514 rather than the environment embedding 510 facilitates the training of the policy neural network and the discriminator neural network of the simulation system.

[0075] The observation embedding system 500 generates a corresponding observation embedding 506 for each agent based on the agent-centric embedding 514 corresponding to the agent. For example, to generate the observation embedding 506 for an agent, the observation embedding system 500 can apply a spatial pooling operation to the corresponding agent-centric embedding 514 and then process the pooled agent-centric embedding using one or more neural network layers to generate the observation embedding 506.

[0076] In some cases, certain areas of the environment may not be visible to the agent. For example, an area of ​​the environment may be invisible to the agent as a result of being obscured by another agent or a static object in the environment (e.g., a safety barrier or a tree). Optionally, the observation embedding system 500 can prune point embeddings corresponding to areas of the environment that are not visible to the agent from the agent-centric embedding 514 of the agent, for example, by replacing the point embeddings with default (predefined) embeddings. For example, if the agent's line of sight to an area of ​​the environment is blocked (e.g., by another agent or by a static object in the environment), the observation embedding system 500 can determine that the area of ​​the environment is not visible to the agent. Generating the observation embedding 506 based only on data from areas of the environment that are visible to the agent can enable the simulation system's policy neural network to more realistically simulate the behavior of a real-world agent.

[0077] In some implementations, rather than using the embedding neural network 508 to generate the environment embedding 510 , the observation embedding system 500 can generate the environment embedding data 510 from the environment data 502 using a set of manually defined rules.

[0078] Figure 66 is a block diagram of an example onboard system 602 of a vehicle 600. Onboard system 602 is composed of hardware and software components, some or all of which are physically located on vehicle 600. Onboard system 602 uses planning system 604 to plan the future trajectory of the vehicle and select actions to be performed by the vehicle. The simulation system described in this specification can be used to train and test planning system 604, as described in more detail below.

[0079] Onboard system 602 includes a sensor system 606 that generates sensor data 608 that enables onboard system 602 to "see" the environment near vehicle 600. More specifically, sensor system 606 includes one or more sensors, some of which are configured to receive reflections of electromagnetic radiation from the environment near vehicle 600. For example, sensor system 606 may include one or more laser sensors (e.g., lidar laser sensors) that are configured to detect reflections of laser light. As another example, sensor system 606 may include one or more radar sensors that are configured to detect reflections of radio waves. As another example, sensor system 606 may include one or more camera sensors that are configured to detect reflections of visible light.

[0080] Sensor system 606 continuously (i.e., at each of a plurality of time points) captures sensor data that may indicate the direction, intensity, and distance traveled of reflected radiation. For example, a sensor in sensor system 606 may transmit one or more pulses of electromagnetic radiation in a particular direction and may measure the intensity of any reflections and the time at which the reflections were received.

[0081] Planning system 604 is configured to use sensor data 608 (in addition to other data, such as road map data) to generate planning decisions for planning a future trajectory of the vehicle, and in particular, to select actions to be performed by the vehicle. The actions 610 selected by planning system 604 can be provided to a control system 612 of vehicle 600. The control system 612 of the vehicle can control some or all operations of the vehicle by implementing the actions 610 selected by the planning system. For example, in response to planning system 604 selecting an action specifying that the speed of the vehicle should be reduced, control system 612 can send an electronic signal to a brake control unit of the vehicle. In response to receiving the electronic signal, the brake control unit can mechanically apply the brakes of the vehicle.

[0082] After being trained, simulation system 300 can be used to safely and efficiently test the effectiveness of planning system 604. For example, planning system 604 can be used to control an agent in a simulated environment, while one or more other agents in the simulated environment are controlled by simulation system 300 (using the policy neural network and discriminator neural network as described earlier). Potential faults in the planning system can be identified by running a large number of simulations and identifying where the planning system fails (e.g., where the planning system selects actions that cause the simulated agent to operate unsafely).

[0083] The planning system 604 can be trained jointly with the simulation system 300 using machine learning techniques. For example, the planning system 604 and the simulation system 300 can be trained jointly by conducting a large number of simulations in which the planning system 604 controls one agent in the simulated environment while the simulation system 300 controls the remaining agents (as described earlier). The planning system 604 can be trained to optimize an objective function that characterizes whether the vehicle controlled by the planning system 604 effectively achieves its goal (e.g., navigates from point A to point B) while operating safely. As described earlier, the simulation system 300 (particularly the policy neural network and the discriminator neural network) can be trained based on a set of recorded real-world data to realistically simulate the behavior of the other agents. Training the planning system 604 jointly with the simulation system 300 enables the planning system 604 to learn to more safely and effectively control the vehicle 600.

[0084] The simulation system 300 can also be implemented by the vehicle's onboard systems 602 and used by the vehicle's planning system 604 to perform predictions about the behavior of other agents in the vehicle's vicinity. For example, before selecting an action to be performed by the vehicle at a point in time, the planning system 604 can use the simulation system 300 to run a simulation that predicts the future behavior of other agents in the vehicle's vicinity over a duration (e.g., 5 seconds). When selecting an action to be performed by the vehicle, the planning system 604 can take the predicted future behavior of other agents into account.

[0085] Figure 7 is a flow chart of an example process 700 for generating a simulation of an environment for multiple agents to interact with. For convenience, process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a simulation system appropriately programmed according to the present specification, such as Figure 3 The simulation system 300 can perform process 700.

[0086] The system obtains one or more current simulation states for the current time step (702). If the current time step is the first time step, the system may obtain the current simulation state from a set of recorded environmental data that characterizes the state of the real-world environment. If the current time step is after the first time step, the system may obtain the simulation state generated for the current time step at the previous time step. Typically, the simulation state for a time step specifies the corresponding state of each agent at that time step. In one example, the agent is a vehicle, and the state of the agent at a time step specifies the position of the agent (e.g., spatial position and heading) and the motion of the agent (e.g., velocity and acceleration) at that time step.

[0087] The system generates a set of candidate next simulation states for the next time step based on the current simulation state (704). In order to generate candidate next simulation states that extend the current simulation state, the system generates a corresponding probability distribution over a set of possible actions for each agent. The system can generate a probability distribution of actions for the agents by processing a representation of the current state of the agents in the environment using a policy neural network (sometimes called an "observation embedding"). The system then samples the corresponding actions of each agent using the probability distribution corresponding to the agent, and uses a motion model to determine the next state of the agent at the next time step if the agent performs the sampled action at the current time step. The set of possible actions may include, for example, an action to adjust the steering angle of the agent and an action to adjust the acceleration of the agent.

[0088] The system generates a corresponding discrimination score for each candidate next simulation state (706). The discrimination score of the candidate next simulation state represents the likelihood that the candidate next simulation state is a true simulation state. The system can generate the discrimination score for the candidate next simulation state by processing the representation of the candidate next simulation state using a discriminator neural network. The discriminator neural network is trained to generate a discrimination score that represents the environment state as true if the environment state is a real-world environment state. The system can generate the representation of the candidate next simulation state by obtaining, for each agent, a representation of the next state of the agent at the next time step corresponding to the candidate next simulation state. To generate the discrimination scores for the candidate next simulation states, the system can generate an agent-specific discrimination score for each agent by processing the representation of the next state of the agent corresponding to the candidate next simulation state using the discriminator neural network. The agent-specific discrimination score of the agent represents the likelihood that the next stage of the agent is a true agent state. The system can then generate an overall discrimination score for the candidate next simulation states based on the agent-specific discrimination scores of the candidate next simulation states, for example, based on the minimum value of the agent-specific discrimination scores.

[0089] The system selects one or more of the candidate next simulation states as the simulation state for the next time step based on the discrimination scores (708). For example, the system can select a predefined number of candidate next simulation states with the highest discrimination scores as the simulation state for the next time step. If the current time step is not the last time step, the system can advance to the next time step and return to step 702. If the current time step is the last time step, the system can output one or more simulations of the environment based on the simulation states generated for each time step.

[0090] This specification uses the term "configured" in connection with systems and computer program components. For a system of one or more computers to be configured to perform a particular operation or action means that the system has software, firmware, hardware, or a combination thereof installed thereon that, when operated, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0091] The embodiments and functional operations of the subject matter described in this specification may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more thereof. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated as encoded information to be transmitted to a suitable receiver device for execution by a data processing device.

[0092] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0093] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and the computer program may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to it, or in multiple collaborating files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one or more computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0094] Throughout this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same computer or computers.

[0095] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, such as an FPGA or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0096] The computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a dedicated logic circuit or can be incorporated therein. In general, a computer will also include one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or be operably coupled to one or more large-capacity storage devices to receive data from it or to transfer data thereto or to receive and transmit both. However, a computer does not necessarily have such a device. In addition, a computer can be embedded in another device (e.g., to give only a few examples, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive)).

[0097] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0098] To provide for user interaction, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. Furthermore, a computer can interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving a response message from the user in return.

[0099] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing the general and computationally intensive parts of the machine learning training or generation (i.e., inference workloads).

[0100] The machine learning model can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).

[0101] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0102] A computing system may include a client and a server. The client and the server are generally remote from each other and typically interact via a communication network. The relationship between the client and the server is formed by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device acting as a client (e.g., for the purpose of displaying data to a user interacting with the device and receiving user input therefrom). Data generated at the user device (e.g., the result of the user interaction) can be received from the device at the server.

[0103] Although this specification contains many specific implementation details, these details should not be interpreted as limiting the scope of any invention or the scope of possible claims, but should be interpreted as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in a single embodiment in combination. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described as working in certain combinations as described above and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may point to a sub-combination or a variant of the sub-combination.

[0104] Similarly, although operations are depicted in a particular order in the drawings and recited in a particular order in the claims, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments. Instead, it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0105] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for generating a simulation of an environment with which a plurality of agents interact over a plurality of time steps, performed by one or more data processing devices, wherein: The simulation includes a respective simulation state at each time step, the respective simulation state specifying a respective state of each agent at the time step, the method comprising, for each time step: Get the current simulation state for the current time step; Generating a plurality of candidate next simulation states for the next time step based on the current simulation state, wherein generating each candidate next simulation state comprises: For each agent, sampling a corresponding action from a set of possible actions that can be performed by the agent; and For each agent, if the agent performs the corresponding sampled action at the current time step, determine the corresponding next state of the agent at the next time step; For each candidate next simulation state, determining a discrimination score characterizing the likelihood that the candidate next simulation state is a true simulation state includes: For each agent, generating an agent-specific discrimination score that characterizes the likelihood that the next state of the agent is the true agent state; and generating a discrimination score representing the likelihood that the candidate next simulation state is the true simulation state by combining agent-specific discrimination scores; selecting the candidate next simulation state as the simulation state for the next time step based on the discrimination score of the candidate next simulation state; and A simulation of the environment is used to test the planning system of the autonomous vehicle.

2. The method according to claim 1, wherein The agents are vehicles in the environment.

3. The method according to claim 2, wherein: The set of possible actions that can be performed by the agent includes an action of adjusting a steering angle of the agent.

4. The method according to claim 2, wherein: The set of possible actions that can be performed by the agent includes an action to adjust the acceleration of the agent.

5. The method according to claim 1, wherein The state of an agent at a time step includes: (i) the position of the agent at the time step, and (ii) the motion of the agent at the time step.

6. The method according to claim 5, wherein: The position of the agent includes: (i) the spatial position of the agent in the environment, and (ii) the heading of the agent in the environment.

7. The method according to claim 5, wherein: The motion of the agent includes: (i) the velocity of the agent, and (ii) the acceleration of the agent.

8. The method of claim 1 , further comprising, for each agent: Obtain a representation of the agent's current state in the environment; and processing a representation of a current state of the agent in the environment using a policy neural network to generate a corresponding probability distribution over the set of possible actions of the agent; in, For each agent, sampling a corresponding action from the set of possible actions includes, for each agent: An action is sampled from the set of possible actions according to a probability distribution over the set of possible actions of the agent.

9. The method of claim 1, wherein: For each agent, if the agent performs the corresponding sampled action at the current time step, determining the corresponding next state of the agent at the next time step includes, for each agent: The motion model is used to process data representing: (i) the current state of the agent, and (ii) the sampled actions of the agent to generate the next state of the agent at the next time step.

10. The method of claim 1, wherein: For each agent, generating an agent-specific discrimination score that characterizes the likelihood that the next state of the agent is the true agent state includes: For each agent, obtaining a representation of the next state of the agent at the next time step corresponding to the candidate next simulation state; and For each agent, a representation of the agent's next state at the next time step corresponding to a candidate next simulated state is processed using a discriminator neural network to generate an agent-specific discrimination score that characterizes the likelihood that the agent's next state is the true agent state.

11. The method according to claim 10, wherein: Generating a discrimination score representing the likelihood that a candidate next simulation state is a true simulation state based on the agent-specific discrimination score includes: Compute the minimum of the agent-specific discrimination scores.

12. The method of claim 10, wherein: The discriminator neural network is trained to generate a discrimination score that characterizes the environment state as real if the environment state is a real-world environment state.

13. The method of claim 1, wherein: Selecting a candidate next simulation state as a simulation state for the next time step based on the discrimination score of the candidate next simulation state includes: The candidate next simulation state with the highest discrimination score is selected as the simulation state for the next time step.

14. A system comprising: one or more computers; as well as One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1-13.

15. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1-13.

Images