Generate spatial embeddings by integrating agent motion and optimizing predicted targets

Abstract

Methods, systems, and apparatus for training a spatial embedding neural network configured to process data representing the motion of an agent interacting with an environment to generate a spatial embedding, including a computer program encoded on a computer storage medium. In one aspect, the method comprises: processing data representing the motion of an agent in an environment at a current time step using the spatial embedding neural network to generate a current spatial embedding for the current time step; determining a predicted score and a target score for each of a plurality of slots in an external memory, wherein each slot stores: (i) a representation of an observation representing a state of the environment and (ii) a spatial embedding; and determining an update to the values ​​of a set of spatial embedding neural network parameters based on an error between the predicted score and the target score.

Description

Background Art

[0001] This specification deals with processing data using machine learning models.

[0002] A machine learning model receives an input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parameterized models and generate an output based on the received input and based on the values ​​of the model's parameters.

[0003] Some machine learning models are deep models that employ multiple layers of a model to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to a received input to generate an output. Summary of the Invention

[0004] This specification generally describes a system and method implemented as a computer program on one or more computers in one or more locations for training a spatial embedding neural network having a set of spatial embedding neural network parameters. The spatial embedding neural network is configured to process data representing the motion of an agent interacting with an environment to generate a spatial embedding.

[0005] As used throughout this specification, "embedding" refers to an ordered collection of values, eg, a vector or matrix of values.

[0006] In more detail, an example method includes, for each of a plurality of time steps, obtaining (e.g., inputting) and processing data representing an agent's motion in an environment at the current time step using a spatial embedding neural network (e.g., a recurrent neural network) to generate a current spatial embedding for the current time step. The method determines a predicted score and a target score for each of a plurality of slots in an external memory, wherein each slot stores: (i) a representation of an observation representing a state of the environment and (ii) a spatial embedding. The predicted score for each slot measures the similarity between (i) the current spatial embedding and (ii) the spatial embedding corresponding to the slot. The target score for each slot measures the similarity between (i) the current observation representing the state of the environment at the current time step and (ii) the observation corresponding to the slot. The method determines an update to the values ​​of a set of spatial embedding neural network parameters based on an error between the predicted score and the target score.

[0007] In various embodiments, the method further includes, for each of a plurality of time steps, processing a current observation and a current spatial embedding using an action selection neural network to generate an action selection output, and selecting an action to be performed by the agent in the current time step using the action selection output.

[0008] In some embodiments, the environment is a real-world environment, the agent is a machine agent that navigates the real-world environment, and the actions control the movement of the agent in the environment, i.e., the action selection system selects actions to enable the agent to perform tasks involving navigation in the environment.

[0009] In some embodiments, the environment is a real-world environment and the agent is a machine agent, or the environment is a simulated environment and the agent is implemented as one or more computer programs. The method may include using a trained spatial embedding neural network to enable the machine agent to navigate a new real-world environment. That is, the spatial embedding neural network can be trained in the real world or in a simulation, but the trained spatial embedding neural network can be used in the real world.

[0010] Navigating in a new real-world environment may include processing data representing the movement of a machine agent in the real-world environment using a trained spatial embedding neural network to generate a spatial embedding. An action selection system (particularly an action selection neural network of the action selection system) may be used to process the spatial embedding to select an action to be performed by the machine agent, thereby controlling the movement of the agent in the new real-world environment to navigate the new real-world environment.

[0011] According to one aspect, a method is provided for training a spatial embedding neural network having a set of spatial embedding neural network parameters, the method comprising: for each of a plurality of time steps: processing data representing the motion of an agent in an environment at a current time step using the spatial embedding neural network to generate a current spatial embedding for the current time step; determining a predicted score and a target score for each of a plurality of slots in an external memory, wherein each slot stores: (i) a representation of an observation representing a state of the environment and (ii) a spatial embedding, wherein the predicted score for each slot measures a similarity between (i) the current spatial embedding and (ii) the spatial embedding corresponding to the slot, wherein the target score for each slot measures a similarity between (i) the current observation representing the state of the environment at the current time step and (ii) the observation corresponding to the slot; and determining an update to the values ​​of the set of spatial embedding neural network parameters based on an error between the predicted score and the target score.

[0012] In some embodiments, the data representing the movement of the agent in the environment in the current time step includes one or more of: velocity data representing the velocity of the agent in the current time step, angular velocity data representing the angular velocity of the agent in the current time step, or translational velocity data representing the translational velocity of the agent in the current time step.

[0013] In some implementations, the current observation representing the state of the environment at the current time step includes an image.

[0014] In some implementations, the image is captured from the agent's perspective at the current time step.

[0015] In some embodiments, determining a target score for each slot in an external memory includes: obtaining corresponding embeddings of a current observation representing a current environmental state and an observation corresponding to the slot; and determining the target score based on a similarity measure between (i) the embedding of the current observation representing the current environmental state and (ii) the embedding of the observation corresponding to the slot.

[0016] In some implementations, obtaining an embedding for the current observation includes processing the current observation using an embedding neural network.

[0017] In some embodiments, the error between a predicted score and a target score comprises a cross-entropy error between the predicted score and the target score.

[0018] In some implementations, the method further includes determining an update to the spatial embedding stored in the external memory based on the error between the predicted score and the target score.

[0019] In some implementations, the spatial embedding neural network does not process the current observation to generate the current spatial embedding for the current time step.

[0020] In some implementations, the method further includes storing a representation of the current observation and the current spatial embedding in a slot in an external memory.

[0021] In some embodiments, the method further includes: processing second data representing the agent's motion in the environment at the current time step using a second spatial embedding neural network having a set of second spatial embedding neural network parameters to generate a second current spatial embedding for the current time step, wherein each slot in the external memory also stores the second spatial embedding, wherein, for each slot in the external memory, the predicted score for the slot further measures a similarity between (i) the second current spatial embedding and (ii) the second spatial embedding corresponding to the slot; and determining an update to the values ​​of the set of second spatial embedding neural network parameters based on an error between the predicted score and the target score.

[0022] In some implementations, data representing the motion of an agent processed by a spatial embedding neural network is a proper subset of second data representing the motion of an agent processed by a second spatial embedding neural network.

[0023] In some embodiments, for each slot in the external memory, determining a predicted score for the slot includes determining the product of (i) a similarity measure between a current spatial embedding and a spatial embedding corresponding to the slot and (ii) a similarity measure between a second current spatial embedding and a second spatial embedding corresponding to the slot.

[0024] In some embodiments, the method further comprises: for each of a plurality of time steps: processing a current observation and a current spatial embedding using an action selection neural network to generate an action selection output; and selecting an action to be performed by the agent in the current time step using the action selection output.

[0025] In some implementations, the action selection output includes a corresponding score for each action in a predetermined set of actions.

[0026] In some implementations, selecting an action to be performed by the agent in the current time step includes selecting the action with the highest score.

[0027] In some embodiments, the action selection neural network is trained using reinforcement learning techniques to encourage the agent to perform tasks in the environment.

[0028] In some implementations, the task is a navigation task.

[0029] In some implementations, for one or more of the plurality of time steps, the action selection neural network also processes a goal spatial embedding generated by the spatial embedding neural network at the time step when the agent was at the goal location in the environment.

[0030] In some embodiments, the spatial embedding neural network comprises a recurrent neural network, and wherein generating a current spatial embedding for a current time step comprises: processing, using the spatial embedding neural network, (i) data characterizing motion of the agent in the environment at the current time step and (ii) an output of the spatial embedding neural network at a previous time step to update a hidden state of the spatial embedding neural network, wherein the updated hidden state defines the current spatial embedding.

[0031] In some embodiments, the method further includes: determining an output of the spatial embedding neural network at the current time step, including: identifying an updated hidden state of the spatial embedding neural network as the output of the spatial embedding neural network at the current time step.

[0032] In some embodiments, the method further includes: determining an output of the spatial embedding neural network for the current time step, comprising: determining a corresponding weight value for each slot in an external memory, which represents (i) a current observation representing the state of the environment at the current time step and (ii) a similarity between the observation corresponding to the slot; determining a corrected embedding as a linear combination of the spatial embeddings corresponding to the slots in the external memory, wherein each spatial embedding is weighted by a corresponding weight value; and determining the output of the spatial embedding neural network based on (i) the updated hidden state of the spatial embedding neural network and (ii) the corrected embedding.

[0033] In some embodiments, the method further comprises: processing data representing the motion of the agent in the environment at the current time step using an integrated embedding neural network to generate a current integrated embedding for the current time step; determining a predicted score and a target score for each of a plurality of slots in an additional external memory, wherein each slot stores: (i) a spatial embedding and (ii) an integrated embedding, wherein the predicted score for each slot measures a similarity between (i) the current integrated embedding and (ii) the integrated embedding corresponding to the slot, wherein the target score for each slot measures a similarity between (i) the current spatial embedding and (ii) the spatial embedding corresponding to the slot; and determining an update to values ​​of a set of integrated embedding neural network parameters based on an error between the predicted score and the target score for the slot in the additional external memory.

[0034] According to another aspect, one or more (non-transitory) computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding methods described herein are provided.

[0035] According to another aspect, a system is provided, comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding methods described herein.

[0036] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0037] The system described in this specification can train a spatial embedding neural network that continuously (i.e., in multiple time steps) processes data representing the motion of an agent (e.g., angles and translational velocities) to generate a corresponding spatial embedding for each time step. Specifically, the system can train a spatial embedding neural network to process motion data to generate a spatial embedding that predicts an observation representing the state of the environment (e.g., an image of the environment captured by the agent's camera). The spatial embedding generated by the spatial embedding neural network can implicitly represent the agent's position in the environment.

[0038] An action selection system can process spatial embeddings generated by a spatial embedding neural network to select actions to solve a task (e.g., involving navigation in a complex, unfamiliar, and changing environment). Processing the spatial embeddings generated by the spatial embedding neural network can enable the action selection system to solve the task more efficiently (e.g., quickly) than it would otherwise because the spatial embeddings encode rich spatial information content and provide an efficient basis set for representing spatial information. For example, processing the spatial embeddings can enable the action selection system to select actions that enable the agent to navigate to a target location using a direct (or nearly direct) route, which can cover areas of the environment that the agent has not previously visited. Specifically, processing the spatial embeddings can enable the action selection system to exploit the rich spatial information encoded in the spatial embeddings to select actions that enable the agent to take shortcuts, resulting in the agent reaching a target location (or otherwise completing a task) more efficiently than some other systems.

[0039] Processing the spatial embeddings generated by the spatial embedding neural network can also enable the action selection system to be trained to an acceptable level of performance (i.e., task solving) in fewer training iterations than some other systems, thereby reducing the consumption of computational resources during training. Computational resources may include, for example, memory and computing power.

[0040] After training, the spatial embedding neural network can generate spatial embeddings that enable the agent to efficiently navigate in new (i.e., previously unseen) environments, without having to retrain the spatial embedding neural network on training data that characterizes the agent's interactions with the new environment.

[0041] The system described in this specification can jointly train multiple spatial embedding neural networks, each of which can be configured to process a different set of data representing the agent's motion at each time step. For example, one spatial embedding neural network can be configured to process data representing the agent's angular velocity at each time step, and another spatial embedding neural network can be configured to process data representing both the agent's angular velocity and the agent's translational velocity at each time step. The spatial embeddings generated by each spatial embedding neural network can have different properties and be complementary to each other, for example. One spatial embedding neural network may generate a spatial embedding that is substantially based on the agent's orientation, while another spatial embedding neural network may generate a spatial embedding that is substantially based on the agent's distance from other objects in the environment at a specific orientation. The set of spatial embeddings generated by the corresponding spatial embedding neural networks can jointly characterize the agent's positioning in various complementary ways. Processing the set of spatial embeddings generated by the spatial embedding neural networks can enable the action selection system to select actions that allow the agent to complete a task more efficiently.

[0042] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter can be seen from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a block diagram of an example action selection system.

[0044] Figure 2 Figure 1 is a block diagram of the example space embedding training system.

[0045] Figure 3 is a data flow illustrating the operations performed by the spatial embedding training system.

[0046] Figure 4 is a flowchart of an example process for training one or more spatial embedding neural networks.

[0047] Figure 5 is a flowchart of an example process for generating spatial embeddings of time steps using a spatial embedding neural network with a recurrent neural network architecture.

[0048] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0049] Figure 1 is a block diagram of an example action selection system 100. Action selection system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0050] The system 100 selects actions 110 to be performed by an agent 112 interacting with an environment 116 at each of a plurality of time steps to perform a task involving navigation in the environment. The task can be, for example, navigating in the environment to locate an object in the environment, navigating in the environment to reach a specified destination in the environment (referred to as a "goal location"), or navigating in the environment to visit as many locations in the environment as possible (e.g., exploring the environment).

[0051] In some embodiments, the environment is a real-world environment, and the agent is a machine agent that navigates the real-world environment. For example, the agent can be a robot or an autonomous or semi-autonomous land, sea, or air vehicle. In other embodiments, the environment is a simulated environment, and the agent is implemented as one or more computer programs that interact with the simulated environment. For example, the simulated environment can be a motion simulation environment (e.g., a driving simulation or a flight simulation), and the agent can be a simulated vehicle that navigates the motion simulation.

[0052] At each time step, system 100 receives motion data 114 representing the current motion of the agent in the environment at the time step and observations 120 representing the current state of the environment at the time step. System 100 processes the motion data 114 and observations 120 for the time step to select an action to be performed by agent 112 at the time step.

[0053] The motion data 114 in each time step may include one or more of the following: velocity data characterizing the agent's velocity in the time step, angular velocity data characterizing the agent's angular velocity in the time step, or translational velocity data characterizing the agent's translational velocity in the time step. The velocity data may be represented as one or more scalar values ​​(e.g., representing the agent's velocity in meters per second or any other suitable units), or as the sine and cosine of the angular velocity in radians per second, for example. The angular velocity data may be represented as, for example, a scalar value representing the rate at which the agent rotates about a vertical axis in radians per second or any other suitable units. The translational velocity data may be represented as a two-dimensional (2D) vector [u, v], for example, where u and v are expressed in meters per second or any other suitable units.

[0054] The observations 120 at each time step may be generated or derived from the agent's sensors at the time step. For example, the observations at the time step may include data representing the visual appearance or geometry of the environment from the agent's perspective at the time step (e.g., one or more images (e.g., color images) captured by the agent's camera sensor, one or more hyperspectral images captured by the agent's hyperspectral sensor, or images in the form of geometric data (e.g., 3D point clouds) captured by the agent's laser sensor (e.g., a lidar sensor), or a combination thereof). In the case of a simulated environment, the observations at each time step may be simulated observations representing the visual appearance or geometry of the simulated environment from the agent's perspective at the time step.

[0055] The actions 110 performed by the agent in each time step can control the agent's motion in the environment (e.g., by changing the agent's translational velocity, the agent's angular velocity, or both). These actions can be represented, for example, as control signals that control the agent. Each action can represent, for example, a corresponding torque that should be applied to the agent's joints, an acceleration action that changes the agent's acceleration, or a steering action that changes the agent's heading. In some embodiments, these actions can be multi-dimensional actions, for example, such that each action includes both a corresponding acceleration control signal and a corresponding steering control signal.

[0056] At each time step, the system 100 may receive a reward 118 based on the current state of the environment 116 at the time step and the action 110 performed by the agent 112. The reward 118 may be represented as a numerical value. The reward 118 may indicate whether the agent 112 has completed a task in the environment, or the agent 112's progress in completing a task in the environment. For example, if the task specifies that the agent should navigate to a goal location in the environment, then the reward at each time step may have a positive value once the agent reaches the goal location, and a zero value otherwise. As another example, if the task specifies that the agent should explore the environment, then the reward at the time step may have a positive value if the agent navigates to a location that was not previously explored in the time step, and a zero value otherwise.

[0057] The action selection system 100 includes one or more spatial embedding neural networks 102, spatial embedding training system 200, and action selection neural networks 106, each described in more detail below.

[0058] Each spatial embedding neural network 102 is configured to process, at each time step, a subset (e.g., a so-called proper subset) of the agent motion data 114 for that time step to generate a corresponding spatial embedding 104 for the time step. The spatial embedding 104 generated by the spatial embedding neural network 102 is an embedding that implicitly represents the position of the agent in the environment at that time step.

[0059] In embodiments where the system 100 includes a single spatial embedding neural network 102, the single spatial embedding neural network 102 can process all of the agent motion data 114 at each time step. In embodiments where the system 100 includes multiple spatial embedding neural networks 102, each spatial embedding neural network can process a different subset of the agent motion data 114 at each time step. For example, one spatial embedding neural network can be configured to process data representing the agent's angular velocity at each time step, while a second spatial embedding neural network can be configured to process data representing the agent's angular velocity and translational velocity at each time step. The spatial embeddings generated by each spatial embedding neural network can have different properties based on the agent motion data they receive and can be complementary to each other. For example, one spatial embedding neural network receiving, for example, angular velocity data might generate a spatial embedding that is substantially based on the agent's orientation, while another spatial embedding neural network receiving, for example, angular velocity and translational velocity data might generate a spatial embedding that is substantially based on the agent's distance from other objects in the environment at a particular orientation.

[0060] In embodiments, a spatial embedding neural network receiving agent motion data comprising only angular velocity may generate a spatial embedding that encodes the agent's orientation, e.g., where activations of neural network units generating the spatial embedding have an "activation bump" that encodes the orientation. The spatial embedding, and in particular, such an activation bump, may encode the orientation with respect to visual cues in the environment.

[0061] Each spatial embedding neural network 102 can have any suitable neural network architecture that enables it to perform its described function (i.e., process the agent motion data 114 to generate the corresponding spatial embedding 104). In particular, each spatial embedding neural network 102 can include any suitable number (e.g., 5 layers, 25 layers, or 125 layers) of any suitable type of neural network layers (e.g., recurrent layers, attention layers, fully connected layers, convolutional layers, etc.) connected in any suitable configuration (e.g., as a linear sequence of layers).

[0062] In some embodiments, each spatial embedding neural network 102 can be a recurrent neural network (e.g., a neural network having one or more recurrent neural network layers (e.g., long short-term memory (LSTM) layers or any other suitable recurrent neural network layers)) that maintains a corresponding hidden state. The hidden state of the spatial embedding neural network 102 at each time step can define the spatial embedding 104 generated by the spatial embedding neural network 102 at that time step. Each spatial embedding neural network can update its hidden state at each time step by processing (i) the agent motion data 114 at that time step and (ii) the data generated by the spatial embedding neural network 102 at the previous time step. Figure 5 An example process for processing a time step of agent motion data 114 using the spatial embedding neural network 102 implemented as a recurrent neural network will be described in more detail. (If the spatial embedding neural network is implemented as a recurrent neural network having multiple recurrent neural network layers that each maintain a corresponding hidden state, the hidden state of the spatial embedding neural network can be understood as a concatenation of the corresponding hidden states of one or more recurrent neural network layers.)

[0063] At each time step, the action selection neural network 106 receives inputs including (i) the current spatial embedding 104 generated by the spatial embedding neural network 102 at the time step and (ii) the current observation 120 representing the state of the environment at the time step. Optionally, the input received by the action selection neural network 106 may include additional data, such as a reward 118 received at a previous time step, a representation of an action 110 performed at a previous time step, or both. In some cases, the task performed by the agent involves repeatedly navigating to a "goal" location in the environment, and the input received by the action selection neural network 106 may include a "goal" spatial embedding. The goal spatial embedding may be the spatial embedding generated by the spatial embedding neural network 102 at the previous time step when the agent was at the goal location in the environment.

[0064] The action selection neural network 106 processes its input to generate an action selection output 108, and the system 100 selects an action 110 to be performed by the agent 112 in the time step based on the action selection output 108. For example, the action selection output 108 may include a corresponding action score for each action in a set of possible actions, and the system 100 may use the action scores to select the action 110 to be performed by the agent in the time step. In one example, the system 100 may select the action with the highest action score as the action to be performed in the time step. In another example, the system 100 may process the action scores (e.g., using a soft-max function) to determine a probability distribution over a set of possible actions, and then sample the actions to be performed in the time step according to the probability distribution.

[0065] In some embodiments, the system 100 can select an action to be performed at each time step according to an exploration strategy (e.g., a ∈ greedy exploration strategy). In the ∈ greedy exploration strategy, the system 100 randomly selects an action from a set of possible actions with probability ∈, and the system 100 selects an action for the time step using the action selection output 108 with probability 1-∈ (where ∈>0 is a small positive value). Selecting actions to be performed by the agent according to the exploration strategy can enable the agent to quickly explore the environment, thereby generating a higher diversity of training data that can facilitate more efficient training of the action selection neural network 106.

[0066] The action selection neural network 106 can have any suitable neural network architecture that enables it to perform its described functions (e.g., processing spatial embeddings and observations to generate an action selection output for selecting an action to be performed by the agent). For example, the action selection neural network architecture can include any suitable number (e.g., 5 layers, 25 layers, or 125 layers) of any suitable type of neural network layers (e.g., recurrent layers, attention layers, fully connected layers, convolutional layers, etc.) connected in any suitable configuration (e.g., as a linear sequence of layers).

[0067] The spatial embedding training system 200 is configured to train the spatial embedding neural network 102 to generate spatial embeddings that encode rich spatial information content and provide an efficient basis set for representing spatial information. Specifically, the training system 200 trains the spatial embedding neural network 102 to process data representing agent motion to generate spatial embeddings 104 that predict observations representing the state of the environment. More specifically, the training system 200 trains the spatial embedding neural network 102 to generate spatial embeddings that predict the visual or geometric appearance of the environment from the agent's perspective based on the agent's motion. Figure 2 An example of a spatial embedding training system 200 for training the spatial embedding neural network 102 will be described in more detail.

[0068] Processing the spatial embeddings 104 generated by the spatial embedding neural network 102 may enable the system 100 to select actions that efficiently solve complex navigation tasks (e.g., involving navigating in unfamiliar and changing environments). For example, processing the spatial embeddings may enable the system 100 to select actions that enable the agent to navigate to a target location using direct (or nearly direct) routes that may cover areas of the environment that the agent has not previously visited.

[0069] The spatial embedding training system 200 can train the spatial embedding neural network 102 based on trajectories representing the interaction of an agent with one or more environments. For each time step, each trajectory can include data representing the agent's motion in the time step and an observation of the state of the environment in the time step. After being trained based on trajectories representing the interaction of an agent with one or more environments, the spatial embedding neural network 102 can be used by the action selection system to control the agent interacting with a new environment without having to retrain on trajectories representing the interaction of the agent with the new environment. In other words, the trained parameter values ​​of the spatial embedding neural network 102 can be generalized to the new environment without having to retrain based on the interaction of the agent with the new environment.

[0070] The system 100 trains the action selection neural network 106 using reinforcement learning techniques to select actions that increase a cumulative measure of reward (e.g., a time-discounted sum of rewards) received by the system 100 as a result of the agent's interaction with the environment. More specifically, the system 100 trains the action selection neural network 106 by iteratively adjusting the values ​​of some or all parameters of the action selection neural network 106 using the gradient of the reinforcement learning objective function. The system 100 can train the action selection neural network 106 using any suitable reinforcement learning technique (e.g., an actor-critic technique or a Q-learning technique). Alternatively, the system 100 can train the action selection neural network 106 independently of the spatial embedding neural network 102, e.g., such that the gradient of the reinforcement learning objective function is not backpropagated into the spatial embedding neural network 102.

[0071] In general, the system 100 can be used to control an agent that interacts with a simulated environment or a real-world environment as described above. In some cases, the system 100 can be used to control an agent that interacts with a simulated environment, and the system 100 (particularly the spatial embedding neural network 102 and the action selection neural network 106) can be trained based on the agent's interaction with the simulated environment. The agent can then be deployed in a real-world environment, and the trained system 100 can be used to control the agent's interaction with the real-world environment. Training the system 100 based on the agent's interaction with the simulated environment (i.e., in place of the real-world environment) can avoid wear and tear on the agent and can reduce the likelihood that the agent may damage aspects of itself or its environment due to performing poorly selected actions.

[0072] Figure 2 is a block diagram of an example spatial embedding training system 200. The spatial embedding training system 200 is an example of a system implemented as a computer program on one or more computers in one or more locations that implements the systems, components, and techniques described below.

[0073] Training System 200 Training Reference Figure 1 One or more spatial embedding neural networks 102 are described (eg, included in an action selection system).

[0074] Each spatial embedding neural network 102 is configured to process corresponding agent motion data 202 representing the motion of the agent in the environment at a time step to generate a spatial embedding 208 implicitly representing the position of the agent in the environment at the time step.

[0075] The training system 200 includes an observation embedding neural network 206, an external memory 220, a scoring engine 212, and a training engine 218, each of which is described in more detail below.

[0076] The observation embedding neural network 206 is configured to process observations 204 representing environmental states to generate embeddings 210 for the observations 204. Prior to using the observation embedding neural network as part of training the spatial embedding neural network 102, the training system 200 can train the observation embedding neural network 206 to perform an autoencoding task on a training set of environmental observations. To perform the autoencoding task, the observation embedding neural network 206 processes the observations to generate corresponding observation embeddings that, when processed by a "decoder" neural network, are capable of reconstructing the original observations. Thus, in various embodiments, the spatial embedding neural network 102 is trained using the trained observation embedding neural network 206.

[0077] The observation embedding neural network can have any suitable neural network architecture that enables it to perform its described function (e.g., process observations to generate observation embeddings). For example, the observation embedding neural network architecture can include any suitable number (e.g., 5 layers, 25 layers, or 125 layers) of any suitable type of neural network layers (e.g., attention layers, fully connected layers, convolutional layers, etc.) connected in any suitable configuration (e.g., as a linear sequence of layers).

[0078] The external memory 220 includes a set of logical data storage spaces, referred to as "slots" 222. Each slot corresponds to a respective time step during the agent's interaction with the environment and stores: (i) the observation embedding 226 for that time step and (ii) the corresponding spatial embedding 224 for each spatial embedding neural network 102 corresponding to that time step.

[0079] The training system 200 generates corresponding observation embeddings 226 stored in each slot of the external memory 220 by processing the observations of the corresponding time steps using the observation embedding neural network 206.

[0080] The training system generates a corresponding spatial embedding 224 stored in each slot of the external memory by processing the agent motion data for the corresponding time step using the spatial embedding neural network 102. As will be described in more detail below, the training system 200 can modify the spatial embeddings stored in the slots of the external memory during the training process (e.g., using the gradient of the objective function).

[0081] At each time step in a sequence of time steps, training system 200 receives: (i) agent motion data 202 characterizing the agent's motion at the time step and (ii) observations 204 characterizing the state of the environment at the time step.

[0082] The training system 200 provides the time-step proxy motion data 202 to the spatial embedding neural networks 102 , and each spatial embedding neural network 102 processes the corresponding input based on the proxy motion data 202 to generate a corresponding current spatial embedding 208 .

[0083] The training system 200 provides the observation 204 for the time step to the observation embedding neural network 206, and the observation embedding neural network 206 processes the observation 204 to generate a current observation embedding 210.

[0084] The scoring engine 212 generates (i) a corresponding target score 216 and (ii) a corresponding predicted score 214 for each slot in the external memory 220 based on the current observation embedding 210 and the current spatial embedding 208 .

[0085] The target score 216 for each slot in the external memory 220 represents the similarity between (i) the current observation embedding 210 and (ii) the observation embedding in the slot stored in the external memory 220. The scoring engine 212 can generate the target score 216 for each slot based on a similarity measure (e.g., a cosine similarity measure, a Euclidean similarity measure, or any other suitable similarity measure) between the current observation embedding 210 and the observation embedding in the slot stored in the external memory. In one example, the scoring engine 212 can generate the target score T for each slot s. s Generates as:

[0086]

[0087] where β is a positive scalar parameter, is the transpose of the current observation embedding 210, is the observation embedding stored in slot s. The parameter β is the inverse-temperature parameter, which can be chosen for sparse selection of memory slots so that the interference between memories is low.

[0088] The predicted score 214 for each slot in the external memory 220 represents the similarity between (i) the current spatial embedding 208 and (ii) the spatial embedding in the slot stored in the external memory 220. To generate the predicted score 214 for the slot in the external memory 220, for each spatial embedding neural network 102, the scoring engine 212 may determine a corresponding similarity measure between (i) the current spatial embedding 208 generated by the spatial embedding neural network and (ii) the spatial embedding stored in the slot corresponding to the spatial embedding neural network. The similarity measure may be, for example, a cosine similarity measure, a Euclidean similarity measure, or any other suitable similarity measure. The scoring engine 212 may then determine the predicted score 214 for the slot by aggregating (e.g., by a product operation, a sum operation, or any other suitable operation) the determined similarity measures between the current spatial embedding 208 and the spatial embeddings in the slot stored in the external memory 220. For example, the scoring engine 212 may generate a predicted score P for each slot s.s Generates as:

[0089]

[0090] Where r indexes the spatial embedding neural network, R is the number of spatial embedding neural networks, is a scalar parameter that determines the relative importance of each spatial embedding in the neural network, is the transpose of the current spatial embedding generated by the spatial embedding neural network r, is the spatial embedding corresponding to the spatial embedding neural network r stored in slot s.

[0091] The training engine 218 receives the predicted score 214 and the target score 216 for the time step and updates the parameter values ​​of the spatial embedding neural network 102 to optimize an objective function that measures the error between the predicted score 214 and the target score 216. The objective function can be any suitable objective function that measures the error between the predicted score 214 and the target score, for example, a cross entropy objective function given by

[0092]

[0093] Where s indexes the slot of the external memory, S is the number of (occupied) slots in the external memory, and T s is the target score for slot s, and P s is the predicted rating of slot s.

[0094] To update the spatial embedding neural network parameters, the training engine 218 may determine the gradient of the objective function with respect to the spatial embedding neural network parameters (e.g., using backpropagation). The training engine 218 may then use the gradient to update the spatial embedding neural network parameters using any suitable gradient descent optimization technique (e.g., RMSprop or Adam).

[0095] Optionally, in addition to updating the spatial embedding neural network parameters, the training engine 218 may also use the gradient of the objective function to update various other system parameters. For example, the training engine 218 may additionally update the gradient described with reference to equation (2). Parameters, spatial embedding 224 stored in external memory 220, or both. The learning rate of spatial embedding 224 can be higher than the learning rate of spatial embedding neural network parameters, for example, the order is 10 -2 instead of 10 -4, because the interference between memories is small. As previously described, in a new environment, the spatial embedding neural network parameters can be frozen while storing the new spatial embedding 224 in the external memory 220 and optionally updating the new spatial embedding 224, and optionally retraining the action selection neural network using reinforcement learning.

[0096] Training the spatial embedding neural network to minimize the error between the predicted score and the target score encourages the spatial embedding neural network to generate a spatial embedding that predicts the observation of the environment. More specifically, training encourages the spatial embedding neural network to integrate the agent motion data to generate an embedding that predicts the visual or geometric appearance of the environment from the agent's perspective (i.e., as represented by the observation). Therefore, in various embodiments, the observation embedding 210 stored in the external memory is not a parameter updated by the training engine 218.

[0097] Optionally, the training system 200 can store the spatial embeddings 208 and observation embeddings 210 generated in the time step in slots in the external memory with some predefined probability. If one or more slots in the external memory are empty, the training system 200 can store the spatial embeddings 208 and observation embeddings 210 in the empty slots. Otherwise, the training system 200 can (possibly with a lower probability) store the spatial embeddings 208 and observation embeddings 210 in the external memory by overwriting the data currently stored in the slots. For example, the slots can be initialized to be empty, overwritten with a first probability (e.g., p=0.01) until the slots are filled, and then overwritten with a second lower probability (e.g., p=0.001).

[0098] In some embodiments, the training system 200 can train one or more additional neural networks (referred to as "ensemble embedding neural networks"), each of which is configured to process some or all of the agent motion data 202 at each time step to generate a corresponding embedding (referred to as the "ensemble embedding" for the time step). Optionally, each ensemble embedding neural network can process one or more additional inputs at each time step (i.e., inputs in addition to the agent motion data for the time step), such as spatial embeddings generated by one or more spatial embedding neural networks at the time step. Similar to the spatial embeddings generated by the spatial embedding neural network, the ensemble embeddings generated by the ensemble embedding neural network can be provided as input to the action selection neural network (i.e., the reference neural network). Figure 1 action selection system described in

[15] .

[0099] As part of training each integrated embedding neural network, the training system 200 can store the corresponding integrated embedding for the time step corresponding to the slot in each slot of the external memory 220 (or in a slot of an additional external memory). The training system generates the corresponding integrated embedding stored in each slot of the external memory by processing the agent motion data for the corresponding time step (and any other appropriate inputs) using the integrated embedding neural network. As will be described in more detail below, the training system can modify the integrated spatial embedding stored in the slot of the external memory during the training process (e.g., using the gradient of the objective function).

[0100] To train the integrated embedding neural network, at each time in a sequence of time steps, the training system determines an "integrated" prediction score and an "integrated" target score for each slot in the external memory. The training system generates an integrated prediction score for the slot based on the similarity between (i) the current integrated embedding generated by the integrated embedding neural network at the time step and (ii) the integrated embedding corresponding to the slot in the external memory. The training system can use any suitable similarity measure (e.g., a Euclidean similarity measure, a cosine similarity measure, or a similarity measure described with reference to Equation (1)) to generate the integrated prediction score.

[0101] The training system also generates an integrated target score for each slot that measures the similarity between (i) one or the current spatial embedding generated by the spatial embedding neural network at the time step and (ii) one or more spatial embeddings corresponding to the slot. For example, the training system can generate the integrated target score using any suitable similarity measure (e.g., a Euclidean similarity measure or a cosine similarity measure) to measure the similarity between (i) the concatenation of one or more current spatial embeddings at the time step and (ii) the concatenation of one or more spatial embeddings corresponding to the slot.

[0102] The training system can update parameter values ​​of the integrated embedding neural network and can optionally update the integrated embedding stored in the slot of the external memory to optimize an objective function that measures the error between the integrated prediction score and the integrated target score. The objective function can be, for example, a cross-entropy objective function (e.g., as described with reference to Equation (3)). The training system can update parameter values ​​of the integrated embedding neural network (e.g., by backpropagating the gradient of the objective function into the integrated neural network).

[0103] Figure 3 Show description reference Figure 2 A data flow 300 depicts the operations performed by the spatial embedding training system 200 in greater detail.

[0104] At each time step, the training system 200 processes the observation 204 of the time step using the observation embedding neural network 206 to generate the current observation embedding yt The training system 200 can then embed y based on (i) the current observation t and (ii) the observation embeddings stored in the corresponding slots of the external memory , determining (eg, as described above with reference to equation (1)) a corresponding target score 216 corresponding to each slot in the external memory.

[0105] At each time step, the training system 200 processes the corresponding agent motion data using each of the spatial embedding neural networks 102-1 to 102-3 to generate a corresponding current spatial embedding x 1,t ,x 2,t ,x 3,t Then, for each of the spatial embedding neural networks r∈{1,2,3}, the training system 200 can train the training system based on (i) the current spatial embedding x generated by the spatial embedding neural network r r,t and (ii) the spatial embedding corresponding to the spatial embedding neural network r stored in the corresponding slot of the external memory , a set of similarity measures (shown as 302 - 1 through 302 - 3 ) is determined by the training system 200 . The training system 200 then aggregates the similarity measurement sets 302 - 1 through 302 - 3 (e.g., via a multiplication operation) to determine a corresponding prediction score 214 corresponding to each slot in the external memory.

[0106] At each time step, the training system 200 determines the gradient of an objective function 304 (e.g., a cross-entropy objective function) that measures the error between the target score 216 and the predicted score 214, and uses the gradient to update the parameter values ​​of the spatial embedding neural network.

[0107] Figure 4 is a flow chart of an example process 400 for training one or more spatial embedding neural networks. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., Figure 2 The spatial embedding training system 200) can perform process 400.

[0108] In general, the steps of process 400 are performed for each time step in a sequence of time steps in which the agent interacts with the environment. For convenience, the following description of process 400 refers to the current time step in the sequence of time steps.

[0109] The system receives data representing the motion of an agent in an environment at a current time step and an observation representing the state of the environment at the current time step (402). The agent motion data may include one or more of: velocity data representing the velocity of the agent at the current time step, angular velocity data representing the angular velocity of the agent at the current time step, or translational velocity data representing the translational velocity of the agent at the current time step. The observation may include, for example, an image captured by the agent's camera depicting the visual appearance of the environment from the agent's perspective at the time step.

[0110] The system processes the observation using an observation embedding neural network to generate an observation embedding (404). The observation embedding neural network can be, for example, a convolutional neural network that is trained to perform an auto-encoding task (i.e., by generating an observation embedding that is capable of reconstructing the original observation when processed by a decoder neural network). In some cases, the observation embedding neural network is a dimensionality reduction neural network, i.e., one that causes the dimension of the observation embedding to be lower than the dimension of the observation itself.

[0111] The system processes a corresponding subset of the agent motion data using each spatial embedding neural network to generate a corresponding spatial embedding (406) using each spatial embedding neural network. Figure 5 An example process for generating spatial embeddings using a spatial embedding neural network is described in more detail.

[0112] The system determines a corresponding target score for each slot in the external memory based on the current observation embedding (408). Each slot in the external memory corresponds to a corresponding previous time step and stores: (i) an observation embedding (representation) that characterizes the state of the environment at the previous time step and (ii) a corresponding spatial embedding corresponding to the time step of each spatial embedding neural network. The system determines a target score for each slot in the external memory (e.g., as described above with reference to equation (1)) based on a similarity measure between (i) the current observation embedding and (ii) the observation embedding stored at the slot in the external memory.

[0113] The system determines a corresponding predicted score for each slot in the external memory based on the current spatial embedding (410). To generate the predicted score for the slot in the external memory, for each spatial embedding neural network, the system can determine a corresponding similarity measure between (i) the current spatial embedding generated by the spatial embedding neural network and (ii) the spatial embedding stored in the slot that corresponds to the spatial embedding neural network. The system can then determine the predicted score for the slot (e.g., as described above with reference to equation (2)) by aggregating the determined similarity measures between the current spatial embedding and the spatial embeddings in the slot stored in the external memory.

[0114] The system updates the parameter values ​​of each spatial embedding neural network parameter based on the error between the predicted score and the target score (412). For example, the system can determine the gradient of an objective function (e.g., a cross entropy objective function) that measures the error between the predicted score and the target score, and backpropagate the gradient of the objective function to the spatial embedding neural network parameters.

[0115] Figure 5 is a flow chart of an example process 500 for generating spatial embeddings of time steps using a spatial embedding neural network having a recurrent neural network architecture. For convenience, process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., Figure 2 The spatial embedding training system 200) can perform process 500.

[0116] The system receives network inputs to a spatial embedding neural network that include: (i) agent motion data representing the agent's motion in a time step and (ii) output generated by the spatial embedding neural network in a previous time step (502). The output generated by the spatial embedding neural network in the previous time step can be, for example, the spatial embedding of the previous time step, or a replacement output generated based in part on the observation in the previous time step. Generating a replacement output of the spatial embedding neural network based in part on the observation in the time step will be described in more detail with reference to steps 508 to 512.

[0117] The system processes the network input using the spatial embedding neural network to update the hidden state of the spatial embedding neural network (504). The updated hidden state of the spatial embedding neural network defines the spatial embedding of the time step. In one example, the updated hidden state h of the spatial embedding neural network t It can be given by:

[0118] h t =σ(Wh t-1 +Vx t +b) (4)

[0119] Where σ(·) is the element-wise sigmoid function, W, V, and b are weight matrices, and h t-1 is the hidden state of the spatial embedding neural network in the previous time step, and x t is the network input.

[0120] The system may provide an updated hidden state as an output of the spatial embedding neural network for the time step (506).

[0121] Alternatively, as an alternative to identifying an updated hidden state as an output of the spatial embedding neural network, the system can generate a replacement output based in part on the observation at the time step, as described with reference to steps 508 to 512. The system can generate a replacement output based in part on the observation at the time step at a randomly selected time step (e.g., with a probability of p=0.1). Generating a replacement output based in part on the observation at the time step can enable the spatial embedding neural network to correct accumulated errors and incorporate position and orientation information from the observation into the hidden state at the next time step.

[0122] To generate a replacement output for the time step based on the observation at the time step, the system determines a corresponding weight value for each slot in the external memory (508). The system can determine the corresponding weight value for each slot based on a similarity measure between (i) the current observation embedding and (ii) the observation embedding stored in the slot. For example, the system can determine the weight value w for slot s s Determined as:

[0123]

[0124] where γ is a positive scalar parameter that determines the entropy of the weight distribution, is the transpose of the current observation embedding, is the observation embedding stored in slot s of the external memory, s′ indexes the slot, and S is the total number of slots. In various embodiments, γ is one of the parameters optimized by the training engine 218.

[0125] The system determines a "correction" embedding (510) based on the weight value of the slot in the external memory. For example, the system may embed the correction Determined as:

[0126]

[0127] Where s indexes the slot in the external memory, S is the number of slots, and w s is the weight value of slot s, and is the spatial embedding corresponding to the spatial embedding neural network stored in slot s.

[0128] The system generates an output for the time step using the corrected embedding (512). For example, the system can use one or more neural network layers (e.g., recurrent layers) of a spatial embedding neural network to process: (i) an updated hidden state of the spatial embedding neural network and (ii) the corrected embedding to generate the output for the time step. The output for the time step can be an embedding having the same dimensions as the hidden state of the spatial embedding neural network. The output for the time step that depends on the observation for the time step can be provided as input to the spatial embedding neural network in the next time step and processed as part of updating the hidden state of the spatial embedding neural network in the next time step. By incorporating information from the observation (e.g., visual information from the observation including the image) into the hidden state of the spatial embedding neural network in the next time step, the spatial embedding neural network can correct errors in the spatial information represented by the hidden state as a result of integrating primarily motion information over a potentially long sequence of time steps.

[0129] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0130] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or additionally, program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to an appropriate receiver device for execution by the data processing apparatus.

[0131] The term "data processing apparatus" refers to data processing hardware and includes various types of devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or include dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0132] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; furthermore, a computer program may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0133] Throughout this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same computer or computers.

[0134] The processes and logic flows described in this specification can be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by dedicated logic circuitry (e.g., an FPGA or ASIC), or by a combination of dedicated logic circuitry and one or more programmed computers.

[0135] A computer suitable for executing a computer program can be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally speaking, the central processing unit will receive instructions and data from a read-only memory or random access memory, or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by, or incorporated into, dedicated logic circuits. Generally speaking, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from, transfer data to, or both mass storage devices. However, a computer does not require such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few.

[0136] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0137] To provide for interaction with a user, embodiments of the subject matter described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including voice, speech, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, a computer may interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and in turn receiving a response message from the user.

[0138] The data processing apparatus for implementing a machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive portions of machine learning training or production (i.e., inference) workloads.

[0139] Machine learning models can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0140] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer having a graphical user interface, a web browser, or an application (app) through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs") (e.g., the Internet).

[0141] A computing system may include a client and a server. The client and the server are generally remote from each other and typically interact via a communication network. The relationship between the client and the server arises from computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device acting as a client (e.g., to display data to a user interacting with the device and to receive user input from the user). Data generated at the device (e.g., the results of user interaction) may be received at the server from the user device.

[0142] Although this specification contains many specific implementation details, these should not be understood as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Specific features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any appropriate sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in a particular combination, or even initially claimed as such, in some cases one or more features in the combination may be removed from the claimed combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0143] Similarly, although the drawings depict operations in a particular order, and the claims recite them in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown, or in sequential order, or that all illustrated operations be performed to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments. It should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0144] Specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the specific order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method, performed by one or more data processing devices, for training a spatial embedding neural network having a set of spatial embedding neural network parameters, the spatial embedding neural network configured to process data representing motion of an agent interacting with an environment to generate a spatial embedding, the method comprising: For each of multiple time steps: processing data representing the motion of the agent in the environment at a current time step using a spatial embedding neural network to generate a current spatial embedding at the current time step that implicitly represents the position of the agent in the environment; determining a prediction score and a target score for each of a plurality of slots in an external memory, wherein each slot stores: (i) a representation of an observation representing a state of an environment and (ii) a spatial embedding, where the predicted score for each slot measures the similarity between (i) the current spatial embedding and (ii) the spatial embedding corresponding to the slot, where the target score for each slot measures the similarity between (i) the current observation representing the state of the environment at the current time step and (ii) the observation corresponding to the slot; and Determine an update to the values ​​of a set of spatial embedding neural network parameters based on the error between the predicted score and the target score, wherein the data representing the motion of the agent in the environment at the current time step comprises one or more of: velocity data representing the velocity of the agent at the current time step, angular velocity data representing the angular velocity of the agent at the current time step, or translational velocity data representing the translational velocity of the agent at the current time step, and The current observation representing the state of the environment in the current time step includes an image.

2. The method according to claim 1, wherein The image is captured from the perspective of the agent at the current time step.

3. The method according to any one of claims 1 to 2, wherein Determining the target score for each slot in external memory involves: Obtain the corresponding embeddings of the current observation representing the current state of the environment and the observation corresponding to the slot; and A target score is determined based on a similarity measure between (i) the embedding of the current observation representing the current state of the environment and (ii) the embedding of the observation corresponding to the slot.

4. The method according to claim 3, wherein: Obtaining an embedding for a current observation includes processing the current observation using an embedding neural network.

5. The method according to any one of claims 1 to 2, wherein The error between the predicted score and the target score includes the cross entropy error between the predicted score and the target score.

6. The method according to any one of claims 1 to 2, further comprising: Based on the error between the predicted score and the target score, an update to the spatial embedding stored in external memory is determined.

7. The method according to any one of claims 1 to 2, wherein The spatial embedding neural network does not process the current observation to generate the current spatial embedding for the current time step.

8. The method according to any one of claims 1 to 2, further comprising: Store the current observation and the current spatial embedded representation in slots in external memory.

9. The method according to any one of claims 1 to 2, further comprising: processing second data representing the motion of the agent in the environment at the current time step using a second spatial embedding neural network having a set of second spatial embedding neural network parameters to generate a second current spatial embedding for the current time step, Each slot in the external memory also stores a second spatial embedding, wherein, for each slot in the external memory, the prediction score for the slot also measures the similarity between (i) the second current spatial embedding and (ii) the second spatial embedding corresponding to the slot; and Based on the error between the predicted score and the target score, an update to the values ​​of a set of second spatial embedding neural network parameters is determined.

10. The method according to claim 9, wherein: The data representing the motion of the agent processed by the spatial embedding neural network is a proper subset of the second data representing the motion of the agent processed by the second spatial embedding neural network.

11. The method according to claim 9, wherein For each slot in the external memory, determining a prediction score for the slot includes determining a product of (i) a similarity measure between a current spatial embedding and a spatial embedding corresponding to the slot and (ii) a similarity measure between a second current spatial embedding and a second spatial embedding corresponding to the slot.

12. The method according to any one of claims 1 to 2, further comprising: For each of multiple time steps: Process the current observation and the current spatial embedding using an action selection neural network to generate an action selection output; as well as Use the action selection output to select the action to be performed by the agent in the current time step.

13. The method according to claim 12, wherein: The action selection output includes a corresponding score for each action in a predetermined set of actions.

14. The method according to claim 13, wherein Selecting the action to be performed by the agent in the current time step consists of selecting the action with the highest score.

15. The method according to claim 12, wherein: The action selection neural network is trained using reinforcement learning techniques to encourage the agent to perform tasks in the environment.

16. The method according to claim 15, wherein The task is a navigation task.

17. The method according to claim 12, wherein: For one or more of the plurality of time steps, the action selection neural network also processes a goal spatial embedding that was generated by the spatial embedding neural network at the time step when the agent was at the goal location in the environment.

18. The method according to any one of claims 1 to 2, wherein The spatial embedding neural network comprises a recurrent neural network, and wherein generating a current spatial embedding for a current time step comprises: Processing, using the spatial embedding neural network, (i) data representing the agent's motion in the environment at a current time step and (ii) the output of the spatial embedding neural network at a previous time step to update a hidden state of the spatial embedding neural network, wherein the updated hidden state defines a current spatial embedding.

19. The method according to claim 18, further comprising: Determine the output of the spatial embedding neural network at the current time step, including: Identify the updated hidden state of the spatial embedding neural network as the output of the spatial embedding neural network at the current time step.

20. The method of claim 18, further comprising: Determine the output of the spatial embedding neural network at the current time step, including: determining a corresponding weight value for each slot in the external memory, the weight value representing the similarity between (i) a current observation representing the state of the environment at the current time step and (ii) the observation corresponding to the slot; determining a rectified embedding as a linear combination of spatial embeddings corresponding to slots in the external memory, where each spatial embedding is weighted by a corresponding weight value; The output of the spatial embedding neural network is determined based on (i) the updated hidden state of the spatial embedding neural network and (ii) the corrected embedding.

21. The method according to any one of claims 1 to 2, further comprising: processing data representing the motion of the agent in the environment at the current time step using an integrated embedding neural network to generate a current integrated embedding for the current time step; determining a prediction score and a target score for each of a plurality of slots in an additional external memory, wherein each slot stores: (i) a spatial embedding and (ii) an integrated embedding, where the predicted score for each slot measures the similarity between (i) the current integrated embedding and (ii) the integrated embedding corresponding to the slot, where the target score for each slot measures the similarity between (i) the current spatial embedding and (ii) the spatial embedding corresponding to the slot; and An update to the values ​​of a set of integrated embedding neural network parameters is determined based on an error between a predicted score and a target score for a slot in an additional external memory.

22. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 21.

23. A system for updating neural network parameters, comprising: one or more computers; and One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 21.

Images