Performing navigation tasks using mesh code
Through the grid unit neural network and action selection neural network, the agent speed and environment state are processed, and the grid code is generated to solve navigation problems in navigation tasks, achieving efficient navigation and resource conservation in complex environments.
Patent Information
- Application Number
- CN202411132347.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-05-09
- Filing Date
- 2019-05-09
- Publication Date
- 2024-12-06
AI Technical Summary
The existing reinforcement learning system is difficult to effectively navigate to a designated target location in complex, unfamiliar and variable environments in navigation tasks, and the training process consumes a lot of computing resources.
The grid unit neural network and action selection neural network are used to process the speed data and environmental state of the agent, and generate grid code to implicitly represent the position of the agent. The reinforcement learning technology is used to train the action selection network and select the appropriate action to Implement navigation tasks.
It realizes faster and more efficient navigation tasks in complex environments, reduces the number of training iterations and computing resource consumption, and can choose more effective shortcuts to reach the target position.
Smart Images

Figure CN119089947A_ABST
Abstract
Description
[0001] This case is a divisional application of the invention patent application with the application date of May 9, 2019, application number 201980020693.1, and invention name “Using grid codes to perform navigation tasks”. Technical Field
[0002] This specification relates to reinforcement learning. Background Art
[0003] In a reinforcement learning system, an agent interacts with an environment by performing actions selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.
[0004] Some reinforcement learning systems select an action to be performed by the agent in response to receiving a given observation, based on the output of a neural network.
[0005] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values of the corresponding set of parameters. Summary of the invention
[0006] This specification describes an action selection system implemented as a computer program on one or more computers in one or more locations that can control an agent by selecting actions to be performed by the agent that enable the agent to solve a task involving navigation through an environment.
[0007] According to a first aspect, a system for selecting an action to be performed by an agent interacting with an environment is provided. The system includes one or more computers and one or more storage devices, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a grid cell neural network. The grid cell neural network may be configured to: receive an input including data representing the speed of the agent; process the input to generate a grid cell representation; and process the grid cell representation to generate an estimate of the position of the agent in the environment. The one or more computers may also implement an action selection neural network. The action selection neural network may be configured to: receive an input including a grid cell representation and an observation representing the state of the environment; and process the input to generate an action selection network output. The one or more computers may also implement a subsystem configured to: receive data representing the current speed of the agent; provide the data representing the current speed of the agent as input to the grid cell neural network to obtain a current grid cell representation; receive a current observation representing the current state of the environment; provide the current grid cell representation and the current observation as input to the action selection neural network to obtain an action selection network output; and use the action selection network output to select an action to be performed by the agent in response to the current observation.
[0008] Some advantages of such a system are described later. For example, where the actions are used to navigate, for example, towards a goal in, for example, a two-dimensional environment, such navigation may be faster, more efficient, and able to adapt to unfamiliar or changing environments. Thus, the agent may be a mechanical agent, such as a robot or vehicle that navigates in an environment, and the actions may control the movement of the agent.
[0009] In some implementations, the action space may be discrete but may give fine-grained motor control, for example, actions that may control rotation, movement (e.g., forward / backward / sideways), or affect rotational acceleration while moving.
[0010] In some implementations, the action selection network output includes a score distribution over actions in the set of possible actions. Selecting an action to be performed by the agent may then include sampling actions in the set of possible actions based on the score distribution over actions in the set of possible actions. The action selection neural network may include a recurrent neural network.
[0011] In some implementations, the grid cell neural network includes a recurrent neural network. Processing the input to generate the grid cell representation may include: processing the input by a recurrent neural network layer to generate a recurrent layer output, and processing the recurrent layer output by a linear neural network layer to generate the grid cell representation.
[0012] In some implementations, the linear neural network layer includes neurons having a spatial activity profile having a hexagonal spatial periodicity, and thus the grid cell representation may have a similar periodicity. In some implementations, the spatial scale of the hexagonal spatial periodicity has a multimodal distribution with a defined ratio between scale clusters and (adjacent) clusters, e.g., about 1.5. That is, there may be hexagonal spatial periodicity at different scales with a fixed ratio between continuous length scales. In some implementations, the linear neural network layer may include neurons having a spatial activity profile that defines a boundary location (of the environment) and / or a location (specific location) and / or heading of an agent. Thus, the grid cell representation may similarly have, for example, a hexagonal spatial periodicity at a different (e.g., fixed) ratio scale, and may also indicate one or both of: the location of the boundary, and one or more specific locations in the environment. In theory, decomposing spatial localization into multi-scale periodic codes can allow the relative position of two points to be determined from the difference in the codes at each scale level, combining the modulus remainders to determine the relative position vector.
[0013] In some implementations, the grid cell neural network can be configured to process the grid cell representation to generate an estimate of the location of the agent in the environment by generating a linear transformation of the grid cell representation and processing the linear transformation of the grid cell through a softmax layer to generate an estimate of the location of the agent in the environment. The estimate of the location of the agent in the environment can include the location of the agent and / or the heading of the agent. Therefore, in an implementation, generating a linear transformation and processing it using a softmax layer can be performed for each of the location and the heading.
[0014] In some implementations, the grid cell neural network is trained by supervised learning techniques. During training regularization, such as dropout, can be applied to the linear neural network layers.
[0015] The data characterizing the velocity of the agent may include data characterizing the translational velocity of the agent and / or data characterizing the angular velocity of the agent. The system may implement a visual neural network configured to receive an input including an observation characterizing a state of an environment and process the input to generate an estimate of the position of the agent in the environment. The grid cell neural network may then receive further input including an estimate of the position of the agent in the environment.
[0016] In some implementations, the subsystem may provide an estimate of the location of the agent in the environment generated by the visual neural network as an input to the grid unit neural network with a first probability. The visual neural network may include a convolutional neural network; which may be trained by supervised learning techniques. The subsystem may process the estimate of the location of the agent in the environment generated by the visual neural network through a masking layer with a second probability, which masking layer zeroes the estimate of the location of the agent in the environment to generate a masked estimate of the location of the agent, and provide the masked estimate of the location of the agent as an input to the grid unit neural network. Each probability may be unity or less than unity. In this way, the system may mimic the provision of imperfect observations of environmental cues.
[0017] In some implementations, the action selection network is trained by a reinforcement learning technique to perform a navigation task in the environment. In an implementation, the action selection network output may also include a predicted expected benefit, the predicted expected benefit being an estimate of the time discounted benefit due to the environment being in the current state, and the reinforcement learning technique may be an actor-evaluator reinforcement learning technique.
[0018] The current observation representing the current state of the environment provided as input to the action selection neural network may include an image. In an implementation, the input received by the action selection neural network may also include a target grid cell representation, i.e., a grid cell representation generated by the grid cell neural network at a (previous) time step when the position of the agent in the environment was the target position. The subsystem may also be configured to maintain data specifying the target grid cell representation and provide the target grid cell representation as input to the action selection neural network.
[0019] More specifically, a method for selecting an action to be performed by an agent interacting with an environment is provided, the method comprising: receiving a current observation representing a current state of the environment; receiving data representing a current velocity of the agent; processing the data representing the current velocity of the agent using an embedded neural network to generate a current grid cell representation; wherein the embedded neural network has been included as a subnetwork of the grid cell neural network when the grid cell neural network is trained to perform the following operations: processing an input representing the velocity of the agent using the embedded neural network to generate a grid cell representation as an intermediate output of the grid cell neural network; and processing the grid cell representation using one or more subsequent neural network layers of the grid cell neural network to generate an estimate of the position of the agent in the environment; processing using an action selection neural network: (i) the current grid cell representation generated by the embedded neural network, and (ii) the current observation to generate an action selection network output; and using the action selection network output, selecting an action to be performed by the agent in response to the current observation.
[0020] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.
[0021] The action selection system described in this specification can select actions for solving a navigation task by processing a "grid code" that implicitly characterizes the agent's position in the environment. The grid code refers to the intermediate output generated by a grid cell neural network that is trained to integrate data defining the agent's velocity over time to estimate the agent's current position, i.e., perform "dead reckoning."
[0022] The action selection systems described in this specification can solve navigation tasks (e.g., tasks involving navigating to a specified "target" location in a complex, unfamiliar, and changing environment) more efficiently (e.g., quickly) than some action selection systems that do not process grid codes. For example, processing grid codes can enable the action selection system to perform "vector navigation" by selecting actions that cause the agent to navigate to the target location using a direct (or nearly direct) route that may cover areas of the environment that the agent has not previously visited. In particular, processing grid codes can enable the action selection system described in this specification to select actions that cause the agent to take shortcuts that cause the agent to reach the target location more efficiently than certain other systems.
[0023] The action selection systems described in this specification can be trained to reach an acceptable level of performance (i.e., in solving a task) with fewer training iterations than some other systems. Thus, training the action selection systems described in this specification can consume fewer computing resources (e.g., memory and computing power) than training some other systems.
[0024] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 An example data flow for using an action selection system to select actions that enable an agent to perform a task involving navigation through an environment is shown.
[0026] Figure 2 An example action selection system is shown.
[0027] Figure 3 An example architecture of a mesh cell network is shown.
[0028] Figure 4A qualitative illustration of one example of the performance gain that can be achieved in a navigation task by providing a grid code rather than position data to the action selection network is provided.
[0029] Figure 5 A qualitative illustration of one example of the performance gain that can be achieved in a navigation task by providing a grid code rather than position data to the action selection network is provided.
[0030] Figure 6 is a flow chart of an example process for selecting an action to be performed by an agent at a time step.
[0031] Like reference numbers and numerals in the various drawings indicate like elements. DETAILED DESCRIPTION
[0032] This specification describes an action selection system that can be used to control an agent by repeatedly selecting actions to be performed by the agent and causing the agent to perform the selected actions. An agent controlled by the action selection system can effectively perform tasks involving navigation through an environment. The action selection system includes a grid cell neural network and an action selection neural network.
[0033] The grid cell network is configured to continuously process data defining the current velocity of the agent to generate an estimate of the current position of the agent in the environment. In generating the estimate of the current position of the agent, the grid cell network generates an intermediate output, referred to in this specification as a "grid code" or "grid cell representation", which implicitly represents the position of the agent in the environment.
[0034] The action selection network is configured to process the grid code generated by the grid cell network and observations representing a current state of the environment to generate an action selection output that can be used to select an action to be performed by the agent.
[0035] These and other features are described in more detail below.
[0036] Figure 1 An example data flow 100 is shown for selecting actions 104 using an action selection system 102 that enable an agent 106 to perform a task involving navigation through an environment 108. The task performed by the agent 106 may be, for example, locating an object in the environment 108, navigating through the environment to reach a specified destination (referred to as "goal localization"), or exploring the environment (e.g., navigating to as many locations as possible in the environment).
[0037] In some implementations, the environment 108 is a real-world environment and the agent is a mechanical agent that navigates through the real-world environment. For example, the agent can be a robot, or an autonomous or semi-autonomous land, sea, or air vehicle. In other implementations, the environment 108 is a simulated environment and the agent 106 is implemented as one or more computer programs that interact with the simulated environment.
[0038] At each time step in a plurality of time steps, the action selection system 102 processes data representing: (i) the current velocity 110 of the agent 106, and (ii) the current state of the environment to determine an action 104 to be performed by the agent at that time step. In this specification, the data representing the current state of the environment is referred to as an "observation" 112.
[0039] The data characterizing the current velocity 110 of the agent may include data defining the current translation and angular velocity of the agent. The translation velocity of the agent may be represented as a two-dimensional (2D) vector [u, v], for example, expressed in meters per second. The angular velocity of the agent characterizes the rate at which the "head direction" of the agent changes, i.e., the rate at which the agent rotates about its vertical axis. The angular velocity of the agent may be represented as a scalar value, for example, expressed in radians per second.
[0040] Observations 112 may be generated by or derived from sensors of agent 106. For example, observations 112 may be color images captured by a camera sensor of agent 106, hyperspectral images captured by a hyperspectral sensor of the agent, or geometric data (e.g., a 3D point cloud) captured by a laser sensor (e.g., a lidar sensor) of the agent. Observations 112 may include text or verbal instructions provided to the agent by a third party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and a user of the autonomous vehicle may provide text or verbal instructions to the agent (e.g., to navigate to a particular location).
[0041] Actions 104 performed by agent 106 may control the movement of agent 106 in environment 108, for example, by changing the translational velocity of the agent, the angular velocity of agent 106, or both. Actions 104 may, for example, be represented as torques that should be applied to the joints of the agent.
[0042] At each time step, the action selection system 102 may receive a reward 114 based on the current state of the environment 108 and the action 104 of the agent 106 at that time step. Typically, the reward 114 is a numerical value. The reward 114 may indicate whether the agent 106 has completed the task, or the progress of the agent 106 toward completing the task. For example, if the task specifies that the agent should navigate through the environment to a target location, then once the agent reaches the target location, the reward at each time step may have a positive value, otherwise it may have a zero value. As another example, if the task specifies that the agent should explore the environment, then if the agent navigates to a previously unexplored location at a time step, then the reward at that time step may have a positive value, otherwise it may have a zero value.
[0043] Reinforcement learning techniques are used to train the action selection system 102 to select actions that increase a cumulative measure of rewards received by the action selection system 102. More specifically, the action selection system 102 is trained by iteratively adjusting the values of some or all parameters of the action selection system 102 using the gradient of the reinforcement learning objective function 116. Training the action selection system 102 will be described in more detail below.
[0044] Figure 2 An example action selection system 102 is shown. Action selection system 102 is an example of a system implemented as a computer program on one or more computers in one or more locations that implement the systems, components, and techniques described below.
[0045] At each time step in the plurality of time steps, an action selection system 102 processes data characterizing the agent's current velocity 110 and observations 112 characterizing the current state of the environment to select an action 104 to be performed by the agent at that time step.
[0046] Action selection system 102 includes grid cell network 202 , action selection network 204 , and optionally vision network 206 .
[0047] Grid cell network 202 is configured to process velocity data 110 to generate an output representing an estimated position 208 of the agent in the environment.
[0048] Velocity data 110 may include data defining, for example, the agent's current translational velocity and angular velocity, as previously described.
[0049] The position data 208 generated by the grid cell network 202 may include: (i) positioning data characterizing the spatial location of the agent in the environment, and (ii) heading data characterizing the orientation of the agent (ie, about a vertical axis). Figure 3The format of location data 208 and an example architecture of grid cell network 202 are described in further detail.
[0050] As part of generating position data 208, grid cell network 202 generates current grid code 210 as an intermediate output. An intermediate output of a neural network refers to an output generated by one or more neurons from one or more intermediate layers (i.e., layers that are not input layers or output layers) of the neural network.
[0051] The grid code 210 may be represented as an ordered set of values, such as a vector or matrix of values, and implicitly represents the position of the agent in the environment.
[0052] The action selection network 204 is configured to process inputs including: (i) the current grid code 210, and (ii) the current observation 112 to generate an action selection output 212. The system 102 uses the action selection output 212 generated by the action selection network 204 to select an action 104 to be performed by the agent at the current time step. The following are some examples.
[0053] In one example, the action selection output 212 may include a corresponding numerical probability value for each action in a set of possible actions that may be performed by the agent. The system 102 may select an action to be performed by the agent, for example, by sampling the actions according to their probability values, or by selecting the action with the highest probability value.
[0054] In another example, the action selection output 212 may directly define the action to be performed by the agent, for example by defining the value of the torque that should be applied to the joints of the robotic agent.
[0055] In another example, the action selection output 212 may include a corresponding Q-value for each action in a set of possible actions that can be performed by the agent. The system 102 may process the Q-values (e.g., using a soft-max function) to generate a corresponding probability value for each possible action, which may be used to select an action to be performed by the agent (as described above). The system 102 may also select the action with the highest Q-value as the action to be performed by the agent.
[0056] The Q-value of an action is an estimate of the return that would result from the agent performing the action in response to the current observation 112 and subsequently selecting future actions performed by the agent 106 based on the current values of the action selection network parameters. Return refers to a cumulative measure of the rewards received by the agent, e.g., a time-discounted sum of rewards.
[0057] In some cases, the system 102 may select an action to be performed by the agent according to an exploration strategy. For example, the exploration strategy may be an ∈-greedy exploration strategy, in which the system 102 selects an action to be performed by the agent according to the action selection output 212 with probability 1-∈, and randomly selects an action to be performed by the agent with probability ∈. In this example, ∈ is a scalar value between 0 and 1.
[0058] In some cases, the system 102 may be used to select actions that enable an agent to perform a navigation task that involves, for example, repeatedly returning to a specified "target" location in the environment from different starting locations in the environment. To facilitate the selection of actions that enable the agent to return to a target location in the environment, the action selection network 204 may be configured to process a "target" grid code 214 in addition to the current grid code 210. The target grid code 214 refers to the grid code generated by the grid cell network 202 when the agent is at the target location.
[0059] At a time step before the agent first navigates to a target location in the environment, the system 102 may provide a “default” grid code (e.g., a grid code having all zero values) to the action selection network 204 in place of the target grid code 214. At the first time step at which the agent navigates to the target location, the system 102 may store the corresponding target grid code 214 generated by the grid cell network 202. At each subsequent time step, the system 102 may provide the stored target grid code 214 as input to the action selection network.
[0060] In one example, the agent may be a robotic agent configured to perform navigation tasks in a real-world environment, such as package delivery from a warehouse. The agent may initially explore the environment by navigating through the environment with the purpose of visiting as many locations in the environment as possible. At each location in a plurality of locations visited by the agent, the system 102 may store a grid code corresponding to the location. After the agent has explored the environment, the agent may be assigned the task of navigating to a specific target location in the environment (e.g., a location representing a package warehouse or a package should be delivered). In order to navigate to the target location in the environment, the agent may provide a target grid code corresponding to the target location in the environment (e.g., a target grid code previously stored by the agent during exploration) to an action selection network. Thereafter, the agent may perform a navigation action specified by the output of the action selection network that causes the agent to navigate to the target location (e.g., to perform package delivery).
[0061] The action selection network 204 may process various inputs in addition to the observations 112, the current grid code 210, and the target grid code 214. For example, the action selection network 204 may additionally process a reward received at a previous time step, and a representation of an action performed by the agent at a previous time step. The action performed by the agent at a previous time step may be represented in any of a variety of ways, for example, as a one-hot vector.
[0062] The action selection network 204 can have any of a variety of neural network architectures. For example, the action selection network 204 can include one or more convolutional neural network layers and recurrent neural network layers (e.g., long short-term memory (LSTM) layers). The convolutional layers can be configured to process observations to generate encoded representations of the observations (e.g., as an ordered set of numerical values, such as a vector or matrix of numerical values). The recurrent layers can be configured to process the encoded representations of the observations, the current grid code, the target grid code, the action performed at the previous time step, and the reward received at the previous time step to generate the action selection output 212.
[0063] Processing the grid code generated by the grid cell network 202 enables the action selection network 204 to select actions that efficiently solve the navigation task. In particular, processing the grid code enables the action selection network 204 to select actions that enable the agent to navigate to the target location using a direct (or approximately direct) route that covers areas of the environment that the agent has not previously visited. In contrast, if the action selection network 204 is configured to process other kinds of data that characterize the location of the agent in the environment, such as position data 208, the action selection network 204 may select actions that are less efficient (or not at all) in solving the navigation task. Figure 4 and Figure 5 An example of the performance gain that can be achieved in a navigation task by providing a grid code rather than, for example, position data 208 to the action selection network is shown.
[0064] When vision network 206 is included in system 102, it is configured to process observations 112 to generate an estimate of the position of the agent in the environment. In one example, observations 112 may include an image of the environment, and vision network 206 may be configured to generate an estimate of the position of the agent based on the visual appearance of the environment (i.e., as represented by the image). Similar to grid cell network 202, vision network 206 may generate an estimate of the position of the agent including: (i) positioning data characterizing the spatial location of the agent, and (ii) heading data characterizing the orientation of the agent. Figure 3Examples of positioning data and heading data that may be generated by vision network 206 and grid cell network 202 are described in more detail.
[0065] The system 102 may provide the output of the visual network 206 as an additional input to the grid cell network 202 at each time step. In some cases, at each time step, the system 102 may sample a Bernoulli random variable before providing the output of the visual network 206 to the grid cell network 202. If the random variable assumes a specified value (e.g., 0), the system 102 may process the output of the visual network 206 using a "masking" neural network layer 216 that zeroes the output of the visual network 206 before providing the output of the visual network 206 to the grid cell network 202. Using a masking layer to zero the output of the visual network 206 at certain time steps (e.g., with a certain probability) may regularize the training of the system. For example, using a masking layer may allow the system to generate a grid code based primarily on an integrated velocity signal (independent of the environment) rather than relying on visual cues (dependent on the environment). Relying on visual cues in the environment may cause the system to overfit the environment during training, thereby reducing the performance of the system in performing tasks (e.g., navigation tasks) in other environments.
[0066] The vision network 206 can have any of a variety of neural network architectures. For example, the vision network 206 can have an architecture that includes one or more convolutional neural network layers, followed by two output layers, each of which processes the output of the final convolutional layer. In this example, one of the output layers can generate positioning data, while the other output layer can generate heading data.
[0067] System 102 may train action selection network 204 , grid cell network 202 , and vision network 206 to determine trained values for their respective parameters.
[0068] The system 102 may use reinforcement learning training techniques to train the action selection network 204. More specifically, the system 102 may iteratively adjust the values of the action selection network parameters using the gradient of the reinforcement learning objective function with respect to the action selection system parameters to increase the cumulative measure of rewards received by the system 102. The system 102 may train the action selection network 204 using any suitable reinforcement learning technique, such as, for example, a Q-learning technique or an actor-critic technique.
[0069] In one implementation, the system 102 uses an actor-critic technique to train the action selection network 204. In this implementation, the action selection network 204 is configured to generate a value estimate in addition to the action selection output 212. The value estimate represents an estimate of the benefit that would result from selecting a future action performed by the agent based on the current values of the action selection network parameters. The system 102 may use a reinforcement learning objective function The gradient of is used to train the action selection network and strengthen the learning objective function Given by:
[0070]
[0071] Among them, α and β are normal values, is the expected value relative to the current action selection policy (i.e., defined by the current value of the action selection network parameter θ), V(s t ,θ) refers to the action selection network for observation s t The generated value estimate, H(π(·|s t ,θ)) is a regularization term, which refers to the action selection network for observation s t The entropy of the probability distribution over the generated possible actions, and is the n-step look-ahead return, for example, given by:
[0072]
[0073] where γ is a discount factor between 0 and 1, r t+i is the reward received at time step t+i, and V(s t+n ,θ) refers to the estimated value at time step t+n.
[0074] System 102 may use supervised learning techniques to train grid cell network 202 and vision network 206 , as will be described in more detail below.
[0075] Figure 3 An example architecture of the grid cell network 202 is shown. At each time step, the grid cell network 202 is configured to process inputs including data defining the current velocity 110 of the agent (and optionally, the output of the vision network) to generate data estimating the current position of the agent in the environment. For example, in some implementations, the inputs at each time step may include, for example, a linear velocity at the center of the self and an angular velocity of the agent, for example, the sine and cosine of its angular velocity.
[0076] exist Figure 3In the depicted example, grid cell network 202 includes recurrent layer 302 , linear layer 304 , localization output layer 306 , and heading output layer 308 .
[0077] The recurrent layer 302 is configured to process the grid cell network input to update the hidden state of the recurrent layer (based at least in part on the value of the hidden state at the previous time step) and generate a recurrent layer output. The recurrent layer 302 can be, for example, a long short-term memory (LSTM) layer.
[0078] The linear layer is configured to apply a linear transformation (eg, defined by a weight matrix and a bias vector) to the recurrent layer output to generate a trellis code 210. In a specific example, the trellis code 210 may be represented as a vector having 512 components.
[0079] The localization output layer 306 processes the grid code 210 to generate localization data 310 that characterizes the estimated spatial localization of the agent in the environment. The localization data 310 may be represented in any of a variety of ways. The following are some examples.
[0080] In one example, the positioning data 310 may represent the estimated proximity of the agent's spatial positioning to each of a plurality of "reference" spatial positionings. For example, the grid cell network 202 may be trained to generate a vector corresponding to the N-dimensional vector Matching positioning data 310, the N-dimensional vector is given by:
[0081]
[0082] Where N is the number of reference spatial locations, x is a 2D vector representing the agent’s current spatial location (e.g., in xy coordinates), and μ i is a 2D vector representing the position of the ith reference space, and σ 2 is a predetermined variance parameter. Before training the grid cell network, the reference space location can be randomly selected in the environment. In this example, the location output layer 306 can generate the location data 310 by applying a linear transformation (e.g., defined by a weight matrix and a bias vector) and a subsequent soft-maximum activation function to the grid code 210.
[0083] In another example, the positioning data 310 can directly estimate the spatial positioning of the agent in the environment as, for example, a 2D vector. In this example, the positioning output layer 306 can generate the positioning data 310 by applying a linear transformation (e.g., defined by a weight matrix and a bias vector) to the grid code 210.
[0084] Heading output layer 308 processes grid code 210 to generate heading data 312 that characterizes the orientation of the agent, for example, about a vertical axis. Heading data 312 may be represented in any of a variety of ways. The following are some examples.
[0085] In one example, the heading data 312 may characterize the proximity of the agent's heading to each of a plurality of "reference" headings. For example, the grid cell network 202 may be trained to generate a vector corresponding to the M-dimensional vector Matching the forward direction data 312, the M-dimensional vector Given by:
[0086]
[0087] Where M is the number of reference headings, φ is a scalar value representing the agent's current heading (e.g., in radians), and α i is a numerical value representing the i-th reference heading (e.g., in radians), and κ is a predetermined positive scalar value. Prior to training the grid cell network, the reference heading may be randomly selected (e.g., in the range [-π, π]). In this example, the heading output layer 308 may generate heading data 312 by applying a linear transformation (e.g., defined by a weight matrix and a bias vector) and a subsequent soft-max activation function to the grid code 210. The weight matrix and bias vector may be learned or optimized during training.
[0088] In another example, heading data 312 may directly estimate the heading of the agent as a scalar value, for example, in the range [-π, π]. In this example, heading output layer 308 may generate heading data 312 by applying a linear transformation (e.g., defined by a weight matrix and a bias vector) to grid code 210. The weight matrix and bias vector may be learned or optimized during training.
[0089] The action selection system 102 can use any suitable supervised learning technique to train the grid cell network 202. More specifically, the action selection system 102 can use the gradient of the supervised learning objective function relative to the grid cell network parameters to adjust the values of the grid cell network parameters. The supervised learning objective function measures the similarity between: (i) the positioning data 310 and the heading data 312 generated by the grid cell network 202, and (ii) the "target" positioning data and heading data that characterize the actual position of the agent in the environment. In a simulated environment, the target positioning data and heading data can be accessible as parameters of the simulated environment that the simulation environment makes available (e.g., through an application programming interface (API)). In a real-world environment, the positioning position data and heading data can be generated by the agent's sensors, for example, the target positioning data can be generated by the agent's radio navigation system. The supervised learning objective function can be, for example, a cross-entropy objective function or a squared-error objective function.
[0090] In a specific example, the localization output layer 306 may generate an N-dimensional vector The forward direction output layer 308 can generate an M-dimensional vector And the supervised learning objective function It can be the cross entropy objective function given by:
[0091]
[0092] in, is given by equation (6) and characterizes the proximity of the agent's spatial localization to each of the multiple reference spatial localizations, It is given by equation (7) and characterizes the closeness of the agent's heading to each of the multiple reference headings.
[0093] In some cases, the action selection system 102 can use regularization techniques during training of the grid cell network 202 , for example, the action selection system 102 can apply dropout to the linear layer 304 that generates the grid code 210 during training.
[0094] Action selection system 102 may train vision network 206 to generate positioning data and heading data using supervised learning techniques similar to those described above with reference to grid cell networks.
[0095] Figure 4Provides a qualitative illustration of one example of the performance gain that can be achieved in a navigation task by providing a grid code rather than position data to the action selection network. Figure 4 The described navigation task requires an agent to repeatedly navigate to a goal location in an environment.
[0096] Diagram 402 depicts an example of a path followed by a "grid cell" agent trained to navigate through a maze using an action selection network configured to process current and target grid codes generated by the grid cell network, as described with reference to FIG. Figure 2 In this example, corridor 1 (indicated by a white "1" in 402) was blocked during training, so the action selection network makes adjustments to enable the grid cell agent to navigate through corridors 2 and 3 (indicated by white "2" and "3" in 402, respectively).
[0097] After training, corridor 1 in the maze is not blocked, i.e., the agent can freely traverse it. Diagram 404 depicts an example of a path followed by a trained grid cell agent. It can be appreciated that, although corridor 1 was never traversed during training, the trained grid cell agent frequently uses an effective "shortcut" to navigate to the target location through corridor 1.
[0098] Diagram 406 depicts an example of a path followed by a "place cell" agent trained to navigate through a maze using an action selection network that is configured to process position data rather than a grid code. In particular, at each time step, the action selection network processes current position data representing the agent's current position in the environment, and once the agent has reached a target location, position data representing the location of the target in the environment is used as a target. The position data consists of positioning data and heading data described with reference to equations (6) and (7), respectively. As before, corridor 1 is blocked during training, and the action selection network is adjusted to enable the place cell agent to navigate through corridors 2 and 3.
[0099] As before, after training the place unit agent, corridor 1 in the maze is unblocked. Diagram 408 depicts an example of a path followed by the trained place unit agent. It can be appreciated that although corridor 1 can be used to more efficiently reach the target location, the trained place unit agent frequently uses corridors 2 and 3 to navigate to the target location.
[0100] Figure 5A qualitative illustration of one example of the performance gain that can be achieved in a navigation task by providing a grid code rather than position data to the action selection network is provided. The horizontal axis of the graph 500 represents the number of training steps that have been performed in training the grid cell agent, the place cell agent, and the "A3C" agent. The vertical axis represents the cumulative reward received by the respective agents and humans while performing the navigation task. The navigation task involves repeatedly navigating to a target location in the environment.
[0101] As described above, the grid-cell agent performs the navigation task using an action selection network configured to process the current grid code and the target grid code generated by the grid-cell network. The place-cell agent performs the navigation task using an action selection network configured to process position data (e.g., location and heading data) rather than the grid code. The A3C agent performs the navigation task using an action selection network that processes neither the grid code nor the position data, but rather the action selection network processes the current observation, the reward received at the previous time step, and the data defining the action performed at the previous time step.
[0102] It can be understood from graph 500 that the grid cell agent achieves the best performance on the navigation task, even surpassing human performance.
[0103] Figure 6 is a flow chart of an example process 600 for selecting an action to be performed by an agent at a time step. For convenience, process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system (e.g., Figure 2 The action selection system 102) can perform processing 600.
[0104] The system receives: (i) data representing a current velocity of an agent, and (ii) a current observation representing a current state of an environment (602). The data representing the velocity of the agent may include data representing a translational velocity of the agent and data representing an angular velocity of the agent. The current observation may be generated by or derived from a sensor of the agent. For example, the current observation may include one or more of: a color image captured by a camera sensor of the agent, a hyperspectral image captured by a hyperspectral sensor of the agent, and geometric data captured by a laser sensor of the agent.
[0105] The system processes the data representing the current velocity of the agent using a grid cell neural network to obtain a current grid cell representation (i.e., a "grid code") (604). More specifically, the grid cell neural network generates a grid code as an intermediate output while processing the data representing the current velocity of the agent to generate an estimate of the position of the agent in the environment. The estimate of the position of the agent in the environment can include the spatial location of the agent in the environment and the heading of the agent.
[0106] In some cases, especially when the grid cell network is no longer trained, the system can use the grid cell network to generate grid code without generating an estimate of the agent's position in the environment. Figure 3 The example grid cell network architecture described above allows the system to generate grid codes but avoid providing the grid codes to the localization output layer or the heading output layer.
[0107] Optionally, the system can provide the output of the visual neural network as an additional input to the grid cell network. The visual network is configured to process the current observation to generate an estimate of the location of the agent in the environment (e.g., the spatial localization of the agent and the direction in which the agent is heading). In some cases, the system provides the output of the visual network directly as input to the grid cell network with a certain probability, otherwise, the output of the visual network is processed using a masking layer before it is provided to the grid cell network. The masking layer is configured to zero its input (i.e., to map any input to an output consisting of zeros) with a defined probability (e.g., 95%).
[0108] The system uses the current grid code and the current observation to select an action to be performed by the agent at the current time step (606). In particular, the system uses an action selection network to process the current grid code and the current observation to generate an action selection output, and thereafter uses the action selection output to select an action to be performed by the agent. In one example, the action selection output includes a score distribution over actions in a set of possible actions, i.e., a corresponding score is defined for each action in the set of possible actions. In this example, the system can select an action to be performed by the agent by sampling actions according to the score distribution over actions in the set of possible actions.
[0109] This specification uses the term "configured to" in connection with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, when operated, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, causes the device to perform the operation or action.
[0110] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.
[0111] The term "data processing apparatus" refers to data processing hardware and covers various apparatuses, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0112] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language, including compiled or interpreted languages or declarative or procedural languages; it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple coordinated files (e.g., files storing portions of one or more modules, subroutines or code). A computer program may be deployed to execute on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.
[0113] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.
[0114] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits (e.g., FPGAs or ASICs) or by a combination of special purpose logic circuits and one or more programmed computers.
[0115] Computers suitable for executing computer programs can be based on general or special microprocessors or both, or can be based on any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a dedicated logic circuit or incorporated into the dedicated logic circuit. Typically, the computer will also include or be operably coupled to receive data from one or more mass storage devices (e.g., magnetic, magneto-optical disks or optical disks) for storing data or transfer data to one or more mass storage devices or both. However, the computer does not have to have such devices. In addition, the computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, for example, a universal serial bus (USB) flash drive, to name a few.
[0116] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
[0117] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from the device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and in turn receiving a response message from the user.
[0118] The data processing apparatus for implementing a machine learning model may also include, for example, a dedicated hardware accelerator unit that is used to process the common and computationally intensive parts of machine learning training or production (i.e., inference, workloads).
[0119] The machine learning model can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).
[0120] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification) or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.
[0121] The computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to the user device for displaying data to a user interacting with the device used as a client and receiving user input from the user. Data generated at the user device, such as the result of the user interaction, can be received from the device at the server.
[0122] Although this specification contains many specific implementation details, these specific implementation details should not be interpreted as limitations on any invention or the scope of protection that can be claimed, but should be interpreted as descriptions of features that are specific to a particular embodiment of a particular invention. Specific features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments or in any suitable sub-combination. In addition, although the features may be described above as working in a certain combination and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.
[0123] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0124] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.
Claims
1. A method for selecting an action to be performed by an agent interacting with an environment, performed by one or more data processing devices, the method comprising: receiving a current observation representing a current state of the environment; Receiving data representing a current velocity of the agent; Processing data representing the agent's current velocity using an embedded neural network to generate a current grid cell representation; Wherein, the embedding neural network has been included as a subnetwork of the grid unit neural network when the grid unit neural network is trained to perform the following operations: processing an input representing the velocity of the agent using an embedding neural network to generate a grid cell representation as an intermediate output of the grid cell neural network; and processing the grid cell representation using one or more subsequent neural network layers of a grid cell neural network to generate an estimate of the agent's position in the environment; using the action selection neural network to process: (i) the current grid cell representation generated by the embedding neural network, and (ii) the current observation to generate an action selection network output; and Using the action selection network output, select an action to be performed by the agent in response to the current observation.
2. The method according to claim 1, wherein: The action selection network output consists of a score distribution over the actions in the set of possible actions, and Selecting an action to be performed by the agent includes sampling actions from the set of possible actions according to a distribution of scores over the actions in the set of possible actions.
3. The method according to claim 1, wherein: Grid cell neural networks are recurrent neural networks.
4. The method according to claim 1, wherein: The action selection neural network is a recurrent neural network.
5. The method according to claim 1, wherein: The data representing the speed of the intelligent body includes data representing the translation speed of the intelligent body and data representing the angular speed of the intelligent body.
6. The method according to claim 1, wherein: The estimation of the position of the agent in the environment includes the positioning of the agent and the direction of the agent's heading.
7. The method according to claim 1, further comprising: Processing an image representing the current appearance of the environment using a visual neural network to generate an estimate of the agent’s position in the environment. as well as The estimate of the agent's position in the environment generated by the vision neural network is provided as an additional input to the action selection neural network.
8. The method according to claim 7, wherein: The visual neural network is a convolutional neural network.
9. The method according to claim 1, wherein: Generating the current grid cell representation includes: Processing the data representing the current speed of the agent through a recurrent neural network layer of the embedded neural network to generate a recurrent layer output; and The recurrent layer output is processed through a linear neural network layer of the embedded neural network to generate the current grid cell representation.
10. The method according to claim 9, wherein: Dropout is applied to linear neural network layers.
11. The method according to claim 9, wherein: Processing the grid cell representation using one or more subsequent neural network layers of a grid cell neural network to generate an estimate of the agent's position in the environment includes: Producing a linear transformation of the grid cell representation; and A linear transformation of the grid cell representation is processed through a softmax layer to produce an estimate of the agent’s position in the environment.
12. The method according to claim 1, wherein: The action selection network is trained via reinforcement learning techniques to perform navigation tasks in the environment.
13. The method according to claim 12, wherein: The action selection network output also includes a predicted expected reward, which is an estimate of the time-discounted reward due to the environment being in the current state, and wherein the reinforcement learning technique is an actor-critic reinforcement learning technique.
14. The method of claim 1 further comprising providing the target grid cell representation as an additional input to the action selection neural network, wherein: The target grid cell representation is the grid cell representation generated by the embedding neural network at the time step when the agent's position in the environment is the target position.
15. The method according to claim 1, wherein: The grid cell neural network is trained using supervised learning techniques.
16. A system comprising: one or more computers; as well as One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for selecting an action to be performed by an agent interacting with an environment, the operations comprising: receiving a current observation representing a current state of the environment; Receiving data representing a current velocity of the agent; Processing data representing the agent's current velocity using an embedded neural network to generate a current grid cell representation; Wherein, the embedding neural network has been included as a subnetwork of the grid unit neural network when the grid unit neural network is trained to perform the following operations: processing an input representing the velocity of the agent using an embedding neural network to generate a grid cell representation as an intermediate output of the grid cell neural network; and processing the grid cell representation using one or more subsequent neural network layers of a grid cell neural network to generate an estimate of the agent's position in the environment; using the action selection neural network to process: (i) the current grid cell representation generated by the embedding neural network, and (ii) the current observation to generate an action selection network output; and Using the action selection network output, select an action to be performed by the agent in response to the current observation.
17. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for selecting an action to be performed by an agent interacting with an environment, the operations comprising: receiving a current observation representing a current state of the environment; Receiving data representing a current velocity of the agent; Processing data representing the agent's current velocity using an embedded neural network to generate a current grid cell representation; Wherein, the embedding neural network has been included as a subnetwork of the grid unit neural network when the grid unit neural network is trained to perform the following operations: processing an input representing the velocity of the agent using an embedding neural network to generate a grid cell representation as an intermediate output of the grid cell neural network; and processing the grid cell representation using one or more subsequent neural network layers of a grid cell neural network to generate an estimate of the agent's position in the environment; using the action selection neural network to process: (i) the current grid cell representation generated by the embedding neural network, and (ii) the current observation to generate an action selection network output; and Using the action selection network output, select an action to be performed by the agent in response to the current observation.
18. The non-transitory computer storage medium of claim 17, wherein: The action selection network output consists of a score distribution over the actions in the set of possible actions, and Selecting an action to be performed by the agent includes sampling actions from the set of possible actions according to a distribution of scores over the actions in the set of possible actions.
19. The non-transitory computer storage medium of claim 17, wherein: Grid cell neural networks are recurrent neural networks.
20. The non-transitory computer storage medium of claim 17, wherein: The action selection neural network is a recurrent neural network.
21. A system for selecting an action to be performed by an agent interacting with an environment, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to: The grid cell neural network is configured as: receiving input including data characterizing a velocity of an agent; processing the input to generate a grid cell representation; as well as Processing the grid cell representation to generate an estimate of the agent’s position in the environment; The action selection neural network is configured as follows: receiving an input comprising a grid cell representation and an observation representing a state of an environment; as well as processing the input to generate an action selection network output; Subsystems are configured as: Receiving data representing a current velocity of the agent; The data representing the current velocity of the agent is provided as input to the grid cell neural network to obtain the current grid cell representation, including: processing the data representing the current velocity of the agent using a grid cell neural network to generate a current grid cell representation as an output of one or more intermediate layers of the grid cell neural network, Wherein, in the architecture of the grid cell neural network, each intermediate layer of the grid cell neural network is located: (i) after an input layer of the grid cell neural network configured to receive data representing a current velocity of the agent, and (ii) before an output layer of the grid cell neural network configured to generate an estimate of a current position of the agent in the environment; receiving a current observation representing a current state of the environment; Let (i) the current grid cell generated by the intermediate layer of the grid cell neural network be represented as (ii) the current observation is provided as input to the action selection neural network to obtain the action selection network output; and Using the action selection network output, select an action to be performed by the agent in response to the current observation.
22. The system of claim 21, wherein: The action selection network output consists of a score distribution over the actions in the set of possible actions, and Selecting an action to be performed by the agent includes sampling actions from the set of possible actions according to a distribution of scores over the actions in the set of possible actions.
23. The system of claim 21, wherein: Grid cell neural networks are recurrent neural networks.
24. The system of claim 21, wherein: The action selection neural network is a recurrent neural network.
25. The system of claim 21, wherein: The data representing the speed of the intelligent body includes data representing the translation speed of the intelligent body and data representing the angular speed of the intelligent body.
26. The system of claim 21, wherein: The estimation of the position of the agent in the environment includes the positioning of the agent and the direction of the agent's heading.
27. The system of claim 21, wherein: The instructions further cause the one or more computers to implement a visual neural network configured to: receiving input comprising an observation characterizing a state of an environment; processing the input to generate an estimate of the agent's position in the environment; and Therein, the grid cell neural network is configured to receive input also including an estimate of the position of the agent in the environment.
28. The system of claim 27, wherein: The subsystem is also configured to provide an estimate of the agent's position in the environment generated by the vision neural network as input to the grid cell neural network.
29. The system of claim 27, wherein: The subsystem is also configured to: providing an estimate of the position of the agent in the environment generated by the visual neural network as input to the grid cell neural network with a first probability; An estimate of the position of the agent in the environment generated by the visual neural network is processed with a second probability through a masking layer, wherein the masking layer zeroes the estimate of the position of the agent in the environment to generate a masked estimate of the position of the agent, and the masked estimate of the position of the agent is provided as an input to the grid cell neural network.
30. The system of claim 27, wherein: The visual neural network is a convolutional neural network.
31. The system of claim 21, wherein: Processing the input to generate a grid cell representation includes: processing the input by a recurrent neural network layer to generate a recurrent layer output; and The recurrent layer output is processed by a linear neural network layer to produce a grid cell representation.
32. The system of claim 31, wherein: Dropout is applied to linear neural network layers.
33. The system of claim 31, wherein: Processing the grid cell representation to generate an estimate of the agent's position in the environment involves: Producing a linear transformation of the grid cell representation; and A linear transformation of the grid cell representation is processed through a softmax layer to produce an estimate of the agent’s position in the environment.
34. The system of claim 21, wherein: The action selection network is trained via reinforcement learning techniques to perform navigation tasks in the environment.
35. The system of claim 34, wherein: The action selection network output also includes a predicted expected reward, which is an estimate of the time-discounted reward due to the environment being in the current state, and wherein the reinforcement learning technique is an actor-critic reinforcement learning technique.
36. The system of claim 34, wherein: The input received by the action selection neural network also includes a target grid cell representation, wherein the target grid cell representation is a grid cell representation generated by the grid cell neural network at a time step when the position of the agent in the environment is the target position; and The subsystem is also configured to maintain data specifying a target grid cell representation and to provide the target grid cell representation as input to the action selection neural network.
37. The system of claim 21, wherein: The grid cell neural network is trained using supervised learning techniques.
38. The system of claim 27, wherein: Visual neural networks are trained using supervised learning techniques.
39. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for selecting an action to be performed by an agent interacting with an environment, the operations comprising: Receiving data representing a current velocity of the agent; The data representing the current velocity of the agent is provided as input to the grid cell neural network to obtain the current grid cell representation, where: The grid cell neural network is trained to: processing an input representing the velocity of the agent to generate a grid cell representation; and Processing the grid cell representation to generate an estimate of the position of the agent in the environment; and obtaining a current grid cell representation includes: processing the data representing the current velocity of the agent using a grid cell neural network to generate a current grid cell representation as an output of one or more intermediate layers of the grid cell neural network, Wherein, in the architecture of the grid cell neural network, each intermediate layer of the grid cell neural network is located: (i) after an input layer of the grid cell neural network configured to receive data representing a current velocity of the agent, and (ii) before an output layer of the grid cell neural network configured to generate an estimate of a current position of the agent in the environment; receiving a current observation representing a current state of the environment; processing (i) the current grid cell representation generated by the intermediate layer of the grid cell neural network and (ii) the current observation using the action selection neural network to generate an action selection network output; and Using the action selection network output, select an action to be performed by the agent in response to the current observation.
40. A method for selecting an action to be performed by an agent interacting with an environment, performed by one or more data processing devices, the method comprising: Receiving data representing a current velocity of the agent; The data representing the current velocity of the agent is provided as input to the grid cell neural network to obtain the current grid cell representation, where: The grid cell neural network is trained to: processing an input representing the velocity of the agent to generate a grid cell representation; and Processing the grid cell representation to generate an estimate of the position of the agent in the environment; and obtaining a current grid cell representation includes: processing the data representing the current velocity of the agent using a grid cell neural network to generate a current grid cell representation as an output of one or more intermediate layers of the grid cell neural network, Wherein, in the architecture of the grid cell neural network, each intermediate layer of the grid cell neural network is located: (i) after an input layer of the grid cell neural network configured to receive data representing a current velocity of the agent, and (ii) before an output layer of the grid cell neural network configured to generate an estimate of a current position of the agent in the environment; receiving a current observation representing a current state of the environment; processing (i) the current grid cell representation generated by the intermediate layer of the grid cell neural network and (ii) the current observation using the action selection neural network to generate an action selection network output; and Using the action selection network output, select an action to be performed by the agent in response to the current observation.
41. A system for selecting actions to be performed by an agent interacting with an environment to navigate to a target location in the environment, wherein the environment is a real-world environment and the agent is a mechanical agent navigating through the real-world environment, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to: The grid cell neural network is configured as: receiving input including data characterizing a velocity of an agent; and The input is processed to generate a grid cell representation, where The grid cell representation is generated as output by the hidden layer of the grid cell neural network; The action selection neural network is configured as follows: Receive input including: (i) a current grid cell representation generated by a grid cell neural network at a current time step, (ii) a target grid cell representation generated by a grid cell neural network at a previous time step when the agent was at a target location in the environment, and (iii) a current observation representing a current state of the environment; and processing the input to generate an action selection network output; Subsystems are configured as: maintaining in memory the data represented by the specified target grid cell; and At each of a number of time steps: Receiving data representing a current velocity of the agent; providing data representing the current velocity of the agent as input to the grid cell neural network to obtain a current grid cell representation; receiving a current observation representing a current state of the environment; Provide (i) the current grid cell representation, (ii) the target grid cell representation, and (iii) the current observation as input to the action selection neural network to obtain the action selection network output; Using the action selection network output, select an action to be performed by the agent in response to the current observation.
