Training camera policy neural network by self-prediction

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training camera policy neural networks.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 352,633, filed on June 15, 2022. The disclosure of the prior application is considered part of and incorporated by reference into the disclosure of the present application. Technical Field

[0002] This specification relates to using machine learning models to process data. Background Art

[0003] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate output based on the received input and the parameter values ​​of the model.

[0004] Some machine learning models are deep models that employ multiple layers of models to generate outputs from received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to a received input to generate an output. Summary of the invention

[0005] This specification generally describes techniques for training a camera policy neural network and using the trained camera policy neural network.

[0006] One example implementation described herein relates to a method for training a camera policy neural network. The camera policy neural network is used to control the position of a camera sensor in an environment in which a robot is interacting. The method includes: obtaining data of one or more target sensors of a specified robot; obtaining a first observation, the first observation comprising one or more images of the environment captured by the camera sensor when in a current position; processing a camera policy input using a camera policy neural network, the camera policy input comprising (i) data of one or more target sensors of a specified robot and (ii) the first observation, the first observation comprising one or more images captured by the camera sensor, to generate a camera policy output defining a camera control action for adjusting the position of the camera sensor; adjusting the current position of the camera sensor based on the camera control action; obtaining a second observation, the second observation comprising one or more images of the environment captured by the camera sensor when in the adjusted position; generating a corresponding prediction for each of the one or more target sensors representing a sensor reading generated by the target sensor based on the second observation; generating a corresponding reward for the camera policy neural network for each target sensor based on an error in the corresponding prediction of the target sensor; and training the camera policy neural network using the rewards of the one or more target sensors.

[0007] In this specification, a "robot" can be a real-world mechanical robot or a computer simulation of a real-world mechanical robot. Therefore, the camera policy neural network can be trained in a real-world environment or a simulated environment, that is, a computer simulation of a real-world environment. In some embodiments, when the camera policy neural network is trained in a simulated environment, the trained camera policy neural network can be used for downstream tasks in the real-world environment. For example, the trained camera policy neural network can be used as part of training a robot policy neural network for controlling a robot. Training the robot policy neural network can be performed in a real-world environment, and after training, the robot policy neural network can be used to control a real-world robot in a real-world environment. Alternatively, training the robot policy neural network can also be performed in a computer simulation, and after training, the robot policy neural network can be used to control a real-world robot in a real-world environment.

[0008] In some embodiments, the camera sensor is part of the robot.

[0009] In some embodiments, the camera sensor is external to the robot within the environment.

[0010] In some implementations, the camera sensor is a foveated camera.

[0011] In some embodiments, the fovea camera includes multiple cameras with different fields of view.

[0012] In some implementations, the corresponding prediction is a prediction of the value of the sensor reading of the target sensor at the time step at which the second observation was generated.

[0013] In some implementations, the corresponding prediction is a prediction of a reward generated from a value of at least a sensor reading of the target sensor at each of one or more time steps after the time step at which the second observation was generated.

[0014] In some embodiments, generating a corresponding prediction for each of the one or more target sensors that characterizes a sensor reading generated by the target sensor based on the second observation includes: processing a predictor input including the second observation using a sensor prediction neural network to generate a predictor output including the corresponding prediction for each of the one or more target sensors.

[0015] In some embodiments, the method further includes training the sensor prediction neural network using errors in corresponding predictions of the one or more target sensors.

[0016] In some embodiments, the robot includes a plurality of sensors, the plurality of sensors including one or more target sensors, the predictor output includes a corresponding prediction for each of the plurality of sensors, and training the sensor prediction neural network includes training the sensor prediction neural network using errors in the corresponding predictions for each of the plurality of sensors.

[0017] In some embodiments, the target sensor comprises one or more proprioceptive sensors of the robot.

[0018] In some implementations, the action specifies a target velocity for each of one or more actuators of the camera sensor.

[0019] In some implementations, training the camera policy neural network using rewards for the one or more target sensors includes training the camera policy neural network through reinforcement learning.

[0020] In some implementations, training the camera policy neural network via reinforcement learning includes training the camera policy neural network jointly with the camera criticism neural network.

[0021] In some embodiments, the robot further comprises one or more controllable elements.

[0022] In some implementations, each of the controllable elements is controlled using a corresponding fixed policy during training of the camera policy neural network.

[0023] In some implementations, during training of the camera policy neural network, each of the controllable elements can be controlled using a robotic policy neural network that receives input including one or more images generated by the camera sensor.

[0024] In some implementations, during training of the camera policy neural network, the robot policy neural network is trained on external rewards for a specified task.

[0025] In some implementations, training of the camera policy neural network is performed as an auxiliary task during training of the robot policy neural network.

[0026] In some embodiments, the method further includes: after training the camera policy neural network: using the trained camera policy neural network to train a robot policy neural network that receives input including one or more images generated by the camera sensor to control each of the one or more controllable elements using external rewards for one or more specified tasks.

[0027] In some embodiments, using a trained camera policy neural network to train a robotic policy neural network that receives input comprising one or more images generated by a camera sensor to control each of one or more controllable elements using external rewards for one or more specified tasks includes: using the trained camera policy neural network to generate training data for training the robotic policy neural network.

[0028] In some embodiments, the one or more controllable elements include one or more manipulators.

[0029] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0030] By training a camera policy neural network as described in this specification, the neural network learns active vision skills for moving a camera to observe a robot sensor from informative viewpoints, without external rewards or labels. Specifically, the camera policy neural network learns to move the camera to a viewpoint that is most predictive for a target sensor, which is specified using a conditioning input to the neural network. The learned policy is capable even when training uses a noisy learned reward function, avoids occlusions, and accurately frames the sensor to a specific location in the view. That is, the learned policy learns to move the camera to avoid occlusions between the camera sensor and the target sensor, and learns to frame the sensor to a location in the view that is most predictive for the sensor readings generated by the sensor.

[0031] Learning these active vision skills can be useful for any of a variety of downstream tasks. For example, learning to visually fit objects into consistent image locations actively reduces the image space variance attributable to object locations. Thus, locking onto object locations within an image can simplify learning downstream robotic skills, i.e., training policy neural networks for controlling a robot to perform a task or learning reusable skills. For example, utilizing a camera policy neural network (or a subnetwork of a neural network) can improve the acquisition of visually guided manipulation policies because they can then focus on the hard-to-learn manipulation aspects of the policy.

[0032] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 An example training system is shown.

[0034] Figure 2Ais a flow chart of an example process for generating training data for training a camera policy neural network.

[0035] Figure 2B is a flowchart of an example process for training a camera policy neural network.

[0036] Figure 3 is a flow chart of an example process for training a sensor prediction neural network.

[0037] Figure 4 An example of training of a neural network is shown.

[0038] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0039] Figure 1 An example training system 100 is shown. Training system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0040] Training system 100 trains camera policy neural network 110 , which controls the position of camera sensor 102 in environment 106 including robot 104 .

[0041] In the present description, robot 104 may be a real-world mechanical robot or a computer simulation of a real-world mechanical robot. Thus, camera policy neural network 110 may be trained in environment 106 as a real-world environment or a simulated environment, i.e., a computer simulation of a real-world environment.

[0042] When the camera policy neural network 110 is trained in a simulated environment, after training, the camera policy neural network 110 can be used for downstream tasks in a real-world environment. For example, the trained camera policy neural network 110 can be used as part of training a robot policy neural network for controlling the robot 104. This training of the robot policy neural network can also be performed in a real-world environment or in a computer simulation, and after training, the robot policy neural network can be used to control a real-world robot in a real-world environment. These downstream tasks are described in more detail below.

[0043] The robot 104 typically includes a collection of sensors for sensing the environment 106 , such as one or more of the following: proprioceptive sensors; exteroceptive sensors such as camera sensors, lidar sensors, audio sensors, etc.; tactile sensors, etc.

[0044] Although the present description generally describes sensors as being sensors of a robot, the system 100 can be used to generate predictions for sensors of any suitable type of agent that has sensors and can move in an environment. That is, more generally, the robot 104 can be any suitable type of agent. For example, when the environment 104 is a simulated environment, examples of other agent types can include simulated humans or animals or other avatars equipped with sensors.

[0045] Specifically, camera policy neural network 110 receives input including observations 110 —i.e., including one or more images 108 captured by a camera sensor—and processes the input to generate camera policy output 112 that defines camera control actions 114 for adjusting the position of camera sensor 102 .

[0046] Specifically, the position of the camera sensor 102 may be adjusted by applying control inputs to one or more actuators, and the camera policy output 112 may specify a corresponding control input for each of the one or more actuators of the camera sensor 102. As a specific example, the camera control action 114 may specify a target velocity for each of the one or more actuators of the camera sensor or a different type of control input for each of the one or more actuators.

[0047] The camera sensor 102 may be any of a variety of types of camera sensors. For example, the camera sensor 102 may be a foveated camera sensor. A foveated camera is a camera that produces images where the image resolution varies across the image, i.e., is different in different parts of the image.

[0048] The foveated camera sensor may be implemented as a single multi-resolution hardware device or as multiple cameras with different fields of view.

[0049] When the environment is a computer simulation, a foveated image can be generated by rendering different regions of the camera's field of view at different resolutions. For example, the "foveated area", i.e., the higher resolution portion of the image, can be rendered at a higher resolution (consuming more computational response to focus on it), while portions outside the foveated area can be rendered at a lower resolution (consuming less computational resources).

[0050] Alternatively, camera sensor 102 may be a single single-resolution camera device.

[0051] As will be described in more detail below, the input to the camera policy neural network 110 also identifies one or more target sensors of the robot 104 , i.e., directs the camera policy neural network 110 to focus the camera on the target sensors of the robot 104 .

[0052] The robot 104 and the camera sensor 102 may be arranged within the environment 106 in any of a variety of configurations.

[0053] For example, the camera sensor 102 may be part of the robot 104. That is, the camera sensor 102 may be attached to or embedded within the body of the robot 104. Thus, the one or more actuators that control the camera position are a subset of the robot 104's actuators.

[0054] As another example, the camera sensor 102 may be external to the robot 104 within the environment 106. Thus, one or more actuators that control the position of the camera are separate from the actuators of the robot 104.

[0055] In particular, the system 100 trains the camera policy neural network 104 so that the camera policy neural network 104 can effectively guide the camera sensor 102 to consistently lock on the target sensor identified in the input of the neural network 104, even when the robot 104 (and therefore the target sensor) is changing position within the environment 106.

[0056] That is, during training of camera policy neural network 110 , the system or another system controls robot 104 to change position within environment 106 .

[0057] For example, a fixed policy, i.e., a fixed policy that is not learned during training, may be used to control the robot 104. The policy may be, for example, a random policy that randomly selects control inputs to the robot 104 at any given time. As another example, a policy may be a policy that has been learned and maximizes the entropy of the target sensor. As another example, a policy learned during training of the camera policy neural network 110 may be used to control the robot 104. For example, the policy being learned may be a policy that attempts to maximize the entropy of the target sensor.

[0058] In some implementations, system 100 uses sensor prediction neural network 120 as part of training of camera policy neural network 104 .

[0059] The sensor prediction neural network 120 is configured to receive input observations 110 including one or more images captured by a camera sensor and generate a predictor output including a corresponding prediction for each sensor of at least a subset of the sensors of the robot 104 .

[0060] The prediction for a given sensor may be any of a variety of different predictions that characterize the current or future state of the sensor.

[0061] In some embodiments, the corresponding prediction is a prediction of the value of the sensor reading of the sensor at the time step when the last image in the one or more images in the input was generated. As will be described below, a "time step" within a sequence of environmental interactions, such as a round, corresponds to the time when the last image in the one or more images in a given observation was generated.

[0062] In some other embodiments, the corresponding prediction is a prediction of the value of the sensor reading of the sensor at the next time step immediately following the time step at which a last image in the one or more images in the input was generated.

[0063] In some other embodiments, the corresponding prediction is a prediction of the reward generated from the value of the sensor reading of at least the target sensor at each of one or more time steps after the last image in the one or more images in the input was generated. The "reward" is the sum or time-discounted sum of the values ​​at one or more time steps. For example, at time step The return can satisfy: in In the round All subsequent time steps or within the episode The range changes for some fixed number of time steps thereafter, is a discount factor greater than zero and less than or equal to one, and is the sensor at time step The value at .

[0064] In any of the above embodiments, the output of the sensor prediction neural network 120 may be a direct regression predicted value (or reward), or may be a parameter of a discrete or continuous distribution over a set of possible values ​​(or rewards). That is, in some cases, the sensor prediction neural network 120 may generate a distribution prediction that defines a distribution over a set of possible values ​​(or rewards). For example, the distribution may be a categorical distribution over possible values ​​(or rewards), and the output may provide support for the categorical distribution, or the distribution may be any suitable type of distribution, and the output may specify a quantile function of the distribution.

[0065] For example, the system 100 may train the camera policy neural network 110 jointly with the camera critic neural network 150 .

[0066] The camera evaluation neural network 150 is a neural network that receives inputs including observations including one or more images captured when the camera sensor 102 is in a particular position and a camera control action generated for a target sensor, and generates as output an evaluation output defining a predicted reward for the target sensor if the camera control action is performed when the camera sensor is in the particular position. For example, the evaluation output may be a regressed reward value, or may specify parameters of a distribution over possible rewards.

[0067] Reference below Figures 2A to 4 Generating sensor predictions and training the camera policy neural network 110 are described in more detail.

[0068] Once system 100 has trained camera policy neural network 110 , system 100 (or another separate system) may use the trained camera policy neural network 110 for any of a variety of purposes.

[0069] As a specific example, system 100 can use trained camera policy neural network 110 to train a robot policy neural network that controls robot 104 to perform one or more specified tasks. The robot policy neural network receives inputs including one or more images generated by a camera sensor and generates a policy output for controlling the robot.

[0070] As a general example, a task may include, for example, one or more of navigating to a specified location in the environment, identifying a specific object in the environment, manipulating a specific object in a specified manner, and the like.

[0071] That is, the robot 104 may have one or more controllable elements, such as one or more manipulators or other elements that may be controlled to move portions of the body of the robot 104 within an environment.

[0072] As described above, in some embodiments, during training of the camera policy neural network 110, each of the controllable elements is controlled using a corresponding fixed policy, such as a random policy.

[0073] After training the camera policy neural network, the trained camera policy neural network can be used to train the robot policy neural network using external rewards for one or more specified tasks.

[0074] In some cases, the robot policy neural network can control both the camera sensor and the robot, for example, when the camera sensor is mounted on the robot, or when the camera sensor is positioned away from the robot, by transmitting control signals to a control system that controls one or more actuators of the camera sensor.

[0075] In these cases, the trained camera policy neural network 110 can be used to generate training data for training of the robotic policy neural network, for example, by controlling a camera to capture images that allow the robotic policy neural network to explore the environment.

[0076] In some other cases, the robot policy neural network 110 can be used to control the robot 104 (or, when the camera sensor 102 is mounted on the robot, one or more other joints or other actuators of the robot 104 other than those that control the camera sensor 102), and the camera policy neural network 110 or a subnetwork of the camera policy neural network 110 together with downstream subnetworks can be used to change the position of the camera sensor 102 during training.

[0077] In some of these cases, camera policy neural network 110 or a sub-network of camera policy neural network 110 can be trained, i.e., fine-tuned, along with the robot policy neural network. In other of these cases, camera policy neural network 110 or a sub-network of camera policy neural network 110 remains fixed during training of the robot policy neural network.

[0078] In some implementations, during this training, the learning or fixed controller may generate inputs to the camera policy neural network 110 (or subnetwork) to cause the camera policy neural network 110 to move the camera to different locations. That is, the learning or fixed controller may identify a target sensor of the robot 104 to provide as input to the neural network 110 at any given time step, or may generate different types of conditioning inputs to specify a target location of the camera sensor 102 in the environment 106.

[0079] However, in some other examples, during training of the camera policy neural network, a robotic policy neural network is used to control each of the controllable elements.

[0080] In these examples, the robot policy neural network can be trained for external rewards for a specified task during training of the camera policy neural network, and training of the camera policy neural network is performed as an auxiliary task during training of the robot policy neural network, i.e., so that the robot can generate more useful images with the help of the camera policy neural network and the robot policy neural network generates more useful policy outputs to improve performance of the task.

[0081] In any of the above cases, the robot policy neural network receives as input observations including one or more observations captured by the camera sensor 102 .

[0082] The robot policy neural network processes the input to generate a policy output that defines a policy for controlling the robot, i.e., defines the actions to be performed by the robot from a set of actions ("control inputs"). For example, the set of actions may include a fixed number of actions, or may be a continuous action space.

[0083] In one example, the policy output may include a corresponding Q-value for each control input in the fixed set. The system may process the Q-values ​​(e.g., using a soft-max function) to generate a corresponding probability value for each control input, which may be used to select a control input or may select the control input with the highest Q-value.

[0084] The Q-value of a control input is an estimate of the “reward” that would result from the agent exercising the control input in response to the current observation, and thereafter controlling using the control input generated by the controller.

[0085] Return refers to a cumulative measure of the "rewards" received by an agent, such as the time-discounted sum of rewards. The agent may receive a corresponding reward at each time step, where the reward is specified by a scalar numerical value and represents, for example, the agent's progress in completing an assigned task.

[0086] In another example, the policy output may include a respective numerical probability value for each control input in the fixed set. The system may select the control input, for example, by sampling the control inputs according to the probability values, by selecting the control input with the highest probability value.

[0087] As another example, when the control input space is continuous, the policy output may include parameters of a probability distribution over the continuous control input space. The system may then select the control input by sampling the control input from the probability distribution or by selecting an average control input.

[0088] In some embodiments, the environment is a real-world environment and the robot is a mechanical agent that interacts with the real-world environment. For example, the robot can be a robot that interacts with the environment to achieve a goal, such as to locate an object of interest in the environment, move an object of interest to a specified location in the environment, physically manipulate an object of interest in the environment in a specified manner, or navigate to a specified destination in the environment.

[0089] In these embodiments, in addition to camera sensor images, observations may optionally include one or more object position data and sensor data to capture the observations as the agent interacts with the environment, such as sensor data from distance or position sensors or from actuators.

[0090] For example, in the case of a robot, observations may include data characterizing the current state of the robot, such as one or more of: joint positions, joint velocities, joint forces, torques or accelerations, such as gravity-compensated torque feedback, and a global or relative pose of an item held by the robot.

[0091] In the case of a robot or other mechanical agent or vehicle, observations may similarly include one or more of position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. Observations may be defined in 1, 2, or 3 dimensions, and may be absolute and / or relative observations.

[0092] Observations may also include, for example, data obtained by one or more sensor devices that sense the real-world environment; for example, sensed electronic signals such as motor current or temperature signals; and / or image or video data, for example, from a camera or LIDAR sensor, for example, data from the agent's sensors or data from sensors located separately from the agent in the environment.

[0093] Observations can also include data characterizing the task, for example, data specifying a target state of the agent, such as target joint positions, velocities, forces, or torques or higher-level states such as the agent's coordinates or the agent's velocity, data specifying target states or positions or both of other objects in the environment, data specifying target positions in the environment, etc.

[0094] The control inputs may be control inputs for controlling a robot, e.g. torques or higher level control commands to a joint of the robot, or an autonomous or semi-autonomous land or air or sea vehicle, e.g. torques or higher level control commands to a control surface or other control element of the vehicle.

[0095] In other words, control inputs may include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or part of another mechanical agent. Control inputs may additionally or alternatively include electronic control data, such as motor control data, or more generally data for controlling one or more electronic devices within an environment, the control of which has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or marine vehicle, control inputs may include actions that control navigation, such as steering, and movement, such as braking and / or acceleration of the vehicle.

[0096] In some implementations, the environment is a simulated environment, and the robot and camera sensor are implemented as one or more computer programs that interact with the simulated environment.

[0097] Typically, in the case of a simulated environment, observations may include simulated versions of one or more of the previously described observations or types of observations, and actions may include simulated versions of one or more of the previously described control inputs or types of control inputs. That is, the simulated environment is a computer simulation of a real-world environment, and the agent is a computer simulation of a robot in the real-world environment.

[0098] In some cases, the system can be used to control the interaction of an agent with a simulated environment, and the system can train parameters of a robot policy neural network (e.g., using reinforcement learning techniques) and a camera policy neural network based on the interaction of the agent with the simulated environment. After training the neural network based on the interaction of the agent with the simulated environment, the agent can be deployed in a real-world environment, and the trained neural network can be used to control the interaction of the agent with the real-world environment. Training the neural network based on the interaction of the agent with a simulated environment (i.e., in place of a real-world environment) can avoid wear and tear on the agent and can reduce the likelihood that the agent can damage aspects of itself or its environment by performing poorly chosen actions.

[0099] Alternatively, after training, the camera policy neural network, the robot policy neural network, or both can continue to be used in a simulated environment, for example, to control a simulated robot or other agent in the simulated environment. As a specific example thereof, the simulated environment can be integrated with or otherwise be part of a video game or other software, where some agents are controlled by human users and other agents are controlled by a computer system. The camera policy neural network, the robot policy neural network, or both can be used as part of the control of the other agents.

[0100] Optionally, in any of the above embodiments, observations at any given time step may include data from previous time steps that may be useful in characterizing the environment, such as actions performed at previous time steps, rewards received at previous time steps, etc.

[0101] Figure 2A 2 is a flow chart of an example process 200 for generating training data for training a camera policy neural network. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system appropriately programmed according to the present specification, such as Figure 1 The training system 100 can perform process 200.

[0102] Typically, to train a camera policy neural network, the system may repeatedly perform control rounds to generate training data for the camera policy neural network.

[0103] For each round, the system obtains data of one or more target sensors of a designated robot (step 202). For example, for each round, the system may select, for example, randomly, one or more of the robot's sensors to serve as target sensors for the round. For example, the system may select one or more proprioceptive sensors, exteroceptive sensors, or tactile sensors of the robot as target sensors.

[0104] The system can then repeatedly perform steps 204 to 210 of process 200 to generate training data for training the camera policy neural network, e.g., until a termination criterion for the episode is met, e.g., a certain amount of time has passed or a certain amount of observations have been generated.

[0105] The system obtains a first observation comprising one or more images of the environment captured by the camera sensor when the camera sensor is in its current position (step 204). For example, the observation may include the two (or more) most recent images captured by the camera sensor.

[0106] The system processes a camera policy input including (i) data specifying one or more target sensors of the robot and (ii) a first observation using a camera policy neural network to generate a camera policy output defining a camera control action for adjusting a position of the camera sensor (step 206 ).

[0107] For example, the camera policy output may define a probability distribution over the space of camera control actions, and the system may sample actions from the probability distribution, or the camera policy output may be the regressed camera control actions directly.

[0108] The system adjusts the current position of the camera sensor based on the camera control action (step 208). That is, the system causes the camera sensor to be moved according to the camera control action. For example, the system can apply a control input to an actuator of the camera to cause the actuator to reach a target speed specified by the action.

[0109] The system obtains a second observation comprising one or more images of the environment captured by the camera sensor while in the adjusted position (step 210). That is, the observation comprises one or more images captured by the camera after the camera has been moved according to the camera control action.

[0110] Thus, the system generates training tuples that specify a first observation, an action, and a second observation.

[0111] The system may then repeat process 200, e.g., by using the "second" observation as the "first" observation for the next iteration of process 200, until a termination criterion for the round has been met, e.g., until a specified number of tuples have been generated or the environment reaches a certain terminal state.

[0112] The system may then store the generated tuples in a memory such as a replay memory. For example, multiple actor processes within the system may generate training tuples in parallel and store the generated tuples in the replay memory. In some cases, the replay memory has a fixed capacity, and during training, tuples that have been used for training are probabilistically deleted to ensure that the fixed capacity is not exceeded.

[0113] Figure 2B is a flow chart of an example process 220 for training a camera policy neural network. For convenience, process 220 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system appropriately programmed according to the present specification, such as Figure 1 The training system 100 can perform process 220.

[0114] For example, a learner process within the system may repeatedly execute process 220 on training tuples generated by one or more actor processes and obtained from a replay memory.

[0115] The system obtains a tuple including a first observation, a camera control action, and a second observation (step 212 ). For example, the system can sample a tuple from a replay memory, such as a tuple that has been generated by executing process 200 .

[0116] The system generates a corresponding prediction for each of the one or more target sensors representing the sensor reading generated by the target sensor from the second observation (step 214). That is, when the camera policy neural network selects a camera control action, the system generates a corresponding prediction for each target sensor that is one of the target sensors.

[0117] As described above, the prediction for a given sensor can be generated by a sensor prediction neural network and can be any of a variety of different predictions that characterize the current or future state of the sensor. Thus, the system generates a prediction by processing the second observation using the sensor prediction neural network.

[0118] In some implementations, the corresponding prediction is a prediction of the value of the sensor reading of the sensor at the time step at which a last image in the one or more images in the input was generated.

[0119] In some other embodiments, the corresponding prediction is a prediction of the value of the sensor reading of the sensor at the next time step immediately following the time step at which a last image in the one or more images in the input was generated.

[0120] In some other embodiments, the corresponding prediction is a prediction of the reward generated from the value of the sensor reading of at least the target sensor at each of one or more time steps after the time step at which the second observation was generated.

[0121] In any of the above embodiments, the output of the sensor prediction neural network may be a direct regression predicted value (or reward), or may be a parameter of a discrete or continuous distribution over a set of possible values ​​(or rewards).

[0122] Typically, when a sensor prediction neural network is used, the sensor prediction neural network is trained jointly with the training of the camera policy neural network. Figure 3 Training a sensor prediction neural network is described in more detail.

[0123] The system generates a corresponding reward for the camera policy neural network for each target sensor based on the error in the corresponding prediction of the target sensor (step 216). Generally, when the error is low, the reward is high, so that the camera policy neural network is rewarded for positioning the camera sensor so as to generate accurate predictions.

[0124] For example, when the corresponding prediction is a prediction of the value of a sensor reading of a sensor at a time step when a last image in one or more images in the input was generated, the system may determine the reward as the negative of the error, e.g., the squared error, between the prediction and the true value of the sensor reading of the sensor at that time step.

[0125] For example, when the corresponding prediction is a prediction of the value of a sensor reading at the next time step, the system may determine the reward as the negative of the error, eg, the squared error, between the prediction and the true value of the sensor reading at the next time step.

[0126] For example, when the corresponding prediction is a prediction of a return, the system may determine the reward as the negative of the time difference loss.

[0127] That is, the temporal difference loss is the loss between the prediction and the target prediction, which is calculated using (i) the discount factor, (ii) the true value of the sensor at the next time step, and (iii) the new prediction at the next time step generated by the sensor prediction neural network by processing the observation at the next time step - that is, the second observation. When the prediction specifies a distribution, the temporal difference loss can be a distributional temporal difference loss using a distributional target prediction.

[0128] The system trains the camera policy neural network using the rewards for the one or more target sensors in the training tuple (step 218 ).

[0129] Specifically, the system trains the camera policy neural network through reinforcement learning to generate actions that maximize the expected return calculated from the rewards of one or more target sensors identified in the input of the camera policy neural network. As described above, the return is the sum or time-discounted sum of future rewards received. For example, at time step The returns can satisfy: in In the round All subsequent time steps or in the episode The range changes for some fixed number of time steps thereafter, is a discount factor greater than zero and less than or equal to one, and is the reward at time step i. The discount factor used to compute the reward for training the camera policy neural network can be the same or different from the discount factor used for the sensor prediction neural network.

[0130] Since the reward of the target sensor depends on the accuracy of the predictions generated based on the images captured by the neural network, as a result of training, the system trains the camera policy neural network to generate images that accurately fit the target sensor at a location within the camera's viewpoint that allows the predictions to be accurately generated.

[0131] For example, the system can train a camera policy neural network jointly with a camera critique neural network. Thus, the system uses rewards to train both the camera policy neural network and the camera critique neural network.

[0132] As described above, a camera evaluation neural network is a neural network that receives inputs including observations including one or more images captured when a camera sensor is in a particular position and a camera control action generated for a target sensor, and generates as output an evaluation output that defines a predicted reward for the target sensor if the camera control action is performed when the camera sensor is in the particular position. For example, the evaluation output can be a regression reward value, or can specify parameters of a distribution over possible rewards. Examples of types of distribution outputs are described above with reference to sensor prediction neural networks.

[0133] The system can train the camera policy neural network jointly with the camera critic neural network using any suitable actor-critic reinforcement learning technique, such as a deterministic policy gradient-based technique or a distributional deterministic policy gradient-based technique. Some examples of such techniques are described in Distributed Distributional Deterministic Policy Gradients, Barth-Maron, et al, arXiv:1804.08617.

[0134] Figure 3 is a flow chart of an example process 300 for training a sensor prediction neural network. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a computer program such as a computer program program that is appropriately programmed according to the present specification Figure 1 The training system 100 of the training system can perform process 200.

[0135] Typically, the system can generate training data for training the sensor prediction neural network during the same control round performed to generate training data for the camera policy neural network. Thus, the system can train the sensor prediction neural network using tuples generated for training the camera policy neural network.

[0136] The system obtains training tuples (step 302 ), eg, the same tuples that were sampled for training a camera policy neural network.

[0137] The system processes the image in the first observation in the tuple using a sensor prediction neural network to generate a corresponding prediction for each of a plurality of sensors of the robot (step 304 ).

[0138] As described above, the prediction for a given sensor may be any of a variety of different predictions that characterize the current or future state of the sensor.

[0139] In some implementations, the corresponding prediction is a prediction of the value of the sensor reading of the sensor at the time step at which a last image in the one or more images in the input was generated.

[0140] In some other embodiments, the corresponding prediction is a prediction of the value of the sensor reading of the sensor at the next time step immediately following the time step at which a last image in the one or more images in the input was generated.

[0141] In some other implementations, the corresponding prediction is a prediction of a reward generated from a value of at least the sensor reading of the target sensor at each of one or more time steps after the time step at which the second observation was generated.

[0142] The system obtains the corresponding true value of each of the sensors (step 306). For example, when generating the tuple, the system may have stored the true value of the sensor in the replay memory along with the training tuple.

[0143] For example, when the prediction is a prediction of the value of a sensor reading of a sensor at a time step at which a last image in one or more images in the input was generated, the system obtains the actual value of the sensor reading at that time step.

[0144] When the corresponding prediction is a prediction of a value of a sensor reading at a next time step or a prediction of a reward, the system obtains an actual value of the sensor reading at the next time step.

[0145] The system trains the sensor prediction neural network using the true values ​​(step 308 ).

[0146] In particular, the system trains the neural network to minimize the error in the predictions generated by the neural network.

[0147] For example, when the neural network predicts a current value or a next value, the system can train the neural network using a regression loss, such as a mean squared error loss, which measures, for each sensor and for each training pair, the error between the prediction of the observations in the training pair and the true value of the sensor reading in the training pair.

[0148] When the neural network predicts a reward, the system can train the neural network on the training tuple by minimizing a loss that is a combination, e.g., a sum, of the temporal difference learning losses (or distributional temporal difference learning losses) of the sensors. Examples of temporal difference learning losses and distributional temporal difference learning losses are described in more detail in, e.g., Playing Atari with Deep Reinforcement Learning, Mnih, et al, arXiv:1312.5602, Distributed Distributional Deterministic Policy Gradients, Barth-Maron, et al, arXiv:1804.08617, etc.

[0149] Figure 2B and Figure 3 The training of the sensor prediction neural network 120 and the camera policy neural network 110 is described when these neural networks are trained "off-policy", for example, on training tuples sampled from memory. In some other embodiments, one or both of the neural networks can be trained on-policy, for example, such that the training tuples are used directly to train the neural network instead of being sampled from memory.

[0150] Figure 4 An example of training of the sensor prediction neural network 120 and the camera policy neural network 110 is shown.

[0151] like Figure 4As shown in the example of , when the sensor prediction neural network 120 generates an output specifying a reward calculated from the value of the sensor reading, for any given training tuple, the system can use one of the temporal difference (TD) losses calculated for the training of the sensor prediction neural network 120 on the training tuple to generate a reward for the training of the camera policy neural network 110 and the camera evaluation neural network 150.

[0152] Specifically, the system can calculate a corresponding TD loss for each of the sensors using the predictions and true value sensor values ​​of the sensor's sensor prediction neural network 120. The system can then select the TD loss of a target sensor, i.e., a sensor included in the input of the camera policy neural network 110 when generating a given training tuple, and use the TD loss of the target sensor to generate a reward, e.g., by setting the reward equal to the negative of the TD loss.

[0153] exist Figure 4 In the example of , the sensor prediction neural network 120 is configured to generate an output that specifies a distribution over possible rewards calculated from the values ​​of the sensor readings. Additionally, the camera policy neural network 110 is trained jointly with the camera evaluation neural network 150, which generates an evaluation output that specifies a distribution over possible rewards calculated from the rewards.

[0154] Specifically, Figure 4 shows the training of a neural network on a tuple specifying the first observation Second observation And in response to the first observation The camera control actions performed , and the robot The sensor at the time step The corresponding actual sensor readings .

[0155] like Figure 4 As shown, for For each sensor in the system, the system The actual sensor reading of the sensor at , time step The distribution of sensors at, the discount factor and time step for sensor prediction training The distribution of sensors at is used to calculate the corresponding distributed time difference (TD) loss of each sensor.

[0156] The system then The corresponding distributed TD losses of the sensors are summed to generate a combined loss, and the combined loss is used to train the sensor prediction neural network 120.

[0157] The system selects the negative value of the distribution TD loss of the target sensor of the episode during which the first observation and the second observation are received as the reward of the camera control policy neural network 110 .

[0158] The system then uses the reward To train the camera policy neural network 110 and the camera evaluation neural network 150.

[0159] Specifically, the system uses rewards To calculate the distribution TD loss of the evaluation, such as Figure 4 As shown, the camera evaluation neural network 150 is trained using the evaluation loss.

[0160] The system also uses the camera-critic neural network 150 to train the camera-policy neural network 110 as part of an actor-critic reinforcement learning technique.

[0161] Table 1 shows the results of the described technique ("ours") in terms of prediction accuracy of the sensor prediction neural network for various sensors after training (lower is better). Table 1

[0162] As can be seen from Table 1, the described technique performs significantly better than a "Blind" strategy where the sensor prediction neural network cannot see the sensor and a random strategy where the location of the camera sensor is randomly selected using both the regular ("c") camera sensor and the foveated camera sensor ("f"). In addition, the described technique is comparable to an "oracle" technique designed with visibility of the target sensor.

[0163] This specification uses the term "configured" in connection with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0164] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, for example, one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical signal, optical signal, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0165] The term "data processing apparatus" refers to data processing hardware and covers all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to the hardware, the apparatus may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0166] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files storing one or more modules, subroutines, or code locals. A computer program may be deployed to execute on one or more computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0167] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same computer or computers.

[0168] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit such as an FPGA or ASIC, or by a combination of a special purpose logic circuit and one or more programmed computers.

[0169] Computers suitable for executing computer programs can be based on general or special microprocessors or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by or incorporated into a special logic circuit. Typically, the computer will also include or be operably coupled to receive data from or transmit data to or both of one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0170] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0171] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device, such as a smart phone running a messaging application, and receiving a response message from the user in return.

[0172] The data processing apparatus for implementing a machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production, such as inference, workloads.

[0173] Machine learning models can be implemented and deployed using a machine learning framework such as the TensorFlow framework or the Jax framework.

[0174] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, for example, as a data server, or includes a middleware component, for example, an application server, or includes a front-end component, for example, a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0175] The computing system may include a client and a server. The client and the server are usually remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to a user device, for example, to display data to a user of the device interacting with the client and to receive user input therefrom. Data generated at the user device, such as the result of a user interaction, may be received from the device at the server.

[0176] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or on the scope that can be claimed, but rather as descriptions of features that can be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination. In addition, although features may be described above as functioning in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve a sub-combination or a variation of the sub-combination.

[0177] Similarly, although operations are depicted in a particular order in the drawings and recited in the claims, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0178] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for training a camera policy neural network for controlling the position of a camera sensor in an environment with which a robot is interacting, the method comprising: obtaining data specifying one or more target sensors of the robot; obtaining a first observation comprising one or more images of the environment captured by the camera sensor while in a current position; processing a camera policy input using the camera policy neural network, the camera policy input comprising (i) the data specifying one or more target sensors of the robot and (ii) the first observation comprising one or more images captured by the camera sensor to generate a camera policy output defining a camera control action for adjusting the position of the camera sensor; adjusting the current position of the camera sensor based on the camera control action; obtaining a second observation comprising one or more images of the environment captured by the camera sensor while in the adjusted position; generating, based on the second observation, a respective prediction for each of the one or more target sensors that characterizes a sensor reading generated by the target sensor; generating, for each target sensor, a corresponding reward for the camera policy neural network based on an error in the corresponding prediction for the target sensor; as well as The camera policy neural network is trained using the rewards of the one or more target sensors.

2. The method according to claim 1, wherein: The camera sensor is part of the robot.

3. The method according to claim 1, wherein: The camera sensor is external to the robot within the environment.

4. A method according to any preceding claim, wherein: The camera sensor is a foveated camera.

5. The method according to claim 4, wherein: The fovea camera includes a plurality of cameras having different fields of view.

6. A method according to any preceding claim, wherein: The corresponding prediction is a prediction of the value of the sensor reading of the target sensor at the time step at which the second observation was generated.

7. The method according to any one of claims 1 to 5, wherein: The respective prediction is a prediction of a reward generated from a value of at least a sensor reading of the target sensor at each of one or more time steps after the time step at which the second observation was generated.

8. A method according to any preceding claim, wherein: Generating a respective prediction for each of the one or more target sensors representing a sensor reading generated by the target sensor based on the second observation includes: A predictor input including the second observation is processed using a sensor prediction neural network to generate a predictor output including the corresponding prediction for each of the one or more target sensors.

9. The method according to claim 8, further comprising: The sensor prediction neural network is trained using the errors in the corresponding predictions of the one or more target sensors.

10. The method according to claim 9, wherein: The robot comprises a plurality of sensors, the plurality of sensors comprising the one or more target sensors, The predictor output includes a respective prediction for each of the plurality of sensors, and Training the sensor prediction neural network includes using an error in the corresponding prediction of each sensor of the plurality of sensors to train the sensor prediction neural network.

11. A method according to any preceding claim, wherein: The target sensor includes one or more proprioceptive sensors of the robot.

12. A method according to any preceding claim, wherein: The action specifies a target velocity for each of one or more actuators of the camera sensor.

13. A method according to any preceding claim, wherein: Training the camera policy neural network using the rewards of the one or more target sensors includes training the camera policy neural network through reinforcement learning.

14. A method according to any preceding claim, wherein: Training the camera policy neural network through reinforcement learning includes training the camera policy neural network jointly with a camera evaluation neural network.

15. A method according to any preceding claim, wherein the robot further comprises one or more controllable elements.

16. The method according to claim 15, wherein: Each of the controllable elements is controlled using a corresponding fixed strategy during the training of the camera policy neural network.

17. The method according to claim 15, wherein: During training of the camera policy neural network, each of the controllable elements may be controlled using a robotic policy neural network that receives input including one or more images generated by the camera sensor.

18. The method according to claim 17, wherein: The robot policy neural network is trained for external rewards for a specified task during the training of the camera policy neural network.

19. The method according to claim 18, wherein: The training of the camera policy neural network is performed as an auxiliary task during the training of the robot policy neural network.

20. The method according to any one of claims 15 to 18, further comprising: After the training of the camera policy neural network: The trained camera policy neural network is used to train a robotic policy neural network that receives input including one or more images generated by the camera sensor to control each of the one or more controllable elements using external rewards for one or more specified tasks.

21. The method according to claim 20, wherein: Using the trained camera policy neural network to train a robotic policy neural network that receives inputs including one or more images generated by the camera sensor to control each of the one or more controllable elements using external rewards for one or more specified tasks includes: The trained camera policy neural network is used to generate training data for the training of the robot policy neural network.

22. The method according to any one of claims 15 to 21, wherein: The one or more controllable elements include one or more manipulators.

23. A system comprising: one or more computers; as well as One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 22.

24. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 22.

Images