Viewpoint invariant visual servoing of robot end effector using recurrent neural network

Abstract

Training and / or using a recurrent neural network model for visual servoing of an end effector of a robot. In visual servoing, the model can be utilized to generate, at each of a plurality of time steps, an action prediction that represents a prediction of how the end effector should be moved to cause the end effector to move toward a target object. The model can be viewpoint invariant in that it can be utilized across a variety of robots having vision components at a variety of viewpoints and / or can be utilized for a single robot even when a viewpoint, of a vision component of the robot, is drastically altered. Moreover, the model can be trained based on a large quantity of simulated data that is based on simulator(s) performing simulated episode(s) in view of the model. One or more portions of the model can be further trained based on a relatively smaller quantity of real training data.

Description

Background technique [0001] Many robots are configured to utilize one or more end effectors to perform one or more robotic tasks, such as grasping and / or other manipulation tasks. For example, a robot may utilize a grasping end effector, such as an "impact" gripper or an "invasive" gripper (e.g., using pins, needles, etc. to physically penetrate an object) to pick from a first location object, moves the object to a second location, and drops the object at the second location. Some other examples of robotic end effectors that can grasp objects include "astrictive" end effectors (e.g., using suction or vacuum to pick up objects) and "contigutive" end effectors (e.g., using surface tension, freezing or adhesives to pick up objects). [0002] Various machine learning-based approaches have been proposed for robotic manipulation tasks such as grasping. Some of these methods train a machine learning model (e.g., a feed-forward deep neural network) to generate one or more predictio...

AI Technical Summary

Problems solved by technology

Although such machine learning models are suitable for robots that capture images from the same or similar viewpoints, they may be inaccurate and / or fail when used in robots that capture images from different viewpoints

[0004] As another example of a shortcoming, various approaches rely heavily or exclusively on training examples generated based on data from real-world physical robots, which requires extensive use of physical robots when attempting robotic grasping or other manipulations

This can be time-consuming (e.g. it takes a lot of time to actually try a lot of grasping), can be resource-intensive (e.g. the electricity required to operate the robot), can cause wear and tear on the robot being used, and / or can require a large human intervention (e.g., placing objects to grasp, correcting error conditions)

Images