Object-centric prediction and control for human to robot skill transfer

Abstract

A system for training a robotic device to perform a task may include a glove to be worn by a user; a sensor array coupled to the glove, a camera to generate images of an object; and a processor configured to execute instructions stored in memory. The processor can receive contact measurements from the sensor array, including tactile signals determined from tactile sensors and positions determined from motion sensors, and an object pose of the object based on an image from the camera, a fusion of images, motion sensing, and / or the contact measurements. The processor can train a machine learning model, based on training data including the contact measurements and the object pose, to generate a prediction of future contact measurements and a future object pose. Other aspects are also described and claimed.

Images