Controller for interactive classification and recognition of objects

Abstract

A controller for interactive classification and recognition of objects in a scene using haptic feedback is provided. The controller includes an interface configured to transmit and receive control signals from a robot arm, gripper signals from a gripper attached to the robot arm, and haptic signals from a sensor attached to the gripper and at least one vision sensor; a memory module for storing a robot control program and a classifier and recognition model; and a processor configured to generate control signals based on the control program and a gripping pose relative to the object to control the robot arm to grip the object with the gripper. The processor is further configured to calculate a haptic feature representation from the haptic sensor signals and to iterate between grasping the object at a set of gripping poses and calculating the haptic feature representation, after which the processor processes the collection of haptic features to learn a model utilized to classify or recognize the object as known or unknown.

Description

[Technical field]

[0001] The present invention generally relates to a controller and method for an interactive robotic haptic perception method for classification and recognition of new object instances. [Background technology]

[0002] Robotic manipulation has evolved over the last few years from simple pick-and-place tasks, where the robot's environment is predominantly well-structured, to deft manipulation, where neither the objects nor their poses are known a priori to the robotic system. Structured pick-and-place tasks benefit from an artificially reduced task complexity and thus require minimal, if any, sensing for the grasping action. Deft manipulation, on the other hand, must rely more heavily on sensing to locate, identify and track graspable objects, as well as to confirm the success of the grasping attempt, and to autonomously plan the grasp. Typically, robotic manipulation systems rely on "old-fashioned" machine vision, e.g., depth cameras, LiDAR, or color cameras, that require line-of-sight with the environment. Although some of the inherent vision problems can be mitigated by using multiple viewpoints, in-wrist camera systems, and visual servoing, the final stage of grasping, i.e., physical contact, remains blind and open-loop. More importantly, estimating the state of the object after grasping and during manipulation is very challenging (e.g., due to material properties).

[0003] Many previous studies have investigated haptic recognition problems, especially those related to supervised learning. Some examples of such haptic recognition problems include the recognition of object instances, surface texture information, and rigidity. Previous studies have focused on recognizing object instances when the number / type of object classes is known a priori. In contrast, in this work, we aim to recognize novel object instances using haptic manipulation in settings where the robot does not have a priori information about the number of classes and associated object labels. Our work helps address the questions of whether interaction with touch can provide important information about novel object identities and whether global shape and appearance properties can be approximated with such information.

[0004] Several previous studies have investigated supervised learning with training datasets for object category classification using tactile sensing. Spiers et al. proposed a gripper hardware including a simple two-fingered underactuated hand equipped with a TakkTile pressure sensor for object classification. They use a random forest (RF) classifier to learn to recognize object instances based on training data for a set of objects. Schneider et al. used an approach that operates on low-resolution intensity images obtained by contact sensing to identify objects using touch sensors placed on the fingertips of a manipulation robot. Such tactile observations, like our work, are generally only partial and local perspectives. They adapt a bag-of-words framework to perform classification using local tactile images as features and create a feature vocabulary for tactile observations using k-means clustering. Drimus et al. proposed a new tactile array sensor based on flexible piezoresistive rubber and presented an approach for classification of several household objects. They represent the array of tactile information as a time series of features for a k-nearest neighbor classifier using dynamic time warping and calculate the distance between different time series.

[0005] More recently, deep learning-based approaches have also been proposed to recognize object instances using touch and vision. Lin et al. proposed a convolutional neural network (CNN) for cross-modality instance recognition that, given visual and tactile observations, recognizes whether these observations correspond to the same object. In their work, they use two GelSight sensors mounted on the fingers of a parallel-jaw gripper. The GelSight tactile sensor provides high-resolution image observations that can detect fine surface features and material details using a deformable gel mounted above a camera in the sensor. Although their approach does not require specific class labels during training, it still requires a large dataset for training, as with all deep learning-based methods. Researchers have also proposed supervised methods to infer object properties from touch. For example, Yuan et al. proposed to estimate the stiffness of an object using a convolutional neural network and a GelSight tactile sensor.

[0006] Beyond recognition problems, tactile sensing has been utilized to improve robotic manipulation and grasping. Calandra et al. proposed a multimodal sensing framework that combines vision and touch to determine the usefulness of touch sensing in predicting grasping outcomes. They use a deep neural network (DNN) that uses inputs from RGB images from a front-facing camera and gel-site sensors to predict whether a grasp will be successful. Hogan et al. proposed a novel regrasp control policy that utilizes tactile sensing to improve grasping using local grasp adjustments. In the next section, we describe the tactile sensing hardware and tactile data generation. Summary of the Invention [Problem to be solved by the invention]

[0007] This disclosure provides a new approach for classification of unencountered object instances from interactive haptic feedback. Our proposed embodiment interactively learns a one-class classification model using 3D haptic descriptors, thus demonstrating advantages over existing approaches that require pre-training on the object. We explain how we derive 3D features from tactile sensor inputs and utilize them to learn a one-class classifier. In addition, since our proposed method uses unsupervised learning, we do not require ground truth labels. This makes our proposed method flexible and more practical for deployment in robotic systems. Furthermore, embodiments of our proposed method demonstrate the usefulness of low-resolution tactile sensor arrays for tactile perception, which may potentially bridge the gap between vision and physical contact for manipulation. The proposed method can also utilize high-resolution camera-based tactile sensors.

[0008] This disclosure proposes a method for classifying new objects based on haptic feedback without the need for pre-training and ground truth labels for supervision. An embodiment of our proposed method uses a one-class SVM to fit a set of features derived from a grasp pressure map obtained from interactive haptic manipulation on an object, and then classifies instances by interacting with the object. [Means for solving the problem]

[0009] According to some embodiments of the present invention, there is provided a controller for interactive classification and recognition of objects in a scene using haptic feedback. The controller controls sensor signals from a robot arm, gripper signals from a gripper attached to the robot arm, and a controller for detecting a target object in a scene using haptic feedback. Touch and an interface configured to transmit and receive tactile signals from the sensor and the at least one visual sensor. The gripper includes fingers, and the tactile sensors are attached to either side of at least one finger of the gripper. The controller further includes:The present invention includes a memory module for storing a robot control program and a classifier and a recognition model, and a processor configured to generate a control signal based on the control program and a gripping posture with respect to an object, and configured to control the robot arm to grip the object with a gripper. nothing The processor is further configured to compute a tactile feature representation from the tactile sensor signals. The processor may further comprise a processor for computing a tactile feature representation from the tactile sensor signals. By touching or grasping Computing tactile feature representations I will The processor is configured to process the population of tactile features to learn a model that is utilized to classify or recognize objects as known or unknown.

[0010] The accompanying drawings, which are included to provide a further understanding of the invention, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 shows the experimental setup for our interactive tactile-based classification and recognition. [Diagram 2] FIG. 2 illustrates an example of a robot used in a preferred embodiment. [Figure 3A] FIG. 1 is a diagram showing an example of a two-fingered robot gripper. [Figure 3B] FIG. 2 shows a close-up of the right finger of the robotic gripper along with a portion of the gripper housing. [Figure 4] FIG. 1 illustrates an example of a system using a robotic arm. [Diagram 5] FIG. 2 illustrates an exemplary layout of a tactile sensor. [Figure 6] FIG. 1 is a block diagram of a haptic-based interactive classification and recognition system according to some embodiments. [Figure 7] FIG. 1 is a block diagram of a tactile-based classification and recognition process. [Figure 8A]FIG. 1 shows an example of candidate gripping poses colored according to several metrics. [Figure 8B] FIG. 13 shows an example of candidate grasping poses colored according to some graspability metric. [Figure 8C] FIG. 13 shows an example of candidate grasping poses colored according to some graspability metric. [Figure 9] 1 shows a block diagram of a haptic feature computation process. [Figure 10A] FIG. 1 is a diagram of the one-class classification fitting and storing process during training. [Figure 10B] FIG. 1 is a diagram of a process for manipulating an object using a tactile sensor and utilizing one-class classification during operation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Various embodiments of the present invention are described below with reference to the drawings. It should be noted that the drawings are not drawn to scale, and elements of similar structure or function are represented by the same numerals throughout the drawings. Furthermore, the drawings are merely intended to facilitate the description of specific embodiments of the present invention. The drawings are not intended as an exhaustive description of the present invention or as limitations on the scope of the present invention. In addition, aspects described in connection with a particular embodiment of the present invention are not necessarily limited to that embodiment, and may be practiced in any other embodiment of the present invention.

[0013] Robotic manipulation has evolved over the last few years from simple pick-and-place tasks, where the robot's environment is predominantly well-structured, to deft manipulation, where neither the objects nor their poses are known a priori to the robotic system. In some cases, taxels, short for tactile element, are similar to pixels. Structured pick-and-place tasks take advantage of an artificially reduced task complexity and thus require minimal, if any, sensing for the grasping action. Deft manipulation, on the other hand, must rely more heavily on sensing to locate, identify, and track graspable objects, as well as to confirm the success of the grasping attempt, and to autonomously plan the grasp.

[0014] Typically, robotic manipulation systems rely on "old-fashioned" machine vision, e.g., depth cameras, LiDAR, or color cameras, that require line-of-sight with the environment. Although some of the inherent vision problems can be mitigated by using multiple viewpoints, in-wrist camera systems, and visual servoing, the final stage of grasping, i.e., physical contact, remains open-loop with no line-of-sight. More importantly, estimating the state of the object after grasping and during manipulation is very challenging (e.g., due to material properties).

[0015] Objects that may appear similar to advanced vision systems may be completely different in terms of their material properties. Haptic feedback can bridge the gap between vision and physical manipulation. There have been recent advances in haptic manipulation, and state-of-the-art approaches use vision-based haptic feedback using deformable gels mounted above a camera that provides high-resolution imagery observations of the grasped object. Such sensors, while effective, are typically bulky and may cause computational overhead during processing of high-resolution images. In this work, inspired by the human fingertip, which can amount to approximately 100 taxels (tactile sensor cells) per square centimeter, we propose the use of a low-resolution haptic device based on a pneumatic MEMS device (Micro Electro-Mechanical System).

[0016] Object classification is an important task for robotic systems. Vision-based approaches require pre-training on a set of objects known a priori for classification. We instead propose to perform classification of novel objects based on interactive tactile perception using unsupervised learning without pre-training. This could make robotic systems more practical and flexible. The contributions described in the attached academic paper can be summarized as follows: Using commercially available pneumatic MEMS devices, we present the construction and integration of a low-cost and low-resolution tactile sensor array for robotic grasping and manipulation; · We present a meaningful 3D representation of the local shape of an object using the proposed low-resolution tactile sensing; We propose an unsupervised machine learning approach to classify novel objects, adapting a model to the tactile representations obtained through interaction without the need for pre-training and labelling ground truth across a training set of objects.

[0017] Figure 1 shows the experimental setup for our interactive tactile-based classification and recognition: a 7-DoF (Degrees of Freedom) robotic arm equipped with a parallel-jaw gripper (enlarged insert) equipped with a pneumatic tactile sensor.

[0018] FIG. 2 illustrates an example of a robot used in the preferred embodiment. The robot arm 200 is composed of a set of rigid links 211, 213, 215, 217 connected to each other by a set of joints 210, 212, 214, 216, 218. In the described embodiment, the joints 210, 212, 214, 216, 218 are revolute joints, but in alternative embodiments, they may be sliding joints or other types of joints. The collection of joints determines the degrees of freedom for the robot arm 200. The robot arm 200 has five degrees of freedom, one for each of the joints 210, 212, 214, 216, 218. These joints have embedded sensors that can report the state of the joints. The reported state may be angle, current, velocity, torque, acceleration, or any combination thereof. The robot arm 200 has a gripper 300 attached. The gripper 300 is described in detail in the description of FIG. 3A. The robotic arm 200 and gripper 300 can grasp and manipulate objects 220. These objects are typically located on a work surface 230. A collection of objects 240 may be present on the work surface. These objects may vary in shape, size, or both. These objects may be separated or may overlap each other. If the collection of objects 240 is not separated according to some separation method, the collection of objects 240 is called cluttered. The robotic arm 200 is also often called a manipulator arm.

[0019] FIG. 3A illustrates an example of a two-fingered robotic gripper 300. The robotic gripper 300 has a mount 310 for mounting the gripper to the robotic arm 200. The mount 310 typically routes power, control and sensor cables between the gripper housing 320, the fingers 330, 340, and the tactile sensor 360. The robotic gripper 300 is further comprised of a motor and housing 320 for controlling the left finger 330 and the right finger 340. In a preferred embodiment, the motor controls the sliding of the fingers 330 and 340. By sliding the fingers towards each other, the gripper is said to close and grip the object 220. By sliding the fingers away from each other, the gripper is said to open and release the object 220 from its grip. In one embodiment, the left finger 330 may be the same as the right finger 340. In another embodiment, the fingers may differ in size, shape, and actuation, or any combination thereof. In the preferred embodiment, we attach an elastic polymer 350 to the fingers 330 and 340. A tactile sensor 360 is attached to the elastic polymer 350. It is understood that a two-fingered robotic gripper is just one example of a gripper in the preferred embodiment. Alternative embodiments may use a three-finger actuated gripper. Yet other embodiments may use a five-fingered fully actuated hand. It is understood that in other embodiments, the tactile sensor may be attached to the fingers and other portions of the gripper.

[0020] 3B shows a close-up of the right finger 340 of the robotic gripper 300 along with a portion of the gripper housing 320. In this embodiment, an elastic polymer 350 is attached to each side of the finger 340. A tactile sensor 360 is also attached to the elastic polymer 350 on each side of the finger 340.

[0021] 4 is a diagram illustrating an example of a control system using a robotic arm. In the following, the control system is described as an example applied to a robotic system 400, but the control system and computer-implemented method according to the present invention are not limited to robotic systems. The robotic arm 200 receives commands or tasks that may be provided externally to the system 460, a robotic control system 460 The robotic system is controlled using a policy 430. An example of a command or task may be to touch or grasp an object 220 using the gripper 300 of the robot arm. The robot control system 460 sends control signals 470 to the manipulator. The control signals 470 may be torques applied at each of the joints 210, 212, 214, 216, 218 of the robot arm, and opening / closing of the gripper 300. The state 415 of the robotic system is derived using sensors. These sensors may include encoders at the robot's joints 210, 212, 214, 216, 218, a camera 410 that can observe the robot's environment, and a tactile sensor 360 that may be attached to the fingers 330, 340 of the gripper 300. The state measurements 415 from the sensors are sent to a data input / output unit 420 that stores the data received from the sensors. The robot control system 460 is told to execute the policy 430 to accomplish some task or command. The program 440 receives input from the data input / output unit 420 and determines updates to the control policy using a controller update system 450. The controller update system 450 then sends the updated policy 430 to the robot control system 460. The policy 430 and the robot control system 460 may also control the amount of opening or closing of the gripper fingers 330 and 340. The amount of closure, i.e., the grip strength, may be determined by the control policy 430. In a preferred embodiment, the grip strength is determined from the signal of the tactile sensor 360, which is part of the state measurements 415 from the sensor.

[0022] In a preferred embodiment, the camera 410 is an RGBD camera capable of providing both RGB color images and depth images. The internal information from the RGBD camera can be used to convert the depth to a 3D point cloud. In another embodiment, the camera can be a stereo camera composed of two color cameras from which the depth and 3D point cloud can be calculated. In yet another embodiment, the camera can be a single RGB camera and the 3D point cloud can be directly estimated using machine learning. In another embodiment, there can be more than one camera 410. Finally, in another embodiment, the camera 410 can be mounted at any point on the robot arm 200 or the gripper 300. Tactile Sensing Hardware and Data Generation

[0023] Figure 5 shows an example layout of the tactile sensors. In a preferred embodiment, our tactile sensing hardware 360 ​​consists of four closely packed arrays of tactile sensor strips arranged as the inner and outer touchpads of a two-fingered parallel-jaw gripper. In this embodiment, the arrays are 2 It is composed of four TakkStrip2 devices (RightHand Robotics, Inc.) connected to the C-Bus. Our tactile array is composed of 48 taxels arranged in a 4x6 array with a dot pitch of approximately 7.5 mm. Note that the size and arrays used in the embodiments are not limited. The shape, sensors, and number of sensors per array can be modified according to the design of the gripper 300.

[0024] The Tactile Sensor is a series of MEMS pressure I / O modules molded in a soft elastomer and packaged as a strip of six tactile sensor cells (tactile sensor cells). 2For this purpose, we use MEMS pressure sensors. The main advantage of these devices is that they provide analog signal conditioning, temperature compensation, and analog to digital conversion (ADC) all on-chip. In contrast to other tactile sensing technologies, barometric pressure sensors directly read tactile pressure and temperature inputs, and thus more closely resemble human touch sensing. Also, compared to vision-based touch sensing alternatives, MEMS pressure sensors communicate over a significantly lower bandwidth, while allowing for more flexible spatial placement of the touch sensor (i.e., not limited to flat touchpads).

[0025] Each finger 330, 340 of the gripper is fitted with eight tactile strips. They are divided into two groups, one for external gripping and the other for internal gripping, for a total of 48 taxels per finger. For convenience, the touchpad is flat, but this is not a design limitation. In fact, each sensor cell can be addressed separately with minimal hardware modifications, while the device footprint can be further reduced by using an equivalent MEMS pneumatic device. The current iteration of the touch-sensing array used in our experiments has dimensions of 30 x 45 mm and contains 4 x 6 taxels (so the dot pitch is 7.5 mm).

[0026] All devices are connected to a single I 2 The master communicates over a standard I / O bus. Data collisions and other transmission safety is handled "on the strip" by the traffic controller. The traffic controller is the master I / O bus. 2 When addressed by the I controller, it launches each connected device in a loop that triggers each device to load its pressure data onto the bus (more information on Tactile's communication protocol can be found here). 2Using a C-USB device interface, the sensors are connected to a Raspberry Pi 4, which acts as a physical ROS node exposing the raw tactile data to our ROS-enabled robot controller. With this setup, we achieve a sampling rate of 64Hz with all 96 taxels connected.

[0027] It should be noted that while these barometric pressure sensors are the preferred embodiment for our tactile sensing hardware, tactile sensing equipment is not limited to barometric pressure arrays, but extends to piezoelectric, capacitive, and fiducial devices, including image-based tactile sensing. Tactile-Based Interactive Classification and Recognition

[0028] FIG. 6 illustrates a block diagram of a haptic-based interactive classification and recognition system according to some embodiments. The haptic-based interactive classification and recognition system 600 is configured to generate haptic features to classify an object as encountered / known or unencountered / unknown, or to recognize an object, i.e., to what object category it belongs, according to some embodiments. We refer to the terms "encountered" or "known" and "unencountered" or "unknown" to indicate whether the system has previously interacted with the object. The system 600 includes a processor 620 configured to execute stored instructions and a memory 640 that stores instructions executable by the processor. The processor 620 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 640 may include a random access memory (RAM), a read only memory (ROM), a flash memory, or any other suitable memory system. The processor 620 is connected to one or more input and output devices through a bus 605.

[0029] The system 600 is configured to perform haptic feature computations and classify or recognize objects manipulated by the robotic arm 200. The system 600 may include a storage device 630 adapted to store haptic-based classification and recognition 631 and robotic control algorithms 632. The storage device 630 may be implemented using a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof.

[0030] The human-machine interface 610 in the haptic-based interactive classification and recognition system 600 can connect the system to a keyboard 611 and a pointing device 612. The pointing device 612 can include a mouse, a trackball, a touchpad, a joystick, a pointing stick, a stylus, or a touch screen, among others. The system 600 can be linked through a bus 605 to a display interface 660 adapted to connect the system 600 to a display device 665. The display device 665 can include a computer monitor, a camera, a television, a projector, or a mobile device, among others.

[0031] The haptic-based interactive classification and recognition system 600 may also be connected to an imaging interface 670 adapted to connect the system to an imaging device 675 providing RGBD images. In one embodiment, images for haptic feature computation are received from an imaging device. In another embodiment, the imaging device 675 may include a depth camera, a thermal camera, an RGB camera, a computer, a scanner, a mobile device, a webcam, or any combination thereof.

[0032] The network interface controller 650 is adapted to connect the haptic-based interactive classification and recognition system 600 to a network 690 through a bus 605. Through the network 690, robot states can be received through a command / state module 695 and stored in the computer storage system 630 through the bus 605 for storage and / or further processing. Through the network 690, commands can be sent to the robot arm 200 through the command / state module 695. In another embodiment, commands are sent through the bus 605.

[0033] In some embodiments, the haptic-based interactive classification and recognition system 600 is connected to the robot interface 680 through a bus 605. The bus 605 is adapted to connect the haptic-based interactive classification and recognition system 600 to the robot arm 200 operable based on commands derived from the robot control algorithm 632 and the received robot state 695. For example, the robot arm 200 is a system that performs policy execution to interact with objects. In another embodiment, the robot interface 680 is connected to the network 690 via the command / state module 695.

[0034] The main objective of the proposed system is to control the robot arm and gripper to grasp an object on a work surface and then record the haptic signals for the grasp. The haptic signals are processed and used for classification and recognition. The robot arm and gripper are commanded to grasp the object multiple times with different grasping poses. The haptic signals for the different grasping poses are different from each other.

[0035] FIG. 7 shows a block diagram of a haptic-based classification and recognition process. The process 700 for classifying or recognizing an object starts by acquiring an RGBD image with the RGBD camera 410 (710). The RGBD image 710 is processed by a gripping pose detection algorithm to determine (720) candidate gripping poses. The candidate gripping poses 720 may include gripping poses that are not obtainable by the robot arm 200 and gripper 300. From the candidate gripping poses 720, a set of valid gripping poses is determined (730). An example of an invalid gripping pose is a gripping pose where the robot arm 200 and / or gripper 300 would collide with itself or with the work surface 230. Other invalid gripping poses are gripping poses where the inverse kinematics calculation cannot find a solution. In addition, due to limitations on the robot's range of motion, some gripping poses are not obtainable by the robot and gripper and are therefore invalid. Another example of an invalid gripping pose is a gripping pose where the gripping may be stable according to some metric.

[0036] For one such gripping posture selected (740) from the valid gripping postures 730, the robot is controlled (750) to grip the object with the selected gripping posture 740. In a preferred embodiment, the robot is a robot arm 200 with an attached gripper 300 and tactile sensor 360. Once the object is gripped, a tactile signal is recorded (760). If it is determined by checking the desired amount (755) that the desired number of valid gripping postures 730 have not been processed, the process repeats selecting a gripping posture from the valid gripping postures 730 (740) and grips the object with the next selected gripping posture using the robot control 750 and stores the tactile signal (760). Once the desired number of gripping postures and tactile signals have been obtained, the check for the desired amount 755 will direct the process to process (770) the stored tactile signals for all candidate grips. The processed (770) haptic signals are then used to classify or recognize (780) the object with which the robotic arm 200 is interacting.

[0037] 8A, 8B, and 8C show examples of candidate gripping poses colored according to some metrics. FIG. 8A shows candidate gripping poses for a stuffed toy object. The red arrows represent poses that are eliminated due to collisions between the robot arm and the work surface as determined by the robot control algorithm 632. The robot control algorithm 632 further determines that the approach angles of the robot marked as blue or magenta are not feasible. The yellow arrows indicate candidate poses for which the robot control algorithm 632 may not be able to calculate valid inverse kinematics. Finally, the remaining valid gripping poses are marked by green arrows. The gripping poses that the robot arm will execute are selected from a normal distribution of valid gripping poses. In another embodiment, the gripping poses selected for the robot arm may be determined according to some desired coverage of the object's surface. FIG. 8B and FIG. 8C show additional examples for other objects. Raw Signal Processing

[0038] Figure 9 shows our haptic sensing processing pipeline. The robot control for grasping an object for the selected (740) pose provides us with the haptic signal for grasping the object under consideration contained in block 901. The haptic raw data in the preferred embodiment consists of an array of 96 pressure and temperature values, corresponding to each taxel. As a first processing step, the individual pressure values ​​are temperature compensated as described in the sensor manufacturer's data sheet. Despite the relatively low noise (0.01 N) and good linearity (typically less than 1%), the barometer cells exhibit a slow drift after a few hours of use. To compensate for noise drift, we use a simple moving average filter with an arbitrarily chosen window of 30 samples and a uniform weight across all data points. This filter also serves as a measure of the unloaded sensor state. Although not a requirement of our haptic sensing pipeline, we offset the measured steady state for each taxel (given by the moving average filter) so that our filtered readings reflect only haptic load and temperature information. In FIG. 9, this 3D signal conditioning is contained in block 902 . Generate pressure maps

[0039]

number

[0040]

number

[0041] Next, we compute 3D surface descriptors (905) from the 3D pressure maps 903 and associated surface normals 904. The 3D surface descriptors are Viewpoint Feature Histograms (VFHs). Each VFH is a 308-dimensional feature vector. We compute one for each finger of the gripper and store the grasp as a combination of the two VFHs, a 616-dimensional feature vector. The 3D surface descriptors 905 are flattened into vectors and stored on disk or in memory (906). Interactive tactile classification of novel objects

[0042] Our goal is to use haptic feedback to classify objects as previously unencountered or encountered. In recent years, DNNs have achieved good performance for various classification tasks. The networks are trained using supervisory signals, i.e., ground truth class labels, and thus fall under the umbrella of supervised learning methods. In addition, DNNs require a huge amount of training data to achieve good performance. Due to these requirements, using DNNs is not a practical solution to achieve our goal.

[0043] We instead propose to learn online, one object at a time, without the need for pre-training. Object instances previously manipulated by the robot should be classified as such, and new objects should be detected, learned, classified and added to the set of previously manipulated objects. The main motivation behind our approach is data efficiency and active exploration. For practical manipulation tasks, real robotic systems can only “spend” a short time to determine whether an object is new, which implies too few haptic samples for deep learning. Moreover, knowing in advance the span of object shapes and material properties, i.e. the tactile range, would be counter to the purpose of a comprehensive haptic manipulation framework and would only be applicable for regular robotic pick-and-place in structured environments. Learning local tactile representations of novel objects

[0044]

number

[0045]

number

[0046]

number

[0047] If classification 1040 classifies the object as unencountered / unknown, the initial process of OC-SVM fitting is repeated. For the current object, we consider the 3D feature descriptors 905 for all grasps simultaneously and fit an OC-SVM to this data (1010). We then store this OC-SVM as a representation for this current object (1020). This process is repeated for each object that was classified as unencountered / unknown (1040). Sampling Grasps on Objects

[0048]

number

[0049] The tactile features for the selected grasp should essentially form some basis in fitting the OC-SVM. The more uniformly we can sample the object across its surface, the more likely the model will be able to classify it accurately. In the next section, we present an evaluation of our proposed method in two experiments.

[0050] The above-described embodiments of the present invention may be implemented in any of many ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether the processors are provided in a single computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit having one or more processors in an integrated circuit component. However, the processor may be implemented using circuits in any suitable format.

[0051] Additionally, embodiments of the invention may be embodied as a method, examples of which are provided. The acts performed as part of the method may be ordered in any suitable manner. Thus, while an example embodiment may be shown as sequential acts, embodiments may be constructed in which acts are performed in an order different from that illustrated.

[0052] The use of ordinal terms such as "first," "second," etc. in the claims to modify claim elements does not, by itself, imply any priority, precedence, or ordering of a claim element over another claim element, or any chronological order in which method operations are performed, but is merely used as a label to distinguish a claim element having a certain name from another element having the same name (except for the use of the ordinal terminology).

[0053] Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.

Claims

1. 1. A controller for interactive classification and recognition of objects in a scene using haptic feedback, comprising: an interface configured to transmit and receive control signals, sensor signals from a robot arm, gripper signals from a gripper attached to the robot arm, tactile signals from a tactile sensor attached to the gripper, and visual signals from at least one visual sensor; a memory module for storing the robot control program and the classifier and recognition model; a processor configured to generate a control signal based on the control program and a gripping attitude with respect to the object, and configured to control the robot arm to grip the object with the gripper; The processor is further configured to calculate a haptic feature from the haptic signal; the processor is configured to touch or grasp the object in a set of grasping poses to calculate the haptic features; Thereafter, the processor processes the haptic features to learn a model that is utilized to classify or recognize the object as known or unknown; The controller: the gripper including fingers; the tactile sensors are attached to either side of at least one finger of the gripper.

2. The controller of claim 1 , wherein the grasp pose for an object is calculated from at least one visual signal.

3. A controller as described in claim 2, wherein a grasping posture used to control the robotic arm is selected from the set of grasping postures.

4. 4. The controller of claim 3, wherein the selected grip pose is selected based on a normal distribution constructed from the set of grip poses, or the selected grip pose is selected according to a metric of surface coverage of the object.

5. The controller of claim 1 , wherein the shape descriptor is based on a representation of haptic information.

6. the shape descriptor is a 3D surface descriptor based on a 3D representation of haptic information; The 3D surface descriptor may be generating a 3D pressure map based on the tactile signals; Calculating surface normals from the 3D pressure map; calculating a 3D surface descriptor from the 3D pressure map and the surface normals; It is calculated as follows: The controller of claim 5 , wherein the 3D pressure map is calculated by use of a spline surface patch method.

7. The controller of claim 1 , wherein processing the haptic features of a shape descriptor for classification is performed using machine learning.

8. The machine learning model is a one-class support vector machine; or the machine learning model is used to classify instances of an object; The machine learning model is used to classify instances of objects as previously encountered or unencountered; and The machine learning model is updated for instances of objects classified as unencountered, or The controller of claim 7 , wherein the machine learning model is used to recognize an instance of an object.

9. the tactile sensor attached to the gripper includes an array of tactile sensor cells; and the tactile sensor cell is a pneumatic microelectromechanical system device; or The controller of claim 1 , wherein the sensor array measures any surface of the fingers of the robotic gripper.

10. The controller of claim 1 , wherein the tactile sensor attached to the gripper comprises a high resolution camera-based tactile device.

11. The controller of claim 1 , wherein a soft elastomer is attached to the gripper and the attached tactile sensor is disposed in the soft elastomer.

12. The controller of claim 1 , wherein the tactile sensor is configured for temperature compensation and analog-to-digital signal conversion.

13. The controller of claim 1 , wherein the tactile sensor measures tactile pressure upon touching the object.

14. The controller of claim 1 , wherein the tactile sensor is configured to be flexible for placement on a non-planar surface.

15. 2. The controller of claim 1, wherein the interface is a single serial I2C standard communications bus.

Images