ROBOTIC GRIP SLIP DETECTION.

Abstract

A plurality of sensors is configured to generate respective outputs that reflect a detected value associated with the engagement of a robotic arm end effector with an item. The corresponding outputs from one or more of the sensors comprising the plurality are used to determine one or more inputs to a multimodal model configured to generate an output associated with the item sliding into or out of a gripper of the robotic arm end effector, based at least in part on one or more inputs. A determination associated with the item sliding into or out of the gripper of the robotic arm end effector is made based, at least in part, on an output from the multimodal model. A response action is taken based at least in part on the determination associated with the item sliding into or out of the reach of the robotic arm end effector.

Description

ROBOTIC GRIP SLIP DETECTION CROSS REFERENCE TO OTHER APPLICATIONS This application claims priority over the Application of United States Provisional Patent No. 62 / 926,162 entitled ROBOTIC GRIP SLIP DETECTION, filed October 25, 2019, which is included herein by reference for all purposes. 10 BACKGROUND OF THE INVENTION A robotic system may be tasked with grasping and placing items from a first location to a second location. The robotic system may use an end effector to grasp an item at the first location. Once the item has been successfully grasped, the robotic system may move the item to the second location. However, the item may slip off the end effector while being moved from the first to the second location. The item may also be damaged by falling from too great a height. BRIEF DESCRIPTION OF THE DRAWINGS The following detailed description and accompanying drawings disclose several embodiments of the invention. FIG. 1 is a block diagram illustrating a system for picking up and placing items according to certain modalities. FIG. 2 is a block diagram illustrating a robotic arm end effector according to some modalities. FIG. 3A is a block diagram illustrating a touch detection unit according to some modalities. FIG. 3B is a diagram illustrating a touch detection unit from different points of view according to some modalities. FIG. 30 is a diagram illustrating a touch detection unit from different points of view according to some modalities. 2o FIG. 3D is a diagram illustrating an example of a magnetic sensor according to some modalities. FIG. 3E is a diagram illustrating a touch detection unit according to some modalities. FIG. 3F is a diagram illustrating a touch detection unit according to some modalities. FIG. 3G is a diagram illustrating a touch detection unit according to some modalities. Figure 4 is a flowchart illustrating a process for picking and placing an item according to certain procedures. 1(1 FIG. 5 is a flowchart illustrating a process for detecting whether an item slips according to some fashion 1ities. Figures 6A-6C are diagrams illustrating an example of a gripping and placing operation that includes a controlled slide. FIG. 7 is a flowchart illustrating a process for grasping an item according to certain modalities. FIG. 8 is a flowchart illustrating a process for training a multimodal model. DETAILED DESCRIPTION The invention can be implemented in numerous ways, including as a process; apparatus; system; material composition; product of a computer program embodied by a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored and / or entered by a memory coupled to the processor. In this specification, these implementations, or any other form the invention may take, may be referred to as techniques. In general, within the scope of the invention, the order of the steps of the described processes may be modified. Unless otherwise indicated, any component, such as a processor or memory, described as being configured to perform a task, may be implemented as a general component that is temporarily configured to perform the task at a given time or as a specific component manufactured to perform the task.As used herein, the term processor refers to one or more processing devices, circuits and / or cores configured to process data, such as computer program instructions. A detailed description of one or more embodiments of the invention is presented below, along with accompanying figures illustrating its principles. The invention is described in relation to such embodiments but is not limited to any one embodiment. The scope of the invention is limited only by the claims, and the invention encompasses numerous alternatives, modifications, and equivalents. Many specific details are set forth in the following description to enable a complete understanding of the invention. These details are presented by way of example, and the invention can be implemented according to the claims without some or all of these specific details. For the sake of clarity, the technical material known in the technical fields related to the invention has not been described in detail so as not to unnecessarily obscure the invention. Techniques are described for determining whether an item is beginning to slip or is slipping from the grip of a robotic arm end effector. A touch sensing unit is presented in several configurations. The touch sensing unit is used to determine whether an item is beginning to slip or is slipping from a robotic arm. The touch sensing unit includes and / or receives sensor output values ​​generated by one or more sensors. The one or more sensors may include one or more magnetic sensors, optical sensors, electromechanical sensors, pressure sensors, strain gauges, force sensors, conductivity sensors, current sensors, voltage sensors, capacitance sensors, resistance sensors, inductance sensors, infrared sensors, temperature sensors, etc. Each of the one or more sensors is configured to generate an output that reflects a detected value associated with the engagement of the robotic arm's end effector with an item (also called an object). The outputs from one or more sensors are used to determine one or more modalities indicative of the engagement of the robotic arm's end effector with an item. These modalities may include weight, deformation, continuity, conductivity, pressure, resistance, inductance, capacitance, or any other factor indicative of the engagement of the robotic arm's end effector. The touch sensing unit may be associated with a robotic arm end effector. A robotic arm end effector may include two or more fingers. A corresponding touch sensing unit may be attached to each of the fingers. The end effector may include one or more suction cups or other structures for grasping an item, and a corresponding touch sensing unit may be attached to each of the suction cups or other structures used to grasp items. In some embodiments, one or more touch sensing units are included in a sensing cover (e.g., glove, mitten, etc.) that is placed over the fingers of the robotic arm end effector so that each finger has an associated touch sensing unit. In some models, one or more tactile detection units are integrated into the end effector of the robotic arm. The touch sensing unit may include a plurality of sensing layers. Each sensing layer may include one or more sensors. The touch sensing unit may include one or more layers such as a conductive layer, a deformation layer, and a substrate layer. When engaged with an item, a sensor in the touch sensing unit may generate a corresponding detected value that differs from the reference detected value (for example, a tare value when the robotic arm's end effector is not engaged with an item). A detected value from the sensor may change if an item begins to slide or if it is sliding off a robotic arm's end effector. The conductive layer may include conductive material (e.g., metal) that allows for the measurement of continuity or other electrical properties, such as conductivity, resistance, capacitance, or inductance. The conductive layer may be an upper layer of the touch-sensing unit, such that when a first touch-sensing unit attached to a first finger of an end effector contacts a second touch-sensing unit attached to a second finger of the end effector, the conductive layers of the first and second touch-sensing units come into contact. When the robotic arm's end effector grasps an item, the item prevents the conductive layers of the touch-sensing units from coming into contact. However, when the robotic arm's end effector drops the item, the conductive layers of the touch-sensing units can come into contact. The conductive layer is coupled to a processor that can use an output from the conductive layer to determine one or more continuity values, one or more conductivity values, one or more resistance values, one or more capacitance values, and / or one or more inductance values. The processor can monitor the determined values ​​over time as the robotic arm's end effector moves the item from the first location to the second location to determine if the item is beginning to slip and / or is slipping out of the end effector's grip. In some configurations, the deformation layer includes a dielectric material (e.g., silicone, plastic, or any other material capable of deforming in response to a force). The objects being measured can be injected into the dielectric material. Reference sensing values ​​can be determined before the robotic end effector engages an item. When the robotic arm's end effector engages an item, the sensing values ​​are compared to the reference sensing values. The sensing values ​​may differ depending on whether the end effector successfully or unsuccessfully grasped the item and / or whether the item is slipping or about to slip. While the robotic arm's end effector is holding an item, the sensing values ​​may fluctuate.However, when an item slips from the gripper of the robotic arm's end effector, the detected values ​​may fluctuate more than the threshold amount or may fluctuate differently than when the item is not slipping. The processor coupled to one or more sensors associated with the deformation layer monitors the detected values ​​over time as the robotic arm's end effector moves the item from the first location to the second location to determine if the item is beginning to slip and / or if it is slipping from the gripper of the robotic arm's end effector. In some embodiments, a plurality of magnets is injected into the dielectric material. The plurality of magnets may be arranged in a grid pattern (e.g., 2D or 3D grid) or without a grid. One or more magnets may be located at a boundary of the dielectric material. The plurality of magnets is associated with a magnetic sensor located in the substrate layer. The magnetic sensor detects a reference magnetic field before the robotic arm's end effector engages an item. When the robotic arm's end effector engages an item, a force and / or moment associated with the engagement causes one or more of the plurality of magnets to displace. This displacement causes a change in the magnetic field, which is detected by the magnetic sensor. The magnetic sensor is coupled to a processor, which can use the change in the magnetic field to determine a weight and / or strain value. The magnetic sensor detects the magnetic field as the robotic arm's end effector moves the item from the first location to the second to determine if the item is beginning to slip and / or is slipping out of the end effector's gripper. The processor monitors the current magnetic field, weight, and strain values ​​for any changes. A change in the magnetic field associated with a successful grip, the weight associated with a successful grip, and / or the strain associated with a successful grip that exceeds the corresponding threshold may indicate that the item is beginning to slip out of the end effector's gripper.A change in the current magnetic field associated with a successful grip, the current weight value associated with a successful grip, and / or the current strain value associated with a successful grip that exceeds the corresponding second threshold may indicate that the item is slipping out of the robotic arm's end effector grip. In some embodiments, a plurality of markers are embedded in the dielectric material. These markers may be arranged in a grid pattern or without a grid. The grid pattern may be either 2D or 3D. The markers may be associated with an image sensor located in the substrate layer. The image sensor can detect a reference position for all the markers before the robotic arm's end effector engages an item. When the robotic arm's end effector engages an item, a force and / or moment associated with the engagement causes one or more of the markers to shift. This displacement results in a change in the markers' positions, which is detected by the image sensor.The image sensor is coupled to a processor, which can use the change in marker position to determine a marker lighter value and / or a deformation value. The image sensor can detect the position of the markers as the robotic arm's end effector moves the item from the first location to the second location to determine if the item is beginning to slip and / or is slipping out of the end effector's grip. The processor can monitor the current positions, weight, and strain values ​​for any changes. A change in the current marker positions associated with a successful grip, the current weight, and / or the strain value exceeding the corresponding threshold may indicate that the item is beginning to slip out of the end effector's grip.A change in the current positions of the markers associated with a successful grip, the current weight value associated with a successful grip, and / or the current strain value associated with a successful grip that exceeds the corresponding second threshold may indicate that the item is slipping out of the robotic arm's end effector grip. In some configurations, a reflective material is embedded in the dielectric material, and a transmitter and receiver are located in the substrate layer. The transmitter can transmit a signal that is reflected by the reflective material, and the reflected signal is received by the receiver. When the robotic arm's end effector engages an object, a force and / or moment associated with the engagement changes the way the signal travels through the dielectric material. The engagement causes the amplitude of the reflected signal received by the receiver to change from the reference signal amplitude. The receiver is coupled to a processor, which can use the change in signal amplitude to determine an ignition amplitude value and / or a deformation value. The receiver can continue receiving a signal as the robotic arm's end effector moves the item from the first location to the second location to determine if the item is slipping out of the end effector's grip. The processor can monitor the received signal amplitude, weight value, and strain values ​​to detect any changes. A change in the current signal amplitude relative to the amplitude associated with a successful grip, a change in the current weight value relative to the value associated with a successful grip, and / or a change in the current strain value relative to the value associated with a successful grip that exceeds the corresponding first threshold may indicate that the item is beginning to slip out of the end effector's grip.A change in the amplitude of the current signal relative to the amplitude of the signal associated with a successful grip, the current weight value associated with a successful grip, and / or the current strain value associated with a successful grip that exceeds the corresponding second threshold may indicate that the item is slipping out of the robotic arm's end effector grip. In some embodiments, one or more electromechanical sensors capable of detecting deformation (e.g., strain gauge) are integrated into the dielectric material. When the end effector of the robotic arm engages with an item, a force and / or moment associated with the engagement causes a resistance that is associated with a change in one or more electromechanical sensors. One or more electromechanical sensors are coupled to a processor, which can use the change in resistance to determine a weight value and / or a deformation value. One or more electromechanical sensors generate resistance as the robotic arm's end effector moves the item from the first location to the second location to determine if the item is slipping out of the end effector's grip. The processor can monitor the resistance, weight, and strain values ​​to detect any changes. A change in the current resistance value from the resistance value associated with a successful grip, a change in the current weight value from the weight value associated with a successful grip, and / or a change in the current strain value from the strain value associated with a successful grip that exceeds the corresponding first threshold may indicate that the item is beginning to slip out of the end effector's grip.A change in the current resistance value from the resistance value associated with a successful grip, the current weight value from the weight associated with a successful grip, and / or the current strain value from a strain value associated with a successful grip that exceeds the corresponding second threshold may indicate that the item is slipping out of the robotic arm's end effector grip. In some embodiments, the dielectric material may include a flexible membrane (e.g., a sack, a bag) containing a gas, air, or liquid. When the robotic arm's end effector engages an item, a force and / or moment associated with the engagement causes a pressure that corresponds to a change in the flexible membrane. This pressure can be detected by a pressure sensor, and the processor can use the change in pressure to determine a weight and / or strain value. The processor can monitor pressure, weight (value 15), and strain values ​​for changes as the robotic arm's end effector moves the item from the first location to the second location to determine if the item is slipping out of the end effector's grip. A change in the current pressure (value 20) from the pressure value associated with a successful grip, a change in the current weight (value 15) from the weight (value 15) associated with a successful grip, and / or a change in the current strain (value 25) from the strain (value 15) associated with a successful grip that exceeds the corresponding first threshold may indicate that the item is beginning to slip out of the end effector's grip (value 25).A change in the current pressure value from the pressure value associated with a successful grip, the current weight value from the weight associated with a successful grip, and / or the current strain value from a strain value associated with a successful grip that exceeds the corresponding second threshold may indicate that the item is slipping out of the robotic arm's end effector grip. One or more touch-sensing units are coupled to a processor. The robotic arm's end effector may include other sensors that are also coupled to the processor. For example, a housing that connects the robotic arm's end effector fingers to the robotic arm may include one or more sensors used to measure force and / or torque. The housing sensors can be used to filter a detected weight to normalize the robotic arm's end effector weight. The sensor output associated with one or more housing sensors can be used to determine one or more modalities. The plurality of sensors (e.g., touch sensing unit sensor(s) and / or housing sensor(s)) provide their corresponding outputs to the processor. The processor can use the sensor outputs to determine one or more forces and / or one or more moments associated with a coupling between the robotic arm's end effector and an item. The processor can use the sensor outputs to determine the forces and moments associated with each of the touch sensing units. The processor can use the sensor outputs to determine the corresponding values ​​for the plurality of modalities. For example, it can determine a detected weight, a detected strain, a detected continuity, a detected conductivity, a detected pressure, a detected resistance, a detected inductance, and / or a detected capacitance. Each of the plurality of modalities is associated with a coefficient. In some modalities, the plurality of modalities are weighted equally (for example, each has an associated coefficient of 1). In some modalities, some of the plurality of modalities have different weights. For example, the item might be a metallic item, and the coefficient associated with a continuity factor might be less than the coefficient associated with a strain factor. The processor can implement a multimodal model to determine whether an item is beginning to slip and / or is slipping off a robotic arm. The multimodal model can be a rule-based model, a predictive model, a machine learning model (e.g., a neural network, linear classifier, support vector machine, linear regression, logistic regression, decision tree, deep learning, etc.), etc. In some modalities, the multimodal model is configured to output a binary decision regarding whether the item is beginning to slip from the grip of the robotic arm's end effector. In some modalities, the multimodal model is configured to output a binary decision regarding whether the item is slipping from the grip of the robotic arm's end effector.In some modes, the multimodal model is configured to output a probability of whether the item is beginning to slip out of the robotic arm's end effector grip. The values ​​associated with some or all of a plurality of modalities and their corresponding coefficients are sent as input to the multimodal model. Input 15 is applied to the multimodal model, and the robotic system is configured to perform a response action (e.g., feedback control, reactive movement, etc.) based on an output from the multimodal model. In some modalities, the robotic system moves the item from the first location to the second location. In some modalities, the robotic system applies suction force to grasp the item. In some modalities, the robotic system lowers the item and grasps it again. In some modalities, the robotic system requests human intervention. In some modalities, the robotic system adjusts how it grasps the item. For example, the robotic arm can be rotated so that gravity is applied to the grasped item along a different axis. In some configurations, the robotic system proactively adjusts the grasping motion to prevent slippage. The use of tactile sensing units and a multimodal model allows the robotic system to accurately determine whether an item is beginning to slip and / or is slipping off a robotic arm. Multiple sensors can provide data points from different perspectives that confirm the same conclusion: that the robotic arm's end effector is beginning to slip and / or is slipping off. A single-factor approach can bias the robotic system's decision-making process. For example, a robotic system might only consider the continuity between the fingers of the robotic arm's end effector as a factor. A robotic system might not determine that an item is slipping from the end effector's gripper, but the item could be conductive.Using continuity as the sole factor can cause the robotic system to determine that the robotic arm's end effector is still gripping the item when it has actually dropped it. Conversely, using multiple modalities provides the robotic system with a balanced approach to its decision-making process. Figure 1 is a block diagram illustrating a system for picking and placing items according to several embodiments. In the example shown, a robotic system 101 operates in environment 100. The robotic system 101 includes a plurality of articulated segments comprising a robotic arm 102 mounted on a stationary base 104, an end effector 108, one or more sensors 134, and a controller 106. In some embodiments, the stationary base 104 is optional. The robotic arm 102 is coupled to a controller 106 configured to manipulate the robotic arm 102 and an end effector 108 mounted on a distal end of the robotic arm 102. In some embodiments, the controller 106 controls the robotic arm 102 and the end effector 108 by generating voltages and / or other signals, inputs, etc.The motors configured at each of the respective joints between the rigid elements comprising the robotic arm 102 and / or the end effector 108 are configured to apply the corresponding torque(s) to move the element coupled to the rotating element of the motor relative to an element coupled to a non-rotating element of the motor. The end effector 108 may include a suction gripper, a parallel gripper, a soft gripper, a dexterous gripper, etc. The robotic system 101 may include a plurality of end effectors and select the one best suited for grasping the object. For example, an end effector may be selected based on the texture of an object. The robotic system 101 may select a parallel gripper instead of a suction gripper if the object has too many corrugated areas. In the example shown in FIG. 1, the robotic arm 102 is used to pick up items from a table or other surface 110 (e.g., a workspace area), which in the example shown includes the items of different shapes 112, 114, and 116, and place them on a conveyor belt 118 (e.g., a delivery area). As shown, the robotic arm 102 has previously been used to place item 120 onto a conveyor belt 118 that rotates in such a direction that item 120 is about to fall off the conveyor belt 118 into a destination 122. A workspace area may include a moving platform, such as a conveyor belt or a turntable, or a stationary area, on which a stack of items (stable or unstable) is located. 2. In several embodiments, the pick-and-place operation shown in FIG. 1 is performed by the robotic system 101, which includes the robotic arm 102, the end effector 108, and the controller 106, at least partially in an autonomous operating mode. For example, in some embodiments, the controller 106 and / or one or more other control devices, such as a computer comprising a processor, memory, and other components, are programmed to perform the pick-and-place operation illustrated in FIG. 1. For example, in some embodiments, a programmer or other operator may have programmed or otherwise configured the robotic system 101 to be aware of its environment 100 and its position relative to the items on the table 110 (or, in some embodiments, a set of coordinates or other locations are associated with the table 110, on the one hand, and the conveyor belt 118, on the other). In some embodiments, the robotic system 101 is programmed or otherwise configured to use a library or other repository of strategies to perform the pick-and-place operation and / or parts thereof. For example, the robotic system 101 may be configured so that, by knowing its current position and the environment 100, it can place the end effector 108 in a location above the table 110. Computer vision or other techniques may be used to identify and select the item to be picked up next, and a strategy may be selected to pick up the item autonomously, for example, based on the item's location, shape, orientation, appearance, texture, stiffness, etc. For example, in the example shown in FIG. 1, the robotic system 101 may have recognized a feature associated with item 112, such as its cube geometry, and selected a grasping strategy for cube geometries before picking up item 112. The robot may recognize that item 114 has a pyramidal geometry and select a grasping strategy for pyramidal geometries. The robot may recognize that item 116 has a cylindrical geometry and select a grasping strategy for cylindrical geometries. Environment 100 includes a plurality of cameras, such as cameras 115 and 117. Although Figure 1 depicts environment 100 with two cameras, environment 100 can include n cameras, where n is a number greater than one. The plurality of cameras can be wired or wirelessly coupled to the robotic system 101. In some embodiments, at least one of the plurality of cameras is in a fixed location. In some embodiments, at least one of the plurality of cameras moves dynamically (for example, it is attached to a moving object, such as a drone). In some embodiments, at least one of the plurality of cameras can be stationary and move to a different location (for example, detecting an item at a first location, moving the camera to a second location, and detecting the item at the second location).In some modalities, different lighting conditions in the environment 100 are used to detect changes in the perceived surface characteristics of one or more items. The use of multiple cameras allows the robotic system 101 to view the environment 100 from different perspectives. This prevents items from being obscured and generates more accurate estimates of item geometries and boundaries. For example, a large item can be positioned in such a way that it prevents a camera from seeing a smaller item next to it, thus hiding the smaller item from the camera. Using multiple cameras from different locations allows the smaller item to be seen and the boundary information associated with it to be determined. A single camera may not cover a large work area. The views from multiple cameras can be combined to give the robotic system 101 a more complete view of the workspace area 110. Even if one of the cameras is blocked, the robotic system 101 can still pick and place items. In some configurations, the Robotic System 101 segments items based on the point cloud generated by one or more cameras. The Robotic System 101 can segment items based on the RGB or multi-spectrum camera image (e.g., a combination of RGB, depth, and / or infrared, etc.). The segmented objects can be projected onto a point cloud to determine potential gripping areas. This generates additional information, such as item type, expected weight / material, preferred gripping strategy, etc., which is not available when an item is segmented based solely on point cloud information.This combined segmentation strategy works well when selecting items that are difficult to distinguish solely by their depth (e.g., small boxes packed together may appear as a single plane in a point cloud), but with image segmentation and point cloud information, the robotic system 101 can identify each box and extract the box using this information. In some embodiments, the robotic system 101 autonomously picks up and places unknown items from table 110 (e.g., a workspace area) onto conveyor belt 118 (e.g., a delivery area). Using cameras 115 and 117, the robotic system 101 can determine that items 112, 114, and 116 are located on table 110. Controller 106 determines the geometric information based on the visual data (e.g., point cloud data) received from cameras 115 and 117. Controller 106 selects the features of items 112, 114, and 116 that can potentially be grasped according to the geometric information determined from the visual data received from cameras 115 and 117.For example, based on visual data received from cameras 115 and 117, controller 106 can determine that item 112 includes a graspable feature corresponding to a cube shape, that item 114 includes a graspable feature corresponding to a pyramid shape, and that item 116 includes a cylindrical shape. Controller 106 can select a grasping feature that, to a certain extent, most closely resembles a geometric object. For example, controller 106 can compare the determined geometric information against a library of known features and select a feature for the item based on the comparison. In some modes, the features are canonical shapes. Controller 106 can overlay the canonical shapes onto the items to be grasped. To determine one or more features to be grasped associated with an item, the 106 controller can randomly slice planes from an item to decompose it into a plurality of subsegments. The item can be sliced ​​into planes using minimal data points from a point cloud (related to grasping a pointed feature on the top of an item). Planes can be sliced ​​from any item based on strong color or appearance gradients. In some modes, a membership function is used to determine if there are outliers in a point cloud within the generically generated subregion. An additional slice plane can be added, or the item can be divided into segregated areas with high residuals. The subsegments can be processed separately. For example, outlier detection techniques can be applied within the subsegments.In some methods, a 5-sigma test fits a Gaussian distribution to the data points and identifies the 5 points that are within 5 sigma (standard deviation) of the median, marking these points as outliers. In other methods, a subsampling method is applied to the data set and refitted to a median. These points are then used to find points that are a certain distance from the median. In some modes, the subsegments of an item are determined based on the reach of the end effector's interaction with the item. For example, if the end effector cannot grasp a wide item, the controller decides not to grasp the item by its wide portion. If a suction gripper end effector is used, a relatively smooth, flat surface is sought. Gap-based picking strategies or minimum-occupancy cutting planes can be avoided. Primitives are readjusted to the new partitioned cloud. The process can be repeated iteratively until some quality level or recursion limit is met. Controller 106 can determine negative spatial information (e.g., voids) associated with an item based on visual data received from cameras 115, 117. For example, controller 106 can determine that the handle of a coffee mug includes a negative space or that a car tire includes a negative space. Computer vision algorithms using data from multiple cameras can determine voids (e.g., holes) in the item, such as cups, coiled wire, tapes, etc. If a void is detected, the item can be grasped by inserting a gripper into the void and lifting it from a side wall. Controller 106 can determine the curvature of an item to be lifted based on visual data received from cameras 115 and 117. If controller 106 determines that the item is curved, it can change the control strategy associated with placing the item, placing it more carefully and releasing it more slowly to prevent movement during placement. In some configurations, the curved item includes a flatter surface. Robotic system 101 can place the item onto the flatter surface and reorient it for more stable placement. If visual data received from cameras 115 and 117 indicate that the placed item is rolling or moving after the initial grip is released, controller 106 can re-grasp it and attempt to reposition it before releasing it again.In the event that controller 106 attempts to grab / re-grab the item more than a certain limit number of times, user 130 may be given a warning and alerted that the item may be rolled away. Controller 106 determines the corresponding characteristics associated with items 112, 114, and 116 based on visual data received from cameras 115 and 117. For example, controller 106 can determine that an item includes a handle. The visual data received from the cameras can be used to determine a minimum and maximum limit associated with an item. The limit associated with an item includes its height, width, or depth. The visual data can provide information that allows one or more of the item's limits to be determined. Controller 106 is associated with a memory (not shown) that stores a data structure associating grasping strategies with features. A grasping strategy can consist of a grasping technique and how to grasp a feature using that technique. In some modalities, the grasping strategy involves grasping a major and minor axis of a bounding box that can be adjusted to the geometric estimate of the item / segment. In other modalities, a grasping strategy involves cutting the item / segment estimate at a height Z and recalculating the bounding box. The major and minor axes of the recalculated bounding box can then be grasped. This is useful when the item has a wide base but a small turret somewhere in the middle, and the robotic system wants to accurately grasp the top.The memory also stores instructions on how to perform the gripping techniques. These instructions may include instructions to partially close a gripper if necessary to avoid impacting other items. The memory also stores instructions on how to perform the placement techniques. These instructions may include instructions to partially open the fingers of the end effector 108 gripper so that the end effector 108 does not affect other items when the item is placed in the unloading area. The memory also stores information about the mechanism and geometry of the end effector (e.g., parallel gripper vs. suction gripper, gripper finger width / length, etc.). A gripping technique can be associated with one or more features. For example, a suction technique can be used on items with a feature that can be gripped in pyramidal shapes, a feature that can be gripped in cubic shapes, or a feature that can be gripped in rectangular prism shapes. A parallel gripping technique can be used for items with a gripping feature for spherical shapes. A feature can be associated with one or more gripping techniques. For example, a parallel gripping technique or a lifting technique can be used for a feature that can be gripped in spherical shapes. Different types of grippers can be used to grasp a feature with a particular shape. For example, a first grasping technique might use a parallel gripper, and a second grasping technique might use a suction gripper. In some modalities, the gripper types are changed autonomously during a grasping and positioning operation. A single grasping technique can be used on different parts of a feature. For example, a parallel grasping technique can be used on the top, middle, or bottom of a feature. Controller 106 determines the corresponding scores for each of the grasping strategies associated with a given feature. In some modalities, an item is associated with multiple features.The 15 controller 106 can determine one or more gripping techniques in each of the plurality of features and determine the corresponding scores for the determined gripping techniques. 2. The score associated with a grasping strategy can be based on the probability that the grasping strategy will result in a successful grasp of the feature. The probability of a successful grasping strategy can be based on one or more 25 modalities, such as contextual information about the environment, historical grasping information based on the environment, the angle at which the robotic arm should grasp the feature (to avoid collisions with other items), the height at which a robotic arm should grasp the feature (to avoid collisions at the top of the gripper), the grip width, the orientation of the normal surface at the grasping points, the amount of the feature that can be grasped, material properties, and so on.Contextual information about the environment includes the existence of other objects near or adjacent to the item, the extent to which those other objects affect the robotic arm's ability to grasp the feature, whether more objects are being continuously added to a workspace area, and so on. Material properties can include an item's center of mass, friction properties, color, flexibility, and so forth. For example, Robotic System 101 can construct a large support surface so that a large item can be stably placed on it.When the robotic system 101 detects that an item could slip off an inclined support surface given the friction coefficients of the item and the placement support surface, the robotic system 101 can be configured to choose to pick up only items with sufficiently high friction coefficients (e.g., to avoid slipping). Controller 106 selects one of the grasping strategies based on the corresponding scores associated with each strategy. The items may be a heterogeneous collection of items placed in a random pile. The items may vary in size, color, robotic light, geometry, texture, stiffness, etc. Each item is removed from the pile individually. Some of the items are at least partially occluded. The contents of the random pile are unknown beforehand. Controller 106 selects the grasping strategy with the highest score. If two or more grasping strategies have the same high score, Controller 106 selects one of the grasping strategies, chooses the feature associated with that strategy, moves the item to a delivery area, and then selects another item associated with a different grasping strategy. Controller 106 causes end effector 108 to grasp some feature associated with the item. In the example shown, controller 106 has caused end effector 108 to grasp item 112. Controller 106 can leverage prior knowledge of the grasping mechanism and geometry to simplify the grasping prediction problem. For example, if end effector 108 approaches an item, such as item 112, from above, controller 106 analyzes the top section of a point cloud to identify protrusions that can be grasped. In some modalities, as the robotic system moves, cameras 115 and 117 collect more data (e.g., closer proximity, different angles, different lighting, reflectivity, etc.), and the robotic system 101 adjusts how it causes end effector 108 to grasp an item based on the new data. s The gripping points of an item can be determined using a mesh or segmented version of it. An approximation of the item to be gripped is created, and a model matching a library or an I / O machine learning method is used to determine an optimal gripping location for the item. The gripping points are then categorized. Controller 106 causes the end effector 108 to grip the item using one of the gripping points. In several configurations, the end effector 108 includes one or more touch-sensing units (not shown) as described herein. The system uses the sensed values ​​from the touch-sensing unit(s) to determine whether an item has been successfully grasped. The system also uses sensed values ​​from the touch-sensing unit(s) to determine whether an item is beginning to slip or is slipping out of the grasp of the end effector 108. In some configurations, the system uses sensed values ​​from the touch-sensing unit(s) and a multimodal model to determine and implement a strategy to reorient and / or otherwise reposition an item into the grasp of the end effector 108.For example, the system can intentionally relax its grip on an item to a point where the item begins to slide in an expected and intended manner, such as sliding down between the fingers or other mating structures of the end effector 108, or rotating while still being held by the end effector 108. Once the item is in the desired position and / or orientation, more force is applied as needed to stop and prevent further sliding. The touch-sensing unit includes one or more sensors. Each of the one or more sensors is configured to provide an output that reflects a detected value associated with the mating of the end effector 108 with an item, such as item 112. The outputs of one or more sensors are used to determine a plurality of mating modes of the end effector 108 with an item. The one or more sensors may include one or more magnetic sensors, optical sensors, electromechanical sensors, pressure sensors, strain gauges, force sensors, conductivity sensors, current sensors, voltage sensors, capacitance sensors, resistance sensors, inductance sensors, infrared sensors, temperature sensors, etc. The plurality of modalities may include weight, deformation, continuity, conductivity, pressure, resistance, inductance, capacitance, or any other factor that is indicative of the coupling of the robotic arm's end effector. One or more touch-sensing units are coupled to a controller 106. The end effector 108 may include other sensors that are also coupled to the controller 106. For example, a housing connecting the fingers of the end effector 108 of the robotic arm 102 may include one or more sensors used to measure force and / or torque. The plurality of sensors (touch-sensing unit sensor(s) and / or housing sensor(s)) provide their corresponding outputs to the controller 106. The controller 106 can use the sensor outputs to determine one or more forces and / or one or more moments associated with a coupling between the end effector 108 and an item, such as item 112. The controller 106 can use the sensor outputs to determine the forces and moments associated with each of the touch-sensing units. Controller 106 can use the sensor outputs to determine the corresponding values ​​for the plurality of modes. For example, it can determine a detected weight, a detected strain, a detected continuity, a detected pressure, a detected resistance, a detected inductance, and / or a detected capacitance. Each of the plurality of modes is associated with a coefficient. In some modes, the plurality of modes are weighted equally (for example, each has an associated coefficient of 1). In some modes, some of the plurality of modes have different weights. For example, the selected item might be a metallic object, and the coefficient associated with a conductivity factor might be less than the coefficient associated with a strain factor. Controller 106 can implement a multimodal model to determine whether the robotic arm's end effector has grasped an item. Controller 106 can also implement a multimodal model to determine whether an item is beginning to slip and / or is slipping out of the grasp of an end effector 108. The multimodal model can be a rule-based model, a predictive model, a machine learning model (e.g., a neural network, linear classifier, support vector machine, linear regression, logistic regression, decision tree, deep learning, etc.), etc. In some modalities, the multimodal model is configured to output a binary decision regarding whether the item is beginning to slip out of the grasp of the end effector 108. In some modalities, the multimodal model is configured to output a binary decision regarding whether the item is slipping out of the grasp of the end effector 108.In some modalities, the multimodal model is configured to output a probability of whether the item is beginning to slip out of a grip on end effector 108. In some modalities, the multimodal model is configured to output a probability of whether the item is slipping out of the grip on end effector 108. The input (for example, the values ​​associated with some or all of the modalities) is applied to the multimodal model and to the robotic system 101 configured to perform a response action based on an output from the multimodal model. In some modalities, the robotic system 101 moves the item from table 110 to conveyor belt 118. In some modalities, the robotic system 101 applies suction force to grasp the item. In some modalities, the robotic system 101 lowers the item and grasps it again. In some modalities, the robotic system 101 requests human intervention. In some modalities, the robotic system 101 adjusts how it holds the item. In some modalities, the robotic system 101 changes its gripping configuration to allow the item to rest on the palm of the robotic arm 102 or another part of the robotic system 101 without dropping the item. 20 The end effector 108 moves an item, in this example item 112, to a discharge area, such as conveyor belt 118. The end effector 108 places the item in the delivery area. The robotic system 101 can use the plurality of cameras to position the item near where the robotic system 101 believes the item should be placed. The robotic system 101 can lower the item into the delivery location and detect when it senses the force of the delivery area. In some embodiments, the robotic system 101 uses mechanical vibrations from the impact between the bottom of an item and the delivery area to confirm that the item is being successfully placed onto the support surface. When the robotic system 101 detects that it has reached the delivery area, it opens the end effector 108 or stops the suction to gently place the item.As the end effector 108 opens, the robotic system 101 can move up or down to control the placement force (sometimes, opening the gripper while in contact can crush the items). This allows the robotic system 101 to stack items or dynamically adjust the placement height when estimating the height of the placement surface might be subject to error or is unknown. This also helps when other items are in the way. In some configurations, the robotic system 101 determines whether items that might roll are being placed in the delivery area. 2. If there are no items that could roll, controller 106 can control robotic arm 102 and end effector 108 to bring items already in the delivery area closer together to create space for one or more items. If there are items that could roll or tip over, controller 106 can control robotic arm 102 and end effector 108 to rotate the item and place its stable support surface in the delivery area. In several modes, the robotic system 101 automatically requests teleoperation intervention. In some modes, if during the pick-and-place operation shown in FIG. 1, the robotic system 101 reaches a state where it cannot determine the (next) strategy to continue the operation, it requests assistance from a remote operator (in this example) via teleoperation. In the example shown, controller 106 is connected via network 124 to a teleoperation computer 126. In some modes, the teleoperation computer 126 can participate in the operation of the robotic system 101 in autonomous mode, for example, by sending high-level instructions to controller 106 via network 124. In several modes, controller 106 and / or the teleoperation computer 126 can request teleoperation intervention, for example, if the robotic system 101 reaches a state where it does not have a strategy to perform (complete) a next task or step in the operation. For example, again referring to FIG. 1, if object 114 were dropped and landed on one of its flat sides, in an orientation that presented a triangular aspect to the robot, in some configurations the robotic system 101 might not have a strategy for picking up item 114 and / or might have timed out or exhausted a configured number of attempts to pick up item 114. In response, the teleoperator 130 could be requested to intervene via teleoperation and could use the manual input device 128 to control the robot's operation. For example, the teleoperator 130 could manipulate the robotic system 101 to pick up item 114 and place it on the conveyor belt 118.Or, the teleoperator 130 can use the robotic system 101 to change the orientation of item 114 to one in which the autonomous robotic system 101 would be expected (or more likely) to have a strategy to pick up item 114. In some modes, the Robotic System 101 monitors teleoperation and updates the multimodal model based on that teleoperation. For example, sensors associated with the Robotic System 101 (e.g., the sensor(s) on the touch sensing unit and / or the sensor(s) on the housing) can output values ​​during teleoperation. If a teleoperation grasp attempt results in a successful grasp of an item, the robotic system can learn the 25 sensor output values ​​that are indicative of a successful grasp and use those learned values ​​for subsequent autonomous grasps to determine whether an item was successfully picked up.In the event that a teleoperation grasp attempt results in a failed grasp of an item, the robotic system can learn the sensor output values ​​indicative of a failed grasp and use those learned values ​​for subsequent autonomous grasps to determine whether an item was not successfully grasped. Teleoperation can be performed in multiple instances. In each instance, the robotic system can learn a range of sensor output values ​​indicative of a successful grasp and a range of sensor output values ​​indicative of a failed grasp. Corresponding ranges can be learned for different types, shapes, and / or sizes of items. In the example shown, teleoperation can be performed by a human operator 130 manipulating a manual input device 128, for example, a haptic input device. To initiate teleoperation, the human operator 130 (sometimes referred to as the teleoperator) can be prompted by information displayed on a display device included with and / or associated with the teleoperation computer 126. Data from one or more sensors 134 can be provided to the human operator 130 via the network 124 and the teleoperation computer 126. In some modalities, the sensors 134 include a camera on the robot (not shown) or cameras 115, 117 and are configured to generate a video feed that is displayed to the teleoperator 130 and used to perform and / or complete the execution of an operation or part thereof via teleoperation.In several configurations, the camera connects via a low-latency, high-performance connection, which, by way of example and without limitation, includes one or more analog communications based on Ethernet, Wi-Fi, Bluetooth, and sub-GHz. Some configurations use a combination of different camera types. For example, cameras with varying communication speeds, bandwidth, and / or other characteristics can be used, such as two RGB visual cameras, four depth cameras, two IR cameras, etc. In several modes, teleoperation can be performed using a variety of 134 different sensors. In some 15 modes, these can guide the robotic system 101 to determine if it is stuck and / or if they can simplify teleoperation. In some modes, the sensors help move the teleoperation mode from direct haptic controls to increasingly abstract execution commands (such as clicking on an object to select it with a mouse or saying "open shelf" on an audio transcription device). Examples of sensors 134 used in various modalities include digital switches configured to detect specific stuck interactions and scenarios within the environment, and / or the presence of unknown agents in the vicinity of the robotic system 101 (or the teleoperator). Other examples include force or pressure sensors on the hand or robot that determine the success or failure of operations such as grasping. After a series of failures, the robotic system 101 determines that it is stuck. Another example is one or more sensors, such as position sensors on the robot's joints, that the robotic system 101 can use to determine whether the planned and / or expected motion path is being followed accurately.When io is not accurately following the expected trajectory, it has likely made contact with the environment 100 and the robotic system 101 can be programmed to conclude that it is stuck and needs human intervention. A vision system is configured that includes a plurality of cameras to track each item within a workspace area through multimodal means (e.g., RGB instance tracking, RGB feature matching, RGB optical flow, RGB feature matching). 2 or point clouds, etc.) and to use methods, such as Hungarian pair matching, to track the items that Robotic System 101 must choose. Robotic System 101 is configured to estimate the states of each tracked item, such as speed, fall / drop potential, and motion trajectory. Robotic System 101 can use other known information, such as current speed and size of transport systems and sensors 134, to more accurately update the item states. Robotic System 101 can use the determined item states to make informed decisions about where and which items to choose, and where / when / how to place the items.For example, the robotic system 101 can select to lift (grasp) more stable items and possibly grasp (even while moving) from an estimated future item location to compensate for the movement time of the robotic arm 102 and the speed of a moving item. The robotic system 101 can place an item in the delivery area of ​​a mobile platform more steadily without dropping or rolling the item, placing it with an estimated initial speed 15 in the environment 100. The robotic system 101 can also choose collision-free zones to place items within the delivery area 118. Collision zones can be determined from the estimated trajectories of the tracked items. Using the data associated with the plurality of cameras, the robotic system 101 can understand the shape of the grasped item and the environment 100.This allows the robotic system 101 to intelligently plan trajectories that will avoid collisions between the selected items and the environment 100. In some models, a variety of robotic systems collaborate to grasp and place items. The use of a plurality of robotic systems can increase the overall performance of the system. FIG. 2 is a block diagram illustrating a robotic arm end effector according to several configurations. In several configurations, end effector 200 can be used to implement end effector 108 of FIG. 1. In the example shown, the end effector 200 includes a housing 202 attached to the robotic arm 204 by means of a swivel coupling. In some embodiments, the connection between the housing 202 and the robotic arm 204 may comprise a motorized joint controlled by a control computer, such as the controller 106 of FIG. 1. The end effector 200 further includes a gripper comprising the joint digits 210 and 212 and a power line 206 running along the robotic arm 204 and to the housing 202 to supply power to the gripper control module 208. In several embodiments, the control module 208 is connected, for example, wirelessly and / or by cable through the communication interface 214 to a control computer external to the end effector 200, for example, the controller 106 of FIG. 1.The control module 208 includes electronic and / or electromechanical elements that are operated to manipulate the digits of the grippers 210, 212, for example, to grasp an item to be lifted, moved and placed using the end effector 200. In the example shown, the camera 216 mounted on one side of the housing 202 provides image data of a field of view below the end effector 200. A plurality of force sensors 218, 220, 222, 224, 226, and 228 measure the force applied to the digit points 210 and 212, respectively. In various modes, the force measurements are transferred via the communication interface 214 to a remote and / or external control computer. The sensor readings are used in various modes to enable the robotic arm 204 and the end effector 200 to be used to place an item in position, adjacent to other items and / or side walls or other structures, and / or to detect its instability (e.g., insufficient push when the item is pressed down while still under suction, but in the location where the item was expected to be placed and held stable). In some models, sensors are used to detect collisions with other items, the container, and / or the environment, and to continue automatic operation by precisely adjusting the trajectory. For example, if a wall or other structure is hit, in some models the robotic arm reduces the force and adjusts the trajectory to continue past the obstacle. The weight sensor 215 can be a force sensor, a load cell, or any other sensor capable of detecting a force applied directly or indirectly to the housing 202. The weight sensor 215 can be configured to measure rotational and / or directional torque. The weight sensor 215 can also be configured to measure a force and a moment when the end effector 202 is attached to an item. The touch sensing units 211 and 213 are associated with the articulated digits 210 and 212, respectively. In some embodiments, the touch sensing units 211 and 213 are attached to the digits 210 and 212, respectively. In some embodiments, the touch sensing units 211 and 213 are included in a sensing cover that is placed over the end effector 200, such that each of the digits 210 and 212 has a corresponding touch sensing unit 211. The sensing cover may cover the digits 210 and 212, as well as the housing 202. In some embodiments, the touch sensing units 211 and 213 are of the same type as the touch sensing units 20. In some embodiments, the touch sensing units 211 and 213 are different types of touch sensing units. For example, the digit 210 may be associated with the touch sensing unit depicted in FIG. 3B, and the digit 212 may be associated with the touch sensing unit 25 depicted in FIG. 3E. Figure 3A is a block diagram illustrating a touch sensing unit according to several modalities. In the example shown, the touch sensing unit 300 can be implemented as a touch sensing unit, as can the touch sensing unit 211 or the touch sensing unit 213. The touch sensing unit 300 includes a conductive layer 302, a deformation layer 304, and a substrate layer 306. In some configurations, the conductive layer 302, deformation layer 304, and substrate layer 306 are flat layers. In some configurations, the conductive layer 302, deformation layer 304, and substrate layer 306 are curved layers. The touch sensing unit 300 is coupled to a processor (not shown). In some configurations, the conductive layer 302 or the substrate layer 306 are optional. In some configurations, the conductive layer 302 and the deformation layer 304 are combined into a single layer. The conductive layer 302 can include conductive material that allows for the detection of continuity or other electrical properties, such as conductivity, resistance, capacitance, or inductance. For example, the conductive layer 302 can be an upper layer of the touch-sensing unit so that when a first touch-sensing unit attached to a first finger makes contact with a second touch-sensing unit attached to a second finger, the conductive layers of the first and second touch-sensing units come into contact. Before a gripping operation, a reference continuity value, a reference conductivity value, a reference resistance value, a reference capacitance value, and / or a reference inductance value can be determined. The detected reference values ​​can be the tare value before the robotic arm end effector engages an item.During a grasping operation, the end effector may use their fingers to grasp an item, and the processor may transmit an electrical signal to a first touch-sensing unit to determine one or more continuity values, one or more conductivity values, one or more resistance values, one or more capacitance values, and / or one or more inductance values. In some modes, the electrical signal varies in frequency. The determined continuity values, conductivity values, resistance values, capacitance values, and inductance values ​​indicate whether or not the end effector grasped the item. When the robotic arm's end effector is grasping an item moving from a first location to a second location, the sensing values ​​associated with the conductive layer may vary. The robotic system's processor can monitor these sensing values ​​over time as the robotic arm's end effector moves the item from the first location to the second location to determine if the item is beginning to slip out of the end effector's grip and / or if it is already slipping out of the grip. When the robotic end effector successfully grasps an item, the successful grasp can have an associated continuity value. When the robotic arm's end effector is grasping an item moving from a first location to a second location, the processor can compare the current continuity value with the continuity value associated with a successful grasp. If the current continuity value differs by a threshold amount from the continuity value associated with a successful grasp (i.e., the conductive layers of two touch-sensing units are in contact), the current continuity value may indicate that the item is slipping from the end effector's grasp. If the current continuity value does not differ by a threshold amount from the continuity value associated with a successful grasp, the current continuity value may indicate that the item is not slipping from the end effector's grasp. When the robotic end effector successfully grasps an item, the successful grasp can have an associated conductivity value. When the robotic arm's end effector is grasping an item moving from a first location to a second location, the processor can compare the current conductivity value with the conductivity value associated with a successful grasp. If the current conductivity value does not change by a threshold amount from the conductivity value associated with a successful grasp, the current conductivity value may indicate that the item is beginning to slip from the end effector's grasp. If the current conductivity value does not change by a threshold amount from the conductivity value associated with a successful grasp, the current conductivity value may indicate that the item is not beginning to slip from the end effector's grasp.If the current conductivity value does not change by a threshold amount from the conductivity value associated with a successful grip, the current conductivity value may indicate that the item is slipping from the end effector's grip. The second threshold amount may be greater than the first threshold amount. In some configurations, when the robotic end effector successfully grasps an item, the successful grasp can have an associated conductivity profile. The conductivity profile can be composed of one or more conductivity values ​​based on the output of one or more tactile sensors. When the robotic arm's end effector is grasping an item moving from a first location to a second location, the processor can compare the current conductivity profile with the conductivity profile associated with a successful grasp. If the current conductivity profile does not change by a threshold amount from the conductivity profile associated with a successful grasp (for example, by a first threshold amount), the current conductivity profile may indicate that the item is beginning to slip from the end effector's grasp.If the current conductivity profile does not change by a threshold amount from the profile associated with a successful grip, the current conductivity profile may indicate that the item is not beginning to slip from the end effector grip. If the current conductivity profile changes from a threshold amount indicating that the item is beginning to slip from the end effector grip (for example, by a second threshold amount), the current conductivity profile may indicate that the item is slipping from the end effector grip. When the robotic end effector successfully grasps an item, the successful grasp can have an associated resistance value. When the robotic arm's end effector is grasping an item moving from a first location to a second location, the processor can compare the current resistance value with the resistance value associated with a successful grasp. If the current resistance value does not change by a threshold amount from the resistance value associated with a successful grasp, the current resistance value may indicate that the item is beginning to slip from the end effector's grasp. If the current resistance value does not change by a threshold amount from the resistance value associated with a successful grasp, the current resistance value may indicate that the item is not beginning to slip from the end effector's grasp.If the current resistance value does not change by a threshold amount from the resistance value associated with a successful grip, the current resistance value may indicate that the item is slipping from the end effector grip. The second threshold amount may be greater than the first threshold amount. In some configurations, when the robotic end effector successfully grasps an item, the successful grasp may have an associated resistance profile. The resistance profile may be composed of one or more resistance values ​​based on the output of one or more tactile sensors. When the robotic arm's end effector is grasping an item moving from a first location to a second location, the processor can compare the current resistance profile with the resistance profile associated with a successful grasp. If the current resistance profile does not change by a threshold amount from the resistance profile associated with a successful grasp (for example, by a first threshold amount), the current resistance profile may indicate that the item is beginning to slip from the end effector's grip.If the current resistance profile associated with a successful grip does not change by a threshold amount, the current resistance profile may indicate that the item is not beginning to slip from the end effector grip. If the current resistance profile changes by a threshold amount indicating that the item is beginning to slip from the end effector grip (for example, by a second threshold amount), the current resistance profile may indicate that the item is slipping from the end effector grip. 15. When the robotic end effector successfully grasps an item, the successful grasp can have an associated capacitance value. The processor can also determine the material of the grasped item based on this associated capacitance value. Secondly, when the robotic arm's end effector is grasping an item moving from a first location to a second location, the processor can compare the current capacitance value with the capacitance value associated with a successful grasp. If the current capacitance value does not change by a threshold amount from the capacitance value associated with a successful grasp, the current capacitance value may indicate that the item is beginning to slip from the end effector's grip.If the current capacitance value does not change by a threshold amount from the capacitance value associated with a successful grip, the current capacitance value may indicate that the item is not slipping from the end effector grip. If the current capacitance value does not change by a threshold amount from the capacitance value associated with a successful grip, the current capacitance value may indicate that the item is slipping from the end effector grip. The second threshold amount may be greater than the first threshold amount. In some modalities, when the robotic end effector successfully grasps an item, the successful grasp can have an associated capacitance profile. The capacitance profile can be composed of one or more capacitance values ​​based on the output of one or more tactile sensors. The processor can also determine the material of the grasped item based on the associated capacitance value. When the robotic arm's end effector is grasping an item that is moving from a first location to a second location, the processor can compare the current capacitance profile with the capacitance profile associated with a successful grasp.If the current capacitance profile (25) does not change by a threshold amount from the capacitance profile associated with a successful grip (for example, at a first threshold amount), the current capacitance profile may indicate that the item is beginning to slip from the end effector grip. If the current capacitance profile (5) does not change by a threshold amount from the capacitance profile associated with a successful grip, the current capacitance profile may indicate that the item is not beginning to slip from the end effector grip. If the current capacitance profile (25) changes by a threshold amount indicating that the item is beginning to slip from the end effector grip (for example, at a second threshold amount), the current capacitance profile may indicate that the item is slipping from the end effector grip. When the robotic end effector successfully grasps an item, the successful grasp can have an associated inductance value. When the robotic arm's end effector is grasping an item moving from a first location to a second location, the processor can compare the current inductance value with the inductance value associated with a successful grasp. If the current inductance value differs by a threshold amount from the inductance value associated with a successful grasp (i.e., the conductive layers of two touch-sensing units are in contact), the current inductance value may indicate that the item is slipping from the end effector's grasp.If the current inductance value does not change by a certain threshold amount from the inductance value associated with a successful grip, the current inductance value may indicate that the item is not beginning to slip from the end effector grip. If the current inductance value does not change by a certain threshold amount from the inductance value associated with a successful grip, the current inductance value may indicate that the item is slipping from the end effector grip. The second threshold amount may be greater than the first threshold amount. In some configurations, when the robotic end effector successfully grasps an item, the successful grasp can have an associated inductance profile. This inductance profile can be composed of one or more inductance values ​​based on the output of one or more tactile sensors. When the robotic arm's end effector is grasping an item moving from a first location to a second location, the processor can compare the current inductance profile with the inductance profile associated with a successful grasp. If the current inductance profile does not change by a certain threshold amount from the inductance profile associated with a successful grasp (for example, by a certain initial threshold amount), the current inductance profile may indicate that the item is beginning to slip from the end effector's grasp.If the current inductance profile associated with a successful grip does not change by a certain threshold amount, the current inductance profile may indicate that the item is not beginning to slip from the end effector grip. If the current inductance profile changes to a level indicating that the item is beginning to slip from the end effector grip (for example, by a second threshold amount), the current inductance profile may indicate that the item is slipping from the end effector grip. The second threshold amount may be greater than the first threshold amount. The deformation layer 304 may include a dielectric material (e.g., silicone, plastic, or any other material capable of deforming in response to a force). The measuring objects may be injected into the dielectric material 15. For example, a plurality of magnets are injected into the dielectric material. The substrate layer 306 may include one or more sensors configured to detect a change associated with the measurement objects injected into the strain layer 304. For example, the substrate layer 306 may include a magnetic sensor configured to detect a change in the magnetic field when the plurality of magnets in the strain layer 304 is displaced. In some embodiments, the substrate layer 306 is a finger associated with a robotic arm end effector. In some configurations, the 306 substrate layer is a material that is configured to support one or more sensors and electrically connects one or more sensors to a processor associated with the robotic system. Figure 3B is a diagram illustrating a touch sensing unit from different viewpoints according to several modalities. In the example shown, the different viewpoints illustrate a top view and two side views of a touch sensor unit. A conductive layer of the touch sensing unit is not shown for explanatory purposes. In the example shown, the deformation layer of the touch-sensing unit includes a plurality of measuring objects. In some modalities, the plurality of measuring objects are magnets. In some modalities, the plurality of measuring objects are markers. The plurality of measuring objects can be arranged in a grid pattern or without a grid. The grid pattern can be a 2D m x no grid pattern or a 3D 1 x m x n grid pattern. The measuring objects can be located on an edge boundary of the deformation layer. In the example shown, measurement objects 312a, 312b, 312c, 312d, 312e, 312f are arranged on a 2 x 25 grid. Although six measurement objects are shown, any number of measurement objects can be injected into the deformation layer. The number of measurement objects that can be injected into the deformation layer can be based on an acceptable amount of crosstalk between the measurement objects. For example, as the number of measurement objects in the deformation layer increases, the amount of crosstalk between the measurement objects also increases. The acceptable amount of crosstalk can be specified by an operator associated with the robotic system. The plurality of measuring objects is associated with a sensor 314 located in substrate layer 306. If the plurality of measurement objects are magnets, sensor 314 is a magnetic sensor. If the plurality of measurement objects are markers, sensor 314 is an image sensor. In some embodiments, a measurement object is associated with a sensor. In some embodiments, a measurement object is associated with a plurality of sensors. Before a gripping operation, a reference magnetic field can be determined for the plurality of magnets or a reference position for each of the markers. When the robotic arm's end effector engages an item, a force and / or moment associated with the engagement causes one or more of the plurality of measurement objects to displace. This displacement causes a change in a measurable value that is configured to be detected by sensor 314. For example, if the plurality of measurement objects are magnets, sensor 314 can detect a change in a magnetic field associated with the plurality of magnets. A processor can be coupled to the magnetic sensor and compare the detected magnetic field with the reference magnetic field. The detected magnetic field may differ depending on whether the robotic arm's end effector successfully or unsuccessfully grasped the item.The processor can determine a weight value and / or a deformation value based on the comparison. When the robotic end effector successfully grasps an item, the successful grasp may have an associated magnetic field value. Sensor 314 can detect the magnetic field as the robotic arm's end effector moves the item from the first location to the second location to determine if the item is beginning to slip and / or is slipping out of the end effector's grasp. The processor can monitor the current magnetic field, current weight, and current strain values ​​to detect any changes. A change in the current magnetic field associated with a successful grasp, the current weight associated with a successful grasp, and / or the current strain associated with a successful grasp that exceeds the corresponding first threshold may indicate that the item is beginning to slip out of the end effector's grasp.A change in the current magnetic field associated with a successful grip, the current weight value associated with a successful grip, and / or the current strain value associated with a successful grip that exceeds the corresponding second threshold may indicate that the item is slipping out of the robotic arm's end effector grip. If the plurality of measurement objects consists of markers, sensor 314 can detect a change in the position associated with one or more of the markers. The position change may differ depending on whether the robotic arm's end effector successfully or unsuccessfully grasped the item. A processor can be coupled to the image sensor and compare the detected positions of the markers with reference positions to determine a weight and / or deformation value. When the robotic end effector successfully grasps an item, the plurality of markers have corresponding positions. Sensor 314 can detect the position of the markers as the robotic arm's end effector moves the item from the first location to the second location to determine if the item is beginning to slip and / or is slipping out of the robotic arm's end effector's grasp. The processor can monitor the current positions, current weight value, and current deformation value for any changes. A change in the current positions associated with a successful grasp, the current weight value associated with a successful grasp, and / or the current deformation value associated with a successful grasp that exceeds the corresponding first threshold 5 may indicate that the item is beginning to slip out of the robotic arm's end effector's grasp.A change in the current positions of the positions associated with a successful grip, the current weight value of the weight associated with a successful grip, and / or the current strain value of a strain value associated with a successful grip that exceeds the corresponding second threshold may indicate that the item is slipping from the grip of the robotic arm's end effector. Figure 3C is a diagram illustrating a touch sensing unit 15 from different viewpoints according to several modalities. In the example shown, the different viewpoints 320 illustrate a top view and two side views of a touch sensor unit. A conductive layer of the touch sensing unit is not shown for explanatory purposes. In the example shown, the measuring objects embedded in the deformation layer 304 are magnets. The touch-sensing unit in FIG. 3C is similar to the touch-sensing unit in FIG. 3B except that a subset 25 of the measuring objects (322a, 322b, 322c, 322g, 322h, 322i) are located on the sides of the deformation layer. A magnitude of the magnetic field associated with the measuring objects 322a, 322b, 322c, 322d, 322e, 322f, 322g, 322h, 322i that is detected by sensor 314 can depend on a depth associated with the deformation layer 304 and a distance between the measuring objects and sensor 314. Placing a subset of the measurement objects on the sides of the deformation layer can increase the magnitude of the magnetic field associated with measurement objects 322a, 322b, 322c, 322d, 322e, 322f, 322g, 322h, 322i that are detected by sensor 314. Figure 3D is a diagram illustrating an example of a magnetic sensor according to several embodiments. The magnetic sensor 330 can be implemented as a sensor, similar to sensor 314. In the example shown, the magnetic sensor 330 includes coils 332a, 332b, and 332c. Coil 332a is configured to measure a magnetic field in the x-axis direction. Coil 332b is configured to measure a magnetic field in the y-axis direction. Coil 332c is configured to measure a magnetic field in the z-axis direction. The magnetic sensor 330 can be coupled to a plurality of magnets, such as those depicted in Figures 3B and 3C. A displacement of at least one of the magnets causes a change in the magnetic field detected by the magnetic sensor 330. The magnetic sensor 330 is configured to detect a change in the magnetic field in the x-axis, y-axis, and z-axis directions. FIG. 3Ξ is a diagram illustrating a touch sensing unit according to several modalities. The touch sensing unit 340 can be implemented as a touch sensing unit, such as the touch sensing unit 211 or the touch sensing unit 213. In the example shown, a reflective material 341 is coupled to the deformation layer 304. The emitter 342 (e.g., a light-emitting diode emitter) and the receiver 343 are located on the substrate layer 306. The emitter 342 can transmit a signal that is reflected by the reflective material 341 and received by the receiver 343. A reference amplitude of the signal received by the receiver 343 can be determined when a robotic arm end effector is not coupled to an item. When the robotic arm's end effector engages an item, a force and / or moment associated with the engagement changes the way the signal travels through the 304 deformation layer.For example, the amplitude of the signal received at receiver 343 may increase or decrease due to reflection, refraction, and / or scattering of the signal within the deformation layer 304. A processor can be coupled to the receiver and compare the amplitude of the reference signal with the amplitude of the received signal to determine a weight value and / or a deformation value. When the robotic end effector successfully grasps an item, the successful grasp can be associated with an amplitude value. Receiver 343 can receive the current signal as the robotic arm's end effector moves the item from the first location to the second location, and the processor can use the current signal amplitude to determine if the item is beginning to slip and / or is slipping out of the robotic arm's end effector's grasp. The processor can monitor the current signal amplitude, current weight value, and / or current strain values ​​to detect any changes.A change in the amplitude of the current signal relative to the amplitude of the signal associated with a successful grip, the current weight value associated with a successful grip, and / or the current strain value associated with a successful grip that exceeds the corresponding first threshold may indicate that the item is beginning to slip out of the robotic arm's end effector grip. A change in the amplitude of the current signal relative to the amplitude of the signal associated with a successful grip, the current weight value associated with a successful grip, and / or the current strain value associated with a successful grip that exceeds the corresponding second threshold may indicate that the item is slipping out of the robotic arm's end effector grip. FIG. 3F is a diagram illustrating a touch sensing unit according to some modalities. The touch sensing unit 350 can be implemented as a touch sensing unit, as can the touch sensing unit 211 or the touch sensing unit 213. In the example shown, a conductive layer and a substrate layer of the touch sensing unit are not shown for illustrative purposes. The tactile sensing unit 350 includes an electromechanical material sensor 352 capable of detecting deformation (e.g., with a strain gauge) of the dielectric material. A reference detected resistance can be determined using the electromechanical sensor 352 when a robotic arm end effector is not coupled to an item. When the robotic arm end effector 15 is coupled to an item, a force and / or moment associated with the coupling causes a resistance that is associated with a change in the electromechanical sensor 352. A processor can be coupled to one or more electromechanical sensors and compare the reference detected resistance value against the changed resistance value to determine a weight and / or deformation value. When the robotic end effector successfully grasps an item, the successful grasp can be associated with a resistance value of 25. As the robotic arm's end effector moves the item from the first location to the second, the processor can use the current resistance value to determine if the item is beginning to slip and / or is slipping out of the robotic arm's end effector's grip. The processor can monitor the current resistance value, current weight value, and / or current strain values ​​to detect any changes.A change in the current resistance value relative to the resistance value associated with a successful grip, a change in the current weight value relative to the weight associated with a successful grip, and / or a change in the current strain value relative to the strain value associated with a successful grip that exceeds the first corresponding threshold may indicate that the item is beginning to slip out of the robotic arm's end effector grip. A change in the current resistance value relative to the resistance value associated with a successful grip, a change in the current weight value relative to the weight associated with a successful grip, and / or a change in the current strain value relative to the strain value associated with a successful grip that exceeds the second corresponding threshold may indicate that the item is slipping out of the robotic arm's end effector grip. FIG. 3G is a diagram illustrating a touch sensing unit according to some modalities. The 360 ​​touch sensing unit can be implemented as a touch sensing unit, as can the 211 or 213 touch sensing unit. In the example shown, a conductive layer and a substrate layer of the touch sensing unit are not shown for illustrative purposes. The deformation layer of the 360 ​​5 touch sensing unit includes a flexible membrane 362 (e.g., a sack, a bag) containing air, a gas, or a liquid. A reference pressure associated with the flexible membrane 362 can be determined when a robotic arm end effector is not coupled to an item. When the robotic arm end effector is coupled to an item, a force and / or moment associated with the coupling causes a pressure that corresponds to a change in the flexible membrane. A processor can be coupled to the flexible membrane, and the reference pressure value can be compared to the changed pressure value to determine a weight and / or deformation value. When the robotic end effector successfully grasps an item, the successful grasp can be associated with a pressure value. As the robotic arm's end effector moves the item from the first location to the second location, the processor can use the current pressure value to determine if the item is beginning to slip and / or is slipping out of the robotic arm's end effector's grasp. The processor can monitor the current pressure value, the current weight value, and / or the current strain values ​​to detect any changes. A change in the current pressure value from the pressure value associated with a successful grasp, the current weight value from the weight value associated with a successful grasp, and / or the current strain value from the strain value associated with a successful grasp that exceeds the corresponding first threshold may indicate that the item is beginning to slip out of the robotic arm's end effector's grasp.A change in the current pressure value from the pressure value associated with a successful grip, the current weight value from the weight associated with a successful grip, and / or the current strain value from a strain value associated with a successful grip that exceeds the corresponding second threshold may indicate that the item is slipping out of the robotic arm's end effector grip. FIG. 4 is a flowchart illustrating a process for picking up and placing an item according to certain procedures. In the example shown, process 403 can be implemented using a robotic system, such as robotic system 101. In step 402, an item is selected to be moved from a first location to a second location. A first location, such as a workspace area, can include one or more items. The robotic environment can include multiple cameras configured to view and detect one or more items from different perspectives. In some configurations, one or more cameras generate one or more point clouds of one or more items. If multiple point clouds are generated, they are combined. In several configurations, one or more items can include multiple items placed in a random stack, multiple items that are spaced out, and / or multiple items where one or more are hidden by another item or items. The geometric information for each of the plurality of items is determined. This geometric information can be determined based on the point cloud data obtained from the data associated with one or more of the plurality of cameras. The corresponding geometric information associated with each of the plurality of features can be compared with a library of geometries for which gripping strategies are known. A strategy associated with a geometry that most closely resembles the geometry of a given feature can be selected, for example, within a certain similarity limit. In some embodiments, an item is associated with only one feature (for example, a roll of paper towels corresponds to a cylinder).In some forms, an article is divided into a plurality of subsegments (also called sub-objects in this document) and then the characteristics of one of the plurality of subsegments are determined (for example, a golf club includes a body segment and a head segment). Items located near the edges or corners of a workspace may have physical or other limitations on where and / or how the item will be picked. In some configurations, a non-optimal but feasible picking angle may be selected, depending on the environmental constraints associated with a particular item. A wider longitudinal grip may be chosen over a narrower latitudinal grip because the wider longitudinal grip may not keep the end effector within the environmental boundaries. The corresponding scores for a successful grasp are determined for each of the defined grasping strategies. For example, a grasping tool, 15, is used as the end effector 200 to grasp an item at its top, middle, or bottom. The score for a successful grasp of a feature can be based on the probability that the grasping strategy will result in a successful grasp. The probabilities of 2ii different combinations of grasping tools (in modalities where multiple tools are available) and grasping locations are determined.The probability that a grasping strategy will result in a successful grip on an item can be based on one or more grasping modalities, such as contextual information about the environment, historical grasping information based on the environment, the angle at which the robotic arm should grasp the item (to avoid collisions with others), the height at which a robotic arm should grasp the item (to avoid collisions at the top of the gripper), the grip width, the orientation of the normal surface at the grasping points, the amount of the item that the arm is capable of grasping, and so on. Contextual information about the environment includes the existence of other items near or adjacent to the item, the extent to which other items near or adjacent to the item affect the robotic arm's ability to grasp it, whether more items are being continuously added to a workspace area, and so forth. One of the predetermined grasping strategies is selected based on the corresponding predetermined scores. The items or features, and the corresponding grasping strategies, are ranked according to these scores. The item or feature with the highest score among the plurality of items and features is selected to grasp. If several potential grasps have the same score, one is selected from among them. Once the grasped object has been moved, one of the other grasps with the same score is selected. In the case of an item being selected from among the plurality of items that have been grasped, moved and placed in a delivery area, the grasp with the next highest score is selected to be attempted. In step 404, the selected item is grasped. A robotic system end effector is attached to the selected item. The end effector may include a plurality of fingers. Each of the plurality of fingers may be associated with a corresponding touch sensing unit. The touch sensing unit may include a plurality of sensing layers. Each of the sensing layers may include one or more sensors. When connected with an item, each of the layers of the touch sensing unit may generate a corresponding sensed value that is different from a reference sensed value (for example, when the robotic arm's end effector is not engaged with an item). The robotic system performs a static measurement using multiple sensors to determine if the robotic arm's end effector successfully grasped the item. The static measurement may involve the robotic arm's end effector grasping the item at rest and measuring the outputs of multiple sensors. The robotic system may also perform a dynamic measurement using multiple sensors to determine if the robotic arm's end effector successfully grasped the item. For example, after an initial grasp, the orientation of the robotic arm's end effector may be changed, and the output of multiple sensors may be measured as the end effector moves to determine if it successfully grasped the item. One or more touch-sensing units are coupled to a processor. The processor can use the sensor outputs to determine one or more forces and / or one or more moments associated with a coupling between the robotic arm's end effector and an object. The sensor outputs (touch-sensing unit sensor(s) and / or housing sensor(s)) can be used to determine the corresponding values ​​for multiple modalities. For example, one can determine a detected weight, a detected strain, a detected continuity, a detected conductivity, a detected pressure, a detected resistance, a detected inductance, and / or a detected capacitance. The plurality of modalities is each associated with a Second coefficient. In some categories, the plurality of categories are weighted equally (for example, each has an associated coefficient of 1). In some categories, some of the plurality of categories have different weights. For example, the selected item may be a metallic object, and the coefficient associated with a continuity factor may be lower than the coefficient associated with a deformation factor. The processor can implement a multimodal model to determine whether the robotic arm's end effector has grasped an item. The multimodal model can be a rule-based model, a predictive model, or a machine learning model. In some modalities, the multimodal model is configured to output a binary decision regarding whether the robotic arm's end effector is engaged with an item (e.g., engaged / unengaged). In other modalities, the multimodal model is configured to output a probability of whether the robotic arm's end effector is engaged with the item. Some or all of the corresponding values ​​associated with the plurality of modalities and their corresponding coefficients are used as input to the multimodal model, and the robotic system is configured to perform a response action based on the multimodal model's output. 2. The robotic system can be trained to apply force to a grasped object. The robotic system can apply a variety of forces to items of different shapes, sizes, types, and weights to learn a basic principle. A variety of forces can be applied to different gripping locations on an item. A variety of forces can be applied for different gripping techniques. The robotic system can learn a force level that will result in a successful grasp of a particular item. The robotic system can also learn a force level that will result in a particular item slipping off a robotic arm. In step 406, the selected item is moved. The selected item is moved in response to an output from the multimodal model indicating that the end effector has grasped the selected item. A processor can determine whether the selected item begins to slide or slides while being moved. The processor can use the current outputs from one or more touch-sensing units to make this determination. The current values ​​for a plurality of modalities (e.g., weight, strain, continuity, conductivity, pressure, resistance, inductance, capacitance, or any other factor indicative of the robotic arm's end-effector engagement) can be determined based on the outputs of one or more touch-sensing units. The corresponding current values ​​for the plurality of modalities and their respective coefficients are sent as input to the multimodal model.In some modes, the current outputs from one or more touch-sensing units are also sent to the multimodal model as input. This input is applied to the multimodal model, and the robotic system is configured to determine whether the selected item begins to slide or slides while moving, based on the multimodal model's output. In 408, it is determined whether or not the object 5 associated with the selected grip has been dropped while moving from the workspace area to the delivery area. Whether the object has been dropped can be determined based on one or more sensor measurements (e.g., pressure, force, capacitance, etc.) from one or more tactile sensor units associated with the end effector of the robotic system. A sensor measurement can be compared to a corresponding threshold value to determine whether the object has been dropped. If the item has been dropped, process 400 returns to 402. If the item has not been dropped, process 400 proceeds to 410. In 410, the selected item is placed in the second location. Items can be placed in a way that prevents the end effector from colliding with the boundaries associated with the second location. In some configurations, the item is placed randomly with other items in the second location. The robotic system can randomly place the item in the second location. The robotic system can use a force sensor on the end effector to gently place the item into the group without triggering a safety stop. The robotic system can then pack items into boxes at the second location by using the force sensor to determine an appropriate separation strategy. In some embodiments, the item is placed separately from other items in the second location. The robotic system can divide a placement space in the second location into multiple sub-areas and place the selected item in one of them. There may be a buffer zone between each of the sub-areas. In some embodiments, the buffer zone is adjustable. In some configurations, a vision system is integrated with the robotic system to determine how to position the item. For example, some items are not rigid, and the surface they are attached to (e.g., fabric or a plush toy) changes after the item has been grasped and moved. The vision system is configured to determine the surface characteristics and material information to choose how to position the item, avoiding crushing it or dropping it from a height that could damage it or cause it to fall in a tangled or unfavorable manner. In step 412, it is determined whether there are more items to move. Visual data from one or more cameras of the robotic system can be used to determine if there are more items to move. If there are more items to move, process 400 returns to step 402. If there are no more items to move, process 400 ends. Figure 5 is a flowchart illustrating a process for detecting whether an item is slipping, according to certain modalities. In some modalities, a robotic system 101 implements process 500, as is the case with robotic system 101. In some modalities, the 500 process is implemented to perform some or all of the 406 stage of the 400 process. In 502, the corresponding outputs of a plurality of sensors are monitored. A robotic system includes a robotic arm end effector and one or more tactile sensor units associated with the fingers of the robotic arm end effector. The robotic arm end effector may include a wrist portion that incorporates one or more sensors. A tactile sensing unit includes one or more sensors as described in this document. When the end effector of the robotic arm grasps an item, each of the plurality of sensors emits corresponding values. A processor can determine the corresponding values ​​for a plurality of modalities based on the sensor values. For example, the processor can determine a weight value, a deformation value, a continuity value, a conductivity value, a pressure value, a resistance value, an inductance value, or a capacitance value, etc. The detected values ​​may fluctuate while the robotic arm's end effector is gripping an item. The processor can monitor the measured values ​​over time as the robotic arm's end effector moves the item from the first location to the second location to determine if the item is beginning to slip and / or is slipping out of the robotic arm's end effector's grip. In some modes, one or more detected values ​​associated with the robotic arm end effector indicate linear slip associated with an item being grasped by the robotic arm end effector. In some modes, one or more detected values ​​associated with the robotic arm end effector indicate rotational slip associated with an item being grasped by the robotic arm end effector. In 504, a feature vector is introduced into a multimodal model. The elements of the feature vector can include one or more detected values ​​and / or one or more modality values ​​determined based on one or more detected values. The multimodal model can be a rule-based model, a predictive model, a machine learning model (e.g., neural network, linear classifier, support vector machine, linear regression, logistic regression, decision tree, deep learning, etc.), etc. In some modes, the processor compares one or more detected values ​​and / or one or more predetermined mode values ​​(κι) with one or more corresponding first thresholds to determine whether the item is beginning to slip out of the robotic arm's end effector grip. In some modes, the processor compares one or more detected values ​​and / or one or more predetermined mode values ​​with one or more corresponding 15-second thresholds to determine whether the item is slipping out of the robotic arm's end effector grip. In some modes, the processor classifies the detected 20 values ​​based on a combination of element values ​​within the mode value. The feature vector can be located in a feature space based on the element values ​​of the mode value. The processor can classify the feature vector as indicative of no item slippage, items beginning to slippage, and / or items slipping, based on the feature vector's location in the feature space. The multimodal model can be trained to classify whether a slip associated with an item is linear or rotational. For example, the end effector of a robotic arm can grasp an item multiple times and allow the item to experience linear slippage so that a range associated with the detected values ​​indicating linear slippage can be determined. A range can also be determined for one or more of the modality values ​​determined based on the detected values ​​indicating linear slippage. This process can be repeated for items with different shapes, sizes, characteristics, and weights. This process can also be repeated at different grasping locations on an item. It can also be repeated for different grasping techniques. The robotic arm's end effector can grasp an item multiple times and allow the item to undergo rotational slip so that a range associated with the detected values ​​indicating rotational slip can be determined. A range can also be determined for one or more of the modality values ​​determined based on the detected values ​​indicating rotational slip. This process can be repeated for items with different shapes, sizes, characteristics, and weights. It can also be repeated at different grasping locations on an item and for different grasping techniques. In 506, it is determined whether the gripped item is beginning to slip out of the robotic arm's end-effector grip or whether it is slipping out of the robotic arm's end-effector grip. In some modalities, the multimodal model is configured to output a binary decision regarding whether the item is slipping out of the robotic arm's end-effector grip. In some modalities, the multimodal model is configured to output a probability of whether the item is beginning to slip out of the robotic arm's end-effector grip. In some modalities, the multimodal model is configured to output a probability of whether the item is slipping out of the robotic arm's end-effector grip. If the multimodal model output indicates that the item is beginning to slide out of the robotic arm's end effector gripper or is sliding out of the robotic arm's end effector gripper, process 500 continues to 512. If the multimodal model output does not indicate that the item is beginning to slide out of the robotic arm's end effector gripper or is sliding out of the robotic arm's end effector gripper, process 500 advances to 508, where the item is moved to a delivery area. In 510, it is determined whether the robotic arm's end effector is located in the delivery area. If the robotic arm's end effector is not located in the delivery area, process 500 returns to 502. If the robotic arm's end effector is located in the delivery area, process 500 continues to 514, and the item is placed in the delivery area. In 512, the robotic system performs one or more response actions. In some modalities, the end effector places the item in a safe location and grasps it again. In some modalities, the speed at which the robotic arm moves the item is reduced. In some configurations, the robotic arm adjusts the end effector's orientation to prevent or reduce slippage. For example, the end effector might grasp an item from its sides (e.g., the left and right sides). The robotic arm can simply adjust the end effector's orientation by 90 degrees so that it grasps the item from both its top and bottom sides. This can reduce the influence of gravity that causes the item to slip from the end effector's grip. In some configurations, the end effector applies an additional gripping force. The amount of additional force applied may depend on the fragility of the item being gripped (e.g., the amount of additional force that can be applied without breaking the item). The fragility of the item being gripped can be determined based on strain values ​​measured when the end effector grips the item. These strain values ​​can indicate whether the item being gripped is fragile. In some configurations, the output from the camera(s) and / or sensor(s) associated with the robotic system, such as weight sensors, can indicate the type of item being gripped (e.g., a cardboard box, a plush toy, a glass, etc.), and the amount of additional force applied may depend on the item type.In some modes, the amount of additional force applied depends on whether linear or rotational slippage is determined to be occurring. In some modes, the amount of additional force applied may depend on a material associated with the grasped item. The material associated with the grasped item can be determined based on a detected capacitance associated with the detected item. The robotic system can be trained to apply additional force to a grasped item when the item begins to slip or is slipping from the end effector's grip. A variety of forces can be applied to items of different shapes, sizes, types, and weights. A variety of forces can be applied to different gripping locations on an item. A variety of forces can be applied for different gripping techniques. The robotic system can learn a threshold amount of force that will not cause damage to a particular grasped item when additional force is applied. In several configurations, the robotic system is set up to learn, for example, through machine learning techniques, how much additional force is required to be applied to a grasped item if the item begins to slip or slips out of the end effector's grip. The robotic system can also be configured to determine whether the additional force damaged the item. Feedback can be provided to the robotic system indicating whether or not it damaged the item. This feedback can be given by a human operator or determined after the robotic system has placed the item in the delivery location. In some configurations, the system can use techniques described in this document to learn and employ strategies for grasping and / or moving an item from a source location to a destination location, including through controlled sliding. This involves allowing the item to slide into and, while remaining within the grip of the end effector, to reposition and / or reorient the item by allowing it to slide within the end effector. For example, an item can be allowed to slide downward within the reach of the end effector, for instance, to increase stability or to move some of the item's mass and / or volume to a position below the end effector.In another example, controlled sliding can be used to allow an item grasped at a certain distance from its center of gravity, such as at the end of an elongated item, to rotate around an axis by which it has been grasped, thus changing the item's orientation within reach of the end effector. In various modalities, such maneuvers can be included in a set of grasping strategies available for grasping the item and moving it to a destination.The robot can use such a strategy to grasp an item by one end protruding from a stack of other items, for example, and use a controlled sliding maneuver to reposition or reorient the item to a position that is more stable and / or more likely to ensure that the item does not come into contact with an obstacle located along a path that the robot has planned to move the item from the origin location to the destination location. Figures 6A-6C are diagrams illustrating an example of a gripping and placing operation that includes a controlled slide. Figure 6A depicts an environment 600 that includes a robotic system with an end effector 602 gripping an item 604. The item 604 is positioned above the item 606. The environment 600 may include one or more cameras (not shown) that enable the robotic system to determine the approximate dimensions of the items 604 and 606. A processor in the robotic system determines one or more gripping strategies for item 604. The one or more gripping strategies may include gripping item 604 from the sides of item 604 or from the top of item 604. The processor may determine that end effector 602 cannot grip item 604 from the top of item 602 due to physical limitations of the robotic system (e.g., the robotic arm cannot be positioned so that end effector 602 can grip item 604 from the top of item 604). The robotic system processor may select a gripping strategy that includes end effector 602 gripping item 604 from the sides of item 604. Figure 6B depicts an environment 630 in which the robotic arm of the robotic system has lifted item 604 from item 606. The end effector 602 includes one or more tactile sensing units (not shown). The robotic system determines that one or more forces and / or one or more moments are applied to the end effector 602 while the end effector 602 is grasping item 604. The robotic system can determine a current weight distribution of item 604 based on one or more determined forces and / or one or more determined moments. The robotic system can determine to perform a controlled slide of item 604 to change the orientation of item 604 relative to the end effector 602 so that the end effector 602 uses less force to grasp item 604. For example, the current weight distribution of item 604 may cause the robotic system to drop item 604.In some embodiments, the robotic system may determine to change the orientation of item 604 as part of a grasping strategy before grasping item 604. In some embodiments, the robotic system may determine to change the orientation of item 604 while the end effector 602 is grasping item 604. 20. Figure 6C depicts an environment 650 in which the robotic arm of the robotic system has moved item 604 to an intermediate position. In the intermediate position, in the example shown, the robotic system has completed a controlled glide maneuver to reposition item 604 in the orientation shown. The maneuver included, in this example, adjusting the force applied by end effector 602 to article 604 so that gravity would cause article 604 to rotate from a horizontal position to a vertical position as shown, without sliding out of reach of end effector 602. The maneuver may have been planned and executed, for example, to allow the robotic system to move item 604 through a restricted space and / or use less force to grip item 604. The robotic system may move item 604 to a delivery area after the orientation of item 604 has been modified. FIG. 7 is a flowchart illustrating a process for grasping an item according to some modalities. In some modalities, a robotic system 101 implements process 700, as in the case of robotic system 101. In some modalities, process 700 is implemented to perform some or all of steps 404 / 406 of process 400. In 702, a strategy is developed to move an item 2D grasping. A robotic system uses detected values ​​from the tactile sensor unit(s) and a multimodal model to determine and implement a strategy to reorient and / or otherwise reposition an item for grasping by the end effector. In some modalities, the item is located in a jumbled stack. In some modalities, an orientation or position of the grasped item prevents a robotic arm from moving the grasped item without accidentally touching other items in the jumbled stack. In some modalities, the required gripping force is too strong to lift an object with an orientation in which the object's center of mass is far from the grasping contact points. A strategy for moving the grasped item may include performing an initial grasp of the item and then readjusting the grip after the item has been moved a threshold distance away from the other items. In 704, the item is moved to an intermediate location. The intermediate location can be a threshold distance from other items in the disordered pile. The intermediate location can be determined based on the dimensions associated with the grasped item and other detected items. In 708, the item is allowed to slide according to the gripping strategy while maintaining a grip on the item. The robotic system intentionally relaxes its grip on the item to a point where the item begins to slide in an expected and intended manner, such as sliding down, between the fingers or other mating structures of the end effector, or rotating while still being held by the end effector. Once the item is in the desired position and / or orientation, additional force is applied as needed to stop and prevent further sliding. In 710, the item is moved to the delivery area. Figure 8 is a flowchart illustrating a process for training a multimodal model. In some modalities, a robotic system 101 implements process 800, as is the case with robotic system 101. In experiment 802, the goal is to grasp and move a variety of items. The robotic system can apply a range of forces to items of different shapes, sizes, types, and weights to learn a fundamental principle. A variety of forces can be applied to different gripping locations for a variety of different items. A variety of forces can be applied using different gripping techniques. The robotic system can include a real-world sensor on the ground. For example, the robotic system can use a motion capture system to track each object and determine the force range combined with different gripping locations that can result in the desired pick-and-place behavior. A force sensor on the wrist can be used to measure whether the item was successfully picked up after each pick-up. Accelerometers and / or gyroscopes can be placed on the items, allowing the robotic system to determine if the gripped item is slipping. The robotic system can learn whether a gripped item slipped or not based on the output from the accelerometers and / or gyroscopes. The robotic system can learn whether a grip was successful or unsuccessful. In step 804, the sensed values ​​are determined. The end effector may include a plurality of sensors. Each sensor outputs a sensed value when an item is grasped and moved to a delivery location. The robotic system can learn the amount of force that will result in a successful or failed grasp for a particular item; that is, the robotic system can lift the grasped item from its initial position. The robotic system can use the sensed outputs to determine one or more modes. As the robotic system moves the grasped item to a different location, it can also learn whether the amount of force applied caused the item to slip from a robotic arm end effector. The robotic system can determine whether the grasped item slips based on the output detected from multiple sensors and / or one or more predefined modalities. In some configurations, the system can learn to apply greater force and / or tolerate more slippage when first grasping an item and pulling it from a stack or other source of items, for example, to allow the item to be removed from overlapping items to move to a destination. In some configurations, the item is moved in a specific way, at least partially, to minimize slippage and / or prevent it from slipping completely out of the robot's grasp. In version 806, a multimodal model is updated. The robotic system can associate a successful or failed grip with the detected values ​​and / or one or more modalities determined based on the detected values. The robotic system generates and saves profiles of successful and failed grips. This allows the robotic system to compare current detected values ​​and / or current factor values ​​with previous detected values ​​and / or factor values ​​to determine whether a current grip will be successful (the item moved to the delivery position without being dropped) or unsuccessful (the item slipped / fell before being moved to the delivery location). Although the above embodiments have been described in some detail to facilitate understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The embodiments described are illustrative and not restrictive.

Claims

1. A robotic system, comprising: 5 a plurality of sensors, each of the sensors in the plurality configured to generate respective outputs reflecting a detected value associated with the engagement of a robotic arm end effector with an article; and ki a processor coupled to the plurality of sensors and configured to: use the corresponding outputs of one or more sensors comprising the plurality of sensors to 15 determine one or more inputs to a multimodal model configured to generate an output associated with the sliding of the article into or out of a grip of the robotic arm end effector, based at least in part on one or more inputs; based at least in part on an output of the multimodal model, make a determination associated with the sliding of the article into or out of the reach of the robotic arm end effector;and take a response action based at least in part on the determination associated with the item sliding into or out of the reach of the robotic arm's end effector.

2. The robotic system of claim 1, wherein the touch sensing unit includes a plurality of sensors, wherein the touch sensing unit is composed of a plurality of layers, wherein each of the plurality of layers includes one or more sensors from the plurality of sensors.

3. The robotic system of claim 1, wherein one or more inputs include one or more modes.

4. The robotic system of claim 3, wherein one or more embodiments include one or more of the characteristics of weight, deformation, continuity, conductivity, resistance, inductance, and capacitance.

5. The robotic system of claim 1, wherein the output of the multimodal model indicates that the item is beginning to slide out of the grip of the end effector of the robotic arm.

6. The robotic system of claim 1, wherein the output of the multimodal model indicates that the item is being slid out of the gripper of the robotic arm's end effector.

7. The robotic system of claim 1, wherein the processor is further configured to monitor the respective outputs of one or more sensors while the end effector moves the item from a first location to a second location.

8. The robotic system of claim 1, wherein the deformation value determined based on the respective outputs of at least one of one or more sensors indicates whether the article is slipping from the gripper of the end effector of the robotic arm.

9. The robotic system of claim 1, wherein the determined deformation value is based on the respective outputs of at least one of one or more of the sensors that 2o indicate whether the article is beginning to slip out of the grip of the end effector of the robotic arm.

10. The robotic system of claim 1, wherein the multimodal model is configured to classify the article's 25 sliding motion as linear sliding. 100 11. The robotic system of claim 1, wherein the multimodal model is configured to classify the article's sliding as rotational sliding.

12. The robotic system of claim 1, wherein the response action includes moving the article from a first location to a second location.

13. The robotic system of claim 1, wherein the response action includes placing the item in a location and re-grasping the item.

14. The robotic system of claim 1, wherein 15 the response action includes adjusting an orientation of the end effector of the robotic arm.

15. The robotic system of claim 1, wherein the response action includes increasing a force applied to the 2nd article by the end effector of the robotic arm.

16. The robotic system of claim 15, wherein the force applied by the end effector of the robotic arm is based on a fragility associated with the article. 101 17. The robotic system of claim 1, wherein the force applied by the end effector of the robotic arm is based on whether an output from the multimodal model indicates that the article's glide is a linear glide or a rotational glide.

18. The robotic system of claim 1, wherein the processor is further configured to use the respective Io outputs of one or more sensors to initiate the sliding of the article into or out of the reach of the end effector of the robotic arm and to adjust the reach of the end effector of the robotic arm relative to the article. 15 19. A method comprising: using corresponding outputs from one or more sensors comprising the plurality of sensors to determine one or more inputs to a multimodal model configured to generate a 20 output associated with the slippage of the item, based at least in part on one or more inputs, wherein each of the plurality of sensors is configured to generate a corresponding output reflecting a detected value associated with the engagement of the robotic arm's end effector with the 25 item; based at least in part on an output from the multimodal model, making a determination associated with the slippage of the item into or out of reach of the robotic arm's end effector; and taking a response action based at least in part on the determination associated with the slippage of the item into or out of reach of the robotic arm's end effector.

20. A product of a computer program contained on a non-transient, computer-readable medium comprising computer instructions for: 15 using corresponding outputs from one or more sensors comprising a plurality of sensors to determine one or more inputs to a multimodal model configured to generate an output associated with the article's slippage, based at least in part on one or more inputs, wherein each of the plurality of sensors is configured to generate a corresponding output reflecting a detected value associated with the engagement of the robotic arm's end effector with the article; 103 based at least in part on an output from the multimodal model, making a determination associated with the article's slippage into or out of the reach of the robotic arm's end effector;and take a response action based at least in part on the determination associated with the item sliding into or out of the reach of the robotic arm's end effector.

Images