Autonomous unknown object picking and placing

Abstract

Based on visual data received from a plurality of cameras, a set of one or more potentially graspable features of one or more objects present in a workspace area is determined. For each of at least a subset of the one or more potentially graspable features, one or more corresponding grasping strategies are determined to grasp the features with a robotic arm and an end effector. With respect to each of at least a subset of the grasping strategies, a score associated with a probability of successfully grasping the corresponding feature is determined. A first feature of the one or more potentially graspable features to be grasped using a selected grasping strategy is selected, based at least in part on a corresponding score associated with the selected grasping strategy with respect to the first feature. The robotic arm and the end effector are controlled to attempt to grasp the first feature using the selected grasping strategy.

Description

[0001] Cross-references to other applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 829,969, filed on April 5, 2019, entitled AUTONOMOUS UNKNOWN OBJECT PICK AND PLACE, which is incorporated herein by reference for all purposes. Technical Background

[0003] For many years, humans have been engaged in tasks that require moving objects from one location to another. The throughput of object movement relies on human strength, endurance, and skill. Tools and machines have been introduced to help increase the throughput of moving objects, but such tools and machines are usually operated or semi-operated by humans. Autonomous robots can be used to move objects from one location to another. Autonomous robots offer significant advantages over humans, such as the ability to work 24 hours a day, 7 days a week without taking breaks. Although autonomous robots offer advantages over human labor, the use of autonomous robots introduces new problems when trying to move objects from one location to another. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0005] Figure 1 is a block diagram illustrating a system for autonomous pick and place of objects in accordance with some embodiments.

[0006] Figure 2 is a flow chart illustrating a process for picking and placing objects according to some embodiments.

[0007] Figure 3 is a flow chart illustrating a process for grasping and moving an object according to some embodiments.

[0008] Figure 4 is a flow chart illustrating a process for grasping a feature according to some embodiments.

[0009] Figure 5 is a flow chart illustrating a process for selecting a gripping technique according to some embodiments.

[0010] Figure 6 is a flow chart illustrating a process for grasping and moving an object according to some embodiments.

[0011] Figure 7 is a flow chart illustrating a process for moving an object according to some embodiments.

[0012] Figure 8is a flow chart illustrating a process for updating the operation of a robotic system according to some embodiments.

[0013] Fig. 9 is a block diagram illustrating a suction-based end effector according to some embodiments.

[0014] Fig.10 is a block diagram illustrating a gripper-type end effector according to some embodiments. DETAILED DESCRIPTION

[0015] The present invention may be implemented in a variety of ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these implementations or any other form that the present invention may take may be referred to as techniques. In general, the order of the steps of the disclosed processes may be changed within the scope of the present invention. Unless otherwise stated, a component such as a processor or memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform a task at a given time or as a specific component manufactured to perform a task. As used herein, the term 'processor' refers to one or more devices, circuits, and / or processing cores that are configured to process data (such as computer program instructions).

[0016] Provided below is a detailed description of one or more embodiments of the present invention and the accompanying drawings illustrating the principles of the present invention. The present invention is described in conjunction with such embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications and equivalents. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. These details are provided for illustrative purposes, and the present invention can be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical materials known in the technical field related to the present invention are not described in detail so that the present invention will not be unnecessarily obscured.

[0017] A robotic system is tasked with autonomously picking up and placing an unknown object from a first location to a second location. The robotic system may use visual data from one or more cameras to make the object visible at the first location. However, in some cases, the objects may be piled up in an arbitrary and / or messy manner, making it difficult or impossible to discern object boundaries and / or obtain shape and position information for all objects in a workspace area.

[0018] In various embodiments, a robotic system as disclosed herein uses a robotic arm with an actuator (e.g., gripper, suction, etc.) at an operating end to pick up arbitrary, previously unknown objects from a first location and then place them at a second location. The throughput of the robotic system depends on the ability of the robotic system to successfully grasp the object on the first attempt and move the object to a drop zone without damaging the object during the process. However, the objects may be haphazardly piled together at the first location, making it challenging to determine the object boundaries based on a camera. The robotic system may grasp the object at the wrong location, causing the robotic system to drop the object, which may damage the object at the first location and / or one or more other objects.

[0019] Other systems may specifically identify all objects located at the first location before starting the pick and place process (e.g., the object is a specific type of soap, the object is a specific type of stapler, etc.). However, such a process reduces the throughput of the robotic system because the computation time required to identify all objects may take a long time to execute. In addition, if an object is incorrectly identified, the robotic system may restart its identification process before the pick and place process resumes.

[0020] The technology disclosed herein is used to increase the throughput of a robotic system that autonomously picks up an unknown object from a first location and places the unknown object to a second location. The robotic system is coupled to a plurality of cameras for viewing a workspace area (e.g., a first location). One or more objects are determined to be in the workspace area. An object of the one or more objects is composed of arbitrary colors, geometric shapes, textures, etc.

[0021] The geometry of each of the one or more objects is determined based on the data of the multiple cameras. For example, the data of the multiple cameras may include point cloud information. Based on the geometry of the one or more objects, potential graspable features (e.g., handles, protrusions, gaps, etc.) are determined for each of the one or more objects in the workspace area.

[0022] A corresponding grasping strategy is determined for each of the graspable features. For example, the robotic system can store one or more grasping techniques for features having various corresponding shapes and / or sizes. In some embodiments, the feature corresponds to a portion of an object. In some embodiments, the feature corresponds to the entire object.

[0023] A corresponding score of successful grasps is determined for each of the grasping strategies determined with respect to each corresponding potentially graspable feature. One of the grasping strategies is selected based on its score, and an attempt is made to pick up an object associated with the selected grasping strategy. In the event that the robotic system successfully grasps the object, the robotic system moves the object from the workspace area to a second location (e.g., a drop zone). In the event that the robotic system fails to successfully grasp the object, the robotic system attempts an alternative grasping strategy for the object or attempts to grasp a different object. The process of picking up objects from the workspace area and placing the objects into the drop zone continues until all objects from the workspace area have been placed in the drop zone.

[0024] The robotic system can increase the throughput of the robotic system by associating objects with different shapes and using a grasping strategy for each of the different shapes. Such a technique is applicable to any set of objects and does not require the robotic system to be programmed for a specific set of objects before picking and placing the objects.

[0025] Figure 1 1 is a block diagram illustrating a system for autonomously picking and placing objects according to some embodiments. In the illustrated example, a robotic system 101 operates in an environment 100. The robotic system 101 includes a plurality of articulated portions, including a robotic arm 102 mounted on a fixed base 104, an end effector 108, one or more sensors 134, and a controller 106. The robotic arm 102 is coupled to the controller 106, which is configured to manipulate the robotic arm 102 and the end effector 108 mounted on the distal end of the robotic arm 102. In some embodiments, the controller 106 controls the robotic arm 102 and the end effector 108 by providing voltage and / or other signals, inputs, etc. to motors configured at each of the respective joints between the rigid elements including the robotic arm 102 and / or the end effector 108, so that the respective motors apply corresponding (one or more) torques to move an element coupled to a rotating element of the motor relative to an element coupled to a non-rotating element of the motor. The end effector 108 may include a suction gripper, a parallel gripper, a soft gripper, a dexterous gripper, etc. The robotic system 101 may include multiple end effectors and select the end effector that best fits the object being gripped. For example, the end effector may be selected based on the texture of the object. In the case where the object has too many wrinkled areas, the robotic system 101 may select a parallel gripper instead of a suction gripper.

[0026] exist Figure 1In the example shown, the robotic arm 102 is used to pick up objects, including objects 112, 114, and 116 of different shapes in the example shown, from a table or other surface 110 (e.g., a workspace area) and place them on a conveyor belt 118 (e.g., a drop zone). As shown, the robotic arm 102 has previously been used to place an item 120 on the conveyor belt 118, which rotates in one direction so that the object 120 is about to fall from the conveyor belt 118 to a destination 122. The workspace area can include a mobile platform, such as a conveyor belt or a rotating platform, or a fixed area in which a pile of objects (stable or unstable) are located.

[0027] In various embodiments, Figure 1 The "pick and place" operation shown in FIG. 1 is performed at least partially in an autonomous operating mode by a robotic system 101 including a robotic arm 102, an end effector 108, and a controller 106. For example, in some embodiments, the controller 106 and / or one or more other control devices, such as a computer including a processor, a memory, and other components, are programmed to perform Figure 1 For example, in some embodiments, a programmer or other operator may have programmed or otherwise configured the robotic system 101 to be aware of its environment 100 and its position relative to objects on the table 110 (or in some embodiments, a set of coordinates or other positions associated with the table 110 and conveyor belt 118 on the one hand).

[0028] In some embodiments, the robotic system 101 is programmed or otherwise configured to use a library or other repository of strategies to perform pick and place operations and / or portions thereof. For example, the robotic system 101 can be configured to use its current position and knowledge of the environment 100 to position the end effector 108 at a location above the table 110. Computer vision or other techniques can be used to identify and select the object to be picked up next, and a strategy for picking up an object can be selected autonomously, for example, based on one or more of the object's position, shape, orientation, aspect, texture, rigidity, etc.

[0029] For example, in Figure 1 In the example shown, the robotic system 101 may have identified the feature associated with object 112 as having a cube geometry and selected a grasping strategy for a cube geometry before picking up object 112. The robot may identify object 114 as having a pyramid geometry and select a grasping strategy for a pyramid geometry. The robot may identify object 116 as having a cylindrical geometry and select a grasping strategy for a cylindrical geometry.

[0030] Environment 100 includes multiple cameras, such as cameras 115, 117. Although Figure 1 Environment 100 is depicted with two cameras, but environment 100 may include n cameras, where n is a number greater than 1. The multiple cameras may be wired or wirelessly coupled to robotic system 101. In some embodiments, at least one of the multiple cameras is located at a fixed location. In some embodiments, at least one of the multiple cameras is in dynamic motion (e.g., attached to a moving object, such as a drone). In some embodiments, at least one of the multiple cameras can be fixed and moved to different locations (e.g., an object is detected at a first location, the camera is moved to a second location, and the object is detected at the second location). In some embodiments, different lighting conditions are used in environment 100 to detect changes in perceived surface features of one or more objects.

[0031] The use of multiple cameras enables the robotic system 101 to observe the environment 100 from different vantage points. This prevents objects from being blocked and gives a more accurate estimate of object geometry and object boundaries. For example, a large object may be placed in such a way that it prevents the camera from seeing smaller objects next to the large object. Using multiple cameras from different positions enables smaller objects to be seen and boundary information associated with smaller objects to be determined. A single camera may not be able to cover a large workspace area. The views associated with multiple cameras can be merged to give the robotic system 101 a more complete view of the workspace area 110. In the event that one of the cameras is blocked, the robotic system 101 is still able to pick up and place objects. The use of cameras also enables the robotic system 101 to determine whether the robotic system needs to be recalibrated as described herein. For example, ArUco markers (e.g., binary square fiducial markers) can be used to initially align multiple cameras.

[0032] In some embodiments, the robotic system 101 segments objects based on point clouds generated by one or more of the multiple cameras. The robotic system 101 can segment objects based on RBG or multispectral camera images (e.g., a combination of RGB, depth, and / or infrared, etc.). The segmented objects can be back-projected into a point cloud so that potential graspable areas can be determined. This provides additional information, such as object type, expected weight / material, preferred grasping strategy, etc., which is not available when segmenting objects based on point cloud information alone. This combined segmentation strategy works well when picking up objects that are difficult to distinguish using depth alone (e.g., small boxes packed tightly together may look like a single plane as a point cloud), but using image segmentation combined with point cloud information, the robotic system 101 can identify each box and extract the box from the input.

[0033] In some embodiments, the robotic system 101 autonomously picks up an unknown object from a table 110 (e.g., a workspace area) and places it on a conveyor 118 (e.g., a drop zone). The robotic system 101 may determine that the objects 112, 114, 116 are located on the table 110 by using the cameras 115, 117. The controller 106 determines geometric shape information based on visual data (e.g., point cloud data) received from the cameras 115, 117. The controller 106 selects corresponding potential graspable features for the objects 112, 114, 116 that correspond to the geometric shape information determined based on the visual data received from the cameras 115, 117. For example, based on the visual data received from the cameras 115, 117, the controller 106 may determine that the object 112 includes a graspable feature corresponding to a cube shape, the object 114 includes a graspable feature corresponding to a pyramid shape, and the object 116 includes a cylindrical shape. The controller 106 may select the graspable feature that most closely resembles the geometric object within a threshold amount. For example, the controller 106 can compare the determined geometric shape information with a known feature library and select a feature of the object based on the comparison. In some embodiments, the feature is a canonical shape. The controller 106 can superimpose the canonical shape on the object to be grasped.

[0034] In order to determine one or more graspable features associated with the object, the controller 106 can randomly cut the plane of the object to decompose the object into multiple sub-segments. The object can be cut at the plane where the data points of the point cloud occupy the least (related to the sharp features on the top of the grasped object). The plane of the object can be cut based on a strong gradient of the color or appearance of the object. In some embodiments, a membership function is used to determine whether there are outliers in the point cloud within the sub-region generated by the category. Additional cutting planes can be added, or the object can be split into isolated areas with high residuals. The sub-segments can be processed separately. For example, outlier detection techniques can be applied to the sub-segments. In some embodiments, 5-sigma fits a Gaussian distribution to these points and identifies points that are 5-sigma (standard deviation) away from the mean, and marks the identified points as outliers. In some embodiments, a subsampling method is used for the point cloud and refitted to a mean. These points are then used to find points that are a certain distance away from the mean.

[0035] In some embodiments, sub-segments of the object are determined based on the extent to which the end effector 108 interacts with the object. For example, if the end effector 108 cannot grasp across a wide object, the controller 108 determines not to grasp the object around the wide portion of the object. If a suction gripper end effector is used, relatively smooth flat surfaces are sought. Gap-based picking strategies or minimum occupied cutting planes may be avoided. Primitives are refitted into the new partitioned cloud. This process may be repeated iteratively until a certain quality level or recursion limit is met.

[0036] The controller 106 can determine negative space information (e.g., voids) associated with an object based on visual data received from the cameras 115, 117. For example, the controller 106 can determine that the handle of a coffee cup includes negative space or that a car tire includes negative space. Computer vision algorithms using data from multiple cameras can determine voids (e.g., holes) in objects such as cups, mugs, rolled wires, tape, etc. In the event that a void is detected, the object can be grasped by inserting a gripper into the void and picking up the object from the side wall of the object.

[0037] The controller 106 may determine the curvature of the object to be picked up based on the visual data received from the cameras 115, 117. In the event that the controller 106 determines that the object is curved, the controller 106 may change the control strategy associated with placing the object so that the curved object is placed more carefully and released more slowly to prevent the object from rolling away when placed. In the event that the visual data received from the cameras 115, 117 indicates that the placed object is rolling or moving after initially releasing the grip of the object, the controller 106 may re-grip the object and attempt to place the object again before releasing the grip. In the event that the controller 106 attempts to grip / re-grip the object more than a threshold number of times, a warning may be provided to the user 130 and alert the user 130 that the object may roll away.

[0038] The controller 106 determines corresponding features associated with the objects 112, 114, 116 based on the visual data received from the cameras 115, 117. For example, the controller 106 may determine that the object includes a handle. The visual data received from the camera may be used to determine the minimum boundary associated with the object and the maximum boundary associated with the object. The boundary of the object may include the height, width, or depth associated with the object. The visual data may provide data that allows one or more of the boundaries of the object to be determined. For example, a first camera may face the object at a first angle. The first camera may be able to provide information about the height and width of the object, but not the depth information of the object. The second camera may face the object at a second angle. The second camera may be able to provide information about the depth and height of the object, but not the width of the object. The controller 106 may combine the visual data received from multiple cameras to determine boundary information (estimated, approximate, or precise) associated with the object. For example, the controller 106 may determine the height and width of the object, but not the depth. The controller 106 may determine that the depth of the object is within a certain value range.

[0039] The minimum bounds associated with an object correspond to the minimum value that the object may have in a particular dimension. The maximum bounds associated with an object correspond to the maximum value that the object may have in a particular dimension. For example, a first camera may detect a first object and a second object behind the first object. The controller 106 may determine, based on visual data from the first camera, that the second object is 12 inches behind the first object. In the case where the first object and the second object are in contact, the maximum value that the first object may have in the depth dimension is 12 inches. The minimum value that the first object may have in the depth dimension is a threshold minimum size for that dimension (e.g., 1 / 8 inch, 1 / 4 inch, 1 / 2 inch, 1 inch, etc.). The threshold minimum size may be based on historically known objects, known objects of similar size, object context, object type, etc.

[0040] The controller 106 is associated with a memory (not shown) storing a data structure associating a grasping strategy with a feature. A grasping strategy may consist of a grasping technique and how to grasp a feature using the grasping technique. In some embodiments, the grasping strategy includes grasping the major and minor axes of a bounding box that can fit into the geometric estimate of the object / fragment. In some embodiments, the grasping strategy includes cutting the object / fragment estimate at a certain Z height and recalculating the bounding box. The major and minor axes of the recalculated bounding box can then be grasped. This is useful when an object has a wide base but has a small tower somewhere in the middle and the robotic system wants to grasp the town accurately. The memory also stores instructions on how to perform grasping techniques. The instructions may include instructions for partially pre-closing the gripper if it is necessary to avoid affecting other objects. The memory also stores instructions on how to perform placement techniques. The instructions may include instructions to partially open the gripper fingers of the end effector 108 so that the end effector 108 does not interfere with other objects when placing the object in the drop zone. The memory also stores information about the mechanism and geometry of the end effector (e.g., parallel gripper versus suction gripper, width / length of gripper fingers, etc.).

[0041] The gripping technique may be associated with one or more features. For example, the suction technique may be used for an object having a grippable feature corresponding to a pyramid shape, a grippable feature corresponding to a cube shape, or a grippable feature corresponding to a rectangular prism shape. The parallel gripping technique may be used for an object having a grippable feature corresponding to a spherical shape. Features may be associated with one or more gripping techniques. For example, the parallel gripping technique or the scooping technique may be used for a grippable feature corresponding to a spherical shape. Different types of grippers may be used to grip features having a specific shape. For example, a first gripping technique may use a parallel gripper, and a second gripping technique may use a suction gripper. In some embodiments, the type of gripper switches autonomously between gripper types during a pick-and-place operation. The gripping technique may be used for different parts of a feature. For example, the parallel gripping technique may be used for the top, middle, or bottom portion of a feature. The controller 106 determines a corresponding score for each of the gripping strategies associated with the feature. In some embodiments, an object is associated with a plurality of features. The controller 106 may determine one or more gripping techniques for each of the plurality of features and determine the corresponding score of the determined gripping technique.

[0042] The score associated with the grasping strategy can be based on the probability that the grasping strategy will result in a successful grasp of the feature. The probability that the grasping strategy will result in a successful grasp of the feature may be based on an additional factor, such as contextual information about the environment, historical grasping information about the environment, the angle at which the robot arm grasps the feature (to avoid collisions with other objects), the height of the robot arm grasping the feature (to prevent collisions at the top of the gripper), the grasp width, the orientation of the surface normal at the grasp point, the amount of the feature that can be grasped, material properties, etc. Contextual information about the environment includes the presence of other objects near or adjacent to the object, the amount by which other objects near or adjacent to the object hinder the ability of the robot arm to grasp the feature, whether more objects are continuously added to the workspace area, etc. Material properties may include the center of mass of the object, the friction properties of the object, the color, the reflectivity, etc. For example, the robotic system 101 can construct a large support surface so that a large object can be stably placed. When the robotic system 101 detects that the object may slide off the inclined placement support surface given the friction coefficient of the object and the placement support surface, the robotic system 101 can be configured to only select to pick up objects with a sufficiently high friction coefficient (e.g., to avoid sliding).

[0043] The controller 106 selects one of the grasping strategies based on the corresponding scores associated with each of the grasping strategies. The objects may be a heterogeneous collection of objects placed in a cluttered pile. The objects may differ in size, color, robotight, geometry, texture, stiffness, etc. The objects are removed individually from the pile. Some of the objects are at least partially occluded. All objects in the cluttered pile are a priori unknown. The controller 106 selects the grasping strategy with the highest score. In the event that two or more grasping strategies have the same high score, the controller 106 selects one of the grasping strategies, picks up the features associated with the grasping strategy, moves the object to the drop zone, and then selects the remaining objects associated with other grasping strategies.

[0044] The controller 106 causes the end effector 108 to grasp features associated with the object. In the example shown, the controller 106 has caused the end effector 108 to grasp the object 112. The controller 106 can use the previous knowledge about the gripper mechanism and geometry to simplify the grasping prediction problem. For example, if the end effector 108 will approach an object, such as object 112, from above, the controller 106 analyzes the top section of the point cloud to identify graspable protrusions. In some embodiments, as the robotic system moves, the cameras 115, 117 collect more data (e.g., closer, different angles, different lighting, reflectivity, etc.) and the robotic system 101 adjusts how it causes the end effector 108 to grasp the object based on the new data.

[0045] A meshed or segmented version of the object may be used to determine the grip points of the object. A close approximation of the object to be gripped is built, and a model matched with a library or machine learning method is used to determine the best grip position of the object. The grip points are ranked. The controller 106 causes the end effector 108 to grip the object at one of the grip points.

[0046] In some embodiments, pressure and / or vacuum sensors are used to detect leaks to assess the quality of the grip. For example, the robotic system 101 can use a suction gripper as the end effector 108 to pick up torn plastic bags. The robotic system 101 can use the pressure / vacuum information to abort the picking action of these items, which can avoid damaging the suction gripper and / or contaminating the packaged goods in these plastic bags.

[0047] The end effector 108 moves the object (in this example, object 112) to a drop zone, such as a conveyor 118. The end effector 108 places the object in the drop zone. The robot system 101 can use multiple cameras to place the object at a location near where the robot system 101 thinks the object should be placed. The robot system 101 can lower the object at the location drop and detect when the robot system 101 feels the force of the drop zone to push it back. When the robot system 101 detects that the drop zone has been reached, the robot system 101 opens the end effector 108 or stops suction to gently put the object down. When the end effector 108 is opened, the robot system 101 can move up or down to control the placement force (sometimes opening the gripper when in contact may crush the object). This enables the robot system 101 to stack objects or dynamically adjust the placement height when the placement surface height estimation is prone to error or unknown. This also helps when other objects are in the way. In some embodiments, the robot system 101 determines whether any objects that may roll away are placed in the drop zone. In the absence of objects that may roll away, the controller 106 may control the robotic arm 102 and the end effector 108 to push objects already in the drop zone closer together, so that space is created to place one or more other objects.

[0048] In various embodiments, the robotic system 101 including the robotic arm 102, the end effector 108, and the controller 106 automatically prompts intervention via teleoperation. Figure 1 During the illustrated pick and place operation, the robotic system 101 reaches a state where the robotic system 101 cannot determine a (next) strategy for (further) performing the operation, and the robotic system 101 prompts a remote operator (in this example) for assistance via teleoperation.

[0049] In the example shown, the controller 106 is connected to a teleoperation computer 126 via a network 124. In some embodiments, the teleoperation computer 126 can participate in the operation of the robotic system 101 in an autonomous mode, for example, by transmitting high-level instructions to the controller 106 via the network 124. In various embodiments, one or both of the controller 106 and the teleoperation computer 126 can prompt teleoperation intervention, for example, if the robotic system 101 reaches a state where it does not have a strategy available to perform (complete) the next task or step in the operation.

[0050] For example, further reference Figure 1, if object 114 is dropped in an orientation that presents a triangular formation to the robot and lands on one of its flat sides, then in some embodiments, the robotic system 101 may not have a strategy available to pick up object 114 and / or may have timed out or exhausted a configured number of attempts to pick up object 114. In response, the teleoperator 130 may be prompted to intervene via teleoperation and may use the manual input device 128 to control the operation of the robot. For example, the teleoperator 130 may manipulate the robotic system 101 to pick up object 114 and place the object on the conveyor belt 118. Alternatively, the teleoperator may use the robotic system 101 to change the orientation of object 114 to an orientation where the autonomous robotic system 101 would be expected (or more likely) to have a strategy available to pick up object 114.

[0051] In the example shown, teleoperation can be performed by manipulation of a manual input device 128 (e.g., a tactile input device) by a human operator 130. The human operator 130 (sometimes referred to as a teleoperator) can be prompted by information to start teleoperation, which is displayed via a display device including a teleoperation computer 126 and / or associated with the teleoperation computer 126. Data from one or more sensors 134 can be provided to the human operator 130 via the network 124 and the teleoperation computer 126. In some embodiments, the sensor 134 includes a camera (not shown) on the robot or cameras 115, 117, and is configured to generate a video feed displayed to the teleoperator 130 and used to perform and / or complete the operation or part thereof via teleoperation. In various embodiments, the camera is connected with a low-latency, high-throughput connection, including, by way of example and not limitation, one or more of analog RF-based communications, WiFi, Bluetooth, and Sub GHz. In some embodiments, a mixture of different types of cameras is used. For example, cameras with different communication rates, bandwidths, and / or other characteristics can be used, such as two RGB vision cameras, four depth cameras, two IR cameras, and the like.

[0052] In various embodiments, teleoperation can be performed using a variety of different sensors 134. In some embodiments, these can guide the robotic system 101 to determine if it is "stuck", and / or can simplify teleoperation. In some embodiments, sensors help transition teleoperation modes from direct tactile control to increasingly abstract execution commands (such as clicking an object with a mouse to pick up, or saying "open the shelf" to an audio transcription device).

[0053] Examples of sensors 134 used in various embodiments include digital switches that are configured to detect interactions with the environment and specific "stuck" scenarios, and / or the presence of unknown agents in proximity to the robotic system 101 (or teleoperator). Additional examples include force or pressure sensors on a hand or robot that determine the success or failure of an operation (such as a grasp). After some series of failures, the robotic system 101 determines that it is "stuck". Another example is one or more sensors, such as position sensors on a robot joint, that the robotic system 101 can use to understand whether a planned and / or otherwise intended trajectory of motion is being accurately followed. When it is not accurately following the expected trajectory, it is likely that it has made contact with the environment 100 and the robotic system 101 can be programmed to conclude that it has been "stuck" and that human intervention needs to be invoked.

[0054] The vision system including multiple cameras is configured to keep tracking each object in the workspace area through multimodal means (e.g., RGB instance tracking, RGB feature matching, RGB optical flow, point cloud matching, etc.), and keep tracking the objects to be picked up by the robot system 101 using methods such as Hungarian pair matching. The robot system 101 is configured to estimate the state of each tracked object, such as speed, possibility of falling / sliding away, and motion trajectory. The robot system 101 can use other known information, such as the current speed and size of the conveyor system and sensors 134, to update the object state with higher accuracy. The robot system 101 can use the determined object state to make informed decisions about where to pick up and what objects to pick up and where / when / how to place the objects. For example, the robot system 101 can select a more stable object to pick up (grasp) and may pick up from the estimated object position in the future (even while moving) to compensate for the movement time of the robot arm 102 and the speed of the moving object. The robotic system 101 can place the object onto the mobile platform drop zone more stably by placing the object at an initial velocity as estimated from the environment 100, without falling and causing the object to roll. The robotic system 101 can also select a collision-free zone to place the object in the drop zone 118. The collision zone can be determined from the estimated trajectory of the tracked object. Using data associated with multiple cameras, the robotic system 101 is able to understand the shape of the grasped object and the environment 100. This enables the robotic system 101 to intelligently plan a trajectory that will avoid collisions between the picked object and the environment 100.

[0055] In some embodiments, multiple robotic systems work together to pick up and place objects. Using multiple robotic systems can increase the overall throughput of the system.

[0056] Figure 2is a flow chart illustrating a process for picking and placing an object according to some embodiments. In some embodiments, process 200 is implemented by a robotic system such as robotic system 101 .

[0057] At 202, sensor data, such as image data, associated with one or more objects located in a workspace area is received. In some embodiments, the sensor data is generated by multiple cameras. The multiple cameras are configured to observe and detect the one or more objects from different vantage points. In some embodiments, one or more of the cameras generate one or more point clouds for the one or more objects. Where multiple point clouds are generated, the multiple point clouds are merged together. In various embodiments, the one or more objects may include multiple objects placed in a cluttered pile, multiple objects spaced apart, and / or multiple objects where one or more are obscured from view by one or more other objects in the multiple objects.

[0058] Geometric information is determined for each of the plurality of objects. The geometric information may be determined based on point cloud information obtained from data associated with one or more of the plurality of cameras.

[0059] At 204, one or more potentially graspable features are determined for one or more objects. Corresponding geometry information associated with each of the plurality of features may be compared to a library of geometries known to its grasping strategy. A strategy associated with a geometry that most closely resembles the geometry of the determined feature (e.g., within a similarity threshold) may be selected. In some embodiments, an object is associated with a single feature (e.g., a paper towel roll corresponds to a cylinder). In some embodiments, an object is divided into a plurality of sub-segments (also referred to herein as sub-objects) and corresponding features are determined for each of the plurality of sub-segments (e.g., a golf club includes a body segment and a head segment).

[0060] Objects near the edges or corners of the workspace area may have physical or other limitations on where and / or how to pick up the object. In some embodiments, a non-optimal but feasible pick angle may be selected depending on the environmental boundaries associated with the object. A wider longitudinal grip may be selected over a narrower latitudinal grip because a wider longitudinal grip may keep the end effector within the environmental boundaries.

[0061] At 206, a corresponding score of successful grasps is determined for each of the determined grasping strategies. The robotic system can be configured to grasp an object or feature using different grasping tools (e.g., a suction grasper, a parallel grasper, other end effectors, etc.). The robotic system can grasp an object at different locations on the object using the grasping tools. For example, the grasping tools can be used to grasp an object at a top portion, a middle portion, or a bottom portion of the object. Certain grasping tools may be more successful than other grasping tools when grasping certain shapes.

[0062] The score of a successful grasp feature may be based on the probability that a grasping strategy will result in a successful grasp. The probabilities of different combinations of grasping tools (in embodiments where multiple tools are available) and grasping locations are determined. The probability that a grasping strategy will result in a successful grasp of an object may be based on an additional factor, such as contextual information about the environment, historical grasp information about the environment, the angle at which the robot arm will grasp the object (to avoid collisions with other objects), the height at which the robot arm will grasp the object (to prevent collisions at the top of the gripper), the grasp width, the surface normal orientation at the grasp point, the amount of the object that can be grasped, etc. Contextual information about the environment includes the presence of other objects near or adjacent to the object, the amount by which other objects near or adjacent to the object impede the robot arm's ability to grasp the object, whether more objects are continuously added to the workspace area, etc.

[0063] At 208, one of the determined grasping strategies is selected to be attempted based on the determined corresponding scores. The objects / features and corresponding grasping strategies are ranked based on the corresponding scores. The object / feature with the highest score is selected for grasping among the multiple objects. In the case where multiple possible grasps have the same score, one of the multiple grasps is selected. After the grasped object is moved, one of the other grasps with the same score is selected.

[0064] In the event that an object from multiple objects has been selected, grasped, moved, and placed in a drop zone, the grasp with the next highest score is selected to be attempted.

[0065] At 210, a selected grasp is performed / attempted. Using visual data from multiple cameras, the robotic system can determine whether the grasped object has moved. If not, the robotic system determines that the grasp has failed. In some embodiments, the end effector has a pressure sensor to determine whether a feature has been grasped. In some embodiments, the end effector uses capacitance to sense deformation of the skin surface of the object to determine whether a feature has been grasped. In some embodiments, the end effector is a suction gripper and the robotic system uses a pressure sensor to detect changes in suction pressure to determine whether a feature has been grasped.

[0066] In some embodiments, for thin objects or gripping features that are very close to the surface or require pinching (e.g., flat cloth), the robotic system can ensure that the end effector is in contact with the pick surface by sensing and controlling a specific contact force. This contact also provides the robotic system with an accurate estimate of the location of a single point on the pick surface, which can be used to improve the calibration of the robotic system.

[0067] At 214, it is determined whether the object associated with the selected grasp has been dropped when the object is moved from the workspace area to the drop area. The determination that the object has been dropped can be based on sensor measurements (e.g., pressure, force, capacitance, etc.) of sensors included in the end effector, which the robotic system uses to determine whether features associated with the object are grasped. The sensor measurements can be compared to a threshold to determine whether the object has been dropped.

[0068] In the event that the object has been dropped, process 200 returns to 208. In the event that the object has not been dropped, process 200 proceeds to 216.

[0069] At 216, the object is placed in the drop zone. The object can be placed in a manner that prevents the end effector from colliding with a boundary associated with the drop zone. In some embodiments, the object is placed in the clutter with other objects in the drop zone. The robotic system can randomly place the object in the drop zone. The robotic system can then use force sensors on the gripper to gently place the object on the clutter without causing the robotic system to perform a protective stop. The robotic system can add random perturbations to the drop zone in small areas to create a better distribution of objects. The robotic system can tightly pack the objects into boxes in the drop zone by using force sensors to achieve a tight slotting strategy.

[0070] In some embodiments, the object is placed to be spaced apart from other objects in the drop zone. The robotic system can divide the placement space in the drop zone into multiple sub-areas and place the selected object in one of the sub-areas. There may be a buffer area between each of the sub-areas. In some embodiments, the buffer area is adjustable.

[0071] In some embodiments, a vision system associated with the robotic system is configured to determine how to put the object down. For example, some objects are not rigid and the range associated with the object (e.g., a cloth or cuddly toy) changes after the object is grasped and moved. The vision system is configured to determine what the range is and the material information to select how to put the object down to prevent crushing the object and to prevent the object from being dropped from a height that could damage the object or cause the object to become entangled or in an unfavorable configuration.

[0072] The robotic system can compare point cloud information at various times during the grasping / picking operation. A vision system associated with the robotic system can determine an initial point cloud, and as the object is grasped and moved, a difference between the initial point cloud and the current point cloud can be determined. If the determined difference indicates that the range change associated with the object is greater than or equal to a threshold amount, the robotic system can determine that the object is a non-rigid object. In response to determining that the object is a non-rigid object, the robotic system can implement a placement strategy that is different from when the object is a rigid object. Depending on the rigidity of the selected object, the object can be moved faster and the way the object is placed can be put down in a more aggressive manner to increase the picking speed / throughput of the robotic system. If the determined difference indicates that the range change associated with the object is less than a threshold amount, the robotic system can determine that the object is a rigid object. The way the object is placed can be different, so that the bottom surface of the object is controlled to approach the drop area to prevent the object from breaking, rolling, and / or ending up in a tangled configuration.

[0073] The point cloud information of the drop zone can be used to determine where there is enough open space for placing objects. The point cloud information can also be used to determine where each object should be placed to maximize packing efficiency.

[0074] At 218, it is determined whether more objects are located in the workspace region. In the event that more objects are located in the workspace region, process 200 returns to step 208. In the event that no more objects are located in the workspace region, process 200 ends.

[0075] Figure 3 300 is a flow chart illustrating a process for grasping and moving an object according to some embodiments. In some embodiments, process 300 is implemented by a robotic system such as robotic system 101. In some embodiments, process 300 is implemented to perform some or all of steps 210 and 212 of process 200.

[0076] At 302, a feature associated with an object is grasped. A controller of a robotic system causes an end effector to grasp the feature associated with the object. The end effector may include one or more sensors, such as a force sensor, a pressure sensor, a capacitive sensor, etc. The feature associated with the object may be grasped at a determined optimal grasping position. The optimal grasping position may be determined using a gridded or segmented version of the object determined based on visual data received from one or more cameras of the robotic system.

[0077] In some embodiments, visual and / or tactile information is used to detect the texture of the object, such as wrinkles. The robotic system can use visual and / or tactile information to select high-quality pick-up points (e.g., probability of successful grasping greater than a threshold), such as avoiding picking up objects in wrinkled areas using a suction gripper.

[0078] At 304, determine whether the grasping is successful. In some embodiments, the robotic system uses computer vision to determine whether the object has moved, that is, whether the grasping is successful. In some embodiments, the robotic system uses a force sensor. In the case where the mass detected by the force sensor increases, and this increase is different from the static threshold or the general estimate based on the volume / appearance of the object, the robotic system determines that the wrong object is grasped and drops the object. In some embodiments, the end effector of the robotic system has a pressure sensor or uses capacitance to sense the deformation of the skin surface of the object to determine whether the object has been grasped. The output of the sensor is compared with the grasping threshold to determine whether the feature associated with the object has been grasped. In some embodiments, the electrical or capacitive sensor on the end effector is used to indicate the fully closed contact of the end effector. This indicates that the end effector does not grasp the object. In some embodiments, the robotic system uses a suction gripper end effector to pick up the object and uses a pressure sensor to detect the suction pressure change. The robotic system can determine whether the object has been grasped based on the output of the pressure sensor. In some embodiments, the robotic system determines whether the object has been grasped based on a combination of sensors and computer vision. A voting system can be used to reduce the detection error rate. If most detection methods determine that the object has been grasped, the robotic system can determine that the feature associated with the object has been grasped.

[0079] In some embodiments, visual information and fingertip sensor information are combined to determine if the object slides and moves after the initial grasping of the object to ensure gentle placement. The combined information can be used to determine when the object slides during the motion from pick / grasp to placement. In the event that it is determined that the object has slipped, the range of the object can be recalculated to avoid crashing the object in the drop zone.

[0080] The robot's geometric model and sensor state information can be used to determine the 3D configuration of the robot arm and end effector. The point cloud or RGB pixels associated with the 3D objects may be filtered out, leaving only successfully picked objects, or nothing if the grasp failed.

[0081] In the event that the grasp is successful, the process 300 proceeds to 306. In the event that the grasp is unsuccessful, the process 300 proceeds to 308. At 306, the object is moved to a drop zone. At 308, it is determined whether a threshold number of grasp attempts have been performed. In the event that a threshold number of grasp attempts have been performed, the process 300 proceeds to 312. In the event that a threshold number of grasp attempts have not been performed, the process 300 proceeds to 310.

[0082] At 310, the same gripping technique or a different gripping technique is implemented. Features of the object may be associated with multiple gripping techniques. Each of the gripping techniques is associated with a corresponding score. The gripping technique among the multiple gripping techniques with the highest score is initially selected. In the event of a gripping technique failure, the gripping technique with the next highest score may be implemented. In some embodiments, the same gripping technique is tried again.

[0083] At 312, a next feature is selected. A plurality of objects having a plurality of features may be located in the workspace area. Each of the plurality of features has a corresponding successful grasp score. Features are selected based on their corresponding scores. The next feature is selected based on their corresponding scores. In some embodiments, the next feature is associated with a different object. In some embodiments, the next feature is associated with the same object. The next feature has a lower score than a previously selected feature, but has the same or higher score than other remaining features of the plurality of features.

[0084] Figure 4 4 is a flow chart illustrating a process for grasping a feature according to some embodiments. In the example shown, process 400 can be implemented by a robotic system such as robotic system 101. In some embodiments, process 400 is implemented to perform some or all of step 210 of process 200 or step 302 of process 300.

[0085] At 402, a grasping strategy is determined for a geometric shape associated with a feature. In some embodiments, a feature is associated with a determined shape. In some embodiments, an object or feature is divided into a plurality of sub-segments and a corresponding shape for each of the plurality of sub-segments is determined. Based on the geometric shape of the feature, the grasping technique for the feature may be different. For example, the grasping technique for a feature having a spherical shape may be different from the grasping technique for a feature having a rectangular prism shape.

[0086] The robotic system may store a data structure associating features with one or more grasping strategies. For example, the data structure may include an entry associating a first feature with a first grasping strategy and a second grasping strategy. The grasping strategies may be applied to different portions of a feature. The data structure may include entries associating features with one or more grasping strategies and one or more portions of the feature. For example, the data structure may include a first entry associating a first feature with a first grasping strategy at a top portion of the feature, a second entry associating the first feature with a first grasping strategy at a middle portion of the feature, a third entry associating the first feature with the first grasping strategy at a bottom portion of the feature, a fourth entry associating the first feature with a second grasping strategy at a top portion of the feature, a fifth entry associating the first feature with a second grasping strategy at a middle portion of the feature, a sixth entry associating the first feature with a second grasping strategy at a bottom portion of the object, and so on.

[0087] In some embodiments, some of the grasping strategies associated with the feature are not available because one or more other objects prevent the end effector from grasping the feature at a particular location. The robotic system is configured to determine one or more available grasping strategies for the feature associated with the object based on a current placement of the one or more objects located in the workspace area (e.g., some grasping strategies are filtered out).

[0088] At 404, a corresponding successful grasp score is determined for each of the grasping strategies. The successful grasp score for a feature may be based on a probability that the grasping strategy will result in a successful grasp of the feature. The probabilities of different combinations of grasping tools and grasping positions are determined. The probability that a grasping strategy will result in a successful grasp of a feature may be based on an additional factor, such as contextual information about the environment, historical grasp information about the environment, the angle at which the robot arm grasps the feature (to avoid collisions with other objects), the height at which the robot arm grasps the feature (to prevent collisions at the top of the gripper), the grasp width, the surface normal orientation at the grasp point, the amount of the feature that can be grasped, etc. The contextual information about the environment includes the presence of other objects near or adjacent to the object, the amount by which other objects near or adjacent to the object obstruct the robot arm's ability to grasp the feature, whether more objects are continuously added to the workspace, etc.

[0089] At 406 , the gripping strategy with the highest score for the geometry associated with the feature is selected.

[0090] Figure 5 5 is a flow chart illustrating a process for selecting a gripping technique according to some embodiments. In the example shown, process 500 may be implemented by a robotic system such as robotic system 101. In some embodiments, process 500 is implemented to perform some of steps 402 of process 400.

[0091] Visual data from one or more cameras may provide data that allows one or more of the boundaries of an object to be determined. For example, a first camera may be facing an object at a first angle. The first camera may be able to provide information about the height and width of the object, but not the depth information of the object. A second camera may be facing the object at a second angle. The second camera may be able to provide information about the depth and height of the object, but not the width of the object. Visual data received from multiple cameras may be combined to determine boundary information (estimated, approximate, or exact) associated with an object.

[0092] At 502, a minimum bound associated with an object is determined. The minimum bound of an object corresponds to the smallest value that the object may have in a particular dimension.

[0093] At 504, a maximum bound associated with the object is determined. The maximum bound of the object corresponds to the maximum value that the object may have in a particular dimension. The maximum bound of the object corresponds to the maximum value that the object may have in a particular dimension.

[0094] At 506, a grasping strategy for the object is selected based on the determined minimum boundary and the determined maximum boundary. A plurality of grasping strategies may be implemented to grasp the object based on the determined minimum boundary and the determined maximum boundary. For example, a feature associated with the object may be grasped along its major axis or minor axis. Due to the size of the end effector, the end effector may not be able to grasp a feature along its major axis, but may be able to grasp a feature along its minor axis. In this case, a grasping strategy that grasps the feature along its minor axis will be selected.

[0095] Historical data may be used to select a grasping strategy. For example, actual sizes of previous objects with similar minimum and maximum bounds and corresponding grasping strategies for the previous objects may be stored in a data structure. After the robotic system moves the object to an area with a non-occluded view, the actual size of the previous object may be determined. In the event that a grasping strategy used for one of the previous objects was successful for an object with similar minimum and maximum bounds, a successful grasping strategy may be selected for a current object with similar minimum and maximum bounds.

[0096] Figure 6 is a flow chart illustrating a process for grasping and moving an object according to some embodiments. In the example shown, process 600 may be implemented by a robotic system such as robotic system 101 .

[0097] Human interaction may be necessary to replenish objects located in the workspace area or to remove objects from the drop zone. This interaction puts the person within the range of the robotic system. The robotic system's controls may detect the entry of a person into the workspace area and react to make the person's conditions safe.

[0098] At 602, a process of picking up and moving a plurality of objects from a workspace area to a drop zone begins.

[0099] At 604, it is determined whether a person is located in the workspace area. One or more of the multiple cameras of the robot system can detect human body parts. In some embodiments, a high-resolution RGB image is used to detect a person. In some embodiments, an IR sensor is used to sense a person based on a person's infrared signature. Data from one or more detection sources can be used to determine the probability that a person is located in the workspace area. If the probability is above a detection threshold, it is determined that the person is located in the workspace area.

[0100] In the event that it is determined that the person is located in the workspace area, process 600 proceeds to 606. In the event that it is determined that the person is not located in the workspace area, process 600 proceeds to 610.

[0101] At 606, it is determined whether a person is located in an area associated with an object. The workspace area may consist of a plurality of zones. The robotic system may pick up an object from a first zone. It is determined whether a person is located in the first zone.

[0102] In the event that it is determined that the person is not located in the zone associated with the object, process 600 proceeds to 610. In the event that it is determined that the person is located in the zone associated with the object, process 600 proceeds to 608.

[0103] At 608, an object in a different zone is selected. The robotic system may slow down the speed of the robotic arm in the zone associated with the object to prevent harm to the person. The robotic system may stop the robotic arm in the zone associated with the object to prevent harm to the person. After the robotic system slows down or stops the robot, the robotic system may change the trajectory of the robotic arm to a different zone so that the trajectory of the robotic arm avoids the person. The robotic system may sound an alarm (audible and / or visual) after detecting the person to alert the person.

[0104] At 610, the object is moved to a drop zone.

[0105] Figure 7 700 is a flow chart illustrating a process for moving an object according to some embodiments. In the example shown, process 700 may be implemented by a robotic system such as robotic system 101. In some embodiments, process 700 is implemented to perform some of steps 304 of process 300.

[0106] A human operator or other objects may bump into the camera or robot mount of a robotic system, which may cause the robotic system to be miscalibrated. Multiple fiducial markers on the robotic system combined with additional information (e.g., depth or point cloud data) can be used to continuously co-locate the fixed camera, workspace, and robotic system while one or more robotic systems perform their normal operations. This continuous co-localization can also be performed using any combination of fixed / mobile markers and / or fixed / mobile cameras. In some embodiments, continuous co-localization is performed without the use of fiducial markers and using information from the normal scene. Continuous co-localization enables detection and correction of such faults as camera misalignment, sensor drift, workspace rearrangement, etc. This calibration method enables hot-swapping, adding, or removing cameras during normal operation.

[0107] At 702, the orientation of the camera is detected. Aruco markers can be used to align multiple cameras of a robotic system. Before the time when the robotic system is to grasp a feature associated with an object, the alignment of the robotic system can be checked. The orientation of the camera is detected and compared to the initial alignment of the camera.

[0108] At 704, it is determined whether any uncalibrated conditions have been met. In some embodiments, an uncalibrated condition occurs when the plurality of cameras detects that the robot base position has moved from a calibrated robot base position. In some embodiments, an uncalibrated condition occurs when a handheld camera in the plurality of cameras detects that one or more of the cameras and / or one or more of the camera supports have moved. In some embodiments, an uncalibrated condition occurs when the robot arm is unable to grasp a feature associated with an object after a threshold number of attempts.

[0109] In the event that any uncalibrated conditions have been met, process 700 proceeds to 708. In the event that any uncalibrated conditions have not been met, process 700 proceeds to 706.

[0110] At 706, the object is moved to a drop zone.

[0111] At 708, the robotic system is recalibrated. In some embodiments, the robotic system is recalibrated using a handheld camera of a plurality of cameras to reposition the workspace Aruco markers. In some embodiments, the robotic system is recalibrated by reestimating using a camera to workspace transformation using a fiducial marker (e.g., Aruco). In some embodiments, the robotic system is recalibrated relative to the markers on the robot. In some embodiments, the robotic system is recalibrated using an Iterative Closest Point (ICP) algorithm using previous calibration parameters as a seed transformation. In some embodiments, the robotic system is recalibrated using hand-eye.

[0112] Figure 8 is a flow chart illustrating a process for updating the operation of a robotic system according to some embodiments. In the example shown, process 800 may be implemented by a robotic system such as robotic system 101 .

[0113] At 802, one or more objects are detected in a workspace area. At 804, it is determined whether one of the one or more detected objects is a new object in the workspace area. In the event that one of the one or more detected objects is not a new object in the workspace area, process 800 proceeds to 806. In the event that one of the one or more detected objects is a new object in the workspace, process 800 proceeds to 808. At 806, operation of the robotic system is maintained according to the current configuration. At 808, operation of the robotic system is updated. In some embodiments, the scene of the workspace area is recalculated to include the new object. In some embodiments, the scene is recalculated when the new object is a person. In some embodiments, the scene of the workspace area is recalculated when there are any changes between one or more existing objects and the new object.

[0114] Fig. 9 is a block diagram illustrating a suction-based end effector according to some embodiments. In various embodiments, Fig. 9 The robot arm end effector 900 can be used to implement Figure 1 The end effector 108 is provided.

[0115] In the example shown, the end effector 900 includes a body or housing 902 attached to a robotic arm 904 via a rotatable coupling. In some embodiments, the connection between the housing 902 and the robotic arm 904 may include a control computer (such as Figure 1 The end effector 900 further includes a suction or other pneumatic line 906 extending through the robotic arm 904 into the housing 902 to provide a vacuum source to a suction control module 908. In various embodiments, the control module 908 is connected to a control computer external to the end effector 900, such as via wireless and / or wired communication via a communication interface 914. Figure 1 The controller 106 of the embodiment of the present invention. The control module 908 includes electronic and / or electromechanical components that are operable to provide suction force to the suction cups 910, 912 comprising the end effector 900, for example, to attach the end effector to an item to be picked up, moved and placed using the end effector 900 by suction.

[0116] In the example shown, a camera 916 mounted on the side of the housing 902 provides image data of the field of view below the end effector 900. A plurality of force sensors 918, 920, 922, 924, 926, and 928 measure the forces applied to the suction cups 910 and 912, respectively. In various embodiments, the force measurements are transmitted to an external and / or remote control computer via the communication interface 914. The sensor readings are used in various embodiments to enable the robotic arm 904 and the end effector 900 to be used to fit an item into a place adjacent to other items and / or a sidewall or other structure, and / or to detect instability (e.g., while the item is still under suction, the back push force of the item being pressed down is insufficient in a place where the item is expected to be placed and is stable). In various embodiments, horizontally mounted force sensor pairs (e.g., 918 and 922, 924 and 928) are placed at right angles in the xy plane to enable the determination of forces in all horizontal directions.

[0117] Fig.10 is a block diagram illustrating a gripper-type end effector according to some embodiments. In various embodiments, Fig.10 The robot arm end effector 1000 can be used to implement Figure 1 The end effector 108 is provided.

[0118] In the example shown, the end effector 1000 includes a body or housing 1002 attached to a robotic arm 1004 via a rotatable coupling. In some embodiments, the connection between the housing 1002 and the robotic arm 1004 may include a control computer (such as Figure 1 The end effector 1000 further includes a gripper including articulated digits 1010 and 1012 and a power cord 1006 extending through the robotic arm 1004 into the housing 1002 to provide power to a gripper control module 1008. In various embodiments, the control module 1008 is connected to a control computer external to the end effector 1000, such as via wireless and / or wired communication through a communication interface 1014, for example. Figure 1 The controller 106 of the embodiment of the present invention. The control module 1008 includes electronic and / or electromechanical components that are operable to manipulate the gripper fingers 1010, 1012, for example, to grasp items to be picked up, moved, and placed using the end effector 1000.

[0119] In the example shown, a camera 1016 mounted on the side of the housing 1002 provides image data of the field of view below the end effector 1000. A plurality of force sensors 1018, 1020, 1022, 1024, 1026, and 1028 measure the forces applied to the mounting points of the fingers 1010 and 1012, respectively. In various embodiments, the force measurements are transmitted to an external and / or remote control computer via the communication interface 1014. The sensor readings are used in various embodiments to enable the robotic arm 1004 and end effector 1000 to be used to fit an item into place adjacent to other items and / or sidewalls or other structures, and / or to detect instability (e.g., insufficient back push force to hold an item down where it is expected to be placed and is stable, while the item is still under suction).

[0120] Although in Fig. 9 A suction type actuator is shown in Fig.10 A gripper-type effector is shown in FIG, but in various embodiments, one or more other and / or different types of end effectors may be used in a robotic system to pick up and place unknown objects as disclosed herein.

[0121] In some embodiments, sensors are used to detect collisions with other objects, containers, and / or the environment, and to continue automated operation by making "compliant" adjustments to the trajectory. For example, in some embodiments, if a collision occurs into a wall or other structure, the robotic arm reduces force and adjusts the trajectory to follow the obstacle until it avoids the obstacle.

[0122] Although the foregoing embodiments have been described in considerable detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A method comprising: determining, based on visual data received from the plurality of cameras, a plurality of potentially graspable features of a plurality of objects present in the workspace area; determining, for each of at least a subset of the plurality of potentially graspable features, one or more corresponding grasping strategies for grasping with a robotic arm and an end effector; determining, for each of at least a subset of the one or more corresponding grasping strategies, a score associated with a probability of successfully grasping the corresponding feature; selecting a first feature of the plurality of potentially graspable features to be grasped using a selected grasping strategy based at least in part on a corresponding score associated with the selected grasping strategy with respect to the first feature, wherein the first feature is associated with a first object of the plurality of objects; and Controlling the robotic arm and the end effector to attempt to grasp the first feature using the selected grasping strategy, wherein grasping the first feature using the selected grasping strategy includes: determining whether grasping of the first feature is successful, wherein in response to determining that grasping of the first feature is determined to be unsuccessful, repeatedly reattempting to grasp the first feature using the selected grasping strategy or implementing a different grasping strategy to grasp the first feature until a threshold number of attempts to grasp the first feature have been attempted; and In response to having attempted the threshold number of attempts, selecting a second feature of the multiple potentially graspable features for grasping, wherein the second feature of the multiple potentially graspable features has a lower score than the first feature but has the same or higher score than other remaining features of the multiple potentially graspable features, and wherein the second feature is associated with the first object or a different object of the multiple objects.

2. The method according to claim 1, wherein: The visual data received from the plurality of cameras includes point cloud data. 3 . The method of claim 1 , further comprising determining boundary information associated with the plurality of objects based on visual data received from the plurality of cameras.

4. The method according to claim 3, wherein: The selected grasping strategy is selected based in part on the determined boundary information associated with the plurality of objects. The method of claim 1 , further comprising segmenting one of the plurality of objects into a plurality of shapes.

6. The method according to claim 1, wherein: The first feature has the highest score associated with a probability of successfully grasping the corresponding feature.

7. The method according to claim 1, wherein: At least one of the plurality of cameras is dynamically movable.

8. The method according to claim 1, wherein: At least one of the plurality of cameras is in a fixed position.

9. The method according to claim 1, wherein: In response to determining that the grasping of the second feature is determined to be successful, an object associated with the second feature is moved to a drop zone.

10. The method according to claim 1, wherein: Visual data received from the plurality of cameras is used to determine negative space information associated with the plurality of objects.

11. The method according to claim 1, wherein: The plurality of potentially graspable features include handles, protrusions, or voids.

12. The method according to claim 1, further comprising: Detecting people in the workspace area; and A second feature associated with the second object is selected based on whether the person is located in the same zone as the first feature associated with the first object.

13. The method according to claim 1, further comprising: Detecting people in the workspace area; and In the event that a person is detected in the same zone as the first feature associated with the object, operation of the robotic arm is stopped.

14. The method of claim 1, further comprising moving the first object to a drop zone. The method of claim 14 , further comprising determining whether the first object has been dropped.

16. The method of claim 1, further comprising placing an object associated with the second feature in a drop zone.

17. The method of claim 1, further comprising recalibrating a robotic system associated with the robotic arm and the end effector based on whether an uncalibrated condition has occurred.

18. The method of claim 1, further comprising updating operation of a robotic system associated with the robotic arm and the end effector based on whether a new object is detected in the workspace.

19. A system comprising: a communication interface; and a processor coupled to the communication interface and configured to: determining, based on visual data received from the plurality of cameras, a plurality of potentially graspable features of a plurality of objects present in the workspace area; determining, for each of at least a subset of the plurality of potentially graspable features, one or more corresponding grasping strategies for grasping with a robotic arm and an end effector; determining, for each of at least a subset of the one or more corresponding grasping strategies, a score associated with a probability of successfully grasping the corresponding feature; selecting a first feature of the plurality of potentially graspable features to be grasped using a selected grasping strategy based at least in part on a corresponding score associated with the selected grasping strategy with respect to the first feature, wherein the first feature is associated with a first object of the plurality of objects; and Controlling the robotic arm and the end effector to attempt to grasp the first feature using the selected grasping strategy, wherein grasping the first feature using the selected grasping strategy includes: determining whether grasping of the first feature is successful, wherein in response to determining that grasping of the first feature is determined to be unsuccessful, repeatedly reattempting to grasp the first feature using the selected grasping strategy or implementing a different grasping strategy to grasp the first feature until a threshold number of attempts to grasp the first feature have been attempted; and In response to having attempted the threshold number of attempts, selecting a second feature of the multiple potentially graspable features for grasping, wherein the second feature of the multiple potentially graspable features has a lower score than the first feature but has the same or higher score than other remaining features of the multiple potentially graspable features, and wherein the second feature is associated with the first object or a different object of the multiple objects.

20. A non-transitory computer readable medium and comprising instructions for: determining, based on visual data received from the plurality of cameras, a plurality of potentially graspable features of a plurality of objects present in the workspace area; determining, for each of at least a subset of the plurality of potentially graspable features, one or more corresponding grasping strategies for grasping with a robotic arm and an end effector; determining, for each of at least a subset of the one or more corresponding grasping strategies, a score associated with a probability of successfully grasping the corresponding feature; selecting a first feature of the plurality of potentially graspable features to be grasped using a selected grasping strategy based at least in part on a corresponding score associated with the selected grasping strategy with respect to the first feature, wherein the first feature is associated with a first object of the plurality of objects; and Controlling the robotic arm and the end effector to attempt to grasp the first feature using the selected grasping strategy, wherein grasping the first feature using the selected grasping strategy includes: determining whether grasping of the first feature is successful, wherein in response to determining that grasping of the first feature is determined to be unsuccessful, repeatedly reattempting to grasp the first feature using the selected grasping strategy or implementing a different grasping strategy to grasp the first feature until a threshold number of attempts to grasp the first feature have been attempted; and In response to having attempted the threshold number of attempts, selecting a second feature of the multiple potentially graspable features for grasping, wherein the second feature of the multiple potentially graspable features has a lower score than the first feature but has the same or higher score than other remaining features of the multiple potentially graspable features, and wherein the second feature is associated with the first object or a different object of the multiple objects.

Images