System and method for automated packaging and processing with static and dynamic payload protection

Abstract

A system for processing an object using a programmable motion device is disclosed. The system includes an end effector of a programmable motion device for gripping objects from a feed container, and a control system for determining payload protection for selected objects. The payload protection includes point cloud data regarding volumetric data, the volumetric data including a volume occupied by the selected object, and the payload protection is determined in response to at least one characteristic of the selected object and provided specific to the selected object.

Description

[0001] Priority

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 433,211, filed on Dec. 16, 2022, the disclosure of which is hereby incorporated herein by reference in its entirety. Background of the Invention

[0003] The present invention generally relates to automated sorting and other processing systems, and particularly to automated systems for handling and processing objects such as packages, wrappers, articles, goods, etc. for e-commerce fulfillment, sorting, facility replenishment, and automated storage and retrieval (AS / RS) systems.

[0004] A shipping center that packages and transports a limited range of objects, for example, from a source company that manufactures the objects, may only need a system and process that repeatedly adapts to a limited range of the same objects. On the other hand, a third-party shipping center that receives multiple types of objects must utilize a system and process that adapts to multiple types of objects.

[0005] For example, in an e-commerce order fulfillment center, personnel package units of objects into shipping containers such as boxes or plastic bags. One of the final steps in the order fulfillment center is to package one or more objects into a shipping container or bag. Order units destined for customers are typically hand-packaged at a packaging station. The order fulfillment center does this for several reasons.

[0006] For example, objects typically need to be packaged into shipping materials. The objects need to be placed into boxes or bags to protect the objects, but generally are not stored in the materials in which they are shipped and instead need to be packaged immediately after an order for the objects has been received.

[0007] In addition, however, handling multiple types of objects on common conveyor and processing systems also presents challenges, particularly when the objects have either low pose authority or low placement authority. Pose authority is the ability to place an object in a desired position and orientation, and placement authority is the ability of an object to maintain its placed position and orientation. For example, if an object with low pose authority or low placement authority is to be moved on a conveyor system that may experience linear or angular acceleration or deceleration, the object may tip over and / or may fall off the conveyor system.

[0008] As the number of goods and the number of destination locations increase and further where the system needs to operate in a busy work cell environment with static and moving obstacles, these requirements become even more challenging. Accordingly, there is still a need for an automated system for handling multiple types of objects in an object handling system in a busy work cell environment with static and moving obstacles. Summary of the Invention

[0009] According to one aspect, the present invention provides a system for handling objects using a programmable motion device. The system includes an end effector of the programmable motion device for grasping an object from a feed container, and a control system for determining a payload protection for the selected object. The payload protection includes point cloud data with respect to volume data that includes the volume occupied by the selected object, and the payload protection is determined in response to at least one characteristic of the selected object and provided specific to the selected object.

[0010] According to another aspect, the present invention provides a method for handling objects using a programmable motion device. The method includes: using an end effector of the programmable motion device to grasp an object from a feed container, lifting the object from the feed container, and determining a payload protection for the grasped object, the payload protection being derived from point cloud data with respect to volume data that includes the volume occupied by the grasped object, the payload protection being determined in response to at least one characteristic of the grasped object and provided specific to the grasped object.

[0011] According to a further aspect, the present invention provides a system for handling objects using a programmable motion device. The system includes an end effector of the programmable motion device for grasping an object, a sensing system for determining sensing data with respect to the object, and a control system for determining a payload protection for the grasped object. The payload protection is derived from point cloud data with respect to volume data that includes the volume occupied by the grasped object, and the payload protection is determined in response to the sensing data. Description of the Drawings

[0012] The following description can be further understood with reference to the accompanying drawings, in which:

[0013] Figure 1 An illustrative diagrammatic view of an object handling system according to one aspect of the present invention is shown, the object handling system including an in-line object handling station;

[0014] Figure 2 An illustrative diagrammatic view of an object handling system according to another aspect of the present invention is shown, the object handling system including a pick-and-place object handling station with an automatic bagging system;

[0015] Figure 3 Is shown Figure 2 An illustrative enlarged view of the object handling system of, showing an opening of the automatic bagging system;

[0016] Figure 4 An illustrative diagrammatic view of an end effector with a vacuum chuck for grasping an object in an object handling system according to one aspect of the present invention;

[0017] Figure 5 Illustrates Figure 4 An illustrative graphical view of multiple positions of an object, showing the relationship between the edge and the radius;

[0018] Figure 6 Illustrates Figure 4 An illustrative graphical view of multiple positions of an object, where the disk of the suction cup does not overlap any edge of the object;

[0019] Figure 7 Illustrates an end effector with a vacuum suction cup for grasping an object in Figure 4 , showing an axially aligned bounding cylinder;

[0020] Figure 8 Illustrates an illustrative graphical view of multiple positions of an object, showing Figure 7 The axially aligned bounding cylinder of;

[0021] Figure 9 Illustrates an illustrative graphical view of an end effector with a conformable cup for holding an object in an unbalanced manner;

[0022] Figure 10 Illustrates an illustrative graphical view of an object in multiple non - equilibrium positions, showing a bounding box;

[0023] Figure 11 Illustrates Figure 10 An illustrative graphical view of an object in multiple non - equilibrium positions in, showing a convex bounding hull;

[0024] Figure 12 Illustrates an illustrative graphical plan view of an object in a handling box, showing a cylindrical payload protection;

[0025] Figure 13 Illustrates an illustrative graphical plan view of an object in a handling box, showing a situation - limited payload protection;

[0026] Figure 14 Illustrates an illustrative graphical representation of an image of a cabinet containing an object to be processed;

[0027] Figure 15 Illustrates Figure 14 An illustrative graphical representation of the point cloud segmentation of the cabinet contents of;

[0028] Figure 16 Illustrates Figure 1 An illustrative graphical enlarged view of a part of the system of, showing a hand - held pose perception system;

[0029] Figure 17 Shows Figure 16 An illustrative diagrammatic bottom view of the end effector of the system of [[ID=]], showing the bottom surface of the object and the end effector;

[0030] Figure 18 Shows Figure 17 The point cloud data of the end effector and the object of [[ID=]];

[0031] Figure 19 Shows Figure 2 An illustrative diagrammatic side view of the end effector of the system of [[ID=]], showing the side surface of the object and the end effector;

[0032] Figure 20 An illustrative diagrammatic enlarged view of the end effector and the vacuum suction cup for grasping an object inside a bag;

[0033] Figure 21 An illustrative diagrammatic enlarged view of the end effector and the vacuum suction cup for grasping an object inside a larger bag that can swing;

[0034] Figure 22 Shows an illustrative diagrammatic view of an object handling system according to one aspect of the present invention, which is related to placing an object into the opening of a bag;

[0035] Figure 23 Shows Figure 22 An illustrative diagrammatic view of the object handling system of [[ID=]], showing the end effector positioned to place an object into the bag opening;

[0036] Figure 24 Shows an illustrative diagrammatic view of an object handling system according to one aspect of the present invention, which is related to placing an object into an open compartment; and

[0037] Figure 25A And Figure 25B Shows Figure 24 An illustrative diagrammatic enlarged view of the object handling system of [[ID=]], showing that the end effector is not aligned with the compartment opening ( Figure 25A ) and showing that the end effector is rotated to be aligned with the compartment opening ( Figure 25B ).

[0038] The drawings are shown for illustrative purposes only. Detailed Description

[0039] According to various aspects, such as Figure 1As shown, the present invention provides an object handling system 10 including a processing station 12 in communication with an input transfer system 14 and a processing transfer system 16. The processing station 12 includes a programmable motion device 18, and at the distal end of the device 18, an end effector 20 is attached. The end effector 20 can be coupled (e.g., via a hose) to a vacuum source 22, and the operation of the programmable motion device can be provided by one or more computer processing systems 100, 101 in communication with the programmable motion device and all sensing units 11, conveyors 14, 16, and further processing systems disclosed herein.

[0040] The object handling system 10 further includes a vacuum chuck replacement rack 22, which is accessible by the programmable motion device 12 to exchange the vacuum chucks attached to the end effector 20, as disclosed, for example, in U.S. Patent Application Publication No. 2019 / 0217471, the disclosure of which is hereby incorporated by reference in its entirety. One or more hand-held pose scanners 24 are also provided in the vicinity of the area where objects are picked up from a feed bin cabinet or tote 15 and placed into a processing container 26 such as a box. The processing container 26 may include a lid 28 that holds the box lid in an open position and against the box, and provides a funnel-shaped opening for each container 26. When containers 26 with lids 28 are adjacent to each other, the lids can contact each other, thereby providing adjacent container openings where there is no area between the containers that is exposed and into which an object could accidentally fall.

[0041] When the system 10 moves an object out of a feed tote 15 on the feed transfer system 14 and into a processing container 26 on the processing transfer system 16, the system needs to ensure that the object it holds does not collide with any other part of the work environment (such as the chuck replacement rack 22). If the object collides with the environment, it may cause damage to the environment (the chuck replacement rack may be fragile), or it may cause damage to the object itself, or it may cause the gripper to drop the object. Therefore, it is important to plan the motion in the environment such that it is unlikely to cause a collision between the object held and the work station environment, such as the lid 28 when lifting and moving the object.

[0042] According to various aspects, the systems and methods of the present invention utilize payload protection in an automated object handling system including a programmable motion device, such as, for example, Figure 1 the in-line object processing station 12 shown, or Figure 2 the pick-up station 30 shown. Figure 2The pick-up station 30 includes an automatic bagging system 32 and a programmable motion device 18 with an attached end effector 20 that moves objects from input conveyor systems 34, 36 to the automatic bagging system 32. Once bagged, the object 47 (in the bag) falls onto a processing conveyor system 38. The end effector 40 can be coupled to a vacuum source 40, and the operations of the programmable motion device 18, sensing units 31, 33, 35, conveyors 34, 36, and the automatic bagging system 32 can be provided by one or more computer processing systems 100, 101.

[0043] The system 30 uses, for example, a vacuum chuck 48 attached to the end effector 20 to pick an object out of the feed handling bins 42, 44 and place it into a chute or opening 46 of the automatic bagging system 32, as Figure 3 shown. When the system 30 moves an object out of and into the automatic bagging system 32 from the feed handling bins 42, 44, it needs to ensure that the object it holds does not collide with any other part of the work station. Moreover, it is important to plan the motion in the environment in such a way that it is unlikely to result in a collision between the held object and the work station environment.

[0044] Motion planning in a programmable motion device (e.g., a robot) typically involves finding a sequence of robot arm joint configurations (e.g., a trajectory) that reaches a desired destination and avoids collisions with the environment at each part of the trajectory. See, for example, Planning Algorithms, Steven M. LaValle, Cambridge University Press, 2006. Some motion planning systems include algorithms such as Rapidly-Exploring Random Trees (RRT) and Rapidly-Exploring Random Trees: A New Tool for Path Planning by Steven M. LaValle, Technical Report, Department of Computer Science, Iowa State University, October 1998, which include collision checking, where the geometric model of the robot is checked against the geometric model of the environment. The geometry of the robot is represented digitally, and a given test configuration is used to transform the robot to a given set of positions, orientations, or joint angles, and then that given set of positions, orientations, or joint angles is tested for collisions against the digital model of the environment as well as the robot itself (to avoid self-collisions).

[0045] However, if the robot holds an object, e.g., with some kind of robotic gripper, care must be taken to avoid collisions of the held object with the robot or the environment. To achieve this, some representation of the object is needed in order to incorporate it into the collision checking tests. This is straightforward if the position and orientation of the object relative to the gripper are known or can be controlled in advance. For example, in some manufacturing applications, special grippers can be designed to pick up parts at specific locations on a part, in which case the geometry of the held object can be directly attached to the end effector / gripper at a position that accurately represents reality.

[0046] However, there are many situations in which information about the held object, such as when picking up products or SKUs from bins in an AS / RS, cannot be tightly controlled. The products may be stored in the bins in many arbitrary, uncontrolled positions and orientations. For example, a robot may pick up an item with a vacuum gripper, and where and how the item is then held may be completely unknown. For example, the geometric model of a bin picked up from the center of the bin will need to be different from the geometric model of the bin picked up from the edge of the bin.

[0047] Another consideration is that there may be compliance in the connection between the end effector and the held object. For example, when the object accelerates or the orientation of the gripper changes, the bellows in the suction cup may deflect or bend. Thus, the position and orientation of the object relative to the end effector of the robot may not be constant. In addition, the object itself may be deformable. Both of these considerations increase the uncertainty in the actual volume occupied by the held object as the end effector of the robot moves through space. All of these considerations are very important for a robotic workstation with little clearance. Thus, in such situations, it is important to consider the geometry of the held object during motion planning to check for potential collisions.

[0048] However, the challenge in providing such planning is that overly conservative bounds may lead to failure in trajectory planning in a crowded work environment. It is also important to realistically adopt the tightest bounds possible, regardless of what geometry is attached to the model. If the attached geometry is too conservative (much larger than the actual object or the volume it may hold), then there may be no collision-free path in the virtual environment. This can lead to trajectory planning failure: either no collision-free path can be generated, or in other cases, a suboptimal path is generated due to overly conservative bounds. Such overly conservative bounds may cause the system to be unable to generate a path in situations where there are many collision-free paths (no bounds or with smaller bounds) in reality.

[0049] The virtual geometry attached to the end effector to perform collision checking is referred to herein as payload protection. Payload protection can be considered the worst-case volume that encloses the item. The motion planner assumes that the payload protection is attached to the robot's end effector when it attempts to find a collision-free plan in an environment from point A to point B. Various ways of calculating and using payload protection with and without uncertainty in the way the object is held are disclosed herein.

[0050] The term trajectory generally refers to a list of joint configurations (e.g., joint angles) and time relative to the start of the motion. The robotic arm controls its motors to achieve a given angle within a given time. For example, a robotic arm can have 6 joint angles (6DOF), and when using a yaw gripper as Figure 1 and Figure 2 shown, the system can have 7 degrees of freedom (7DOF). Forward kinematics is the mapping from joint configuration (6 or 7DOF) to a coordinate system (usually at the end effector or suction cup). Inverse kinematics determines the joint configuration that can reach a given coordinate system at the end effector (or vacuum suction cup). The acronym AABB used herein refers to an axis-aligned bounding box, and the axes to which the box is aligned depend on the choice of coordinate system. The term pose refers to a position (x,y,z) and an orientation (e.g., roll, pitch, yaw) or other orientation representation such as a quaternion, matrix, or axis angle, typically providing a total of six degrees of freedom.

[0051] For a sensor that captures an image or point cloud, the external parameters are six parameters that encode the position and orientation of the sensor relative to the robot's coordinate system. External calibration allows the system to place points from the point cloud in the robot's coordinate system, for example, or project virtual points in the robot's coordinate system onto an image. The term stock keeping unit (SKU) refers to the identification of a certain product. As described above, an automated storage and retrieval system (ASRS) refers to an automated system of cranes, shuttles, or robots that transports tote boxes to stations. A warehouse management system (WMS) provides a source of information about the size or weight of an SKU. The term sorted dimensions refers to a triple (d1,d2,d3), where d1 ≤ d2 ≤ d3, representing the dimensions of an SKU, e.g., in cm. A point cloud is a list of 3D points generated by a 3D camera such as a stereo camera system, LIDAR imaging system, time-of-flight (TOF) camera system, or structured light camera system. If an object is presented to the picking system such that the end effector is presented with the largest face up (LFU) for each object, the system knows the largest face up of the object that encloses the minimum volume of a cuboid. This means that the shortest dimension d1 is parallel to the vertical direction.

[0052] In various aspects of the present invention, systems and methods are provided for generating and using payload protection in various applications with different fidelity (accuracy) requirements. Different methods of providing payload protection provide varying degrees of conservatism, or in other words, increased fidelity levels of the bounds on worst-case outcomes. A set of kinematic and logic-based concepts discussed below use increased model fidelity. Additional disclosed methods increase the information from sensors, and further methods are hybrid methods.

[0053] System parameters can provide that the object handling system is capable of handling products of a specific size and weight. For example, a robot may only be able to handle items in the shape of a rectangular parallelepiped that are less than 35 cm × 25 cm × 25 cm. Figure 4 An end effector 20 with a vacuum chuck 48 for grasping an object 50 is shown. A virtual bounding box shown at 52 can be defined with respect to the distal end of the vacuum chuck 48. For planning the motion after picking up the object, the system can construct an axis-aligned bounding box (AABB) 52, the top plane of which is centered on the gripper, and the length of the horizontal sides of the bounding box is 2*d, where d 2 =(d2) 2 +(d3) 2 , that is, the hypotenuse of the two longest dimensions. Figure 5 Multiple positions of the object 50 are shown, which result in a relationship such that each of the four sides has a length of 2*d.

[0054] If it can be guaranteed that the object is LFU, then the system defines d1 as the height of the bounding box, otherwise the system sets the height to d3. Then this bounding cuboid is virtually attached to the gripper coordinate system and used to avoid collisions during motion planning. The coordinate system for axis alignment purposes is selected at the time of grasping and can be a fixed coordinate system relative to the robot, or a coordinate system derived, for example, from the pose of the tote box from which it is picked (which may vary from tote box to tote box and pick to pick). This AABB defines the payload protection, and due to its construction, it will enclose the object regardless of which part of the object 50 is picked up.

[0055] If the gripper is a vacuum chuck and if it is assumed that the vacuum chuck is entirely within one of the pick-up surfaces, the tightness of the model can be improved. The system can adjust d according to the radius d d of the suction cup such that d 2 =(d2 - d d ) 2 -(d3 - d d ) 2 , as Figure 6 shown, where the disk of the suction cup does not overlap any edge of the object. This results in a slightly smaller bounding box 54, asFigure 6 as shown, where d d is the radius of the vacuum suction cup.

[0056] The benefit of using a bounding box (cube) is that it facilitates the speed of mathematical processing. A tighter boundary can be constructed by using a cylinder with a radius of d instead of a box, but with a more complex geometry. Figure 7 An end effector 20 with a vacuum suction cup 48 grasping an object 50 is shown. A virtual bounding cylinder shown at 56 is defined with respect to the distal end of the vacuum suction cup 48. To plan the motion after picking up an object, the system can construct an axis-aligned bounding cylinder 56, the top plane of which is centered on the gripper, where the cylinder 56 has a radius of d, where again, d 2 =(d2 - d d ) 2 -(d3 - d d ) 2 , the hypotenuse of the two longest dimensions. Figure 8 Multiple positions of the object 50 are shown, showing that the bounding cylinder 56 has a radius r.

[0057] The above method uses the maximum size that can be handled by the robot. The generated boundary may be much larger than the actual item being picked up. This can result in slower motion than in the case where the motion planning system has a tighter boundary and, therefore, less restricted motion in the virtual environment.

[0058] In some applications, SKUs arrive at the picking robot in like-kind tote boxes, for example when they come from an AS / RS. The tote box can be subdivided into multiple sections, each of which has multiple individual SKUs. Additionally, when an order to pick up one or more objects is sent to the picking robot, size and weight data about the SKUs can also be sent. Alternatively, the robot picking system software itself can maintain a database of SKU sizes. Thus, in such applications, the robot picking system can use the size of the object itself rather than the maximum size to generate a payload guard using any of the methods described herein. This has the potential to generate a significantly smaller payload guard volume.

[0059] According to a further aspect, a kinematic gripping model can be used to generate a payload guard that may be more accurate in some applications. If information about the SKU and the gripper is available, a more accurate payload guard can be obtained. For example, a vacuum-based gripper with a suction cup 48 typically has a bellows that causes the suction cup to deflect as shown when holding an object 50 in an unbalanced manner. In this case, the suction cup 48 acts like a torsion spring, the torque and angular deflection of which follow Hooke's law: the torque is proportional to the angle of deflection (d Figure 9 as shown. ddd= d×d). Static balance means that the torque from the suction cup 48 balances the torque generated by the gravity acting on the center of mass of the object 50.

[0060] After knowing the size and weight of the object, the system can calculate a set of possible gripping methods by changing the position where the gripper holds the object. For example, Figure 10 and Figure 11 shows multiple positions of the object 50 and the vacuum suction cup 48b relative to the programmable motion device, and these positions can be determined by the computer processing system. Then, for payload protection, the system has various options, including: (a) using a bounding box 60 that contains the union of the so-generated cuboids, as shown in Figure 10 ; (b) using the smallest ellipsoid that contains all the cuboids (as shown at 62 in Figure 11 ); (c) using the convex hull of the cuboids (as shown at 64 in Figure 11 ); and (d) using the union of the so-generated cuboids (which may be computationally slower). These diagrams are 2D, but can be easily extended computationally to 3D and to objects that are not necessarily boxes.

[0061] According to further applications, the payload protection of the object can be generated at least partially based on the inference of the object in-situ in the container. For example, the system can logically infer the geometry of the payload protection during picking. If a SKU of known size is picked from a homogeneous handling box, then we can intersect the internal volume of the handling box with one of the payload protections calculated above. For example, referring to Figure 12 , if the robot picks up the object 61 from the handling box 63 through the vacuum suction cup at the position shown at 65, then it can infer a cylindrical payload protection based on the size of the object, as shown at 66 (in top view). For example, in the case where the contour of the object is unknown. Referring to Figure 13 , the contour of the object 61 may still be unknown, but since the object is close to the wall (actually two walls) of the handling box 63, the system can determine the restricted payload protection 68 as the union of the payload protection 66 of Figure 12 and the restriction on its available positions due to the in-situ environment (the internal volume of the handling box) of the object. In other words, it can be assumed that the object does not pass through any wall of the handling box 63. The position and orientation can be determined from the same 3D imaging sensor that generates the grasp.

[0062] This method relies on the tote being one of known geometric shapes and the system being able to estimate the pose of the tote relative to the robot coordinate system. This method can be generalized as follows: the principle is that an object must not be found straddling the boundaries of the interior volume of the tote. Thus, any volume that can be safely assumed to contain all positions and orientations of the SKU can be used as the intersection volume.

[0063] According to a further aspect, a payload safeguard can be developed based on perception data prior to picking. The above method of generating a payload safeguard is based on kinematic principles. In a further aspect, the system can consider a method of generating a payload safeguard by sensing prior to picking and performing further operations after the object has been grasped and lifted.

[0064] Various segmentation algorithms can be used to segment 3D point cloud data and 2D images. The 3D point cloud can be segmented using local convexity (e.g., see Object Partitioning Using Local Convexity, Simon Christoph Stein, Marcus Schoeler, Jeremie Pappon, and the Computer Vision Foundation, CVPR 2014). Deep learning algorithms (e.g., see Learning Orientation-Estimation Convolution Neural Network for Building Detection in Optical Remote Sensing Image, Yongliang Chen, arXiv:1903.05862v1 [cs.CV] Mar 14, 2019) can infer the orientation of common objects. These algorithms can be used directly to generate grasps, but can also be used to segment an object from its background in order to generate a payload safeguard. Figure 15 Shown at 72 is the use of Figure 14 an example of such a segmentation of the point cloud data of the contents of the cabinet using the image shown at 70.

[0065] Using the segmented tote content, there are various further ways in which the system can construct payload protection. One way to do this is to center the payload protection around the center of the segmented section and align the long dimension of the payload protection with the long dimension of the segmented section. However, in some applications, this method may not adequately account for noise and / or occlusion in the segmented section. For example, segmentation may typically be over-segmented, meaning that depending on the tuning of the system, the system may generate more segmented sections than objects. Another method is to generate the payload protection from the union of the bounding boxes of all SKU sizes that contain the segmented section. Such a volume can be approximated by sampling.

[0066] Point cloud data from the tote sensors and / or the segmented object can be fed through a deep learning algorithm that is trained on previous general pickups or objects of the same type. For example, the deep learning algorithm takes as input tote sensor data and segmented section data represented as an image, as well as the expected grasp location and yaw orientation relative to the segmented section. The output is an estimate of the object bounding box size and pose relative to the gripper. For training purposes, the ground truth object size and relative pose can be explicitly determined via an auxiliary sensor such as a hand pose. The deep learning model can be conservatively trained, e.g., the model is 99% certain that the payload protection encloses the item.

[0067] According to a further aspect, the system can develop the payload protection based on inferences made after picking up the object using sensor and hand pose estimation. Some of the methods of generating the payload protection before picking up discussed above may be challenged by the presence of clutter in the pick-up environment, which may make it difficult to visually or geometrically distinguish the object from its background. For example, different segmented sections may overlap multiple objects, which may result in an inaccurate payload protection developed before picking up. Inaccuracy may lead to collisions, which can be compensated for by adding a margin around the payload protection, however, this will provide a less tight boundary. An alternative or additional method is for the system to generate the payload protection after the picked-up object has been removed from the tote.

[0068] Figure 16 is shown Figure 1An enlarged view of a portion of the system 10 shows a hand pose sensing system 24 having a field of view of a hand pose position pointing to the programmable motion system 18 shown. The hand pose position can be associated with known positions of joints of the programmable motion device and the end effector, thereby providing a defined known position and pose of the end effector at which the volume occupied by the end effector 20 (and the vacuum chuck) is known. Thus, the system can scan an object during transfer by stopping or moving through the hand pose position. The pick box 15 is a pick-and-place box, and the container 26 having a lid 28 is a place box.

[0069] During transfer from the pick box to the place box, one or more 3D cameras 24 are placed below the area where the SKU is transferred in order to capture the hand pose of the object. Thus, the object 74 held by the end effector 20 can be scanned or imaged, and the background including the end effector 20 can be easily removed because the pose and position of the end effector are known, as Figure 17 shown. Figure 18 The robot and end effector point cloud data at 76 and the object point cloud data at 78 are shown. The point cloud data from the 3D camera is filtered by depth, so not all points in the background are shown. The object 74 held by the robot is as shown at 78.

[0070] According to a further aspect, once the object is removed from the tote, one or more 3D cameras can be used to generate one or more point clouds of the object from multiple directions. Figure 19 One such sensor layout is shown in, showing Figure 2 and Figure 3 a portion of the system. The positions and orientations of the three cameras 33, 35, 37 are such that once the object is picked up and removed from the tote 44, the object is centered in the fields of view of the cameras 33, 35, 37. Furthermore, point cloud data is captured from each of the cameras 33, 35, 37 in order to obtain information about the object 94 from multiple angles.

[0071] Although the above method may only provide a very loosely adapted payload protection, capturing the actual volume occupied by the object will result in a more accurate and tightly adapted payload protection. If the dimensions of the object are known, then the sensor data enables the position and orientation of an item of known size to be estimated relative to the gripper. Since this is an estimate of the pose of an object of known size relative to the gripper when held by the gripper, the system can refer to the transformation from the coordinate system of the item to the end effector coordinate system as the in-hand pose. The in-hand pose estimate can then be used by the system to generate the payload protection. If the object is unknown at the time of grasping (and thus the dimensions are not available), then the sensor data can be used to construct an oriented bounding box from the dimensions, orientation, and position, which becomes the payload protection.

[0072] However, it should be noted that the in-hand pose estimate and thus the tightly adapted payload protection will not be available until the item has been scanned by the sensor and then processed from the sensor data. Thus, until that point, another payload protection generated from one of the previous methods may have to be used to influence the motion planning until the in-hand pose scan point. For example, in Figure 16 the system, this will be necessary in the case of an in-hand pose scan being performed mid-transfer.

[0073] Thus, according to various aspects, the sequence can operate as follows. After picking an object from a pick-and-place bin of like or unlike SKUs, the robot moves to one of a set of fixed known arm positions, where at this time the scanner is triggered to capture an image. To plan the movement, it can use one of the kinematically-derived payload protections with or without the known SKU dimensions. At capture, the robot arm joint configuration is recorded. The image and / or point cloud is then processed, and the processing can involve a sequence of configurable filters and estimators, which are then applied to the point cloud, and at the end of the sequence, an in-hand pose estimate is provided. The filter masks its input cloud and returns a new point cloud with an equal or fewer number of points. The estimator takes the point cloud and returns a bounding box around such point cloud.

[0074] The filter removes irrelevant data from the inputs of various estimators and is designed to improve the performance (e.g., processing time) or accuracy of the estimators. Examples of filters include, but are not limited to: cropping the point cloud to a fixed region of 3D space; downsampling filters; removing points from the point cloud in a fixed region of 3D space; removing points that are known to be part of the background because background scan data has been acquired previously; removing points that are known to be part of known objects in the space such as tote boxes or robots, and the poses of these objects can change, but the poses of these objects can be known from the sensor; outlier filters such as removing isolated points; smoothing filters; or methods of clustering or segmenting parts of the cloud and filtering the segments.

[0075] The estimator infers an estimate of the position and orientation of the object from the filtered point cloud data and can also optionally estimate the size of the object, e.g., using a deep learning process. For example, see 3D ShapeNets: A Deep Representation for Volumetric Shape Modeling by Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015). Examples of estimators include, but are not limited to: a yaw-based bounding box that flattens the point cloud along the z dimension and then computes the minimum area oriented bounding box (with length, width, and rotation of an OBB) to enclose all points, assuming the height of the OBB is the distance from the bottom of the cloud to the known height of the gripper; a minimum volume 3D box computed directly on the cloud. The estimator can incorporate and propagate uncertainty about the hand pose estimate, e.g., using a multi-dimensional (six for position and orientation only; nine when including the three dimensions of the bounding box as well) covariance matrix; or can represent uncertainty using samples (as in particle filtering). However, such uncertainty can be propagated to the payload protection, thus representing a volume larger than the volume of a single object.

[0076] The final bounding box returned by the known system is at a given position relative to the end effector of the robot. To determine this, the system must first use the extrinsic parameters of the sensor to transform the bounding box determined from the sensor into the robot's coordinate system. Then, since the robot is in a known state at the time of capture, the position and orientation of the item relative to the robot, and the size and shape of the item, the hand pose can be calculated by subtracting the pose of the end effector. The result provided by the system is the transformation of the bounding box to the end effector. If multiple sensors are used, then typically these sensors should be triggered simultaneously, e.g., with a common hardware trigger, so that all images or point clouds of the item are temporally corresponding to the same recorded robot state.

[0077] As described above, according to various aspects, the system can perform a hand pose scan while the robot is stopped at a hand pose position and location or while it is moving. If the robot reaches a stop at the hand pose position and location, then the system knows the position of each joint in the robot's joints and thus knows the precise position and location of the end effector. Specifically, the robot holds an object, reaches a stop at a known position, and then once the robot is stopped, the software can trigger the sensor. Since the robot is stopped, the joint configuration is known accurately.

[0078] However, if the robot does not have to reach a stop and can scan while the arm segment is in motion, then the process will determine the association more quickly, resulting in faster robot operation because the robot will not need to decelerate, pause, and then accelerate again. But to do this, the system will have to accurately obtain the position of the robot during the scan. If there is an error of one centimeter in the position of the robot end effector, then there is at least a corresponding one centimeter error in hand pose estimation and payload protection, which results in inefficiency (and thus requires an increase in margin to compensate). Two sources of inaccuracy are: (a) Generally, the robot controller provides the state of the robot to the processor at a low bandwidth such as 100 Hz; (b) It may be difficult to accurately determine the time of the scan because the sensing unit will have to provide a direct notification and the system will have to be ready to receive the direct notification immediately. In the latter case, if the robot is moving at one meter per second during the scan and the time error between the time of the scan and the time of the joint configuration used is fifty milliseconds, this results in at least a five centimeter error in hand pose estimation and payload protection.

[0079] If an object is scanned during motion, the error distribution of the pose of the end effector is actually one-dimensional, which means that the true pose lies somewhere along the executed trajectory. The spatial distribution of this error can be incorporated into payload protection by expanding along the direction tangent to the trajectory at the estimated capture time without significant expansion in the orthogonal direction. This results in a more conservative payload protection than that produced by uniform expansion in all directions. Specifically, to address the time of the scan, the system triggers the camera from a software application that controls a digital I / O device, sets the pin on the sensor to high level for 20 to 80 ms. Then various methods can be used to recover the joint state at the time of the trigger, such as the following.

[0080] First, the same digital I / O output signals are also routed to the robot controller. The drivers on the robot controller can be programmed to record the time of the I / O trigger and the joint state at the time of the trigger, and then forward the time of the I / O trigger and the joint state at the time of the trigger back to the software application via a message. To solve this problem, the system can interpolate based on the time found above. Assuming that the time of the trigger and the time of the joint configuration are collected relative to a single clock, and two samples of the joint configuration are known before and after the trigger time, linear (or higher-order) interpolation can be performed on the joint configuration to obtain a more accurate estimate of the configuration at the time of transmission.

[0081] Second, after the software system commands the digital I / O device to trigger, it can immediately record the latest joint state received from the robot drive.

[0082] Third, if the motion during the scan is consistent, there may be a contact device mounted on the robot (e.g., the base link of the robot) such that when the first joint of the arm swings through, for example, 135 degrees, it causes the camera to trigger. Then, since the scan trajectory may be monotonic in the base joint, the true joint state at the time of the trigger can be found by interpolating the commanded trajectory as a function of the base joint angle.

[0083] According to a further aspect, the system can employ multi-shot scans for swing detection. As described above, there may be compliance in the end effector (the suction cup may deflect) or the object may be deformable. Thus, the object may swing or otherwise exhibit some dynamics while in motion. To detect this motion, multiple scans of the object can be collected in a timely manner. For each time instance, a hand-held pose estimate can be collected, and since the joint configuration should be known, the motion and swing relative to the end effector can be estimated. A pendulum model or other dynamic models derived from the mechanics of the gripper can be used to extrapolate further motion. Then, by taking the union of the optionally extrapolated volumes measured, these volumes can be used to enlarge the payload protection. Additionally, the system can use the detection of significant swings (such as determined by the presence of a swing sufficient to put the motion at risk of collision) to reduce the speed of the motion. The multi-shot data can also be used to train the parameters of an SKU-specific deformability model. Then, such a model can be used to estimate payload protection before capturing hand-held pose data (e.g., payload protection can be used when planning a trajectory from an initial grasp to a hand-held pose data capture configuration). The payload protection for a hand-held rigid item will be less than that for an object of the same size but with greater hand-held deformability.

[0084] According to a further aspect, the system can employ a size lookup table for pre-computed trajectories. To optimize the time required for planning robotic motions and trajectories, the system can pre-compute various motions in advance to reduce the computational time required during real-time operation. In particular, for robotic trajectories that can be executed while holding an SKU, there can be many discrete options from which to select a trajectory, based on factors such as: (a) the size of the given object being picked up, which is crucial for ensuring a collision-free trajectory of the item with the environment, and / or (b) the orientation of such an object relative to the gripper (handheld posture) while being held, which is also important for ensuring a collision-free trajectory but in some cases is also important for determining how to fit the object into a tightly fitting container / box / etc. Based on the results of the detected handheld posture, the system can intelligently select a pre-computed trajectory to match the requirements for how the system is holding a given object at that time.

[0085] According to a further aspect, the system can employ techniques for relaxation margins to avoid motion planning failures. The concept of relaxed constraints can be employed. To compute a successful, collision-free trajectory, the system can, in some cases, assume a more conservative payload safeguard, where, if successful, the system will note that the most nominal case provides the safest trajectory. However, if not successful, the system can iteratively relax the constraints until a given limit is reached, increasing the chance of finding a successful solution while maintaining an optimal solution in the nominal case.

[0086] According to a further aspect, the system can employ techniques for satisfactory placement. In some applications, for any general robotic trajectory with a payload safeguard, there can be expected to be many acceptable solutions. For example, it may be desirable to ensure that an object is fully contained within a target tote bin when being transported. The system can encode the goal of the motion planning to ensure that the payload safeguard (however constructed) is within the walls of the destination tote bin. Then, after a successfully executed placement, the object should be within the tote bin. If the object is much smaller than the tote bin, then there can be a very large number of valid placements (e.g., some placements to the far part of the tote bin, some placements to the near part of the tote bin). The system can be further encoded to generate many hypothesized placements that meet the goal of placing the payload safeguard within the tote bin; then determine which of those hypothesized positions have feasible, or collision-free, trajectories; and then select the best plan among those plans based on criteria such as speed or distance. This is referred to herein as satisfiability, or selecting an optimal solution among many acceptable options.

[0087] According to a further aspect, the system can employ techniques for adjusting off-gantry grasping. One of the factors in the shape of the payload protection is the nature of the movement of the robot when gripping the payload. The system mainly assumes that the gripper performs a gantry movement, i.e., it does not roll or pitch. However, if the robot gripper rolls or pitches significantly (e.g., more than 30 degrees), then for some suction cup grasps, the payload protection should roll or pitch accordingly.

[0088] In addition, if the robot has to roll more than 90 degrees to bypass an obstacle in the environment to place something in a slot, or has to roll 90 degrees, for example, to perform a hand-held scan so that the barcode points upward, then the kinematically derived payload protection should take into account the mechanical properties of the gripper. Heavy or non-rigid objects will tend to deflect or rotate in the opposite direction to the roll or pitch of the robot gripper in order to minimize the energy of the system. By finding the equilibrium angle that balances the torques due to the suction cups and gravity, the amount of deflection of the payload protection as a function of the angle from the vertical can be determined.

[0089] According to a further aspect, the system can use hand-held pose estimation to employ empirical payload protection. Specifically, if the system has a hand-held pose scanning sensor, the payload protection for the pre-hand-held pose can be improved by leveraging a set of representative post-hand-held poses. In other words, instead of using a potentially overly large payload protection, the system constructs a probabilistic occupancy grid in the coordinate system of the gripper and computes the probability that a given voxel is occupied from the hand-held pose estimation. The probability that each voxel is occupied is the fraction of the cuboid that contains it. Then, the payload protection can be defined as a set of voxels with an occupancy probability greater than, for example, 99%. To achieve this, some systems can be designed to have a hand-held pose scanner, or some systems for training or evaluation may have a hand-held pose scanning sensor. These systems can collect this probabilistic payload protection information for use by other systems.

[0090] According to a further aspect, the system can employ a teardrop model for bagged objects. The object in the bag (or the bag itself) tends not to behave like a rigid object. The following options exist for employing payload protection: (1) know in advance that the object is a bag and employ a geometric model specifically applicable to bags; (2) infer from the sensor data that the object is a bag and then employ a geometric model specifically applicable to bags; or (3) without any prior or reasonable information about the bag, but use the geometric model of the bag as part of a worst-case bound. Figure 20 An end effector 20 with a vacuum suction cup 48 for grasping an item 80 within a bag 82 is shown. Figure 21Shows an end effector 20 with a vacuum suction cup 48 grasping the same item 80 inside a larger bag 84, showing that the bag 84 may swing significantly during handling.

[0091] A possible geometric model of the bag is a teardrop model with two parameters: (1) the surface area of the item inside the bag; (2) the surface area enclosing the bag. The ratio of these parameters determines the degree of item fall, as well as other dynamic properties, such as the degree to which the item can swing (the period of a pendulum depends only on the length of the pendulum). In any case, the swept volume of the possible swing states (2D angles, roll, and pitch) can be used to generate payload protection.

[0092] According to a further aspect, the system can employ pick verification by checking how many objects the robot has picked up from a handling box or other container. It is typically desired that the robot pick up only one object, and thus a verification step is set up to ensure that one and only one object has been picked up, and if it is determined that more than one object has been picked up, the objects are placed back into the handling box. The system receives a 3D point cloud of the product from a depth camera and attempts to fit the point cloud within a bounding box, which is a rectangular cuboid having length, width, and height dimensions of the SKU retrieved from a database.

[0093] In the case where a point cloud sensor obtains data from below the gripper, since a single point cloud sensor can only see the bottom of the product, the point cloud of the product is supplemented with points centered on the suction cup. Then, the system checks multiple orientations and positions for which the supplemented point cloud (the point cloud with increased points) fits the bounding box. If the system cannot find a pose in which the rectangular cuboid encloses the point cloud, the pick is considered to have picked up more than one item, and the object is placed back into the source handling box. A margin can be added to the dimensions in order to reduce the occurrence of false positives and balance these false positives with false negatives.

[0094] According to a further aspect, the system can provide payload protection related to placing an object into a bag. An example application is one where a robot picks up an SKU from a homogeneous or heterogeneous handling box and places the item directly into a grocery bag in order to fulfill a customer order. Handling boxes containing inventory are conveyed to the system, for example, by a conveyor, where each handling box, for example, contains three grocery bags, as Figure 22 shown. The three grocery bags 90 in each handling box 92 can correspond to one or more customer orders. The job of the robot is to transfer a single object 94 from the inventory handling box 96 to the grocery bag 90. The system is commanded by the WMS to fulfill a particular SKU. Figure 23 Shows a robotic arm 98 rolling the object 94 in order to laterally fit it into the grocery bag 90.

[0095] The system is capable of picking an object from the inventory tote 96 and it needs to determine a trajectory to place the picked item into one of three grocery bags. A placement planner is used to compute the trajectory. According to some aspects, three plastic bags can be stretched over the shipping tote and each bag has an opening, for example, of approximately 15 cm by 34 cm and a depth of about 30 cm. The placement planner solves the following problems. The placement planner finds a trajectory to place the item into the grocery bag to avoid the SKU hooking on the grocery bag (if the SKU hooks, it may tear the bag). Additionally, the maximum size of the SKU may be larger than the opening of the bag, so the placement planner may have to compute a reorientation of the SKU to insert the SKU into the bag. Further, yawing the item may not be sufficient. Some SKUs and some ways of grasping them may further require the gripper to roll the SKU in order to fit the SKU into the slot. Thus, the planner is capable of performing roll- and / or pitch-based planning to place the item into the bag. Additionally, the placement planner must also avoid overfilling the grocery bag as this will cause delays or require manual intervention. The placement planner uses hand-held pose estimation and compensates for free and non-free space within the bag and the dynamics of placing the item.

[0096] For example, the workflow can be as follows. First, pick the SKU from the inventory tote. Using a motion planning algorithm that uses kinematics-derived payload protection (i.e., not based on sensor data), move the SKU from the pick location to the hand-held pose scan location in the robot work cell.

[0097] Next, at the hand-held pose scan location, determine the bounding box of the volume held by the gripper. Determine both the pose of the bounding box relative to the gripper and the dimensions of the bounding box.

[0098] Then optionally compare the dimensions of the bounding box with the known dimensions of the SKU. If the observed dimensions or volume exceed the known values, then it can be considered to hold more than one SKU, in which case the robot returns the SKU to the tote.

[0099] In parallel, scan the placement tote (grocery bag) to obtain a point cloud. Generate a height map of the placement bag region (i.e., a discretized height grid above the bottom of the tote) from the point cloud. This involves performing point cloud filtering (via clustering / ML methods) to remove corners that extend through the corners of the plastic bag. Additionally, the edges of the point cloud are filtered out, expecting the item to be large enough to be seen even with the edge filtering.

[0100] Next, a height map is used to generate candidate SKU placement poses that will not overfill the container. There is a first set of preferred candidates that do not involve a roll gripper, followed by a second set of candidates that may involve a pitch or roll gripper (and are slower). The goal is to generate multiple possibilities because some poses may not be achievable - for example, a trajectory may not be found due to collisions.

[0101] To this end, the system first considers yawing the SKU, i.e., rotating the SKU about the axis of the gripper. Two item poses corresponding to four robot poses are tested, where the bounding box axis with the longest horizontal dimension is parallel and perpendicular to the bag. Second, if no placement is found in the first set, the system rolls the item 90 degrees and again considers two yaws 90 degrees apart. Since there are other constraints such as feasibility and object drop height that come into play, more than one robot yaw can be considered for each object. In all cases, the system aligns the base of the rolled bounding box with the rectangular slot corresponding to the grocery bag. This produces a set of candidate placement poses to send to the motion planning algorithm in the next step.

[0102] Each candidate SKU placement pose is used to generate a corresponding candidate robot placement pose. Note that many of these robot placement poses (especially the rolled placements) are infeasible. The system concurrently plans in the joint space of the robot from the hand-held pose node to the robot placement pose. The system also plans candidate robot placement poses in the workspace from these configurations and tries to get as close as possible to the candidate robot placement pose while avoiding collisions. The system then executes the selected trajectory. These measurements result in rolled placements that are experimentally accurate to ~1 - 2 cm.

[0103] According to a further aspect, the present invention provides that the system can develop payload protection related to placing an object into a compartment or chute. Specifically, the application sorts goods into compartments corresponding to customer orders, for example. The walls of an open cube station (or compartment) are referred to as a putwall; this application is a dual-robot putwall. Refer Figure 24 ring to, the object handling system 120 includes a robotic arm 18 having an end effector 20 with a vacuum chuck 48 for picking up an object from a feed tote 122 on a feed conveyor 124 and placing the object into an open cube station 136 provided in one of an array of two open cube stations 130, 132. A person can remove the object from the cube station, or, according to a further aspect, the open cube station can be provided as a chute leading to a further processing location.

[0104] Robot 18 retrieves the SKU from the heterogeneous tote 128, scans its barcode using any one of the multiple scanners 134, and determines into which compartment to place the SKU based on the decoded barcode, and then performs the placement. Generally, it is desirable to have as many compartments as possible that can be reliably and accurately placed by the robot, as this improves the efficiency of picking items from the shelves. Payload guarding can be used to place items into compartments, and thus, having a more closely fitting payload guarding will allow for more compartments.

[0105] Reference Figure 25A , the system uses the hand-held pose information (along with the device 18 joint information) to know that certain placement methods for the selected compartment 136 may not work (be ill-fitting), while other methods as Figure 25B shown will work. The system has information about the size and location of all compartments and uses the hand-held pose information to ensure that an object (e.g., object 140) is placed into the compartment in a placement pose that will fit.

[0106] The processing steps can be as follows. The robot picks one of the SKUs from the heterogeneous tote. Then, when lifted out of the tote, the robot uses the barcode scanner array 128 in the workstation to scan the barcode on the SKU. If no barcode is found, it places it back into the tote or into a separate processing location, such as a compartment designated as an exception compartment. If multiple barcodes are found, which indicates that multiple SKUs have been picked, the SKUs are placed back into the tote.

[0107] If a barcode is found, the system will construct a payload guard using one of three methods. First, if the hand-held pose scanner is not available and if the size of the SKU is unknown, then a worst-case payload guard is constructed from the maximum disposable size of the robot. Second, if the hand-held pose scanner is not available and if the size of the SKU is known, then a worst-case payload guard is constructed from the known size of the SKU. Third, if the hand-held pose scanner is available, then a rectangular prism or other fitting geometry is used as the payload guard.

[0108] Using the payload protection that has now been constructed, the system plans a trajectory from where the robot is after scanning in the handheld position to the target, which is placed within the desired dispensing wall compartments 130, 132. Placement can be defined as the payload protection device being adapted within the rectangle of the compartment entrance and substantially extending beyond the front plane of the dispensing wall, where substantially can mean, for example, any one of the following: one, the payload protection extends beyond the front plane; two, the trailing edge of the payload protection extends at most some distance from the front dispensing wall plane; or three, the center of the payload protection extends at least some distance beyond the front plane. Other manipulations of the payload protection can be employed such that the virtual placement defined for the purpose of motion planning results in a successful placement into the correct compartment without jamming the SKU and it does not result in the wrong compartment. Other techniques can also be used here, such as the satisfiability and mechanical models of the gripper for trajectories that may require significant pitching. The robot then executes the trajectory to place the object, and the system repeats until the tote is empty and waits for a new tote.

[0109] Those skilled in the art will appreciate that many modifications and variations can be made to the embodiments disclosed above without departing from the spirit and scope of the invention.

Claims

1. A system for handling an object using a programmable motion device, the system comprising: an end effector of the programmable motion device for gripping the object from a feed container; and a control system for determining a payload protection for a selected object, the payload protection comprising point cloud data regarding volume data, the volume data including the volume occupied by the selected object, the payload protection being determined in response to at least one characteristic of the selected object and provided specific to the selected object.

2. The system according to claim 1, wherein, The payload protection is provided as an axis-aligned bounding box.

3. The system according to any one of claims 1 to 2, wherein, The payload protection is provided as an axis-aligned bounding cylinder.

4. The system according to any one of claims 1 to 3, wherein, The system is capable of detecting deflection data representative of a deflection of a vacuum suction cup attached to the end effector and holding the selected object, and wherein the payload protection is provided in response to the detected deflection data.

5. The system according to any one of claims 1 to 4, wherein, The end effector includes a vacuum suction cup of diameter d, and wherein the payload protection is reduced in at least one dimension by a distance of the same order of magnitude as the diameter d.

6. The system according to any one of claims 1 to 5, wherein When lifting the selected object from the feed container, the payload protection is provided based on the sensed data of the object.

7. The system according to any one of claims 1 to 6, wherein, When the selected object is held at a defined position, the payload protection is provided based on the sensed data of the selected object.

8. The system according to any one of claims 1 to 7, wherein, When the selected object is moved through a defined position, the payload protection is provided based on the sensed data of the selected object.

9. The system according to any one of claims 1 to 8, wherein, The payload protection of the selected object is determined at least in part based on the position within the feed container occupied by the selected object prior to gripping.

10. The system according to claim 9, wherein, The position within the feed container occupied by the selected object prior to gripping is a position adjacent to the wall of the feed container.

11. The system according to any one of claims 1 to 10, wherein, The system is capable of detecting wobble data representative of a wobble of the selected object, and wherein the payload protection is provided in response to the detected wobble data.

12. The system according to any one of claims 1 to 11, wherein, The payload protection is provided in a substantially teardrop shape in response to sensed data indicating that the selected object includes a non-rigid bag.

13. The system according to any one of claims 1 to 12, wherein, The payload protection includes placement restrictions regarding the position where the programmable motion device places the selected object.

14. The system according to claim 13, wherein, The placement restrictions include either a horizontally or vertically defined opening.

15. A method for handling an object using a programmable motion device, the method comprising: using an end effector of the programmable motion device to grip an object from a feed container; lifting the object from the feed container; and determining a payload protection for the gripped object, the payload protection being derived from point cloud data regarding volume data, the volume data including the volume occupied by the gripped object, the payload protection being determined in response to at least one characteristic of the gripped object and provided specific to the gripped object.

16. The method according to claim 15, wherein, When the gripped object is held by the end effector, determining the payload protection of the object.

17. The method according to any one of claims 15 to 16, wherein, The payload protection is provided as an axis-aligned bounding box.

18. The method according to any one of claims 15 to 17, wherein The payload protection is provided as an axis-aligned bounding cylinder.

19. The method according to any one of claims 15 to 18, wherein, The method further includes detecting data representative of a deflection of a vacuum suction cup attached to the end effector and holding the grasped object, and determining that the payload protection is responsive to the detected deflection data.

20. The method according to any one of claims 15 to 19, wherein The end effector includes a vacuum suction cup having a diameter d, and wherein the payload protection is reduced in at least one dimension by a distance on the order of magnitude of the diameter d.

21. The method according to any one of claims 15 to 20, wherein, When lifting the grasped object from the feed container, the payload protection is provided based on the sensed data of the object.

22. The system according to any one of claims 15 to 21, wherein, When the grasped object is held at the defined location, the payload protection is provided based on the sensed data of the object.

23. The method according to any one of claims 15 to 22, wherein When the grasped object is moved through the defined location, the payload protection is provided based on the sensed data of the object.

24. The method according to any one of claims 15 to 23, wherein, When the grasped object is positioned near the wall of the feed container, the payload protection is limited by at least a portion of the feed container.

25. The method according to any one of claims 15 to 24, wherein The method further includes detecting wobble data representative of a wobbling of the grasped object, and determining that the payload protection is responsive to the detected wobble data.

26. The method according to any one of claims 15 to 25, wherein In response to sensed data indicating that the grasped object includes a non-rigid bag, the payload protection is provided in a generally teardrop shape.

27. The method according to any one of claims 15 to 26, wherein The payload protection includes placement restrictions regarding the location where the grasped object is placed by the programmable motion device.

28. The method according to claim 27, wherein, The placement restrictions include either a horizontally or vertically defined opening.

29. A system for handling an object using a programmable motion device, the system comprising: an end effector of the programmable motion device for grasping the object; a sensing system for determining sensed data regarding the object; and a control system for determining a payload protection for the grasped object, the payload protection being derived from point cloud data regarding volume data, the volume data including the volume occupied by the grasped object, the payload protection being determined responsive to the sensed data.

30. The system according to claim 29, wherein, When lifting the grasped object from the feed container, the sensed data is obtained.

31. The system according to any one of claims 29 to 30, wherein When the grasped object is held at the defined location, the sensed data is obtained.

32. The system according to any one of claims 29 to 31, wherein, When the grasped object is moved through the defined location, the sensed data is obtained.

33. The system according to any one of claims 29 to 32, wherein, The sensed data includes wobble data regarding movement of the object detected when the end effector is not moving.

Images