System and method for automated packaging and processing using object placement attitude control

Abstract

A method of processing an object is disclosed. The method includes: gripping an object with an end effector of a programmable motion device; determining an estimated pose of the object when gripped by the end effector; determining a pose adjustment for repositioning the object to place at a destination position in a destination pose; determining a pose adjustment to be applied to the object; and placing the object at the destination location in a destination pose according to the pose adjustment.

Description

[0001] Priority

[0002] This invention claims priority to U.S. Provisional Patent Application 63 / 419,932, filed Oct. 27, 2022, the disclosure of which is hereby incorporated by reference in its entirety. Background of the Invention

[0003] This invention generally relates to automated sorting and other handling systems and, more particularly, to automated systems for loading, unloading, and handling objects such as packages, parcels, goods, etc. for e-commerce fulfillment, sorting, facility replenishment, and automated storage and retrieval systems (AS / RS).

[0004] A shipping center that packages and ships an object, for example, from a source company that manufactures a limited range of objects, may only need a system and process that repeatedly accommodates that limited range of the same object. On the other hand, a third-party shipping center that receives a variety of objects must utilize a system and process that can accommodate that variety of objects.

[0005] For example, in an e-commerce order fulfillment center, human workers pack units of objects into shipping containers such as boxes or plastic bags. One of the final steps in the order fulfillment center is to pack one or more objects into a shipping container or bag. Order units destined for customers are typically manually packed at a packing station. There are many reasons for the order fulfillment center to do so.

[0006] Objects generally need to be packed in shipping materials. The objects need to be placed in a box or bag to protect the objects, but they are not typically stored in the materials in which they are shipped and instead need to be packed on-the-fly after receiving an order for the object.

[0007] However, there are challenges in loading and unloading a variety of objects on a common conveyance and handling system, particularly where the object has either low pose authority or low placement authority. Pose authority is the ability to place an object in a desired position and orientation (pose), and placement authority is the ability of an object to maintain its placed position and orientation. For example, if an object with low pose authority (e.g., a soft bag) or an object with low placement authority (e.g., a cylindrical object) is to be moved on a conveyance system that may experience changes in shape and / or linear or angular acceleration or deceleration, the object may tip over and / or may fall off the conveyance system.

[0008] As the number of goods and the number of destination locations increase, and further in cases where the system needs to place an object in a relatively small place such as a cubby or a bag or a slot, these requirements become more challenging. Therefore, an automated system for handling objects with low posture authority and / or low placement authority in a loading and unloading object handling system is needed, and an automated system that can place an object into a container, cubby, bag, or slot more easily and conveniently is also needed. Summary of the Invention

[0009] According to one aspect, the present invention provides a method for handling an object, the method comprising: grasping the object with an end effector of a programmable motion device; determining an estimated pose of the object when grasped by the end effector; determining a pose adjustment for repositioning the object to be placed in a destination location in a destination pose; determining the pose adjustment to be applied to the object; and placing the object in the destination location in the destination pose according to the pose adjustment.

[0010] According to another aspect, the present invention provides a method for handling an object, the method comprising: grasping the object with an end effector of a programmable motion device; determining an estimated pose of the object when grasped by the end effector; determining estimated joint positions of a plurality of joints of the programmable motion device associated with the estimated pose of the object; associating the estimated pose with the estimated joint positions to provide placement pose information; and placing the object in the destination location in the destination pose based on the placement pose information.

[0011] According to another aspect, the present invention provides an object handling system for handling an object, the object handling system comprising: an end effector of a programmable motion device for grasping the object; at least one pose-in-hand sensing system for assisting in determining an estimated pose of the object held by the end effector; a control system for determining estimated joint positions of a plurality of joints of the programmable motion device associated with the estimated pose of the object, and for associating the estimated pose with the estimated joint positions to provide placement pose information; and a destination location for placing the object in the destination pose based on the placement pose information. Brief Description of the Drawings

[0012] The following description can be further understood with reference to the accompanying drawings, in which:

[0013] Figure 1 An illustrative schematic diagram of an object handling system according to one aspect of the present invention is shown;

[0014] Figure 2 Shows Figure 1An exemplary schematic enlarged view of a part of the system, which shows a hand gesture sensing system;

[0015] Figure 3A and Figure 3B shows an exemplary schematic bottom view of an object when the object is being held by the Figure 1 end effector, showing that the object is being held in a stationary first hand gesture position ( Figure 3A ) and showing that the object is being held in a stationary second hand gesture position ( Figure 3B );

[0016] Figure 4A and Figure 4B shows the Figure 3A and Figure 3B object in an exemplary schematic side view, showing the object in a first position before sensing data capture during movement ( Figure 4A ), and showing the object in a second position after sensing data capture during movement ( Figure 4B );

[0017] Figure 5 shows an exemplary schematic diagram of an object placement attitude control system used with a box packaging system according to an aspect of the present invention;

[0018] Figure 6A and Figure 6B shows an exemplary schematic plan view of an object covering a part of a box packaging, showing the part of the box packaging material before cutting ( Figure 6A ) and after cutting ( Figure 6B );

[0019] Figure 7A and Figure 7B shows an exemplary schematic plan view of another object covering a part of a box packaging, showing the part of the box packaging material before cutting ( Figure 7A ) and after cutting ( Figure 7B );

[0020] Figure 8 shows an exemplary schematic diagram of an end effector used according to an aspect of the present invention, wherein a vacuum chuck is attached in a coordinate environment;

[0021] Figure 9 shows an exemplary schematic diagram of an object shown in various face-up positions in a coordinate environment;

[0022] Figure 10A and Figure 10B shows the Figure 8 end effector in an exemplary schematic diagram, showing that the object has been placed face-up on the conveyor with a minimum area ( Figure 10A), and shows the object subsequently falling as the conveyor moves ( Figure 10B );

[0023] Figure 11A and Figure 11B shows Figure 8 an exemplary schematic view of an end effector, showing an object placed with its smallest face up rotating on a conveyor ( Figure 11A ), and shows the object subsequently falling as the conveyor moves ( Figure 11B );

[0024] Figure 12A and Figure 12B shows an object being placed on a conveyor ( Figure 12A ) such that its center of mass is offset by a trailing distance from the contact area on the moving conveyor, provided that the object undergoes a controlled fall on the conveyor ( Figure 12B );

[0025] Figure 13A and Figure 13B shows an object being placed on a conveyor ( Figure 13A ) such that its center of mass is offset by a distance perpendicular to the direction of movement of the conveyor, assuming that the object undergoes a controlled fall on the conveyor ( Figure 13B );

[0026] Figure 14 shows an exemplary schematic view of an object being placed in a shallow box to initiate a controlled fall;

[0027] Figure 15 shows an exemplary schematic view of an object being placed in a taller box that already contains other objects to initiate a controlled fall;

[0028] Figure 16 shows an exemplary schematic view of a part of an object handling system that includes a bag into which an object falls, where the object is grasped with its largest face up;

[0029] Figure 17 shows an exemplary schematic view of an object handling system that includes a bag as shown in Figure 16 , where the object is grasped with its largest face up and is positioned to drop the object into the bag;

[0030] Figure 18 shows an exemplary schematic view of a part of an object handling system that includes an auto - bagging system into which an object is dropped, where the object is grasped with its largest face up;

[0031] Figure 19 shows an exemplary schematic view of an object handling system that includes a bag as shown in Figure 18Exemplary schematic diagram of an object handling system of an automatic bagging system, wherein the largest face of the object is grasped upwards and positioned to drop the object into an opening of the automatic bagging system;

[0032] Figure 20 Exemplary schematic front view showing an object with its largest face upwards being held by an end effector, the object being moved to a specified container of known dimensions;

[0033] Figure 21 Shows Figure 20 of the object when it can be placed into Figure 20 the specified container in multiple exemplary schematic plan views of its orientations;

[0034] Figure 22 Exemplary schematic rear view showing an object handling system according to another aspect of the present invention, the physical handling system including an array of vertically stacked compartments;

[0035] Figure 23 Shows Figure 20 an exemplary schematic enlarged front view of the system;

[0036] Figure 24A And Figure 24B Shows an exemplary schematic plan view of an object handling system according to one aspect of the present invention, wherein the system attempts to place an object into a compartment where the object is not aligned with the opening ( Figure 24A ), and wherein the system places the object into a compartment where the object is aligned with the opening ( Figure 24B );

[0037] Figure 25A And Figure 25B Shows an exemplary schematic plan view of an object handling system according to one aspect of the present invention, wherein an object is being loaded into a container with similar objects at a first position ( Figure 25A ), and wherein an object is being loaded into a container with similar objects at a second position ( Figure 25B );

[0038] Figure 26A And Figure 26B Shows an exemplary schematic side view of an object being placed onto a surface for repositioning while being held by its largest face in a first orientation ( Figure 26A ), and after repositioning the object is re - grasped by an end effector ( Figure 26B );

[0039] Figure 27A And Figure 27BIllustrates an exemplary schematic plan view of another object handling system according to another aspect of the present invention, where an object is being loaded into a container with similar objects at a first position ( Figure 27A ), and where an object is being loaded into a container with similar objects at a second position ( Figure 27B );

[0040] Figure 28A and Figure 28B Illustrates an exemplary schematic side view of an object held in a second orientation by its largest face being placed onto a surface for repositioning ( Figure 28A ), and after repositioning the object is re - grasped by an end - effector ( Figure 28B ); and

[0041] Figure 29A and Figure 29B Illustrates an exemplary schematic plan view of another object handling system according to another aspect of the present invention, where an object is loaded into a container with similar objects at a first position ( Figure 29A ), and where an object is loaded into a container with similar objects at a second position ( Figure 29B ).

[0042] The drawings are shown for illustrative purposes only. Detailed Description

[0043] According to various aspects, the present invention provides an object handling system 10 that includes a processing station 12 in communication with an input conveyor system 14 and a processing conveyor system 16, as Figure 1 shown. The processing station 12 includes a programmable motion device 18 having an end - effector 20 attached at its distal end. The end - effector 20 can be coupled (e.g., via a hose) to a vacuum source 22, and the operation of the programmable motion device can be provided by one or more computer processing systems 24 that communicate with one or more control systems 100, which communicate with all of the sensing units, conveyors, and additional processing systems disclosed herein.

[0044] The object handling system 10 further includes a hand - in - hand pose sensing system 26 that can be used to determine the pose of an object held by the end - effector 20. Figure 2Shows a hand pose sensing system 26 (having one or more sensing units), which is pointing upward and has an observation area indicated schematically at 27. The sensing system 26 can include either a 2D or 3D sensor and / or a camera, which calculates a virtual bounding box around the point cloud data captured (e.g., by one or more 3D cameras), with the constraint that the bounding box touches the gripper. The geometry or size of the object may be previously known or previously unknown. If known, the geometry or size can be used a priori to fuse with the noisy hand pose estimate.

[0045] An additional sensing system 28 can also be employed to observe the object on the end effector 20, each additional sensing system having an observation area indicated schematically at 29. The end effector 20 includes a vacuum chuck 30, and the sensing systems 26, 28 are pointed at a virtual bounding box 31 defined to touch the vacuum chuck 30. The object 32 is grasped and moved from the input container 34 on the input conveyor system 14, and the object moves above the hand pose sensing system 26.

[0046] According to some aspects, when the object is above the hand pose sensing system 26, the programmable motion device 18 can stop moving so that the pose of the object 32 on the vacuum chuck 30 can be determined. The determined hand pose is associated with the joint positions of each joint of the articulated arm portion of the programmable motion device. In this way, the system not only records the hand pose of the object held by the gripper, but also records the exact position of each articulated part of the programmable motion device. In particular, this means that the exact positions of the end effector 20 and the gripper 30 are known. Knowing these positions (in space), the system can be subtracted from any sensing data associated with the object. Thus, the system can also know all the positions, locations, and orientations where the object can be moved to in the robotic environment and oriented in the robotic environment. The sensing units 26, 28 are set at known external calibration positions. In response to the determined hand pose, the system can move the object (e.g., 32) to a desired position (e.g., a bin or a conveyor surface) among any of various positions, locations, and orientations in response to the determined hand pose.

[0047] For example, Figure 3A and Figure 3B Shows a bottom-up hand pose view when the object is held by the end effector. For example, Figure 3A Shows the situation when the programmable motion device 18 holds the object 32 in a hand pose position (stationary). Since the programmable motion device is not moving, data regarding all joint positions can be easily determined. According to additional aspects, the programmable motion device 18 can then move to a secondary position (as Figure 3BAs shown, at this secondary position, the hand pose is determined again to be associated with additional data sets regarding the new joint positions. Robust hand pose data can thus be determined to be associated with specific joint positions of the programmable motion device 18, thereby providing information about the volume in the space occupied by the end effector and the gripper, and providing information about the potentially available (and unavailable) positions and orientations of the arm portion of the programmable motion device 18.

[0048] According to another aspect, the system can determine the hand pose while the object is moving. However, one challenge is that the response time between capturing the hand pose image and determining the joint positions of the articulated arm (either before or after image capture) may introduce significant errors. According to one aspect, the system can record the position of each joint (e.g., 40, 42, 44, 46, 48) immediately before the hand pose sensing system 26 captures the sensing data, and the position of each joint (e.g., 40, 42, 44, 46, 48) immediately after the sensing data is captured. For example, the joints can include joint 40 (rotation of the bracket 41 relative to the support structure), joint 42 (pivoting of the first arm portion relative to the bracket 41), joint 44 (pivoting of the arm portion), joint 46 (pivoting of the arm portion), and joint 48 (rotation and yaw of the end effector). For example, Figure 4A shows a system in which the programmable motion device 18 is in a first position before the sensing data is captured, and Figure 4B shows the programmable motion device 18 in a second position after the sensing data is captured. Then, the system can interpolate the set of joint positions to estimate the position of each joint at the time of sensing data capture. In particular, the system can interpolate the joint positions of each of the joints 40, 42, 44, 46, and 48 between the corresponding joint positions (e.g., Figure 4A ) before the sensing data is captured and the corresponding joint positions (e.g., Figure 4B ) after the sensing data is captured.

[0049] According to another aspect, the trajectory from the hand pose node to the placement position can be pre-computed. In particular, the system can discretize the desired placement position (x,y) and its orientation at the center of the position where the gripper is located. Then for each of X x Y x θ probabilities, the system can pre-compute the motion plan. Then, when the system looks up (x,y,θ) in the look-up table, the system can interpolate or blend the motion plans between two or more nearby pre-computed trajectories in order to increase the placement accuracy.

[0050] The object placement pose control system can be associated with, as Figure 5used in conjunction with the box packaging system 50 shown. The box packaging system 50 can receive objects (e.g., 33, 35) that have been purposefully positioned on the conveyor 16 and individually wrap each object in box packaging material 52, which is supplied as a continuous stack of raw material into the box material cutter and assembler 54. See, for example, the CMC CartonWrap system sold by CMC S.P.A, Perugia, Italy. The objects (e.g., on the conveyor belt) are received and wrapped with packaging material such as cardboard. The packaged objects (e.g., 56) are then provided on the output section 58 of the conveyor. The goal of such a system is to not only cleanly and protectively package each input stream in the object input stream but also use the least amount of box packaging material (e.g., cardboard) in the process. The box packaging material 52 can be provided in the form of panels (e.g., 53), which are supplied to the box material cutter and assembler 54. The panels 53 can be releasably joined together such that they are easily separable from each other. Inside the box material cutter and assembler 54, the panels are cut to the desired size to form a box around each individual object.

[0051] By determining the hand posture when the object is held, the object can be placed on the conveyor (e.g., 16) in an orientation designed to minimize waste of the box packaging material. For example, Figure 6A shows an object 33 placed longitudinally and covering (schematically) a portion 53 of the box packaging material. In the case where the object is placed longitudinally, the panel 53 can be cut in a way that produces a smaller panel 53' and additional material 55 as shown. Figure 6B Figure 6B Figure 7A shows an object 35 placed transversely and covering (schematically) a portion 53 of the box packaging material. In the case where the object is placed transversely, the panel 53 can be cut in a way that produces an even smaller panel 53” and a greater amount of additional material 55’ as shown. Figure 7B Figure 7B

[0052] Determining whether the object is placed longitudinally or transversely on the conveyor depends on the specific application, but once the hand posture has been determined, the system can appropriately supply the object to the box packaging system (e.g., 50). Specific rules can be developed, such as not placing the object transversely in the case where the width W O of the object is greater than the width W P of the panel. Further rules can include: if W O+2*H O +Margin > W P , then place the object vertically (as Figure 6A shown), and if W O +2*H O +Margin < W P , then place the object horizontally (as Figure 7A shown), where H O is the height of the object. The system provides a closed-loop placement system that is based on in-hand pose analysis when processing each object. The objects can be set in heterogeneous or homogeneous totes, and can be placed on the conveyor before the box packaging system in a way that minimizes the use of box material (e.g., cardboard) raw materials by obtaining and using in-hand pose analysis to properly place the objects. In additional applications, such as for heterogeneous totes, the system can also include additional scanners (e.g., fly-over scanners) to identify SKU or other barcode information about the incoming objects.

[0053] Figure 8 Shows an end effector 20 attached with a vacuum chuck 30 in a coordinate environment, showing the width (W) or X direction, the length (L) or Y direction (and the conveyor direction), and the height (H) or Z direction. As shown, the system describes a virtual bounding box 31 in the area of the vacuum chuck 30, and uses an in-hand pose sensing system to record all the point cloud data points of all the points in the virtual bounding box of the point cloud. If the object is placed at a position where the end effector is held at almost the same position, the object will be described as having a width in the W (or X direction), a length in the L (or Y direction or conveyor direction), and a height in the H (or Z direction).

[0054] This placement orientation defines a tipping risk factor based on both the relative size of the face-up and the size of the object's dimensions in the conveyor direction. Figure 9 Shows the minimum face-up (as shown at 60), with a higher tipping risk, while the medium face-up (as shown at 62) has a lower tipping risk. The tipping risk is the lowest when the face-up is the largest (as shown at 64). The boundary sphere can be described as being close to the Z-axis as shown at 66, and outside this boundary sphere the tipping risk becomes a greater concern.

[0055] As described above, when an object is taken out from an inventory tote or an input conveyor and placed on a processing conveyor belt for feeding into a subsequent system (such as a box packaging system), the object must have sufficient placement authority, especially since the receiving surface (e.g., the processing conveyor system 16) is moving. Therefore, the system can evaluate both the pose authority and the placement authority for grasping the object.

[0056] The end effector of the programmable motion device picks up an object from an input area (e.g., from a tote box) and places it on a conveyor of a processing conveyor system. If the SKU is packed in the tote box with its shortest dimension vertical, everything will proceed smoothly. As described above, the robot will use the in-hand pose to orient the object (e.g., in a way that minimizes cardboard usage). However, if the object is packed such that its largest dimension is vertical, there may be a problem that the SKU may tend to tip over after being placed on the conveyor belt. Tipping not only causes problems such as those in the subsequent processing stations described above, but also causes the object to jam in the conveyor system.

[0057] For example, Figure 10A FIG. shows the end effector 20 of the programmable motion device just placing the object 70 on the conveyor belt 72 of the processing conveyor system using the vacuum chuck 30. When the vacuum chuck 30 moves away from the object, the conveyor belt 72 and the object 70 move in the processing direction as shown at A. Due to the largest dimension being vertical and / or due to the movement of the conveyor belt (especially in the case where the smallest dimension is in the direction of the conveyor belt), the object may tip over (fall over). This may occur as Figure 10B shown, or the object may fall forward. Additionally, the object may land unevenly on its bottommost corner, causing the object to rotate as it falls. Further, if the object 70 is placed on the conveyor belt 72 with its largest dimension vertical and its smallest dimension in the lateral direction of the conveyor belt (as Figure 11A shown), the object 70 may tip over in the lateral direction of the conveyor belt (as Figure 11B shown), potentially causing a jam downstream in the processing conveyor system. Any of these events results in uncertainty in the system regarding the placement of the object, and this uncertainty hinders the effort to control the pose of the object during processing.

[0058] According to various aspects, the present invention provides an object that can be placed entirely in a laid-down position or can be placed such that its center of mass is offset from the contact point in the direction in which it is desired to be laid down. Thus, according to one aspect, the object can be reoriented such that it can be gently dropped (or placed) such that its shortest dimension is vertical.

[0059] Referring again to Figure 8 and Figure 9 , the system can determine whether the object is being held with its largest face up (LFU), medium face up (MFU), or smallest face up (SFU) based on an in-hand pose estimate. First, the dimensions (d 1 , d 2 , d 3 ) in the product database (when available) can be sorted such that d 3 ≤d 2 ≤d1 Then, an e can be assigned to the estimated dimensions from the hand pose sensing system 1 , e 2 , e 3 such that e 1 ≥ e 2 and e 3 is arbitrary. Any one of several poses (p 1 , p 2 , p 3 ) can be determined as follows:

[0060] SFU: e 3 > e 1 && e 3 < e 2 => d 1 , d 3 , d 2 -> p 1

[0061] MFU: e 1 > e 3 && e 3 < e 2 => d 2 , d 3 , d 1 -> p 2

[0062] LFU: e 1 > e 2 > e 3 => d 1 , d 2 , d 3 -> p 3

[0063] The hand pose estimate may not always be completely accurate, and in some applications, it may be desirable to compare this estimate with the hand pose estimate or otherwise employ database measurements when evaluating the hand pose estimate.

[0064] The tipping tendency is further determined by the acceleration of the object once it is transferred to the conveyor (since the end effector does not travel with the conveyor) and any acceleration or deceleration of the conveyor while conveying the object. According to another aspect, the system can move the end effector at the speed and direction of the conveyor during transfer. The tipping tendency can be determined in a variety of ways, including whether the height dimension is greater than twice the width or length (H > 2W or H > 2L), and this can be modified by any acceleration of the conveyor belt to (H > 2W - ɑ|Acc| or H > 2WL - ɑ|Acc|), where ɑ is a factor applied to any acceleration or deceleration |Acc| of the conveyor belt during processing.

[0065] As described above, when it is desired to change the orientation of an object from a determined hand orientation (e.g., SFU to LFU), the system can place the object in its MFU orientation (e.g., on the intermediate side). However, in some applications, this may require movement of a large number of joints of a programmable motion device. Refer to Figures 12A to 12B , the system can place the object 70 on the conveyor 72 at a determined angle such that the center of mass (CM) of the object (e.g., as schematically shown at 74) is offset by a trailing distance from the contact area 76 (e.g., line) of the object 70 on the conveyor 72. As the conveyor 72 moves as shown at A, the object will fall in the direction of the trailing distance in a controlled manner as Figure 12B shown. Figure 13A shows the object 70 placed along the cross-belt direction, and Figure 13B shows the object falling along the cross-belt direction in a controlled manner.

[0066] For any given object, it may be sufficient to place the CM above the edge; there is no need to hold the object at 90 degrees to reorient the object. If the object is placed on its edge, the object will tip over on the remaining path unless the acceleration of the object when placed on the conveyor disrupts the fall of the object. In some applications, it is necessary to place the object such that the CM is behind the contact edge in the direction of movement of the conveyor ( Figure 12A ). The object can be placed such that, for example, the CM can be at least 1 cm for a smaller object and at least about 3 cm to about 5 cm for a larger object. In some applications, the distance between the CM and the contact edge can be based on the height H when the object is being held by the end effector, e.g., the CM is at least 1 / 10H to 1 / 2H from the contact edge.

[0067] Thus, the strategy can be to place the object at a height where the edge just touches the conveyor and at a fixed angle according to the worst-case scenario of H = 2W, or at an angle depending on the CM. Either can be chosen to balance the execution of the trajectory and minimize bouncing. Another strategy can be to reorient the object by a certain number of degrees away from the vertical direction, e.g., about 15 degrees, 20 degrees, 30 degrees, or 45 degrees away from the vertical direction. Taller items may require a smaller angle but will also tend to fall through a larger total angle, which may result in undesirable bouncing and unpredictable behavior. A further strategy can be to rotate 90 degrees (or any required angle) completely to make the LFU face parallel to the conveyor.

[0068] Further referring to Figure 14 , the object 80 can be placed in the bin 82 on the conveyor 84 to start falling in a controlled manner as described above. AsFigure 15 As shown, the object can be further placed into the bin 86 that already contains the object 88 such that the object is placed on the other object 88 in a manner that begins a controlled fall onto the object 88 in the bin 86.

[0069] According to another aspect, the hand pose placement pose control system can be used in conjunction with a bagging station where an object may need to be positioned in a desired orientation to be placed into one of a plurality of bags. For example, Figure 16 a system is shown where an object (e.g., 90) is removed from an input container 34 and placed (or dropped) into a bag 92 in a processing container 94. When the object 90 is grasped by the vacuum suction cup 30 of the end effector 20 of the programmable motion device 18 on the MFU (or SFU) surface, the object 90 will neatly fit into the top opening of the selected bag 92. However, referring to Figure 16 , when the object 90 is grasped by the vacuum suction cup 30 of the end effector 20 of the programmable motion device 18 on the LFU surface, the object 90 will not neatly fit into the top opening of the selected bag 92. The system will identify this situation based on hand pose analysis and will rotate the end effector and the object to position the object 90 such that the object will enter the bag opening via its MFU (or alternatively its SFU) side, as Figure 17 shown.

[0070] Thus, the object is transferred by the programmable motion device to a hand pose scan location where the relative pose (orientation and position) of the object with respect to the gripper is determined. Optionally, a height map of the destination bag is generated at the same time. This includes performing point cloud filtering (via a clustering / machine learning method) to remove the corner regions that extend across the corners of the plastic bag. Additionally, the edges of the point cloud are filtered, with the expectation that even with edge filtering the object will be large enough to be seen.

[0071] Next, candidate object placement poses that will not overfill the container are generated using the height map. The system considers yawing the object both parallel and perpendicular to the bag. If no position is found, the system rolls the object 90 degrees and again considers two yaws 90 degrees apart. In all cases, the system aligns the base of the object with the base of the container to minimize the bounce force during placement.

[0072] Each candidate object placement pose is used to generate a corresponding candidate robot placement pose. Note that many of these robot placement poses (especially for rolling placements) are infeasible. Next, the system simultaneously plans the joint space from the in-hand pose node to the TSR above the candidate robot placement pose. The system also uses greedy inverse kinematics to plan candidate placement poses of the robot in the workspace from these configurations and attempts to approach the candidate robot placement pose as closely as possible while avoiding collisions.

[0073] During a rolling placement, a default release may eject the item with a significant force and may make precise placement difficult. Therefore, the system takes the following steps to reduce the ejection force: 1) use a disengaged gripper to reduce the gripper spring force; 2) add a one-way valve to the cup to reduce the air ejection generated when the valve opens; 3) harden the bellows to reduce the spring force; and 4) use a multi-stage valve release that quickly opens the valve halfway and then continues to open the valve slowly until the item drops. These measures result in a rolling placement that is experimentally accurate to approximately 1 cm to 2 cm. The yaw gripper adds an additional degree of freedom, reducing both the trajectory duration and the planning failures (cases where the robot cannot find a trajectory given the PIH and the target).

[0074] According to yet another aspect, the in-hand pose placement pose control system can be used in conjunction with an automated bagging station where objects may need to be positioned in a desired orientation to be placed into a slot of an automated bagging system (such as the Sharp system sold by Pregis Corporation, NY, NY). The bags thus formed can be shipping packages for non-rigid objects (such as envelopes), or the object itself can be a non-rigid object inside an envelope. Figure 18 A system is shown where an object (e.g., 1O2) is removed from an input container 34 and placed (or dropped) into an automated bagging system 104. When the object 102 is grasped by the vacuum chuck 30 of the end effector 20 of the programmable motion device 18 on the MFU (or SFU) surface, the object 90 will neatly fit into the top opening of the automated bagging system 104. However, referring to Figure 19 , when the object 102 is grasped by the vacuum chuck 30 of the end effector 20 of the programmable motion device 18 on the LFU surface, the object 102 will not neatly fit into the top opening of the automated bagging system 104. The system will identify this situation based on in-hand pose analysis and will rotate the end effector and the object to position the object 102 such that the object will enter the opening of the automated bagging system via its MFU (or alternatively, its SFU) side, as Figure 19As shown. As shown in 106, a bag will be formed around the object, and the bag will be separated from the system 104 and stored on the handling conveyor system 16. The object must be correctly oriented to fit into the opening of the automated bagging system, and knowing the hand pose and the joint positions of the device 18 allows the system to achieve this.

[0075] In some applications, the system can try several different poses, but in some cases, the system can also optionally sample from a continuous (i.e., infinite) number of possible valid poses. This will give the system more possibilities in cases where some inverse kinematic solutions fail (e.g., due to collisions). Additionally, some poses that still achieve the goal of placing the object into a slot / bag / compartment may be faster than those that align it precisely. However, the tighter the fit, the smaller the satisfactory region. When the dimension of the pose space is small such as only one angle, the system can calculate the angle limits and sample the range discretely. When the pose space is 6D (e.g., x, y, z, roll, pitch, yaw), the system can sample randomly around the centered and axis-aligned poses. Inverse kinematics can be employed here.

[0076] Specifically, there may be multiple joint configurations that result in the same pose. Inverse kinematics (IK) typically returns all roots. The inverse kinematic solution can be found by transforming from the joint space (j 1 ,j 2 ,j 3 ,j 4 ,j 5 ,j 6 ) to the gripper pose space (x, y, z, roll, pitch, yaw) using forward kinematics. Thus, inverse kinematics can transform from the joint space (j 1 ,j 2 ,j 3 ,j 4 ,j 5 ,j 6 ) to any of the following:

[0077]

[0078] Once the inverse kinematic solutions are found, they are checked for self-collisions (robot to itself) and collisions with the environment (no part of the robot or the item it holds collides with the virtual model of the workspace).

[0079] In applications where the object is placed into a container (e.g., box, crate, or tote), the system can select (which placement pose is suitable) from a set of determined placement poses (or a database of possible placement poses) that specify the object in the container. For example, Figure 20A system is shown where an object 110 held by a vacuum chuck 30 of an end effector is to be moved to a specified container 112. The allowed placement postures (e.g., as shown at 114) can be known or dynamically determined. In some cases, there may even be a large (possibly infinite) number of possible valid postures. This will give the planning system more possibilities in the case where some solutions fail (e.g., due to collisions). Additionally, some placement postures can achieve the goal of placing the object in a slot / bag / compartment faster than other placement postures that place the object in a precisely aligned position (low tolerance). However, the tighter the fit, the smaller the satisfactory region. When the dimension of the placement posture space is small such as only one angle, the system can calculate the angle limits and discretely sample the range. When the placement posture space is 6D (e.g., x, y, z, roll, pitch, yaw, etc.), the system can randomly sample around the centered and axis-aligned postures. When seeking to place an object in a container, the system can select the most efficient orientation suitable for the container. As Figure 21 shown, many different placement postures are acceptable.

[0080] According to another aspect, the object handling system can additionally use in-hand posture information to assist in placing an object into a vertically stacked compartment. For example, Figure 22 and 23 show a system that includes a vertical array of eighteen compartments adjacent to a processing conveyor system 16. Figure 22 The system is shown from the back, showing that the compartments are open at the back, and Figure 23 the system is shown from the front, showing that the compartments can be accessed by the end effector from the front. Some of the objects handled by the object handling system can be selected to be placed into one or another of the compartments (e.g., for handling by a human operator).

[0081] Referring to Figure 24A , the system uses in-hand posture information (along with the joint information of device 18) to know that certain placement methods for a selected compartment 122 may be infeasible (not suitable), while other methods as shown in Figure 24B will be feasible. The system has information about the size and positioning of all compartments and uses in-hand posture information to ensure that an object (e.g., object 124) is placed into a compartment in a suitable placement posture.

[0082] According to certain aspects, the system can include an application to a cell where two stock bins can reach the cell (similar to one of the packing cells in a packing unit). The cell can be specifically designed for tote consolidation, or tote consolidation can be a part-time job when it is idle.

[0083] In the tote merge mode, two totes arrive at the cell, one tote is the source and the other is the destination, and both totes can come from an AS / RS (Automated Storage and Retrieval System). They can be homogeneous or heterogeneous, and the tote can be subdivided or not. The source tote / subdivision typically has only a few remaining SKUs. In the homogeneous case, the job is to pick up all the remaining SKUs from the source and place them in the destination, presumably with other units of the same SKU. In the heterogeneous case, all units or all units of a given set of SKUs are transferred from the source to the destination. The aim is to improve storage efficiency in the AS / RS. If two totes in the AS / RS have the same SKU, the system can merge those SKUs into one tote to make room for more SKUs.

[0084] Thus, according to some aspects, the object handling system can additionally use in-hand pose information to assist in merging objects in a container (e.g., a tote or a bin), and in managing the efficient packing of the container. Figure 25A Illustrated is an object 130 being loaded into a container 132 that has similar objects positioned in an LFU in a first orientation; and Figure 25B Illustrated is an object 130 being loaded into a container 134 that has similar objects positioned in an LFU in a second orientation. The in-hand pose information is used to assist in placing the object into containers 132, 134 to efficiently pack the containers.

[0085] However, in some applications, it may be desirable to change the in-hand pose position of an object on the vacuum chuck 30 of the end effector 20 from, for example, LFU to MFU. Figure 26A Illustrated is the end effector placing an object on a support surface (e.g., the conveyor belt of the handling conveyor system 16 when stopped), Figure 26B Illustrated is the vacuum chuck 30 of the end effector 20 re-gripping the object 130, this time from MFU.

[0086] Figure 27A Illustrated is an object 130 being loaded into a container 136 that has similar objects positioned in an MFU in a first orientation; and Figure 27B Illustrated is an object 130 being loaded into a container 138 that has similar objects positioned in an MFU in a second orientation. The in-hand pose information is used to assist in placing the object into containers 136, 138 to efficiently pack the containers.

[0087] In additional applications, it may be desirable to change the in-hand pose position of an object on the vacuum chuck 30 of the end effector 20 from, for example, LFU to SFU. Figure 28AShows the end effector placing an object on a support surface (e.g., the conveyor belt of the handling transfer system 16 when stopped). Figure 28B Shows the vacuum chuck 30 of the end effector 20 re - grasping the object 130, this time from the SFU.

[0088] Figure 29A Shows the object 130 being loaded into a container 140 that has similar objects positioned with SFUs in a first orientation; and Figure 29B Shows the object 130 being loaded into a container 142 that has similar objects positioned with SFUs in a second orientation. In - hand pose information is used to assist in placing the objects into the containers 140, 142 in order to efficiently pack the containers. As can be seen from the comparison of the merges from the LFU, MFU, and SFU, packing with the SFU can provide the maximum number of objects in the container.

[0089] Thus, according to various aspects, the system can perform the following steps: scan the input container; pick up an object from the input container with a gripper; perform an in - hand pose sensing analysis on the object while the object is held by the gripper; scan the destination container with a 3D scanner; perform a packing plan for each packing plan job given the in - hand pose of the object; place the object and repeat. Exceptions include: if picked up twice, detect conventionally with a scale or PIH; if dropped, call for intervention; if the conveyor is jammed, call for intervention.

Claims

1. A method of handling an object, the method comprising: grasping the object with an end effector of a programmable motion device; determining an estimated pose of the object when grasped by the end effector; determining a pose adjustment for repositioning the object to be placed in a destination location in a destination pose; determining the pose adjustment to be applied to the object; and placing the object in the destination location in the destination pose according to the pose adjustment.

2. The method according to claim 1, wherein the method further comprises determining a joint position of each of a plurality of joints of the programmable motion device.

3. The method according to claim 2, wherein the joint position is associated with the estimated pose.

4. The method according to claim 3, wherein the joint position is determined when the end effector is positioned in a hand pose position.

5. The method according to claim 3, wherein the joint position is an estimated joint position determined by interpolation.

6. The method according to claim 1, wherein determining the estimated pose of the object is performed while the end effector is moving.

7. The method according to any one of claims 1 to 6, wherein determining the pose adjustment includes determining a tipping risk factor.

8. The method according to any one of claims 1 to 7, wherein placing the object in the destination location in the destination pose involves positioning the object with the end effector such that the center of mass of the object is outside a contact area where the object contacts the destination location.

9. The method according to any one of claims 1 to 8, wherein positioning the object in the destination location in the destination pose involves moving the end effector at least about 15 degrees from a vertical direction before releasing the object to the destination location.

10. The method according to any one of claims 1 to 9, wherein determining the pose adjustment includes determining that any one of the largest face, smallest face, or other face of the object faces upward when placed in the destination location.

11. The method according to any one of claims 1 to 11, wherein adjusting the pose of the object in response to the pose adjustment involves re-grasping the object.

12. The method according to claim 11, wherein re-grasping the object involves re-grasping the object on a face different from an initial face on which the object was initially grasped.

13. The method according to any one of claims 1 to 12, wherein adjusting the pose of the object in response to the pose adjustment involves placing the object on a repositioning surface.

14. The method according to claim 13, wherein the repositioning surface is part of a conveyor.

15. The method according to claim 13, wherein placing the object on the receiving surface involves tipping the object on the receiving surface.

16. The method according to any one of claims 1 to 15, wherein the destination location includes a bag, and the pose adjustment involves aligning opposite sides of the object with inner sidewalls of an opening of the bag.

17. The method according to any one of claims 1 to 16, wherein the destination positioning includes a slot, and the attitude adjustment involves aligning opposite sides of the object with the inner sidewalls of the slot.

18. The method according to any one of claims 1 to 17, wherein the destination positioning includes a compartment, and the attitude adjustment involves aligning opposite sides of the object with the sidewalls of the compartment.

19. A method of handling an object, the method comprising: grasping the object with an end effector of a programmable motion device; determining an estimated attitude of the object when grasped by the end effector; determining estimated joint positions of a plurality of joints of the programmable motion device associated with the estimated attitude of the object; associating the estimated attitude with the estimated joint positions to provide placement attitude information; and placing the object in the destination positioning in a destination attitude based on the placement attitude information.

20. The method according to claim 19, wherein the joint positions are determined when the end effector is positioned in a hand attitude position.

21. The method according to claim 19, wherein the joint positions are estimated joint positions determined by interpolation.

22. The method according to claim 19, wherein determining the estimated attitude of the object is performed while the end effector is moving.

23. The method according to any one of claims 19 to 22, wherein determining the attitude adjustment includes determining a tipping risk factor.

24. The method according to any one of claims 19 to 23, wherein placing the object in the destination attitude in the destination positioning involves positioning the object with the end effector such that the center of mass of the object is outside the contact area where the object contacts the destination positioning.

25. The method according to any one of claims 19 to 24, wherein positioning the object in the destination attitude in the destination positioning involves moving the end effector from a vertical direction by at least about 15 degrees before releasing the object into the destination positioning.

26. The method according to any one of claims 19 to 25, wherein determining the estimated attitude includes determining that any one of the largest face, smallest face, or other face of the object faces upward when placed in the destination positioning.

27. The method according to any one of claims 19 to 26, wherein the placement attitude information includes information related to re-grasping the object.

28. The method according to claim 27, wherein re-grasping the object involves re-grasping the object on a face different from the initial face on which the object was initially grasped.

29. The method according to any one of claims 19 to 28, wherein placing the object on a receiving surface involves tipping the object on the receiving surface.

30. The method according to any one of claims 19 to 29, wherein the destination positioning includes a bag, and the attitude adjustment involves aligning opposite sides of the object with the inner sidewalls of the opening of the bag.

31. The method according to any one of claims 19 to 30, wherein the destination positioning includes a slot, and the attitude adjustment involves aligning opposite sides of the object with the inner sidewalls of the slot.

32. The method according to any one of claims 19 to 31, wherein the destination positioning includes a compartment, and the attitude adjustment involves aligning opposite sides of the object with the sidewalls of the compartment.

33. An object handling system for handling an object, the object handling system comprising: an end effector of a programmable motion device, the end effector for gripping the object; at least one in-hand attitude sensing system for assisting in determining an estimated attitude of the object held by the end effector; a control system for determining estimated joint positions of a plurality of joints of the programmable motion device associated with the estimated attitude of the object, and for associating the estimated attitude with the estimated joint positions to provide placement attitude information; and a destination positioning for placing the object in a destination attitude based on the placement attitude information.

34. The physical handling system according to claim 33, wherein the joint positions are determined when the end effector is positioned in an in-hand attitude positioning.

35. The object handling system according to claim 33, wherein the joint positions are estimated joint positions determined by interpolation.

36. The object handling system according to claim 33, wherein the estimated attitude of the object is performed while the end effector is moving.

37. The object handling system according to any one of claims 33 to 35, wherein the control system further determines a tipping risk factor.

38. The object handling system according to any one of claims 33 to 36, wherein the object handling system positions the object using the end effector such that the center of mass of the object is outside the contact area where the object contacts the destination positioning.

39. The object handling system according to any one of claims 33 to 38, wherein the object handling system positions the object at least about 15 degrees from the vertical direction using the end effector before releasing the object to the destination positioning.

40. The object handling system according to any one of claims 33 to 39, wherein the object handling system further determines that any one of the largest face, smallest face, or other face of the object faces upward when placed at the destination positioning.

41. The object handling system according to claim 33, wherein the placement attitude information includes information related to re-gripping the object.

42. The object handling system according to claim 40, wherein re-gripping the object involves re-gripping the object on a face different from the initial face on which the object was initially gripped.

43. The object handling system according to any one of claims 33 to 42, wherein placing the object on a receiving surface involves tipping the object over on the receiving surface.

44. The object handling system according to any one of claims 33 to 43, wherein the destination positioning includes a bag, and the attitude adjustment involves aligning opposite sides of the object with inner side walls of an opening of the bag.

45. The object handling system according to any one of claims 33 to 44, wherein the destination positioning includes a groove, and the attitude adjustment involves aligning opposite sides of the object with inner side walls of the groove.

46. The object handling system according to any one of claims 33 to 45, wherein the destination positioning includes a compartment, and the attitude adjustment involves aligning opposite sides of the object with side walls of the compartment.

Images