Object processing system and method using end effector

Abstract

An object processing system includes: an input area for receiving a plurality of objects to be processed; an output area comprising a plurality of destination containers for receiving any of the plurality of objects; a programmable motion device proximate to the input region and the output region, the programmable motion device comprising an end effector for gripping a selected object of the plurality of objects; and a perception system for detecting an unexpected occurrence of any of the plurality of objects that is not related to the end effector of the programmable motion device.

Description

[0001] priority

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 358,302 filed on July 5, 2022, the disclosure of which is hereby incorporated by reference in its entirety. Background Art

[0003] The present invention relates generally to object handling systems, and in particular to object handling systems for handling various objects, such as automated storage and retrieval systems, distribution center systems, and sorting systems.

[0004] Current object handling systems generally involve the processing of large quantities of objects, where the objects are received in organized or unorganized batches and must be routed to a desired destination based on a manifest or a specific address on the object (eg, in a mail system).

[0005] For example, an automated storage and retrieval system (AS / RS) typically includes a computer-controlled system for automatically storing (placing) and retrieving objects from defined storage locations. Conventional AS / RS typically use totes (or bins), which are the smallest loading unit in the system. In these systems, the tote is brought to a person who picks individual objects from the tote. When a person picks the required number of objects from the tote, the tote is returned to the AS / RS.

[0006] For example, current distribution center sorting systems typically present an inflexible sequence of operations in which a disorganized stream of incoming objects is first segmented into a single stream of isolated objects, which are then presented one at a time to a scanner that identifies the objects. An inductive element (e.g., a conveyor, tilt tray, or manually movable bin) transports the objects to a desired destination or additional processing station, which may be a bin, tilt shelf, chute, bag, or conveyor, among others.

[0007] In a typical package sorting system, a manual or automated system retrieves packages, usually in the order they arrive, and sorts each package or object into a collection bin based on a given set of heuristics. For example, all objects of similar types, or all objects in a single customer order, or all objects shipped to the same shipping destination, etc., may go into one collection bin. A manual or automated system is required to receive the objects and move each object to its designated collection bin. If the number of input (received) objects of different types is large, a large number of collection bins may be required.

[0008] Automated handling systems may employ programmable motion devices, such as robotic systems, that grasp objects and move them to another location (e.g., from a tote to a destination container). However, during such grasping and moving, errors may occur, such as, for example, more than one object being picked, objects being picked being lower than other objects (which may then be ejected from the tote), and objects being dropped or knocked out of the end effector of the robotic system. Any of these events may result in errors in the automated handling system.

[0009] In addition to these challenges, there are situations where some objects may have object information incorrectly entered into a manifest or shipping label. For example, if a manifest at a distribution center includes incorrect object dimensions or weights (e.g., because they were manually entered incorrectly), or if a shipping shipper enters incorrect dimensions or weights on a shipping label, the processing system may reject the object as unknown. Additionally, for incorrect information on a shipping label, for example, if a shipper incorrectly enters dimensions or weights, the shipper may be undercharged due to the incorrect information.

[0010] There remains a need for a more efficient and cost-effective object handling system that handles objects of various sizes and weights into appropriate collection bins or boxes, but still efficiently handles such objects of varying sizes and weights. Summary of the invention

[0011] According to one aspect, the present invention provides an object processing system, comprising: an input area, the input area being used to receive a plurality of objects to be processed; an output area, the output area comprising a plurality of destination containers for receiving any of the plurality of objects; a programmable motion device, the programmable motion device being proximate to the input area and the output area, the programmable motion device comprising an end effector for grasping a selected object of the plurality of objects; and a perception system, the perception system being used to detect the unexpected presence of any object of the plurality of objects that is not associated with the end effector of the programmable motion device.

[0012] According to another aspect, the present invention provides an object handling system comprising: an input area, the input area being used to receive a plurality of objects to be processed, the input area including a weight sensing conveyor portion, and the plurality of objects being provided in at least one input container; an output area, the output area including a plurality of destination containers for receiving any of the plurality of objects; a programmable motion device, the programmable motion device being close to the input area and the output area, the programmable motion device including an end effector for grasping any of the plurality of objects; and a perception system, the perception system being used to detect whether any of the plurality of objects on the weight sensing conveyor portion is not within the at least one input container on the weight sensing conveyor portion.

[0013] According to another aspect, the present invention provides an object handling system, comprising: an input area, the input area being used to receive a plurality of objects to be processed; an output area, the output area comprising a plurality of destination containers for receiving any of the plurality of objects; a programmable motion device, the programmable motion device being close to the input area and the output area, the programmable motion device comprising an end effector for grasping any of the plurality of objects; and a perception system, comprising at least one camera system and a plurality of scanning systems for detecting any identification marks on any of the plurality of objects falling toward a portion of the ground of the object handling system, and for detecting the number of any of the plurality of objects falling toward the portion of the ground of the object handling system.

[0014] According to another aspect, the present invention provides a method for handling objects, comprising: providing a plurality of objects in a container on a weight sensing conveyor portion; grabbing a selected object from the plurality of objects using a programmable motion device to move to a destination container; and monitoring whether any object from the plurality of objects other than the selected object has fallen or shifted using a sensing system and the weight sensing conveyor portion. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The following description may be further understood with reference to the accompanying drawings, in which:

[0016] Figure 1 An illustrative diagrammatic view of an object handling system according to an aspect of the present invention is shown;

[0017] Figure 2A and Figure 2B an illustrative diagrammatic side view showing the transfer of an object from one container to another container in accordance with an aspect of the present invention;

[0018] Figure 3 Shows Figure 1An illustrative diagrammatic view of an object handling system of FIG. 1 , wherein an object is positioned in a pose-in-hand position;

[0019] Figure 4 Shows Figure 1 An illustrative diagrammatic view of an object handling system of Figure 1 Weight sensing of containers on conveyor belt systems;

[0020] FIG. 5A to FIG. 5G An illustrative diagrammatic view showing processing steps in a processing system according to an aspect of the present invention;

[0021] Fig. 6A and Figure 6B Shows Figure 1 An illustrative diagrammatic lower side view of an object handling system having Figure 1 The processing station of the system, see ( Fig. 6A )and( Figure 6B );

[0022] Figure 7 An illustrative diagrammatic side view showing the transfer of objects from one container to another container wherein multiple picks occur in accordance with an aspect of the present invention;

[0023] Figure 8 a schematic diagrammatic side view showing the transfer of objects from one container to another container in accordance with one aspect of the present invention, wherein the objects have been lifted, resulting in the discharge of other objects;

[0024] Fig.9A and 9B Shows Figure 1 An illustrative diagrammatic plan view of a processing station of a system showing object movement operations ( Fig.9A ) and shows that the object has fallen onto the weight sensing conveyor system ( Fig. 9B );

[0025] FIG. 10A to FIG. 10D Shows Fig.9A and Fig. 9B An illustrative diagrammatic side view of a processing station of , showing a side view of an object having been dropped ( Fig. 10A ), showing multiple pick objects returned to the processing bin ( Fig. 10B ), showing the end effector returning to pick up a dropped object ( Fig. 10C ), and shows the end effector grabbing and moving a dropped object ( Fig. 10D );

[0026] Fig.11A and Fig. 11B Shows Figure 1An illustrative diagrammatic view of a placement portion of a processing station showing a drop on a placement conveyor belt portion ( Fig.11A ), and shows the end effector grabbing a dropped object ( Fig. 11B );

[0027] Fig.12 Shows Figure 1 an illustrative diagrammatic view of a lower portion of a processing station of a system of showing a receiving box; and

[0028] Fig.13 Shows Fig.12 An illustrative diagrammatic view of a receiving box of FIG. 1 , wherein an object has been dropped into the receiving box.

[0029] The drawings are shown for illustrative purposes only. DETAILED DESCRIPTION

[0030] The present invention provides an efficient and economical object handling system that can be used, for example, to provide any of the following: shipping orders for various objects, object groupings for shipping to various locations, and local specific object groupings for collection and shipping to large locations with local specific areas, such as merchandise areas in retail stores. Each system can be designed to meet key performance indicators (KPIs) while meeting industrial and system safety standards.

[0031] According to one aspect, the system provides an object handling system that maintains knowledge of the objects as they are handled, including the number of objects picked, whether any objects were dropped or displaced, which objects were dropped or displaced, and how the objects were dropped or displaced. Figure 1 An object handling system 10 is shown that includes an input conveyor system 12, an object handling station 11, and two output conveyor systems 14, 16. Located between the conveyor systems 12, 14, 16 is a programmable motion device 18, such as an articulated arm robotic system with an end effector (e.g., a vacuum end effector including a vacuum cup). The input conveyor system 12 includes a weight sensing belt conveyor section 40, and the output conveyor systems 14, 16 also each include a weight sensing belt conveyor section 42, 44, respectively.

[0032] The system 10 also includes a plurality of upper sensing units 20, 22, 24, 26 and a floor receiving box 28. Input objects arrive in an input container 30 on the input conveyor 13 of the input conveyor system 12 and are provided by the programmable motion device 18 to a destination container 32 on the output conveyor 15 of the output conveyor system 14 or a destination container 34 on the output conveyor 17 of the output conveyor system 16. The operation of the system is provided by one or more computer processing systems 100, including the conveyor systems 12, 14, 16 and all sensing systems (including the sensing units 20, 22, 24, 26 and the weight sensing belt conveyor sections 40, 42, 44) and the programmable motion devices. According to various aspects, any of the roller conveyors, belt conveyors and other conveyor systems (e.g., moving plates) can have weight sensing capabilities by mounting on load cells or force torque sensors according to various aspects of the present invention.

[0033] The goal of the system is to accurately and reliably move objects from an input container 30 to either of the destination containers 32, 34 using, for example, an end effector 46 having a vacuum cup 48. As discussed in more detail herein, the system employs a set of powerful perception processes that use weight sensing, imaging, and scanning to maintain knowledge of the location of all objects at all times. Figure 2A and Figure 2B , when the object 50 is input from the container 30 ( Figure 2A ) is transferred to the output container 32, the initial weight (W Ai ) plus the initial weight of the output container 32 before transfer (W Bi ) should be equal to the weight of the input container 30 after transfer (W Ap )( Figure 2B ) plus the weight of the output container 32 before transfer (W Bp ). When the object 50 is transferred to the container 32, this utilizes the principle of conservation of mass.

[0034] According to one aspect of the invention, the system may take an initial weight measurement immediately before an event (pick or place) and then wait for a buffer time before taking a post-event weight measurement. The buffer time may be, for example, 1 second, 1.5 seconds, 2 seconds, 2.5 seconds, 3 seconds, or 5 seconds to allow any force applied by the end effector to the bin during pick or place to not change the post-event weight measurement.

[0035] Figure 3An object 52 is shown in an in-hand pose position of the object handling station 11, en route to being moved to a container 32 on the weight sensing belt conveyor section 42. All transfers may include moving the end effector to an in-hand pose position (with or without a stop), and an in-hand pose camera 55 may be aimed at any object held by the end effector in the in-hand pose position. Figure 4 An object 54 is shown being moved to a container 34 on a weight sensing belt conveyor section 44 at an object handling station 11. Each time an object is moved from a source location (location A), the system verifies that the correct object is grasped and lifted into the hand pose position (e.g., as shown in FIG. Figure 1 The in-hand pose position is the position (typically near the input conveyor system) where the pose (position and orientation) of the object on the gripper is determined (e.g., by multiple perception systems). The in-hand pose position can also be selected so that the upper camera has an unobstructed view of the weight sensing belt conveyor (not obstructed by the programmable motion device), as described below with reference to Fig.9A and Fig. 9B Detailed description. The system then moves the object to the destination location (position B) and confirms receipt of the object at the destination location. As described in detail below, objects dropped on any of the weight sensing belt conveyor sections 40, 42, 44 or roller conveyors of the conveyor system 12, 14, 16 (typically mounted on load cells or force torque sensors) are detected, and objects dropped in the floor receiving bin 28 are also detected, also as described in detail below. For example, an object may be dropped or displaced by any of a drop, multiple pick, or grab and lift operation that causes other objects to be lifted out of the input container 30 along with the selected object.

[0036] Although steady-state mass readings on a picking conveyor are typically taken at some time X before the expected collision of the robot with the picking container, the system is not truly in steady state at the expected time X due to imperfect motion approximations or because the objects in the tote may still be dynamically moving based on previous actions. Therefore, the following technique can be used: after time X, based on the readings thereafter, new readings at time X+T can be considered if such new readings are more accurate for the steady-state system mass before the collision. Specifically, one such method considers all readings between time X and the time when the robot is sensed to have collided with the object it is picking, and uses the minimum reading in that time span. This is most useful in examples where a rapid retry occurs. A rapid retry is when the robot attempts to pick object A, does not grab it, and then quickly retries to pick object B. The time span between attempts A and B is typically very fast, so the weigh scale may not have stopped (and may have a positive mass spike) at time X before attempt B due to the robot's interference with the container in attempt A. Therefore, minimizing the reading of the timed callback before picking B until picking B occurs solves this problem of finding a stable reading before picking B.

[0037] According to another aspect, the system of the present invention may employ continuous moving object detection. In order to minimize the time it takes to detect that an unexpected number of objects have been picked, continuous detection may be performed after the robot has picked an object and while the robot is moving to periodically check whether an unexpected number of objects have been picked. This has the advantage that an unexpected number of picked objects may be detected as quickly as possible. If this condition is detected sooner, the robot may be told to stop sooner, which increases the speed of the system. In addition, if Figure 8 As shown, an undesired pick may involve the robot extracting an object from the container in an unstable manner, such as objects 64 and 68 may be lifted and risk falling out of the container. Therefore, detecting such a desired pick as quickly as possible means minimizing the risk of such objects falling outside the container.

[0038] In order to detect undesired picks while the robot is moving (and while objects within the pick container may change position as a result of the pick), careful algorithmic techniques are derived that balance detecting undesired picks as quickly as possible while not introducing false positives. A false positive is defined as when the system believes an undesired pick has occurred when in fact a valid pick has occurred. Due to the dynamics of real-world systems, when the robot picks one or more objects, the objects may change position and move in a non-uniform manner, thus presenting a risk of false positives.

[0039] This continuous detection can take many forms when the system is in a dynamic state, but some non-limiting examples are provided herein. A clearing time can be used so that when an invalid mass difference is detected, such mass difference must remain above a provided threshold for a specified clearing time. This technique helps to reduce the possibility of misjudgment of the number of objects picked due to mass spikes (sudden changes) caused by objects tipping over, hitting container walls, etc. when the robot performs picking and withdrawal. In addition, a confirmation method can be used, that is, if the recorded mass difference is considered invalid and the difference remains within the stability limit for a specified stabilization time, then the system can be considered to have reached a stable state, and a determination can be made immediately.

[0040] Specifically, FIG. 5A to FIG. 5G Detailed steps of a process for moving an object from bin A to bin B (e.g., from bin 30 to bin 32 or from bin 30 to bin 34) are shown. The process begins by determining the weight sensing belt conveyor section A (e.g., section 40) (step 1000). Fig.9A and Fig. 9B As further shown, the rollers in the weight sensing belt conveyor section 40 may be mounted on load cells or force torque sensors and may be covered by elastic bands. In addition, the rollers of the conveyor systems 12, 14, 16 outside the belt conveyor sections 40, 42, 44 may also be mounted on load cells or force torque sensors for weight sensing. The weight of the entire section (e.g., 40) is determined (step 1002), which includes the input box (e.g., 30) and all the contents therein. In the case where each roller has weight sensing capabilities, each weight sensing system for each roller is cleared to adjust the weight of the associated roller. Any portion of the bin on such a roller will be subtracted from the post-picking weight measurement to confirm the object picking. The system then grabs and lifts the selected object from box A (step 1004), and then determines the weight of the conveyor section A again (step 1006). The system has information about each object, including the expected weight of the selected object. The system has previously recorded the weight of the conveyor and box before picking. At steps 1002 and 1006, the system determines the weight pick delta, i.e., the difference in weight before and after the pick, which represents the weight that has (potentially) been removed from bin A. If the weight pick delta is within a tolerance (e.g., 3% or 5%) of the expected weight of the selected object, the system proceeds to step 1010. If not, the process moves to Figure 5E , as indicated and discussed below.

[0041] If the weight pick Δ for grabbing and lifting the selected object is within tolerance, the system determines whether the camera system has detected any new objects not associated with the end effector (step 1010). In particular, the upper camera system can run continuous background subtraction to identify different consecutive image pairs. The interval between the capture of consecutive images can be, for example, 0.5 seconds, 1 second, 3 seconds, or 5 seconds. The images can be captured continuously or at discrete times, and background subtraction is performed on each consecutive image pair to deduct any changes associated with the movement of the end effector. Refer to the following Fig.9A and Fig. 9B The object detection analysis is discussed in more detail. The upper camera system may include multiple cameras at the upper perception units 20, 22, 24, 26. If the camera system does detect a new object that is not associated with the end effector, the process will move to Figure 5G , as indicated and discussed below.

[0042] If the system does not detect any new objects not associated with the end effector, the process moves to Figure 5B , and the system reviews all recent scans of the lower scanning unit 60 in the floor receiving box 28 (step 1012). The scanning unit 60 can detect any identification marks on the dropped object, thereby identifying each object that falls into the floor receiving box 28. The system then reviews images from each of the multiple lower camera units 62 that are pointed at the floor receiving box 28 (step 1014). The camera unit 62 is pointed toward the interior of the floor receiving box 28, thereby identifying or confirming the identity of each object located in the floor receiving box 28.

[0043] The system then confirms (using the upper camera system and / or sensors in the end effector) whether the object is still held by the gripper (step 1016). If not, the process moves to Figure 5C , as indicated and discussed below. The system then moves the object (along with the end effector) to a hand pose position (e.g., Figure 1 ) (step 1018), and the weight of weight sensing belt conveyor section B (e.g., section 42 or 44) is determined (step 1020). The object is then placed in box B (e.g., box 32 or box 34) (step 1022), and reference is made to Figure 5C, the system then again determines the weight of the sensing belt conveyor section B (step 1024). Once again, the weight of the entire section (e.g., 42, 44) is determined (step 1026), which includes the output bin (e.g., 32, 34) and all the contents therein. Again, the system has information about each object, including the expected weight of the selected object. At steps 1020 and 1024, the system determines a weight placement Δ, which represents the weight that has (likely) been placed in bin B. If the weight placement Δ is within a tolerance (e.g., 3%, 5%, or 10%) of the expected weight of the selected object, the system proceeds to step 1028. If not, the process moves to Fig. 5F , as indicated and discussed below.

[0044] If the weight placement Δ of the selected object is within the tolerance, the system determines whether the camera system has detected any new objects that are not associated with the end effector (step 1028). Similarly, the upper camera system can run continuous background subtraction to identify different consecutive image pairs. The capture interval of consecutive images can be, for example, 0.5 seconds, 1 second, 3 seconds, or 5 seconds, and the images can be captured continuously or at discrete times, and background subtraction is performed on each consecutive image pair to deduct any changes associated with the movement of the end effector. The upper camera system includes multiple cameras at the upper perception units 20, 22, 24, and 26. If the camera system does detect a new object that is not associated with the end effector, the process will move to Figure 5G , as indicated and discussed below.

[0045] If the system does not detect any new objects not associated with the end effector (step 1028), the system then looks at all recent scans of the lower scanning unit 60 in the floor receiving box 28 (step 1030). The scanning unit 60 can detect any identification marks on the dropped object, thereby identifying each object that falls into the floor receiving box 28. The system then looks at images from each of the plurality of lower camera units 62 that are pointed at the floor receiving box 28 (step 1032). The camera unit 62 is directed toward the interior of the floor receiving box 28, thereby identifying or confirming the identity of each object located in the floor receiving box 28.

[0046] refer to Figure 5D, the system then records the identity of any object detected on belt conveyor section A and returned to bin A (described below) (step 1034). The system then records the identity of any object detected on belt conveyor section A and dropped into the floor pickup box (described below) (step 1036). The system then records the identity of any object detected on belt conveyor section B and returned to bin A (described below) (step 1038), and then records the identity of any object detected on belt conveyor section B and dropped into the floor pickup box (described below) (step 1040). The system then records the identity and quantity of any object received by the floor pickup box 28 (step 1042), and then ends (step 1044).

[0047] refer to Figure 5E , if the system determines that the weight picking Δ is not Figure 5A , then the system determines if any object is on the gripper (step 1046), and if so, returns the object to bin A or drops the object into the floor catch bin (step 1048). If the object is returned to bin A, further attempts to grasp and move the object may be made. If more than a limited number of previous attempts have been made (e.g., 3 or 4), the system may drop the object into the floor catch bin 28. The system then returns to Figure 5C Step 1030 in .

[0048] If the system determines that the weight placement Δ is not within tolerance (step 1026), the system determines if there are any objects on the gripper (step 1050), and if so, returns the object to bin A or drops the object into the floor catch bin (step 1052). Again, if the object is returned to bin A, further attempts to grasp and move the object may be made, and if more than a limited number of previous attempts have been made (e.g., 3 or 4), the system may drop the object into the floor catch bin 28. The system may then retrieve the last object from bin B (step 1054), and then either return the last object to bin A or drop the object into the floor catch bin (step 1056), as described above. The system then returns to Figure 5C Step 1030 in .

[0049] As described above, if the camera system has detected motion that is not associated with the motion of the end effector (step 1010 or 1028), the system uses the upper camera system (and / or the end effector sensor) to determine whether any object is being gripped by the gripper (step 1060), and if so, the system either returns the object to bin A or drops the object into a floor receiving box (step 1062). The system then determines whether any object is detected on conveyor section A instead of in bin A (step 1064). If so, the system returns one or more objects on conveyor section A to bin A, or drops the one or more objects into a floor receiving box (step 1066). Regardless of whether any object is detected on conveyor section A, the system determines whether any object is detected on conveyor section B instead of in bin B (step 1068). If so, the system returns one or more objects on conveyor section A to bin A, or drops the one or more objects into a floor receiving box (step 1070).

[0050] Fig. 6A and Figure 6B The upper sensing unit 20, 22, 24 is shown, which includes a camera 56 and a scanning unit 58 facing the object processing area. The camera can be used in part to detect the movement of the object that is not related to the movement of the end effector. For example, Fig. 6A A multiple pick is shown where two objects 60, 62 have been gripped by the vacuum cup 48 of the end effector 46. If an object (e.g., 48) is not adequately gripped, it may fall during transport (e.g., Figure 6B ). Before the object falls, the only motion detected by the camera 56 is the motion of the end effector along its trajectory. The scanning unit 54 may be used to facilitate the capture of any identifying marks while the object is being processed, or may be a 3D scanner for providing volumetric information about the volume of the object within any of the bins 30, 32, 34 during processing.

[0051] So, based on the run time example, the system may pick an object with a mass of 100g. The system may then measure the pick scale 0.4 seconds before impact and receive a reading of 1kg. The system may then successfully pick the object and move it to the hand pose position. As this motion occurs, the system periodically receives pick scale readings and fits a model to determine if more than one object has been picked. If continuous checking does not register a double pick, the system will reach the hand pose node and receive a pick scale reading of 0.895kg. The system will confirm that 105g has been removed from the pick tote, which is within the threshold to consider that we have picked a 100g object. The system will proceed to place the object and place it successfully. Then, before placing the object, the system will read a place scale reading of, say, 200g, place the object, and then the system will read another place scale reading of, say, 295g. Therefore, the system verifies that 95g has been added to the place box, which is within the tolerance for a 100g object.

[0052] Figure 7 Multiple picks are schematically shown where the end effector 46 picks two objects 60, 62 from the bin 30 to be placed into the bin 32. If one object is to be picked, the system should record multiple picks when the conveyor section 40 is weighed after the pick. Figure 6B Before any drop of the object (shown as 62), the system may first determine whether both objects are intended to be moved to the bin 32. If so, the system may move to the in-hand pose position, and if both objects are still held by the gripper, the system may move both objects to the bin 32, re-adjusting the expected weight of the objects to the combined weight of the two objects. If neither object is intended to be moved to the bin 32, the system may return both objects to the bin 30, or if the grasping attempt is not the first grasping attempt and more than a limited number of grasping attempts (e.g., 3 to 5) have been made, the system may discharge both objects into the floor receiving bin 28.

[0053] According to another run time example involving a double pick, the system may attempt to pick an object with a mass of 100g. The system may measure the pick scale 0.4 seconds before impact and receive a reading of 1kg. The system may successfully pick the object and move it to the hand pose position. As this movement occurs, the system will periodically receive pick scale readings and fit a model to determine if more than one object has been picked. The system determines that during the pull back pick, the pick scale recorded 810g. This indicates that the system has picked 2 objects. As described above, the system can interrupt the pick and return the object.

[0054] Figure 8The end effector 48 is schematically shown grasping a predetermined object 66, which cannot be lifted freely during lifting to avoid inadvertently ejecting one or more additional objects (e.g., 64, 68) from the bin 30. If either of the objects 64, 68 falls outside the bin 32 while the object 66 is being lifted, the ejected object 64, 68 should fall to the weight sensing belt conveyor section 40 or the floor receiving box 28. If the object falls to the floor receiving box 28, the camera should detect that the motion is not associated with the end effector motion described above. If the ejected object 64, 68 falls onto the weight sensing belt conveyor section 40. As described above, the upper camera system (camera 56) can detect the presence of the objects 64, 68 on the conveyor section 40. In some aspects, the system can continuously determine the weight of the conveyor section 40 during lifting. In this case, the system will confirm that more than one object (66) has been lifted from the bin 32. If the total weight of the conveyor section 40 includes the object ( 64 , 68 ), the system will engage the upper camera system to locate the object ( 64 , 68 ) on the conveyor section 40 .

[0055] According to another runtime example, the system may use scale verification to verify that the object is displaced. The system may attempt to pick an object with a mass of 100g. The system will measure the pick scale 0.4 seconds before impact and receive a reading of 1kg. The system will successfully pick the object and move it to the hand pose position. As this movement occurs, the system will periodically receive pick scale readings and fit a model to determine if more than one object has been picked. Assuming that continuous checks do not record double picks, the system will reach the hand pose node and receive a pick scale reading of 0.895kg. The system confirms that 105g has been removed from the pick tote, which is within the threshold for believing that the system has picked a 100g object. The system will then read the placement scale reading, which is 200g. The system will continue to place the object, but for some reason in this example, the object falls off the gripper. The system will read the pick scale reading and see that the reading is 895g. This indicates that the object did not end up in the pick tote. The system will read the placement scale reading and see that it is still 200g. This indicates that the object did not end up in the placement box. Therefore, the object was displaced and should be found by any of the perception units described above.

[0056] Fig.9A and Fig. 9B A top view of the system is shown, showing the upper sensing units 20, 22, 24 and the weight sensing belt conveyors 40, 42, 44. Fig.9A , Fig. 9BAs shown, when the end effector of the programmable motion device is in the in-hand pose position, the upper perception units 20, 22, 24 have views of the weight sensing belt conveyor portions 40, 42, 44, respectively, which are not obstructed by the programmable motion device. According to further aspects, the system (which always knows the location of the programmable motion device) can track when any part of the programmable motion device is not above the conveyor portions 40, 42, 44 and perform background subtraction during these times.

[0057] The perception units 20, 22, 24 may use RGB cameras and computer vision to detect whether a new object is on the conveyor sections 40, 42, 44 (e.g., Fig. 9B In particular, to suppress the detection of unexpected ambient light changes, several images before and after are taken (R 之前 , G 之前 , B 之前 , R 之后 , G 之后 , B 之后 ). 之前 and R 之后 is the value of the red channel of each image pixel, G 之前 and G 之后 is the value of the green channel of each image pixel, and B 之前 and B 之后 is the value of the blue channel for each image pixel. ΔImage is calculated using the following formula:

[0058] Δ=abs(R 之前 -R 之后 )+abs(G 之前 -G 之后 )+abs(B 之前 -B 之后 )

[0059] If Δ exceeds a threshold, the pixel is considered to have changed. Dilation is used to clean the calculated difference between the images from noise, and spots in the cleaned difference image are searched for within the area of ​​interest (e.g., conveyor sections 40, 42, 44). The area, roundness, convexity, and inertia of the spots are restricted to prevent noise detection. If one or more qualified spots are detected, it is considered that one or more objects have fallen into the area of ​​interest between before and after the event. According to another aspect, the outer surface of the belt conveyor sections 40, 42, 44 can be formed of a colored or reflective material to detect and isolate any spots that may represent one or more objects.

[0060] Fig. 10AAn object 68 is shown on the conveyor portion 40 after the end effector 46 has lifted the object 66. Once an object is detected on the conveyor portion 40, the system can respond in a number of ways. First, if the identity of the object 68 can be determined, the system can move the selected object 66 to the intended destination location of the object 66, and then the end effector can return to grab the object 68. Another possible response is to use the end effector 46 to return the object 66 to the input bin 30 (e.g., Fig. 10B ), the end effector can then position itself to grab the object 68. The conveyor section 40 can also move forward and backward to facilitate the end effector to reach the object 68. In another response, the system can eject the object 66 from the end effector 46, and then can position itself to grab the object 68 while the conveyor section 40 moves to position the object 68 closer to the end effector (as shown). Fig. 10C In either case, the end effector is then used to grab the object 68 from the conveyor portion 40 (as shown in FIG. Fig. 10D ), and either returns it to the box 30 or drops it into the floor receiving box 28. The conveyor section 40 may then move in the opposite direction to return the box 30 to the unloading position.

[0061] A dropped object may also land on the weight sensing belt conveyor portion at a target location (eg, 42, 44) and be detected by either the upper camera 56 or the scanner 58 as described above. Fig.11A An object 70 is shown on the conveyor portion 42. If another object is still on the end effector 46, the system can respond in any of the processes described above (conveying it to the bin 32, returning it to the bin 30, or dropping it into the floor bin 28). Fig. 11B As shown, the conveyor section 42 may be moved to accommodate the end effector 46 to grasp the object 70. The conveyor section 42 may then be moved in the opposite direction to return the bin 32 to the loading position.

[0062] As objects are dropped into the floor bin 28, the system obtains the identity and quantity of the objects received by the floor bin 28. In particular, the system includes a scanner 78 mounted on the robot support structure 76 and a scanner 80 on the inner wall of the floor bin 28. These scanners detect each dropped object (e.g., Fig.12 The object 72 shown in the figure) is detected by the structure 76, and the identification of each object is determined when the object falls into the receiving box 28 (through identification marks, such as bar codes, QR codes, etc.), and the number of objects that have fallen into the receiving box 28 is determined. Once each object is stationary, the lower camera detection system 82 on the structure 76 will confirm (for example, through image recognition or volume analysis) that the identified receiving object (for example, 76) is indeed present in the receiving box 28, as shown in the figure. Fig.13 As shown. Additional cameras may also be positioned in the inner wall of the receiving box 28 below the scanner 80.

[0063] Thus, according to various aspects, the present invention provides an object handling system that includes a perception system for detecting the motion of any of a plurality of objects that is not associated with the motion of an end effector of a programmable motion device; a perception system for detecting whether any of a plurality of containers on a weight sensing conveyor portion is not within at least one input container on the weight sensing conveyor portion; or a perception system that includes at least one camera system and a plurality of scanning systems for detecting any identification marks on any of a plurality of objects that fall toward a portion of a floor of the object handling system, and for detecting the number of any of a plurality of objects that fall toward the floor of the portion of the object handling system. These perception systems are provided by the scanners 56, 78, 80 and cameras 58, 82 described above in combination with one or more computer processing systems 100 that communicate with the programmable motion device 18, the conveyors 13, 15, 17, and the conveyor portions 40, 42, 44.

[0064] Those skilled in the art will appreciate that many modifications and variations can be made to the disclosed embodiments without departing from the spirit and scope of the invention.

Claims

1. An object processing system, the object processing system comprising: an input area for receiving a plurality of objects to be processed; an output area including a plurality of destination containers for receiving any of the plurality of objects; a programmable motion device proximate the input region and the output region, the programmable motion device comprising an end effector for grasping a selected object of the plurality of objects; as well as A perception system is provided for detecting the unexpected presence of any of the plurality of objects that is not associated with the end effector of the programmable motion device.

2. The object handling system of claim 1, wherein the perception system detects an unexpected presence of any of the plurality of objects within the defined region of interest.

3. An object handling system as described in claim 2, wherein the defined area of ​​interest includes at least a portion of the roller conveyor system.

4. The object processing system of claim 2, wherein the defined area of ​​interest includes at least a portion of a belt conveyor system.

5. An object handling system as claimed in any one of claims 1 to 4, wherein the input area includes a weight sensing conveyor section, on which there is an input container, the input container including the plurality of objects to be processed.

6. The object handling system of claim 5, wherein the sensing system further detects whether any of the plurality of objects on the weight-sensing conveyor portion is not within the at least one input container on the weight-sensing conveyor portion.

7. The object handling system of claim 5, wherein the weight sensing conveyor portion is a belt conveyor.

8. An object handling system as described in any one of claims 1 to 7, wherein the perception system also includes any one of a camera system and a scanning system for detecting any identification mark on any object of the plurality of objects that fall toward a portion of the ground of the object handling system.

9. An object handling system as described in any one of claims 1 to 8, wherein the perception system also includes any one of a camera system and a scanning system for detecting the number of any objects among the plurality of objects falling toward the ground of the portion of the object handling system.

10. An object processing system, the object processing system comprising: an input area for receiving a plurality of objects to be processed, the input area including an input weight sensing conveyor portion, and the plurality of objects being provided in at least one input container; an output area including a plurality of destination containers for receiving any of the plurality of objects; a programmable motion device proximate the input region and the output region, the programmable motion device including an end effector for grasping any object of the plurality of objects; as well as A sensing system for detecting whether any of the plurality of objects on the weight-sensing conveyor section is not within the at least one input container on the weight-sensing conveyor section.

11. The object handling system of claim 10, wherein the output area includes an output weight sensing conveyor portion.

12. The object handling system of claim 10, wherein the perception system further detects the unexpected presence of any of the plurality of objects that are not associated with the end effector of the programmable motion device.

13. The object processing system of claim 12, wherein the perception system detects an unexpected presence of any of the plurality of objects within the defined region of interest.

14. The object handling system of claim 13, wherein at least one of the input weight sensing conveyor section and the output weight sensing conveyor section comprises at least a portion of a roller conveyor system.

15. The object handling system of claim 13, wherein at least one of the input weight sensing conveyor section and the output weight sensing conveyor section comprises at least a portion of a belt conveyor system.

16. An object handling system as described in any one of claims 10 to 15, wherein the perception system further includes any of a camera system and a scanning system for detecting any identification mark on any object of the plurality of objects that fall toward a portion of the ground of the object handling system.

17. An object handling system as described in any one of claims 10 to 16, wherein the perception system further includes any of a camera system and a scanning system for detecting the number of any objects among the plurality of objects falling toward the ground surface of the portion of the object handling system.

18. An object processing system, the object processing system comprising: an input area for receiving a plurality of objects to be processed; an output area including a plurality of destination containers for receiving any of the plurality of objects; a programmable motion device proximate the input region and the output region, the programmable motion device including an end effector for grasping any object of the plurality of objects; as well as A perception system comprising at least one camera system and a plurality of scanning systems for detecting any identification marks on any of the plurality of objects falling toward a portion of the ground surface of the object handling system, and for detecting the number of any of the plurality of objects falling toward the portion of the ground surface of the object handling system.

19. The object handling system of claim 18, wherein the perception system further detects the unexpected presence of any of the plurality of objects that are not associated with the end effector of the programmable motion device.

20. The object processing system of claim 19, wherein the perception system detects an unexpected presence of any of the plurality of objects within the defined region of interest.

21. An object handling system as claimed in any one of claims 18 to 20, wherein the input area includes an input weight sensing conveyor portion.

22. The object handling system of claim 21, wherein the output area includes an output weight sensing conveyor portion.

23. The object handling system of claim 22, wherein at least one of the input weight sensing conveyor section and the output weight sensing conveyor section comprises at least a portion of a roller conveyor system.

24. The object handling system of claim 22, wherein at least one of the input weight sensing conveyor section and the output weight sensing conveyor section comprises at least a portion of a belt conveyor system.

25. An object handling system as described in claim 22, wherein the sensing system also detects whether any of the multiple objects on either the input weight sensing conveyor section and the output weight sensing conveyor section is not within the at least one container on the corresponding input weight sensing conveyor section or the output weight sensing conveyor section.

26. A method of processing an object, the method comprising: providing a plurality of objects in a container on a first weight sensing conveyor section; grasping a selected object of the plurality of objects using a programmable motion device for movement to a destination container; as well as A perception system is used to detect whether any of the plurality of objects other than the selected object has been dropped or displaced.

27. The method of claim 26, further comprising detecting an unexpected presence of any of the plurality of objects that is not associated with an end effector of the programmable motion device.

28. The method of any one of claims 26 to 27, wherein the monitoring comprises detecting the unexpected presence of any of the plurality of objects in a plurality of defined regions of interest comprising a belt conveyor portion.

29. The method of any one of claims 26 to 28, wherein the method further comprises detecting whether any of the plurality of objects on the first weight-sensing conveyor section is not within a container on the corresponding weight-sensing conveyor section.

30. The method of any one of claims 26 to 29, wherein the method further comprises detecting any identification marks on any of the plurality of objects falling toward a portion of the ground, and detecting the number of any of the plurality of objects falling toward the ground.

31. The method of any one of claims 26 to 30, wherein the method further comprises detecting at least one feature about the selected object as the selected object continues to move through the in-hand gesture position.

32. A method as described in any one of claims 26 to 31, wherein the method also includes detecting a first weight through the first weight sensing conveyor portion, and wherein monitoring whether any object other than the selected object in the plurality of objects has fallen or shifted includes detecting a second weight through a second weight sensing conveyor portion.

33. The method of claim 32, wherein the method further comprises determining whether any difference between weight decreases at the first weight-sensing conveyor portion is within a tolerance of any weight increase at the second weight-sensing conveyor portion.

Images