Systems and methods for learning to extrapolate optimal object transport and handling parameters
Through object classification system and transportation system, combined with machine learning technology, the picking station and processing parameters are optimized, and the efficiency and resource consumption problems of traditional systems when dealing with large amounts of handbags are solved, and efficient object processing and handling are achieved.
Patent Information
- Application Number
- CN201980070907.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-06-24
- Filing Date
- 2019-10-25
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2039-10-25
AI Technical Summary
Traditional automated storage and retrieval systems consume high efficiency and resource when processing large amounts of handbags, and the manual processing speed is limited, making it difficult to meet the needs of efficient storage and retrieval of objects.
The object classification system and transportation system are adopted, and through identification systems, data warehouses and programmable motion equipment, combined with machine learning technology, the selection and processing parameters of the pickup station are optimized, the object handling performance is improved, and the efficient transportation and processing of automated material handling systems are realized.
It improves the efficiency and economy of the object processing system, reduces manual intervention, realizes efficient classification and handling of various objects, and reduces the time and resource consumption of transportation and processing.
Smart Images

Figure CN112930547B_ABST
Abstract
Description
[0001] priority
[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 62 / 750,777 filed on October 25, 2018 and U.S. Provisional Patent Application Serial No. 62 / 865,421 filed on June 24, 2019, the disclosures of which are incorporated herein by reference in their entirety. Background Art
[0003] The present invention relates generally to object handling systems, and more particularly to object handling systems for handling various objects, such as automated storage and retrieval systems, distribution center systems, and sorting systems.
[0004] For example, an automated storage and retrieval system (AS / RS) typically includes a computer-controlled system for automatically storing (placing) and retrieving items from defined storage locations. Conventional AS / RS typically use a tote (or box), which is the system's smallest load unit. In these systems, the tote is brought to a person who picks individual items from the tote. When the person has picked the required number of items from the tote, the tote is reintroduced back into the AS / RS.
[0005] In these traditional systems, a tote is brought to a person and the person can remove items from the tote or add items to the tote. The tote is then returned to the storage location. Such systems can be used, for example, in libraries and warehouse storage facilities. AS / RS is not involved in handling the items in the tote because the person handles the objects when the tote is brought to the person. This separation of work allows any automated transport system to do what it does well (moving totes) and allows the person to do what it does well (picking items from a cluttered tote). It also means that the person can stand in one place while the transport system brings the tote to the person, which increases the speed at which the person can pick up the goods.
[0006] However, with such conventional object handling systems, there are limitations in the time and resources required to move the tote to and from each person, as well as the speed at which the person can handle the tote in this manner in applications where each person may need to handle a large number of totes. Thus, there remains a need for an object handling system that can more efficiently and economically store and retrieve objects while also facilitating the handling of a wide variety of objects. Summary of the invention
[0007] According to an embodiment, the present invention provides a system for object processing. The system includes a supply location with one or more objects to be processed, a plurality of object processing stations for processing the objects, an object classification system, and a transportation system. The object classification system includes an identification system for identifying objects at the supply location, a first data warehouse with a first object information set, and a second data warehouse with a second object information set, the second object information set including processing parameters. The first object information set is provided using the object, and the second object information set is determined by the system, and the object classification system assigns the object to a category based on the first information set and the second information set. A transportation system is provided for transporting the object from the supply location to a selected object processing station capable of processing objects in the assigned category, wherein the selected object processing station processes the object according to the determined processing parameters.
[0008] According to another embodiment, the present invention provides a method for processing an object. The method includes the following steps: presenting an object to a transport station; evaluating provided object information and determined object information associated with the object, wherein the determined object information includes processing parameters; classifying the object based on the provided object information and the determined object information; selecting a pick-up station for processing the object based on the object classification; transporting the object from the transport station to the selected pick-up station; processing the object at the pick-up station according to the processing parameters; observing the object being processed at the processing station; scoring the object handling performance based on the observed criteria; and updating the processing parameters based on the object handling performance score.
[0009] According to a further embodiment, the present invention provides a method for processing objects. The method includes the following steps: providing one or more objects to be processed at a supply location; providing an object classification system, wherein the object classification system includes: an identification system for identifying objects at the supply location, a first data repository with a first object information set, and a second data repository with a second object information set, wherein the second object information set includes feature information related to multiple objects. The method also includes the following steps: providing a first object information set using an object; providing a second object information set via a feedback learning system, wherein the second object information set includes feature information related to object features; assigning objects to categories based on the first information set and the second information set; and updating the second object information set after engaging with the object.
[0010] According to yet a further embodiment, the present invention provides a method for processing an object. The method includes the following steps: providing one or more objects to be processed at a supply location; capturing identification data of the object; querying a first data repository having a first object information set; determining that the identification data of the object is not included in the first object information set; identifying object feature data related to the object; querying a second data repository having a second object information set, the second object information set including feature information related to a plurality of objects; identifying associated object information within the second object information set, the associated object information including learned feature data that closely matches the object feature data, the learned feature data being associated with the associated object; and engaging with a programmable motion device, the object using grasping and acquisition data related to the associated object. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The following description may be further understood with reference to the accompanying drawings, in which:
[0012] Figure 1 An illustrative diagrammatic view of a framework for use in an object handling system according to an embodiment of the present invention is shown;
[0013] Figure 2 An illustrative diagrammatic view showing a process used in an object handling system according to an embodiment of the present invention;
[0014] Figure 3 An illustrative diagrammatic view of an object handling system according to an embodiment of the present invention is shown;
[0015] Figure 4 Shows Figure 3 An illustrative diagrammatic view of a portion of an object handling system is shown;
[0016] Figure 5 Shows Figure 4 An illustrative diagrammatic bottom view of a detection system and a capture system of a processing station;
[0017] Fig. 6A -6D shows an illustrative diagrammatic plan view of a container undergoing homogeneous object volume and / or density analysis prior to the analysis ( Fig. 6A ), an illustrative plan view before picking up an object ( Figure 6B ), an illustrative plan view after picking up an object ( Figure 6C ) and an illustrative diagrammatic plan view isolating the volume of the picked object from the volume of the container ( Fig.6D );
[0018] Figure 7 An illustrative diagrammatic view of a plurality of sensing units positioned around a scanning volume is shown;
[0019] Figure 8 Shows Figure 7 A diagrammatic view of a plurality of sensing units located around a scanning volume, wherein the sensing units and illumination sources are engaged in pairs;
[0020] Fig. 9 Shows Figure 7 An illustrative diagrammatic side view of a pair of sensing units and an illumination source;
[0021] Fig.10 A pair of sensing units and an illumination source are shown. Fig. 9 illustrative diagrammatic side view of;
[0022] Fig.11 An illustrative diagrammatic view of an object from the perspective of a first perceptual unit is shown;
[0023] Fig.12 Shows the second sensor unit's perspective Fig.11 illustrative diagrammatic views of objects;
[0024] Fig.13 Shows the perspective of the third sensor unit Fig.11 illustrative diagrammatic views of objects;
[0025] Fig.14 Shows the perspective of the fourth sensory unit Fig.11 illustrative diagrammatic views of objects;
[0026] Fig.15 shows an illustrative diagrammatic view of a 3D scanner and scan field for use in a system according to an embodiment of the present invention;
[0027] Fig.16 shows an illustrative diagrammatic view of a system including three 3D scanners according to an embodiment of the present invention;
[0028] Fig.17 shows an illustrative diagrammatic view of a 3D scanning system for use in accordance with an embodiment of the present invention, the 3D scanning system scanning an object and scanning a portion of an end effector grasping the object;
[0029] Fig.18 an illustrative diagrammatic side view showing a scanned object and a scanned portion of an end effector to be subtracted from the overall scan volume;
[0030] Fig.19 shows an illustrative diagrammatic view of an end effector system for use in accordance with an embodiment of the present invention, the end effector system including sensors for detecting potential errors in grasping (such as, for example, multi-picking);
[0031] Fig. 20 shows an illustrative diagrammatic view of a processing system according to a further embodiment of the present invention, the processing system including a programmable motion system suspended from an XY hanger;
[0032] Fig.21 shows an illustrative diagrammatic view of a system according to a further embodiment of the present invention, the system comprising a U-shaped conveyor belt and including a programmable motion system;
[0033] Fig. 22 Shows Fig.21 An illustrative diagrammatic view of a portion of a dispensing station shown in , wherein a carriage moves along a track and is tilted to allow objects to fall from the carriage;
[0034] Fig.23 Shows Fig.21 an illustrative diagrammatic view of a portion of a processing system;
[0035] Fig.24A – Fig.24D An illustrative diagrammatic view of a cargo box undergoing volume and / or density analysis of a foreign object prior to the analysis is shown. Fig.24A ), illustrative diagram view before picking up an object ( Fig. 24B ), an illustrative diagrammatic view after picking up an object ( Fig.24C ) and an illustrative diagrammatic view of isolating the volume of the picked object from the volume of the container ( Fig.24D );
[0036] Fig.25 shows an illustrative diagrammatic view of another weight sensing bracket for use in a system according to another embodiment of the present invention;
[0037] Fig.26 Shows Fig.25 An illustrative diagrammatic side view of a weight sensing bracket;
[0038] Fig. 27 Shown are multiple Fig.21 an illustrative diagrammatic view of a system of processing systems;
[0039] Fig.28 an illustrative diagrammatic view showing process steps of a process control system according to an embodiment of the present invention; and
[0040] Fig.29 An illustrative diagrammatic view of an object handling system according to a further embodiment of the present invention is shown.
[0041] The drawings are shown for illustrative purposes only. DETAILED DESCRIPTION
[0042] Unstructured and dynamic physical environments, such as fulfillment centers, present unique challenges to operators. There is a nearly endless supply of goods that need to be identified, picked up, and delivered. Human workers set a high bar for identifying properties and similarities between objects they encounter. In order for robots to be as fast or faster than humans, artificial intelligence (AI), especially machine learning, is needed to improve system performance in the face of operational complexity.
[0043] According to an embodiment, the present invention provides an automated material handling system, part of which is to transport objects carried in containers to stations where objects are transferred from one container to another at automated stations using one or more programmable motion devices (such as articulated arms), and the stations may further include manual stations. The containers may be containers, totes, or boxes, etc. The overall goal of the system may be to sort and transport goods, perform order fulfillment, replenish inventory, or provide any general system that requires the transfer of individual objects from one container to another.
[0044] The objects may be packages, boxes, flats, or plastic bags in a shipping center, or consumer products in an e-commerce fulfillment center, or product and warehouse packaging in a retail distribution center (DC). The containers may be containers from an automated storage and retrieval system (AS / RS), supplier cases, store supply boxes, or any other containment mechanism designed to separate items from one another. The transport of containers may take many forms, including conveyor belts or roller conveyors, chutes, mobile robots, or human workers. The pick-up station where the items are transferred may be an automated system including programmable motion equipment such as an articulated arm robotic system.
[0045] The automated material handling system uses an execution algorithm or process to determine a strategy for how to transport the cargo box to the pick-up station. One such algorithm may be a balanced load so that all pick-up stations are utilized and none are idle. However, in some cases, uniquely configured pick-up stations with different capabilities for handling different object types may be used throughout the system. For example, pick-up station A may be worse than pick-up station B in picking up item X, and therefore it is best to send item X to pick-up station B. In this case, another algorithm may be used to transport objects through the system, taking into account the pros and cons of various pick-up stations. In addition, it may be that the robot pick-up station A has pick-up modes 1 and 2, and mode 2 is better at picking up item X. According to various embodiments, the present invention provides a system that combines a set of processes for learning how to optimize both the transport to the pick-up station and the parameters related to the pick-up station operation in order to improve the performance of the overall operation. The optimization selects the transport and handling parameters according to different criteria (including throughput and accuracy) and conditional on the contents of the cargo box to improve the expected overall performance of the pick-up.
[0046] Learning is a key aspect of the system of an embodiment of the present invention, in the sense of improving over time. The system provides performance of picking based on the item, pick station, and handling parameters. Further, objects will be encountered periodically that have not been picked before. However, it is possible that new objects similar to previously picked items will have similar performance characteristics. For example, object X may be a shampoo in a twenty ounce bottle, and object Y may be a conditioner in a twenty ounce bottle. If distributed by the same company, the shape of the bottles may be the same. Therefore, the handling of the bottles may be the same or very similar. The system of an embodiment of the present invention includes the following processes: using observations of past performance of similar items to predict future performance; and learning what characteristics of the items available to the system are reliable predictors of future performance.
[0047] Each pick is a new data point. Throughout the picking process, there are many variables that can be changed to provide new data points. For example, the system can change the speed of the robot arm movement, the position at which the object is picked up, the type of gripper used, the gripping force applied, etc. The main outcome checked is whether the object was successfully moved from its source location to the desired destination.
[0048] Figure 1 The framework of the present invention is shown, which enables reliable deployment of AI-based solutions and enables the system to utilize millions of picks, which provide more power to machine learning techniques. Due to uncertainty about the best action or strategy, multiple strategies are adopted to test speed, reliability and accuracy.
[0049] Specifically, Figure 1 The stages of a cyclical iterative process for operational testing and improvement are shown. In one embodiment, the cycle is divided into six stages that constitute different stages of the machine learning and implementation process. These six stages constitute the main features of the cyclical process. One main feature 11 of the cycle utilizes millions of picks in a warehouse setting to provide real world data for the system. The next feature 12 highlights the reduction of the labeling work required to evaluate the performance of the system. The next feature 13 attempts to use the collected data to improve the performance of the system. And the last feature 14 attempts to ensure the reliability of the new strategy before reintroducing it into the commercial warehouse setting.
[0050] The top phase of the cycle begins at phase 21, which provides routine data collection within a warehouse or order fulfillment center. The system uses previously successful arrangements to process objects and collects data about these arrangements to improve the model. In this phase, safe and gradual changes are made to the arrangements, which are compounded over millions of picks. In some embodiments, the changed arrangements constitute a very small percentage of the total picks (e.g., less than one percent) to provide reliability of the system.
[0051] Stage 22 also occurs in the warehouse and uses a large number of sensors to extensively determine whether each pick was successful. The sensors also provide detailed information about various sensed attributes throughout the process. This continuous monitoring allows the system to adjust certain attributes to improve results or correct errors in general. This automated sensing saves workers from having to manually identify and mark the placement used as successful or failed. In most cases, the system is able to operate more reliably than manual operations, miss fewer failure instances, and provide more and almost instant data reporting on success or failure.
[0052] Stage 23 represents the act of getting the collected data from the warehouse. The collected data may include sensed data, as well as order data or other data to provide a complete context for each pick-up that can be broken down and analyzed. Data transmission out of the warehouse may be compressed and sub-sampled to reduce network impact.
[0053] Phase 24 occurs offline and off-site, using the recorded data (including context and outcome pairs) to learn strategies for controlling the various operational variables used. For example, in some embodiments, strategies for grasping locations and robot motions and behaviors (such as arm speed and trajectory) can be developed, or previously used strategies can be enhanced, based on patterns found in the data. This may lead to associations between SKUs that indicate potentially successful strategies for newer objects for which the system has little or no information.
[0054] Phase 25 tests and validates new policies before they are introduced into the warehouse. New policies can be tested using real systems and commonly encountered objects, or they can be tested using simulations of real-world processing. By testing and validating in isolation, operational risk is isolated from the warehouse. For example, errors made offline outside the warehouse do not affect actual order fulfillment. During this exploration phase, millions of situations are practiced or simulated, both to improve policy execution and to ensure appropriate behavior in the learned behaviors.
[0055] Stage 26 orchestrates the deployment of the new policy into the warehouse to serve as the primary operating policy during processing. Deployment is the gradual rollout of the learned policy to the sites. In some embodiments, to provide another layer of reliability in the new policy, the rollout is only provided to certain areas within the warehouse or certain warehouses (if multiple warehouses will be affected by the new policy). After the rollout (whether partial or full) is complete, the cycle returns to the top stage 10 again to continue the cycle.
[0056] According to certain embodiments, the system of the present invention provides certain information or truth (GT) information about various parameters that is determined to be absolutely correct through such feedback learning. For example, as follows, a weight sensor can be used to determine or confirm the occurrence of a multi-object pick-up. For example, if an item is identified but the weight experienced is much greater than the weight of the identified object, the weight sensor can be used to determine that more than one object has been grasped. If such a grasp is maintained (even for a short time), the system will record any one or all of the following: airflow at the end effector, airflow changes at the end effector, vacuum level at the end effector during grasping, vacuum changes at the end effector during grasping, changes in any of weight, linear acceleration, angular acceleration, or rotation. Such recorded changes (or events where changes have not occurred) are then associated with performance (successful grasping and acquisition, or unsuccessful grasping and acquisition), and further details can be provided as to whether such grasps repeated over time are generally successful and / or whether movements (acquisitions) repeated over time are generally successful. The use of such sensing can allow the system to determine the true value (GT) about a particular object, stock keeping unit (SKU), or other known set of information. For example, if an object with a known SKU is grasped and the system subsequently detects any of the following associated with a multi-pick: a change in vacuum at the end effector during the grasp, a change in any of the recorded weight, linear acceleration, angular acceleration, or rotation, the system will know that the acquired object is a multi-pick and does not need to be processed as the identified SKU or other known information may require, but rather like a multi-pick that needs to be re-processed or handled by a person (GT). The sooner the system knows that the pick violates the GT, the sooner the system can re-process or post-process the multi-pick. If the weight of the object is unknown but the object is identified by SKU, the system can record the weight load when the end effector engages the object, and then record the weight load on the end effector again after the object is placed down. Further, a weight scale (such as a weight sensing bracket) can be used. In a short period of time, the known weight of the object (the true weight) will be recorded with certainty for that SKU.
[0057] According to a further embodiment, the system provides a learning process that (a) extrapolates the performance of newly seen objects and (b) continuously updates the data that the system uses to learn to perform the extrapolation so as to continuously improve performance. The potential picking parameters are varied. Several controllable picking parameters can govern the process, such as which picking stations can pick up a given item, which actuators (vacuum cup size or gripper type) are effective for the item, what rules can be used to select the location on the item for grasping, etc. Because these processing parameters can be changed on a per-SKU basis and will determine the efficiency and speed of the picking station, and further can be determined on a per-SKU basis, these parameters must be estimated correctly. In particular, the correct values of the processing parameters depend on: the nature of the item, the weight and size of the item, the packaging of the item, the material properties of the item (such as whether it is deformable or transparent), whether the vacuum gripper is effective in holding the item, where a good grasping location on the object is, and whether the item is easily damaged.
[0058] However, in many operating situations, this can be challenging because there may be new SKUs, which means there is no known set of parameters available. While these parameters will be learned from repeated interactions with the object, this can significantly slow down handling time. In order to speed up the time it takes to learn the appropriate parameters, it can be useful to use previously recorded data based on similar SKUs. According to certain embodiments, the system of the present invention can employ 3-D modeling of the object and train the generated data, for example by attempting to correlate the 3-D model with data about known objects.
[0059] According to a further embodiment, the system of the present invention may not indicate relevant information related to SKU, but rather indicate relevant information related to packaging. In such systems, a summary or description of packaging (e.g., shape, size, weight, length, width, height, perimeter or center or mass) may be provided as an associated event, and the system may operate according to the closest associated event parameter. For example, some packaging features (such as known rectangular surface areas on common boxes, known marking shapes on common boxes, known contours of common objects, and known features of certain objects such as clamshell boxes) may be learned and become the subject of learned knowledge. Similarly, the execution of such actions whether the execution is successful (successful grasping and collection) or unsuccessful (not successful grasping and collection) is also recorded.
[0060] According to various embodiments, the present invention provides a process for an automated material handling system that delivers boxes to a picking station, and the present invention provides the following. The system can predict object-specific parameters for new objects based on previously seen objects. For new objects that are similar to previously handled objects, the process predicts what are expected to be good shipping and handling parameters. In this example, the object is easily identified as being very similar to objects with which the system has extensive experience. From a barcode or SKU number or product catalog or instruction text or from appearance or other features, the system can identify the object and index information in a database, which may include processing parameters, or will at least include information from which processing parameters can be determined with high confidence.
[0061] Further, for completely unknown objects, the system can explore the parameter space. For new objects that are not sufficiently similar to any previously handled objects, the system can propose multiple candidate transport and handling parameters with the goal of finding good transport and handling parameters. When an unfamiliar object is first introduced, handling parameters must be determined.
[0062] The system can also update a predictive model of object-specific handling performance from observed item handling performance. As experience is gained with the object, the process refines the shipping and handling parameters based on the object. As experience is gained, the predictive model is refined.
[0063] The system can further update the predictive model of similarity between objects from the observed object handling performance. As experience is gained with all available items, the parameters that affect the schemes and processes used to classify and / or cluster objects are improved. Further, the system can identify and correct for persistent differences in actual performance versus predicted performance. When supplemented by the manufacturer, some objects have different weights, packaging, or other features that affect the handling execution of the object. Old shipping and handling parameters that previously produced good performance may not be suitable for the changed objects. When actual performance repeatedly exceeds the range of predicted performance, the system prefers exploration of parameter space.
[0064] According to various embodiments, the object information set includes feature information about the features of the object, and feature information associated with any one of the object SKU, text description, product category, object manufacturer, object quality, object material, object shape, packaging details, object image, and object color can be developed. For example, the system can collect such feature information related to the object when the object is captured and collected. The system can engage cameras and scanners to collect any text description, product category, object manufacturer, object quality, object material, object shape, packaging details, object image, and object color information, and associate the information with the SKU. Such collected information is recorded in association (e.g., listed, networked, commonly connected or grouped, etc.), and is therefore associated with each other and with the SKU. The system can further participate in machine learning in pixel processing of images from the camera by learning how to best segment object images at the pixel level (e.g., in a hierarchical representation).
[0065] According to further embodiments, feature information associated with any of the following may be developed: sensed information, object weight, object material, object shape, object opacity, object size, object volume, actuator to be used, grasping position, object motion restrictions, object fragility, and object handling restrictions. Similarly, the system may collect such feature information about the object as it is grasped and collected. The system may engage sensors to collect any sensed information, object weight, object material, object shape, object opacity, object size, object volume, actuator to be used, grasping position, object motion restrictions, object fragility, and object handling restrictions, and associate the information with the SKU. Such collected information is recorded in association (e.g., tabulated, networked, commonly connected or grouped, etc.), and is therefore associated with each other and with the SKU. In any of these embodiments, the association number of any of the above factors may be used to identify the grasping position and the action to be taken to provide active grasping and collection, and such further actions may be recorded for further machine learning.
[0066] In such embodiments of the invention, the machine learning system may provide feedback such that when an item (identified as described above) is grasped and successfully transported, such event is recorded as a successful execution, and all parameters regarding the entire action are recorded as being correlated. When unknown objects are later acquired, the system may, for example, determine the number of commonalities of such text descriptions, product categories, object manufacturers, object mass, object materials, object shapes, packaging details, object images and object color information or sensed information, object weight, object material, object shape, object opacity, object size, object volume, actuators to be used, grasping locations, object movement limitations, object fragility, and object handling limitations.
[0067] According to a further embodiment, an association of commonalities in achieving performance of a task (successfully moving an object) can be generated so that a set of parameters is recorded as having achieved positive performance. Through such a reward feedback process, the system can learn from actions that did not achieve positive performance and learn from actions that achieved positive performance.
[0068] For example, Figure 2 The overall process according to an embodiment of the present invention is shown. The process starts (step 100) and accesses a prediction model (step 102). The prediction model is previously compiled by any combination of manually input data, or is obtained through experience in handling a specific object. When an object is encountered, the system first determines whether the object is new (step 104). If the object is new, the system can explore the parameter space of the object, for example, by trying different grasping positions and / or different routes. If the object is new but similar in size, shape, or SKU, etc. as described above, the system can process the object according to the parameters of similar objects for which the system has processing parameters (step 108). If the object is not new, the system processes the object according to the processing parameters of the object (also step 108). As discussed further below, the system records a large amount of data about both the object and its characteristics during processing, and records this information as feedback (step 110). The system can then determine whether there is a persistent difference with the processing parameters and the recorded feedback (step 112). If not, the feedback information is added to the knowledge base of the processing parameters (step 114), and the prediction model is updated (step 116). If persistent differences with respect to process parameters are identified, such information is added to the predictive model (step 116), for example by marking such information as uncertain model parameters or replacing model parameters with recorded feedback information.
[0069] An object handling system according to an embodiment can provide a cargo to robot product sorting system, wherein a cargo box is brought to a pick and place robot, which must pick up an object from one cargo box and place it in another cargo box. For example, each source cargo box can provide a set of homogeneous retail products, and the source cargo box is filled by an earlier process (dumping the supplier box into the cargo box). These cargo boxes can be directly transferred to the picking station, stored in a short-term buffer storage, or placed in an automatic storage and retrieval system (AS / RS). As part of the dumping process, the product SKU is recorded and the cargo box ID number is also recorded, and the two numbers are associated in the facility management software database. The cargo boxes filled with items are sent back to the store to replenish the store's inventory of items. When processing store orders, a set of source cargo boxes and destination cargo boxes are transported to the picking station. The picking station can be automated by a robot, or it can be a manual picking station where manual handling is performed.
[0070] For example, there may be various robot types for handling different categories of items, or the robots may be able to change the actuator from one size of vacuum cup to another, for example. The robot picks up the object from the source bin, transports it to the destination bin, and drops or places the object therein. After the operation, the sensory information collected throughout the interaction (including depth sensors, RGB cameras, flow sensors, pressure sensors, wrist status, etc.) is used to determine whether the transfer operation was successful and, if necessary, the type of failure, and this information is stored in the system database.
[0071] In addition to sensor information, operators can also provide manual feedback based on observations of the handling, for example, the operator selects a list of items that might include "the robot could move faster / slower" or "it picked up more than one item when it should have picked up one item" or "is the grip point okay?" or "the item is damaged" or "the item was dropped too high."
[0072] For example, Figure 3 An object handling system 200 is shown that includes one or more storage areas 202, a tote pick and place system 204, and a processing station 206. The input totes 208 and output boxes 210 in the storage area 202 may be separate or intermingled, and the pick and place system 204 includes a gripper device 212 on an XY hanger that can access each input tote and output box and bring them to the processing station 206. The operation of the system and the processing and learning discussed herein may be provided by one or more processing systems 201. Further reference Figure 4The processing station 206 includes a programmable motion device 214 (such as an articulated arm) having an end effector 216 and a sensing unit 218. Further reference Figure 5 The sensing unit 218 includes a light 74 and one or more sensing units 76 (e.g., a scanner or a camera) for detecting any identification marks (e.g., barcodes, QR codes, RFID, tags, etc.) on the objects in the cargo box 46 and for guiding the programmable motion device 56 to move the objects with the end effector 216 ( Figure 4 ) grabs the object in the cargo box. Through this system, the selected object is collected from the cargo box and transported via a carrier 58, and then transported to a desired box 64 via a carrier 62.
[0073] The sensing unit 218 also includes scanning and receiving units 80, 82, and an edge detection unit 84 for capturing various features of the selected objects throughout the container. Fig. 6A A view is shown from the perspective of a capture system, which may include a set of similar or dissimilar objects 90, 92, 94, 96, 98, depending on the embodiment. Figure 6B The scan shown in Figure 6C The volume difference between the scans shown in is the estimated volume V of the removed object 94 94 The volume is compared with recorded data or recorded object data concerning the item identified by the identification mark provided by the detection system 72.
[0074] In particular, if Figure 6B As shown, before the object is removed from the container 208, the contents of the container are scanned volumetrically and as Figure 6C As shown, after the object 94 is removed from the cargo box 208, the contents of the cargo box are scanned volumetrically. The volumetric scanning can be performed using scanning and receiving units 80, 82 together with the processing system 42, and the scanning and receiving units 80, 82 send and receive signals (e.g., infrared signals). According to an embodiment, from Figure 6B The volume captured in Figure 6C The volume captured in , and the difference is evaluated as the estimated volume (V 94 According to other embodiments, the system (knowing that it is about to collect object 94) may collect object 94 while it is still in the container (e.g. Figure 6B 94). This can replace the above ( Figure 6B and Figure 6C According to a further embodiment, the scanning and receiving units 80, 82 can also be used to determine the density D of the object from the known mass and volume of the object. 94The volumetric data may be obtained, for example, using any of a Light Detection and Ranging (LIDAR) scanner, a pulsed time-of-flight camera, a continuous wave time-of-flight camera, a structured light camera, or a passive stereo camera.
[0075] According to further embodiments, the system may additionally employ an edge detection sensor 84 that (again in conjunction with the processing system 201) is used to detect the edges of any object in the container, for example using data regarding any of intensity, shadow detection, or echo detection, and may be used, for example, to determine any of size, shape, and / or contour, such as Fig.6D shown.
[0076] In accordance with further embodiments, the system can estimate the volume of an object while the object is held by the end effector. While for some types of object handling systems (e.g., package sorting for shipping / mailing) volume may not be helpful (e.g., when handling deformable plastic bags), in other systems (such as store replenishment or e-commerce applications) volume scanning will be very valuable. In particular, the system can estimate the volume of a picked item while the picked item is held by the gripper and compare the estimated volume to the known volume. One approach is to estimate the volume of one or more items while the gripper holds an object (or multiple objects) and then compare the observed volume to the a priori known volume of the item. Reference Figure 7 and Figure 8 , one or more sensing units 152, 154, 156, 158 (e.g., cameras or 3D scanners) are placed around the scanning volume. Further reference Fig. 9 and Fig.10 Opposite each sensing unit is an illumination source 162, 164, 166, 168 and a diffuser screen 172, 174, 176, 178 in front of each illumination source.
[0077] like Fig.10 As shown in , sensory data related to object 94 backlit by an illumination source (e.g., 168) and a diffuser (e.g., 178) will be captured by each sensory unit (e.g., 158). Fig.11 shows a view of object 94 from camera 158, Fig.12 shows a view of an object from camera 154, Fig.13 shows a view of an object from camera 152, while Fig.14 A view of an object is shown from camera 156. According to various embodiments, three perception units spaced one hundred twenty degrees apart may be used, and according to further embodiments, fewer perception units (e.g., one or two) may be used, and the object may be rotated between data acquisition captures.
[0078] The scan volume can be the volume above the area from which the item is picked; or the scan volume can be strategically placed between the pick-up location and the drop location to minimize travel time. Within the scan volume, the system takes a snapshot of the volume of items held by the gripper. As described above, the volume can be estimated in a variety of ways depending on the sensor type.
[0079] For example, if the sensors are cameras, two or more cameras can be placed in a ring around the volume and oriented slightly upward toward a backlit screen (as described above), which may be in the shape of a cross-section of a donut, with the held volume held between all the cameras and a bright white screen. The bright screen backlight illuminates the held object or objects, leaving the interior volume black. Each sensing unit and associated illumination source can be activated in sequence so that no two illumination sources are on at the same time. This allows for easy segmentation of the held volume in the image.
[0080] The illumination can be provided as a specific wavelength that is not present in the room, or the illumination can be modulated and the detector can demodulate the received sensory data so that only the illumination from the associated source is provided. The black areas, once projected back into space, become viewing cones, and objects are known to be located in the solid viewing cones. Each camera generates a separate viewing cone with the property that the volume of an item is a subset of all viewing cones. The intersection of all viewing cones produces an upper limit on the volume of (multiple) objects. The addition of cameras improves the accuracy of the volume estimate. The gripper can be visible within the camera and since its position is known, its volume can be subtracted from the viewing cone or volume estimate.
[0081] According to other embodiments, a 3D scanner that obtains a 3D image of the scanned volume can be used, and then a volume estimate can be obtained in a similar manner by fusing the point clouds received from each sensor together, without the need to segment the image from the background using backlighting. Each 3D scanner returns a 3D image that returns the depth for each pixel in the image, and any of a light detection and ranging (LIDAR) scanner, a pulsed time-of-flight camera, a continuous wave time-of-flight camera, a structured light camera, or a passive stereo camera, etc. can be used again.
[0082] For example, Fig.15 3D scanner 182 is shown projecting a grid 188 onto the field of view. 3D scanner 182 may be used in conjunction with one, two, or three other 3D scanners (two more are shown at 184, 186) to Fig.16180 is used in the system 180 shown. The 3D scanners are pointed at a common volume in which the object 94 is positioned when attached to the end effector 216. Likewise, for three such 3D scanners, the scanners can be positioned one hundred and twenty degrees apart (90 degrees apart if four are used; facing each other if only two are used). Fig.17 and Fig.18 Each 3D scanner (e.g., 182) captures 3D data related to the object. Likewise, the volume of the end effector can be removed from the captured data.
[0083] According to further embodiments, the system can detect multiple pick-ups by automatically sensing leaks from flow or pressure data. Fig.19 , the system can use an end effector 170 that includes a sensor 160 (such as a flow sensor or a pressure sensor). The system can calculate statistics based on observations of flow and pressure while holding an item and compare them to statistics collected when the same item was previously collected. For example, the system may detect a much larger flow (or increase in vacuum pressure) than expected for object 94, which may be due to the grasping of two objects (92, 94), thereby causing a large amount of air to be drawn into the end effector 216'. Therefore, the system can calculate variance and other statistics based on the time series data of flow and / or pressure while holding the object for comparison with statistics when the same or similar items were previously gripped. The system can also compare the values obtained, and if the difference is above a certain threshold, the system can treat the instance as an instance of picking up multiple items. As described above, the system can use a linear classifier, a support vector machine, or other machine learning based classifier to distinguish between single or multiple picks using flow or pressure data.
[0084] At several stages during the flow of goods throughout the material handling system, there are several processing parameters associated with these actions, including (a) whether to use a person or a robot, (b) which type of robot, (c) which type of actuator, (d) the method and parameters of the algorithm for selecting the grasping point, (e) how aggressively or quickly the object can be moved with the robot arm, (f) how the outcome of the movement is determined, (g) sensor thresholds for various decisions (such as whether to grasp an item based on a vacuum seal), and (h) methods for planning the path used to transfer the object.
[0085] The facility handles tens of thousands of SKUs. The problem of generating processing parameters for each SKU is a challenge. In addition, there may be tens of thousands of new SKUs introduced each year, so an efficient process for associating processing parameters with new SKUs is critical. When a new SKU is introduced, a worker scans the object in the supplier box and transmits the SKU to the facility software system (e.g., running on one or more computer processing systems 201). The worker or system sensors may generate additional information, such as images, weights, and dimensions that are stored in the system software database. The system also stores information provided by the supplier, which may include names, text descriptions, weights, dimensions, images, and other information.
[0086] The classifier software then generates an initial estimate of the parameters, which may take several different probabilities p i The form of is the probability that an item is in the i-th category. New SKUs may also introduce entirely new categories of items. When an object is handled for the first time, initial handling parameters are adopted and the results of the operation are saved. During the handling process, the system obtains additional sensory information that is saved in the system database. The results of the operation are also saved. If the initial grasp fails, the system may adjust the handling parameters to prevent future grasps at the same location. If the object is dropped during the picking process, the system may adjust the movement to be less aggressive. If damage is recorded, parameters can be adjusted for more careful drops, different picking stations, and different sizes of vacuum cups.
[0087] Each time an object is handled, a similar process is followed, allowing the system to build a record of all experiences in a database. As this record grows, the class of the item can be determined with greater confidence, and other information including handling parameters can be refined. This embodiment includes many variations. Within a single distribution center, the experience of all robots is shared by using a common database. By linking distribution centers together, the experience of all robots can be shared across the enterprise. For example, Fig. 20 An object handling system 300 is shown, which includes one or more storage areas 302, a tote pick and place system 304, and a plurality of processing stations 306. The input totes 308 and output boxes 310 in the storage areas 302 can be separate or intermingled, and the pick and place system 304 includes a gripper device 312 on an XY hanger that can access each input tote and output box and bring them to the processing station 306, as described above with reference to Figure 3 Each processing station 306 is similar to processing station 206 described above and includes a programmable motion device (such as an articulated arm) having an end effector and a perception unit under the control of one or more computer processing systems 301 as described above.
[0088] Another variation is that when the system is idle or lightly loaded, the system can request objects from the AS / RS, send the objects to various picking stations, collect more experience with the items, and improve the associated data. Another variation is that in addition to or instead of robotic picking, the handling parameters also include which person does the manual picking. Person X may be better at handling small SKUs, while person Y may be more capable of lifting heavy objects. The system learns from the person's throughput data on an object-specific basis which objects the person handles best and worst, and uses this information to deliver goods to manual stations, or manual stations and robotic stations, and optimize handling activities throughout the warehouse.
[0089] According to a further embodiment, the present invention provides a system in a shipping center. The items may vary greatly, including boxes of different sizes and weights, plastic bags, padded mailers, envelopes, shipping tubes, etc. All objects are sorted by destination, loaded into plastic bags, and then these plastic bags are loaded into trailers. Some packages arrive at the facility from local suppliers (including local e-commerce order fulfillment centers). Other packages come from other shipping centers. All of these packages are loaded onto a conveyor belt that carries the packages to a robotic picking station, as discussed in more detail below. The robot is equipped with various sensors, including cameras and optical barcode scanners. In some cases, the barcode is visible, allowing the system to retrieve the identity of the object and associated information. In some other cases, the camera allows the system to estimate the category of the item and other information related to the picking operation.
[0090] For example, Fig.21 A processing system 400 is shown that includes a programmable motion system 402. The programmable motion system 402 includes an articulated arm 404 and an end effector 406. The system 400 can pick up objects from a tote 410 disposed on a conveyor belt 412 and place the picked up objects into a reciprocating carriage 414 that travels along a track 416 between rows of boxes 420 and as shown in FIG. Fig. 22 As further shown, the system 400 is adapted to drop the objects into one of the output bins 420. The completed bins may be pushed onto an output conveyor 422 which directs the completed bins to a collection output conveyor 424.
[0091] System 402 includes a sensing unit 408, and further reference is made to Fig.23 and Figure 24A-Figure 24D , the sensing unit 408 is similar to the above reference Figure 5 The perception unit 218 discussed, and comprises a light and perception unit, and a scanning and receiving unit, and an edge detection unit. Each perception unit 408 can therefore capture identification marks, and provide a volumetric 3D scan as described above.
[0092] The perception unit 408 includes the scanning and receiving units 80, 82, and the edge detection unit 84 (described above) for capturing various features of a selected object or the entire container. Fig.24A 1 shows a view from the perspective of a volume detection system, which may include a set of similar or dissimilar objects 190, 192, 194, 196, 198, depending on the embodiment. Fig. 24B As shown, before the object is removed from the cargo box 46, the contents of the cargo box are scanned volumetrically, and as shown in FIG. Fig.24C As shown, after the object 194 is removed from the cargo box 46, a volumetric scan of the contents of the cargo box is performed.
[0093] Volume scanning may again be accomplished using scanning and receiving units 80, 82 that send and receive signals (e.g., infrared signals) and processing system 42. Fig. 24B The volume captured in Fig.24C , and the difference is evaluated as the volume of the removed object 194 (V 194 According to other embodiments, the system that knows that object 194 is about to be collected can collect the object 194 while the object 194 is still in the container (e.g. Fig. 24B ) captures volume data about the object 194. This can replace the volume subtraction method ( Fig. 24B and Fig.24C According to a further embodiment, the scanning and receiving units 80, 82 can also be used to determine the density D of the object by knowing the mass and volume of the object. 194 .
[0094] According to a further embodiment and with reference to Fig.24D The system may additionally employ an edge detection sensor 84 which (again in conjunction with the processing system 42) is used to detect the edges of any object in the cargo box using, for example, any of image intensity data, shadow detection, or echo detection, and may, for example, be used to determine any of a size, shape and / or contour.
[0095] Fig.25 and 26 A cradle 430 is shown having a body 432 including a tall rear wall 434 against which an object can be redirected into the generally V-shaped body of the cradle in accordance with an embodiment of the present invention. In particular, the programmable motion system 402 can drop an object into the cradle having the body 432 such that the articulated arm 404 is located on a side of the cradle 430 opposite to a side of the tall rear wall 434.
[0096] The carriage 430 is mounted on a frame 446 via load sensors 442, 444, and its movement along the track and tilting is controlled by an actuation system 448. In particular, a pair of load sensors 442, 444 are coupled to the frame 446, and the carriage body 432 is mounted to (and suspended from) the load sensors. By positioning the load sensors in the carriage body close to the object(s), a highly reliable weight measurement can be obtained. Once the beam interruption transmitter and receiver pair 452, 454 detects an object, the system according to an embodiment will average the weight values of the two load sensors (W1, W2) together, double the result, and subtract the weight of the body 432. In this way, the weight of the object can also be estimated. According to other embodiments, the load sensors themselves can record changes indicating that the carriage has received or discharged an object.
[0097] Communications and electronic control are provided by electronic processing and communications system 450. The cradle body 432 is adapted to rotate about axis 447 (to empty its contents) and is attached to the top 449 of the load cells 442, 444 above the axis of rotation 447. Fig.21 ) provides communications and electronic control. Similarly, load sensors 442, 444 can be used to determine the weight of the contents of the tray. Fig. 27 As shown, the system can be scaled so that multiple programmable motion systems 402 can process objects into multiple carriers 414 and output bins 420.
[0098] Therefore, reference Fig.28 , the processing control system 500 may include an object classification system 510, which includes: an identification system 520 for identifying objects placed at a supply location; a first data repository 530 having a first set of information, the first set of information including information sent with the object; and a second data repository 540, the second data repository 540 including a second set of information about the object determined by the system. The identification system 520 uses a variety of different methods to identify the object. For example, the object identification system 520 can read an identification mark on the object, such as a barcode, an object SKU, a distinguishing label, an RFID tag, or any other method of identifying an object. If the identification system 520 does not recognize the object, the object is transferred to a human operator or a different system for processing. Once the object is identified, the object information from the first data repository 530 and the second data repository 540 is evaluated by the object classification system 510.
[0099] The object classification system 510 uses a first set of information and a second set of information to assign a particular category to an object. The first set of information includes information sent with the object. The first set of information may include identification tags used by the identification system 510, as well as a textual description of the object, the product category of the object, manufacturer information, information about size, shape, mass, volume, materials used, packaging or item information (including color, pattern, image or other source indication), structural information (such as deformability, fragility, required orientation ("this side is up"), etc.). The system can determine a second set of information related to the object, such as using sensors to measure characteristics (such as mass, volume, size, shape, stiffness, color, image, etc.). It can also associate processing variables with the object, such as actuators that can be used with the object, motion or inertial properties of the object that may affect processing, acceptable drop heights or orientations when releasing the object, or other time, temperature or handling details.
[0100] The category information assigned by the classification system 510 in turn defines processing parameters, which include the type and manner of processing that occurs at the object processing station 402. The type and manner of processing may include a specific grasping planning method that may be targeted at, for example, placement on an object associated with the object's center of mass, or information about flat areas on an object that can be grasped by a vacuum cup, or a grasping method that indicates the need for exploratory trial and error to find an appropriate grasping position. The processing parameters may also indicate various motion planning methods that balance the speed at which the actuator moves the object with the ability to keep the object in the actuator's grasp. The processing parameters may also include information related to releasing the object from the actuator, including release height, release orientation, release location, etc. These processing parameters may indicate specific processing methods or hardware requirements available at some but not all object processing stations 402. For example, particularly heavy objects may require the attachment of a larger vacuum cup that can only be found at a single object processing station. Alternatively, particularly fragile objects may be assigned to specific people at a manual object processing station, and objects that require another skill set may go to other people at a manual processing station. Manual Pick-up Station 700 Fig.29 7 and includes a manual pick station 710 where personnel can process items from a source container 720 into a destination container 730. Any processing instructions can be displayed on an interactive display 715, and any observations displayed as to the success or failure of processing parameters can be recorded by the interactive display 715 and sent to a process control center 740.
[0101] The process control system 500 also includes a transport system 550 that transports objects from their supply location to the selected item processing station 402. The transport system 550 uses the classification information provided by the object classification system 510 and the system information 556 to determine the best destination and route to the destination. The transport system 550 optimizes the transport using the object processing station information, such as the object queue, station hardware, station distance, and estimated travel time to the station. The transport information 555 is then sent to the system to be used.
[0102] When the object is sent to the object processing station and is processed according to the determined processing parameters, the evaluation system 560 uses the evaluation information 575 sent from the system input 570 to evaluate the selected category and shipping information. The system input 570 may include input from sensors (such as depth sensors, 3d cameras, RGB cameras, flow sensors, pressure sensors, position or force sensors attached to actuators, weight sensors, microphones or other feedback sensors) to evaluate whether the interaction between the system and the object is successful. The system input can also be manually input into the system by an operator in the case of visual routing and object processing station interaction. For example, if the person observing sees that the object looks very firmly held by the actuator and is likely to be processed faster, the person can send the feedback through the system. A successful interaction can be an interaction that occurs as expected and expected within the expected or expected tolerance and threshold and does not have perceived damage. An unsuccessful interaction can be an interaction that fails to grasp during initial engagement, fails to grasp during transfer, damage to the object, an accidental collision of the actuator or object with another object or structure, or exceeds the expected or expected tolerance and threshold.
[0103] Successful or failed interactions can be quantified by a score indicating whether the interaction was successful. The score can be based on sensory input, threshold proximity, target processing time versus actual processing time, delivery time, object damage, or other measurable information. The score can then be used by the object classification system 510 to reclassify the object.
[0104] The object classification system can change processing parameters regardless of whether the interaction succeeded or failed. For example, after a successful interaction, in an attempt to improve speed or efficiency by trying different actuators, the object classification system can assign a different category to the object to allow for faster or riskier motion plans, less accurate grasping plans, more accurate and less accurate release characteristics. After a failed interaction, a new category can be assigned that uses different processing parameters that are more likely to succeed.
[0105] As the system continues to process objects, the categories assigned to the objects will have a higher probability of being successfully assigned. As new objects enter the system, the object classification system 510 can compare the information of the new object with the information of previously processed objects with similar characteristics to select a category for the new object that has a higher probability of successfully interacting at the selected object processing station.
[0106] For example, a large number of objects may come from a specific retailer that specializes in smartphone cases and sends them all in packages, which presents a special challenge for the robotic picking station. These packages are uniquely colored and easily identifiable, allowing the system to identify these items and handle them in a way that ensures a secure grip and careful movement.
[0107] As another example, there is a great deal of standardization in cardboard envelopes and padded mailers. Objects can be classified based on camera images and their handling attributes inferred based on experience gained across the categories. In this embodiment, the system is able to learn from experience based on recognizable packaging appearance patterns, even in the absence of SKUs. The system is also able to leverage the experience it has gained with individual objects. As an object moves through the network, it may pass through several shipping centers and be sorted many times. When a barcode is captured, and the item is not displayed in the system for the first time, previous handling experience can now be used to select the correct processing parameters.
[0108] According to various embodiments, different use cases may be provided as follows. As an example involving the introduction of a new SKU at a distribution center, assume that a cereal distributor introduces a variation on the packaging of a popular breakfast cereal. The only change may be the appearance of a local sports hero on the box. When a case with several of these items is introduced into the system, the system uses the barcode to obtain the SKU and associated product information. Based on this information, the system correctly classifies the product as a cereal box with the same handling properties as multiple cereal boxes that the system is already familiar with. The system, based on its experience with similar products and its ability to correctly identify this similarity, can immediately process the new SKU.
[0109] As mentioned above, the classification process occurs autonomously without human involvement. Alternatively, this classification process can be performed when a human worker submits a new SKU to the system, giving the human the opportunity to confirm or correct the classification.
[0110] In another example involving reclassifying items based on data obtained during processing, as described above, a new SKU may have been introduced and classified as a cereal box. However, due to the new packaging of the SKU, the size has changed and must be picked using the small cup size. Since the system has never seen this SKU before, it will use the processing parameters of the previous SKU. After the SKU is unsuccessfully picked the first time, the system will update its estimated parameters, causing it to deviate from the original parameters.
[0111] At the same time, this change in the assortment structure can be applied to other objects that have been introduced into the system but have not yet been delivered to a pickup station with a camera. So when a distributor introduces new packaging for an entire product line, the system may not have to look at every item but can make reasonable inferences just by examining the first such item.
[0112] In another example involving learning to avoid placing suction cups on seams, there are several SKUs (several types of shampoo bottles corresponding to dry hair, oily hair, fresh scent, herbal, etc.). They are all packaged in the same type of bottle. When handling these shampoos, the suction gripper is sometimes placed at the seam between the cap and the bottle, which is effectively invisible to the system's perception system. When these grasps fail, the system can determine this based on images of the destination container, pressure sensing in the actuator, force / torque sensors in the wrist, or a combination of these sensors. The system records the location of the failed grasp and adjusts downward in the hope of a good grasp nearby. The system can therefore learn better grasping behavior for all items in a category, and can do this very quickly because it can aggregate experience across the entire category.
[0113] In another example involving learning better movements, several SKUs may be packaged similarly to bottles packed in cardboard boxes. Typically, the boxes arrive at a pick station placed side by side in a bin and must be grasped by a suction cup placed on top of the box, which is a hinged lid. The robot obtains a secure grasp and lifts the box away from the other boxes in the bin. The robot then begins motion to transport the box to the destination bin. Based on the trajectory (which includes acceleration, tilt of the actuator, and orientation of the box), the lid sometimes becomes open. The system detects this event from image and force / torque data. The system adjusts the movement to a more conservative one by keeping the actuator at a greater right angle so that the center of mass remains under the suction cup.
[0114] In another example involving learning to avoid damage, suppose a new SKU is introduced, but sometimes, perhaps one in five times, the product is damaged when it is placed in the destination bin. The vibration of the weight hanging from the bellows cup may exacerbate the severity of the drop. In this case, the result may not be observable with the system's sensors, but it may be observed by a human worker and reported as an input to the system. The system modifies the drop motion, moving the actuator closer to the destination bin and timing the release so that the vibration of the object is at its lowest point.
[0115] In a further example involving learning to use a new actuator, certain categories of items were generally handled well, but some items might be heavier than average. The system learned to carry them by moving slowly without dropping them, but this resulted in reduced productivity. The engineering team introduced an additional robotic picking station with a larger suction cup. As items were delivered to the new station, a variety of items and movements were employed, and based on the observed results, the system learned which items were suitable for the new actuator and which movements were acceptable. In the end, the heavier items could now be carried quickly, and productivity was restored. Additionally, when a new actuator is introduced, its characteristics can be manually entered to speed up the process of the system integrating the new actuator into the operation.
[0116] In a further example involving learning to recognize outcomes, various sensors can be used to detect success or failure and the type of failure. These sensors include pressure sensors, force / torque sensors, and image sensors. There are software programs for processing these sensor signals to recognize outcomes. These software programs can be improved by using information obtained from one sensor to improve another sensor, or by using results determined by sensor fusion. As a simple example, suppose a failure is signaled by a sensed pressure change, but after several examples, it becomes apparent from other data that the pressure sensor signals the failure a little later. Signal processing, one or a set of features, or thresholds can be adjusted to advance the pressure sensor's detection of failure.
[0117] In a further example involving learning to detect anomalies, trash may sometimes enter the vacuum gripper and rest against the screen, reducing the efficiency of the actuator. The anomaly detection software may record when the sensing signal deviates from a given area. When the source of the anomaly is identified, such as when a human operator records that trash has entered the vacuum gripper, the anomaly detection software can adjust itself to correctly classify such anomalies in the future and alert the human operator in a more timely manner.
[0118] In a further example involving learning to interpret a force / torque sensor, the force / torque sensor gets signals from a variety of sources. A vacuum supply hose moves around, and its movement varies with robot movement and hose pressure. An item moves around, and sometimes contents are lost and moved around within the item. There are time constants and other characteristics associated with these signals, which provides an opportunity to put the signal processing in a place to tune it for a specific item and suppress noise.
[0119] Those skilled in the art will appreciate that various modifications and variations can be made to the embodiments disclosed above without departing from the spirit and scope of the present invention.
Claims
1. A system for object processing, the system comprising: a supply location having one or more objects to be processed; a plurality of object processing stations for processing objects; An object classification system, wherein the object classification system comprises: an identification system for identifying objects at the supply location; a first data repository having a first object information set including identification tags and predetermined characteristics associated with the identified object; and a second data repository having a second set of object information including sensed characteristics of the identified object determined using one or more sensors at the supply location; wherein, when the identification tag of the identified object is a new identification tag, the object classification system assigns the identified object to a category associated with a previously processed object, wherein the previously processed object has similar characteristics to the first object information set and the second object information set; and A transport system transports the identified objects from the supply location to a selected object processing station capable of processing objects in the assigned category, wherein the selected object processing station processes the objects according to processing parameters of the assigned category.
2. The system of claim 1, further comprising an evaluation system for evaluating the interaction of the selected object processing station with the identified object according to the processing parameters of the assigned category, wherein The evaluation system determines whether the interaction is a success or a failure.
3. The system of claim 2, wherein: The interactions evaluated by the evaluation system include any of: object grasping, object movement, and object placement using an end effector of a programmable motion device.
4. The system of claim 3, wherein: The evaluation system receives sensor input to evaluate the interaction of the selected object processing station with the identified object according to the processing parameters of the assigned category.
5. The system of claim 3, wherein: The evaluation system receives manual input to evaluate the interaction of the selected object processing station with the identified object according to the processing parameters of the assigned category.
6. The system of claim 2, wherein: The processing parameters are updated in response to the determination of the evaluation system.
7. The system of claim 1, wherein: The first object information set includes any of the following: object stock keeping unit (SKU), text description, product category, object manufacturer, object quality, object material, object shape, packaging details, object image, and object color.
8. The system of claim 1, wherein: The second object information set includes any of the following: sensed information, actuators to be used, grasping positions, object movement limitations, object fragility and object handling limitations, wherein the sensed information includes any of the following: object weight, object material, object shape, object opacity, object size and object volume.
9. A method for processing an object, the method comprising: Identify objects at the delivery station; determining characteristics of the identified object using one or more sensors at the delivery station; When the identified object is associated with the new identification tag, assigning the identified object to an object class associated with a previously processed object having characteristics similar to the determined characteristics of the identified object; selecting a pick station for processing the identified object, wherein the selected pick station is capable of processing the assigned class of objects; transporting the identified object from the delivery station to a selected pickup station; processing the identified object at the selected pick station according to processing parameters defined in the assigned object category; generating an object handling performance score for the identified object being processed at the selected pick station; as well as The processing parameters are updated based on the object handling performance score.
10. The method of claim 9, wherein: The object handling performance score indicates a successful interaction or a failed interaction between the identified object and the selected pick station.
11. The method of claim 10, wherein: The object handling performance score is generated based on sensory information provided by one or more of: a depth sensor, a scanner, a camera, a flow sensor, a position sensor, a force sensor, a scale, an acceleration sensor, and a vibration sensor.
12. The method of claim 10, wherein: The object handling performance score is generated based on sensory information provided by a human observer.
13. The method of claim 9, wherein: The object handling performance score indicator evaluates initial grasp failure, object transfer failure, object placement failure, and object damage.
14. The method of claim 9, wherein: The selected pick station includes an articulated arm having an end effector for engaging and moving the identified object.
15. The method of claim 9, wherein: The identified object is further provided with any of: object stock keeping unit (SKU), text description, product category, object manufacturer, object mass, object material, object shape, packaging details, object image, and object color.
16. The method of claim 9, wherein the determined characteristics of the identified object include any of: object weight, object material, object shape, object opacity, object size, and object volume.
17. A method for processing an object, the method comprising: providing an object to be processed at a supply location; capturing identification data of the object; querying a first data repository having a first set of object information; determining that the identification data of the object is not included in the first object information set; identifying object characteristic data related to the object; querying a second data repository having a second set of object information, the second set of object information including feature information related to a plurality of objects; identifying associated object information within the second object information set, the associated object information comprising learned feature data that closely matches the object feature data, the learned feature data being associated with a related object; assigning the object to an object class associated with the related object; and The object is engaged with a programmable motion device using grasping and acquisition data defined in the assigned object class.
18. The method of claim 17, wherein: The method further includes transporting the objects from the supply location to a selected object handling station based on the assigned category, wherein the selected object handling station includes the programmable motion device for engaging objects in the assigned category.
19. The method of claim 18, wherein the method further comprises evaluating whether the programmable motion device of the selected object handling station successfully engaged the object based on the grasping and acquisition data of the assigned category.
20. The method of claim 18, wherein: The feature information includes any of the following: object stock keeping unit (SKU), text description, product category, object manufacturer, object quality, object material, object shape, packaging details, object image, and object color.
21. The method of claim 18, wherein: The characteristic information includes any of the following: sensed information, actuators to be used, grasping positions, object movement limitations, object fragility and object handling limitations, wherein the sensed information includes any of the following: object weight, object material, object shape, object opacity, object size and object volume.
22. The method of claim 19, wherein evaluating whether the programmable motion device of the selected object handling station successfully engages the object comprises evaluating object grasping, object transfer, and object placement.
23. The method of claim 19, wherein the selected object handling station comprises a robotic picker.
24. The method of claim 19, further comprising: When the programmable motion device of the selected object handling station fails to engage the object, the grasping and acquisition data is altered.
25. The method of claim 19, wherein: Whether the programmable motion device of the selected object handling station successfully engages the object is evaluated based on input from one or more sensors at the selected object handling station.
26. The method of claim 25, wherein the one or more sensors include one or more of the following: a depth sensor, a scanner, a camera, a flow sensor, a position sensor, a force sensor, a scale, an acceleration sensor, and a vibration sensor.
