System and method for robotic package handling

Abstract

One embodiment relates to a robotic package handling system comprising: a. a robotic arm having a distal portion and a proximal base portion; b. an end effector coupled to the distal portion of the robotic arm; c. a place structure positioned in geometric proximity to the distal portion of the robotic arm; d. a pick structure contacting one or more packages and positioned in geometric proximity to the distal portion of the robotic arm; e. a first imaging device positioned and oriented to capture image information regarding the pick structure and the one or more packages; and f. a first computing system operably coupled to the robotic arm and the first imaging device and configured to receive image information from the first imaging device and command movement of the robotic arm based at least in part on the image information.

Description

Technical Field

[0001] (Cross-Reference to Related Application) This application claims the priority and benefit of U.S. Provisional Patent Application No. 63 / 193,775, filed on May 27, 2021, the entire disclosure of which is incorporated herein by reference.

[0002] The present invention relates generally to the field of robotics, and more specifically to new and useful systems and methods for planning and adapting object manipulation by robotic systems. More specifically, the present invention relates to robotic systems and methods for managing and handling packages. Background Art

[0003] Many industries have adopted forms of automation. In particular, robotic systems and robotic arms are increasingly being used to assist with automating manual tasks. However, the cost and complexity involved in integrating robotic automation limits such adoption.

[0004] Due to the wide variety of potential uses, many robotic systems are either highly customized and specifically designed for particular implementations, or are very general robotic systems. Highly specialized solutions can only be used in limited applications. General systems will often require a large amount of integration work to program and configure for a particular implementation, which can be expensive and time-consuming.

[0005] Further complicating matters is the fact that many potential uses of robotic systems involve changing circumstances. Traditionally, robots have been designed and configured for various uses in industrial and manufacturing settings. These robotic systems generally perform highly repetitive and clearly defined tasks. However, the rise of e-commerce has created a greater demand for forms of automation that must deal with highly changing or unknown conditions. Many robotic systems are incapable of handling a wide variety of objects and / or constantly changing objects, which can make such robotic systems an inadequate solution for product handling tasks resulting from e-commerce. Thus, in the field of robotics, there is a need to create new and useful systems and methods for planning and adapting object manipulation by robotic systems. This invention provides such new and useful systems and methods. [Overview of the project] [Means for solving the problem]

[0006] One embodiment relates to a system and method for planning and adapting object manipulation by a robotic system function, using dynamic planning for the control of the robotic system when interacting with an object. The system and method preferably employs robotic gripping planning combined with dynamic tool selection. The system and method can also be dynamically configured in relation to the environment, which can enable the workstation implementation of the system and method to be quickly integrated and configured in a new environment.

[0007] The system and method are preferably operated to optimize or otherwise improve the performance throughput of automated object-related tasks. Alternatively, the problem can be shaped to increase or maximize the success of grasping and object manipulation tasks per unit task. For example, the system and method could improve the ability of a robotic system to pick an object from a first area (e.g., a container), move the object to a new location or orientation, and place the object within a second area.

[0008] In certain modifications, the system and method employ the use of selectable and / or interchangeable end effectors, leveraging dynamic instrument selection for improved manipulation of objects. In such multi-instrument modifications, the system and method may use a variety of different end effector heads, whose design and capabilities may vary. The system and method may use a multi-instrument with a set of selectively activated end effectors, as shown in Figures 7 and 8. In another modification, the system and method may use a changeable end effector head, in which the end effector used can be changed between a set of compatible end effectors.

[0009] Through overall optimization, the system and method can enable unique robotic capabilities. The system and method can rapidly plan with respect to various end-effector elements and dynamically determine when to change the end-effector head and / or how to use selected fixtures. Preferably, the system and method consider the time cost of switching fixtures and the predicted success probability for different actions of the robotic system.

[0010] The unique robotic capabilities enabled by this system and method can be used to allow a wide variety of instruments and more specialized instruments to be used as end effectors. These capabilities also make the robotic system more adaptable and easier to configure in environments or scenarios where a wide variety of objects are encountered, and / or when it is beneficial to use automated instrument selection. In e-commerce applications, there may be many situations where a robotic system is used for collecting different types of objects, such as when sorting returned products or when centrally managing products by a worker or robot for order processing.

[0011] The system and method are preferably used to grasp an object and perform at least one object manipulation task. One preferred sequence of the object manipulation task may include grasping the object (e.g., picking the object), moving the object to a new position, and placing the object, and the robotic system of the system and method operates as a pick-and-place system. The system and method may, alternatively, be applied to a variety of other object handling tasks such as object inspection, object sorting, performing manufacturing tasks, and / or other suitable tasks. Although the system and method are described primarily in the context of pick-and-place applications, variations of the system and method described herein may also be applied to any suitable use cases and applications.

[0012] This system and method can be particularly useful in scenarios where a variety of objects need to be processed, and / or when little or no prior information is available for at least a subset of the objects to be processed.

[0013] This system and method can be used in a variety of use cases and scenarios. A robotic pick-and-place implementation of this system and method can be used in warehouses, product handling facilities, and / or other environments. For example, a warehouse used to fulfill shipping orders may need to process and handle a wide variety of products. Robotic systems handling these products would generally lack available 3D CAD or models, have little to no prior image data, and have no explicit information on barcode locations. This system and method can address such challenges, enabling the handling of a wide variety of products.

[0014] This system and method may offer several potential benefits. This system and method are not necessarily limited to offering such benefits at all times, but are presented merely as illustrative examples of how this system and method may be used. The enumeration of benefits is not intended to be exhaustive, and other benefits may exist, either in addition or alternatively.

[0015] One potential benefit is that the system and method can be used to improve the throughput of robotic systems. Gripping planning and dynamic instrument selection can be used to automatically modify the operation of different end-effectors for the selection of specific objects and to leverage their capabilities. Preferably, the system and method can increase or further maximize the success rate of object manipulation (e.g., successfully gripping an object) while reducing or even minimizing the time spent changing instruments.

[0016] Another potential benefit is that the system and method can interact with objects more reliably. Predictive modeling can be used in interacting with objects more successfully. The added flexibility for changing fixtures can further be used to improve the chances of success when performing object tasks such as picking and installing objects.

[0017] As a related potential benefit, the system and method can work more efficiently with products in an automated manner. Generally, a robotic system will perform some processing of an object as an intermediate step to some other action performed using the object being grasped. For example, a product may be grasped, its barcode scanned, and then the product placed in the appropriate box or container based on its barcode identifier. By more reliably selecting objects, the system and method can reduce the number of failed attempts. This can result in faster handling of objects, thereby improving the efficiency of processing objects.

[0018] Another potential benefit is that the system and method can be adapted to various environments. In some modifications, the system and method can be easily and efficiently configured for use in new environments using the configuration approach described herein. Another aspect is that multi-device modifications can also enable the handling of a wide variety of objects. The system and method may not rely on collecting large amounts of data or information prior to being set up for a particular site. Thus, a pick-and-place robot system using the system and method can be moved into a new warehouse and begin handling products in that warehouse without a lengthy configuration process. Furthermore, the system and method can handle a wide variety of types of objects. The system and method is preferably very suitable for situations where there is a wide variety of species and types of products that need to be handled. However, examples of the system and method may be equally useful even when the diversity of objects is low.

[0019] As a related benefit, the system and method may also learn and improve their performance over time as they learn and adapt to the objects encountered for a particular facility.

[0020] Another embodiment is a robotic package handling system comprising: a robotic arm having a distal portion and a proximal base portion; an end effector coupled to the distal portion of the robotic arm; a place structure positioned geometrically close to the distal portion of the robotic arm; a pick structure positioned geometrically close to the distal portion of the robotic arm and in contact with one or more packages; a first imaging device positioned and oriented to capture image information relating to the pick structure and one or more packages; and a first computing system operably coupled to the robotic arm and the first imaging device, configured to receive image information from the first imaging device and to command the movement of the robotic arm at least partially based on the image information. The present invention relates to a robotic package handling system comprising a stem configured to operate a robotic arm and an end effector to grasp a targeted package from one or more packages from a pick structure, release the targeted package, and place it on a place structure, the end effector comprising a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing system, the first suction cup assembly comprising defining a first inner capture chamber, wherein grasping the targeted package includes, when the vacuum load is controllably activated adjacent to the targeted package, pulling a portion of the targeted package into the first inner capture chamber and using it to at least partially encapsulate it.

[0021] Another embodiment is a robotic package handling system comprising: a robotic arm having a distal portion and a proximal base portion; an end effector coupled to the distal portion of the robotic arm; a place structure positioned geometrically close to the distal portion of the robotic arm; a pick structure positioned geometrically close to the distal portion of the robotic arm and in contact with one or more packages; a first imaging device positioned and oriented to capture image information relating to the pick structure and one or more packages; and a first computing system operably coupled to the robotic arm and the first imaging device, configured to receive image information from the first imaging device and to command the movement of the robotic arm at least partially based on the image information, wherein the first computing system is configured to detect a targeted package from the pick structure to one or more packages. The present invention relates to a robotic package handling system comprising a first inner chamber, a first outer seal edge, and a first vacuum permeable distal wall member, wherein the robotic arm and end effector are configured to operate to grasp a targeted package, release the targeted package, and place it on a place structure, the end effector comprising a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing device, the first suction cup assembly collectively, having an outer seal edge that can be removably coupled to at least one surface of the targeted package in response to grasping the targeted package using the controllably activated vacuum load, while a vacuum permeable distal wall member is configured to prevent excessive protrusion of the surface of the targeted package into the inner chamber of the suction cup assembly.

[0022] Another embodiment is a robotic package handling system comprising: a robotic arm having a distal portion and a proximal base portion; an end effector coupled to the distal portion of the robotic arm; a place structure positioned geometrically close to the distal portion of the robotic arm; a pick structure positioned geometrically close to the distal portion of the robotic arm and in contact with one or more packages; a first imaging device positioned and oriented to capture image information relating to the pick structure and one or more packages; and a first computing system operably coupled to the robotic arm and the first imaging device, configured to receive image information from the first imaging device and to command the movement of the robotic arm at least partially based on the image information, wherein the first computing system performs a grip of a targeted package from the pick structure, releases the targeted package, and places it on the place structure. The robotic package handling system is configured to operate an arm and an end effector, the end effector comprising a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing system, the first suction cup assembly being configured to perform a grip, which includes engaging with a targeted package when the vacuum load is controllably activated adjacent to the targeted package, prior to performing a grip, the computing device being configured to analyze a plurality of candidate grips and, at least in part, select an execution grip to be performed to remove the targeted package from the pick structure, based on the use of a neural network operated by the computing device, the neural network being trained using views unfolded from synthetic data, including rendered images of three-dimensional models of one or more synthetic packages, such as those contained by the synthetic pick structure.

[0023] Another embodiment is a robotic package handling system comprising: a robotic arm having a distal portion and a proximal base portion; an end effector coupled to the distal portion of the robotic arm; a place structure positioned geometrically close to the distal portion of the robotic arm; a pick structure positioned geometrically close to the distal portion of the robotic arm and in contact with one or more packages; a first imaging device positioned and oriented to capture image information relating to the pick structure and one or more packages; and a first computing system operably coupled to the robotic arm and the first imaging device, configured to receive image information from the first imaging device and to command the movement of the robotic arm at least partially based on the image information, wherein the first computing system performs a grip of a targeted package from the pick structure, releases the targeted package, and places it on the place structure, thereby controlling the robotic arm and the The system comprises a first suction cup assembly configured to operate an end effector, the end effector comprising a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing system, the first suction cup assembly configured to perform a grip, which includes engaging with the targeted package when the vacuum load is controllably activated adjacent to the targeted package, the system further comprising a second imaging device operably coupled to the first computing system, which is positioned and oriented to capture an image of one or more of the targeted packages after a grip has been performed using the end effector, to fit a 3D rectangular prism around the targeted package, to estimate the outer dimensional boundary of the targeted package by estimating the LWH of the rectangular prism, and to use the fitted 3D rectangular prism to estimate the position and orientation of the targeted package relative to the end effector, the first computing system, the targeted package,This invention relates to a robotic package handling system configured to operate a robotic arm and end effector so as to position and orient the package structure in a specific location relative to the package structure.

[0024] Another embodiment relates to a system comprising a robotic pick-and-place machine, a system having an actuation system configured to facilitate selection and switching between a plurality of end-effector heads, a modifiable end-effector system, a sensing system, and a grip planning processing pipeline used under the control of the robotic pick-and-place machine.

[0025] Another embodiment provides a method for robotic package handling, comprising: a. a robotic arm having a distal portion and a proximal base portion; an end effector coupled to the distal portion of the robotic arm; a place structure positioned geometrically close to the distal portion of the robotic arm; a pick structure positioned geometrically close to the distal portion of the robotic arm and in contact with one or more packages; a first imaging device positioned and oriented to capture image information relating to the pick structure and one or more packages; and a first computing system operably coupled to the robotic arm and the first imaging device, configured to receive image information from the first imaging device and to command the movement of the robotic arm at least partially based on the image information. The method relates to a first effector comprising a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing system, the first suction cup assembly defining a first inner capture chamber, and b. using the first computing system to operate a robotic arm and an end effector to grasp a targeted package from one or more packages from a pick structure, release the targeted package and place it on a place structure, wherein grasping the targeted package includes, when the vacuum load is controllably activated adjacent to the targeted package, drawing a portion of the targeted package into the first inner capture chamber and using it to at least partially seal it.

[0026] Another embodiment provides a method for robotic package handling, comprising: a. a robotic arm having a distal portion and a proximal base portion; an end effector coupled to the distal portion of the robotic arm; a place structure positioned geometrically close to the distal portion of the robotic arm; a pick structure positioned geometrically close to the distal portion of the robotic arm and in contact with one or more packages; a first imaging device positioned and oriented to capture image information relating to the pick structure and one or more packages; and a first computing system operably coupled to the robotic arm and the first imaging device, configured to receive image information from the first imaging device and to command the movement of the robotic arm at least partially based on the image information; and b. utilizing the first computing system to capture one or more packages from the pick structure The method relates to operating a robotic arm and an end effector to grasp a targeted package from a range, release the targeted package, and place it on a place structure, wherein the end effector comprises a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing device, the first suction cup assembly collectively defining a first inner chamber, a first outer seal edge, and a first vacuum permeable distal wall member, wherein, in response to grasping the targeted package using the controllably activated vacuum load, the outer seal edge may be in a state where it is removably coupled to at least one surface of the targeted package, while a vacuum permeable distal wall member is configured to prevent excessive protrusion of the surface of the targeted package into the inner chamber of the suction cup assembly.

[0027] Another embodiment is directed to a method for robotic package handling, comprising: a. providing a robotic arm comprising a distal portion and a proximal base portion, an end effector coupled to the distal portion of the robotic arm, a placing structure positioned in geometric proximity to the distal portion of the robotic arm, a picking structure positioned in geometric proximity to the distal portion of the robotic arm and in contact with one or more packages, a first imaging device positioned and oriented to capture image information relating to the picking structure and the one or more packages, and a first computing system operably coupled to the robotic arm and the first imaging device, configured to receive image information from the first imaging device and command movement of the robotic arm based at least in part on the image information; and b. utilizing the first computing system to operate the robotic arm and the end effector to grasp a targeted package among the one or more packages from the picking structure, release the targeted package and place it onto the placing structure, wherein the end effector comprises a first suction cup assembly coupled to a controllably activatable vacuum load that is operably coupled to the first computing system, the first suction cup assembly is configured such that performing grasping comprises engaging the targeted package when the vacuum load is controllably activated adjacent the targeted package, and before performing grasping, the computing device is configured to analyze a plurality of candidate grasps and select an execution grasp to be performed for removing the targeted package from the picking structure based at least in part on inference time usage of a neural network operated by the computing device, the neural network being trained using views developed from synthetic data including rendered images of three-dimensional models of one or more synthetic packages as contained in a synthetic picking structure.

[0028] Another embodiment provides a method for robotic package handling, comprising: a. a robotic arm having a distal portion and a proximal base portion; an end effector coupled to the distal portion of the robotic arm; a place structure positioned geometrically close to the distal portion of the robotic arm; a pick structure positioned geometrically close to the distal portion of the robotic arm and in contact with one or more packages; a first imaging device positioned and oriented to capture image information relating to the pick structure and one or more packages; and a first computing system operably coupled to the robotic arm and the first imaging device, configured to receive image information from the first imaging device and to command the movement of the robotic arm at least partially based on the image information; and b. utilizing the first computing system to grasp a targeted package from one or more packages from the pick structure, release the targeted package, and the place structure a. To operate a robotic arm and an end effector to rest on top of, the end effector comprising a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing system, wherein the first suction cup assembly is configured to perform a grip, which includes engaging with the targeted package when the vacuum load is controllably activated adjacent to the targeted package; c. To provide a second imaging device operably coupled to a first computing system, which is positioned and oriented to capture an image of one or more of the targeted packages after a grip has been performed using the end effector, to fit a 3D rectangular prism around the targeted package, to estimate the outer dimensional boundary of the targeted package by estimating the LWH of the rectangular prism, and to utilize the fitted 3D rectangular prism to estimate the position and orientation of the targeted package relative to the end effector; and d.using a first computing system to operate a robot arm and an end effector to place a targeted package onto a placement structure at a specific position and orientation relative to the placement structure..

[0029] Another embodiment is directed to a method, comprising: a. collecting image data of an object capture area; b. planning a grasp, which consists of evaluating the image data through a grasp quality model to generate a set of candidate grasp plans, processing the candidate grasp plans, and selecting a grasp plan; c. implementing the selected grasp plan using a robot system; and d. performing an object interaction task. The present invention provides, for example, the following items: (Item 1) A robotic package handling system, a. A robotic arm comprising a distal portion and a proximal base portion, b. An end effector connected to the distal portion of the robot arm, c. A place structure positioned geometrically in close proximity to the distal portion of the robot arm, d. A pick structure, the pick structure is in contact with one or more packages and is positioned geometrically close to the distal portion of the robot arm, e. A first imaging device, wherein the first imaging device is positioned and oriented to capture image information relating to the pick structure and one or more packages, f. A first computing system, wherein the first computing system is operably coupled to the robot arm and the first imaging device, and is configured to receive the image information from the first imaging device and to command the movement of the robot arm at least partially based on the image information. Equipped with, The first computing system is configured to operate the robotic arm and end effector to grasp a targeted package from the pick structure, one or more of the packages, release the targeted package, and place it on the place structure. The end effector comprises a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing system, wherein the first suction cup assembly defines a first inner capture chamber configured to grasp the targeted package, which, when the vacuum load is controllably activated adjacent to the targeted package, includes drawing a portion of the targeted package into the first inner capture chamber and using it to at least partially seal it. (Item 2) The system according to item 1, further comprising a frame structure configured to fix and connect the robot arm to the place structure. (Item 3) The system according to item 2, wherein the pick structure is detachably coupled to the frame structure. (Item 4) The placement structure is the system described in item 1, comprising an installation tray. (Item 5) The system according to item 4, wherein the installation tray comprises first and second rotatably coupled members, the first and second rotatably coupled members are configured to form a substantially flat tray base surface when in a first rotatably coupled configuration relative to each other, and to form a lifting fork configuration when in a second rotatably coupled configuration relative to each other. (Item 6) The system according to item 4, wherein the installation tray is operably coupled to one or more actuators configured to controllably change the orientation of at least a portion of the installation tray, and the one or more actuators are operably coupled to the first computing system. (Item 7) The system according to item 1, wherein the pick structure comprises elements selected from the group consisting of a container, a tray, a fixed surface, and a movable surface. (Item 8) The system according to item 7, wherein the pick structure comprises a container configured to define a package-containing volume bounded by a bottom and a plurality of walls, and an open access opening configured to accommodate the entry and exit of at least the distal portion of the robot arm. (Item 9) The system according to item 8, wherein the first imaging device is configured to capture the image information relating to the pick structure and one or more packages through the open access aperture. (Item 10) The first imaging device is the system described in item 1, comprising a depth camera. (Item 11) The system described in item 1, wherein the first imaging device is configured to capture color image data. (Item 12) The first computing system is the system according to item 2, comprising a VLSI computer operably coupled to the frame structure. (Item 13) The system according to item 1, wherein the first computing system comprises a network of interconnected computing devices, at least one of which is remotely located relative to the robotic arm. (Item 14) The system according to item 1, further comprising a second computing system operably coupled to the first computing system. (Item 15) The system according to item 14, wherein the second computing system is located remotely from the first computing system, and the first and second computing systems are operably coupled via a computer network. (Item 16) The first computing system according to item 1, wherein performing the gripping includes analyzing a plurality of candidate grippings and selecting an execution gripping to be performed to remove the targeted package from the pick structure. (Item 17) The system according to item 16, which includes analyzing multiple candidate grips to examine locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package. (Item 18) The system according to item 17, comprising analyzing multiple candidate grips, which involves examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package from multiple different end-effector approach orientations. (Item 19) The system according to item 17, comprising analyzing multiple candidate grips, which involves examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement with the surface of the targeted package from multiple different end-effector approach positions. (Item 20) The system according to item 17, wherein the first suction cup assembly comprises a first outer seal rim, and the seal engagement portion with the surface includes a substantially complete engagement portion between the first outer seal rim and the surface. (Item 21) The system described in item 17 involves examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package, in a purely geometric manner. (Item 22) The system according to item 16, wherein the first computing system is configured to select the executed grip based on candidate grip factors selected from a group consisting of estimated required time, estimated required calculation, and estimated success rate of grip. (Item 23) The first suction cup assembly comprises a bellows structure, as described in item 1. (Item 24) The bellows structure comprises a plurality of wall portions adjacent to and connected to the bent edge, as described in item 23. (Item 25) The bellows structure comprises a material selected from the group consisting of polyethylene, polypropylene, rubber, and thermoplastic elastomer, as described in item 24. (Item 26) The system according to item 1, wherein the first suction cup assembly comprises an outer housing and an internal structure coupled thereto. (Item 27) The system according to item 26, wherein the internal structure of the first suction cup assembly comprises a wall member coupled to a proximal base member. (Item 28) The system according to item 27, wherein the wall member has a substantially cylindrical shape with a proximal end and a distal end, and the proximal base member forms a substantially circular interface with the proximal end of the wall member. (Item 29) The system according to item 27, wherein the proximal base member defines one or more inlet openings through which the one or more inlet openings are configured to allow airflow through them in accordance with the activation of the controllably activated vacuum load. (Item 30) The system according to item 29, wherein the internal structure further comprises a distal wall member, the distal wall member comprising a structural opening ring portion configured to define an access portion to the inner capture chamber, and one or more transition air channels configured to allow airflow through it in accordance with the activation of the controllably activated vacuum load. (Item 31) The system according to item 30, wherein the one or more inlet openings and the one or more transition air channels function to allow a defined flow of air through the capture chamber such that it facilitates the releaseable coupling of the first suction cup assembly and the targeted package. (Item 32) The system according to item 1, wherein the one or more of the aforementioned packages are selected from the group consisting of bags, "poly bags", "poly", fiber bags, fiber envelopes, bubble wrap bags, bubble wrap envelopes, "Jiffy" bags, "Jiffy" envelopes, and substantially rigid rectangular parallelepiped structures. (Item 33) The aforementioned one or more packages include a fibrous bag containing a paper composite or polymer composite, as described in item 32. (Item 34) The aforementioned one or more packages include a fibrous envelope containing a paper composite or polymer composite, as described in item 32. (Item 35) The aforementioned one or more packages include a substantially rigid rectangular structure, including a box, as described in item 32. (Item 36) The system according to item 1, wherein the end effector comprises a second suction cup assembly coupled to the controllably activated vacuum load. (Item 37) The system according to item 36, wherein the second suction cup assembly defines a second internal capture chamber, the second internal capture chamber being configured to pull in and at least partially encapsulate a portion of the targeted package when the vacuum load is controllably activated adjacent to the targeted package. (Item 38) The system according to item 1, further comprising a second imaging device, the second imaging device being operably coupled to the first computing system and positioned and oriented to capture an image of one or more of the targeted packages after the gripping is performed using the end effector. (Item 39) The system according to item 38, wherein the first computing system and the second imaging device are configured to capture one or more images such that the outer dimensional boundary of the targeted package can be estimated. (Item 40) The system according to item 39, wherein the first computing system is configured to determine the dimensional boundary of the targeted package by utilizing one or more images to fit a 3D rectangular prism around the targeted package and estimating the LWH of the rectangular prism. (Item 41) The system according to item 40, wherein the first computing system is configured to use the adapted 3D rectangular prism to estimate the position and orientation of the targeted package relative to the end effector. (Item 42) The system according to item 38, further comprising a third imaging device, the third imaging device being operably coupled to the first computing system and positioned and oriented to capture an image of one or more of the targeted packages after the gripping is performed using the end effector. (Item 43) The system according to item 38, wherein the second imaging device and the first computing system are further configured to estimate whether the targeted package is deformable by capturing a sequence of images of the targeted package during the motion of the targeted package and analyzing the deformation of the targeted package in the sequence of images. (Item 44) The system according to item 38, wherein the first computing system and the second imaging device are configured to capture and utilize the one or more images after the grip has been performed using the end effector, and to estimate whether multiple packages or zero packages have been brought about by the grip that has been performed. (Item 45) The system according to item 44, wherein the first computing system is configured to terminate a grip in response to a determination that multiple packages or zero packages have been brought about by the grip that has been performed. (Item 46) The system according to item 1, wherein the end effector comprises an instrument switching head portion, the instrument switching head portion is configured to be controllably coupled to and then discoupled from the first suction cup assembly using an instrument holder mounted geometrically in close proximity to the distal portion of the robot arm. (Item 47) The system according to item 46, wherein the instrument holder is configured to hold and detachably connect to one or more additional suction cup assemblies or one or more other package interface instruments, so that the first computing device may be configured to perform instrument switching using the instrument switching head portion. (Item 48) A robotic package handling system, a. A robotic arm comprising a distal portion and a proximal base portion, b. An end effector connected to the distal portion of the robot arm, c. A place structure positioned geometrically in close proximity to the distal portion of the robot arm, d. A pick structure, the pick structure is in contact with one or more packages and is positioned geometrically close to the distal portion of the robot arm, e. A first imaging device, wherein the first imaging device is positioned and oriented to capture image information relating to the pick structure and one or more packages, f. A first computing system, wherein the first computing system is operably coupled to the robot arm and the first imaging device, and is configured to receive the image information from the first imaging device and to command the movement of the robot arm at least partially based on the image information. Equipped with, The first computing system is configured to operate the robotic arm and end effector to grasp a targeted package from the pick structure, one or more of the packages, release the targeted package, and place it on the place structure. The end effector comprises a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing device, the first suction cup assembly defining a first inner chamber, a first outer seal rim, and a first vacuum permeable distal wall member, the first inner chamber, the first outer seal rim, and the first vacuum permeable distal wall member collectively, in response to gripping the targeted package using the controllably activated vacuum load, the outer seal rim may be removably coupled to at least one surface of the targeted package, while the vacuum permeable distal wall member is configured to prevent excessive protrusion of the surface of the targeted package into the inner chamber of the suction cup assembly. (Item 49) The system according to item 48, further comprising a frame structure configured to fix and connect the robot arm to the place structure. (Item 50) The system according to item 49, wherein the pick structure is detachably coupled to the frame structure. (Item 51) The aforementioned placement structure comprises an installation tray, as described in item 48. (Item 52) The system according to item 51, wherein the installation tray comprises first and second rotatably coupled members, the first and second rotatably coupled members are configured to form a substantially flat tray base surface when in a first rotatably coupled configuration relative to each other, and to form a lifting fork configuration when in a second rotatably coupled configuration relative to each other. (Item 53) The system according to item 51, wherein the installation tray is operably coupled to one or more actuators configured to controllably change the orientation of at least a portion of the installation tray, and the one or more actuators are operably coupled to the first computing system. (Item 54) The system according to item 48, wherein the pick structure comprises elements selected from the group consisting of a container, a tray, a fixed surface, and a movable surface. (Item 55) The system according to item 54, wherein the pick structure comprises a container configured to define a package-containing volume bounded by a bottom and a plurality of walls, and an open access opening configured to accommodate the entry and exit of at least the distal portion of the robot arm. (Item 56) The system according to item 55, wherein the first imaging device is configured to capture the image information relating to the pick structure and one or more packages through the open access aperture. (Item 57) The first imaging device is the system described in item 48, comprising a depth camera. (Item 58) The system described in item 48, wherein the first imaging device is configured to capture color image data. (Item 59) The first computing system is the system according to item 49, comprising a VLSI computer operably coupled to the frame structure. (Item 60) The system according to item 48, wherein the first computing system comprises a network of interconnected computing devices, at least one of which is remotely located relative to the robotic arm. (Item 61) The system according to item 48, further comprising a second computing system operably coupled to the first computing system. (Item 62) The system according to item 61, wherein the second computing system is located remotely from the first computing system, and the first and second computing systems are coupled together operably via a computer network. (Item 63) The first computing system according to item 48, wherein performing the grip includes analyzing a plurality of candidate grips and selecting an execution grip to be performed to remove the targeted package from the pick structure. (Item 64) The system according to item 63, which includes analyzing multiple candidate grips, and examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package. (Item 65) The system according to item 64, comprising analyzing multiple candidate grips, scrutinizing locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package from multiple different end-effector approach orientations. (Item 66) The system according to item 64, comprising analyzing multiple candidate grips, which involves examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement with the surface of the targeted package from multiple different end-effector approach positions. (Item 67) The system according to item 64, wherein the first suction cup assembly comprises a first outer seal rim, and the seal engagement portion with the surface includes a substantially complete engagement portion between the first outer seal rim and the surface. (Item 68) The system described in item 64 involves examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package, in a purely geometric manner. (Item 69) The system according to item 63, wherein the first computing system is configured to select the executed grip based on candidate grip factors selected from a group consisting of estimated required time, estimated required calculation, and estimated success rate of grip. (Item 70) The first suction cup assembly comprises a bellows structure, as described in item 48. (Item 71) The system according to item 70, wherein the bellows structure comprises a plurality of wall portions adjacent to and connected to the bent edge. (Item 72) The system according to item 71, wherein the bellows structure comprises a material selected from the group consisting of polyethylene, polypropylene, rubber, and thermoplastic elastomer. (Item 73) The system according to item 48, wherein the first suction cup assembly comprises an outer housing and an internal structure coupled thereto. (Item 74) The system according to item 73, wherein the internal structure of the first suction cup assembly comprises a wall member coupled to a proximal base member. (Item 75) The system according to item 74, wherein the wall member has a substantially cylindrical shape having a proximal end and a distal end, and the proximal base member forms a substantially circular interface with the proximal end of the wall member. (Item 76) The system according to item 74, wherein the proximal base member defines one or more inlet openings through which the one or more inlet openings are configured to allow airflow through them in accordance with the activation of the controllably activated vacuum load. (Item 77) The system according to item 76, wherein the vacuum permeable distal wall member comprises a structural opening ring portion configured to define an access portion to the inner chamber, and one or more transition air channels configured to allow airflow through it in accordance with the activation of the controllably activated vacuum load. (Item 78) The system according to item 77, wherein the one or more inlet openings and the one or more transition air channels function to allow a defined flow of air through the capture chamber to facilitate the releaseable coupling of the first suction cup assembly and the targeted package. (Item 79) The system according to item 48, wherein the one or more of the aforementioned packages are selected from the group consisting of bags, "poly bags", "poly", fiber bags, fiber envelopes, bubble wrap bags, bubble wrap envelopes, "Jiffy" bags, "Jiffy" envelopes, and substantially rigid rectangular parallelepiped structures. (Item 80) The aforementioned one or more packages include a fibrous bag containing a paper composite or polymer composite, as described in item 79. (Item 81) The aforementioned one or more packages include a fiber-based envelope containing a paper composite or polymer composite, as described in item 79. (Item 82) The aforementioned one or more packages include a substantially rigid rectangular structure, including a box, as described in item 79. (Item 83) The system according to item 48, wherein the end effector comprises a second suction cup assembly coupled to the controllably activated vacuum load. (Item 84) The system according to item 83, wherein the second suction cup assembly defines a second inner chamber, a second outer seal rim, and a second vacuum permeable distal wall member, and the second inner chamber, the second outer seal rim, and the second vacuum permeable distal wall member are configured such that, in response to gripping the targeted package using the vacuum load which is activated in a controllable manner, the second outer seal rim may be removably bonded to at least one surface of the targeted package, while the second vacuum permeable distal wall member prevents excessive protrusion of the surface of the targeted package into the inner chamber of the suction cup assembly. (Item 85) The system according to item 48, further comprising a second imaging device, the second imaging device being operably coupled to the first computing system and positioned and oriented to capture an image of one or more of the targeted packages after the gripping is performed using the end effector. (Item 86) The system according to item 85, wherein the first computing system and the second imaging device are configured to capture one or more images such that the outer dimensional boundary of the targeted package can be estimated. (Item 87) The first computing system according to item 86, configured to determine the dimensional boundary of the targeted package by utilizing one or more images to fit a 3D rectangular prism around the targeted package and estimating the LWH of the rectangular prism. (Item 88) The system according to item 87, wherein the first computing system is configured to use the adapted 3D rectangular prism to estimate the position and orientation of the targeted package relative to the end effector. (Item 89) The system according to item 85, further comprising a third imaging device, the third imaging device being operably coupled to the first computing system and positioned and oriented to capture an image of one or more of the targeted packages after the gripping is performed using the end effector. (Item 90) The system according to item 85, wherein the second imaging device and the first computing system are further configured to estimate whether the targeted package is deformable by capturing a sequence of images of the targeted package during the motion of the targeted package and analyzing the deformation of the targeted package in the sequence of images. (Item 91) The system according to item 85, wherein the first computing system and the second imaging device are configured to capture and utilize the one or more images after the grip has been performed using the end effector, and to estimate whether multiple packages or zero packages have been brought about by the grip that has been performed. (Item 92) The system according to item 91, wherein the first computing system is configured to terminate a grip in response to a determination that multiple packages or zero packages have been brought about by the grip that has been performed. (Item 93) The system according to item 48, wherein the end effector comprises an instrument switching head portion, the instrument switching head portion is configured to be controllably coupled to and then discoupled from the first suction cup assembly using an instrument holder mounted geometrically in close proximity to the distal portion of the robot arm. (Item 94) The system according to item 93, wherein the instrument holder is configured to hold and detachably coupled to one or more additional suction cup assemblies or one or more other package interface instruments, so that the first computing device may be configured to perform instrument switching using the instrument switching head portion. (Item 95) A robotic package handling system, a. A robotic arm comprising a distal portion and a proximal base portion, b. An end effector connected to the distal portion of the robot arm, c. A place structure positioned geometrically in close proximity to the distal portion of the robot arm, d. A pick structure, the pick structure is in contact with one or more packages and is positioned geometrically close to the distal portion of the robot arm, e. A first imaging device, wherein the first imaging device is positioned and oriented to capture image information relating to the pick structure and one or more packages, f. A first computing system, wherein the first computing system is operably coupled to the robot arm and the first imaging device, and is configured to receive the image information from the first imaging device and to command the movement of the robot arm at least partially based on the image information. Equipped with, The first computing system is configured to operate the robotic arm and end effector to grasp a targeted package from the pick structure, one or more of the packages, release the targeted package, and place it on the place structure. The end effector comprises a first suction cup assembly, the first suction cup assembly being coupled to a controllably activated vacuum load operably coupled to the first computing system, and the first suction cup assembly being configured such that the gripping includes engaging with the targeted package when the vacuum load is controllably activated adjacent to the targeted package. Before performing the gripping, the computing device is configured to analyze a plurality of candidate grippings and, at least in part, select an execution gripping to be performed to remove the targeted package from the pick structure, based on the startup time usage of the neural network operated by the computing device. The neural network is a system trained using views unfolded from synthetic data, including rendered images of three-dimensional models of one or more synthetic packages, such as those contained by a synthetic pick structure. (Item 96) The system according to item 95, further comprising a frame structure configured to fix and connect the robot arm to the place structure. (Item 97) The system according to item 96, wherein the pick structure is detachably coupled to the frame structure. (Item 98) The aforementioned placement structure comprises an installation tray, as described in item 95. (Item 99) The system according to item 98, wherein the installation tray comprises first and second rotatably coupled members, the first and second rotatably coupled members are configured to form a substantially flat tray base surface when in a first rotatably coupled configuration relative to each other, and to form a lifting fork configuration when in a second rotatably coupled configuration relative to each other. (Item 100) The system according to item 98, wherein the installation tray is operably coupled to one or more actuators configured to controllably change the orientation of at least a portion of the installation tray, and the one or more actuators are operably coupled to the first computing system. (Item 101) The system according to item 95, wherein the pick structure comprises elements selected from the group consisting of a container, a tray, a fixed surface, and a movable surface. (Item 102) The system according to item 101, wherein the pick structure comprises a container configured to define a package-containing volume bounded by a bottom and a plurality of walls, and an open access opening configured to accommodate the entry and exit of at least the distal portion of the robot arm. (Item 103) The system according to item 102, wherein the first imaging device is configured to capture the image information relating to the pick structure and one or more packages through the open access aperture. (Item 104) The system described in item 95, wherein the first imaging device comprises a depth camera. (Item 105) The system according to item 95, wherein the first imaging device is configured to capture color image data. (Item 106) The first computing system is the system described in item 95, comprising a single VLSI computer. (Item 107) The first computing system, as described in item 95, wherein at least one of the computing devices constitutes a network of interconnected computing devices located remotely from the robotic arm. (Item 108) The system according to item 95, further comprising a second computing system operably coupled to the first computing system. (Item 109) The system according to item 108, wherein the second computing system is located remotely from the first computing system, and the first and second computing systems are coupled together operably via a computer network. (Item 110) The neural network is trained using a view unfolded from synthetic data, which includes rendered color images of three-dimensional models of one or more synthetic packages, such as those contained by a synthetic pick structure, as described in item 95. (Item 111) The neural network is trained using a view unfolded from synthetic data, including rendered depth images of three-dimensional models of one or more synthetic packages, such as those contained by a synthetic pick structure, as described in item 95. (Item 112) The neural network is trained using a view unfolded from synthetic data, which includes rendered images of three-dimensional models of one or more randomized synthetic packages, such as those contained by a synthetic pick structure, as described in item 95. (Item 113) The aforementioned synthesis package is randomized by a color texture, as described in item 112. (Item 114) The system described in item 112, wherein the synthetic package is randomized by a physically based rendering mapping selected from the group consisting of reflection, diffusion, translucency, transparency, metallicity, and micro-surface scattering. (Item 115) The neural network is trained using a view unfolded from synthetic data, which includes rendered images of three-dimensional models of one or more synthetic packages in random positions and orientations, such as those contained by a synthetic pick structure, as described in item 95. (Item 116) The first computing system is configured such that performing the gripping includes analyzing a plurality of candidate grippings and selecting an execution gripping to be performed to remove the targeted package from the pick structure, as described in item 95. (Item 117) The system according to item 116, comprising analyzing multiple candidate grips, and examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package. (Item 118) The system according to item 117, comprising analyzing multiple candidate grips, which involves examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package from multiple different end-effector approach orientations. (Item 119) The system according to item 117, comprising analyzing multiple candidate grips, which involves examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package from multiple different end-effector approach positions. (Item 120) The system according to item 117, wherein the first suction cup assembly comprises a first outer seal rim, and the seal engagement portion with the surface includes a substantially complete engagement portion between the first outer seal rim and the surface. (Item 121) The system according to item 117, wherein the location on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement with the surface of the targeted package is examined in a purely geometric manner. (Item 122) The system according to item 116, wherein the first computing system is configured to select the executed grip based on candidate grip factors selected from a group consisting of estimated required time, estimated required calculation, and estimated success rate of grip. (Item 123) The first suction cup assembly comprises a bellows structure, as described in item 95. (Item 124) The bellows structure comprises a plurality of wall portions adjacent to and connected to the bent edge, as described in item 123. (Item 125) The system according to item 124, wherein the bellows structure comprises a material selected from the group consisting of polyethylene, polypropylene, rubber, and thermoplastic elastomer. (Item 126) The system according to item 95, wherein the first suction cup assembly comprises an outer housing and an internal structure coupled thereto. (Item 127) The internal structure of the first suction cup assembly comprises a wall member coupled to a proximal base member, the wall member and the proximal base member defining an inner chamber, as in the system described in item 126. (Item 128) The system according to item 127, wherein the wall member has a substantially cylindrical shape having a proximal end and a distal end, and the proximal base member forms a substantially circular interface with the proximal end of the wall member. (Item 129) The system according to item 127, wherein the proximal base member defines one or more inlet openings through which the one or more inlet openings are configured to allow airflow through them in accordance with the activation of the controllably activated vacuum load. (Item 130) The system according to item 129, wherein the internal structure further comprises a distal wall member, the distal wall member comprising a structural opening ring portion configured to define an access portion to the inner chamber, and one or more transition air channels configured to allow airflow through it in accordance with the activation of the controllably activated vacuum load. (Item 131) The system according to item 130, wherein the one or more inlet openings and the one or more transition air channels function to allow a defined flow of air through the inner chamber such that it facilitates the releaseable coupling of the first suction cup assembly and the targeted package. (Item 132) The system according to item 95, wherein the one or more of the aforementioned packages are selected from the group consisting of bags, "poly bags", "poly", fiber bags, fiber envelopes, bubble wrap bags, bubble wrap envelopes, "Jiffy" bags, "Jiffy" envelopes, and substantially rigid rectangular parallelepiped structures. (Item 133) The aforementioned one or more packages include a fibrous bag containing a paper composite or polymer composite, as described in item 132. (Item 134) The aforementioned one or more packages include a fibrous envelope containing a paper composite or polymer composite, as described in item 132. (Item 135) The aforementioned one or more packages include a substantially rigid rectangular structure, including a box, as described in item 132. (Item 136) The system according to item 95, wherein the end effector comprises a second suction cup assembly coupled to the controllably activated vacuum load. (Item 137) The system according to item 136, wherein the second suction cup assembly defines a second inner chamber, the second inner chamber being configured to pull in and at least partially seal a portion of the targeted package when the vacuum load is controllably activated adjacent to the targeted package. (Item 138) The system according to item 95, further comprising a second imaging device, the second imaging device being operably coupled to the first computing system and positioned and oriented to capture an image of one or more of the targeted packages after the gripping is performed using the end effector. (Item 139) The system according to item 138, wherein the first computing system and the second imaging device are configured to capture one or more images such that the outer dimensional boundary of the targeted package can be estimated. (Item 140) The first computing system according to item 139, configured to determine the dimensional boundary of the targeted package by utilizing one or more images to fit a 3D rectangular prism around the targeted package and estimating the LWH of the rectangular prism. (Item 141) The system according to item 140, wherein the first computing system is configured to use the adapted 3D rectangular prism to estimate the position and orientation of the targeted package relative to the end effector. (Item 142) The system according to item 138, further comprising a third imaging device, the third imaging device being operably coupled to the first computing system and positioned and oriented to capture an image of one or more of the targeted packages after the gripping is performed using the end effector. (Item 143) The system according to item 138, wherein the second imaging device and the first computing system are further configured to estimate whether the targeted package is deformable by capturing a sequence of images of the targeted package during the motion of the targeted package and analyzing the deformation of the targeted package in the sequence of images. (Item 144) The system according to item 138, wherein the first computing system and the second imaging device are configured to capture and utilize the one or more images after the grip has been performed using the end effector, and to estimate whether multiple packages or zero packages have been brought about by the grip that has been performed. (Item 145) The system according to item 144, wherein the first computing system is configured to terminate a grip in response to a determination that multiple packages or zero packages have been brought about by the grip that has been performed. (Item 146) The system according to item 95, wherein the end effector comprises an instrument switching head portion, the instrument switching head portion being configured to be controllably coupled to and then discoupled from the first suction cup assembly using an instrument holder mounted geometrically in close proximity to the distal portion of the robot arm. (Item 147) The system according to item 146, wherein the instrument holder is configured to hold and detachably connect to one or more additional suction cup assemblies or one or more other package interface instruments, so that the first computing device may be configured to perform instrument switching using the instrument switching head portion. (Item 148) A robotic package handling system, a. A robotic arm comprising a distal portion and a proximal base portion, b. An end effector connected to the distal portion of the robot arm, c. A place structure positioned geometrically in close proximity to the distal portion of the robot arm, d. A pick structure, the pick structure is in contact with one or more packages and is positioned geometrically close to the distal portion of the robot arm, e. A first imaging device, wherein the first imaging device is positioned and oriented to capture image information relating to the pick structure and one or more packages, f. A first computing system, wherein the first computing system is operably coupled to the robot arm and the first imaging device, and is configured to receive the image information from the first imaging device and to command the movement of the robot arm at least partially based on the image information. Equipped with, The first computing system is configured to operate the robotic arm and end effector to grasp a targeted package from the pick structure, one or more of the packages, release the targeted package, and place it on the place structure. The end effector comprises a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to the first computing system, wherein the first suction cup assembly is configured to perform the gripping, which includes engaging with the targeted package when the vacuum load is controllably activated adjacent to the targeted package. The system further comprises a second imaging device, the second imaging device operably coupled to the first computing system, and is positioned and oriented to capture an image of one or more of the targeted packages after the gripping is performed using the end effector, to fit a 3D rectangular prism around the targeted package, to estimate the outer dimensional boundary of the targeted package by estimating the LWH of the rectangular prism, and to use the fitted 3D rectangular prism to estimate the position and orientation of the targeted package relative to the end effector. The first computing system is configured to operate the robotic arm and end effector to place the targeted package on the place structure in a specific position and orientation relative to the place structure. (Item 149) The system according to item 148, further comprising a frame structure configured to fix and connect the robot arm to the place structure. (Item 150) The system according to item 149, wherein the pick structure is detachably coupled to the frame structure. (Item 151) The aforementioned placement structure comprises an installation tray, as described in item 148. (Item 152) The system according to item 151, wherein the installation tray comprises first and second rotatably coupled members, the first and second rotatably coupled members are configured to form a substantially flat tray base surface when in a first rotatably coupled configuration relative to each other, and to form a lifting fork configuration when in a second rotatably coupled configuration relative to each other. (Item 153) The system according to item 151, wherein the installation tray is operably coupled to one or more actuators configured to controllably change the orientation of at least a portion of the installation tray, and the one or more actuators are operably coupled to the first computing system. (Item 154) The system according to item 148, wherein the pick structure comprises elements selected from the group consisting of a container, a tray, a fixed surface, and a movable surface. (Item 155) The system according to item 154, wherein the pick structure comprises a container configured to define a package-containing volume bounded by a bottom and a plurality of walls, and an open access opening configured to accommodate the entry and exit of at least the distal portion of the robot arm. (Item 156) The system according to item 155, wherein the first imaging device is configured to capture the image information relating to the pick structure and one or more packages through the open access aperture. (Item 157) The first imaging device is the system described in item 148, comprising a depth camera. (Item 158) The system described in item 148, wherein the first imaging device is configured to capture color image data. (Item 159) The first computing system is the system described in item 148, comprising a single VLSI computer. (Item 160) The first computing system, as described in item 148, wherein at least one of the interconnected computing devices is located remotely from the robotic arm. (Item 161) The system according to item 148, further comprising a second computing system operably coupled to the first computing system. (Item 162) The system according to item 161, wherein the second computing system is located remotely from the first computing system, and the first and second computing systems are operably coupled via a computer network. (Item 163) The first computing system according to item 148, wherein performing the gripping includes analyzing a plurality of candidate grippings and selecting an execution gripping to be performed to remove the targeted package from the pick structure. (Item 164) The system according to item 163, which includes analyzing multiple candidate grips to examine locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package. (Item 165) The system according to item 164, which includes analyzing multiple candidate grips, and examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package from multiple different end-effector approach orientations. (Item 166) The system according to item 164, comprising analyzing multiple candidate grips, which involves examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement with the surface of the targeted package from multiple different end-effector approach positions. (Item 167) The system according to item 164, wherein the first suction cup assembly comprises a first outer seal rim, and the seal engagement portion with the surface includes a substantially complete engagement portion between the first outer seal rim and the surface. (Item 168) The system described in item 164 involves examining locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package, in a purely geometric manner. (Item 169) The system according to item 163, wherein the first computing system is configured to select the executed grip based on candidate grip factors selected from a group consisting of estimated required time, estimated required calculation, and estimated success rate of grip. (Item 170) The first suction cup assembly comprises a bellows structure, as described in item 148. (Item 171) The bellows structure comprises a plurality of wall portions adjacent to and connected to the bent edge, as described in item 170. (Item 172) The system according to item 171, wherein the bellows structure comprises a material selected from the group consisting of polyethylene, polypropylene, rubber, and thermoplastic elastomer. (Item 173) The system according to item 148, wherein the first suction cup assembly comprises an outer housing and an internal structure coupled thereto. (Item 174) The internal structure of the first suction cup assembly comprises a wall member coupled to a proximal base member, the wall member and the proximal base member defining an inner chamber, as described in item 173. (Item 175) The system according to item 174, wherein the wall member has a substantially cylindrical shape having a proximal end and a distal end, and the proximal base member forms a substantially circular interface with the proximal end of the wall member. (Item 176) The system according to item 174, wherein the proximal base member defines one or more inlet openings through which the one or more inlet openings are configured to allow airflow through them in accordance with the activation of the controllably activated vacuum load. (Item 177) The system according to item 176, wherein the internal structure further comprises a distal wall member, the distal wall member comprising a structural opening ring portion configured to define an access portion to the inner chamber, and one or more transition air channels configured to allow airflow through it in accordance with the activation of the controllably activated vacuum load. (Item 178) The system according to item 177, wherein the one or more inlet openings and the one or more transition air channels function to allow a defined flow of air through the inner chamber such that it facilitates the releaseable coupling of the first suction cup assembly and the targeted package. (Item 179) The system according to item 148, wherein the one or more of the aforementioned packages are selected from the group consisting of bags, "poly bags", "poly", fiber bags, fiber envelopes, bubble wrap bags, bubble wrap envelopes, "Jiffy" bags, "Jiffy" envelopes, and substantially rigid rectangular parallelepiped structures. (Item 180) The aforementioned one or more packages include a fibrous bag containing a paper composite or polymer composite, as described in item 179. (Item 181) The aforementioned one or more packages include a fibrous envelope containing a paper composite or polymer composite, as described in item 179. (Item 182) The aforementioned one or more packages include a substantially rigid rectangular prism structure, including a box, as described in item 179. (Item 183) The system according to item 148, wherein the end effector comprises a second suction cup assembly coupled to the controllably activated vacuum load. (Item 184) The system according to item 183, wherein the second suction cup assembly defines a second inner chamber, the second inner chamber being configured to pull in and at least partially seal a portion of the targeted package when the vacuum load is controllably activated adjacent to the targeted package. (Item 185) The system according to item 148, further comprising a third imaging device, the third imaging device being operably coupled to the first computing system and positioned and oriented to capture an image of one or more of the targeted packages after the gripping is performed using the end effector. (Item 186) The system according to item 148, wherein the second imaging device and the first computing system are further configured to estimate whether the targeted package is deformable by capturing a sequence of images of the targeted package during the motion of the targeted package and analyzing the deformation of the targeted package in the sequence of images. (Item 187) The system according to item 148, wherein the first computing system and the second imaging device are configured to capture and utilize the one or more images after the grip has been performed using the end effector, and to estimate whether multiple packages or zero packages have been brought about by the grip that has been performed. (Item 188) The system according to item 187, wherein the first computing system is configured to terminate a grip in response to a determination that multiple packages or zero packages have been brought about by the grip that has been performed. (Item 189) The system according to item 148, wherein the end effector comprises an instrument switching head portion, the instrument switching head portion being configured to be controllably coupled to and then discoupled from the first suction cup assembly using an instrument holder mounted geometrically in close proximity to the distal portion of the robot arm. (Item 190) The system according to item 189, wherein the instrument holder is configured to hold and detachably coupled to one or more additional suction cup assemblies or one or more other package interface instruments, so that the first computing device may be configured to perform instrument switching using the instrument switching head portion. (Item 191) The system according to item 148, wherein the first computing system is configured to operate the robotic arm and end effector on the place structure together with the targeted package such that the targeted package is dragged into the inclined portion including the place structure. (Item 192) The system according to item 148, wherein the first computing system is configured to operate the robotic arm and end effector on the place structure together with the targeted package so that the targeted package is intentionally placed on the edge of the targeted package so that it will tip over onto the surface of the place structure in a preferred orientation and position. (Item 193) The system according to item 148, wherein the first computing system is configured to operate the robotic arm and end effector on the place structure together with the targeted package so that the targeted package is swept across the surface of the place structure such that the targeted package remains substantially flat with respect to the surface. (Item 194) It is a system, a. A robotic pick-and-place machine comprising an actuation system configured to facilitate selection and switching between multiple end effector heads, and a changeable end effector system, b. Sensing system, c. A gripping planning processing pipeline used under the control of the robot pick-and-place machine. A system equipped with these features. (Item 195) The changeable end effector system comprises a head selector integrated into the distal end of the actuation system, a set of end effector heads, and a head holding device, wherein the head selector is mounted together with one of the set of end effector heads on an individual mounting surface, as described in item 194. (Item 196) The changeable end effector system further comprises at least one magnet, the at least one magnet surrounding the center of either the head selector or the effector head, and providing initial seating and retention of the end effector head, as described in item 195. (Item 197) The system according to item 196, wherein at least one of the head selector or the set of end effector heads comprises a seal positioned along the outer edge of an individual mounting surface. (Item 198) The head selector and the end effector head set are provided with a complementary alignment structure, as described in item 195. (Item 199) The system according to item 195, wherein the head selector and the set of end effector heads are provided with a lateral support structure geometry selected to assist in gripping the flexible package. (Item 200) The set of end effector heads comprises a set of suction end effectors, as described in item 195. (Item 201) The aforementioned operating system comprises an articulated arm, as described in item 195. (Item 202) The grip planning pipeline comprises one or more processors that include machine-readable instructions, and when the machine-readable instructions are executed, they are sent to the one or more processors. a. Collecting image data of the object capture region, b. Having the system perform a grip planning, the system performing the grip planning includes evaluating image data through a grip quality model, generating a set of candidate grip plans, processing the candidate grip plans, and selecting a grip plan. c. To carry out the selected gripping plan using the robot pick-and-place machine, d. Perform object interaction tasks related to targeted packages. The system described in item 194, which enables the following. (Item 203) The system described in item 202, which evaluates image data through a grip quality model and generates a set of candidate grip plans, includes segmenting the image data into a region of interest mask and evaluating the image data and region of interest mask through a neural network architecture to generate a detailed prediction of grip quality for a set of instruments at multiple locations in the image data, with associated probabilities of success. [Brief explanation of the drawing]

[0030] [Figure 1] Figure 1 illustrates a schematic diagram of the robot package handling system configuration.

[0031] [Figure 2] Figure 2 illustrates an embodiment with a modifiable end-effector configuration.

[0032] [Figure 3] Figure 3 illustrates an embodiment in which a head selector engages with an end effector head.

[0033] [Figure 4] Figure 4 illustrates an embodiment of a head selector that engages with an end effector head having a lateral support.

[0034] [Figure 5] Figure 5 illustrates an embodiment of an end-effector head having multiple selectable end-effectors.

[0035] [Figure 6] Figure 6 illustrates an embodiment of an end-effector head having multiple selectable end-effectors.

[0036] [Figure 7A] Figures 7A-7G illustrate various aspects of one embodiment of a robot package handling configuration. [Figure 7B]Figures 7A-7G illustrate various aspects of one embodiment of a robot package handling configuration. [Figure 7C] Figures 7A-7G illustrate various aspects of one embodiment of a robot package handling configuration. [Figure 7D] Figures 7A-7G illustrate various aspects of one embodiment of a robot package handling configuration. [Figure 7E] Figures 7A-7G illustrate various aspects of one embodiment of a robot package handling configuration. [Figure 7F] Figures 7A-7G illustrate various aspects of one embodiment of a robot package handling configuration. [Figure 7G] Figures 7A-7G illustrate various aspects of one embodiment of a robot package handling configuration.

[0037] [Figure 8A] Figures 8A-8B illustrate various aspects of the suction cup assembly end effector. [Figure 8B] Figures 8A-8B illustrate various aspects of the suction cup assembly end effector.

[0038] [Figure 9A] Figures 9A-9B illustrate various aspects of the suction cup assembly end effector. [Figure 9B] Figures 9A-9B illustrate various aspects of the suction cup assembly end effector.

[0039] [Figure 10A] Figures 10A-10F illustrate various aspects of an embodiment of the place structure configuration. [Figure 10B] Figures 10A-10F illustrate various aspects of an embodiment of the place structure configuration. [Figure 10C] Figures 10A-10F illustrate various aspects of an embodiment of the place structure configuration. [Figure 10D] Figures 10A-10F illustrate various aspects of an embodiment of the place structure configuration. [Figure 10E] Figures 10A-10F illustrate various aspects of an embodiment of the place structure configuration. [Figure 10F] Figures 10A-10F illustrate various aspects of an embodiment of the place structure configuration.

[0040] [Figure 11A] Figures 11A-11C illustrate various aspects of embodiments of a robotic package handling configuration featuring one or more interconnected computing systems. [Figure 11B] Figures 11A-11C illustrate various aspects of embodiments of a robotic package handling configuration featuring one or more interconnected computing systems. [Figure 11C] Figures 11A-11C illustrate various aspects of embodiments of a robotic package handling configuration featuring one or more interconnected computing systems.

[0041] [Figure 12] Figure 12 illustrates one embodiment of a computing architecture that may be used to implement aspects of the thematic configuration.

[0042] [Figure 13] Figures 13-19 illustrate various embodiments of the method. [Figure 14] Figures 13-19 illustrate various embodiments of the method. [Figure 15] Figure 13-19 illustrates various embodiments of the method. [Figure 16] Figures 13-19 illustrate various embodiments of the method. [Figure 17] Figures 13-19 illustrate various embodiments of the method. [Figure 18] Figures 13-19 illustrate various embodiments of the method. [Figure 19] Figures 13-19 illustrate various embodiments of the method.

[0043] [Figure 20A] Figures 20A and 20B illustrate images of the composite data. [Figure 20B] Figures 20A and 20B illustrate images of the composite data. [Modes for carrying out the invention]

[0044] Detailed explanation The following sequentially numbered U.S. patent applications, namely, No. 17 / 220,679 (Publication No. 2021 / 0308874), No. 17 / 220,694 (Publication No. 2021 / 0308875), No. 17 / 404,748 (Publication No. 2022 / 0048707), and No. 17 / 468,220 (Publication No. 2022 / 0072587), are incorporated herein by reference as a whole.

[0045] Referring to Figure 1, a system for planning and adapting object manipulation may include a robotic pick-and-place machine (2) with an actuation system (8) and a modifiable end-effector system (4), and a sensing system and a grip planning processing pipeline (6) used under the control of the robotic pick-and-place machine. The system and method may also include a work station configuration module used in dynamically defining the environmental configuration of the robotic system. The system is preferably used in situations where a set of objects in one area needs to be processed or manipulated in a certain way.

[0046] In many pick-and-place applications, this system is used when a set of objects (e.g., products) is presented in a way that is within the environment. The objects may be stored and presented in containers, transport boxes, bags, boxes, and / or other storage elements. The objects may also be presented through an item supply system such as a conveyor belt. In addition, the system may need to manipulate the objects to place them within such storage elements, such as by moving them from a container into a box specific to that object. Similarly, the system may be used to move objects into a bagging system or another object handling system such as a conveyor belt.

[0047] The system may be implemented in an integrated workstation, which is a single unit in which various elements are physically integrated. However, some parts of the computing infrastructure and resources may be remote and accessed via a communication network. In one embodiment, the integrated workstation includes a robotic pick-and-place machine (2) with a physically coupled sensing system. Thus, the integrated workstation can be moved and fixed in place and begin operating on objects in the environment. Alternatively, the system may be implemented as a collection of discrete components that operate collaboratively. For example, the sensing system in one implementation may be physically removed from the robotic pick-and-place machine. The workstation configuration modules described below may be used in customized configurations and settings of such a workstation.

[0048] A robotic pick-and-place machine functions as an automated system used to interact with an object. The robotic pick-and-place machine (2) preferably includes an actuation system (8) and an end effector (4) used to temporarily physically bind to an object (e.g., grasp or attach) and perform some operation on that object. The actuation system is used to move the end effector and, once bound to one or more objects, move and orient the object in space. Preferably, the robotic pick-and-place machine is used to pick up an object, manipulate the object (move the object, and / or reorient it), and then, upon completion, place the object. Hereinafter, robotic pick-and-place machines are more commonly referred to as robotic systems. A variety of robotic systems may be used. In one preferred implementation, the robotic system is an articulated arm using a pressure-based suction cup end effector. The robotic system may include a variety of features or designs.

[0049] The actuation system (8) functions to translate the end effector through space. The actuation system will preferably move the end effector to various locations for interaction with various objects. In addition, or alternatively, the actuation system may be used to move the end effector and the object being grasped along a specific path, to orient the end effector and / or the object being grasped, and / or to provide any preferred operation of the end effector. Generally, the actuation system is used for the overall movement of the end effector.

[0050] The actuation system (8) may be one of various types of machines used to facilitate the movement of the end effector. In one preferred modification, the actuation system is a robotic articulated arm having multiple acting degrees of freedom coupled through interconnected arm compartments. One preferred modification of the actuated robotic arm is a 6-axis robotic arm having 6 degrees of freedom, as shown in Figure 1. The actuation system may alternatively be a robotic arm with fewer degrees of freedom, such as a 4-axis or 5-axis robotic arm, or with additional articulated degrees of freedom, such as a 7-axis robotic arm.

[0051] In other variations, the actuation system may be any various robotic systems such as Cartesian coordinate robots, cylindrical coordinate robots, polar coordinate robots, horizontal articulated robots, delta robots, and / or any other variations of robotic systems for controlled actuation.

[0052] The actuation system (8) preferably includes an end arm section. The end arm section is preferably a rigid structure extending from the final operating degree of freedom portion of the actuation system. In an articulated robotic arm, the last arm section is coupled to the end effector (4). As described below, the end of the end arm section may include a head selector, which is part of a modifiable end effector system.

[0053] In one modified example, the end arm compartment may also include, or be connected to, at least one flexible joint.

[0054] The flexible joint preferably functions as at least one additional degree of freedom, located near the end effector. The flexible joint is preferably located at the distal end of the end arm section of the actuation system, and the flexible joint can function as a “wrist” joint. The flexible joint preferably provides an auxiliary amount of dexterity near where the end effector interacts with an object, which can be useful in various situations when interacting with an object.

[0055] In the multi-device modification of this system, the flexible joint preferably precedes the head selector component so that each mountable end effector head can be used in conjunction with controllable flexibility. Alternatively, one or more end effectors may have flexible joints.

[0056] In variations of the multi-head device, the flexible joint may be integrated into a shared mounting point of the multi-head end effector. Thus, the use of connected end effectors allows for the sharing of common degrees of freedom in the flexible joint. Alternatively, one or more end effectors of the multi-head end effector may include the flexible joint. In this way, each individual end effector can have independent flexibility.

[0057] A flexible joint is preferably a controllable flexible joint, which can be selectively manufactured to move in at least partially flexible mode. When moving in flexible mode, the flexible joint can preferably act in response to an external force. Preferably, the flexible joint has a controllable degree of rotational freedom so that the flexible joint can rotate in response to an external force. The flexible joint can also preferably be selectively manufactured to act in a controlled mode. In one preferred modification, the controllable flexible joint has a single degree of rotational freedom, which, when engaged in flexible mode, can rotate freely (at least within a certain angular range) and, when engaged in controlled mode, can be actuated to rotate in a controlled mode. Linear flexible action may also be designed in the flexible joint, either in addition to or alternatively. The flexible joint may also be controlled for variable or partially flexible forms of action, which the flexible joint can actuate but is flexible to forces above a certain threshold.

[0058] The end effector (4) functions to facilitate direct interaction with the object. Preferably, the system is used to grasp an object, and the grasp describes a physical coupling with the object for physical manipulation. Controllable grasping preferably allows the end effector to selectively connect / couple to ("grasp" or "pick") an object and selectively disconnect / decouple from ("drop" or "set") an object. The end effector may controllably "grasp" an object through attractive force, pinch an object, apply a magnetic field, and / or through any suitable force. In this specification, the system is described primarily in terms of attractive-based grasping of an object, but the modifications described herein are not necessarily limited to attractive-based end effectors.

[0059] In one preferred modification, the end effector (4) includes a suction end effector head (24) (which may be more simply referred to as a suction head) connected to a pressure system. The suction head preferably includes one or more suction cups (26, 28, 30, 32). The suction cups may be stocked in a variety of sizes, rigidities, shapes, and other configurations. Some embodiments of the suction head configuration may include a single suction cup configuration, a four-suction cup configuration, and / or other modifications. The size, material, and geometric shape of the suction head may also be modified to target different applications. The pressure system will generally include at least one vacuum pump connected to the suction head through one or more hoses.

[0060] In one preferred modification, the end effector of the system includes a multi-head end effector fixture, which includes a plurality of selectable end effector heads, as shown in the exemplary modifications of Figures 5(34) and 6(24). Each end effector head can be connected to an individually controlled pressure system. The system can selectively activate one or more pressure systems and grip using one or more end effectors of the multi-head end effector fixture. The end effector heads are preferably selected and used based on dynamic control inputs from a grip planning model. The pressure system may, alternatively, use a controllable valve to redirect the airflow. Different end effectors are preferably spaced apart. They may be angled substantially in the same direction, but the end effectors may, alternatively, be oriented outward in a non-parallel direction from the end arm section.

[0061] As shown in the cross-sectional view of Figure 5, one exemplary modification of the multi-head end-effector instrument may be a two-head gripper (34). This modification may be specialized to reach into the corners of deep containers or vessels and to pick up small objects (e.g., small items such as pencils) and larger objects (e.g., boxes). In one modification, each gripping head end-effector may be able to slide linearly on a spring mechanism. The end-effector heads may be coupled to a hose that connects to a pressure system. The hose may be spirally coiled around a central shaft and allow movement of the suction head to connect to a vacuum generator.

[0062] As shown in Figure 6, another exemplary modification of the multi-head end effector device (24) can be a multi-four-head gripper. As shown in this modification, various sensors, such as cameras or barcode readers, can be integrated into the multi-head end effector device, shown here in the palm. Suction cup end effector heads can be selected collectively to have a wide range of applications (e.g., one for small boxes, one for large boxes, one for loose poly bags, one for more rigid poly bags). Multiple gripper combinations can pick up objects of different sizes. In some modifications, this multi-head end effector device may be connected to a robot by a spring plunger to allow for positioning errors.

[0063] Another preferred modification of the system includes a changeable end-effector system that functions to allow the end-effector to be changed. The changeable end-effector system preferably includes a head selector (36), a set of end-effector heads, and a head holding device (38), i.e., an instrument holder for so-called “instrument switching,” which are preferably integrated at the distal end of the actuation system (e.g., an end arm compartment). The end-effector heads are preferably selected and used based on dynamic control input from a grip planning model. The head selector and end-effector heads are preferably mounted together at the selector and head mounting sites. One or more end-effector heads may be stored in the head holding device (38) when not in use and when in use. The head holding device may also be able to orient the stored end-effector heads for easier selection during storage. The head holding device may also partially restrict the movement of the end-effector heads in at least one direction to facilitate mounting to or removal from the head selector.

[0064] The head selector system functions to selectively attach and detach multiple end-effector heads. The end-effector heads function as physical parts for engaging with an object. The end-effectors can be configured specifically for different situations. In some modifications, the head selector system may be used in combination with a multi-head end-effector fixture. For example, one or more end-effector heads may be removable and changeable through the head selector system.

[0065] A changeable end-effector system can employ various designs in allowing the end-effector to be changed. In one variation, the changeable end-effector is a passive variation in which the end-effector head is attached to and removed from a robotic system without the use of a controlled mechanism. In the passive variation, the actuation and / or pneumatic control capability of the robotic system may be used to engage and disengage different end-effector heads. Static magnets (44, 46), physical fasteners (48) (threads, feed / matching structures, friction mating, or snap-fit ​​fasteners) and / or other static mechanisms may also be used to temporarily attach the end-effector head and head selector.

[0066] In another modification, the changeable end effector is an active system that uses an activated mechanism (e.g., mechanical, electromechanical, electromagnetic, etc.) to engage with and disengage a selected end effector head. While passive modifications are primarily used in this description, modifications of the system and method may also be used in conjunction with active or alternative modifications.

[0067] One preferred modification of the modifiable end-effector system is designed for use with a robotic system, employing a pressure system with a suction head end-effector. A head selector can further function to direct pressure to the end-effector head. The head selector may include defined internal through-holes so that the pressure system is coupled to the end-effector head. The end-effector head will generally be a suction head. A set of suction end-effector heads can have a variety of designs, as shown in Figure 2.

[0068] The head selector and / or end effector head may include a seal (40, 42) element surrounding the defined through-hole. The seal can allow the pressure system to reinforce the mounting of the head selector and end effector head. This force is activated when the end effector is used to pick up an object and should help the end effector head remain mounted when the outer object is loaded.

[0069] The seals (40, 42) are preferably integrated within the mounting surface of the head selector, but the seals may also be integrated within the end effector head, in addition or alternatively. The seals may be O-rings, gaskets, or other sealing elements. Preferably, the seals are positioned along the outer edge of the mounting surface. The outer edge is preferably a mounting location along the mounting surface where more surface area of ​​the mounting surface exists on the inner portion compared to the outer portion. For example, in one implementation, the seals may be positioned such that more than 75% of their surface area is within the inner portion. This can increase the surface area over which the pressure system can apply force.

[0070] Magnets (44, 46) may be used in a modifiable end-effector system to facilitate passive mounting. Preferably, the magnets are integrated into a set of head selectors and / or end-effector heads. In a preferred modification, the magnets are integrated into both the head selectors and the end-effector heads. Alternatively, the magnets may be integrated into one of the head selectors or end-effector heads, the other having a ferromagnetic metal piece instead of a magnet.

[0071] In one implementation, the magnet has a single magnetic pole (e.g., the N-plane of the magnet, oriented upward on the head selector, and the S-plane of a second magnet, oriented outward on each end-effector head) that is aligned in the mounting direction. The use of both poles in the head selector and end-effector heads can increase the attractive force.

[0072] The magnets can be centered or aligned around the center of the mounting site. In one mounting, the magnets can surround the center and a defined cavity through which air can flow for a pressure-based end effector. In another modification, multiple magnets may be positioned around the center of the mounting point, which can be used to facilitate some degree of alignment between the head selector and the end effector head. In one modification, the magnets may be asymmetrical outward with respect to the center, and / or the polar alignment may be modified to further facilitate the desired alignment between the head selector and the end effector head.

[0073] In one implementation, the magnet can provide initial seating and retention of the end effector head when not engaged with an object (e.g., not under pressure), while the seal and / or pressure system can provide the primary attractive force when holding the object.

[0074] A modifiable end-effector system may include various structural elements that function in a variety of ways, including providing reinforcement during loading, facilitating better physical coupling when mounted, aligning the end-effector head when mounted (and / or when it is in a head retention device), or providing other features to the system.

[0075] In one structural element modification, the head selector and end effector head may include complementary alignment structures, as shown in Figure 3. The alignment structure may be a protruding or recessed feature on the mounting surface of the head selector and / or end effector. In one modification, the alignment structure is a groove or a tooth. The alignment structure may be used to restrict the way in which the head selector and end effector head are mounted. The set of head selector and end effector head may include one set or more pairs of alignment structures. The alignment structure may, in addition or alternatively, prevent rotation of the end effector head. In a similar manner, the alignment structure may allow torque to be transmitted through the coupling of the head selector and end effector head.

[0076] In another structural element modification, the modifiable end effector system may include a lateral support structure (50) integrated into one or both of the head selector and the end effector head. The lateral support structure provides structural support and functions to restrict rotation (e.g., rotation about an axis perpendicular to a defined central axis of the end arm section). Preferably, the lateral support structure provides support when the end effector is positioned horizontally and holding an object. The lateral support structure can prevent or mitigate situations in which the torque applied when gripping an object causes the end effector head to detach.

[0077] The lateral support structure (50) may be an extending structural piece having a form that engages with the surface of the head selector and / or end arm section. The lateral support structure may be located on one or both of the head selector and the end effector head (4). Preferably, the complementary lateral support structure is part of the body of the head selector and the end effector arm. In one modification, the complementary lateral support structures of the end effector and the head selector engage in a complementary manner when connected as shown in Figure 4.

[0078] A single lateral support structure may exist. Using a single lateral support structure, the robotic system can actively position the lateral support structure along the principal axis, benefiting from the lateral support when moving an object. In this modification, the robotic system may include position tracking and planning configurations for appropriately picking up an object and orienting the end effector head so that the lateral support is appropriately positioned to provide the desired support. In some cases, this may be used solely for selecting an object (e.g., a large and / or heavy object). In another modification, a set of lateral support structures may exist. The set of lateral support structures may be positioned around the periphery so that some degree of lateral support is provided regardless of the rotational orientation of the end effector head. For example, there may be three or four lateral support structures uniformly distributed around the periphery. In yet another modification, a continuous support structure may exist surrounding the edge of the end effector section.

[0079] A head holder or instrument holder (38) device functions to hold an end effector head when not in use. In one modification, the holder is a rack with a defined set of open slots that can hold multiple end effector heads. In one implementation, the holder includes slots that are open so that an end effector head can slide into the slot. The holder slots can also engage around the tapered portion of the end effector head so that a robotic system can pull the head selector perpendicularly to engage and disengage the current end effector head. Conversely, when selecting a new end effector head, the actuation system can move the head selector to an approximate position around the opening of the end effector head, allowing the end effector head to slide out of the holder slot, and a magnetic element can pull the end effector head onto the head selector.

[0080] The head retainer device may include a feed structure that, when engaged, moves the end effector head to a desired position. This can be used when a feature of the modifiable end effector system requires that the orientation of the end effector be in a known position.

[0081] The sensing system functions to collect data on objects and the environment. The sensing system preferably includes an imaging system that functions to collect image data. The imaging system preferably includes at least one imaging device (10) with a field of view within a first region. The first region may be a location where object interaction is expected. The imaging system may also include a plurality of imaging devices (12, 14, 16, 18), such as digital camera sensors, used to collect image data from a plurality of viewpoints of distinctly different regions, overlapping regions, and / or distinctly different non-overlapping regions. The set of imaging devices (e.g., one or more imaging devices) may include visual imaging devices (e.g., cameras). The set of imaging devices may also include, or alternatively, other types of imaging devices, such as depth cameras. Other suitable types of imaging devices may also be used, or alternatively.

[0082] The imaging system preferably captures an overhead or aerial view of the location from which the object will be initially positioned and moved. More generally, the image data collected is from the general direction from which the robotic system will approach and grasp the object. In one variation, the collection of objects presented for processing is presented in a substantially unorganized collection. For example, the collection of various objects may be temporarily stored in a box or transport container (in a stack and / or in an unorganized bundle). In another variation, the objects may be presented in a substantially organized or systematic manner. In one variation, the objects may be placed on a constructed conveyor that moves within the range of the robotic system. In this variation, the objects may be substantially separate from adjacent objects so that each object can be handled individually.

[0083] The system preferably includes a grip planning processing pipeline (6) used to determine a method for gripping an object from a set of objects and, optionally, an instrument to be used for gripping the object. The processing pipeline may use heuristic models, conditional checks, statistical models, machine learning or other data-based modeling, and / or other processes. In one preferred modification, the pipeline includes an image data segmenter and a grip quality model is used to generate an initial set of candidate grip plans, and then a grip plan selection process or multiple processes using the set of candidate grip plans.

[0084] Image data segmenters can segment image data and generate one or more image masks. The set of image masks may include object masks, object collection masks (e.g., segmenting multiple containers, carriers, shelves, etc.), object feature masks (e.g., barcode masks), and / or other suitable types of masks. The image masks can be used within a grip quality model and / or grip plan selection process.

[0085] A grip quality model functions to transform image data and, optionally, other input data into an output set of candidate grip plans. The grip quality model may include parameters of deep neural networks, support vector machines, random forests, and / or other machine learning models. In one variation, training a grip quality model may involve, or be, a convolutional neural network (CNN). The parameters of the grip quality model will generally be optimized to substantially maximize (or otherwise improve) performance on a training dataset that may include a set of images, grip plans for a set of points on the images, and grip results (e.g., success or failure) for those grip plans.

[0086] In one exemplary implementation, a grip quality CNN is a model that, given image data (e.g., visual or depth) as input, is trained to output tensors / vectors characterizing specific instruments, orientations (position and / or orientation for gripping), and success probabilities. A grip planning model and / or additional processing models may further integrate modeling of object selection order, material-based instrument selection, and / or other determinants.

[0087] The training dataset may include real or synthetic images that are manually or automatically labeled. In one variation, simulated reality transfer learning can be used to train a grasp quality model. Synthetic images may be created by generating a virtual scene in the simulation using a database of thousands of 3D object models with randomized textures, and rendering a virtual image of the scene using a graphics technique.

[0088] The grip plan selection process preferably assesses a set of candidate grip plans from a grip quality model and selects a grip plan for execution. Preferably, a single grip plan is selected, but in some variations, such as when there are multiple robotic systems operating simultaneously, multiple grip plans may be selected and executed in coordination to avoid interference. If some of the top-tier candidate grip plans are for an instrument that is not currently installed, the grip plan selection process may assess the success probability of the top-tier candidate grip plans and evaluate the time impact of changing the instrument.

[0089] In some modifications, the system may include a workstation configuration module. The workstation configuration module may be software, implemented as machine-interpretable instructions stored on a data storage medium, which, when implemented by one or more computer processors, causes the workstation configuration to output a user interface instructing the definition of annular conditions. The configuration device may be mounted as an end-effector and used to mark and locate the coordinates of important features of various environmental objects.

[0090] The system may also include API interfaces to various environmental implementation systems. The system may include API interfaces to external systems such as warehouse management systems (WMS), warehouse control systems (WCS), warehouse execution systems (WES), and / or any other suitable systems, which may be used to receive commands and / or information regarding the location and identification of objects. In another variation, there may be API interfaces to various order requests, which may be used to determine how to pack assemblies of products into boxes for different orders. [[******]]

[0091] Referring to Figures 7A-7F, various aspects of a robotic package handling configuration are illustrated. Referring to Figure 7A, a central frame (64) with multiple elements may be used to connect various components such as a robotic arm (54), a place structure (56), a pick structure (62), and a computing enclosure (60). As described in the incorporated references above, a movable component (58) of the place structure may be used to capture items from the place structure (56) and deliver them to various other locations within the system (52). Figure 7B illustrates a closer view of an embodiment of the system (52) in which the illustrated pick structure (62) comprises a container defining a package-containing volume bounded by a bottom and multiple walls, and, in addition to being visible by an imaging device (66), can define an open access opening to accommodate the entry and exit of a portion of the robotic arm. In other embodiments, the pick structure may comprise a fixed surface such as a table, a movable surface such as a conveyor belt system, or a tray. Referring to Figure 7C, the system may comprise multiple imaging devices configured to capture images of various aspects of the operation. Such imaging devices may comprise monochrome, grayscale, or color devices, and may comprise depth camera devices such as those marketed by Intel Corporation under the trademark RealSense (RTM). A first imaging device (66) may be fixedly coupled to an element of the frame (64), as shown in Figure 7C, and may be positioned and oriented to capture images with its field of view (80) oriented downward within the pick structure (62). A second imaging device (66) may be coupled to an element of the frame (64) and may be positioned and oriented to capture image information relating to the end effector (4) of the robot arm (54), and image information relating to the captured or grasped package, which can be detachably coupled to the end effector (4) after successful grasping. Such image information may be used to estimate the outer dimensional boundaries of the item or package being held by fitting a 3D rectangular prism around the targeted package and estimating the length-width-height (LWH) of the rectangular prism.A 3D rectangular prism is used to estimate the position and orientation of the targeted package relative to the end effector. The imaging device may be automatically triggered by an interconnected computing system (60). The computing system may be configured to estimate whether the targeted package is deformable by analyzing the deformation of the targeted package in the sequence of images, such as by capturing a sequence of images of the targeted package during motion of the targeted package and observing the motion within the image region of the package during motion or acceleration of the package by the robotic arm (i.e., rigid packages generally have regions that move in harmony, while flexible packages may have regions that do not move in harmony with acceleration and motion). As shown in Figures 7C and 7D, various additional imaging devices (74, 76, 78) may be positioned and oriented to provide a field of view (84, 86, 88) which may be useful in observing the activity of the robotic arm (54) and the associated package.

[0092] Referring to Figure 7E, a vacuum load source (90), such as a pressurized air or gas source, may be controllably circulated through a venturi configuration (by an electromechanically controllable input valve, etc., operably coupled to a computing system with integrated pressure and / or velocity sensors for closed-loop control), and operably coupled to an end effector assembly (via a conduit, etc.) to produce a controlled vacuum load for a suction cup assembly and a suction-based end effector (4).

[0093] Figure 7F shows a closer view of a robotic arm (54) with an end effector assembly (24) comprising two suction cup assemblies (26, 28) configured to assist in gripping a package, as further described in the incorporated references mentioned above. Referring to Figures 8A, 8B, and 7G, one embodiment of the suction cup assembly (26) is illustrated, showing a vacuum coupling (104) coupled to an outer housing (92), which may have a bellows structure comprising a plurality of foldable wall portions joined at the bent edges, and such a bellows structure may include a material selected from the group consisting of polyethylene, polypropylene, rubber, and thermoplastic elastomers. The interconnected inner internal structure (94) may comprise a proximal base member (112) that can define a wall member (114), such as a substantially cylindrical wall member as shown, and a plurality of inlet openings (102) passing through it, which may further comprise a distal wall member (116) that defines an inner structural opening ring portion, a plurality of transition air channels (108), and an outer seal edge member (96), which may further define an inner chamber (100). The gap (106) may be defined between the outer housing member (92) and the internal structure (94) such that a vacuum from a vacuum source draws air through the inner chamber (100) and associated inlet openings (102) and transition air channels using a defined path configured to assist in gripping, and generally tends to prevent excessive protrusion of a package surface using a non-flexible package.

[0094] Referring to Figures 9A and 9B, as described in the aforementioned incorporated references, the system may be configured to ensure a relatively reliable grip using the flexible package, to the extent that the flexible portion (122) is pulled upward into the inner chamber (100), and the package portion (122) is at least partially enclosed as shown in Figure 9B.

[0095] Referring to Figures 10A–10F, as described above, the place structure (56) may include a component (58) that is rotatably and / or detachably coupled to the remainder of the place structure (56) and can assist in the distribution of articles from the place structure (56). As shown in Figure 10C, the place structure (56) may include a grill-like or detached surface configuration (128) with a retaining inclined portion (132) configured to adapt to rotatable and / or detachable engagements of complementary components (58), such as those shown in Figure 10D, which may have branched or detached configurations (126) for engaging with other place structure components (56). Figure 10F schematically illustrates a side view of a movable and rotatable engagement between structures (56, 58) as described in the incorporated references above.

[0096] Referring to the system (52) configuration in Figure 11A, a computing system such as a VLSI computer may be housed within a computing system enclosure structure (60), as described above. Figure 11B illustrates an overview of the system in Figure 11A, which includes an enclosure shown as transparent to illustrate the computing system (134) coupled inside. Referring to Figure 11C, in other embodiments, additional computing resources may be operablely coupled (142, 144, 146) (by fixed network connectivity or wireless connectivity such as a configuration under the IEEE 802.11 standard), for example, the system may include an additional VLSI computer (136) and / or some cloud computing-based computing resource (138) that may be located in one or more remote / non-local (148) locations.

[0097] Referring to Figure 12, this is an exemplary computer architecture diagram of one implementation of the system. In some implementations, the system is implemented within multiple devices that communicate over communication channels and / or networks. In some implementations, the elements of the system are implemented within separate computing devices. In some implementations, two or more of the system elements are implemented within the same device. The system and parts of the system may be integrated into computing devices or systems that can function as a system or within it.

[0098] The communication channel 1001 interfaces with processors 1002A-1002N, memory (e.g., random access memory (RAM)) 1003, read-only memory (ROM) 1004, processor-readable storage medium 1005, display device 1006, user input device 1007, and network device 1008. As shown, the computer infrastructure may be used to connect the robot system 1101, sensor system 1102, grip planning pipeline 1103, and / or other suitable computing devices.

[0099] Processor 1002A-1002N can take many forms, including CPU (Central Processing Unit), GPU (Graphical Processing Unit), microprocessor, ML / DL (Machine Learning / Deep Learning) processing unit such as tensor processing unit, FPGA (Field Programmable Gate Array), custom processor, and / or any suitable type of processor.

[0100] Processors 1002A-1002N and main memory 1003 (or some secondary combination) can form a processing unit 1010. In some embodiments, the processing unit includes one or more processors communicatively coupled to one or more of RAM, ROM, and machine-readable storage media, one or more processors of the processing unit receiving instructions via a bus stored by one or more of RAM, ROM, and machine-readable storage media, and one or more processors executing the received instructions. In some embodiments, the processing unit is an ASIC (Application-Specific Integrated Circuit). In some embodiments, the processing unit is a SoC (System on a Chip). In some embodiments, the processing unit includes one or more of the elements of the System.

[0101] The network device 1008 may provide one or more wired or wireless interfaces for exchanging data and commands between this system and / or other devices such as devices in external systems. Such wired and wireless interfaces include, for example, a Universal Serial Bus (USB) interface, a Bluetooth® interface, a Wi-Fi interface, an Ethernet® interface, a Near Field Communication (NFC) interface, and equivalents.

[0102] Computer and / or machine-readable executable instructions, comprising a configuration for software programs (such as an operating system, application programs, and device drivers), can be stored in memory 1003 from a processor-readable storage medium 1005, ROM 1004, or any other data storage system.

[0103] When executed by one or more computer processors, individual machine-executable instructions may be accessed via communication channel 1001 by at least one of processors 1002A-1002N (of processing unit 1010) and then executed by at least one of processors 1002A-1002N. Data, databases, data records, or other forms of data to be stored, created or used by software programs, may also be stored in memory 1003, and such data may be accessed by at least one of processors 1002A-1002N during the execution of machine-executable instructions of software programs.

[0104] The processor-readable storage medium 1005 is one of the following: a hard drive, a flash drive, a DVD, a CD, an optical disc, a floppy disk, a flash memory device, a solid-state drive, a ROM, an EEPROM, an electronic circuit, a semiconductor memory device, and equivalents (or a combination of two or more of these). The processor-readable storage medium 1005 may include an operating system, a software program, a device driver, and / or other suitable subsystem or software.

[0105] As used herein, “first,” “second,” “third,” etc., are used to characterize and distinguish various elements, components, regions, layers, and / or divisions. These elements, components, regions, layers, and / or divisions should not be limited by these terms. The use of numerical terms may be used to distinguish one element, component, region, layer, and / or division from another element, component, region, layer, and / or division. The use of such numerical terms does not imply a sequence or order unless explicitly indicated by the context. Such numerical references may be used synonymously without departing from the teachings of embodiments and modifications herein. [[******]]

[0106] As shown in Figure 13, a method for planning and adapting object manipulation by a robotic system may include planning a grip (S200), which consists of collecting image data of the object acquisition area (S110), evaluating the image data through a grip quality model to generate a set of candidate grip plans (S210), processing the candidate grip plans and selecting a grip plan (S220), executing the selected grip plan using the robotic system (S310), and performing the object interaction task (S320). The grip quality model preferably integrates grip quality across different sets of robotic instruments, and therefore, the selection of a grip plan can trigger a change in the instrument. With respect to a pick-and-place robot, this may include changing the end effector head based on the selected grip plan.

[0107] In the more detailed implementation shown in Figure 14, the method may include planning a grip (S200), which includes training a grip quality model (S120), configuring a robot system work station (S130), receiving an object interaction task request (S140), triggering the collection of image data of an object acquisition region (S110), segmenting the image data into a focus region mask (S202), evaluating the image data through the grip quality model and generating a set of candidate grip plans (S210), processing the candidate grip plans and selecting a grip plan (S220), executing the selected grip plan using the robot system (S310), and executing the object interaction task (S320).

[0108] This method may be implemented by a system such as the system described herein, but it may also be implemented by any suitable alternative system.

[0109] In one variation, the method may include training a grip quality convolutional neural network (S120) which functions to build a data-based model for scoring different grip plans on a given set of image data.

[0110] The grip quality model may include parameters for deep neural networks, support vector machines, random forests, and / or other machine learning models. In one variation, training the grip quality model may include, or does include, a convolutional neural network (CNN). The parameters of the grip quality model will generally be optimized to substantially maximize (or otherwise improve) performance on a training dataset, which may include a set of images, grip plans for a set of points on the images, and grip results (e.g., success or failure) for those grip plans.

[0111] In one exemplary implementation, a grip quality CNN is trained so that, given image data (e.g., visual or depth) as input, the model can output tensors / vectors characterizing a specific instrument, posture (position and / or orientation for achieving a grip), and probability of success.

[0112] The training dataset may include real or synthetic images that are manually or automatically labeled. In one variation, simulated reality transfer learning can be used to train a grasp quality model. Synthetic images may be created by generating a virtual scene within the simulation using a database of thousands of 3D object models with randomized textures, and rendering a virtual image of the scene using a graphics technique.

[0113] The grip quality model may also integrate other features or grip planning scores into the model. In one variation, the grip quality model integrates object selection order into the model. For example, a CNN can be trained using the above metrics, but it may also prioritize the selection of larger objects to reveal smaller objects directly below, potentially revealing other grip points with a higher probability. In other variations, various algorithmic heuristic methods or processes can be integrated to take into account object size, object material, object features such as barcodes, or other features.

[0114] During the execution of this method, the grip quality model may also be updated and refined as object image data is collected, the grip plan is executed, and the object interaction results are determined. In some modifications, a grip quality model may be provided in which training and / or updating the grip quality model is not performed by the entity performing this method.

[0115] In one modification, the method may include configuring a robot system workspace (S130) which functions to set up a robot system workspace for operation. Configuring a robot system workspace preferably involves configuring the setting up of environmental features for the robot system. For example, in a warehouse embodiment, configuring a robot system workspace involves setting the coordinate locations of a wall, a set of shelves, a box, an output bagger, a conveyor belt, or other area where objects may be located or will be installed.

[0116] In one modification, the robot system may be configured to accept manual operation of a component used as an end-effector to define various geometric shapes. A user interface may preferably guide the user through the process. For example, a set of standard environmental objects may be presented in a menu within the user interface. After selecting an object, commands may be presented to guide the user through a set of measurements to be performed using the component end-effector.

[0117] The configuration may also define the properties of defined objects within the environment. This can provide useful information for avoiding collisions, defining how to plan movement in different areas, and interacting with objects based on relevant environmental objects. Environmental objects may be defined as static to indicate that they do not move. Environmental objects may be defined as movable. For some movable environmental objects, the areas in which they are expected to appear may also be defined. For example, a robot system workstation may be configured to understand the general areas in which a box of objects may appear, and the expected dimensions of the box. Various object-specific characteristics, such as the size and dimensions of movable parts (e.g., doors, box flaps), can also be configured. For example, in addition to the conveyor path, the position of the conveyor can also be configured. The robot system may also be integrated with a suitable API to have data regarding the conveyor state.

[0118] In one modification, the method may include receiving an object interaction task request (S140), which functions to have several signal-initiated object interactions by the robotic system. The request may specify the location of an object, more typically, the location of a collection of objects. The request may also specify an action to be taken on the object, either by supplying a command or otherwise. The object interaction task request may be received through an API. In one implementation, an external system such as a warehouse management system (WMS), warehouse control system (WCS), warehouse execution system (WES), and / or any preferred system may be used to direct the interactions, such as specifying the transport boxes to be used for picking the objects.

[0119] In one variation, the method may include receiving one or more requests. The requests may be shaped around an intended use case. In one embodiment, the request may be an order request defining a group of sets of objects. The objects defined within the order request will generally need to be tied together, packaged, or otherwise grouped together for further order processing. The selection of objects may be based, at least in part, on the set of requests, the priority of the requests, and the planned fulfillment of these orders. For example, an order with two objects, which may be selected from one or more containers with high reliability, may be selected for picking and installation of the objects by the system prior to objects from order requests where the objects are not identified or have lower reliability in picking ability at this point.

[0120] Block S110, which includes collecting image data of an object acquisition area, functions to observe and sense objects to be handled by a robotic system for processing. In some use cases, the set of objects may include one or more types of products. Collecting image data preferably includes collecting visual image data using a camera system. In one variation, a single camera may be used. In another variation, multiple cameras may be used. Collecting image data may also include, or alternatively, collecting depth image data or other forms of 2D or 3D data from a specific area.

[0121] In one preferred implementation, image data collection involves capturing image data from an overhead or aerial viewpoint. More generally, image data is collected from a general direction from which the robotic system would approach and grasp an object. Image data is preferably collected in response to some signal, such as an object interaction task request. Alternatively, image data may be processed continuously or periodically to automatically detect when action should be taken.

[0122] Block S200, which includes grip planning, functions to determine the object to be grasped, the method for grasping the object, and optionally, the instrument to be used. Grip planning can utilize a grip planning model in densely generating different grip options and scoring them based on reliability and / or other metrics. In one variation, grip planning may include segmenting image data into a region mask of interest (S202), evaluating the image data through a neural network architecture to generate a set of candidate grip plans (S210), and processing the candidate grip plans to select one (S220). Preferably, the modeling used in grip planning attempts to increase object interaction throughput. This can function to address the challenge of weighing the probability of success with the current instrument against the time cost of switching to an instrument with a higher probability of success.

[0123] Block S202, which includes segmenting image data into a region of interest mask, functions to generate masks used in evaluating the image data in block S210. Preferably, one or more segmentation masks are generated from the supplied image data input. Segmenting image data may include segmenting image data into an object mask. Segmenting image data may also include, or alternatively, segmenting image data into an object collection (e.g., segmenting onto a transport box, container, shelf, etc.). Segmenting image data may also include, or alternatively, segmenting image data into an object feature mask. An object feature mask may be used to segment detected or predicted object features such as barcodes or other object elements. There are some use cases where it is desirable to avoid gripping on specific features, or to endeavor to grip on specific features.

[0124] Block S210, which includes evaluating image data through a grip quality model and generating a set of candidate grip plans, functions to output a set of grip options from a set of input data. The image data is preferably one input into the grip quality model. One or more segmentation masks from block S202 may also be supplied as input. Alternatively, the segmentation masks may be used to exclude or select segments of the image data from which candidate grips should be evaluated.

[0125] Preferably, evaluating image data through a grip quality model includes evaluating image data through a grip quality CNN architecture. The grip quality CNN can densely predict, for multiple locations in the image data, the grip quality for each instrument and the probability of success if a grip were to be performed. The output is preferably a tensor / vector map characterizing the instrument, orientation (position and / or orientation for aligning the grip), and probability of success.

[0126] As described above, the grip quality CNN may also model the selection order of objects, and therefore the output may also score the grip plan according to the training data that reflects the object order. In another variation, object material planning can be integrated into the grip quality CNN as an additional planning model used in determining grips. The material planning process may classify image data as a map for handling a collection of objects of different materials. Processing image data using the material planning process may be used in the selection of new equipment. For example, if the material planning model represents an object packaged in multiple plastic bags, the equipment change may be triggered based on the classified material properties from the material model.

[0127] Block S220, which includes processing candidate gripping plans and selecting a gripping plan, functions to apply various heuristic methods and / or modeling in prioritizing and / or selecting candidate gripping plans. The output of the gripping quality model is preferably fed into a subsequent processing stage in which different factors are compared. A subset of candidate gripping plans with a high probability of success may be evaluated. Alternatively, all gripping plans may be processed in S220.

[0128] Part of selecting a candidate gripping plan involves choosing a gripping plan based, at least partially, on the time cost of changing instruments and the change in the probability of a successful grip. This can be considered in relation to the current state of the object, but can also be considered across past and potential future activities. In one preferred modification, the current instrument state and gripping history (e.g., a history of successful grips with a given instrument) can be supplied as input. For example, if there have been multiple failures with a given instrument, this may signal the selection of a gripping plan using a different instrument. When processing candidate gripping plans, there may be a bias to keep the same instrument. Changing instruments takes time, and therefore the change in the probability of a successful grip is compared with the time cost of changing instruments.

[0129] Several additional heuristic techniques, such as collision checks, feature avoidance, and other grip detection criteria, can be assessed when planning grips. In variations of multi-head end-effector devices, collision checks may also potentially consider collisions and obstacles, which are taken into account by the end-effector heads when not in use.

[0130] Block S310, which includes using a robotic system to implement a selected gripping plan, functions to control the robotic system to grip an object in a manner defined within the selected gripping plan.

[0131] Since grip plans are preferably associated with different instruments, implementing a selected grip plan using the instrument indicated in the grip plan may include selecting and / or changing the instrument.

[0132] In a modified multi-head end-effector apparatus, the apparatus (or multiple apparatuses) shown may be appropriately activated or used as target points for alignment with an object. Since the end-effector head may be offset from the central axis of the end-arm section, the motion planning of the actuation system preferably modifies the actuation to appropriately align the correct head to the desired position.

[0133] In the case of a changeable instrument modification, if the current instrument differs from the instrument in the selected gripping plan, the robotic system uses the instrument change system to change the instrument and then executes the gripping plan. If the current instrument is identical to the instrument shown in the selected gripping plan, the robotic system moves directly to execute the gripping plan.

[0134] When implementing a gripping plan, the operating system moves the instrument (e.g., an end-effector suction head) to a fixed position and performs the gripping action. In the case of a pressure-based pick-and-place machine, performing the gripping action includes activating the pressure system. During the gripping, the instrument of the robotic system (i.e., the end-effector) will bond with the object. The object can then be moved and manipulated for subsequent interactions. Depending on the type of robotic system and end-effector, gripping can be performed through various gripping mechanisms and / or end-effectors.

[0135] If no suitable gripping plan is identified in block S200, the method may include gripping and reorienting the object to present other gripping plan options. After reorientation, the object's field of view can be re-evaluated to detect a suitable gripping plan. In some cases, multiple objects may be reoriented. In addition, or alternatively, the robotic system may be configured to disturb the collection of objects to perturb the positions of multiple objects with the aim of revealing a suitable gripping point.

[0136] Once an object is grasped, it is preferably extracted from the set of objects and then translated to a different position and / or orientation, which serves to move and orient the object for the next stage.

[0137] If, after executing a grip plan (for example, when gripping an object or during an object interaction task), the object is dropped or otherwise released from the robotic system, a failure can be recorded. This data can be further used to update the system, which may include re-evaluating the object collection for a new grip plan. Similarly, data records regarding grip successes can also be used to update the system, grip quality modeling, and other grip planning processes.

[0138] Block S320, which includes performing an object interaction task, functions to perform arbitrary object manipulation using a robotic system with a grasped object. The object interaction task may involve placing the object within a target destination (e.g., within another container or box), changing the orientation of the object prior to placing it, moving the object for a certain object action (e.g., barcode scanning), and / or performing any preferred action or set of actions. In one embodiment, performing an object interaction task may involve scanning a barcode or other identification marker on the object to detect the object identifier, and then placing the object within the destination location based on the object identifier. When used in a facility used to fulfill shipping orders, a product ID obtained using barcode information is used to look up the corresponding order, then to determine the container to map to that order, and the object can then be placed within that container. When performed repeatedly, multiple products for an order can be packed in the same container. In other applications, other preferred subsequent steps may be performed. A failure to grasp during an object interaction task may result in re-grasping the object and / or returning it to the assembly of objects for the planning and execution of a new object interaction. Re-grasping the object may involve a modified grasp planning process that focuses on a single object at the site where the dropped object fell. [ ]

[0139] Referring to Figures 15-19, various method configurations are illustrated. Referring to Figure 15, one embodiment provides a robot arm comprising a distal portion and a proximal base portion; an end effector coupled to the distal portion of the robot arm; a place structure positioned geometrically close to the distal portion of the robot arm; a pick structure positioned geometrically close to the distal portion of the robot arm and in contact with one or more packages; a first imaging device positioned and oriented to capture image information relating to the pick structure and one or more packages; and a first computing system operably coupled to the robot arm and the first imaging device, configured to receive image information from the first imaging device and to command the movement of the robot arm at least partially based on the image information, wherein the end effector is the (402) A first suction cup assembly is coupled to a controllably activated vacuum load which is operably coupled to a computing system, the first suction cup assembly defines a first internal capture chamber, and the first computing system is used to operate a robotic arm and an end effector to grasp a targeted package from one or more packages from a pick structure, release the targeted package and place it on a place structure, wherein grasping the targeted package includes, when the vacuum load is controllably activated adjacent to the targeted package, pulling a portion of the targeted package into the first internal capture chamber and using it to at least partially seal it.

[0140] Referring to Figure 16, one embodiment provides a robot arm comprising a distal portion and a proximal base portion; an end effector coupled to the distal portion of the robot arm; a place structure positioned geometrically close to the distal portion of the robot arm; a pick structure positioned geometrically close to the distal portion of the robot arm and in contact with one or more packages; a first imaging device positioned and oriented to capture image information relating to the pick structure and one or more packages; and a first computing system operably coupled to the robot arm and the first imaging device and configured to receive image information from the first imaging device and to command the movement of the robot arm at least partially based on the image information (408); and utilizing the first computing system to capture one or more packages from the pick structure The method involves operating a robotic arm and an end effector to grasp a targeted package, release the targeted package, and place it on a place structure, the end effector comprising a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing device, the first suction cup assembly collectively defining a first inner chamber, a first outer seal edge, and a first vacuum permeable distal wall member, wherein, in response to grasping the targeted package using the controllably activated vacuum load, the outer seal edge may be in a state where it is removably coupled to at least one surface of the targeted package, while a vacuum permeable distal wall member is configured to prevent excessive protrusion of the surface of the targeted package into the inner chamber of the suction cup assembly.

[0141] Referring to Figure 17, one embodiment provides a robot arm comprising a distal portion and a proximal base portion; an end effector coupled to the distal portion of the robot arm; a place structure positioned geometrically close to the distal portion of the robot arm; a pick structure positioned geometrically close to the distal portion of the robot arm and in contact with one or more packages; a first imaging device positioned and oriented to capture image information relating to the pick structure and one or more packages; and a first computing system operably coupled to the robot arm and the first imaging device, configured to receive image information from the first imaging device and to command the movement of the robot arm at least partially based on the image information (414); and using the first computing system to grasp a targeted package from one or more packages from the pick structure, release the targeted package, and place it on the place structure. (416) The invention relates to operating a robotic arm and an end effector, the end effector comprising a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing system, the first suction cup assembly being configured to perform a grip, which includes engaging with a targeted package when the vacuum load is controllably activated adjacent to the targeted package, prior to performing a grip, the computing device being configured to analyze a plurality of candidate grips and, at least in part, based on the use of startup times of a neural network operated by the computing device, select a run grip to be performed to remove the targeted package from the pick structure, the neural network being trained using views unfolded from synthetic data, including rendered images of three-dimensional models of one or more synthetic packages such as those contained by the synthetic pick structure.

[0142] Referring to Figure 18, one embodiment provides a robot arm comprising a distal portion and a proximal base portion; an end effector coupled to the distal portion of the robot arm; a place structure positioned geometrically close to the distal portion of the robot arm; a pick structure positioned geometrically close to the distal portion of the robot arm and in contact with one or more packages; a first imaging device positioned and oriented to capture image information relating to the pick structure and one or more packages; and a first computing system operably coupled to the robot arm and the first imaging device and configured to receive image information from the first imaging device and to command the movement of the robot arm at least partially based on the image information (420); and using the first computing system, the robot arm and end effector to grasp a targeted package from the pick structure, release the targeted package, and place it on the place structure. (422) To operate an end effector, the end effector comprises a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to a first computing system, wherein the first suction cup assembly is configured to perform a grip, which includes engaging with the targeted package when the vacuum load is controllably activated adjacent to the targeted package; (424) To operate an end effector, the end effector comprises a first suction cup assembly coupled to a first computing system, wherein the end effector comprises a first suction cup assembly coupled to a first suction cup assembly, which includes performing a grip, which includes engaging with the targeted package when the vacuum load is controllably activated adjacent to the targeted package; and to provide a second imaging device operably coupled to a first computing system, which is positioned and oriented to capture an image of one or more of the targeted packages after a grip has been performed using the end effector, to fit a 3D rectangular prism around the targeted package, to estimate the outer dimensional boundary of the targeted package by estimating the LWH of the rectangular prism, and to utilize the fitted 3D rectangular prism to estimate the position and orientation of the targeted package relative to the end effector; and to utilize the first computing system to process the targeted package.This includes (426) operating the robotic arm and end effector to position them on the place structure in a specific location and orientation relative to the place structure.

[0143] Referring to Figure 19, one embodiment includes collecting image data relating to the acquisition region (430), planning a grip (432) which consists of evaluating the image data through a grip quality model and generating a set of candidate grip plans, processing the candidate grip plans and selecting a grip plan, executing the selected grip plan using a robotic system (434), and performing an object interaction task (436). [***xxxxxx]

[0144] Referring to Figures 20A and 20B, two synthetic training images (152, 154) are shown, each featuring a synthetic pick structure container (156, 158) containing multiple synthetic packages (160, 162). The synthetic volume may be created and used to rapidly train a neural network to facilitate the automated movement of a robotic arm in picking targeted packaging materials from the pick structure and placing them on the place structure, generating a large number of synthetic image data, such as those shown in Figures 20A and 20B. The view may be created from multiple viewing vectors and positions, and the synthetic volume may be varied accordingly. For example, a neural network may be trained using a view unfolded from synthetic data, which includes rendered color images of 3D models of one or more synthetic packages contained by a synthetic pick structure; this may also be trained using a view unfolded from synthetic data, which includes rendered depth images of 3D models of one or more synthetic packages contained by a synthetic pick structure; this may also be trained using a view unfolded from synthetic data, which includes rendered images of 3D models of one or more randomized synthetic packages contained by a synthetic pick structure; this may also be trained using synthetic data in which the synthetic packages are randomized by a color texture; furthermore, this may also be trained using synthetic data in which the synthetic packages are randomized by a physically based rendering mapping selected from the group consisting of reflection, diffusion, translucency, transparency, metallicity, and micro-surface scattering; furthermore, a neural network may be trained using a view unfolded from synthetic data, which includes rendered images of 3D models of one or more synthetic packages at random positions and orientations, as contained by a synthetic pick structure.

[0145] The first computing system may be configured such that performing a grip includes analyzing a plurality of candidate grips and selecting a perform grip to be performed to remove the targeted package from the pick structure. Analyzing the plurality of candidate grips may include scrutinizing locations on the targeted package where the first suction cup assembly is expected to be able to form a seal engagement with the surface of the targeted package. Analyzing the plurality of candidate grips may include scrutinizing locations on the targeted package where the first suction cup assembly is expected to be able to form a seal engagement with the surface of the targeted package from a plurality of different end-effector approach orientations. Analyzing the plurality of candidate grips includes scrutinizing locations on the targeted package where the first suction cup assembly is expected to be able to form a seal engagement with the surface of the targeted package from a plurality of different end-effector approach positions. The first suction cup assembly may have a first outer seal rim, and the seal engagement with the surface includes a substantially complete engagement between the first outer seal rim and the surface. The location on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement with the surface of the targeted package may be examined in a purely geometric manner. The first computing system may be configured to select an executed grip based on candidate grip factors selected from a group consisting of estimated time required, estimated requirement calculation, and estimated grip success rate.

[0146] The system may be configured such that a single neural network can predict gripping for multiple types of end-effectors or instrument configurations (i.e., various combinations of several suction cup assemblies, and various vectors of approaches). Specifically, the system may be configured not to analyze torque and load on the robot arm or other components for the targeted package for the sake of system processing speed (i.e., in various embodiments, such as for packages for mailing, it may be desirable to prioritize speed over torque or load-based analysis).

[0147] As described above, in various embodiments, in order to randomize the visual appearance of an object within the training data being synthesized / simulated, the system may be configured to randomize several properties used to construct a visual representation (including, but not limited to, a color texture which may include base red / green / blue values ​​which can be applied to a 3D model, and, but not limited to, a physically based rendering map which may be available which may be applied to a surface which may include reflection, diffusion, translucency, transparency, metallicity, and / or micro-surface scattering). "***********"

[0148] Various exemplary embodiments of the present invention are described herein. These embodiments are used by reference in a non-limiting sense. They are provided to illustrate broader applicable aspects of the present invention. Various modifications may be made to the invention described, and equivalents may be substituted without departing from the true spirit and scope of the invention. In addition, many modifications may be made to adapt specific situations, materials, composition of substances, processes, process actions, or steps to the object, spirit, or scope of the invention. Furthermore, as will be understood by those skilled in the art, each individual modification described and illustrated herein has discrete components and features that can be readily separated from or combined with any of the features of several other embodiments without departing from the scope or spirit of the invention. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[0149] The present invention includes methods that can be carried out using subject devices. These methods may include the act of providing such suitable devices. Such provision may be carried out by an end user. In other words, the act of “providing” simply requires the end user to acquire, access, approach, position, configure, activate, launch, or otherwise act to provide the device required in the subject method. The methods enumerated herein may be carried out in any logically possible order of the enumerated events, and in the enumerated order of the events.

[0150] Exemplary aspects of the present invention, along with details relating to the selection and manufacture of materials, are described above. Other details of the present invention are to be understood in connection with the patents and published documents referred to above, and are generally graspable or understandable to those skilled in the art. The same as that generally or theoretically adopted may also apply to the method-based aspects of the present invention in terms of additional actions.

[0151] In addition, while the present invention is described with reference to several embodiments that optionally incorporate various features, the present invention should not be limited to those described or shown as possible for each modification of the invention. Various modifications may be made to the described invention (whether listed herein or not included for certain brevity), and equivalents may be substituted without departing from the true spirit and scope of the invention. Furthermore, it should be understood that, where a range of values ​​is provided, all intervening values, i.e., those between the upper and lower limits of that range, and any other described or intervening values ​​within that described range, are encompassed within the present invention.

[0152] Furthermore, it should be assumed that any optional feature of a variation of the invention described may be described or claimed independently or in combination with any one or more of the features described herein. A reference to a single object includes the possibility that multiple identical articles exist. More specifically, as used herein and in the claims relating herein, the singular forms “a,” “an,” “said,” and “the” include multiple referents unless otherwise specifically stated. In other words, the use of articles allows for “at least one” of the subject articles in the above description and in the claims relating to this disclosure. Furthermore, it should be noted that such claims may be drafted to exclude any optional element. Thus, this statement is intended to serve as an antecedent for the use of such exclusive terms, or “negative” restrictions, such as “alone,” “only,” and equivalents, in connection with the enumeration of claim elements.

[0153] Without the use of such exclusive technical terms, the term “comprising” in the claims associated with this disclosure shall allow for the inclusion of any additional elements, regardless of whether a given number of elements are enumerated in such claims, or the addition of features may be considered as transforming the essence of the elements described in such claims. Unless specifically defined herein, all technical and scientific terms used herein should be given the broadest possible, generally understood meaning while maintaining the validity of the claims.

[0154] The scope of the present invention should not be limited to the provided examples and / or specification, but rather should be limited only to the scope of the claim language associated with this disclosure.

Claims

1. A robotic package handling system, a. A robotic arm comprising a distal portion and a proximal base portion, b. An end effector coupled to the distal portion of the robot arm, c. A place structure positioned geometrically in close proximity to the distal portion of the robot arm, d. A pick structure, wherein the pick structure is in contact with one or more packages and is positioned geometrically close to the distal portion of the robot arm, e. A first imaging device, wherein the first imaging device is positioned and oriented to capture image information relating to the pick structure and one or more packages, f. A first computing system, wherein the first computing system is operably coupled to the robot arm and the first imaging device, receives the image information from the first imaging device, and is configured to at least partially command the movement of the robot arm based on the image information. Equipped with, The first computing system is configured to operate the robotic arm and end effector to grasp a targeted package from the pick structure, release the targeted package, and place it on the place structure. The end effector comprises a first suction cup assembly coupled to a controllably activated vacuum load operably coupled to the first computing system, wherein the first suction cup assembly is configured to perform the gripping, which includes engaging with the targeted package when the vacuum load is controllably activated adjacent to the targeted package. The robotic package handling system further comprises a second imaging device, the second imaging device being operably coupled to the first computing system, and positioned and oriented to capture one or more images of the targeted package after the gripping is performed using the end effector, to estimate the outer dimensional boundary of the targeted package by fitting a 3D rectangular prism around the targeted package and estimating the L-W-H of the rectangular prism, and to estimate the position and orientation of the targeted package relative to the end effector using the fitted 3D rectangular prism, A robotic package handling system, wherein the first computing system is configured to operate the robotic arm and end effector to place the targeted package on the place structure in a specific position and orientation relative to the place structure.

2. The robot package handling system according to claim 1, further comprising a frame structure configured to fix and connect the robot arm to the place structure.

3. The robotic package handling system according to claim 1, wherein the placement structure comprises an installation tray.

4. The robotic package handling system according to claim 3, wherein the installation tray comprises first and second rotatably coupled members, the first and second rotatably coupled members are configured to form a substantially flat tray base surface when in a first rotatably coupled configuration relative to each other, and to form a lifting fork configuration when in a second rotatably coupled configuration relative to each other.

5. The robotic package handling system according to claim 1, wherein the pick structure comprises a container, a tray, a fixed surface, or a movable surface.

6. The robotic package handling system according to claim 5, wherein the pick structure comprises a container configured to define a package-containing volume bounded by a bottom and a plurality of walls, and an open access opening configured to accommodate the entry and exit of at least the distal portion of the robotic arm.

7. The robotic package handling system according to claim 6, wherein the first imaging device is configured to capture the image information relating to the pick structure and one or more packages through the open access opening.

8. The robot package handling system according to claim 1, wherein the first imaging device comprises a depth camera.

9. The robot package handling system according to claim 1, wherein the first imaging device is configured to capture color image data.

10. The robot package handling system according to claim 1, wherein the first computing system comprises a single VLSI computer.

11. The robot package handling system according to claim 1, wherein the first computing system constitutes a network of interconnected computing devices, and at least one of the interconnected computing devices is located remotely from the robot arm.

12. The robot package handling system according to claim 1, further comprising a second computing system operably coupled to the first computing system.

13. The robotic package handling system according to claim 12, wherein the second computing system is located remotely from the first computing system, and the first and second computing systems are operably coupled via a computer network.

14. The robotic package handling system according to claim 1, wherein the first computing system is configured to perform the gripping by analyzing a plurality of candidate grips and selecting an execution grip to be performed to remove the targeted package from the pick structure.

15. The robotic package handling system according to claim 14, comprising analyzing multiple candidate grips to examine locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package.

16. The robotic package handling system according to claim 15, comprising analyzing multiple candidate grips to examine locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package from multiple different end-effector approach orientations.

17. The robotic package handling system according to claim 15, comprising analyzing multiple candidate gripping positions to examine locations on the targeted package where it is predicted that the first suction cup assembly can form a sealing engagement portion with the surface of the targeted package from multiple different end-effector approach positions.

18. The robotic package handling system according to claim 15, wherein the first suction cup assembly comprises a first outer seal edge, and the seal engagement portion with the surface includes a substantially complete engagement portion with the first outer seal edge with the surface.

19. The robotic package handling system according to claim 15, wherein the examination of locations on the targeted package where the first suction cup assembly is expected to be able to form a sealing engagement portion with the surface of the targeted package is performed in a purely geometric manner.

20. The robot package handling system according to claim 14, wherein the first computing system is configured to select the execution grip based on candidate grip factors selected from a group consisting of estimated required time, estimated required calculation, and estimated success rate of gripping.

21. The robotic package handling system according to claim 1, wherein the first suction cup assembly comprises a bellows structure.

22. The robot package handling system according to claim 21, wherein the bellows structure comprises a plurality of wall portions adjacent to and connected to the bent edge.

23. The robotic package handling system according to claim 22, wherein the bellows structure comprises a material selected from the group consisting of polyethylene, polypropylene, rubber, and thermoplastic elastomer.

24. The robotic package handling system according to claim 1, wherein the first suction cup assembly comprises an outer housing and an internal structure coupled thereto.

25. The robotic package handling system according to claim 24, wherein the internal structure of the first suction cup assembly comprises a wall member coupled to a proximal base member, and the wall member and the proximal base member define an inner chamber.

26. The robot package handling system according to claim 25, wherein the wall member has a cylindrical shape and comprises a proximal end and a distal end.

27. The robotic package handling system according to claim 25, wherein the proximal base member defines one or more inlet openings through which the one or more inlet openings are configured to allow airflow through them in accordance with the activation of the controllably activated vacuum load.

28. The robotic package handling system according to claim 27, wherein the internal structure further comprises a distal wall member, the distal wall member comprising a structural opening ring portion configured to define an access portion to the inner chamber, and one or more transition air channels configured to allow airflow through it in accordance with the activation of the controllably activated vacuum load.

29. The robotic package handling system according to claim 28, wherein the one or more inlet openings and the one or more transition air channels function to allow a defined flow of air through the inner chamber that facilitates the releaseable coupling of the first suction cup assembly with the targeted package.

30. The robotic package handling system according to claim 1, wherein the one or more packages are selected from the group consisting of bags, "poly bags", "poly", fiber bags, fiber envelopes, bubble packaging bags, bubble packaging envelopes, "Jiffy" bags, "Jiffy" envelopes, and substantially rigid rectangular parallelepiped structures.

31. The robotic package handling system according to claim 30, wherein the one or more packages include a fiber bag containing a paper composite or a polymer composite.

32. The robotic package handling system according to claim 30, wherein the one or more packages include a fiber envelope containing a paper composite or a polymer composite.

33. The robotic package handling system according to claim 30, wherein the one or more packages include a substantially rigid rectangular parallelepiped structure including a box.

34. The robotic package handling system according to claim 1, wherein the end effector comprises a second suction cup assembly coupled to the controllably activated vacuum load.

35. The robotic package handling system according to claim 34, wherein the second suction cup assembly defines a second inner chamber, the second inner chamber being configured to pull in and at least partially encapsulate a portion of the targeted package when the vacuum load is controllably activated adjacent to the targeted package.

36. The robotic package handling system according to claim 1, further comprising a third imaging device, the third imaging device being operably coupled to the first computing system and positioned and oriented to capture one or more images of the targeted package after the gripping is performed using the end effector.

37. The robotic package handling system according to claim 1, further configured to capture a sequence of images of the targeted package during the motion of the targeted package and to estimate whether the targeted package is deformable by analyzing the deformation of the targeted package in the sequence of images.

38. The robotic package handling system according to claim 1, wherein the first computing system and the second imaging device are configured to capture and utilize one or more images after the gripping is performed using the end effector, and to estimate whether multiple packages or zero packages have been brought in using the gripping performed.

39. The robotic package handling system according to claim 38, wherein the first computing system is configured to discontinue the grip in response to a determination that multiple packages or zero packages have been brought in using the grip that has been performed.

40. The robotic package handling system according to claim 1, wherein the end effector comprises an instrument switching head portion, the instrument switching head portion is configured to be controllably coupled to and uncoupled from the first suction cup assembly using an instrument holder mounted geometrically in close proximity to the distal portion of the robotic arm.

41. The robotic package handling system according to claim 40, wherein the instrument holder is configured to hold one or more additional suction cup assemblies or one or more other package interface instruments, and is detachably coupled to the one or more additional suction cup assemblies or the one or more other package interface instruments, so that the first computing system may be configured to perform instrument switching using the instrument switching head portion.

42. The robotic package handling system according to claim 1, wherein the first computing system is configured to operate the robotic arm and end effector to place the targeted package on the place structure together with the targeted package such that the targeted package is dragged into the inclined portion including the place structure.

43. The robotic package handling system according to claim 1, wherein the first computing system is configured to operate the robotic arm and end effector to place the targeted package on the place structure together with the targeted package so that the targeted package is intentionally placed on the edge of the targeted package so that it will tip over onto the surface of the place structure in a preferred orientation and position.

44. The robotic package handling system according to claim 1, wherein the first computing system is configured to operate the robotic arm and end effector to place the targeted package on the place structure together with the targeted package so that the targeted package is swept across the surface of the place structure such that the targeted package remains substantially flat with respect to the surface of the place structure.

Images