State estimation utilizing geometrical data and vision system for palletizing

Abstract

To provide a robot system.SOLUTION: A system comprises a communication interface that receives sensor data indicating a current state of a work space from a sensor arranged in the work space. The work space includes a pallet or other container, and a plurality of items stuck on or in the container. The system comprises a processor that: controls a robot arm so as to place a first set of items on or in the pallet or other container, or take out the first set of items from the pallet or other container; updates a geometrical model on the basis of the first set of items placed on or in the container; estimates a stack of one or more items existing on or in the container by using both of the geometrical data and the sensor data; and generates or updates a plan for controlling the robot arm so as to place a second set of the items by using the estimated state.SELECTED DRAWING: Figure 4

Description

CROSS-REFERENCE TO OTHER APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 211,361, entitled "STATE ESTIMATION USING GEOMETRIC DATA AND VISION SYSTEM FOR PALLETIZING," filed June 16, 2021, which is incorporated herein by reference for all purposes. [Background technology]

[0002] Shipping and distribution centers, warehouses, loading docks, air cargo terminals, large retail stores, and other activities that ship and receive heterogeneous sets of items utilize strategies such as packing and unpacking heterogeneous items into boxes, crates, containers, conveyor belts, pallets, etc. Packing heterogeneous items into boxes, crates, pallets, etc. allows the resulting set of items to be handled by lifting equipment (forklifts, cranes, etc.), allowing the items to be packed more efficiently for storage (e.g., in a warehouse) and / or shipment (e.g., in a truck, cargo hold, etc.).

[0003] In some contexts, items may vary so much in size, weight, density, bulk, rigidity, packaging strength, etc. that they may or may not have attributes that allow any given item or set of items to support the size, weight, weight distribution, etc. of given other items that they may need to pack (e.g., in a box, container, pallet, etc.). When assembling heterogeneous items onto a pallet or other set, the items must be carefully selected and stacked to ensure that the palletized stack does not collapse, tip, or otherwise become unstable (e.g., so that it cannot be handled by a machine such as a forklift), and to avoid damage to the items.

[0004] Currently, pallets are typically loaded and / or unloaded manually. Human workers select items to stack based on, for example, shipping invoices or manifests, and use human judgment and intuition to select, for example, larger, heavier items to be placed at the bottom. However, in some cases, items simply arrive on a conveyor or other mechanism and / or are picked from bins in an ordered list, resulting in an unstable palletized or otherwise packed set.

[0005] The use of robotics is made more difficult in many environments due to the variety of items, for example, the variability in the order, number, and combination of items packed on a given pallet, and the variety in the types and locations of containers and / or delivery mechanisms from which items must be picked up to place them on a pallet or other container. [Brief explanation of the drawings]

[0006] Various embodiments of the present invention are disclosed in the following detailed description and the accompanying drawings.

[0007] [Figure 1] FIG. 1 illustrates a robotic system for palletizing and / or depalletizing heterogeneous items in accordance with various embodiments.

[0008] [Figure 2] FIG. 1 illustrates a robotic system for palletizing and / or depalletizing heterogeneous items in accordance with various embodiments.

[0009] [Figure 3] 1 is a flowchart illustrating a process for palletizing one or more items according to various embodiments.

[0010] [Figure 4]10 is a flowchart illustrating a process for determining the current state of a pallet and / or stack of items according to various embodiments.

[0011] [Figure 5] 10 is a flowchart illustrating a process for determining the current state of a pallet and / or stack of items according to various embodiments.

[0012] [Figure 6] 10 is a flowchart illustrating a process for determining the current state of a pallet and / or stack of items according to various embodiments.

[0013] [Figure 7A] FIG. 10 illustrates the state of a palette / stack of items using geometric data in accordance with various embodiments.

[0014] [Figure 7B] FIG. 1 illustrates the state of a pallet / stack of items using vision system or sensor data in accordance with various embodiments.

[0015] [Figure 7C] FIG. 1 illustrates the state of a pallet / stack of items using geometric data and vision system or sensor data, according to various embodiments.

[0016] [Figure 8] 10 is a flow chart illustrating one embodiment of determining an estimate of the state of a pallet and / or stack of items.

[0017] [Figure 9] 1 is a flowchart illustrating a process for palletizing / depalletizing a set of items according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0018] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments, or any other form the present invention may take, may be referred to herein as technology. In general, the order of steps in a disclosed process may be varied within the scope of the present invention. Unless otherwise noted, components, such as a processor or memory, described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0019] The following is a detailed description of one or more embodiments of the present invention with reference to figures that illustrate the principles of the invention. While the present invention has been described in connection with such embodiments, it is not limited to any particular embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters that are well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0020] As used herein, a geometric model may refer to a model of the state of a workspace (e.g., a programmatically determined state). For example, the geometric model is generated using geometric data determined in connection with generating a plan for moving items in the workspace and an expected result if the items are moved according to the plan. For example, the geometric model corresponds to the state of the workspace that is changed by controlling a robotic arm to pick, move, and / or place items in the workspace, and the picking, moving, and placing of the items is considered to be performed according to the plan (e.g., without errors (e.g., errors or noise) that may be caused by (i) misconfiguration or misalignment of the robotic arm or another component in the workspace, (ii) deformation of the items due to interaction with the robotic arm, (iii) other items in the workspace, other objects in the workspace, or (iv) collisions between the robotic arm or items being moved by the robotic arm and other objects in the workspace).

[0021] As used herein, a "pallet" includes a platform, receptacle, or other container on or within which one or more items can be stacked or placed. Additionally, as used herein, a pallet may be used in connection with packaging and dispensing a set of one or more items. By way of example, the term pallet includes a typical flat transport structure that supports items and is movable by a forklift, pallet jack, crane, or the like. As used herein, a pallet may be made from a variety of materials, including wood, metal, alloys, polymers, and the like.

[0022] As used herein, palletizing an item or set of items includes picking the items from a source location (e.g., a transport structure) and placing the items onto a pallet (e.g., onto a stack of items on a pallet).

[0023] As used herein, depalletization includes picking an item from a pallet (e.g., from a stack of items on a pallet), moving the item, and placing the item at a destination location (e.g., a conveying structure). Examples of palletization / depalletization systems and / or processes for palletizing / depalletizing sets of items are further described in U.S. Patent Application No. 17 / 343,609, which is incorporated herein by reference in its entirety for all purposes.

[0024] As used herein, singulation of items includes picking items from a source pile / stream and placing the items on a conveying structure (e.g., a partitioned conveyor or similar conveying means). Optionally, singulation may include sorting various items on a conveying structure, such as placing items one by one from the source pile / stream into slots or trays on a conveyor. Examples of singulation systems and / or processes for singulating sets of items are further described in U.S. Patent Application No. 17 / 246,356, which is incorporated herein by reference in its entirety for all purposes.

[0025] Kitting, as used herein, includes picking one or more items / objects from corresponding locations and placing one or more items in predetermined locations such that a set of one or more items corresponds to a kit. Examples of kitting systems and / or processes for kitting sets of items are further described in U.S. Patent Application No. 17 / 219,503, which is incorporated herein by reference in its entirety for all purposes.

[0026] As used herein, a vision system comprises one or more sensors that acquire sensor data (e.g., sensor data about a workspace). The sensors may include one or more of a camera, a high-resolution camera, a 2D camera, a 3D (e.g., RGBD) camera, an infrared (IR) sensor, other sensors for generating a three-dimensional view of the workspace (or a portion of the workspace, such as a pallet and a stack of items on the pallet), any combination of the foregoing, and / or a sensor array including multiple sensors as described above.

[0027] Techniques are disclosed that programmatically utilize a robotic system including one or more robots (e.g., robotic arms with suckers, grippers, and / or other end effectors at their working ends) to palletize / depalletize and / or otherwise pack and / or unpack any set of heterogeneous items (e.g., different sizes, shapes, weights, weight distributions, stiffness, fragility, types, packaging, etc.).

[0028] Various embodiments include a system, method, and / or apparatus for picking and placing items. The system includes a communications interface and one or more processors coupled to the communications interface. The communications interface is configured to receive sensor data from one or more sensors disposed in a workspace indicative of a current state of the workspace, the workspace including a pallet or other container and a plurality of items stacked on or in the container. The one or more processors are configured to (i) control a robotic arm to place or retrieve a first set of items on or in the pallet or other container, (ii) update a geometric model based on the first set of items placed on or in or retrieved from the pallet or other container, (iii) use the geometric model in conjunction with the sensor data to estimate a state of the pallet or other container and one or more items stacked on or in the container, and (iv) generate or update a plan for controlling the robotic arm to place or retrieve a second set of items on or in the pallet or other container using the estimated state.

[0029] Related art systems may control a robot based solely on geometric data (e.g., control a robot by driving motors without using workspace sensor data as feedback) or solely on sensor data (e.g., control a robot based on sensor data). However, related art systems do not use both geometric data and sensor data in connection with determining or updating a plan for moving an item and controlling a robot arm to move the item according to the plan. Furthermore, related art systems do not combine geometric data and sensor data to determine / update a plan and / or control a robot arm to move an item. For related art robotic systems that rely solely on geometric data in connection with controlling a robot arm to move an item, the robotic systems have very tight tolerances for manufacturing the robot / robot arm. Such tight tolerances generally increase the cost of the robot / robot arm.

[0030] A robotic system may be implemented to assemble and / or disassemble items onto a pallet. The robotic system may utilize sensors and a robot (e.g., a robotic arm with an end effector for engaging the items). The items to be assembled onto the pallet may be a heterogeneous set of items. The robotic system typically determines a plan for placing the set of items and then controls the robotic arm to place the set of items according to the plan. For example, for each item, the robotic system typically determines a destination location on the pallet where the item will be placed. When determining where to place an item (e.g., the destination location), the robotic system typically acquires sensor data directed toward the workspace (e.g., sensor data directed toward the pallet or a stack of items on the pallet). The robotic system determines the destination location based on the sensor data.

[0031] Relying on sensor data to determine where to place an item or a plan for moving the item to that location can be subject to error or otherwise inaccurate due to, at least, noise inherent in the sensor data and / or misalignment or miscalibration of the sensors used to obtain the sensor data. Thus, assembly and / or disassembly of items onto and / or from pallets can be unpredictable.

[0032] Some challenges that arise from utilizing sensor data to determine destination locations to place items and / or to determine assembly / disassembly of items to pallets (e.g., palletization / depalletization of items) include the following: Noise in Sensor Data. Examples of sources of noise in sensor data include: (i) light reflected from items or objects in the workspace, (ii) dust in the workspace, on items on the pallet / stack, and / or on the item being placed, (iii) poor quality reflective surfaces (such as mirrors), (iv) reflective surfaces in the workspace (robot, frame, shelf, another item, item on pallet, etc.), (v) heat or other temperature changes in the environment, (vi) moisture in the environment, (vii) vibrations at the sensor, and (viii) the sensor being bumped or moved while capturing sensor data. Inaccurate sensor data based at least in part on miscalibrated sensors. Inaccurate sensor data based at least in part on sensor misalignment · Inaccurate sensor data based at least in part on sensor errors. Sensor data is inaccurate or incomplete based at least in part on the field of view of the sensor, which may be incomplete or blocked by items or objects within the robot's workspace. Inaccurate / inexact control of the robot arm based on mechanical misalignment or miscalibration.

[0033] Various embodiments include a robotic system that utilizes sensor data and geometric data (e.g., a geometric model) in connection with determining a location to place one or more items on a pallet (or in connection with depalletizing one or more items from a pallet). The system may use different data sources to model the state of the pallet (or stack of items on the pallet). For example, the system may estimate the location of one or more items on the pallet and one or more characteristics (or attributes) associated with the one or more items (e.g., item size). The one or more characteristics associated with the one or more items may include item size (e.g., item dimensions), center of gravity, item stiffness, package type, deformability, shape, identifier location, etc.

[0034] According to various embodiments, the system estimates a state of the workspace (also referred to herein as an estimated state) based at least in part on geometric data (e.g., a geometric model of the workspace) and sensor data (e.g., data acquired by one or more sensors deployed in the workspace). In response to obtaining the estimated state of the workspace, the system utilizes the estimated state in connection with moving items within the workspace. For example, the system uses the estimated state to determine a plan and / or strategy for picking an item from a source location and placing it at a target location (also referred to herein as a destination location).

[0035] According to various embodiments, the system determines the estimated state at least in part by performing interpolation based on at least a portion of the geometric data and at least a portion of the sensor data. In some embodiments, the interpolation process performed on the first portion of the geometric model and the first portion of the sensor data to obtain the first portion of the estimated state is different from the interpolation performed on the second portion of the geometric model and the second portion of the sensor data to obtain the second portion of the estimated state.

[0036] In some embodiments, the system utilizes the estimated state in connection with determining a plan for palletizing one or more items. For example, the system utilizes the estimated state in connection with selecting an item from a conveyor (or other item source) containing a set of items (e.g., a series of items input into a workspace), controlling a robotic arm to pick the item, selecting a destination location corresponding to a location on a pallet or within a stack of items on a pallet, moving the item, and placing the item at the destination location.

[0037] In some embodiments, the system utilizes the estimated state in connection with determining a plan for depalletizing one or more items. For example, the system utilizes the estimated state in connection with selecting an item from a pallet or from among a stack of items on a pallet, controlling a robotic arm to pick the item, selecting a destination location corresponding to a bin or location on a conveyor (e.g., carrying the item from the workspace), moving the item, and placing the item at the destination location.

[0038] In some embodiments, the system uses the estimated state in connection with determining a plan for singulating one or more items. For example, the system uses the estimated state in connection with selecting an item from a chute containing a set of items, controlling a robotic arm to pick the item, selecting a destination location (e.g., a tray, a conveyor segment, etc.), moving the item, and placing the item at the destination location.

[0039] In some embodiments, the system utilizes the estimated state in connection with determining a plan for performing kitting for one or more items. For example, the system utilizes the estimated state in connection with selecting an item from a kitting shelving system having a plurality of shelves on which the items are located, controlling a robotic arm to pick the item from the shelf, a destination location (e.g., a container such as a box, tote, etc.), moving the item, and placing the item at the destination location.

[0040] According to various embodiments, the geometric model is determined based at least in part on one or more attributes of one or more items in the workspace. For example, the geometric model reflects the attributes of each of a set of items (e.g., one or more of the first palletized / stacked set and the second set of items to be palletized / stacked, etc.). Examples of items include item size (e.g., item dimensions), center of gravity, item rigidity, packaging type, identifier location, item deformability, item shape, etc. Various other attributes of items or objects in the workspace may also be implemented. As another example, the geometric model includes the predicted stability of one or more items stacked on or in a container (e.g., a pallet). The geometric model may include the predicted stability of a set of items (e.g., a stack of items) and / or the predicted stability of individual items included in the stack of items. In some embodiments, the system determines the predicted stability of an item based at least in part on (i) one or more attributes of the item and (ii) one or more predicted interactions with the item and another item or object (e.g., a pallet) in the workspace. For example, the system may determine the predicted stability based on a determination of the attributes of another item or object that contacts (or is positioned proximate to) the item whose predicted stability is being calculated. Examples of attributes of other items that may affect the predicted stability of a particular item include stiffness, deformability, size, etc. As one example, if a particular item rests on another item that is rigid, the particular item may have a higher predicted stability than if the particular item rests on another item that is not rigid or has a low rigidity. As another example, if a particular item rests on another item that is deformable (such as one made of soft packaging), the particular item may have a lower predicted stability than if the particular item rests on another item that is not deformable or has a low deformability.As another example, if a particular item rests on another item that has a top surface area that is larger than the bottom surface area of ​​the particular item, or if a relatively high percentage of the bottom surface of the particular item is supported by the top surface of the other item, the predicted stability of the item will be relatively higher, or at least greater, than if the particular item has a top surface area that is smaller than the bottom surface area of ​​the particular item, or if a relatively high percentage of the bottom surface of the particular item is not supported by / interacting with the top surface of the other item.

[0041] In some embodiments, the system adjusts the sensor data to account for noise (e.g., sensor noise). The system can estimate the noise present in the sensor data based at least in part on an empirical analysis of the vision system. For example, an empirical analysis of the vision system's performance can be performed to determine the noise present in (e.g., inherent in) the sensor data. In some embodiments, the system stores a predetermined sensor noise profile associated with the vision system. The system can utilize the sensor noise profile in connection with adjusting the sensor data to account for noise. For example, the system can apply adjustments to counteract the predicted noise based at least in part on the sensor profile. The empirical analysis of the vision system's performance can include (i) manually / physically measuring an item or workspace, (ii) capturing the same using a vision system, and (iii) determining (e.g., using digital processing, etc.) a difference between the manual / physical measurement of the item / workspace and a measurement of the same using sensor data. The system can consider the difference between the manual / physical measurement of the item / workspace and a measurement of the same using sensor data as a noise profile. As an example, the system determines variability in the sensor data and determines a sensor noise profile based at least in part on the variability. The empirical analysis may be performed on a set of statistically significant experiments / measurements. Examples of noise (or sensor data inaccuracies) may include (i) image inaccuracies at the edge of the vision system's field of view, (ii) glare / reflections from items or other objects in the workspace, etc.

[0042] In some embodiments, the system adjusts the geometric model to account for noise (e.g., geometric noise or inaccuracies resulting from the transition of the geometric model to the physical world, such as by controlling a robotic arm). The system can estimate the noise contained in the geometric model based at least in part on an empirical analysis of the accuracy of the robotic control or other objects in the workspace (e.g., estimated deformation of a pallet, deviations in the placement of the pallet relative to the location utilized in the geometric model, etc.). For example, an empirical analysis of the performance of the control of the robotic arm (e.g., when performing a task such as placing an item) can be performed to determine the noise incorporated (e.g., inherent) in the geometric model. As an example, the system determines variance in the geometric model and determines a geometric noise profile based at least in part on the variance. In some embodiments, the system stores a predetermined geometric noise profile associated with the vision system. The system can utilize the geometric noise profile in connection with adjusting the geometric model to account for noise. For example, the system can apply adjustments to counteract predicted noise contained in the geometric model (e.g., noise generated based on controlling the robot according to a plan determined based on the geometric model).

[0043] According to various embodiments, the system uses the geometric model and the sensor data to determine a best estimate of the state of the workspace. The system can adjust for noise (e.g., noise cancellation) in one or more of the geometric model and / or the sensor data. In some embodiments, the system detects anomalies or discrepancies between the state according to the geometric model and the state according to the sensor data. In response to determining that there is an anomaly or discrepancy between the geometric model and the sensor data, the system can make a best estimate of the state despite the anomaly or discrepancy. For example, the system can determine whether to use the geometric model or the sensor data, or whether to use a combination of the geometric model and the sensor data (e.g., interpolation therebetween). In some embodiments, the system determines the estimated state on a segment-by-segment basis (e.g., for each voxel in the workspace, each item, or each object). For example, a first portion of the workspace may be estimated using only the geometric model, a second portion of the workspace may be estimated using only the sensor data (e.g., if there is an anomaly in the geometric model), and / or a third portion of the workspace may be estimated based on a combination of the geometric model and the sensor data.

[0044] According to various embodiments, the system combines the geometric model and the sensor data to determine an estimated state of the workspace based at least in part on a predetermined bias. In some embodiments, the system utilizes the predetermined bias in connection with determining whether to consider the geometric model or the sensor data as ground truth modified by the other (e.g., to correct anomalies between the geometric model and the sensor data). As an example, if the predetermined bias indicates that the geometric model is biased, the system may utilize the geometric model and supplement the state according to the geometric model with the sensor data. Examples of modifications the system may perform regarding the state according to the geometric model include using the sensor data to confirm the placement of items predicted by the geometric model, and, if the difference between the sensor data and the geometric model exceeds a predetermined threshold, utilizing the sensor data for the portion of the workspace where the difference occurs, or correcting the geometric model by interpolating the data using the geometric model and the sensor data (e.g., the portion of the workspace where the anomaly / difference exists may be subject to interpolation, or the entire workspace may be subject to interpolation, etc.).

[0045] In some embodiments, the biases for the geometric model and the sensor data (or the corresponding weightings applied to each type of information) are determined based at least in part on a determination of whether the geometric model or the sensor data is more accurate. The accuracy of the geometric model or the sensor data may be determined through empirical trials or based on variations in information or noise contained in the geometric model and / or the sensor data. Robots are generally considered to be more accurate than sensors. Thus, in some embodiments, the biases are set to give priority or higher weighting to the geometric model compared to the sensor data. For example, a default bias may be set to preferentially weight the geometric model rather than the sensor data.

[0046] An example of a situation in which the geometric model and the sensor data may be inconsistent (e.g., differ by more than a predetermined difference threshold) includes a situation in which the system, using the geometric model, determines that a set of 10 items are stacked in corresponding positions (e.g., the system programmatically stores 10 item arrangements), but the sensor data may suggest that only 8 items are stacked in those positions (e.g., where the items are stacked according to the geometric model). In response to determining that the sensor data indicates that two boxes are missing, the system can determine whether the inconsistency between the geometric data and the sensor data occurred because (i) a box fell off the stack (e.g., the sensor data may include a box on the floor or to the side of the stack), (ii) the sensor's field of view is partially obstructed by another object (e.g., a robotic arm), or (iii) a box not included in the sensor data is considered by the geometric model to be stacked behind other boxes and therefore not visible in the sensor data.

[0047] Another example of a situation in which the geometric model and the sensor data may be inconsistent (e.g., the difference is greater than a predetermined difference threshold) includes a situation in which the system places a particular item in / on top of a stack of items, but after placing the particular item, the particular item falls from the stack of items. According to the geometric model in such a situation, the system considers the item to be on the stack of items (e.g., at a position where the robot arm placed the item in response to being controlled according to the plan) according to a plan determined based on the estimated state. However, the sensor data does not include the particular item placed in the expected destination position (e.g., within the stack of items). Instead, the sensor data reflects that the item has fallen from the stack of items (e.g., the item may be beside the stack of items or otherwise outside the field of view of the sensor data). In response to determining that the sensor data indicates that an item is missing, the system can determine whether the item has fallen from the stack (e.g., the sensor data may include a box on the floor or to the side of the stack).

[0048] An example of a situation in which the geometric model and sensor data may deviate from one another (e.g., differ by more than a predetermined difference threshold) includes when the system places an item on another item, which may deform the other item. If the other item's deformability exceeds the predicted deformation (if any) according to the geometric model, placing a particular item on another deformable item may cause the actual placement of the particular item to deviate from the geometric model (e.g., the placement of the item may be distorted by the extent to which the other item has deformed based at least in part on its interaction with the particular item placed on top of it, or the extent to which the other item's deformation exceeds the predicted deformation).

[0049] The predetermined bias may be set based on user selection (e.g., administrator definition) or a predetermined state determination policy. The predetermined state determination policy may be set based on empirical data or simulations of item pick-and-placement. For example, the predetermined bias corresponds to the best bias (e.g., optimal bias) for minimizing a cost function associated with moving or palletizing an item. In some embodiments, the cost function is based at least in part on one or more of the efficiency of moving an item, the predicted time to move an item, the likelihood of successfully moving / stacking an item, the predicted stability associated with placing an item (e.g., the stability of an item after placing a particular item, the stability of one or more other items or stacks after placing a particular item, etc.), etc. In some embodiments, the predetermined bias is selected from a set of potential biases corresponding to item placements whose cost function is less than a predetermined cost threshold.

[0050] According to various embodiments, the system uses a predetermined bias in connection with determining respective weightings for the geometric model and the sensor data. For example, the system determines a first weighting corresponding to the geometric model and a second weighting corresponding to the sensor data, and the system determines an estimated state of the workspace using a combination of the geometric model and the sensor data (e.g., combined according to the first weighting and the second weighting).

[0051] In response to obtaining the current estimated state, the system uses the estimated state to determine a first plan for moving a first item in the workspace (e.g., stacking the item on a stack of items), and the system controls the robot to move the first item according to the first plan. In response to placing the first item, the system uses geometric data corresponding to the item placement of the first item in connection with determining an initial updated estimated state. The system acquires sensor data (e.g., from a vision system), segments the sensor data to determine objects (e.g., identify objects in a sensor model of the workspace), and determines an updated estimated state based at least in part on the initial updated estimated state (e.g., a geometric model) and the sensor data. For example, the system combines the geometric model and the sensor data in connection with determining the updated estimated state. In response to determining the updated estimated state, the system uses the updated estimated state to determine a second plan for moving a second item in the workspace, and the system controls the robot to move the second item based at least in part on the second plan.

[0052] The use of geometric models and sensor data (e.g., a combination of geometric models) allows various embodiments to be less costly than related art systems. For example, the system can use the geometric model and sensor data to detect anomalies (e.g., caused by the geometric model or the sensor data) and determine an estimated state of the workspace despite the anomaly (e.g., based on selectively using the geometric model or the sensor data, or interpolation between both). Thus, while related art systems require either high-precision (e.g., high-tolerance) robots or high-precision cameras, various embodiments can utilize relatively low-precision (e.g., low-tolerance) robots or lower-precision (e.g., low-tolerance) vision systems. The high-tolerance requirements for robots or cameras in related art systems result in the robots and / or cameras used by such systems being very expensive. According to various embodiments, the combination of both geometric models and sensor data can reduce the cost and precision / accuracy needs of the robot or vision system.

[0053] According to various embodiments, a system for estimating the state of a pallet and / or a stack of items on a pallet records information about the placement of items (e.g., item location, item size, etc.) in response to a robot placing items on a pallet. For example, the system has logical knowledge of the state of the system based on where the robot placed various items on the pallet. The logical knowledge may correspond to geometric data (e.g., information obtained based on how the robot is controlled). However, the logical knowledge may differ from the actual state of the pallet and / or the stack of items on the pallet. Similarly, as described above, the state of the pallet and / or the stack of items on the pallet detected by the vision system (e.g., the actual state modeled based on sensor data) may differ from the actual state based on noise in the sensor data, inaccurate / incomplete sensor data, etc. Various embodiments combine a view of the world using geometric data (e.g., logical knowledge) and sensor data. Modeling the world using both geometric data and sensor data fills gaps in each dataset's view of the world. For example, sensor data obtained based on a vision system may be utilized in connection with determining whether the predicted state of a pallet or a stack of items on a pallet needs to be updated / improved. State estimation according to various embodiments provides a better estimate of the state of the pallet and / or stack of items on the pallet than would be possible using sensor data alone or geometric data alone. Furthermore, the estimated state of the pallet and / or stack of items on the pallet may be utilized in connection with palletizing / depalletizing items to / from the pallet. Utilizing a better estimate of the state of the pallet and / or stack of items on the pallet in connection with determining to palletize / depalletize items to / from the pallet may provide better placement, leading to a better final pallet (e.g., a tighter / tighter packed pallet, a more stable pallet, etc.).

[0054] In some embodiments, in response to determining that the predicted state of the pallet or stack of items using the geometric data is sufficiently different from the state of the pallet or stack of items using the vision system (e.g., sensor data), the system may provide a suggestion to the user. For example, the system may prompt the user to confirm whether the predicted state using the geometric data is correct, the state using the vision system is correct, or whether neither is correct and the user should modify the current state. The user may use a user interface to adjust the difference between the predicted state using the geometric data and the state using the vision system. A determination that the predicted state using the geometric data is sufficiently different from the state using the vision system may be based on a determination that the difference between the two states or models of the state exceeds a predetermined threshold.

[0055] According to various embodiments, the system for estimating the state of the pallet and / or stack of items on the pallet is implemented on a separate computer system (e.g., a server). Modules or algorithms for modeling the state using geometric data, modeling the state using a vision system (e.g., sensor data), and updating the predicted state (e.g., based on reconciling differences between states) may be expensive and require relatively high computational power. Furthermore, the robot's workspace or the processing power in the computer system controlling the robot may impose computational power and / or bandwidth constraints for executing the modules. In some embodiments, the system controlling the robot may obtain sensor data and information regarding the items to be placed and transmit the sensor data and information regarding the items to be placed to a server over one or more networks. In some embodiments, multiple servers may be utilized in connection with implementing various modules for estimating the state of the system (e.g., the state of the pallet and / or the state of the stack of items on the pallet). For example, a module for modeling a state using geometric data may be executed by a first server, a module for modeling a state using a vision system may be executed by a second server, and a module for updating a predicted state based at least in part on a difference between the states may be executed by a third server. The various servers that may implement the various modules for determining the predicted state may be in communication with each other.

[0056] In some embodiments, a system that determines an estimated state (e.g., a pallet state estimation module / service, etc.) may store and / or manage the difference between (i) a state modeled based on geometric data and (ii) a state modeled based on a vision system. For example, a pallet state estimator may store / manage the current state of a pallet and / or a stack of items on a pallet. The pallet state estimator may be a module executed by a computer system, such as a robotics system controlling a robot that is palletizing / depalletizing items to / from a pallet, or by a server with which the robotics system controlling the robot communicates.

[0057] According to various embodiments, a pallet state estimator may be utilized in connection with determining the current state during the planning / placing of each item. For example, a robotic system controlling a robot and / or determining a plan for moving an item may query the pallet state estimator in connection with determining a plan for moving an item. The pallet state estimator may be queried using (i) information about the item to be placed and (ii) current sensor data acquired by a vision system. The current state using geometric data may be stored by or accessible to the pallet state estimator. In some embodiments, information about the current geometric data may be communicated to the pallet state estimator in connection with querying the pallet state estimator for the current state. In some embodiments, due to the computational intensity of determining the current state using both geometric data and the vision system (e.g., sensor data), the pallet state estimator may be queried for the current state after placing a predetermined number of items (e.g., N items, where N is an integer). For example, the pallet state estimator may be queried at a fixed frequency. As another example, the pallet state estimator may be queried in response to determining that a placed item (e.g., a previously placed item) has an irregular shape or a particular type of packaging (e.g., a less rigid packaging type such as a polybag).

[0058] In some embodiments, the pallet state estimator iteratively updates its internal model of the state (e.g., of the world, the pallet, and / or the stack of items on the pallet, etc.) after placing an item. The internal model of the state may correspond to the current state of the pallet and / or the stack of items on the pallet. The pallet state estimator may update its internal model after placing each item. For example, a robotic system controlling the robot may provide information about the item (e.g., item characteristics such as dimensions, weight, center of gravity, etc.), sensor data obtained by a vision system, and / or geometric data corresponding to the location where the robot is controlled to place the item. In response to receiving information about the item, the sensor data, and / or the geometric data, the pallet state estimator may update its internal model. For example, the pallet state estimator may (i) provide sensor data to a module for modeling a state using sensor data, (ii) provide geometric data and / or information about the item to a module for modeling a state using geometric data, and (iii) update its internal model based on the sensor data and / or the model of the state based on the geometric data (e.g., or the transition between two states).

[0059] A model of the state of the pallet and / or stack of items may be determined using the pallet as a reference (e.g., the bottom of the pallet may be used as a reference point for the bottom of the stack, etc.). The model may be updated after items are placed so that the items are represented in their placed locations. The model of the state may be stored as a two-dimensional grid (e.g., 10x10). The model may further include information related to each item in the stack of items on the pallet. For example, one or more characteristics associated with the item may be stored in association with the item in the model. In some embodiments, the stability of the stack of items may be determined / calculated based on the positions of various items on the stack and one or more characteristics associated with the items. The model of the state of the pallet may be converted to a representation in the physical world based on a predetermined transformation between units of the stored model and units of the physical world.

[0060] Various embodiments include a robotic system including a communications interface and one or more processors coupled to the communications interface. The one or more processors may be configured to receive, via the communications interface, data related to a plurality of items to be stacked on or in a destination location; generate a plan for stacking the items on or in the destination location based at least in part on the received data; and execute the plan by, at least in part, picking up the items and stacking them on or in the destination location according to the plan. Generating the plan for stacking the items on or in the destination location may include, for each item, determining a destination location based at least in part on a feature associated with the item and at least one of (i) a feature of a platform or container on which the one or more items are stacked and (ii) a current state of the stack of items on the platform or container, where the current state of the stack of items on the platform or container is based at least in part on geometric data regarding the arrangement of one or more items included in the stack of items and sensor data regarding the stack of items acquired by a vision system. The current state of the stack of items may be determined based at least in part on a correlation between geometric data regarding the arrangement of one or more items included in the stack of items and sensor data regarding the stack of items acquired by the vision system. The current state of the stack of items on a platform or bin may be obtained based at least in part on the robotic system querying a pallet state estimator executing on a server.

[0061] 1 illustrates a robotic system for palletizing and / or depalletizing heterogeneous items, according to various embodiments. In some embodiments, the robotic system 100 is implemented, at least in part, by process 300 of FIG. 3, process 400 of FIG. 4, process 500 of FIG. 5, process 600 of FIG. 6, process 800 of FIG. 8, and / or process 900 of FIG. 9.

[0062] In the illustrated example, the robotic system 100 includes a robotic arm 102. In this example, the robotic arm 102 is fixed; however, in various alternative embodiments, the robotic arm 102 may be fully or partially movable, e.g., mounted on rails, fully movable on a motor-driven chassis, etc. As shown, the robotic arm 102 is used to pick random and / or heterogeneous items (e.g., boxes, packages, etc.) from a conveyor (or other source) 104 and stack them on a pallet (e.g., a platform or other container) 106. The pallet (e.g., a platform or container) 106 comprises a pallet, container, or base with wheels on the four corners and at least partially closed on three of its four sides, and is also referred to as a three-sided "roll pallet," "roll cage," and / or "roll," "cage," or "trolley." In other embodiments, rolls with more, fewer, and / or no sides or pallets without wheels may be used. In some embodiments, other robots not shown in FIG. 1 may be used to push the container 106 into position for loading / unloading and / or onto a truck or other destination for transport, etc.

[0063] In some embodiments, multiple bins 106 may be positioned around the robotic arm 102 (e.g., within a range near a threshold of the robotic arm or within the range of another robotic arm). The robotic arm 102 may simultaneously (e.g., side-by-side and / or contemporaneously) stack one or more items onto multiple pallets. Each of the multiple pallets may be associated with a manifest and / or order. For example, each of the pallets may be associated with a pre-defined destination (e.g., a customer, an address, etc.). In some examples, some of the multiple pallets may be associated with the same manifest and / or order. However, each of the multiple pallets may be associated with a different manifest and / or order. The robotic arm 102 may place multiple items, each corresponding to the same order, onto multiple pallets. The robotic system 100 may determine the arrangement (e.g., stacking of items) on the multiple pallets (e.g., how items for an order are divided among the multiple pallets, how items on any one pallet are stacked, etc.). The robotic system 100 may store one or more items (e.g., items for an order) in a buffer area or staging area while one or more other items are stacked on a pallet. As an example, one or more items may be stored in a buffer area or staging area until the robotic system 100 determines that the placement of each of the one or more items on the pallet (e.g., into a stack) meets (e.g., exceeds) a threshold suitability or stability. The threshold suitability or stability may be a predetermined value or may be a value empirically determined based at least in part on historical information. Machine learning algorithms may be implemented in connection with determining whether the placement of an item on a stack is expected to meet (e.g., exceed) the threshold suitability or stability and / or in connection with determining the threshold suitability or stability (e.g., a threshold against which a simulation or model is compared to evaluate whether to place an item on a stack).

[0064] In the illustrated example, the robotic arm 102 is equipped with a suction-type end effector (e.g., end effector 108). The end effector 108 has a plurality of suction cups 110. The robotic arm 102 is used to position the suction cups 110 of the end effector 108 over the item to be picked up, as shown, and a vacuum source provides a suction force to grasp the item, lift the item from the conveyor 104, and place the item at a destination location on the bin 106. Various types of end effectors may be implemented.

[0065] In various embodiments, the robotic system 100 includes a vision system that is used to generate a model of the workspace (e.g., a 3D model and / or a geometric model of the workspace). For example, one or more of a 3D camera or other camera 112 mounted on the end effector 108 and cameras 114, 116 mounted in the space in which the robotic system 100 is deployed are used to identify items on the conveyor 104 and / or determine a plan for grasping, picking / placing, and stacking items onto the bins 106 (or placing the items in a buffer or staging area, if applicable). In various embodiments, additional sensors not shown may be used to identify (e.g., determine) attributes of the item, grasp the item, pick up the item, move the item through the determined trajectory, and / or place the item at a destination location on or in the item in the bin 106 on the conveyor 104 and / or other source and / or staging area where the item may be placed and / or relocated, for example, by the system 100. Examples of such additional sensors not shown may include weight or force sensors embedded in and / or adjacent to the conveyor 104 and / or the robotic arm 102, force sensors in the xy plane and / or z direction (vertical direction) of the suction cup 110.

[0066] In the illustrated example, camera 112 is mounted to the side of the body of end effector 108, although in some embodiments camera 112 and / or additional cameras may be mounted in other locations (e.g., facing downward from between suction cups 110, mounted on the underside of the body of end effector 108, mounted on a segment or other structure of robotic arm 102, or mounted in other locations, etc.). In various embodiments, cameras such as 112, 114, and 116 may be used to read text, logos, photographs, drawings, images, marks, barcodes, QR codes, or other encoded and / or graphic information or content visible on and / or comprising items on conveyor 104.

[0067] In some embodiments, the robotic system 100 includes a dispenser device (not shown) configured to dispense a quantity of spacer material from a supply of spacer material in response to a control signal. The dispenser device may be located on the robot arm 102 or may be located proximate to the workspace (e.g., within a threshold distance of the workspace). For example, the dispenser device may be disposed within the workspace of the robot arm 102 such that the dispenser device dispenses spacer material onto or around a container 106 (e.g., a pallet) or within a predetermined distance of the end effector 108 of the robot arm 102. In some embodiments, the dispenser device includes a mount configured to attach the dispenser device on or adjacent to the end effector 108 of the robot arm 102. The mount may be at least one of a bracket, a strap, one or more fasteners, etc. As an example, the dispenser device may include a biasing device / mechanism that biases the supply of material within the dispenser device to be extruded / dispensed from the dispenser device. The dispenser device may comprise a gating structure that is used to control the dispensing of spacer material (e.g., no actuation of the gating structure prevents dispensing of spacer material, and actuation of the gating structure allows dispensing of dispensed spacer material).

[0068] The dispenser device may include a communications interface configured to receive control signals. For example, the dispenser device may communicate with one or more terminals (e.g., control computer 118). The dispenser device may communicate with one or more terminals via one or more wired connections and / or one or more wireless connections. In some embodiments, the dispenser device communicates information to one or more terminals. For example, the dispenser device may send an indication of the status of the dispenser device (e.g., an indication of whether the dispenser device is operating normally), an indication of the type of spacer material included within the dispenser device, an indication of the supply level of spacer material within the dispenser device (e.g., an indication of whether the dispenser device is full, empty, half full, etc.), etc. to the control computer 118. The control computer 118 may be utilized in connection with controlling the dispenser device to dispense a quantity of spacer material. For example, the control computer 118 may determine that a spacer is utilized in connection with palletizing one or more items, such as to improve the predictive stability of the stack of items on / in the container 106. Control computer 118 may determine an amount of spacer material to utilize in connection with palletizing one or more items (e.g., number of spacers, amount of spacer material, etc.). For example, the amount of spacer material to utilize in connection with palletizing one or more items may be determined based at least in part on determining a plan for palletizing the one or more items.

[0069] In some embodiments, the dispenser device includes an actuator configured to dispense a quantity of spacer material from a supply of spacer material in response to a control signal. In response to determining that a spacer / spacer material is to be utilized in connection with palletizing one or more items, the control computer 118 may generate a control signal to cause the actuator to dispense a quantity of spacer material. The control signal may include an indication of the amount of spacer material to be utilized as a spacer.

[0070] According to various embodiments, the spacer or spacer material is a rigid block. For example, the spacer or spacer material may be a rigid block of foam. In some embodiments, the spacer or spacer material comprises polyurethane.

[0071] In some embodiments, the supply of spacer material includes a plurality of cut blocks, which may be pre-loaded into a spring-loaded cartridge that biases the plurality of cut blocks toward the dispensing end, such that in response to dispensing a cut block from the cartridge, another block of the plurality of cut blocks is pushed out of the cartridge into a position to be next dispensed.

[0072] In some embodiments, the supply of spacer material includes one or more of a larger block of spacer material, a strip of spacer material, and a roll of spacer material. The dispenser device or robotic system 100 may include a cutter configured to cut a quantity of spacer material from the supply of spacer material. In response to providing a control signal to the actuator, the actuator may cause the cutter to cut a quantity of spacer material from the supply of spacer material.

[0073] In some embodiments, the source of spacer material includes a liquid precursor, and in response to providing a control signal to the actuator, the actuator causes a quantity of spacer material to be dispensed onto a surface of the pallet or a stack of items on the pallet, the dispensed precursor being capable of curing after being dispensed onto the surface of the pallet or a stack of items on the pallet.

[0074] In some embodiments, the source of spacer material includes an extruded material. In response to providing a control signal to the actuator, the extruded material is filled to one or more of a desired size and a desired hardness. The extruded material may be sealed in response to determining that the extruded material has filled to one or more of a desired size and a desired hardness. In some embodiments, the extruded material is filled with a fluid. The fluid may be one or more of air, water, etc. In some embodiments, the extruded material is filled with a gel.

[0075] In various embodiments, a robotically controlled dispenser, tooling, or machine fills gaps between and / or adjacent to boxes to prepare the surface area for the next box / layer to be placed. In some embodiments, the robotic system 100 may use the robotic arm 102 to pick / place pre-cut material and / or dynamically trim spacer material to fit the surface area needs of the next item. In some embodiments, the robotically controlled dispenser device or a robotic palletization system including the robotically controlled dispenser device includes equipment for trimming cuboids to size from long tubes and / or packages and placing the cuboids onto an existing pallet in connection with preparing the surface area for the next box or item that the system determines may not normally fit on the pallet surface area (e.g., on top of the previous layer). Spacers may include, but are not limited to, foam, air-inflated plastic bags, wood, metal, plastic, etc. The dispenser device may place (e.g., push, feed, etc.) the cuboids (e.g., spacers) directly onto the pallet, and / or the device may feed the cuboids (e.g., spacers) near a robotic arm and an end effector may reposition / place the cuboids (e.g., spacers) on the pallet surface area. The dispenser device may feed a predetermined amount (e.g., the correct amount or expected amount) of spacer material to correct or improve surface area discrepancies between boxes or items on a layer (e.g., on the top surface of a layer) to prepare the surface area for the next box or item.

[0076] 1 , in the depicted example, robotic system 100 includes a control computer 118 configured to communicate with elements such as robotic arm 102, conveyor 104, end effector 108, and sensors (such as cameras 112, 114, and 116 and / or weight, force, and / or other sensors not shown in FIG. 1 ), in this example via wireless communication (although in various embodiments, via one or both of wired and wireless communication). In various embodiments, control computer 118 is configured to use input from the sensors (such as cameras 112, 114, and 116 and / or weight, force, and / or other sensors not shown in FIG. 1 ) to observe, identify, and determine one or more attributes of items being loaded into and / or unloaded from container 106. In various embodiments, the control computer 118 identifies the item and / or its attributes using item model data in a library stored in the control computer 118 and / or accessible to the control computer 118, for example, based on image and / or other sensor data. The control computer 118 uses the model corresponding to the item to determine and implement a plan for stacking the item, along with other items, in / on a destination (e.g., a container 106). In various embodiments, the item attributes and / or model are utilized to determine a strategy for grasping, moving, and placing the item at a destination location (e.g., a location where the item has been determined to be placed as part of the planning / re-planning process for stacking the item in / onto the container 106).

[0077] In the illustrated example, control computer 118 is connected to an "on-demand" telemanipulator 122. In some embodiments, if control computer 118 is unable to continue in fully automated mode, e.g., if a strategy for grasping, moving, and placing an item becomes indeterminable and / or fails such that control computer 118 has no strategy for completing the pick and place of the item in fully automated mode, control computer 118 instructs human user 124 to intervene, e.g., by manipulating robotic arm 102 and / or end effector 108 using telemanipulator 122 to grasp, move, and place the item.

[0078] A user interface for operation of the robotic system 100 may be provided by the control computer 118 and / or the remote operator 122. The user interface may provide the current status of the robotic system 100, including information related to the current state of the pallet (or stack of items associated therewith), the current order or manifest being palletized or depalletized, the performance of the robotic system 100 (e.g., number of items palletized / depalletized by time), etc. A user may select one or more elements on the user interface or otherwise provide input to the user interface to activate or deactivate the robotic system 100 and / or particular robotic arms within the robotic system 100.

[0079] According to various embodiments, the robotic system 100 performs a machine learning process to model the condition of the pallet, such as to generate a model of the stacks on the pallet. The machine learning process may include an adaptive and / or dynamic process to model the condition of the pallet. The machine learning process may define and / or update / refine the process by which the robotic system 100 generates the model of the condition of the pallet. The model may be generated based at least in part on input (e.g., information obtained from the sensors) from one or more sensors in the robotic system 100 (e.g., one or more sensors or sensor arrays in the workspace of the robot arm 102). The model may be generated based at least in part on the shape of the stack, the vision response (e.g., information obtained by one or more sensors in the workspace), and the machine learning process. The robotic system 100 may utilize the model in connection with determining an efficient (e.g., maximize / optimize efficiency) method for palletizing / depalletizing one or more items, where the method for palletizing / depalletizing may be constrained by a minimum threshold stability value. The process for palletizing / depalletizing one or more items may be configurable by a user / administrator. For example, one or more metrics at which the process for palletizing / depalletizing is maximized may be configurable (e.g., set by a user / administrator).

[0080] In the context of palletizing one or more items, the robotic system 100 may generate a model of the pallet's condition in connection with determining whether to place an item on a pallet (e.g., on a stack), selecting a plan for placing the item on the pallet, such as a destination location where the item will be placed, and a trajectory along which the item will be moved from a source location (e.g., a current destination, such as a conveyor) to the destination location. The robotic system 100 may also use the model in connection with determining a strategy for releasing or otherwise placing the item on the pallet (e.g., applying forces to the item to secure it on the stack). Modeling the pallet's condition may include simulating the placement of the item at different destination locations on the pallet (e.g., on a stack) and determining corresponding different predicted fitnesses and / or predicted stabilities (e.g., stability indices) that are predicted to result from placing the item at the different locations. The robotic system 100 may select destination locations where the predicted fitnesses and / or predicted stabilities meet (e.g., exceed) corresponding thresholds. Additionally or alternatively, the robotic system 100 may select a destination location that optimizes predicted fitness (e.g., of the items on the stack) and / or predicted stability (e.g., of the stack).

[0081] Conversely, in the context of depalletizing one or more items from a pallet (e.g., a stack on a pallet), the robotic system 100 (e.g., control computer 118) may generate a model of the state of the pallet in connection with determining whether to remove an item from the pallet (e.g., a stack) and selecting a plan for removing the items from the pallet. The model of the state of the pallet may be utilized in connection with determining the order in which items are removed from the pallet. For example, the control computer 118 may utilize the model to determine whether removing an item is predicted to reduce the stability of the state of the pallet (e.g., a stack) below a threshold stability. The robotic system 100 (e.g., control computer 118) may simulate the removal of one or more items from the pallet and select an order for removing items from the pallet that optimizes the stability of the state of the pallet (e.g., a stack). The robotic system 100 may use the model to determine the next item to remove from the pallet. For example, the control computer 118 may select an item as the next item to be removed from the pallet based at least in part on a determination that the predicted stability of the stack during and / or after removal of the item exceeds a threshold stability. Models and / or machine learning processes may be utilized in connection with determining a strategy for picking an item from a stack. For example, after an item is selected as the next item to be removed from a stack, the robotic system 100 may determine a strategy for picking the item. The strategy for picking the item may be based at least in part on the condition of the pallet (e.g., the determined stability of the stack), attributes of the item (e.g., size, shape, weight or predicted weight, center of gravity, type of packaging, etc.), the location of the item (e.g., position relative to one or more other items in the stack), attributes of other items on the stack (e.g., attributes of adjacent items, etc.), etc.

[0082] According to various embodiments, machine learning processes are performed in connection with improving a gripping strategy (e.g., a strategy for gripping an item). The robotic system 100 may acquire attribute information about one or more items to be palletized / depalletized. The attribute information may include one or more of the item's orientation, material (e.g., packaging type), size, weight (or predicted weight), or center of gravity. The robotic system 100 may also acquire information about a source location (e.g., information about an input conveyor from which the item is picked) and a pallet on which the item is placed (or a set of pallets from which a destination pallet is determined, such as a set of pallets corresponding to an order in which the item is stacked). In connection with determining a plan for picking and placing an item, the robotic system 100 may use information about the item (e.g., attribute information, destination location, etc.) to determine a strategy for picking the item. The pick strategy may include a suggestion of a pick location (e.g., a location on the item where the robotic arm 102 engages the item via an end effector, etc.). The pick strategy may include a force applied to pick the item and / or a holding force to grasp the item while the robotic arm 102 moves the item from a source location to a destination location. The robotic system 100 may utilize machine learning processes to improve the pick strategy based at least in part on associations between information about the item (e.g., attribute information, destination location, etc.) and performance in picking the item (e.g., historical information related to past iterations of picking and placing the item or similar items (e.g., items sharing one or more similar attributes)).

[0083] According to various embodiments, the robotic system 100 may determine to utilize a spacer or an amount of spacer material in connection with palletizing one or more items in response to determining that utilizing a spacer or an amount of spacer material improves the stacking results of the items on the pallet (e.g., improves the stability of the stack of items). In some embodiments, the determination that placing one or more spacers in connection with placing a set of N items on a pallet improves the stacking of the items on the pallet is based at least in part on one or more of packing density, flat top surface, and stability. In some embodiments, the determination that placing one or more spacers in connection with placing a set of N items on a pallet improves the stacking of the items on the pallet is based at least in part on determining that the packing density of a stack of items including the set of N items is higher than the packing density if the set of N items were placed on the pallet without the one or more spacers. In some embodiments, the determination that placing one or more spacers in association with placing a set of N items on a pallet improves stacking of items on the pallet is based at least in part on a determination that the top surface is flatter than the top surface of the set of N items placed on the pallet without the one or more spacers. In some embodiments, the determination that placing one or more spacers in association with placing a set of N items on a pallet improves stacking of items on the pallet is based at least in part on a determination that the stability of a stack of items including the set of N items is greater than the stability of the set of N items placed on the pallet without the one or more spacers. N may be a positive integer (e.g., a positive integer less than the total number of items to be palletized on the completed pallet).

[0084] As an example, because N may be less than the total number of items to be palletized, the robotic system 100 may be limited in optimizing the stack of items (e.g., the robotic system 100 may only plan to place N items at a time). Therefore, utilizing one or more spacers increases the number of degrees of freedom associated with placing the N items. The robotic system 100 may utilize one or more spacers to optimize the stack of N items (or to achieve a "good enough" stack of N items, such as a stack that meets a minimum stability threshold). The robotic system 100 may utilize a cost function in connection with determining whether to utilize one or more spacers, the number of spacers to utilize, the placement of the spacers, etc. For example, the cost function may include one or more of a stability value, a time to place one or more items, a packing density of the stack of items, a top flatness or variability of the top surface of the stack of items, a cost of the feed material, etc.

[0085] According to various embodiments, the control computer 118 controls the robotic system 100 to place a spacer on the container 106 (e.g., a pallet) or a stack of items in connection with improving the stability of the stack of items on the container 106. As one example, the spacer may be placed in response to a determination that it is estimated (e.g., it is likely, such as having a probability above a predetermined likelihood threshold) that the stability of the stack of items will be improved if the spacer is utilized. As another example, the control computer 118 may control the robotic system 100 to utilize a spacer in connection with placing a set of items (e.g., a set of N items, where N is an integer) in response to a determination that the stability of the stack of items is less than a threshold stability value and / or that the stability of the stack of items is estimated to be less than a threshold stability value.

[0086] According to various embodiments, the control computer 118 may determine the stability of the stack of items based at least in part on a model of the stack of items and / or a simulation of placing one or more sets of items. The computer system may obtain (e.g., determine) a current model of the stack of items and model (e.g., simulate) the placement of the set of items. In connection with modeling the stack of items, a predicted stability of the stack of items may be determined. Modeling the stack of items may include modeling the placement of spacers in connection with modeling the placement of the set of items.

[0087] In some embodiments, the control computer 118 may determine the stability of a stack of items (or a simulated stack of items) based at least in part on one or more attributes of the top surface of the stack of items (or the simulated stack of items) and / or the spacer. For example, a measure of the degree to which the top surface is flat may be utilized in connection with determining the stability of the stack of items. Placing a box on a flat surface may result in a stable arrangement and / or stack of items. As another example, the surface area of ​​the flat area above the top surface may be utilized in connection with determining the stability or predicted stability of the arrangement of items on the stack of items. The greater the flat area above the top surface of the stack of items relative to the bottom surfaces of items placed on the stack of items, the more likely the stability of the stack of items will meet (e.g., exceed) a threshold stability value.

[0088] According to various embodiments, the robotic system 100 generates a model of a pallet or a stack of one or more items on the pallet, and spacers or spacer material are determined to be placed in association with palletizing the one or more items based at least in part on the model of the pallet or the stack of one or more items on the pallet. The robotic system 100 may generate a model of at least a top surface of the pallet or the stack of one or more items on the pallet, determine a set of N items to be next placed on the pallet (e.g., N is a positive integer), determine that placing one or more spacers in association with placing the set of N items on the pallet improves the stack of items on the pallet compared to a stack resulting from placing the set of N items without the spacers, generate one or more control signals to cause an actuator to supply an amount of spacer material corresponding to the one or more spacers, and provide the one or more control signals to the actuator in association with placing the set of N items on the pallet.

[0089] According to various embodiments, item (e.g., item type) variability among palletized items can complicate palletizing the items in a stable manner (e.g., in a manner where the stability of the stack of items meets a threshold stability value). In some embodiments, the control computer 118 may only be able to predict a certain number of items to be palletized. For example, the system may have a queue / buffer of N items to be palletized, where N is a positive integer. N may be a fraction of the total number of items to be stacked on the pallet. For example, N may be relatively small compared to the total number of items to be stacked on the pallet. Thus, the robotic system 100 may only be able to optimize the stacking of items utilizing the next N known items. For example, the robotic system 100 may determine a plan for stacking one or more items according to the current state of the stack of items (e.g., current model) and one or more attributes associated with the next N items to be stacked. In some embodiments, the use of one or more spacers may provide flexibility in how the next N items are stacked and / or improve the stability of the stack of items.

[0090] Various embodiments include palletizing a relatively large number of mixed boxes or items onto a pallet. The various boxes and items being palletized may have different attributes (height, shape, size, stiffness, packaging type, etc.). Variations in one or more attributes of the various boxes or items may make it difficult to place the items on the pallet in a consistent manner. In some embodiments, the robotic system 100 (e.g., control computer 118) may determine a destination location (e.g., a location where the item is to be placed) for an item that has a larger surface area (e.g., a larger base) than the box or other item below the item being placed. In some embodiments, items having different heights (e.g., different bin heights) may be placed in relatively tall areas of the pallet (e.g., above a height threshold equal to the maximum pallet height multiplied by 0.5, above a height threshold equal to the maximum pallet height multiplied by 2 / 3, above a height threshold equal to the maximum pallet height multiplied by 0.75, or above a height threshold equal to the maximum pallet height multiplied by another predetermined value).

[0091] According to various embodiments, machine learning processes are performed in connection with improving a spacer material supply / utilization strategy (e.g., a strategy for utilizing spacer material in connection with palletizing one or more items). The robotic system 100 may obtain attribute information about one or more items to be palletized / depalletized and attribute information about one or more spacers utilized in connection with palletizing / depalletizing one or more items. The attribute information may include one or more of the item's orientation, material (e.g., type of spacer material), size, weight (or predicted weight), center of gravity, stiffness, dimensions, etc. The robotic system 100 may also obtain information about a source location (e.g., information about an input conveyor from which an item is picked) and a pallet onto which an item is placed (or a set of pallets from which a destination pallet is determined, such as a set of pallets corresponding to an order in which items are stacked). In connection with determining a plan for picking and placing items, the robotic system 100 may use information about the items (e.g., attribute information, destination locations, etc.) to determine a strategy for palletizing the items (e.g., picking and / or placing the items). The palletizing strategy may include suggestions for pick locations (e.g., locations on the items where the robotic arm 102 engages the items, such as via an end effector) and destination locations (e.g., locations on the pallet / bin 106 or stack of items). The palletizing strategy may include suggestions for the force to be applied to pick the items and / or the holding force with which the robotic arm 102 grips the items while moving them from the source location to the destination location, the trajectory by which the robotic arm moves the items to the destination location, the amount of spacer material (if any) to be utilized in connection with placing the items at the destination location, and a plan for disposing the spacer material.The robotic system 100 may utilize machine learning processes to improve palletizing strategies based at least in part on relationships between information about the item (e.g., attribute information, destination location, etc.) and one or more of: (i) performance in picking and / or placing the item (e.g., historical information related to past iterations of picking and placing the item or similar items (e.g., items sharing one or more similar attributes)); (ii) performance of the item's stability after it has been placed in its destination location, e.g., against predicted stability generated using a model of the stack of items (e.g., historical information related to past iterations of palletizing the item or similar items (e.g., items sharing one or more similar attributes)); and (iii) performance of the item's stack stability after it has been placed in its destination location, e.g., against predicted stability generated using a model of the stack of items (historical information related to past iterations of palletizing the item or similar items and / or spacers (e.g., items / spacers sharing one or more similar attributes)). In some embodiments, the robotic system 100 may use machine learning processes to improve the utilization of one or more spacers in association with a palletizing strategy based at least in part on a relationship between information related to the spacers and / or one or more items to be palletized in association with the palletizing strategy (e.g., attribute information, destination location, etc.) and the stability performance of palletizing the set of items using one or more spacers relative to the predicted stability of palletizing the set of items using one or more spacers (e.g., predicted stability based on a simulation of palletizing the items using a model of the stack of items).

[0092] The model generated by the robotic system 100 may correspond to or be based at least in part on a geometric model. In some embodiments, the robotic system 100 generates the geometric model based at least in part on one or more placed items (e.g., items placed by the robotic system 100 via the robotic arm 102), one or more attributes associated with at least a portion of the one or more items, and one or more objects in the workspace (e.g., predetermined objects such as a pallet, a robotic arm, a shelving system, a chute, or other infrastructure included in the workspace). The geometric model may be determined at least in part based on executing a physics engine on the control computer 118 to model the stacking of items (e.g., modeling the state / stability of a stack of items, etc.). The geometric model may be determined based on predicted interactions of various components of the workspace (e.g., interactions of an item with another item, object, or simulated forces applied to the stack (e.g., to model the use of a forklift or other equipment to lift / move a pallet or other container on which the stack of items is located)).

[0093] In some embodiments, the system updates the geometric model after each item movement (e.g., placement). For example, the system maintains (e.g., stores) a geometric model corresponding to the state of the workspace (e.g., the state / stability of the stack of items and the position of one or more items within the stack of items, etc.). The geometric model utilizes the current geometric model in connection with determining a plan for moving the items and controlling the robotic arm to move the items. In response to item movement, the system updates the geometric model to reflect the item movement. For example, when depalletizing a stack of items, in response to a particular item being picked and removed from the stack of items, the system updates the geometric model so that the particular item is no longer represented as being on the stack but is included within the geometric model at the particular item's placed destination location, or if the destination location is outside the workspace, the geometric model is updated to remove the item. Additionally, the geometric model is updated to reflect the stability of the stack of items after the particular item has been removed from the stack. As another example, when palletizing a set of items, the system updates the geometric model to reflect the placement of the particular item on / in the stack of items. The system can update the geometric model to include an updated stability of the stack of items based at least in part on the placement of items on / in the stack of items (e.g., to reflect interactions that particular items have with other items, or interactions between other items based on the placement of particular items, etc.).

[0094] In some embodiments, the system updates the current state (e.g., based on updates to the geometric model) after (i) the movement (e.g., placement) of a predetermined number of items, or (ii) the earlier of the movement (e.g., placement) of a predetermined number of items or the detection of an anomaly (e.g., an anomaly that meets one or more anomaly criteria (e.g., the degree of anomaly exceeds an anomaly threshold, etc.)). The predetermined number of items (e.g., X items, where X is a positive integer) may be set based on user selection, robot control system policy, or otherwise determined based on empirical analysis of item placement. As an example, the predetermined number of items is set based on a determination that the number of items will yield optimal / best results with respect to a predetermined cost function (e.g., a cost function reflecting efficiency, stability, predicted changes in stability, etc.). As an example, the system determines a current estimated state and uses the current estimated state to determine a plan for moving the next X items, and after the movement (e.g., stacking or de-stacking) of the X items, the system determines an updated estimated state (e.g., geometric updates / model to reflect the placement of the X items). The system determines an updated state based at least in part on a combination of the geometric model and the sensor data (e.g., the current geometric model and the current sensor data, etc.) The system then utilizes the updated state in connection with determining a plan and controlling the robot to place the next set of items according to the plan.

[0095] In some embodiments, the frequency with which the system updates the estimated state is dynamically determined. For example, the system dynamically determines a value X corresponding to the number of items for which the system updates the estimated state after being moved. In some embodiments, the system dynamically determines the value X (e.g., corresponding to the estimated state update frequency) based at least in part on one or more attributes of the items (e.g., attributes of previously moved / placed items and / or attributes of the item being moved). As an example, the system dynamically determines the value X based on a determination that an irregularly placed item or a deformable item was placed before (e.g., immediately before) placing the set of X items using the current estimated state, or that the set of X items includes an irregularly shaped item or a deformable item.

[0096] Examples of dynamically determining / updating an estimated state include (i) controlling a robotic arm to place or remove a first set of X items on or in a pallet or other container, (ii) determining an estimated state (e.g., reflecting the placement / removal of the first set of X items), (iii) using the estimated state to generate or update a plan for controlling a robotic arm to place or remove a second set of Y items on or in a pallet or other container, and / or (iv) controlling a robotic arm to place or remove a first set of Y items on or in a pallet or other container. In some embodiments, the state of one or more items stacked on or in a pallet or other container is estimated after N items have been stacked.

[0097] Although the above examples are discussed in the context of the system palletizing a set of items onto one or more pallets, the robotic system can be utilized in connection with depalletizing a set of items from one or more pallets.

[0098] 2 is a diagram illustrating a robotic system for palletizing and / or depalletizing heterogeneous items, according to various embodiments. In some embodiments, system 200 is implemented, at least in part, by process 300 of FIG. 3, process 400 of FIG. 4, process 500 of FIG. 5, process 600 of FIG. 6, process 800 of FIG. 8, and / or process 900 of FIG. 9.

[0099] In the illustrated example, system 200 includes robotic arm 205. While robotic arm 205 is fixed in this example, in various alternative embodiments, robotic arm 205 may be fully or partially movable, e.g., mounted on rails, fully movable on a motor-driven chassis, etc. In other implementations, system 200 may include multiple robotic arms having workspaces. As shown, robotic arm 205 is utilized to pick any and / or disparate items from one or more conveyors (or other sources) 225 and 230 and items on pallets (e.g., platforms or other containers), such as pallet 210, pallet 215, and / or pallet 220. In some embodiments, other robots not shown in FIG. 2 may be used to push pallets 210, pallet 215, and / or pallet 220 into position for loading / unloading and / or into a truck or other destination for transport, etc.

[0100] As shown in FIG. 2 , system 200 may include one or more predetermined zones. For example, pallet 210, pallet 215, and pallet 220 are shown disposed within the predetermined zones. The predetermined zones may be structurally indicated by markings or labels on the ground or in another manner, such as using a frame as shown in system 200. In some embodiments, the predetermined zones may be radially arranged around robotic arm 205. In some cases, a single pallet is inserted into a predetermined zone. In other cases, one or more pallets are inserted into a predetermined zone. Each predetermined zone may be located within the range of robotic arm 205 (e.g., so that robotic arm 205 can place items onto or depalletize items from the corresponding pallet). In some embodiments, a predetermined zone or one of the pallets disposed within a predetermined zone is utilized as a buffer or staging area where items are temporarily stored (e.g., until the items are placed on a pallet in the predetermined zone).

[0101] One or more items may be provided (e.g., transported) to the workspace of the robotic arm 205, such as via conveyor 225 and / or conveyor 230. The system 200 may control the speed of conveyor 225 and / or conveyor 230. For example, the system 200 may control the speed of conveyor 225 independently of the speed of conveyor 230, or the system 200 may control the speed of both conveyor 225 and / or conveyor 230. In some embodiments, the system 200 may pause conveyor 225 and / or conveyor 230 (e.g., to allow sufficient time for the robotic arm 205 to pick and place an item). In some embodiments, the conveyor 225 and / or conveyor 230 transport items for one or more manifests (e.g., orders). For example, conveyor 225 and conveyor 230 may carry items for the same manifest and / or different manifests. Similarly, one or more of the pallets / predetermined zones may be associated with a particular manifest. For example, pallet 210 and pallet 215 may be associated with the same manifest. As another example, pallet 210 and pallet 220 may be associated with different manifests.

[0102] System 200 may control robotic arm 205 to pick items from a conveyor (e.g., conveyor 225 or conveyor 230) and place the items on a pallet (e.g., pallet 210, pallet 215, or pallet 220). Robotic arm 205 may pick items and move the items to corresponding destination locations (e.g., a location on a pallet or a stack on a pallet) based at least in part on a plan associated with the items. In some embodiments, system 200 may determine a plan associated with an item, such as while the item is on the conveyor, and system 200 may update the plan upon picking up the item (e.g., based on acquired attributes of the item (e.g., weight) or in response to information acquired by sensors in the workspace, such as an indication of an expected collision with another item or person). System 200 may acquire an identifier associated with the item (e.g., a barcode, QR code, or other identifier or information on the item). For example, system 200 may scan / acquire an identifier for an item as it is transported on the conveyor. In response to acquiring the identifier, system 200 may utilize the identifier in connection with determining a pallet on which the item will be placed, such as by performing a lookup on a mapping of item identifiers to manifests and / or a mapping of manifests to pallets. In response to determining one or more pallets corresponding to the manifest / order to which the item belongs, system 200 may select a pallet on which to place the item based at least in part on a model or simulation of the stack of items on the pallet and / or the placement of the item on the pallet. System 200 may also determine a specific location (e.g., destination location) where the item will be placed on the selected pallet. Additionally, a plan for moving the item to the destination location may be determined, including a planned path or trajectory along which the item may be moved.In some embodiments, the plan is updated as the robotic arm 205 is moving items, such as in connection with taking proactive measures to change or adapt to detected states or conditions associated with one or more items / objects in the workspace (e.g., to avoid anticipated collision events, to account for an item's measured weight being greater than its predicted weight, to reduce shear forces on the item as it is moved, etc.).

[0103] According to various embodiments, system 200 includes one or more sensors and / or sensor arrays. For example, system 200 may include one or more sensors (such as sensor 240 and / or sensor 241) near conveyor 225 and / or conveyor 230. The one or more sensors may acquire information related to items on the conveyor, such as identifiers or information on the item's label or item attributes such as the item's dimensions. In some embodiments, system 200 includes one or more sensors and / or sensor arrays that acquire information related to a predetermined zone and / or pallets within the zone. For example, system 200 may include sensor 242 that acquires information related to pallet 220 or a predetermined zone in which pallet 220 is located. The sensors may include one or more 2D cameras, 3D (e.g., RGBD) cameras, infrared, and other sensors for generating a three-dimensional view of the workspace (or a portion of the workspace, such as the pallet and stacks of items on the pallet). The information about the pallet may be utilized in connection with determining the status of the pallet and / or the stack of items on the pallet. As one example, the system 200 may generate a model of the stack of items on the pallet based at least in part on the information about the pallet. The system 200 may then utilize the model in connection with determining a plan for placing the items on the pallet. As another example, the system 200 may determine that the stack of items is complete based at least in part on the information about the pallet.

[0104] According to various embodiments, system 200 determines (or updates) a plan for picking and placing items based at least in part on a determination of the stability of stacks on the pallet. System 200 may determine a model of the stacks for one or more of pallets 210, 215, and / or 220, and system 200 may utilize the model in connection with determining the stack in which to place the item. As an example, if the next item to be moved is relatively large (e.g., the item has a large surface area relative to the footprint of the pallet), system 200 may determine that placing the item on pallet 210 may result in an unstable stack thereon (e.g., due to a non-planar surface of the stack). In contrast, system 200 may determine that placing a relatively large (e.g., flat) item on a stack on pallet 215 and / or pallet 220 may result in a relatively stable stack. The top surface of the stack for pallet 215 and / or pallet 220 is relatively flat, and placing a relatively large item thereon may not lead to stack instability. System 200 may determine that the predicted stability of placing the item on pallet 215 and / or pallet 220 may be greater than a predetermined stability threshold, or that placing the item on pallet 215 or pallet 220 may result in optimal placement (e.g., at least in terms of stability) of the item.

[0105] System 200 may communicate the status of pallets within a given zone and / or the status of the operation of robotic arm 205. The status of the pallets and / or the status of the operation of the robotic arm may be communicated to a user or other human operator. For example, system 200 may include a communication interface (not shown) through which information regarding the status of system 200 (e.g., the status of pallets, given zones, robotic arms, etc.) is communicated to a terminal (e.g., an on-demand remote control and / or a terminal utilized by a human operator). As another example, system 200 may include a status indicator (e.g., status indicator 245 and / or status indicator 250) located near a given zone.

[0106] Status indicators 250 may be utilized in connection with communicating the status of a pallet within a corresponding predetermined zone and / or the status of the operation of robotic arm 205. For example, if system 200 is active for a predetermined zone in which pallet 220 is located, the status indicator may indicate so by illuminating a green light or otherwise communicating information or indication of the active status via status indicator 250. System 200 may be determined to be active for a predetermined zone in response to determining that robotic arm 205 is actively palletizing one or more items on a pallet within the predetermined zone. As another example, if system 200 is inactive for a predetermined zone in which pallet 220 is located, the status indicator may indicate so by illuminating a red light or otherwise communicating information or indication of the active status via status indicator 250. The system 200 may be determined to be inactive in response to a determination that the robotic arm 205 is not actively palletizing one or more items on a pallet in a given zone (e.g., a user has paused the given zone (or cell)) or in response to a determination that palletization of items on the pallet 220 is complete. A human operator or user may utilize the status indicator as an indication of whether it is safe to enter the corresponding given zone in advance. For example, a user working to remove a completed pallet or insert an empty pallet into or from a corresponding given zone may refer to the corresponding status indicator and safely enter the given zone when the status indicator indicates that operation within the given zone is inactive.

[0107] According to various embodiments, system 200 may use information obtained by one or more sensors in the workspace to determine an abnormal condition related to the pallet and / or items stacked on the pallet. For example, system 200 may determine that the pallet is out of alignment with robotic arm 205 and / or a corresponding predetermined zone based at least in part on information obtained by the sensors. As another example, system 200 may determine that a stack is unstable, that items on the pallet are experiencing turbulence, etc. based at least in part on information obtained by the sensors. In response to detecting an abnormal condition, the system may communicate an indication of the abnormal condition, such as to an on-demand remote control or other terminal utilized by an operator. In some embodiments, in response to detecting an abnormal condition, system 200 may automatically set the pallet and / or the corresponding zone to an inactive state. In addition to or instead of notifying an operator of the abnormal condition, system 200 may take proactive measures. The proactive measures may include controlling robotic arm 205 to at least partially correct the abnormal condition (e.g., re-stack a fallen item, re-align the pallet, etc.). In some implementations, in response to detecting that an inserted pallet is misaligned (e.g., incorrectly inserted in a predetermined zone), system 200 may calibrate the process for modeling the stack and / or the process for placing items on the pallet to correct the misalignment. For example, system 200 may generate and utilize an offset corresponding to the misalignment when determining and implementing a plan for placing items on the pallet. In some embodiments, in response to determining that the degree of the abnormality is less than a threshold, system 200 takes proactive measures to partially correct the abnormal condition.Examples of determining that the degree of abnormality is less than a threshold include (i) determining that the pallet misalignment is less than a threshold misalignment value, (ii) determining that the number of misaligned, misplaced, or dropped items is less than a threshold number, and (iii) determining that the size of the misaligned, misplaced, or dropped items meets a size threshold.

[0108] A human operator may communicate with system 200 via a network (e.g., a wired network and / or a wireless network). For example, system 200 may include a communication interface for connecting system 200 to one or more networks. In some embodiments, a terminal connected to system 200 via a network provides a user interface through which the human operator can provide instructions to system 200 and / or obtain information regarding the status of system 200 (e.g., the status of the robotic arm, the status of a particular pallet, the status of the palletization process for a particular manifest, etc.). The human operator may provide instructions to system 200 by inputting to the user interface. For example, the human operator may use the user interface to pause the robotic arm, pause the palletization process for a particular manifest, pause the palletization process for a particular pallet, toggle the status of a pallet / certain zone between active and inactive, etc.

[0109] In various embodiments, elements of system 200 may be added, removed, replaced, etc. In such an example, the control computer initializes and registers the new element, performs operational tests, and initiates / resumes kitting operations to incorporate, for example, the newly added element.

[0110] According to various embodiments, system 200 determines (e.g., calculates, maintains, stores, etc.) an estimated state for each pallet (e.g., pallet 210, pallet 215, and / or pallet 220) within multiple zones, or an aggregate estimated state for a set of pallets among multiple zones, or both individual and aggregate estimated states. In some embodiments, the individual and aggregate estimated states are determined similarly to the estimated states described in connection with robotic system 100 of FIG. 1.

[0111] According to various embodiments, system 200 includes a vision system with one or more sensors (e.g., sensor 240, sensor 241, etc.). In various embodiments, system 200 utilizes sensor data and geometric data (e.g., a geometric model) in connection with determining a location to place one or more items on a pallet (or in connection with depalletizing one or more items from a pallet). System 200 uses different data sources to model the condition of a pallet (or a stack of items on a pallet). For example, system 200 may estimate the location of one or more items on a pallet and one or more characteristics (or attributes) associated with the one or more items (e.g., item size). The one or more characteristics associated with the one or more items may include item size (e.g., item dimensions), center of gravity, item stiffness, package type, identifier location, etc.

[0112] The system 200 determines the geometric model based at least in part on one or more attributes of one or more items in the workspace. For example, the geometric model reflects attributes of each of a set of items (e.g., one or more of a first set that are palletized / stacked and a second set of items to be palletized / stacked, etc.). Examples of items include item size (e.g., item dimensions), center of gravity, item rigidity, packaging type, identifier location, item deformability, item shape, etc. Various other attributes of items or objects in the workspace may also be implemented.

[0113] The model generated by system 200 may correspond to or be based at least in part on a geometric model. In some embodiments, system 200 generates the geometric model based at least in part on one or more placed items (e.g., items placed by system 200 controlling robotic arm 205), one or more attributes associated with at least a portion of one or more items, one or more objects in the workspace (e.g., predetermined objects such as pallets, robotic arms, shelving systems, chutes, or other infrastructure included in the workspace), etc. The geometric model may be determined at least in part based on executing a physics engine on the control computer to model the stacking of items (e.g., modeling the state / stability of a stack of items, etc.). The geometric model may be determined based on predicted interactions of various components of the workspace (e.g., interactions of items with other items, objects, simulated forces applied to the stack (e.g., to model the use of a forklift or other equipment to lift / move a pallet or other container on which the stack of items is located), etc.

[0114] According to various embodiments, system 200 uses the geometric model and sensor data to determine a best estimate of the state of the workspace. System 200 can adjust for noise (e.g., noise cancellation) in one or more of the geometric model and / or the sensor data. In some embodiments, system 200 detects anomalies or discrepancies between the state according to the geometric model and the state according to the sensor data. In response to determining that there is an anomaly or discrepancy between the geometric model and the sensor data, system 200 can make a best estimate of the state despite the anomaly or discrepancy. For example, system 200 determines whether to use the geometric model or the sensor data, or a combination of the geometric model and the sensor data (e.g., interpolation therebetween). In some embodiments, system 200 determines the estimated state on a segment-by-segment basis (e.g., for each voxel in the workspace, each item, or each object, etc.). For example, a first portion of the workspace may be estimated using only the geometric model, a second portion of the workspace may be estimated using only the sensor data (e.g., if there is an anomaly in the geometric model), and / or a third portion of the workspace may be estimated based on a combination of the geometric model and the sensor data. Using the example shown in Figure 2, system 200 may, in connection with determining the collective estimated state, utilize only the geometric model to determine individual estimated states for the stacks of items on pallet 210, utilize only the sensor data to determine individual estimated states for the stacks of items on pallet 215, and utilize a combination of the respective geometric model and sensor data for the stacks of items on pallet 220.

[0115] Although the above examples are discussed in the context of the system palletizing a set of items onto one or more pallets, the robotic system can be utilized in connection with depalletizing a set of items from one or more pallets.

[0116] 3 is a flowchart illustrating a process for palletizing one or more items, according to various embodiments. In some embodiments, process 300 is performed, at least in part, by robotic system 100 of FIG. 1 and / or system 200 of FIG. 2.

[0117] At step 310, a list of items is obtained. The list of items may correspond to a set of items to be palletized together onto one or more pallets. According to various embodiments, the set of items to be palletized is determined at least in part based on a manifest or instructions for fulfilling an order. For example, in response to receiving an order, a list of items for the order may be generated. As another example, a list of items may be generated corresponding to multiple orders to be sent to the same recipient.

[0118] The items may be located on shelves or other locations within the warehouse. To palletize the items, the items are moved to a robotic system that palletizes the items. For example, the items may be placed on one or more conveyors that move the items into range of one or more robotic arms that palletize the items onto one or more pallets. In response to obtaining the list of items, at least some of the items are associated with a particular robotic arm, a predetermined zone corresponding to the particular robotic arm, and / or a particular pallet (e.g., a pallet identifier, a pallet located in a predetermined zone), etc.

[0119] At step 320, planning (or re-planning) is performed to generate a plan for picking and placing items based on the list of items and available sensor information. The plan may include one or more strategies for fetching one or more items on the list of items and placing such items on corresponding one or more conveyors for transporting the items to the robotic arms. According to various embodiments, the order in which items on the list of items are presented to appropriate robotic arms for palletizing is determined based at least in part on the list of items.

[0120] The order in which items are placed on a conveyor may be based at least loosely on the items and the expected stack of the items on one or more pallets. For example, a system that determines the order in which items are placed on a conveyor may generate a model of the expected stack of items and determine the order based on the model (e.g., to transport items that will form the base / bottom of the stack first and incrementally transport items toward the top of the stack). If the items on the list of items are palletized onto multiple pallets, items that are expected to form the base / bottom of each stack (or are otherwise relatively close to the bottom of the stack) may be placed on the conveyor before items that are expected to be substantially in the middle or at the top of the stack. Various items palletized onto multiple pallets may be interspersed with one another, and the robotic system may sort the items upon arrival at the robotic arm (e.g., the robotic arm may pick and place items on the appropriate pallet based at least on the items (e.g., item identifiers or item attributes)). Thus, corresponding items at the base / bottom of corresponding stacks may be interspersed with one another, and the various items for each pallet / stack may be placed on a conveyor as the corresponding stack is built.

[0121] The computer system may generate one or more predictive stacking models for items in the list of items. The models may be generated based at least in part on one or more thresholds (e.g., a fitness threshold or a stability threshold, other packing metrics (e.g., density), etc.). For example, the computer system may generate models for stacks whose predicted stability values ​​meet (e.g., exceed) the stability threshold. The models may be generated using a machine learning process. The machine learning process may be iteratively updated based on historical information, such as previous stacks of items (e.g., attributes of items in previous stacks, performance metrics (stability, density, fitness, etc.) for previous stacks). In some embodiments, the stacking models for palletizing items on the list of items are generated based at least in part on one or more attributes of the items.

[0122] Various attributes of the items may be obtained before or during determining the plan. The attributes may include the size of the item, the shape of the item, the type of packaging for the item, an identifier for the item, the center of gravity of the item, an indication of whether the item is fragile, an indication of the top or bottom of the item, etc. As an example, one or more attributes associated with at least some of the items may be obtained based at least in part on a list of items. The one or more attributes may be obtained based at least in part on information obtained by one or more sensors and / or by performing a lookup in a mapping of attributes to items (e.g., item type, item identifier (serial number, model number, etc.)).

[0123] In some embodiments, generating a model of one or more predictive statistics for items belonging to the list of items includes generating (e.g., determining) an estimated state for a workspace (e.g., a workspace including one or more stacks of items). A computer system determines a plan for moving (e.g., palletizing or depalletizing, etc.) one or more sets of items, and the computer system controls a robot (e.g., a robotic arm) to move the one or more sets of items according to the plan. In response to moving the one or more sets of items according to the plan, the computer system determines an estimated state of the workspace. For example, the computer system updates the estimated state based at least in part on the movement of the sets of items. In some embodiments, the estimated state is determined based at least in part on the geometric model or the sensor data, or a combination of the geometric model and the sensor data, in response to determining that the geometric model and the sensor data are inconsistent (e.g., a difference between the geometric model and the sensor data is greater than a predetermined difference threshold, or includes an anomaly, etc.). The updated / current estimated state reflects the movement of the set of one or more items (e.g., in the case of palletizing, the updated estimated state includes information about the placement of the set of one or more items on a stack, etc.). In response to determining the updated / current estimated state, the computer system determines a plan for moving another set of one or more items, and the computer system controls the robot to move the another set of one or more items according to the plan.

[0124] In some embodiments, the computer system updates the current state (e.g., based on updates to the geometric model) after (i) the movement (e.g., placement) of a predetermined number of items, or (ii) the earlier of the movement (e.g., placement) of a predetermined number of items or the detection of an anomaly (e.g., an anomaly that meets one or more anomaly criteria (e.g., the degree of anomaly exceeds an anomaly threshold, etc.)). The predetermined number of items (e.g., X items, where X is a positive integer) may be set based on user selection, robot control system policy, or otherwise determined based on empirical analysis of item placement. As an example, the predetermined number of items is set based on a determination that the number of items will yield optimal / best results with respect to a predetermined cost function (e.g., a cost function reflecting efficiency, stability, predicted changes in stability, etc.). As an example, the computer system determines a current estimated state and uses the current estimated state to determine a plan for moving the next X items, and after the movement (e.g., stacking or de-stacking) of the X items, the computer system determines an updated estimated state (e.g., geometric updates / model to reflect the placement of the X items). The computer system determines an updated state based at least in part on a combination of the geometric model and the sensor data (e.g., the current geometric model and the current sensor data, etc.) The computer system then utilizes the updated state in connection with determining a plan and controlling the robot to place the next set of items according to the plan.

[0125] According to various embodiments, the computer system determines the estimated state based at least in part on performing an interpolation between the geometric model and the sensor data. For example, the system performs an interpolation between a particular portion of the geometric model and a corresponding portion of the sensor data (e.g., the particular portion may correspond to a difference between the geometric model and the sensor data that exceeds a difference threshold or includes an anomaly).

[0126] Various interpolation techniques may be implemented. A particular portion of the geometric model may correspond to a particular point (or set of points) in the point cloud for the geometric model, the corresponding portion of the sensor data may be the sensor data for that particular point in the point cloud for the sensor data, etc. In some embodiments, the system performs adaptive interpolation between the geometric model and the sensor data. In some embodiments, the system performs non-adaptive interpolation between the geometric model and the sensor data. Examples of adaptive interpolation processes include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, spline interpolation, sinc interpolation, Lanczos interpolation, etc. Various other interpolation processes may be performed in connection with determining the estimated state.

[0127] In step 330, the item is picked according to the plan determined and / or updated in step 320, moved through a (predetermined / planned) trajectory to a location near where the item will be placed on the corresponding conveyor, and placed at the destination location.

[0128] In the illustrated example, (re)planning and plan execution (steps 320, 330) continues until the high-level goal of providing an item on the list of items is completed (step 340), at which point process 300 ends. In various embodiments, replanning (step 320) may be triggered by conditions such as the arrival of an unexpected and / or unidentifiable item, a sensor reading indicating an attribute having a value other than that predicted based on the item identification and / or associated item model information, etc. Other examples of unexpected conditions include, but are not limited to, determining that an expected item is missing, reevaluating an item identification to determine that the item is other than that originally identified, detecting an item weight or other attribute that does not match the identified item, dropping an item or needing to be re-grasped, determining that a later-arriving item is too heavy to be stacked on one or more other items as planned by the original plan and / or the current plan, and detecting instability in a set of items stacked in a container.

[0129] 4 is a flowchart illustrating a process for determining the current state of a pallet and / or stack of items, according to various embodiments. In some embodiments, process 400 is performed, at least in part, by robotic system 100 of FIG. 1 and / or system 200 of FIG. 2.

[0130] According to various embodiments, process 400 is invoked in response to a determination that the system determines a plan for moving one or more items. In some embodiments, process 400 is invoked for the placement of each item in a set of items to be stacked. In some embodiments, process 400 is invoked at a predetermined frequency / interval, such as after a predetermined number of items have been moved since the last determination of the estimated state. In addition to any of the above, the system may invoke process 400 based at least in part on attributes of a previously placed item or a currently placed item. For example, process 400 is invoked in response to a determination that a previously placed item or a current item has an irregular shape (e.g., a size / dimension exceeding a threshold size, a non-rectangular shape, etc.) and / or is deformable (e.g., the item has an expected deformability exceeding a deformability threshold, or the item has soft packaging (e.g., a polybag), etc.). As another example, the system invokes process 400 in response to a determination that a previously placed item or a current item to be placed may cause instability (e.g., a threshold instability) within the stack of items.

[0131] At step 410, information related to the placed item is obtained. The system obtains the information related to the item based on pre-stored information related to the item (e.g., if the item is known in advance to be palletized, such as on a manifest), or based on information obtained by the vision system (e.g., item identifier, size, type of packaging, weight, etc.). The information related to the placed item may correspond to attributes of the item, etc.

[0132] In step 420, geometric data is received for the stack of items, including the placed item. In some embodiments, the system obtains a current geometric model that is updated to reflect the placement of the item. The current geometric model may be an estimated state utilized in connection with determining the placement of the item or the placement of a set of items to which the item belongs (e.g., a predicted placement based on the robot's movement to place the item according to a plan).

[0133] At step 430, sensor data acquired by the vision system is acquired. In some embodiments, the system acquires the sensor data acquired by the vision system. For example, the system instructs the vision system to capture the current state of the workspace, and the system uses information about such capture to acquire the sensor data.

[0134] At step 440, the current state or model is updated based at least in part on the geometric data and the sensor data. The system determines a predicted state based at least in part on the geometric model or the sensor data or both the geometric model and the sensor data, such as in response to determining differences between the geometric model and the sensor data.

[0135] According to various embodiments, the system determines the estimated state (e.g., an updated estimated state) at least in part by performing interpolation based on at least a portion of the geometric data and at least a portion of the sensor data. In some embodiments, the interpolation process performed on the first portion of the geometric model and the first portion of the sensor data to obtain the first portion of the estimated state is different from the interpolation performed on the second portion of the geometric model and the second portion of the sensor data to obtain the second portion of the estimated state.

[0136] In some embodiments, the system determines a plan for moving one or more items and utilizes the updated estimated state in connection with controlling the robot to move the one or more items in accordance with the plan.

[0137] At step 450, a determination is made as to whether process 400 is complete. In some embodiments, process 400 is determined to be complete in response to a determination that no further updates of the estimated state are to be performed, no further items are to be moved, a user has exited the system, an administrator has indicated that process 400 should be paused or stopped, etc. In response to a determination that process 400 is complete, process 400 ends. In response to a determination that process 400 is not complete, process 400 returns to step 410.

[0138] 5 is a flowchart illustrating a process for determining the current state of a pallet and / or stack of items, according to various embodiments. In some embodiments, process 500 is performed, at least in part, by robotic system 100 of FIG. 1 and / or system 200 of FIG. 2.

[0139] According to various embodiments, process 500 is invoked in response to a determination that the system determines a plan for moving one or more items. In some embodiments, process 500 is invoked with respect to the placement of each item in a set of items to be stacked. In some embodiments, process 500 is invoked at a predetermined frequency / interval, such as after a predetermined number of items have been moved since the last determination of the estimated state. In addition to any of the above, the system may invoke process 500 based at least in part on attributes of a previously placed item or a currently placed item. For example, process 500 is invoked in response to a determination that a previously placed item or a current item has an irregular shape (e.g., a size / dimension exceeding a threshold size, a non-rectangular shape, etc.) and / or is deformable (e.g., the item has an expected deformability exceeding a deformability threshold, or the item has soft packaging (e.g., a polybag), etc.). As another example, the system invokes process 500 in response to a determination that a previously placed item or a current item to be placed may cause instability (e.g., a threshold instability) within the stack of items.

[0140] According to various embodiments, process 500 is performed by one or more servers that service one or more robotic systems that move items (e.g., palletization / depalletization systems, singulation systems, kitting systems, etc.). For example, the system receives queries from one or more robotic systems and provides estimated states. The system may maintain a current estimated state of one or more stacks of items (e.g., based on the robotic systems' actions picking and placing items relative to the stacks of items and / or based on sensor data).

[0141] At step 510, information associated with the placed item is received. In some embodiments, the system obtains the information associated with the item based on pre-stored information associated with the item. For example, the system performs a lookup regarding a mapping of attributes to the item (e.g., based on an identifier associated with the item). In some embodiments, the information associated with the placed item is received based at least in part on item attributes (e.g., item or package stiffness / deformability, size, weight, etc.) obtained by one or more sensors (e.g., a vision system) when placing the item.

[0142] In step 520, geometric data about the stack of items, including the placed item, is received. In some embodiments, the system obtains a current geometric model that is updated to reflect the placement of the items. The current geometric model may correspond to a predicted state of the stack of items (e.g., the state of the stack of items where the system estimates that the items were placed correctly according to the plan, etc.). For example, the system obtains a geometric model used in connection with determining a plan for moving the placed item (or the set of items to which the placed item belongs), and the system updates the geometric model to reflect the placement of the placed item.

[0143] At step 530, sensor data acquired by a vision system is acquired. In some embodiments, the system acquires sensor data acquired by a vision system. For example, the system instructs the vision system to capture the current state of the workspace, and the system uses information about such capture to acquire the sensor data.

[0144] At step 540, the current state or model is updated based at least in part on the geometric data and the sensor data. The system determines a predicted state based at least in part on the geometric model or the sensor data or both the geometric model and the sensor data, such as in response to determining differences between the geometric model and the sensor data.

[0145] According to various embodiments, the system determines the estimated state (e.g., an updated estimated state) at least in part by performing interpolation based on at least a portion of the geometric data and at least a portion of the sensor data. In some embodiments, the interpolation process performed on the first portion of the geometric model and the first portion of the sensor data to obtain the first portion of the estimated state is different from the interpolation performed on the second portion of the geometric model and the second portion of the sensor data to obtain the second portion of the estimated state.

[0146] In some embodiments, the updated estimated state is utilized in connection with determining a plan for moving one or more items and controlling the robot to move the one or more items according to the plan.

[0147] At step 550, the updated current state or model is provided to the robotic system. In some embodiments, in response to determining the estimated state (e.g., an initial estimated state or an updated estimated state based on a previous pick / place of the item), the system provides the estimated state to the robotic system, which uses the estimated state to determine a plan for moving one or more other items. In some embodiments, the system provides the estimated state in response to a request from the robotic system. For example, the system includes one or more servers (e.g., remote servers) that provide services to one or more robotic systems that move items (e.g., a palletization / depalletization system, a singulation system, a kitting system, etc.).

[0148] At step 560, a determination is made as to whether process 500 is complete. In some embodiments, process 500 is determined to be complete in response to a determination that no further updates of the estimated state are to be performed, no further items are to be moved, a user has exited the system, an administrator has indicated that process 500 should be paused or stopped, etc. In response to a determination that process 500 is complete, process 500 ends. In response to a determination that process 500 is not complete, process 500 returns to step 510.

[0149] 6 is a flowchart illustrating a process for determining the current state of a pallet and / or stack of items, according to various embodiments. In some embodiments, process 600 is performed, at least in part, by robotic system 100 of FIG. 1 and / or system 200 of FIG. 2.

[0150] At step 610, a model of the state is obtained based on the geometric data. In some embodiments, the system obtains a current geometric model of the system. For example, the system updates the previous geometric model to include item movements (e.g., placing items into stacks, removing items from stacks, etc.) that the system has performed by controlling the robotic arm since the previous geometric model / estimated state was determined.

[0151] A model of the state is obtained based on the sensor data in step 620. The system obtains the sensor data based on information captured by one or more sensors (e.g., a vision system) provided in the workspace.

[0152] At step 630, differences between the model based on the geometric data and the model based on the sensor data are determined. The system calculates the differences between the geometric model and the sensor data, or qualitatively determines the differences between geometric center sensor data, etc. For example, the system calculates the differences between the positions / boundaries of various items in the stack of items. As another example, the system determines whether an item is included in one of the geometric model and the sensor data and is missing from the other of the geometric model and the sensor data, such as when the vision system's field of view is obstructed or when an item falls out of the stack of items (e.g., not expected or reflected in the geometric model).

[0153] At step 640, a determination is made whether the difference exceeds a predetermined threshold. In some embodiments, the system compares the difference between the geometric model and the sensor data to a predetermined difference threshold. The predetermined difference threshold may correspond to anomalies that are not expected to be resolved by interpolation or combination of the geometric and sensor data (e.g., anomalies that the system determines require human intervention).

[0154] In response to determining in step 640 that the difference does not exceed the predetermined threshold, process 600 proceeds to step 650 where the current model or state is updated based at least in part on the difference between the model determined based on the geometric data and the model determined based on the sensor data.

[0155] In response to determining in step 640 that the difference exceeds a predetermined threshold, process 600 proceeds to step 660, where a suggestion is provided to the user. For example, in response to determining that the difference exceeds a predetermined threshold, the system decides to require manual human intervention. The system can prompt the user to confirm whether to use the geometric model or the sensor data, a combination of the geometric model and the sensor data, or neither (e.g., to utilize user-defined aspects of the r-model, etc.).

[0156] In some embodiments, the indication provided to the user is an indication of a difference between the geometric model and a model based on the sensor data for a portion of the workspace (such as a portion of a stack of items). For example, the indication may indicate a particular portion of the stack of items where the difference between the geometric model portion and the sensor data exceeds a predetermined threshold.

[0157] In some embodiments, the suggestion provided to the user is a suggestion of differences for the entire workspace (e.g., the entire geometric model and the entire sensor data). For example, the system prompts the user to clarify anomalies that appear between the geometric model and the sensor data for the workspace. As another example, the system prompts the user to define biases or weightings to be applied with respect to the geometric model and the sensor data. As a further example, the system prompts the user to define an interpolation process to be performed in connection with determining an estimated state based on both the geometric model and the sensor data.

[0158] At step 670, input is received from a user. The user input can be a selection to utilize a geometric model or sensor data or a combination of a geometric model and sensor data. In some embodiments, the user defines whether to utilize a geometric model or sensor data for a particular portion of the workspace state (e.g., the top of a stack of items, a portion of a stack of items where the vision system's view is obstructed, etc.).

[0159] At step 680, the current state / model is updated based at least in part on user input, for example, the estimated state is updated to reflect a user selection of whether to use the geometric model, the sensor data, or a combination of the geometric model and the sensor data as the ground truth for the current state / model.

[0160] At step 690, a determination is made as to whether process 600 is complete. In some embodiments, process 600 is determined to be complete in response to a determination that no further updates of the estimated state are to be performed, no further items are to be moved, a user has exited the system, an administrator has indicated that process 600 should be paused or stopped, etc. In response to a determination that process 600 is complete, process 600 ends. In response to a determination that process 600 is not complete, process 600 returns to step 610.

[0161] Figures 7A and 7B are diagrams illustrating the state of a pallet / stack of items using geometric data and vision system or sensor data, respectively, in accordance with various embodiments.

[0162] The state 700 corresponds to a state of a stack of items according to geometric data. For example, the state 700 corresponds to a geometric model of the stack of items. As shown in FIG. 7A , the geometric model includes correctly placed items. The geometric model may correspond to an ideal state of the stack of items if the robotic arm picks / places items relative to the stack of items according to a plan (e.g., without deviating from the modeled plan).

[0163] State 750 corresponds to the state of a stack of items according to sensor data. For example, state 750 corresponds to a stack of items based on information captured by a vision system included in the workspace. As shown in FIG. 7B, state 750 includes incorrectly positioned items or portions of the stack of items for which no information is provided (e.g., portions of the stack that are obscured from the vision system's view, etc.).

[0164] As shown in FIG. 7A , state 700 includes item 704a stacked on top of a set of items including items 706a, 708a, 710a, 712a, and 714a. However, state 750 includes a gap 752 where the sensor data does not reflect the presence of an item. As an example, gaps in the stack of items may be included in the sensor data if the sensor data is very noisy or if the vision system's view of that portion of the workspace is obstructed. In other words, the sensor data in state 750 does not include information about items 708a, 710a, 712a, and 714a that are included in the geometric model. If sensor data were used rather than a geometric model or had a strong bias, the system may determine that the stack of items is unstable based on item 704b being placed on top of an item that is not supported by gap 752.

[0165] State 700 includes item 706a, which appears to be precisely positioned relative to various other items in the stack of items. In contrast, state 750 includes item 706b, which is not precisely / straightly positioned on the item below it. For example, sensor data reflects the space between item 706b and an adjacent item. Examples of causes of the location / positioning of item 706b include (i) movement of the item after placement by the robotic arm or inaccurate placement of the item by the robotic arm (e.g., relative to a plan determined according to the current estimated state), (ii) inaccurate placement of the item by the robotic arm, (iii) noise in the sensor data, and / or (iv) inaccuracies in the sensor data.

[0166] State 700 includes item 716a, which appears to be precisely positioned relative to various other items in the stack of items. In contrast, state 750 includes item 716b, which is not precisely positioned relative to adjacent items. For example, sensor data reflects a gap 754 between item 716b and an adjacent item. Examples of causes of the location / positioning of item 716b include (i) movement of the item after placement by the robotic arm or inaccurate placement of the item by the robotic arm (e.g., relative to a plan determined according to the current estimated state), (ii) inaccurate placement of the item by the robotic arm, (iii) noise in the sensor data, and / or (iv) inaccuracies in the sensor data.

[0167] State 700 includes item 718a, which appears to be correctly positioned relative to various other items in the stack of items. In contrast, state 750 includes a gap 758 in the stack at the same location where the geometric model includes item 718a. For example, sensor data reflects gap 758 below item 702b and an adjacent item. Example causes of gap 758 include (i) movement of the item after placement by the robotic arm or imprecise placement of the item by the robotic arm (e.g., relative to a plan determined according to the current estimated state), (ii) imprecise placement of the item by the robotic arm, (iii) noise in the sensor data, and / or (iv) inaccuracies in the sensor data. If the system uses sensor data for the portion of the stack of items corresponding to gap 758, the system may consider the stack of items to be unstable. For example, the system may consider item 702b to be partially unsupported and unstable.

[0168] State 700 includes item 702a, which appears to be positioned correctly relative to various other items in the stack of items. In contrast, state 750 includes item 702b, which is not positioned correctly / straight on the item below it. For example, sensor data reflects misalignment (e.g., overhang) of item 702b relative to the item below it. Example causes of misalignment 756 include (i) movement of the item after placement by the robotic arm or incorrect placement of the item by the robotic arm (e.g., relative to a plan determined according to the current estimated state), (ii) incorrect placement of the item by the robotic arm, (iii) noise in the sensor data, and / or (iv) inaccuracies in the sensor data.

[0169] FIG. 7C illustrates a diagram showing the state of a pallet / stack of items using geometric data and vision system or sensor data, according to various embodiments.

[0170] As shown in FIG. 7C , the estimated state 775 includes a state of the stack of items based on a combination of the geometric model (e.g., state 700) and the sensor data (e.g., state 750). In some embodiments, the system determines differences between the geometric model and the sensor data and determines how to reflect the differences in the estimated state. For example, with respect to differences in a particular portion of the stack of items, the system determines whether to (i) use only the geometric model to model the particular portion of the stack in the estimated state, (ii) use only the sensor data to model the particular portion of the stack in the estimated state, or (iii) use a combination of the geometric model and the sensor data. In response to determining to use a combination of the geometric model and the sensor data for a particular portion of the estimated state, the system determines how to combine the geometric model and the sensor data. For example, the system determines an interpolation process. The system may determine how to combine the geometric model and the sensor data based at least in part on particular differences between the geometric model and the sensor data (e.g., the system may determine that the differences are due to noise in the sensor data, an obstructed field of view of a vision system, etc.).

[0171] Inferred state 775 includes item 704c, where corresponding item 704a in state 700 is shown to be located exactly next to item 702a, and corresponding item 704b in state 750 is shown to be located a distance from item 702b.

[0172] In some embodiments, the system determines that the difference in the apparent positions of items 704a and 704b is due to one or more of: (i) movement of the items after placement by the robot arm or inaccurate placement of the items by the robot arm (e.g., relative to a plan determined according to the current estimated state); (ii) inaccurate placement of the items by the robot arm; (iii) noise in the sensor data; and / or (iv) inaccuracies in the sensor data. The system may also determine other causes of the difference. In response to determining the likely cause of the difference in the positions of items 704a and 704b, the system determines how to model item 704c. For example, the system determines whether to (i) use only a geometric model to model the specific portion of the stack in the estimated state, (ii) use only sensor data to model the specific portion of the stack in the estimated state, or (iii) use a combination of a geometric model and sensor data. In some embodiments, the system determines how to model item 704c based at least in part on the determined probable causes of the difference in the locations of items 704a and 704b.

[0173] In some embodiments, the system determines to model item 704c based on a combination of the geometric model and the sensor data. For example, the location of item 704c is determined based on interpolation of the geometric model (e.g., the location of item 704a) and the sensor data (e.g., the location of item 704b). As another example, the location of item 704c is determined as the midpoint between the geometric model and the sensor data.

[0174] In some embodiments, the system determines that the difference in the positions of items 704a and 704b is considered statistically relevant but not perfect. For example, the system attributes the difference in the positions of items 704a and 704b to inaccuracies in the placement of the items. Thus, the system determines to use the position of item 704b as the position of item 704c in the estimated state.

[0175] The estimated state 775 includes item 718c. While the corresponding item 718a in state 700 is shown positioned correctly below item 702a, state 750 has a gap 758 where item 718a was included in state 700. The system determines the differences between states 700 and 750 (e.g., the presence of item 718a and the presence of gap 758), and the system determines how to model that corresponding portion of the stack in the estimated state. In the illustrated example, the system determines that the sensor data is inaccurate (e.g., because the data includes gap 758 where item 718c would be placed). For example, the system may determine that the vision system's view of that portion of the stack is obstructed. Thus, the system determines to utilize information about item 718a in the geometric model in connection with modeling the position of item 718c in the estimated state. For example, the system determines to utilize only the geometric model (and not combine corresponding information, such as sensor data).

[0176] Estimated state 775 includes items 708c, 710c, 712c, and 714c. While corresponding items 708a, 710a, 712a, and 714a in state 700 are shown positioned precisely below item 704a, state 750 has gaps 752 at the locations of such items. The system determines differences between states 700 and 750 (e.g., the presence of item 718a and the presence of gap 758), and the system determines how to model such corresponding portions of the stack in the estimated state. In the illustrated example, the system determines that the sensor data is inaccurate (e.g., because the data includes gaps 752 where items 708a, 710a, 712a, and 714a would be positioned). For example, the system may determine that the vision system's view of such portions of the stack is obstructed. The system may determine that the sensor data is inconsistent because the sensor data (e.g., state 750 in FIG. 7B ) includes item 704b positioned above the set of items without instability resulting from gap 752. Therefore, the system may decide to use information about items 708a, 710a, 712a, and 714a in the geometric model in connection with modeling the positions of items 708c, 710c, 712c, and 714c in the estimated state. For example, the system may decide to use only the geometric model (and not combine corresponding information, such as sensor data).

[0177] Similar to the above example, the system determines the estimated state to include item 716c positioned with a gap 754 between adjacent items, item 702c positioned with an offset 756, and item 706c positioned with an offset. The system determines whether to use a geometric model, sensor data, or both to model the placement of items 702c and 716c. In some embodiments, the system interpolates the positions of items 702a and 702b to determine the position of 702c and / or interpolates the positions of items 716a and 716b to determine the position of item 716c. In some embodiments, the system uses the midpoint between the geometric model and the sensor data to determine the corresponding information for the estimated state.

[0178] 8 is a flow chart illustrating one embodiment of determining an estimate of the state of a pallet and / or stack of items. In some embodiments, process 800 is performed, at least in part, by robotic system 100 of FIG. 1 and / or system 200 of FIG. 2.

[0179] In some embodiments, process 800 is performed by one or more of an app 802, a server 804, a state estimator 806, a vision system 808, and a position determiner 810 running on a control system for the robotic arm.

[0180] At step 820, the app 802 sends a request to the server 804. The request may correspond to a placement request that calls for a plan and / or strategy for placing an item.

[0181] In response to receiving the placement request, the server 804 invokes state determination at step 822. For example, the server 804 sends a request or command to the state estimator 806 to determine (and provide) an estimated state. In some embodiments, the state estimator 806 is a module running on the server 804. In some embodiments, the state estimator 806 is a service that is queried by multiple different server / robot systems. For example, the state estimator 806 may be a cloud service.

[0182] In response to invoking the state determination, the state estimator 806 obtains the vision state. In some embodiments, the state estimator 806 sends a request for the vision state to the vision system 808.

[0183] In response to receiving the vision state request in step 824, the vision system 808 provides the vision state to the state estimator 806 in step 826. For example, in response to receiving the vision state request, the vision system 808 captures a snapshot of the workspace using one or more sensors in the workspace.

[0184] In response to receiving the vision state, the state estimator 806 determines a pallet state (e.g., an estimated state of the pallet and / or stack of items). The state estimator 806 may determine the estimated state based on one or more of the geometric model and the vision state. In some embodiments, the state estimator 806 combines the geometric model and the vision state (for at least a portion of the stack).

[0185] In step 828 , the state estimator 806 provides the palette state to the server 804 .

[0186] In step 830, server 804 sends a placement request including the pallet status to placement determiner 810. In some embodiments, placement determiner 810 is a module running on server 804. In some embodiments, placement determiner 810 is a service that is queried by multiple different server / robot systems. For example, placement determiner 810 may be a cloud service.

[0187] In step 832, the placement determiner 810 provides the set of one or more potential placements to the server 804. The set of one or more potential placements may be determined based at least in part on the items to be placed (e.g., attributes associated with the items) and pallet conditions (e.g., available locations and attributes of items within the stack of items), etc.

[0188] In some embodiments, the set of one or more potential placements is a subset of all possible placements. For example, the placement determiner 810 may use a cost function to determine the set of one or more potential placements for provision to the server 804. The placement determiner 810 may determine potential placements that meet a cost criterion with respect to the cost function (e.g., have a cost less than a cost threshold).

[0189] In response to receiving the set of one or more potential placements, in step 834, the server 804 selects a placement and sends the selected placement to the app 802. For example, the selected placement may be provided as a response to the initial placement request in step 820.

[0190] The app 802 controls the robotic arm to place the item in step 836. In some embodiments, the app 802 determines a plan for moving the item to the selected location (e.g., based on attributes of the item and a location (e.g., coordinates in the workspace) corresponding to the selected location).

[0191] In step 838, the app 802 provides instructions to the server 804 to perform updates related to the geometric state. For example, the app 802 provides confirmation that the placing of the item was performed in step 836, and the server 804 considers such confirmation an indication that an update to the geometric state (e.g., the geometric model) is to be invoked.

[0192] In step 840, the server 804 sends a request to update the geometric state to the state estimator 806. For example, the server 804 requests that the state estimator 806 update the geometric model to reflect the placement of items according to the corresponding plan.

[0193] In response to receiving a request to update the geometric state, the state estimator 806 performs the corresponding update. At step 842, the state estimator 806 provides an indication to the server 804 that the geometric state was successfully updated.

[0194] In step 844, the server 804 provides an indication to the app 802 that the geometric state has been successfully updated to reflect the placement of the items.

[0195] The process 800 may be repeated for the set of items to be stacked.

[0196] 9 is a flowchart illustrating a process for palletizing / depalletizing a set of items, according to various embodiments. In some embodiments, process 900 is performed, at least in part, by robotic system 100 of FIG. 1 and / or system 200 of FIG. 2.

[0197] Sensor data indicative of the current state of the workspace is obtained at step 910. In some embodiments, the system receives the sensor data from a vision system located within the workspace.

[0198] The robotic arm is controlled to move the first set of items at step 920. In some embodiments, the system determines a plan for moving the first set of items and controls the robotic arm to move (e.g., pick and place) the first set of items according to the plan.

[0199] At step 930, the geometric model is updated based on the movement of the first set of items. In some embodiments, in response to moving the first set of items, the system updates the geometric model to reflect the movement (e.g., the placement of the first set of items). In the case of palletizing items, the geometric model is updated to reflect the placement of the first set of items on the stack.

[0200] At step 940, an estimated state (e.g., a state of the stack of items) is determined based at least in part on the geometric model and the sensor data. The sensor data may correspond to the sensor data obtained at step 910 or updated sensor data captured after / during the movement of the first set of items. In some embodiments, the system determines the estimated state based on a combination of the geometric model and the sensor data.

[0201] At step 950, a plan for controlling the robotic arm to move the second set of items is determined based at least in part on the estimated state. In some embodiments, in response to determining the plan for moving the second set of items, the system controls the robotic arm to move the second set of items according to the corresponding plan.

[0202] According to various embodiments, the system determines an estimated state after each set of items is moved (e.g., the set of items determined based on the number of items, or based on attributes of the current or previously placed items, etc.).

[0203] At step 960, a determination is made as to whether process 900 is complete. In some embodiments, process 900 is determined to be complete in response to a determination that no further updates of the estimated state are to be performed, no further items are to be moved, a user has exited the system, an administrator has indicated that process 900 should be paused or stopped, etc. In response to a determination that process 900 is complete, process 900 ends. In response to a determination that process 900 is not complete, process 900 returns to step 910.

[0204] While the above examples are described in the context of palletizing or depalletizing a set of items, various embodiments may be implemented in connection with singulating a set of items and / or kitting a set of items. For example, various embodiments are implemented to determine / estimate a state of a workspace (e.g., a chute, conveyor, bin, etc.) based at least in part on geometric data and sensor data (e.g., a combination of geometric data and sensor data, such as interpolation between the geometric data and the sensor data).

[0205] Various example embodiments described herein are described with reference to flowcharts. While the examples may include some steps performed in a particular order, according to various embodiments, various steps may be performed in different orders and / or various steps may be combined into a single step or performed in parallel.

[0206] Although the above-described embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and are not intended to be limiting.

Claims

1. 1. A robotic system comprising: a communications interface configured to receive sensor data from one or more sensors disposed in a workspace indicative of a current state of the workspace, the workspace including a pallet or other container and a plurality of items stacked on or in the container; one or more processors connected to the communication interface; Equipped with the one or more processors: controlling a robotic arm to place a first set of items onto or into said pallet or other container, or to remove said first set of items from said pallet or other container; updating a geometric model based on the first set of items placed on or in or removed from the pallet or other container; using the geometric model in conjunction with the sensor data to estimate a state of the pallet or other container and one or more items stacked on or in the container; a robotic system configured to use the estimated state to generate or update a plan for controlling the robotic arm to place or remove a second set of items on or into the pallet or other container;

2. The robotic system of claim 1 , wherein the geometric model reflects one or more respective attributes of the first set of items and the second set of items.

3. 3. The robotic system of claim 2, wherein the geometric model reflects a weight of each of the items included in the one or more of the first set of items and the second set of items.

4. 2. The robotic system of claim 1, wherein the geometric model reflects a rigidity compressibility of each of the items included in one or more of the first set of items and the second set of items.

5. 2. The robotic system of claim 1, wherein the geometric model reflects a packaging strength of each of the items included in one or more of the first set of items and the second set of items.

6. 10. The robotic system of claim 1, wherein the geometric model reflects a distribution of weight across multiple items included in one or more of the first set of items and the second set of items.

7. 10. The robotic system of claim 1, wherein the geometric model includes a predicted stability of the one or more items stacked on or in the container.

8. 8. The robotic system of claim 7, wherein the predicted stability is based at least in part on a set of locations where the robotic system considers each item to be placed and one or more attributes of each item stacked on or in the container.

9. 10. The robotic system of claim 1, wherein the geometric model includes predicted stability of the one or more items stacked on or in the container and the one or more simulated items upon which a simulation of the stacking of the one or more simulated items has been performed.

10. The robotic system of claim 1 , wherein the one or more sensors include an image sensor.

11. The robotic system of claim 10 , wherein the image sensor includes a 3D camera.

12. The robotic system of claim 1 , wherein a point cloud is determined based at least in part on the sensor data.

13. 10. The robotic system of claim 1, wherein the estimated state of the pallet or other container and the one or more items stacked on or in the container is determined based at least in part on interpolation between the sensor data and the geometric model.

14. 14. The robotic system of claim 13, wherein the position of each item is determined based on interpolation between (i) the position of the respective item according to the sensor data and (ii) the position of the respective item according to the geometric model.

15. 10. The robotic system of claim 1, wherein the estimated state of the pallet or other container and the one or more items stacked on or in the container is determined based at least in part on a midpoint between the sensor data and the geometric model.

16. 10. The robotic system of claim 1, wherein the one or more processors further: a robotic system configured to use the geometric model to update the current state of the workspace according to the sensor data in response to determining that the current state of the workspace according to the sensor data includes an anomaly.

17. 17. The robotic system of claim 16, wherein the anomaly is caused by one or more of: (i) a sensor of the one or more sensors having an obstructed or obscured view of a particular point in the workspace; or (ii) noise or gaps in the sensor data causing a gap in the current state of the workspace.

18. 10. The robotic system of claim 1, wherein the one or more processors further: and determining that a difference between a state predicted based on the geometric model and a state observed via the one or more sensors exceeds a threshold.

19. 20. The robotic system of claim 18, wherein, in response to determining that the difference exceeds the threshold, the robotic system communicates a warning to a user to ensure that the stack of the one or more items on or in the container is OK.

20. 2. The robotic system of claim 1, wherein the one or more processors: a first subset of the one or more processors, generating or updating the plan for controlling the robotic arm to place or remove the second set of items onto or into the pallet or other container; a first subset of the one or more processors configured to generate or update the plan for controlling the robot arm based at least in part on the estimated state; and a second subset of the one or more processors, a second subset of the one or more processors configured to update the geometric model and determine the estimate of the state of the pallet or other container and the one or more items stacked on or in the container based at least in part on a combination of the geometric model and the sensor data; A robot system comprising:

21. 21. The robotic system of claim 20, wherein the second subset of the one or more processors is remote from the workspace.

22. The robot system according to claim 1, the first set of items includes N items; N is a dynamically determined positive integer for the robotic system.

23. 23. The robotic system of claim 22, wherein the state of the one or more items stacked on or in the pallet or other container is estimated after the N items are stacked.

24. 24. The robotic system of claim 23, wherein the state is estimated before the N items are stacked in response to determining that an irregularly shaped item has been placed.

25. The robot system according to claim 1, the second set of items corresponds to the next M items on the source to be placed on the pallet or other container; M is a positive integer for the robot system.

26. 26. The robotic system of claim 25, wherein the source is a conveyor.

27. 1. A method for controlling a robot, comprising: receiving sensor data from one or more sensors disposed in a workspace indicative of a current state of the workspace, the workspace including a pallet or other container and a plurality of items stacked on or in the container; controlling, by one or more processors, a robotic arm to place a first set of items onto or into the pallet or other container, or to remove the first set of items from the pallet or other container; updating a geometric model based on the first set of items placed on or in or removed from the pallet or other container; using the geometric model in conjunction with the sensor data to estimate a state of the pallet or other container and one or more items stacked on or in the container; using the estimated state to generate or update a plan for controlling the robotic arm to place or remove a second set of items onto or into the pallet or other container; A method comprising:

28. 1. A computer program product for controlling a robot, embodied in a non-transitory computer-readable medium, comprising: computer instructions for receiving sensor data from one or more sensors disposed in a workspace indicative of a current state of the workspace, the workspace including a pallet or other container and a plurality of items stacked on or in the container; computer instructions, by one or more processors, for controlling a robotic arm to place a first set of items onto or into the pallet or other container, or to remove the first set of items from the pallet or other container; computer instructions for updating a geometric model based on the first set of items placed on or in or removed from the pallet or other container; computer instructions for using the geometric model in conjunction with the sensor data to estimate the state of the pallet or other container and one or more items stacked on or in the container; computer instructions for using the estimated state to generate or update a plan for controlling the robotic arm to place or remove a second set of items on or into the pallet or other container; and A computer program product comprising:

Images