Robot system, method for controlling a robot, and computer program product for controlling a robot

Abstract

A robotic system is disclosed that includes a communication interface that receives sensor data from one or more sensors deployed in a workspace indicative of a current state of the workspace, the workspace including a pallet or other container and a plurality of items stacked on or in the container, and one or more processors that control a robotic arm to place or remove a first set of items on or in the pallet or other container, update a geometric model based on the first set of items placed on or in the container, use the geometric model in conjunction with the sensor data to estimate a stack of one or more items on or in the container, and generate or update a plan for controlling the robotic arm to place a second set of items using the estimated state.

Description

Cross-reference to other applications

[0001] This application claims priority based on U.S. Provisional Patent Application No. 63 / 211,361, entitled "STATE ESTIMATION USING GEOMETRIC DATA AND VISION SYSTEM FOR PALLETIZING", filed on June 16, 2021, which provisional patent application is incorporated herein by reference for all purposes. BACKGROUND OF THE INVENTION

[0002] Shipping centers, distribution centers, warehouses, freight docks, air cargo terminals, large retailers, and other operations that ship and receive heterogeneous item sets utilize strategies such as packing and unpacking heterogeneous items into boxes, wooden crates, containers, conveyor belts, and pallets, etc. Packing heterogeneous items into boxes, wooden crates, pallets, etc. enables the resulting item set to be handled by a hoisting machine (such as a forklift, crane, etc.), and enables the items to be packed more efficiently for storage (e.g., within a warehouse) and / or shipping (e.g., within a truck, cargo hold, etc.).

[0003] In some contexts, items can vary so much in size, weight, density, bulk, rigidity, package strength, etc. that any given item or item set may or may not have attributes that enable it to support the size, weight, weight distribution, etc. of some other item that it may need to be packed with (e.g., into a box, container, pallet, etc.). When grouping heterogeneous items onto a pallet or other set, care must be taken in selecting and stacking the items to ensure that the palletized stack does not become unstable (such as by collapsing, tipping, or otherwise) so that it can be handled by a machine (such as a forklift) and to avoid damage to the items.

[0004] Currently, on pallets, loading and / or unloading is typically done manually. A human worker selects the items to stack based on, for example, a shipping invoice or manifest, and using human judgment and intuition, selects, for example, to place larger and heavier items at the bottom. However, in some cases, the set packed in a palletized or other manner becomes unstable as a result of, for example, items simply arriving by conveyor or other mechanism and / or being selected from bins in an order list.

[0005] Due to the diversity of items and the variations in the order, number, and combination of items packed on a given pallet, for example, and the diversity in the type and location of the container and / or supply mechanism (from which items must be picked up to place them on the pallet or other container), the use of robotics has become more difficult in many environments.

Brief Description of the Drawings

[0006] In the following detailed description and the accompanying drawings, various embodiments of the present invention are disclosed.

[0007]

Figure 1

[0008]

Figure 2

[0009]

Figure 3

[0010]

Figure 4

[0011]

Figure 5

[0012]

Figure 6

[0013]

Figure 7A

[0014]

Figure 7B

[0015]

Figure 7C

[0016]

Figure 8

[0017]

Figure 9

DETAILED DESCRIPTION OF THE INVENTION

[0018] The present invention can be implemented in various forms, including a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor (a processor configured to execute instructions stored in and / or provided by a memory connected to the processor). In this specification, these embodiments or any other form that the present invention can take may be referred to as a technique. Generally, the order of the disclosed process steps may be changed within the scope of the present invention. Unless otherwise specified, components such as processors or memories described as being configured to perform a task may be implemented as general components temporarily configured to perform the task at a certain time or as specific components manufactured to perform the task. In this specification, the term "processor" shall refer to one or more devices, circuits, and / or processing cores configured to process data such as computer program instructions.

[0019] Hereinafter, a detailed description of one or more embodiments of the present invention will be given with reference to the drawings showing the principles of the present invention. The present invention is described in relation to such embodiments, but is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, many specific details are set forth in order to provide a complete understanding of the present invention. These details are for illustrative purposes only, and the present invention can be practiced according to the claims without some or all of these specific details. For the sake of brevity, technical matters well known in the technical field related to the present invention are not described in detail so as not to make the present invention unnecessarily difficult to understand.

[0020] The geometric model used in this specification may mean a model of the state of the working space (such as a programmatically determined state). For example, the geometric model is generated using geometric data determined in relation to generating a plan for moving an item within the working space and the results expected when the item is moved according to the plan. For example, the geometric model corresponds to the state of the working space that changes by controlling a robotic arm to pick, move, and / or place an item within the working space, and the pick, move, and place of the item are (for example, (i) misconfiguration or misalignment of the robotic arm or another component in the working space, (ii) deformation of the item based on interaction with the robotic arm, (iii) another item within the working space, another object within the working space, (iv) a collision between the robotic arm or the item being moved by the robotic arm and another object within the working space) considered to be executed according to the plan without errors (such as errors or noise) that may be caused.

[0021] As used herein, "pallet" includes a platform, container, or other container on or in which one or more items can be stacked or placed. Further, the pallet used herein may be used in connection with packaging and distributing a set of one or more items. As an example, the term "pallet" includes a typical flat transport structure that supports items and is movable by a forklift, pallet jack, crane, etc. The pallet used herein may be made of various materials such as wood, metal, alloy, polymer, etc.

[0022] Palletization of an item or a set of items as used herein includes picking an item from a source location (such as a transport structure) and placing the item on a pallet (such as on a stack of items on the pallet).

[0023] The depalletization used herein includes picking an item from a pallet (such as from a stack of items on a pallet), moving the item, and placing the item at a destination location (such as a conveying structure). Examples of depalletization / depalletization systems and / or processes for palletizing / depalletizing a set of items are further described in U.S. Patent Application No. 17 / 343,609, which is hereby incorporated by reference in its entirety for all purposes.

[0024] The singulation of items used herein includes picking an item from a source stack / flow and placing the item on a conveying structure (such as a partitioned conveyor or similar conveying means). Optionally, singulation can include sorting various items on the conveying structure, such as placing items one by one from the source stack / flow into slots or trays on the conveyor. Examples of singulation systems and / or processes for singulating a set of items are further described in U.S. Patent Application No. 17 / 246,356, which is hereby incorporated by reference in its entirety for all purposes.

[0025] The kitting used herein includes picking one or more items / objects from corresponding locations and placing the one or more items at predetermined locations such that the set of one or more items corresponds to a kit. Examples of kitting systems and / or processes for kitting a set of items are further described in U.S. Patent Application No. 17 / 219,503, which is hereby incorporated by reference in its entirety for all purposes.

[0026] The vision system used in this specification includes one or more sensors that acquire sensor data (e.g., sensor data regarding the work space). The sensors may include cameras, high-resolution cameras, 2D cameras, 3D (e.g., RGBD) cameras, infrared (IR) sensors, other sensors for generating a three-dimensional view of the work space (or a part of the work space such as a pallet and a stack of items on the pallet), any combination of the above, and / or a sensor array including a plurality of the above sensors, etc., and may include one or more of them.

[0027] Techniques are disclosed for programmatically utilizing a robotic system including one or more robots (e.g., suction cups, grippers, and / or robotic arms having other end effectors at the working end) to palletize / depalletize and / or otherwise package and / or unpack any set of heterogeneous items (e.g., different sizes, shapes, weights, weight distributions, rigidities, breakabilities, types, packages, etc.).

[0028] Various embodiments include a system, method, and / or apparatus for picking and placing items. The system includes a communication interface and one or more processors connected to the communication interface. The communication interface is configured to receive sensor data indicative of a current state of a work space from one or more sensors deployed in the work space, which includes a pallet or other container and a plurality of items stacked on or in the container. The one or more processors are configured to (i) control a robotic arm to place a first set of items on or in a pallet or other container or remove a first set of items therefrom, (ii) update a geometric model based on a first set of items placed on or in or removed from a pallet or other container, (iii) use the geometric model and the sensor data in combination to estimate the state of a pallet or other container and one or more items stacked on or in the container, and (iv) generate or update a plan for controlling the robotic arm to place a second set of items on or in a pallet or other container or remove a second set of items therefrom using the estimated state.

[0029] Related art systems can control a robot based on only geometric data (e.g., by driving motors to control the robot without using sensor data of the working space as feedback), or can control a robot based on only sensor data (e.g., controlling the robot based on sensor data). However, related art systems do not use both geometric data and sensor data in relation to determining or updating a plan for moving an item and controlling a robot arm to move the item according to the plan. Further, related art systems do not combine geometric data and sensor data to determine / update a plan and / or control a robot arm to move an item. For related art robot systems that rely only on geometric data in relation to controlling a robot arm to move an item, the robot system has very strict tolerances regarding the manufacture of the robot / robot arm. Such strict tolerances generally increase the cost of the robot / robot arm.

[0030] A robot system can be implemented to assemble items onto a pallet and / or disassemble items from a pallet. The robot system can utilize sensors and a robot (e.g., a robot arm with an end effector for engaging an item). The items to be assembled on the pallet can be a set of heterogeneous items. The robot system generally determines a plan for placing a set of items and then controls the robot arm to place the set of items according to the plan. For example, for each item, the robot system generally determines a destination position on the pallet where the item is to be placed. When determining where to place an item (e.g., the destination position), the robot system generally acquires sensor data directed to the working space (sensor data directed to the pallet or a stack of items on the pallet). The robot system determines the destination position based on the sensor data.

[0031] Relying on sensor data to determine the location to place an item or a plan to move the item to that location can, at a minimum, cause errors or be inaccurate due to noise inherent in the sensor data and / or misalignment or miscalibration of the sensors used to acquire the sensor data. Thus, the assembly of items onto a pallet and / or the disassembly of items from a pallet can become unpredictable.

[0032] Some of the problems resulting from using sensor data to determine the destination location to place an item and / or to determine the assembly / disassembly of items with respect to a pallet (e.g., palletization / depalletization of items) include the following. · Noise in sensor data Examples of noise sources in sensor data include: (i) light reflected from items or objects in the work space, (ii) dust in the work space, on items on the pallet / stack, and / or on the item to be placed, (iii) low-quality reflective surfaces (such as mirrors), (iv) reflective surfaces in the work space (robots, frames, shelves, other items, items on the pallet, etc.), (v) heat or other temperature changes in the environment, (vi) moisture in the environment, (vii) vibration in the sensors, (viii) the sensors being bumped or moved during capture of the sensor data, etc. · Sensor data is inaccurate based at least in part on a miscalibrated sensor. · Sensor data is inaccurate based at least in part on a misaligned sensor. · Sensor data is inaccurate based at least in part on a sensor error. · Sensor data is inaccurate or incomplete based at least in part on the field of view of the sensor. The field of view of the sensor can be occluded or blocked by items or objects in the robot's work space. · Control of the robotic arm is inaccurate / imprecise based on mechanical misalignment or miscalibration.

[0033] Various embodiments relate to determining positions to place one or more items on a pallet (or depalletizing one or more items from a pallet) and include a robotic system that utilizes sensor data and geometric data (e.g., a geometric model). The system may model the state of the pallet (or a stack of items on the pallet) using different data sources. For example, the system may estimate the positions of one or more items on the pallet and one or more features (or attributes) associated with the one or more items (e.g., the size of the item). The one or more features associated with the one or more items may include the size of the item (e.g., the dimensions of the item), the center of gravity, the rigidity of the item, the type of package, deformability, shape, the position of an identifier, etc.

[0034] According to various embodiments, the system estimates the state of the work space (also referred to herein as the estimated state) at least in part based on geometric data (e.g., a geometric model of the work space) and sensor data (e.g., data obtained by one or more sensors deployed in the work space). In response to obtaining the estimated state of the work space, the system utilizes the estimated state in connection with moving items within the work space. For example, the system uses the estimated state to determine a plan and / or strategy for picking an item from a source position and placing it at a target position (also referred to herein as the destination position).

[0035] According to various embodiments, the system determines the estimated state by performing interpolation at least in part based on at least a portion of the geometric data and at least a portion of the sensor data. In some embodiments, the interpolation process performed with respect to the first portion of the geometric model and the first portion of the sensor data to obtain the first portion of the estimated state is different from the interpolation performed with respect to the second portion of the geometric model and the second portion of the sensor data to obtain the second portion of the estimated state.

[0036] In some embodiments, the system utilizes an estimated state in connection with determining a plan for palletizing one or more items. For example, the system selects an item from a conveyor (or other item source) that includes a set of items (e.g., a series of items entering a work space), controls a robotic arm to pick the item, selects a destination location corresponding to a location on a pallet or within a stack of items on the pallet, moves the item, and utilizes the estimated state in connection with placing the item at the destination location.

[0037] In some embodiments, the system utilizes an estimated state in connection with determining a plan for depalletizing one or more items. For example, the system selects an item from a pallet or from within a stack of items on the pallet, controls a robotic arm to pick the item, selects a destination location corresponding to a location on a container or on a conveyor (e.g., that carries items away from the work space), moves the item, and utilizes the estimated state in connection with placing the item at the destination location.

[0038] In some embodiments, the system utilizes an estimated state in connection with determining a plan for singulating one or more items. For example, the system selects an item from a chute that includes a set of items, controls a robotic arm to pick the item, selects a destination location (e.g., a tray, a segment of a conveyor, etc.), moves the item, and utilizes the estimated state in connection with placing the item at the destination location.

[0039] In some embodiments, the system utilizes an estimated state in connection with determining a plan for performing kitting on one or more items. For example, the system selects an item from a kitting shelf system that includes a plurality of shelves on which items are disposed, controls a robotic arm to pick the item from the shelf, and a destination location (e.g., a container such as a box, tote, etc.)to Utilize an estimated state in connection with moving an item and placing the item at a destination location.

[0040] According to various embodiments, the geometric model is determined based at least in part on one or more attributes of one or more items within a work space. For example, the geometric model reflects the respective attributes of a set of items (e.g., one or more within a first set that is palletized / stacked, as well as a second set of items that are palletized / stacked, etc.). The of attributesExamples include the size of the item (e.g., the dimensions of the item), the center of gravity, the rigidity of the item, the type of package, the position of the identifier, the deformability of the item, the shape of the item, and the like. Various other attributes of the item or object within the work space may be implemented. As another example, the geometric model includes the predicted stability of one or more items stacked on or within a container (e.g., a pallet). The geometric model may include the predicted stability of a set of items (e.g., a stack of items), and / or the predicted stability of the individual items included in the stack of items. In some embodiments, the system determines the predicted stability of an item based at least in part on (i) one or more attributes of the item and (ii) one or more predicted interactions regarding the item and another item or object (e.g., a pallet) within the work space. For example, the system may determine the predicted stability based on the determination of the attributes of another item or object that contacts (or is disposed in proximity to) the item for which the predicted stability is being calculated. Examples of other item attributes that can affect the predicted stability of a particular item include rigidity, deformability, size, and the like. As an example, if a particular item is placed on top of another item that is rigid, the particular item may have a higher predicted stability compared to the case where the particular item is placed on top of another item that is not rigid or has low rigidity. As another example, if a particular item is placed on top of another item that is deformable (such as one composed of a soft package), the particular item may have a lower predicted stability compared to the case where the particular item is placed on top of another item that is not deformable or has low deformability.As another example, if a particular item is placed on top of another item that has a larger top surface area than the bottom surface area of the particular item, or if a relatively high percentage of the bottom surface of the particular item is supported by the top surface of another item, the predicted stability of the item is relatively high or at least higher than if the particular item had a smaller top surface area than the bottom surface area of the particular item, or if a relatively high percentage of the bottom surface of the particular item is not supported / does not interact with the top surface of another item.

[0041] In some embodiments, the system is noisy (e.g., sensor noise s) To account for , the sensor data is adjusted. The system can estimate the noise included in the sensor data, at least in part, based on a validation analysis of the vision system. For example, a validation analysis of the performance of the vision system can be performed to determine the noise (e.g., inherent in the sensor data) included in the sensor data. In some embodiments, the system stores a predetermined sensor noise profile associated with the vision system. The system can utilize the sensor noise profile in connection with adjusting the sensor data to account for the noise. For example, the system can apply an adjustment to cancel out the noise predicted, at least in part, based on the sensor profile. A validation analysis of the performance of the vision system can include (i) manually / physically measuring an item or the work space, (ii) capturing the same using the vision system, and (iii) determining (e.g., using digital processing, etc.) the difference between the manual / physical measurement of the item / work space and the measurement of the same using the sensor data. The system may consider the difference between the manual / physical measurement of the item / work space and the measurement of the same using the sensor data as the noise profile. As an example, the system determines the variation of the sensor data and determines the sensor noise profile based, at least in part, on the variation. The validation analysis can be performed with respect to a statistically significant set of experiments / measurements. Examples of noise (or inaccuracy of sensor data) can include (i) inaccuracy of an image at the edge of the field of view of the vision system, (ii) glare / reflection from an item or other object within the work space, etc.

[0042] In some embodiments, the system is noise (e.g., controlling a robotic arm doAdjust the geometric model to account for geometric noise or inaccuracies (resulting from, e.g., the transition of the geometric model to the physical world). The system can estimate the noise included in the geometric model based at least in part on the accuracy of robot control or the empirical analysis of other objects in the work space (e.g., the estimated deformation of the pallet, the deviation of the pallet's placement from the position used in the geometric model, etc.). For example, an empirical analysis of the performance of the robot arm control (when performing tasks such as, e.g., placing an item) can be performed to determine the noise (e.g., inherent) incorporated in the geometric model. As an example, the system determines the variation of the geometric model and determines a geometric noise profile based at least in part on the variation. In some embodiments, the system stores a predetermined geometric noise profile associated with the vision system. The system can utilize the geometric noise profile in connection with adjusting the geometric model to account for the noise. For example, the system can apply the adjustment to cancel out the predicted noise included in the geometric model (e.g., the noise that occurs based on controlling the robot according to a plan determined based on the geometric model).

[0043] According to various embodiments, the system determines a best estimate of the state of the work space using a geometric model and sensor data. The system can make adjustments due to noise (e.g., cancellation of noise) in one or more of the geometric model and / or sensor data. In some embodiments, the system detects an anomaly or difference between the state according to the geometric model and the state according to the sensor data. In response to determining that there is an anomaly or difference between the geometric model and the sensor data, the system can make a best estimate of the state regardless of the anomaly or difference. For example, the system can determine whether to use the geometric model or the sensor data, or a combination (e.g., interpolation between them) of the geometric model and the sensor data. In some embodiments, the system determines an estimated state for each segment (e.g., for each voxel, item, or object in the work space, etc.). For example, a first portion of the work space can be estimated using only the geometric model, a second portion of the work space can be estimated using only the sensor data (e.g., if there is an anomaly in the geometric model), and / or a third portion of the work space can be estimated based on a combination of the geometric model and the sensor data.

[0044] According to various embodiments, the system combines a geometric model and sensor data to determine an estimated state of the work space based at least in part on a predetermined bias. In some embodiments, the system utilizes the predetermined bias in connection with determining whether to treat the geometric model or sensor data as ground truth modified by the other of the geometric model or sensor data (e.g., to correct anomalies between the geometric model and sensor data). As an example, if the predetermined bias indicates that the geometric model is biased, the system can utilize the geometric model and supplement the state according to the geometric model with sensor data. Examples of corrections that the system can perform with respect to the state according to the geometric model include using sensor data to confirm the placement of items predicted by the geometric model and, if the difference between the sensor data and the geometric model exceeds a predetermined threshold, using sensor data with respect to the portion of the work space where the difference is occurring, or correcting the geometric model with interpolation of data using the geometric model and sensor data (e.g., the portion of the work space where the anomaly / difference exists may be the subject of interpolation, or the entire work space may be the subject of interpolation, etc.).

[0045] In some embodiments, the bias with respect to the geometric model and sensor data (or the corresponding weighting applied to each type of information) is determined based at least in part on a determination of which of the geometric model and sensor data is more accurate. The accuracy of the geometric model or sensor data can be determined through empirical trials or based on variations in the information or noise included in the geometric model and / or sensor data. The robot is generally considered to be more accurate than the sensor. Thus, in some embodiments, the bias is set to give the geometric model a higher priority or greater weighting compared to the sensor data. For example, the default bias may be set to preferentially weight the geometric model rather than the sensor data.

[0046] An example of a situation where the geometric model and the sensor data can be inconsistent (e.g., the difference is greater than a predetermined difference threshold) is a situation where the system determines, using the geometric model, that a set of 10 items are stacked in their corresponding positions (e.g., the system stores the 10-item arrangement programmatically), but the sensor data may suggest that only 8 items are stacked in those positions (e.g., the positions where the items are stacked according to the geometric model). Depending on the determination that the sensor data indicates that two boxes are missing, the system can determine whether the inconsistency between the geometric data and the sensor data is due to (i) the boxes falling out of the stack (e.g., the sensor data may include boxes on the floor or to the side of the stack), (ii) the sensor's field of view being partially blocked by another object (e.g., a robotic arm), (iii) boxes not included in the sensor data being stacked behind other boxes and thus not visible in the sensor data, as determined by the geometric model, etc.

[0047] Another example of a situation where the geometric model and the sensor data can be inconsistent (e.g., the difference is greater than a predetermined difference threshold) is a situation where the system places a particular item in / on a stack of items, but after placing the particular item, the particular item falls from the stack of items. According to the geometric model in such a situation, the system considers that the item is on the stack of items (e.g., the position where the item is placed according to the plan in response to the robot arm being controlled according to the plan). However, the sensor data does not include the particular item placed at the expected destination position (e.g., inside the stack of items). Instead, the sensor data reflects that the item has fallen from the stack of items (e.g., the item is next to the stack of items or may be outside the field of view of the sensor data in other forms). In response to determining that the sensor data indicates that the item cannot be found, the system can determine whether the item has fallen from the stack (e.g., the sensor data may include a box on the floor or to the side of the stack).

[0048] An example of a situation where the geometric model and the sensor data can deviate from each other (e.g., the difference is greater than a predetermined difference threshold) includes a case where the system places an item on top of another item, and the other item can be deformed. According to the geometric model, if the deformability of the other item exceeds the predicted deformation (if any), placing a particular item on the deformable other item can cause the actual placement of the particular item to deviate from the geometric model (e.g., the placement of the item can be distorted to the extent that the other item deforms at least partially based on the interaction with the particular item placed on it, or to the extent that the deformation of the other item exceeds the predicted deformation).

[0049] The predetermined bias may be set based on user selection (e.g., defined by an administrator) or a predetermined state determination policy. The predetermined state determination policy may be set based on empirical data or simulation of the pick and place of items. For example, the predetermined bias corresponds to the best bias (e.g., the optimal bias) for minimizing a cost function related to moving an item or palletizing an item. In some embodiments, the cost function is at least partially based on one or more of the efficiency of moving an item, the predicted time to move an item, the likelihood that the movement / stacking of an item will be successful, the predicted stability related to the placement of an item (e.g., the stability of an item after a particular item is placed, the stability of one or more other items or stacks after a particular item is placed, etc.), and the like. In some embodiments, the predetermined bias is selected from a set of potential biases corresponding to the placement of an item where the cost function is less than a predetermined cost threshold.

[0050] According to various embodiments, the system uses a predetermined bias in connection with determining respective weightings for the geometric model and the sensor data. For example, the system determines a first weighting corresponding to the geometric model and a second weighting corresponding to the sensor data, and the system uses a combination of the geometric model and the sensor data (e.g., combined according to the first weighting and the second weighting) to determine an estimated state of the work space.

[0051] In response to obtaining the current estimated state, the system determines a first plan for moving a first item in the work space (e.g., stacking the item on a stack of items) using the estimated state, and the system controls the robot to move the first item according to the first plan. In response to placing the first item, the system uses geometric data corresponding to the placement of the first item in relation to determining an initial updated estimated state. The system obtains sensor data (e.g., from a vision system), segments the sensor data to determine objects (e.g., identify the objects within a sensor model of the work space), and determines an updated estimated state based at least in part on the initial updated estimated state (e.g., geometric model) and the sensor data. For example, the system combines the geometric model and the sensor data in relation to determining the updated estimated state. In response to determining the updated estimated state, the system determines a second plan for moving a second item in the work space using the updated estimated state, and the system controls the robot to move the second item based at least in part on the second plan.

[0052] The use of geometric models and sensor data (e.g., combinations of geometric models) enables various embodiments to be less costly compared to systems of the related art. For example, a system can use geometric models and sensor data to detect anomalies (e.g., caused by geometric models or sensor data) and determine an estimated state of the workspace regardless of the anomalies (e.g., by selectively using geometric models or sensor data, or interpolation between both). Thus, systems of the related art require either a high-precision (e.g., high-resilience) robot or a high-precision camera, while various embodiments can utilize a relatively low-precision (e.g., low-resilience) robot or a lower-precision (e.g., low-resilience) vision system. As a result of the high-resilience requirements for robots or cameras in systems of the related art, the robots and / or cameras used by such systems are very expensive. According to various embodiments, combining both geometric models and sensor data can reduce the cost and accuracy / precision requirements of a robot or vision system.

[0053] According to various embodiments, in response to a robot placing an item on a pallet, a system for estimating the state of the pallet and / or a stack of items on the pallet records information regarding the placement of the item (e.g., the position of the item, the size of the item, etc.). For example, the system has logical knowledge of the state of the system based on where the robot has placed various items on the pallet. The logical knowledge may correspond to geometric data (such as information obtained based on how the robot is controlled). However, the logical knowledge may be different from the actual state of the pallet and / or a stack of items on the pallet. Similarly, as described above, the state of the pallet and / or a stack of items on the pallet detected by the vision system (e.g., the actual state modeled based on sensor data) may be different from the actual state based on noise in the sensor data or inaccurate / incomplete sensor data. Various embodiments combine a view of the world using geometric data (e.g., logical knowledge) and sensor data. By using both geometric data and sensor data to model the world, the gap in the view of the world of each dataset is filled. For example, sensor data obtained based on a vision system may be utilized in connection with determining whether the predicted state of the pallet or a stack of items on the pallet needs to be updated / improved. State estimation according to various embodiments provides a better estimate than would be possible using only sensor data or only geometric data regarding the state of the pallet and / or a stack of items on the pallet. Further, the estimated state of the pallet and / or a stack of items on the pallet may be utilized in connection with palletizing / depalletizing items to / from the pallet. Utilizing a better estimated state of the state of the pallet and / or a stack of items on the pallet in connection with determining the palletizing / depalletizing of items to / from the pallet can provide a better arrangement and may lead to a better final pallet (e.g., a more gapless / tightly packed pallet, a more stable pallet, etc.).

[0054] In some embodiments, in response to a determination that the predicted state of a pallet or stack of items using geometric data is sufficiently different from the state of the pallet or stack of items using a vision system (e.g., sensor data), the system may provide a suggestion to the user. For example, the system may prompt the user to confirm whether the predicted state using geometric data is correct, whether the state using the vision system is correct, or whether neither is correct and the user should correct the current state. The user may use the user interface to adjust the difference between the predicted state using geometric data and the state using the vision system. The determination that the predicted state using geometric data is sufficiently different from the state using the vision system may be based on a determination that the difference between the two states or models of the states exceeds a predetermined threshold.

[0055] According to various embodiments, a system for estimating the state of a pallet and / or a stack of items on the pallet is implemented on another computer system (such as a server). Modules or algorithms for modeling the state using geometric data, for modeling the state using a vision system (e.g., sensor data), and for updating a predicted state (e.g., based on adjustment of differences between states) can be costly and may require relatively high computing power. Further, the processing power in the robot's workspace or the computer system controlling the robot can have constraints on the computing power and / or bandwidth for executing the modules. In some embodiments, the system controlling the robot may acquire sensor data and information regarding the item to be placed and transmit the sensor data and information regarding the item to be placed to a server via one or more networks. In some embodiments, multiple servers may be utilized in connection with implementing various modules for estimating the state of the system (e.g., the state of the pallet and / or the state of a stack of items on the pallet). For example, a module for modeling the state using geometric data may be executed by a first server, a module for modeling the state using a vision system may be executed by a second server, and a module for updating the predicted state based at least in part on differences between states may be executed by a third server. The various servers that can implement the various modules for determining the predicted state may communicate with each other.

[0056] In some embodiments, a system for determining an estimated state (e.g., a pallet state estimation module / service, etc.) may store and / or manage the difference between (i) a state modeled based on geometric data and (ii) a state modeled based on a vision system. For example, a pallet state estimator may store / manage the current state of a pallet and / or a stack of items on the pallet. The pallet state estimator may be a module executed by a computer system such as a robot system that controls a robot that is palletizing / depalletizing items to / from the pallet, or by a server that the robot system that controls the robot communicates with.

[0057] According to various embodiments, the pallet state estimator may be utilized in connection with determining the current state while planning / placing each item. For example, a robot system that controls a robot and / or determines a plan for moving an item may query the pallet state estimator in connection with determining a plan for moving an item. The pallet state estimator may be queried using (i) information regarding the item to be placed and (ii) current sensor data obtained by a vision system. The current state using geometric data may be stored by the pallet state estimator or may be accessible by the pallet state estimator. In some embodiments, information regarding the current geometric data may be communicated to the pallet state estimator in connection with querying the pallet state estimator for the current state. In some embodiments, due to the computational intensity in determining the current state using both geometric data and a vision system (e.g., sensor data), the pallet state estimator may be queried for the current state after placing a predetermined number of items (e.g., N items, where N is an integer). For example, the pallet state estimator may be queried at a regular frequency. As another example, the pallet state estimator may be queried in response to a determination that the placed item (e.g., a previously placed item) has an irregular shape or a particular type of package (e.g., a low-rigidity package type such as a polybag).

[0058] In some embodiments, after an item is placed, the pallet state estimator repeatedly updates an internal model of the state (e.g., the world, the pallet, and / or a stack of items on the pallet, etc.). The internal model of the state may correspond to the current state of the pallet and / or a stack of items on the pallet. The pallet state estimator may update the internal model after each item is placed. For example, a robot system that controls a robot may provide information about the item (e.g., characteristics of the item such as dimensions, weight, center of gravity, etc.), sensor data obtained by a vision system, and / or geometric data corresponding to the position where the robot was controlled to place the item. In response to receiving the information about the item, the sensor data, and / or the geometric data, the pallet state estimator may update its internal model. For example, the pallet state estimator may (i) provide the sensor data to a module for modeling the state using the sensor data, (ii) provide the geometric data and / or the information about the item to a module for modeling the state using the geometric data, and (iii) update its internal model based on a model of the state based on the sensor data and / or a model of the state based on the geometric data (e.g., or, in between two states).

[0059] The model of the state of the pallet and / or the stack of items may be determined using the pallet as a reference (e.g., the bottom of the pallet may be used as a reference point as the bottom of the stack, etc.). The model may be updated after the items are placed so that the items are represented at the positions where the items are placed. The model of the state may be stored as a two-dimensional grid (e.g., 10×10). The model may further include information related to each item within the stack of items on the pallet. For example, one or more features related to the item may be stored associated with the item within the model. In some embodiments, the stability of the stack of items may be determined / calculated based on the positions of the various items on the stack and one or more features related to the item. The model of the state of the pallet may be converted into a representation in the physical world based on a predetermined conversion between the stored model unit and the physical world unit.

[0060] Various embodiments include a robotic system having a communication interface and one or more processors connected to the communication interface. The one or more processors receive data related to a plurality of items to be stacked on or in a destination location via the communication interface, and generate a plan for stacking the items on or in the destination location based at least in part on the received data, and are configured to execute the plan, at least in part, by picking up the items according to the plan and stacking the items on or in the destination location. Generating a plan for stacking items on or in a destination location may include, for each item, determining a destination location based at least in part on a feature associated with the item and at least one of (i) a feature of a platform or container on which one or more items are stacked and (ii) a current state of the stack of items on the platform or container, where the current state of the stack of items on the platform or container is based at least in part on geometric data regarding the arrangement of one or more items included in the stack of items and sensor data regarding the stack of items acquired by a vision system. The current state of the stack of items may be determined based at least in part on an interplay between geometric data regarding the arrangement of one or more items included in the stack of items and sensor data regarding the stack of items acquired by a vision system. The current state of the stack of items on the platform or container may be obtained based at least in part on the robotic system querying a pallet state estimator running on a server.

[0061] FIG. 1 is a diagram showing a robotic system for palletizing and / or depalletizing heterogeneous items according to various embodiments. In some embodiments, robotic system 100 is implemented, at least in part, in process 300 of FIG. 3, process 400 of FIG. 4, process 500 of FIG. 5, process 600 of FIG. 6, process 800 of FIG. 8, and / or process 900 of FIG. 9.

[0062] In the example of the figure, the robotic system 100 includes a robotic arm 102. In this example, the robotic arm 102 is fixed, but in various other embodiments, the robotic arm 102 may be fully or partially movable, for example, mounted on a rail, fully movable on a motor-driven chassis, and so on. As shown in the figure, the robotic arm 102 is used to pick any and / or heterogeneous items (such as boxes, packages, etc.) from a conveyor (or other source) 104 and stack them on a pallet (such as a platform or other container) 106. The pallet (such as a platform or container) 106 includes wheels at its four corners and a pallet, container, or base in which three of the four side faces are at least partially closed, and is also called a three-sided "roll pallet", "roll cage", and / or "roll" or "cage" "trolley". In other embodiments, pallets with more side faces, fewer side faces, and / or no side faces and no rolls or wheels may be used. In some embodiments, other robots not shown in FIG. 1 may be used to push the container 106 into a predetermined position for loading and unloading and / or a track or other destination for conveyance, etc.

[0063] In some embodiments, a plurality of containers 106 may be arranged around the robotic arm 102 (e.g., within a range near the threshold of the robotic arm or within the range of other robotic arms). The robotic arm 102 may stack one or more items onto a plurality of pallets simultaneously (e.g., in parallel and / or contemporaneously). Each of the plurality of pallets may be associated with a manifest and / or an order. For example, each pallet may be associated with a preset destination (e.g., a customer, an address, etc.). In some examples, some of the plurality of pallets may be associated with the same manifest and / or order. However, each of the plurality of pallets may be associated with different manifests and / or orders. The robotic arm 102 may place a plurality of items corresponding to the same order onto a plurality of pallets. The robotic system 100 may determine an arrangement (e.g., a stack of items) on the plurality of pallets (e.g., how a plurality of items for an order are divided among the plurality of pallets, how items on any one pallet are stacked, etc.). The robotic system 100 may store one or more items (e.g., items for an order) in a buffer area or a staging area while one or more other items are being stacked onto a pallet. As an example, one or more items may be stored in the buffer area or the staging area until the robotic system 100 determines that the placement of each of the one or more items onto the pallet (e.g., onto the stack) meets (e.g., exceeds) a threshold conformance or threshold stability. The threshold conformance or threshold stability may be a predetermined value or a value empirically determined based at least in part on historical information. A machine learning algorithm may be implemented in connection with determining whether the placement of items on a stack is expected to meet (e.g., exceed) a threshold conformance or threshold stability and / or in connection with determining the threshold conformance or threshold stability (e.g., the threshold against which a simulation or model is compared to evaluate whether to place an item on a stack).

[0064] In the example of the figure, the robotic arm 102 includes an adsorption type end effector (for example, end effector 108). The end effector 108 has a plurality of adsorption cups 110. As shown in the figure, the robotic arm 102 is used to place the adsorption cups 110 of the end effector 108 on the item to be picked up, and a vacuum source provides the adsorption force for gripping the item, lifting the item from the conveyor 104, and placing the item at the destination position on the container 106. Various na ta types of end effectors may be implemented.

[0065] In various embodiments, the robotic system 100 includes a vision system used to generate a model of the working space (for example, a 3D model of the working space and / or a geometric model). For example, one or more of a 3D camera or other camera 112 attached to the end effector 108 and cameras 114, 116 attached within the space where the robotic system 100 is deployed are used to identify the item on the conveyor 104 and / or to determine a plan for gripping the item, picking / placing it, and stacking it on the container 106 (or, if applicable, placing the item in a buffer area or staging area). In various embodiments, additional sensors (not shown) may be used to identify (for example, determine) the attributes of the item, grip the item, pick up the item, move the item through the determined trajectory, and / or place the item on or in the item on the container 106 on the conveyor 104 and / or other sources and / or staging areas where the item can be arranged and / or rearranged, for example, by the system 100. Examples of such additional sensors that are not shown may include weight sensors or force sensors embedded in and / or adjacent to the conveyor 104 and / or the robotic arm 102, and force sensors in the x-y plane and / or the z-direction (vertical direction) of the adsorption cup 110.

[0066] In the example of the figure, camera 112 is attached to the side surface of the main body of end effector 108. However, in some embodiments, camera 112 and / or additional cameras may be attached at other positions (e.g., downward from the position between suction cups 110, attached to the lower surface of the main body of end effector 108, or attached to a segment of robot arm 102 or other structure, or attached at other positions, etc.). In various embodiments, cameras such as 112, 114, and 116 may be used to read text, logos, photos, diagrams, images, marks, barcodes, QR codes (registered trademarks), or other encoded and / or graphic information or content that is visible on and / or constitutes an item on conveyor 104.

[0067] In some embodiments, robot system 100 includes a dispenser device (not shown) configured to supply a certain amount of spacer material from a source of spacer material in response to a control signal. The dispenser device may be disposed on robot arm 102 or in the vicinity of the work space (e.g., within a threshold distance of the work space). For example, the dispenser device is within the work space of robot arm 102 such that the dispenser device supplies spacer material onto or around container 106 (e.g., a pallet) or within a predetermined distance of end effector 108 of robot arm 102. arrangementThis may be done. In some embodiments, the dispenser device comprises a fixture configured to mount the dispenser device on or adjacent to the end effector 108 of the robotic arm 102. The fixture is at least one of a bracket, a strap, one or more fasteners, etc. As an example, the dispenser device may comprise a biasing device / mechanism that biases the supply material within the dispenser device so as to be extruded / supplied from the dispenser device. The dispenser device may comprise a gating structure used to control the supply of the spacer material (e.g., the supply of the spacer material is prevented without the actuation of the gating structure, and the supply of the supplied spacer material becomes possible in response to the actuation).

[0068] The dispenser device may include a communication interface configured to receive a control signal. For example, the dispenser device may communicate with one or more terminals (such as control computer 118). The dispenser device may communicate with one or more terminals via one or more wired connections and / or one or more wireless connections. In some embodiments, the dispenser device communicates information to one or more terminals. For example, the dispenser device may transmit to the control computer 118 an indication of the state of the dispenser device (e.g., an indication of whether the dispenser device is operating properly), an indication of the type of spacer material provided within the dispenser device, an indication of the supply level of the spacer material within the dispenser device (e.g., an indication of whether the dispenser device is full, empty, half-full, etc.), and the like. The control computer 118 may be utilized in connection with controlling the dispenser device to supply a certain amount of spacer material. For example, the control computer 118 may determine that a spacer is to be utilized in connection with palletizing one or more items, such as to improve the predicted stability of the stack of items in the upper / middle of the container 106. The control computer 118 may determine the amount of spacer material to be utilized in connection with palletizing one or more items (e.g., the number of spacers, the amount of spacer material, etc.). For example, the amount of spacer material to be utilized in connection with palletizing one or more items may be determined based at least in part on determining a plan for palletizing one or more items.

[0069] In some embodiments, the dispenser device includes an actuator configured to supply an amount of spacer material from a source of spacer material in response to a control signal. In response to determining that a spacer / spacer material is to be utilized in connection with palletizing one or more items, the control computer 118 may generate a control signal to cause the actuator to supply an amount of spacer material. The control signal may include an indication of the amount of spacer material to be utilized as a spacer.

[0070] According to various embodiments, the spacer or spacer material is a rigid block. For example, the spacer or spacer material may be a rigid block of foam. In some embodiments, the spacer or spacer material includes polyurethane.

[0071] In some embodiments, the source of the spacer material includes a plurality of cut blocks. The plurality of cut blocks may be pre-loaded into a spring-loaded cartridge that biases the plurality of cut blocks towards the supply end. In response to a cut block being supplied from the cartridge, another block within the plurality of cut blocks is pushed into the position to be supplied next from the cartridge.

[0072] In some embodiments, the source of the spacer material includes one or more of a larger spacer material block, a strip of spacer material, and a roll of spacer material. The dispenser device or robot system 100 may include a cutter configured to cut a quantity of spacer material from the source of the spacer material. In response to a control signal being supplied to the actuator, the actuator may cause the cutter to cut a quantity of spacer material from the source of the spacer material.

[0073] In some embodiments, the source of the spacer material includes a liquid precursor. In response to a control signal being supplied to the actuator, the actuator causes an amount of the spacer material to be supplied onto the surface of the pallet or a stack of items on the pallet. The supplied precursor may cure after being supplied onto the surface of the pallet or a stack of items on the pallet.

[0074] In some embodiments, the source of the spacer material includes an extruded material. In response to a control signal being supplied to the actuator, the extruded material is filled to one or more of a desired size and a desired hardness. The extruded material may be sealed in response to a determination that the extruded material is filled to one or more of a desired size and a desired hardness. In some embodiments, the extruded material is filled with a fluid. The fluid may be one or more of air, water, and the like. In some embodiments, the extruded material is filled with a gel.

[0075] In various embodiments, a robot-controlled dispenser, a tooling, or a machine fills the gaps between and / or adjacent to the boxes to prepare a surface area for placement of the next box / layer. In some embodiments, the robot system 100 may pick / place a predetermined cut material using the robot arm 102 and / or dynamically trim spacer material according to the need for placement of the surface area of the next item. In some embodiments, a robot-controlled dispenser device, or a robot palletization system comprising a robot-controlled dispenser device, is related to preparing a surface area for the next box or item that the system has determined may not normally fit on the pallet surface area (e.g., the upper surface of the previous layer), and comprises a device for trimming a rectangular parallelepiped size from a long tube and / or package and placing the rectangular parallelepiped on an existing pallet. The spacer may include, but is not limited to, foam, a plastic bag inflated with air, wood, metal, plastic, etc. The dispenser device may place (e.g., extrude, supply, etc.) a rectangular parallelepiped (e.g., a spacer) directly onto the pallet, and / or the device may supply a rectangular parallelepiped (e.g., a spacer) near the robot arm, and the end effector may reposition / place the rectangular parallelepiped (e.g., a spacer) onto the pallet surface area. The dispenser device may supply a predetermined amount (e.g., the correct amount or the expected amount) of spacer material to correct or improve the surface area mismatch between the boxes or items on the layer (e.g., on the upper surface of the layer) to prepare a surface area for the next box or item.

[0076] Referring further to FIG. 1, in the example of the figure, the robot system 100 includes, in this example via wireless communication (however, in various embodiments, via one or both of wired and wireless communication), a robot arm 102, a conveyor 104, an end effector 108, and a control computer 118 configured to communicate with elements such as sensors (cameras 112, 114, and 116, and / or weight sensors, force sensors, and / or other sensors not shown in FIG. 1). In various embodiments, the control computer 118 is configured to observe, identify, and determine one or more attributes of items being loaded onto and / or unloaded from the container 106 using inputs from sensors (cameras 112, 114, and 116, and / or weight sensors, force sensors, and / or other sensors not shown in FIG. 1). In various embodiments, the control computer 118 identifies items and / or their attributes using item model data within a library stored in the control computer 118 and / or within a library accessible to the control computer 118, based on, for example, images and / or other sensor data. The control computer 118 determines and implements a plan for stacking items together with other items within / onto a destination (such as the container 106) using a model corresponding to the item. In various embodiments, item attributes and / or models are utilized to determine strategies for gripping, moving, and placing the item at a destination location (e.g., the location determined for the item to be placed as part of a plan / replan process for stacking items within / onto the container 106).

[0077] In the example of the figure, the control computer 118 is connected to the "on-demand" remote operation device 122. In some embodiments, if the control computer 118 cannot continue in the fully automatic mode, for example, when the control computer 118 no longer has a strategy to complete the pick and place of items in the fully automatic mode, and the strategy for gripping, moving, and placing items cannot be determined and / or fails, the control computer 118 instructs the human user 124 to intervene by operating the robotic arm 102 and / or the end effector 108 using the remote operation device 122, for example, to grip, move, and place the items.

[0078] A user interface regarding the operation of the robot system 100 may be provided by the control computer 118 and / or the remote operation device 122. The user interface may provide the current state of the robot system 100, including information related to the current state of the pallet (or stack of related items), the current order or manifest being palletized or depalletized, the performance of the robot system 100 (e.g., the number of items palletized / depalletized by a certain time), etc. The user may select one or more elements on the user interface or otherwise provide input to the user interface to activate or pause the robot system 100 and / or a specific robotic arm within the robot system 100.

[0079] According to various embodiments, the robotic system 100 performs machine learning processing to model the state of a pallet, such as to generate a model of a stack on the pallet. The machine learning processing may include adaptive and / or dynamic processing to model the state of the pallet. The machine learning processing may define and / or update / improve the processing for the robotic system 100 to generate a model of the state of the pallet. The model may be generated based at least in part on inputs (e.g., information obtained from sensors) from one or more sensors within the robotic system 100 (such as one or more sensors or sensor arrays within the workspace of the robotic arm 102). The model may be generated based at least in part on the shape of the stack, vision responses (e.g., information obtained by one or more sensors within the workspace), and the machine learning processing, etc. The robotic system 100 may utilize the model in connection with determining an efficient (e.g., maximizing / optimizing efficiency) way to palletize / depalletize one or more items, and the way to palletize / depalletize may be limited by a minimum threshold stability value. The processing for palletizing / depalletizing one or more items may be settable by a user administrator. For example, one or more metrics for which the processing for palletizing / depalletizing is maximized may be settable (e.g., set by a user / administrator).

[0080] In the context of palletizing one or more items, the robot system 100 may generate a model of the state of the pallet in connection with determining whether to place the item on the pallet (e.g., on the stack), and selecting a plan for placing the item on the pallet, such as the destination position where the item is to be placed and the trajectory by which the item is to be moved from the source position (e.g., the current destination such as a conveyor) to the destination position. Also, the robot system 100 may use the model in connection with determining a strategy for releasing the item or otherwise placing the item on the pallet (e.g., applying a force to the item to stow it on the stack). Modeling the state of the pallet may include simulating the placement of the item at different destination positions on the pallet (e.g., on the stack) and determining the corresponding different predicted fitnesses and / or predicted stabilities (e.g., stability metrics) that are expected to result from the placement of the item at the different positions. The robot system 100 may select a destination position where the predicted fitness and / or predicted stability meets (e.g., exceeds) the corresponding threshold. Additionally or alternatively, the robot system 100 may select a destination position that optimizes the predicted fitness of the item (e.g., on the stack) and / or the predicted stability of the stack (e.g., of the stack).

[0081] Conversely, in the context of depalletizing one or more items from a pallet (e.g., a stack on the pallet), the robotic system 100 (e.g., the control computer 118) may generate a model of the state of the pallet in connection with determining whether to remove an item on the pallet (e.g., on the stack) and selecting a plan for removing an item from the pallet. The model of the state of the pallet may be utilized in connection with determining the order in which items are removed from the pallet. For example, the control computer 118 may utilize the model to determine whether removing an item is predicted to reduce the stability of the state of the pallet (e.g., stack) below a threshold stability. The robotic system 100 (e.g., the control computer 118) may simulate the removal of one or more items from the pallet and select an order for removing items from the pallet that optimizes the stability of the state of the pallet (e.g., stack). The robotic system 100 may use the model to determine the next item to remove from the pallet. For example, the control computer 118 may select an item as the next item to remove from the pallet based at least in part on a determination that the predicted stability of the stack during and / or after removal of the item exceeds the threshold stability. The model and / or machine learning process may be utilized in connection with determining a strategy for picking an item from the stack. For example, after an item is selected as the next item to be removed from the stack, the robotic system 100 may determine a strategy for picking the item. The strategy for picking an item may be based at least in part on the state of the pallet (e.g., the determined stability of the stack), the attributes of the item (e.g., size, shape, weight or predicted weight, center of gravity, type of package, etc.), the position of the item (e.g., its position relative to one or more other items in the stack), the attributes of other items on the stack (e.g., the attributes of adjacent items, etc.), and the like.

[0082] According to various embodiments, machine learning processing is performed in connection with improving a grasping strategy (e.g., a strategy for grasping an item). The robotic system 100 may obtain attribute information regarding one or more items to be palletized / depalletized. The attribute information may include one or more of the item's orientation, material (such as package type), size, weight (or predicted weight), or center of gravity, etc. Also, the robotic system 100 may obtain the source location (e.g., information regarding the input conveyor from which the item is picked), and may obtain information regarding the pallet to which the item is to be placed (or a set of pallets from which the destination pallet is determined, such as a set of pallets corresponding to the order on which the items are stacked). In connection with determining a plan for picking and placing an item, the robotic system 100 may use information regarding the item (e.g., attribute information, destination location, etc.) to determine a strategy for picking the item. The pick strategy may include a suggestion of a pick location (e.g., a location on the item where the robotic arm 102 engages the item via an end effector or the like). The pick strategy may include the force applied to pick the item and / or the holding force for grasping the item while the robotic arm 102 moves the item from the source location to the destination location. The robotic system 100 may utilize machine learning processing to improve the pick strategy based at least in part on the relevance between information regarding the item (e.g., attribute information, destination location, etc.) and the performance of picking the item (e.g., historical information related to past iterations of picking and placing that item or similar items (such as items sharing one or more similar attributes)).

[0083] According to various embodiments, the robotic system 100 may determine to utilize a spacer or an amount of spacer material in connection with palletizing one or more items in response to a determination that the use of the spacer or the amount of spacer material improves the result of the stack of items on the pallet (e.g., improves the stability of the stack of items). In some embodiments, the determination that placing one or more spacers in connection with placing a set of N items on a pallet improves the stack of items on the pallet is at least partially based on one or more of packing density, flat top surface, and stability. In some embodiments, the determination that placing one or more spacers in connection with placing a set of N items on a pallet improves the stack of items on the pallet is at least partially based on the determination that the packing density of the stack of items including the set of N items is higher than the packing density when the set of N items is placed on the pallet without one or more spacers. In some embodiments, the determination that placing one or more spacers in connection with placing a set of N items on a pallet improves the stack of items on the pallet is at least partially based on the determination that the top surface is flatter than the top surface when the set of N items is placed on the pallet without one or more spacers. In some embodiments, the determination that placing one or more spacers in connection with placing a set of N items on a pallet improves the stack of items on the pallet is at least partially based on the determination that the stability of the stack of items including the set of N items is higher than the stability when the set of N items is placed on the pallet without one or more spacers. N may be a positive integer (e.g., a positive integer less than the total number of items to be palletized on the pallet at completion).

[0084] As an example, since N may be less than the total number of items to be palletized, the robotic system 100 may be limited in optimizing the stack of items (e.g., the robotic system 100 may only plan to place N items at a time). Thus, by utilizing one or more spacers, the number of degrees of freedom associated with placing N items increases. The robotic system 100 may utilize one or more spacers to optimize the stacking of N items (or to achieve a "sufficiently good" stack with N items, such as a stack that meets a minimum stability threshold). The robotic system 100 may utilize a cost function in connection with determining whether to utilize one or more spacers, the number of spacers to utilize, the placement of the spacers, etc. For example, the cost function may include one or more of a stability value, the time to place one or more items, the packing density of the stack of items, the flatness value or rate of variation of the top surface of the stack of items, and the cost of the supply material, etc.

[0085] According to various embodiments, the control computer 118 controls the robotic system 100 to place a spacer on the container 106 (e.g., a pallet) or on the stack of items in connection with improving the stability of the stack of items on the container 106. As an example, the spacer may be placed in response to a determination that the stability of the stack of items is likely to be improved when the spacer is utilized (e.g., having a probability exceeding a predetermined likelihood threshold). As another example, the control computer 118 may control the robotic system 100 to utilize a spacer in response to a determination that the stability of the stack of items is less than a threshold stability value and / or is likely to be less than the threshold stability value in connection with the placement of a set of items (e.g., a set of N items, where N is an integer).

[0086] According to various embodiments, the control computer 118 may determine the stability of a stack of items based at least in part on a model of the stack of items and / or a simulation of placing a set of one or more items. The computer system may obtain (e.g., determine) a current model of the stack of items and model (e.g., simulate) the placement of the set of items. In connection with modeling the stack of items, a predicted stability of the stack of items may be determined. Modeling the stack of items may include modeling the placement of spacers in connection with modeling the placement of the set of items.

[0087] In some embodiments, the control computer 118 may determine the stability of a stack of items (or a simulated stack of items) based at least in part on one or more attributes of the top surface of the stack of items and / or spacers. For example, a measurement of the degree to which the top surface is flat may be utilized in connection with determining the stability of the stack of items. Placing a box on a flat surface may lead to a stable placement and / or stack of items. As another example, the surface area of a flat region on the top surface may be utilized in connection with determining the stability or predicted stability of the placement of items on the stack of items. The larger the flat region on the top surface of the stack of items is with respect to the bottom surface of the item placed on the stack of items, the more likely it is that the stability of the stack of items satisfies (e.g., exceeds) a threshold stability value.

[0088] According to various embodiments, the robotic system 100 generates a model of a pallet or a stack of one or more items on a pallet, and a spacer or spacer material is determined to be placed in relation to the palletization of the one or more items based at least in part on the model of the pallet or the stack of one or more items on the pallet. The robotic system 100 generates a model of at least the upper surface of the pallet or a stack of one or more items on the pallet, determines a set of N items to be next placed on the pallet (e.g., N is a positive integer), and determines that placing one or more spacers in relation to placing the set of N items on the pallet improves the stack of items on the pallet as compared to the resulting stack from placing the set of N items without spacers, generates one or more control signals for causing an actuator to supply an amount of spacer material corresponding to the one or more spacers, and may provide the one or more control signals to the actuator in relation to placing the set of N items on the pallet.

[0089] According to various embodiments, variations in the items (e.g., item types) among the items to be palletized can complicate the palletization of the items in a stable manner (e.g., in a manner where the stability of the stack of items meets a threshold stability value). In some embodiments, the control computer 118 may only be able to predict a certain number of items to be palletized. For example, the system may have a queue / buffer of N items to be palletized, where N is a positive integer. N can be a portion of the total number of items to be stacked on the pallet. For example, N may be relatively small compared to the total number of items to be stacked on the pallet. Thus, the robotic system 100 may only be able to optimize the stacking of items using the next N known items. For example, the robotic system 100 may determine a plan for stacking one or more items according to the current state (e.g., current model) of the stack of items and one or more attributes associated with the next N items to be stacked. In some embodiments, the use of one or more spacers can provide flexibility in the way the next N items are stacked and / or improve the stability of the stack of items.

[0090] Various embodiments include the palletization of a relatively large number of mixed boxes or items on a pallet. The various boxes and items to be palletized can have different attributes (height, shape, size, rigidity, package type, etc.). Variations in one or more attributes of the various boxes or items can make it difficult to place the items on the pallet in a stable manner. In some embodiments, the robotic system 100 (e.g., the control computer 118) may determine a destination location (e.g., the location where an item is to be placed) for an item having a larger surface area (e.g., a larger bottom surface) than a box or other item beneath the item to be placed. In some embodiments, items having different heights (e.g., different box heights) may be placed in a relatively high region of the pallet (e.g., a height greater than a height threshold equal to a value multiplied by 0.5 of the maximum pallet height, a height greater than a height threshold equal to a value multiplied by 2 / 3 of the maximum pallet height, a height greater than a height threshold equal to a value multiplied by 0.75 of the maximum pallet height, a height greater than a height threshold equal to a value multiplied by another predetermined value of the maximum pallet height).

[0091] According to various embodiments, machine learning processing is performed in connection with improving a supply / usage strategy of spacer material (e.g., a strategy for utilizing spacer material in connection with palletizing one or more items). The robot system 100 may obtain attribute information regarding one or more items to be palletized / depalletized and attribute information regarding one or more spacers utilized in connection with palletizing / depalletizing the one or more items. The attribute information may include one or more of the item's orientation, material (e.g., the type of spacer material), size, weight (or predicted weight), center of gravity, rigidity, dimensions, etc. Also, the robot system 100 may obtain a source location (e.g., information regarding the input conveyor from which the item is picked) and may obtain information regarding the pallet (or set of pallets from which the destination pallet is determined, such as a set of pallets corresponding to an order on which the items are stacked) where the item is to be placed. In connection with determining a plan for picking and placing the item, the robot system 100 may use the information regarding the item (e.g., attribute information, destination location, etc.) to determine a strategy for palletizing the item (e.g., picking and / or placing the item). The palletizing strategy may include suggestions for a pick location (e.g., a location on the item where the robot arm 102 engages the item via an end effector, etc.) and a destination location (e.g., a location on the pallet / container 106 or stack of items). The palletizing strategy may include the force applied to pick the item and / or the holding force for the robot arm 102 to grip the item while moving the item from the source location to the destination location, the trajectory for the robot arm to move the item to the destination location, suggestions for the amount of spacer material (if any) utilized in connection with placing the item at the destination location, and a plan for placing the spacer material.The robot system 100 may utilize machine learning processing to improve the palletizing strategy based at least in part on the relationship between information about an item (e.g., attribute information, destination location, etc.) and one or more of: (i) the performance of picking and / or placing the item (e.g., historical information related to past iterations of picking and placing that item or similar items (items sharing one or more similar attributes, etc.)), (ii) the stability performance of the item after being placed at the destination location with respect to predicted stability generated using a model of the stack of items (e.g., historical information related to past iterations of palletizing that item or similar items (items sharing one or more similar attributes, etc.)), and (iii) the stability performance of the stack of items after the item and / or spacer material have been placed at the destination location with respect to predicted stability generated using a model of the stack of items (historical information related to past iterations of palletizing that item or similar items and / or spacers (items / spacers sharing one or more similar attributes, etc.)). In some embodiments, the robot system 100 may use machine learning processing to improve the utilization of one or more spacers in relation to the palletizing strategy based at least in part on the relationship between information related to the spacer and / or one or more items to be palletized (e.g., information related to the spacer and / or one or more items to be palletized (e.g., attribute information, destination location, etc.) in relation to the palletizing strategy and the stability performance when palletizing a set of items using one or more spacers with respect to the predicted stability (e.g., predicted stability based on a simulation of palletizing the items using a model of the stack of items)).

[0092] The model generated by the robot system 100 may correspond to a geometric model or be at least partially based on a geometric model. In some embodiments, the robot system 100 includes one or more placed items (e.g., items placed by controlling the robot arm 102 of the robot system 100), one or more attributes each associated with at least a portion of the one or more items, and one or more objects within the work space (e.g., a pallet, a robot arm, a shelving system, a chute, or other infrastructure provided within the work space, such as a predetermined object). The geometric model is generated based at least in part on these elements. The geometric model may be determined based at least in part on executing a physics engine on the control computer 118 to model the stacking of items (e.g., modeling the state / stability of a stack of items, etc.). The geometric model may be determined based on the predicted interactions of the various components of the work space (such as the interaction of an item with another item, object, or a simulated force applied to the stack (e.g., to model the use of a forklift or other device to raise / move a pallet or other container on which the stack of items is located)).

[0093] In some embodiments, the system updates the geometric model after the movement (e.g., placement) of each item. For example, the system maintains (e.g., stores) a geometric model corresponding to the state of the work space (such as the state / stability of the stack of items and the position of one or more items within the stack of items). The geometric model is utilized in determining a plan to move an item and controlling the robotic arm to move the item, in relation to the current geometric model. In response to the movement of the item, the system updates the geometric model to reflect the movement of the item. For example, when depalletizing a stack of items, in response to a particular item being picked and moved from the stack of items, the system updates the geometric model such that the particular item is no longer represented as being on the stack and is included within the geometric model at the moved destination position where the particular item is placed, or if the destination position is outside the work space, the geometric model is updated to remove the item. Further, the geometric model is updated to reflect the stability of the stack of items after a particular item has been removed from the stack. As another example, when palletizing a set of items, the system updates the geometric model to reflect the placement of a particular item on / within the stack of items. The system can update the geometric model to include the updated stability of the stack of items, at least in part based on the placement of the items on / within the stack of items (e.g., to reflect the interactions that a particular item has with other items or the interactions between other items based on the placement of a particular item).

[0094] In some embodiments, the system updates the current state (e.g., updates based on an update to a geometric model) either (i) after a predetermined number of item movements (e.g., placements), or (ii) after either a predetermined number of item movements or detection of an anomaly (such as an anomaly that meets one or more anomaly criteria, e.g., the degree of anomaly exceeds an anomaly threshold, etc.). The predetermined number of items (e.g., X items, where X is a positive integer) can be set based on user selection, a robot control system policy, or determined in some other way based on an empirical analysis of the item placement. As an example, the predetermined number of items is set based on a determination that the number of items results in an optimal / best outcome with respect to a predetermined cost function (e.g., a cost function that reflects efficiency, stability, predicted changes in stability, etc.). As an example, the system determines the current estimated state, uses the current estimated state to determine a plan for moving the next X items, and after moving the X items (e.g., stacking or destacking items), the system determines an updated estimated state (e.g., a geometric update / model to reflect the placement of the X items). The system determines the updated state based at least in part on a combination of a geometric model and sensor data (e.g., the current geometric model and current sensor data, etc.). The system then uses the updated state in connection with determining a plan and controlling the robot to place the next set of items according to the plan.

[0095] In some embodiments, the frequency at which the system updates the estimated state is determined dynamically. For example, the system dynamically determines a value X corresponding to the number of items for which the system updates the estimated state after movement. In some embodiments, the system dynamically determines the value X (e.g., corresponding to the frequency of updating the estimated state) based at least in part on one or more attributes of the item (e.g., attributes of items previously moved / placed and / or attributes of the item being moved). As an example, the system determines dynamically the value X based on a determination that an irregularly placed item or a deformable item is placed before (e.g., immediately before) a set of X items with the current estimated state is placed, or that the set of X items includes an irregularly shaped item or a deformable item.

[0096] Examples of dynamically determining / updating the estimated state include (i) controlling a robotic arm to place or remove a first set of X items on or in a pallet or other container, (ii) determining an estimated state (e.g., reflecting the placement / removal of the first set of X items), (iii) using the estimated state to generate or update a plan for controlling the robotic arm to place or remove a second set of Y items on or in the pallet or other container, and / or (iv) controlling the robotic arm to place a first set of Y items on or in the pallet or other container or to remove the first set of Y items from the pallet or other container. In some embodiments, the state of one or more items stacked on or in a pallet or other container is estimated after N items are stacked.

[0097] The above example has been discussed in the context of a system palletizing a set of items on one or more pallets, but the robotic system can be utilized in connection with depalletizing a set of items from one or more pallets.

[0098] FIG. 2 is a diagram showing a robotic system for palletizing and / or depalletizing heterogeneous items according to various embodiments. In some embodiments, system 200 is implemented, at least in part, by process 300 of FIG. 3, process 400 of FIG. 4, process 500 of FIG. 5, process 600 of FIG. 6, process 800 of FIG. 8, and / or process 900 of FIG. 9.

[0099] In the example of the figure, system 200 includes a robotic arm 205. In this example, the robotic arm 205 is fixed, but in various other embodiments, the robotic arm 205 may be fully or partially movable, for example, mounted on a rail, fully movable on a motor-driven chassis, etc. In other implementation examples, system 200 may include a plurality of robotic arms having a working space. As shown in the figure, the robotic arm 205 is utilized to pick any and / or heterogeneous items from one or more conveyors (or other sources) 225 and 230 and items on pallets (e.g., platforms or other containers) such as pallet 210, pallet 215, and / or pallet 220. In some embodiments, other robots not shown in FIG. 2 may be used to push pallets 210, pallet 215, and / or pallet 220 into predetermined positions for loading and unloading and / or to a track or other destination for conveyance, etc.

[0100] As shown in FIG. 2, system 200 may include one or more predetermined zones. For example, pallets 210, 215, and 220 are shown to be disposed within a predetermined zone. The predetermined zone may be structurally indicated in another way, such as by marking or labeling the ground, or using the frames shown in system 200. In some embodiments, the predetermined zones may be arranged radially around robot arm 205. In some cases, a single pallet is inserted into a predetermined zone. In other cases, one or more pallets are inserted into a predetermined zone. Each of the predetermined zones may be disposed within the range of robot arm 205 (e.g., such that robot arm 205 can place an item on the corresponding pallet or depalletize an item from the corresponding pallet). In some embodiments, one of the predetermined zones or a pallet disposed within a predetermined zone is utilized as a buffer area or a staging area where items are temporarily stored (e.g., temporary storage until the items are placed on a pallet within the predetermined zone).

[0101] One or more items may be provided (e.g., conveyed) into the workspace of the robotic arm 205 via the conveyor 225 and / or the conveyor 230. The system 200 may control the speed of the conveyor 225 and / or the conveyor 230. For example, the system 200 may control the speed of the conveyor 225 independently of the speed of the conveyor 230, and the system 200 may control the speeds of both the conveyor 225 and / or the conveyor 230. In some embodiments, the system 200 may stop the conveyor 225 and / or the conveyor 230 (e.g., to ensure sufficient time for the robotic arm 205 to pick and place an item). In some embodiments, the conveyor 225 and / or the conveyor 230 convey items for one or more manifests (e.g., orders). For example, the conveyor 225 and the conveyor 230 may convey items for the same manifest and / or different manifests. Similarly, one or more of the pallets / predetermined zones may be associated with a particular manifest. For example, the pallet 210 and the pallet 215 may be associated with the same manifest. As another example, the pallet 210 and the pallet 220 may be associated with different manifests.

[0102] System 200 may control robot arm 205 to pick items from a conveyor (such as conveyor 225 or conveyor 230) and place the items on a pallet (such as pallet 210, pallet 215, or pallet 220). Robot arm 205 may pick up an item and move the item to a corresponding destination position (e.g., a position on a pallet or on top of a stack on a pallet) based at least in part on a plan related to the item. In some embodiments, System 200 may determine a plan related to an item while the item is on the conveyor, and System 200 may update the plan when picking up the item (e.g., based on an acquired attribute of the item such as weight, or in response to information obtained by sensors in the workspace such as an indication that a collision with another item or a person is expected). System 200 may acquire an identifier associated with the item (such as a barcode, QR code, or other identifier or information on the item). For example, System 200 may scan / acquire the identifier when the item is being carried on the conveyor. In response to acquiring the identifier, System 200 may utilize the identifier in connection with determining the pallet on which the item is to be placed, such as by performing a lookup against a mapping of item identifiers to a manifest and / or a mapping of the manifest to a pallet. In response to determining one or more pallets corresponding to the manifest / order to which the item belongs, System 200 may select the pallet on which to place the item based at least in part on a model or simulation of the stack of items on the pallet and / or the placement of items on the pallet. Also, System 200 may determine a specific location (e.g., a destination position) on the selected pallet where the item is to be placed. Further, a plan for moving the item to the destination position may be determined, including a planned path or trajectory along which the item may be moved.In some embodiments, the plan is updated when the robotic arm 205 is moving an item, e.g., to avoid a predicted collision event, to account for the measured weight of the item being greater than the predicted weight, to reduce the shear force applied to the item when the item is moved, etc., in connection with taking proactive measures to change or adapt to the detected state or condition associated with one or more items / objects within the work space.

[0103] According to various embodiments, system 200 comprises one or more sensors and / or sensor arrays. For example, system 200 may include one or more sensors (sensor 240 and / or sensor 241 na etc.) in the vicinity of conveyor 225 and / or conveyor 230. The one or more sensors may obtain information related to an item on the conveyor, such as an identifier or information on a label of the item, or an attribute of the item such as the dimensions of the item. In some embodiments, system 200 comprises one or more sensors and / or sensor arrays that obtain information regarding a predetermined zone and / or a pallet within the zone. For example, system 200 may include sensor 242 that obtains information related to pallet 220 or a predetermined zone in which pallet 220 is located. The sensors may include one or more 2D cameras, 3D (e.g., RGBD) cameras, infrared, and other sensors for generating a three-dimensional view of the work space (or a portion of the work space such as a pallet and a stack of items on the pallet). Information regarding the pallet may be utilized in connection with determining the state of the pallet and / or a stack of items on the pallet. As an example, system 200 may generate a model of a stack of items on the pallet, at least in part based on information regarding the pallet. System 200 may then utilize the model in connection with determining a plan for placing an item on the pallet. As another example, system 200 may determine that a stack of items is complete, at least in part based on information regarding the pallet.

[0104] According to various embodiments, system 200 determines (or updates) a plan for picking and placing items based at least in part on the determination of the stability of stacks on the pallets. System 200 may determine a model of the stack for one or more of pallets 210, 215, and / or 220, and system 200 may utilize the model in connection with determining the stack on which to place an item. As an example, if the next item to be moved is relatively large (e.g., such that the surface area of the item is large relative to the footprint of the pallet), system 200 may determine that placing the item on pallet 210 may cause the stack thereon to become unstable (e.g., because the surface of the stack is non-planar). In contrast, system 200 may determine that placing a relatively large (e.g., flat) item on the stack of pallet 215 and / or pallet 220 may result in a relatively stable stack. The upper surface of the stack for pallet 215 and / or pallet 220 is relatively flat, and placing a relatively large item thereon may not lead to stack instability. System 200 may determine that the predicted stability when placing an item on pallet 215 and / or pallet 220 may be greater than a predetermined stability threshold, or that placing the item on pallet 215 or pallet 220 may result in an optimal placement of the item (e.g., at least with respect to stability).

[0105] System 200 may communicate the state of the pallet within a predetermined zone and / or the state of the operation of the robotic arm 205. The state of the pallet and / or the state of the operation of the robotic arm may be communicated to a user or other human operator. For example, System 200 may include a communication interface (not shown) through which information regarding the state of System 200 (e.g., the state of the pallet, a predetermined zone, the robotic arm, etc.) is communicated to a terminal (such as an on-demand remote operation device and / or a terminal utilized by a human operator). As another example, System 200 may include a status indicator (such as status indicator 245 and / or status indicator 250) in the vicinity of a predetermined zone.

[0106] The status indicator 250 may be utilized in connection with communicating the status of the pallet within the corresponding predetermined zone and / or the status of the operation of the robotic arm 205. For example, if the system 200 is active with respect to a predetermined zone in which the pallet 220 is disposed, the status indicator may indicate so by illuminating a green light or otherwise communicating information or indication of the active state via the status indicator 250. The system 200 may be determined to be in an active state with respect to a predetermined zone in response to determining that the robotic arm 205 is actively palletizing one or more items on the pallet within the predetermined zone. As another example, if the system 200 is inactive with respect to a predetermined zone in which the pallet 220 is disposed, the status indicator may indicate so by illuminating a red light or otherwise communicating information or indication of the active state via the status indicator 250. The system 200 may be determined to be inactive in response to a determination that the robotic arm 205 is not actively palletizing one or more items on the pallet within the predetermined zone (e.g., in response to the user pausing the corresponding predetermined zone (or cell)) or in response to a determination that the palletization of the items on the pallet 220 is complete. A human operator or user may utilize the status indicator as an indication of whether it is safe to enter the corresponding pre - determined zone. For example, a user performing an operation of removing a completed pallet from and / or inserting an empty pallet into the corresponding predetermined zone may refer to the corresponding status indicator and safely enter the predetermined zone when the status indicator indicates that the operation within the predetermined zone is inactive.

[0107] According to various embodiments, system 200 may determine an abnormal condition regarding a pallet and / or items stacked on the pallet using information obtained by one or more sensors within the work space. For example, system 200 may determine that the pallet is displaced with respect to the robotic arm 205 and / or a corresponding predetermined zone, at least in part based on information obtained by the sensors. As another example, system 200 may determine, at least in part based on information obtained by the sensors, that the stack is unstable, that the items on the pallet are experiencing turbulent flow, etc. In response to detecting an abnormal condition, the system may communicate a suggestion of the abnormal condition to an on-demand remote operation device or other terminal utilized by an operator. In some embodiments, in response to detecting an abnormal condition, system 200 may automatically set the pallet and / or corresponding zone to an inactive state. In addition to, or instead of, notifying the operator of the abnormal condition, system 200 may perform proactive measures. Proactive measures may include controlling the robotic arm 205 to at least partially correct the abnormal condition (e.g., re-stack dropped items, realign the pallet, etc.). In some implementations, in response to detecting that an inserted pallet is displaced (e.g., inaccurately inserted into a predetermined zone), system 200 may calibrate the process for modeling the stack and / or the process for placing items on the pallet to correct the displacement. For example, system 200 may generate and utilize an offset corresponding to the displacement when determining and implementing a plan for placing items on the pallet. In some embodiments, system 200 performs proactive measures to at least partially correct the abnormal condition in response to determining that the degree of the abnormality is less than a threshold.Examples of determining that the degree of abnormality is less than the threshold value include: (i) determining that the deviation of the pallet is less than the threshold deviation value; (ii) determining that the number of misaligned items, misarranged items, or dropped items is less than the threshold number; (iii) determining that the size of the misaligned items, misarranged items, or dropped items meets the size threshold, etc.

[0108] A human operator may communicate with the system 200 via a network (such as a wired network and / or a wireless network). For example, the system 200 may include a communication interface for connecting the system 200 to one or more networks. In some embodiments, a terminal connected to the system 200 via a network provides a user interface through which a human operator can provide commands to the system 200 and / or obtain information regarding the state of the system 200 (such as the state of a robotic arm, the state of a specific pallet, the state of the palletization process for a specific manifest, etc.). The human operator may provide instructions to the system 200 by inputting to the user interface. For example, the human operator may use the user interface to temporarily stop the robotic arm, temporarily stop the palletization process for a specific manifest, temporarily stop the palletization process for a specific pallet, switch the state of the pallet / predetermined zone between active / inactive, etc.

[0109] In various embodiments, elements of the system 200 may be added, removed, replaced, etc. In such examples, the control computer initializes and registers the new element, executes an operation test, and starts / resumes the kitting operation to incorporate, for example, the newly added element.

[0110] According to various embodiments, system 200 determines (e.g., calculates, maintains, stores, etc.) an estimated state for each pallet (e.g., pallet 210, pallet 215, and / or pallet 220) within a plurality of zones, or a collective estimated state for a set of pallets between a plurality of zones, or both individual estimated states and collective estimated states. In some embodiments, the individual estimated states and collective estimated states are determined in a manner similar to the estimated states described in connection with the robotic system 100 of FIG. 1.

[0111] According to various embodiments, system 200 comprises a vision system with one or more sensors (e.g., sensor 240, sensor 241, etc.). In various embodiments, system 200 utilizes sensor data and geometric data (e.g., geometric models) in connection with determining a location to place one or more items on a pallet (or in connection with depalletizing one or more items from a pallet). System 200 models the state of a pallet (or a stack of items on a pallet) using different data sources. For example, system 200 may estimate the location of one or more items on a pallet and one or more features (or attributes) associated with the one or more items (e.g., the size of an item). The one or more features associated with the one or more items may include the size of the item (e.g., the dimensions of the item), the center of gravity, the rigidity of the item, the type of package, the location of an identifier, etc.

[0112] System 200 determines a geometric model based at least in part on one or more attributes of one or more items within the work space. For example, the geometric model reflects the respective attributes of a set of items (e.g., one or more within a first set that is palletized / stacked and a second set of items that is to be palletized / stacked, etc.). The of attributesExamples include the size of the item (e.g., the dimensions of the item), the center of gravity, the rigidity of the item, the type of package, the position of the identifier, the deformability of the item, the shape of the item, etc. Various other attributes of the item or object within the work space may be implemented.

[0113] The model generated by system 200 may correspond to, or be at least partially based on, a geometric model. In some embodiments, system 200 is at least partially based on one or more placed items (e.g., items placed by system 200 controlling robot arm 205), one or more attributes each associated with at least a portion of the one or more items, one or more objects within the work space (e.g., a pallet, a robot arm, a shelving system, a chute, or other infrastructure provided within the work space, such as a predetermined object), etc., to generate a geometric model. The geometric model may be determined based at least in part on executing a physics engine on a control computer to model the stacking of items (e.g., modeling the state / stability of a stack of items, etc.). The geometric model may be determined based on the predicted interactions of the various components of the work space (the interaction of an item with another item, object, (e.g., modeling the use of a forklift or other device to raise / move a pallet or other container on which a stack of items is located) the simulated forces applied to the stack, etc.).

[0114] According to various embodiments, system 200 determines a best estimate of the state of the work space using a geometric model and sensor data. System 200 can perform adjustments due to noise (e.g., cancellation of noise) in one or more of the geometric model and / or sensor data. In some embodiments, system 200 detects an anomaly or difference between a state according to the geometric model and a state according to the sensor data. In response to determining that there is an anomaly or difference between the geometric model and the sensor data, system 200 can perform a best estimate of the state regardless of the anomaly or difference. For example, system 200 determines whether to use the geometric model or the sensor data, or a combination of the geometric model and the sensor data (e.g., interpolation between them), etc. In some embodiments, system 200 determines an estimated state for each segment (e.g., for each voxel, item, or object, etc. within the work space). For example, a first portion of the work space is estimated using only the geometric model, a second portion of the work space is estimated using only the sensor data (e.g., if there is an anomaly in the geometric model), and / or a third portion of the work space may be estimated based on a combination of the geometric model and the sensor data. Using the example shown in FIG. 2, system 200 utilizes only the geometric model to determine individual estimated states for the stack of items on pallet 210 in connection with determining a collective estimated state, utilizes only the sensor data to determine individual estimated states for the stack of items on pallet 215, and may utilize a combination of the respective geometric model and sensor data for the stack of items on pallet 220.

[0115] The above example has been discussed in the context of a system palletizing a set of items on one or more pallets, but the robotic system can be utilized in connection with depalletizing a set of items from one or more pallets.

[0116] FIG. 3 is a flowchart showing a process for palletizing one or more items according to various embodiments. In some embodiments, process 300 is performed, at least in part, by the robot system 100 of FIG. 1 and / or the system 200 of FIG. 2.

[0117] In step 310, a list of items is obtained. The list of items can correspond to a set of items to be palletized together on one or more pallets. According to various embodiments, the set of items to be palletized is determined based at least in part on a manifest or an instruction to fulfill an order. For example, in response to receiving an order, a list of the order's items may be generated. As another example, a list of items corresponding to multiple orders being sent to the same recipient may be generated.

[0118] The items may be located on shelves or other locations within the warehouse. To palletize the items, the items are moved to a robot system that palletizes the items. For example, the items may be placed on one or more conveyors that move the items within the range of one or more robot arms that palletize the items onto one or more pallets. In response to obtaining the list of items, at least some of the items are associated with a particular robot arm, a predetermined zone corresponding to the particular robot arm, and / or a particular pallet (e.g., a pallet identifier, a pallet placed in a predetermined zone).

[0119] In operation 320, planning (or replanning) is performed to generate a plan for picking / placing items based on a list of items and available sensor information. The plan may include one or more strategies for fetching one or more items on the list of items and placing the corresponding item(s) on one or more corresponding conveyors for transporting the item(s) to a robotic arm. According to various embodiments, the order in which items on the list of items are provided to an appropriate robotic arm for palletizing is determined at least in part based on the list of items.

[0120] The order in which items are placed on the conveyor may be based at least roughly on the items and the predicted stack of items on one or more pallets. For example, a system for determining the order in which to place items on a conveyor may generate a model of the predicted stack of items (e.g., to first convey the item(s) forming the base / bottom of the stack and then convey items stepwise towards the top of the stack) and determine the order based on the model. If the items on the list of items are palletized onto multiple pallets, the items expected to form the base / bottom of each stack (or otherwise be relatively close to the bottom of the stack) may be placed on the conveyor before the items expected to be stacked substantially in the middle or at the top of the stack. The various items palletized onto multiple pallets may be interspersed with each other, and the robotic system may sort the items upon arrival at the robotic arm (e.g., the robotic arm may pick and place the items on the appropriate pallet based at least on the item (such as the item identifier or item attributes)). Thus, the items corresponding to the base / bottom of the corresponding stack may be interspersed with each other, and the various items for each pallet / stack may be placed on the conveyor as the corresponding stack is being built.

[0121] A computer system may generate a model of one or more prediction stacks for items belonging to a list of items. The model may be generated at least in part based on one or more thresholds (such as a fitness threshold or a stability threshold, other packing metrics (e.g., density), etc.). For example, the computer system may generate a model of a stack where the predicted stability value meets (e.g., exceeds) the stability threshold. The model may be generated using machine learning processing. The machine learning processing may be repeatedly updated based on historical information such as previous stacks of items (e.g., the attributes of the items within the previous stack, performance metrics regarding the previous stack (stability, density, fitness, etc.)). In some embodiments, a model of a stack for palletizing items on a list of items is generated at least in part based on one or more attributes of the items.

[0122] Various attributes of the items may be obtained before or during determining the plan. The attributes may include the size of the item, the shape of the item, the type of the item's package, the identifier of the item, the center of gravity of the item, an indication of whether the item is fragile, an indication of the top or bottom of the item, etc. As an example, one or more attributes related to at least a part of the item may be obtained at least in part based on the list of items. The one or more attributes may be obtained at least in part based on information obtained by one or more sensors and / or by performing a lookup in an attribute mapping for the item (e.g., item type, item identifier (serial number, model number, etc.)).

[0123] In some embodiments, generating a model of one or more predictive statistics for items belonging to a list of items includes generating (e.g., determining) an estimated state for a work space (e.g., a work space including one or more stacks of items). The computer system determines a plan for moving (e.g., palletizing or depalletizing, etc.) a set of one or more items, and the computer system controls a robot (e.g., a robotic arm) to move the set of one or more items according to the plan. In response to moving a set of one or more items according to the plan, the computer system determines the estimated state of the work space. For example, the computer system updates the estimated state at least partially based on the movement of the set of items. In some embodiments, the estimated state is determined at least partially based on a geometric model, sensor data, or a combination of the geometric model and sensor data in response to a determination that the geometric model and the sensor data are inconsistent (e.g., the difference between the geometric model and the sensor data is greater than a predetermined difference threshold, or includes an anomaly, etc.). The updated / current estimated state reflects the movement of the set of one or more items (e.g., in the case of palletizing, the updated estimated state includes information regarding the placement of the set of one or more items onto the stack, etc.). In response to determining the updated / current estimated state, the computer system determines a plan for moving another set of one or more items, and the computer system controls the robot to move the another set of one or more items according to the plan.

[0124] In some embodiments, the computer system updates the current state (e.g., updates based on an update to a geometric model) either (i) after a predetermined number of item movements (e.g., placements), or (ii) after either a predetermined number of item movements or detection of an anomaly (such as an anomaly that meets one or more anomaly criteria (e.g., the degree of anomaly exceeds an anomaly threshold, etc.)). The predetermined number of items (e.g., X items, where X is a positive integer) can be set based on user selection, a robot control system policy, or determined in some other way based on an empirical analysis of the item placement. As an example, the predetermined number of items is set based on a determination that the number of items results in an optimal / best result with respect to a predetermined cost function (e.g., a cost function that reflects efficiency, stability, predicted changes in stability, etc.). As an example, the computer system determines the current estimated state, uses the current estimated state to determine a plan for moving the next X items, and after moving the X items (e.g., stacking or destacking items), the computer system determines an updated estimated state (e.g., a geometric update / model to reflect the placement of the X items). The computer system determines the updated state based at least in part on a combination of a geometric model and sensor data (e.g., the current geometric model and current sensor data, etc.). The computer system then utilizes the updated state in connection with determining a plan and controlling the robot to place the next set of items according to the plan.

[0125] According to various embodiments, the computer system determines an estimated state based at least in part on performing an interpolation between a geometric model and sensor data. For example, the system performs an interpolation on a particular portion of the geometric model and a corresponding portion of the sensor data (e.g., the particular portion can correspond to a difference between the geometric model and sensor data that exceeds a difference threshold or includes an anomaly).

[0126] Various interpolation techniques may be implemented. Certain portions of the geometric model may correspond to certain points (or sets of points) in the point cloud for the geometric model, and the corresponding portions of the sensor data may be the sensor data for that particular point in the point cloud for the sensor data, and so on. In some embodiments, the system performs adaptive interpolation between the geometric model and the sensor data. In some embodiments, the system performs non - adaptive interpolation between the geometric model and the sensor data. Examples of adaptive interpolation processes include nearest - neighbor interpolation, bilinear interpolation, bicubic interpolation, spline interpolation, sinc interpolation, Lanczos interpolation, and the like. Various other interpolation processes may be performed in connection with determining the estimated state.

[0127] In step 330, the item is picked and moved along a (predetermined / planned) trajectory to a location near where the item is to be placed on the corresponding conveyor, and placed at the destination location, according to the plan determined and / or updated in step 320.

[0128] In the example of the figure, (re)planning and plan execution (steps 320, 330) continue until the high-level goal of providing the items on the list of items is completed (step 340), and when completed, process 300 ends. In various embodiments, replanning (step 320) may be triggered by conditions such as the arrival of unexpected and / or unidentifiable items, sensor readings indicating attributes having values other than those predicted based on item identification and / or associated item model information, and the like. Other examples of unexpected conditions include determining that an expected item is missing, re-evaluating item identification to determine that the item is other than originally identified, detecting an item weight or other attribute that does not match the identified item, dropping the item, or having to re-grasp the item, determining that a later-arriving item is too heavy to be stacked on one or more other items as originally planned and / or as currently planned, and detecting instability in a set of items stacked in a container, but are not limited thereto.

[0129] Figure 4 is a flowchart showing a process for determining the current state of a pallet and / or stack of items, according to various embodiments. In some embodiments, process 400 is performed, at least in part, by the robot system 100 of FIG. 1 and / or the system 200 of FIG. 2.

[0130] According to various embodiments, process 400 is invoked in response to a determination that the system is to determine a plan for moving one or more items. In some embodiments, process 400 is invoked with respect to the placement of each item in a set of items to be stacked. In some embodiments, process 400 is invoked at a predetermined frequency / interval, such as after a predetermined number of items have been moved from the last decision of the estimated state. In addition to any of the foregoing, the system may invoke process 400 based at least in part on the attributes of previously placed items or currently placed items. For example, process 400 is invoked in response to a determination that a previously placed item or the current item has an irregular shape (e.g., a size / dimension exceeding a threshold size, a non-rectangular shape, etc.) and / or is deformable (e.g., the item has a predicted deformability exceeding a deformability threshold, or the item has a soft package (such as a polybag), etc.). As another example, the system invokes process 400 in response to a determination that a previously placed item or the current item to be placed can cause instability (e.g., a threshold instability) in the stack of items.

[0131] In step 410, information related to the placed item is obtained. The system obtains information related to the item based on pre-stored information related to the item (e.g., if it is known in advance on a manifest that the item is to be palletized, etc.) or information obtained by a vision system (e.g., item identifier, size, package type, weight, etc.). The information related to the placed item may correspond to the attributes of the item, etc.

[0132] In operation 420, geometric data regarding stacks of items including the placed items is received. In some embodiments, the system obtains a current geometric model updated to reflect the placement of the items. The current geometric model may be an estimated state utilized in connection with determining the placement of the items or the placement of the set of items to which the item belongs (e.g., a predicted placement based on the actions of a robot placing the items according to a plan).

[0133] In operation 430, sensor data obtained by the vision system is acquired. In some embodiments, the system obtains the sensor data obtained by the vision system. For example, the system commands the vision system to capture the current state of the work space, and the system uses the information regarding such capture to obtain the sensor data.

[0134] In operation 440, the current state or model is updated based at least in part on the geometric data and the sensor data. The system determines a predicted state based at least in part on the geometric model or the sensor data or both the geometric model and the sensor data, such as in response to determining the difference between the geometric model and the sensor data.

[0135] According to various embodiments, the system determines an estimated state (e.g., an updated estimated state) by performing interpolation based at least in part on at least a portion of the geometric data and at least a portion of the sensor data. In some embodiments, the interpolation process performed with respect to the first portion of the geometric model and the first portion of the sensor data to obtain the first portion of the estimated state is different from the interpolation performed with respect to the second portion of the geometric model and the second portion of the sensor data to obtain the second portion of the estimated state.

[0136] In some embodiments, the system utilizes an updated estimated state in connection with determining a plan to move one or more items and controlling a robot to move one or more items according to the plan.

[0137] In step 450, a determination is made as to whether process 400 is complete. In some embodiments, process 400 is determined to be complete in response to a determination that no further updates to the estimated state are performed, no further items are moved, the user has terminated the system, the administrator has indicated a pause or stop of process 400, and the like. In response to a determination that process 400 is complete, process 400 ends. In response to a determination that process 400 is not complete, process 400 returns to step 410.

[0138] FIG. 5 is a flowchart showing a process for determining the current state of a pallet and / or stack of items according to various embodiments. In some embodiments, process 500 is performed, at least in part, by the robot system 100 of FIG. 1 and / or the system 200 of FIG. 2.

[0139] According to various embodiments, process 500 is invoked in response to a determination that the system determines a plan for moving one or more items. In some embodiments, process 500 is invoked with respect to the placement of each item in a set of items to be stacked. In some embodiments, process 500 is invoked at a predetermined frequency / interval, such as after a predetermined number of items have been moved from the last decision of the estimated state. In addition to any of the foregoing, the system may invoke process 500 based at least in part on the attributes of previously placed items or currently placed items. For example, process 500 is invoked in response to a determination that a previously placed item or the current item has an irregular shape (e.g., a size / dimension exceeding a threshold size, a non-rectangular shape, etc.) and / or is deformable (e.g., the item has a predicted deformability exceeding a deformability threshold, or the item has a soft package (such as a polybag), etc.). As another example, the system invokes process 500 in response to a determination that a previously placed item or the current item to be placed may cause instability (e.g., a threshold instability) in a stack of items.

[0140] According to various embodiments, process 500 is executed by one or more servers that provide services to one or more robotic systems (e.g., a palletization / depalletization system, a singulation system, a kitting system, etc.) for moving items. For example, the system receives a query from one or more robotic systems and provides an estimated state. The system may maintain the current estimated state of one or more stacks of items (e.g., based on the operation of a robotic system that picks and places items with respect to a stack of items and / or based on sensor data).

[0141] In operation 510, information related to the placed item is received. In some embodiments, the system obtains information related to the item based on pre-stored information related to the item. For example, the system performs a lookup regarding the mapping of attributes for the item (e.g., based on an identifier associated with the item). In some embodiments, the information related to the placed item is received based at least in part on attributes of the item (e.g., rigidity / deformability, size, weight, etc. of the item or package) obtained by one or more sensors (e.g., a vision system) when placing the item.

[0142] In operation 520, geometric data regarding a stack of items including the placed item is received. In some embodiments, the system obtains a current geometric model updated to reflect the placement of the items. The current geometric model may correspond to a predicted state of the stack of items (e.g., the state of the stack of items that the system infers the items are placed accurately according to the plan, etc.). For example, the system obtains a geometric model used in connection with determining a plan to move the placed item (or a set of items to which the placed item belongs), and the system updates the geometric model to reflect the placement of the placed item.

[0143] In operation 530, sensor data obtained by the vision system is obtained. In some embodiments, the system obtains sensor data obtained by the vision system. For example, the system commands the vision system to capture the current state of the work space, and the system uses the information regarding such capture to obtain the sensor data.

[0144] In operation 540, the current state or model is updated based at least in part on geometric data and sensor data. The system determines a predicted state based at least in part on the geometric model or the sensor data or both the geometric model and the sensor data, in response to, for example, determining a difference between the geometric model and the sensor data.

[0145] According to various embodiments, the system determines an estimated state (e.g., an updated estimated state) by performing interpolation based at least in part on at least a portion of the geometric data and at least a portion of the sensor data. In some embodiments, the interpolation process performed with respect to the first portion of the geometric model and the first portion of the sensor data to obtain the first portion of the estimated state is different from the interpolation performed with respect to the second portion of the geometric model and the second portion of the sensor data to obtain the second portion of the estimated state.

[0146] In some embodiments, the updated estimated state is utilized in connection with determining a plan to move one or more items and controlling a robot to move one or more items according to the plan.

[0147] In operation 550, the updated current state or model is provided to the robotic system. In some embodiments, in response to determining an estimated state (e.g., an initial estimated state, or an estimated state updated based on a previous pick / place of an item), the system provides the estimated state to a robotic system that determines a plan to move one or more other items using the estimated state. In some embodiments, the system provides the estimated state in response to a request from the robotic system. For example, the system includes one or more servers (e.g., a remote server) that provide services to one or more robotic systems (e.g., a palletization / depalletization system, a simulation system, a kitting system, etc.) that move items.

[0148] In operation 560, a determination is made as to whether process 500 has been completed. In some embodiments, process 500 is determined to be complete in response to decisions such as no further update of the estimated state is performed, no further items are moved, the user has terminated the system, the administrator has indicated a pause or stop of process 500, etc. In response to a determination that process 500 has been completed, process 500 ends. In response to a determination that process 500 has not been completed, process 500 returns to operation 510.

[0149] FIG. 6 is a flowchart showing a process for determining the current state of a pallet and / or stack of items according to various embodiments. In some embodiments, process 600 is performed, at least in part, by the robot system 100 of FIG. 1 and / or the system 200 of FIG. 2.

[0150] In operation 610, a model of the state is obtained based on geometric data. In some embodiments, the system obtains the current geometric model of the system. For example, the system updates the previous geometric model such that it includes item movements (e.g., placing an item onto a stack, removing an item from a stack, etc.) that the system has controlled the robotic arm to perform since the previous geometric model / estimated state was determined.

[0151] In operation 620, a model of the state is obtained based on sensor data. The system obtains sensor data based on information captured by one or more sensors (e.g., a vision system) provided in the work space.

[0152] In operation 630, a difference between the model based on geometric data and the model based on sensor data is determined. The system calculates the difference between the geometric model and the sensor data, or the geometric model andQualitatively determine the differences between sensor data, etc. For example, the system calculates the differences between the positions / boundaries of various items in a stack of items. As another example, the system determines whether an item is included in one of the geometric model and the sensor data and not found in the other when the field of view of the vision system is blocked, or when an item has fallen from a stack of items (e.g., not predicted or reflected in the geometric model).

[0153] In step 640, it is determined whether the difference exceeds a predetermined threshold. In some embodiments, the system compares the difference between the geometric model and the sensor data with a predetermined difference threshold. The predetermined difference threshold may correspond to an abnormality that is not expected to be resolved by interpolation or combination of the geometric data and the sensor data (e.g., an abnormality that the system determines requires human intervention).

[0154] In response to determining that the difference does not exceed the predetermined threshold in step 640, process 600 proceeds to step 650 where the current model or state is updated based at least in part on the difference between the model determined based on the geometric data and the model determined based on the sensor data.

[0155] In response to determining that the difference exceeds the predetermined threshold in step 640, process 600 proceeds to step 660 where a suggestion is provided to the user. For example, in response to determining that the difference exceeds the predetermined threshold, the system decides to request manual human intervention. The system can prompt the user to confirm whether to use either the geometric model or the sensor data, a combination of the geometric model and the sensor data, or neither (e.g., , mo to utilize user-defined aspects of the model, etc.).

[0156] In some embodiments, the suggestion provided to the user is a suggestion of a difference between a geometric model for a part of the work space (such as a part of a stack of items) and a model based on sensor data. For example, the suggestion indicates a specific part of a stack of items where the difference between the geometric model part and the sensor data exceeds a predetermined threshold.

[0157] In some embodiments, the suggestion provided to the user is a suggestion of a difference regarding the entire work space (e.g., the entire geometric model and the entire sensor data). For example, the system prompts the user to clarify an anomaly that has appeared between the geometric model and the sensor data for the work space. As another example, the system prompts the user to define a bias or weighting applied to the geometric model and the sensor data. As a further example, the system prompts the user to define an interpolation process to be performed in connection with determining an estimated state based on both the geometric model and the sensor data.

[0158] In step 670, an input is received from the user. The user input can be a selection for utilizing the geometric model or the sensor data or a combination of the geometric model and the sensor data. In some embodiments, the user defines whether to utilize the geometric model or the sensor data for a specific part of the work space state (e.g., the top of a stack of items, a part of a stack of items where the vision system's field of view is blocked, etc.).

[0159] In step 680, the current state / model is updated based at least in part on the user input. For example, the estimated state is updated to reflect the user selection indicating which of the geometric model, or the sensor data, or a combination of the geometric model and the sensor data to use as the ground truth for the current state / model.

[0160] In operation 690, a determination is made as to whether process 600 has been completed. In some embodiments, process 600 is determined to be complete in response to a determination that no further updates to the estimated state are performed, no further items are moved, the user has terminated the system, the administrator has indicated a pause or stop of process 600, etc. In response to a determination that process 600 has been completed, process 600 ends. In response to a determination that process 600 is not complete, process 600 returns to operation 610.

[0161] FIG. 7A is a diagram showing the state of a pallet / item stack when using geometric data, according to various embodiments. FIG. 7B is a diagram showing the state of a pallet / item stack when using vision system or sensor data, according to various embodiments.

[0162] State 700 corresponds to the state of a stack of items according to geometric data. For example, state 700 corresponds to a geometric model of a stack of items. As shown in FIG. 7A, the geometric model includes items placed precisely. The geometric model can correspond to the ideal state of a stack of items when a robotic arm picks / places items with respect to the stack of items according to a plan (e.g., without deviating from the modeled plan, etc.).

[0163] State 750 corresponds to the state of a stack of items according to sensor data. For example, state 750 corresponds to a stack of items based on information captured by a vision system provided in the work space. As shown in FIG. 7B, state 750 includes a stack of items that are misaligned or a portion of the stack of items for which no information is provided (e.g., a portion of the stack where the vision system's field of view is blocked, etc.).

[0164] As shown in FIG. 7A, state 700 includes item 704a stacked on a set of items including item 706a, item 708a, item 710a, item 712a, and item 714a. However, state 750 has a gap 752 where the sensor data does not reflect the presence of an item. As an example, when there is a lot of noise in the sensor data, or when the field of view of the vision system is blocked for such a portion of the work space, a gap in the stack of items may be included in the sensor data. In other words, the sensor data for state 750 does not include information about items 708a, 710a, 712a, and 714a included in the geometric model. If sensor data rather than the geometric model is used, or if it has a strong bias, the system may determine that the stack of items is unstable based on item 704b being placed on an item not supported by gap 752.

[0165] State 700 includes item 706a, which appears to be accurately placed relative to various other items in the stack of items. In contrast, state 750 includes item 706b, which is not accurately / straight placed on the item below it. For example, the sensor data reflects the space between item 706b and an adjacent item. Examples of the causes of the position / placement of item 706b include (i) movement of the item after placement by a robotic arm, or inaccurate placement of the item by a robotic arm (e.g., relative to a plan determined according to the estimated state at that time), (ii) imprecise placement of the item by a robotic arm, (iii) noise in the sensor data, and / or (iv) inaccuracies in the sensor data.

[0166] State 700 includes item 716a, which appears to be precisely positioned relative to various other items in the stack of items. In contrast, state 750 includes item 716b, which is not precisely positioned relative to adjacent items. For example, the sensor data reflects gap 754 between item 716b and an adjacent item. Examples of causes for the position / placement of item 716b include (i) movement of the item after placement by a robotic arm, or inaccurate placement of the item by a robotic arm (e.g., relative to a plan determined according to the estimated state at that time), (ii) imprecise placement of the item by a robotic arm, (iii) noise in the sensor data, and / or (iv) inaccuracies in the sensor data.

[0167] State 700 includes item 718a, which appears to be precisely positioned relative to various other items in the stack of items. In contrast, state 750 includes gap 758 at the same position within the stack where the geometric model includes item 718a. For example, the sensor data reflects gap 758 under item 702b and an adjacent item. Examples of causes for gap 758 include (i) movement of the item after placement by a robotic arm, or inaccurate placement of the item by a robotic arm (e.g., relative to a plan determined according to the estimated state at that time), (ii) imprecise placement of the item by a robotic arm, (iii) noise in the sensor data, and / or (iv) inaccuracies in the sensor data. If the system uses sensor data for the portion of the stack of items corresponding to gap 758, the system may consider the stack of items to be unstable. For example, the system may consider item 702b to be partially unsupported and unstable.

[0168] State 700 includes item 702a, which appears to be precisely positioned relative to various other items in the stack of items. In contrast, state 750 includes item 702b, which is not precisely / straightly positioned on the item below it. For example, the sensor data reflects a deviation (e.g., overhang) of item 702b relative to the item below it. Examples of the cause of deviation 756 include (i) movement of the item after placement by the robotic arm, or inaccurate placement of the item by the robotic arm (e.g., relative to the plan determined according to the estimated state at that time), (ii) imprecise placement of the item by the robotic arm, (iii) noise in the sensor data, and / or (iv) inaccuracy in the sensor data.

[0169] Figure 7C is a diagram showing the state of a pallet / item stack when using geometric data and vision system or sensor data according to various embodiments.

[0170] As shown in FIG. 7C, the estimated state 775 includes the state of a stack of items based on a combination of a geometric model (e.g., state 700) and sensor data (e.g., state 750). In some embodiments, the system determines the difference between the geometric model and the sensor data and determines how to reflect that difference in the estimated state. For example, with respect to the difference in a particular portion of the stack of items, the system determines whether to (i) use only the geometric model to model the particular portion of the stack in the estimated state, (ii) use only the sensor data to model the particular portion of the stack in the estimated state, or (iii) use a combination of the geometric model and the sensor data. In response to determining to use a combination of the geometric model and the sensor data for a particular portion of the estimated state, the system determines how to combine the geometric model and the sensor data. For example, the system determines an interpolation process. The system may determine how to combine the geometric model and the sensor data based at least in part on a particular difference between the geometric model and the sensor data (e.g., the system may determine that the difference is due to sensor data noise, or an occluded field of view of the vision system, etc.).

[0171] The estimated state 775 includes item 704c. The corresponding item 704a in state 700 is shown to be placed exactly next to item 702a, and the corresponding item 704b in state 750 is shown to be placed at a distance from item 702b.

[0172] In some embodiments, the system determines that the apparent positional difference between items 704a and 704b is due to one or more of (i) movement of the items after placement by the robotic arm, or inaccurate placement of the items by the robotic arm (e.g., relative to the plan determined according to the estimated state at that time), (ii) imprecise placement of the items by the robotic arm, (iii) noise in the sensor data, and / or (iv) inaccuracies in the sensor data. The system may perform a determination of other causes of the difference. In response to determining the predicted cause of the difference in the positions of items 704a and 704b, the system determines how to model item 704c. For example, the system determines whether to (i) use only a geometric model to model a particular portion of the stack in the estimated state, (ii) use only sensor data to model a particular portion of the stack in the estimated state, or (iii) use a combination of the geometric model and the sensor data. In some embodiments, the system determines how to model item 704c based at least in part on the determined predicted cause of the difference in the positions of items 704a and 704b.

[0173] In some embodiments, the system determines to model item 704c based on a combination of a geometric model and sensor data. For example, the position of item 704c is determined based on an interpolation of a geometric model (e.g., the position of item 704a) and sensor data (e.g., the position of item 704b). As another example, the position of item 704c is determined as the midpoint between the geometric model and the sensor data.

[0174] In some embodiments, the system determines that the difference in the positions of items 704a and 704b is statistically related but not complete. For example, the system considers that the difference in the positions of items 704a and 704b is due to the inaccuracy of the item placement. Thus, the system determines to use the position of item 704c in the estimated state as the position of item 704b.

[0175] The estimated state 775 includes item 718c. In state 700, the corresponding item 718a is shown to be accurately placed under item 702a, and state 750 has a gap 758 at the position where item 718a is included in state 700. The system determines the difference between states 700 and 750 (e.g., the presence of item 718a and the presence of gap 758), and the system determines how to model the corresponding part of the stack in the estimated state. In the example shown in the figure, the system determines that the sensor data is inaccurate (e.g., because the data includes gap 758 at the location where item 718 is placed). For example, the system may determine that the field of view of the vision system for such a part of the stack is blocked. Thus, the system determines to utilize the information of item 718a in the geometric model in relation to modeling the position of item 718c in the estimated state. For example, the system determines to use only the geometric model (and not combine corresponding information such as sensor data). a For example, the system may determine that the field of view of the vision system for such a part of the stack is blocked. Thus, the system determines to utilize the information of item 718a in the geometric model in relation to modeling the position of item 718c in the estimated state. For example, the system determines to use only the geometric model (and not combine corresponding information such as sensor data).

[0176] The estimated state 775 includes items 708c, 710c, 712c, and 714c. In state 700, the corresponding items 708a, 710a, 712a, and 714a are shown to be accurately placed under item 704a, and state 750 has a gap 752 at the position of such items. The system determines the difference between states 700 and 750 (e.g., the presence of the item models 708a, 710a, 712a, and 714a and the gap 75 2determine the existence), and the system determines a way to model the corresponding part of the stack in the estimated state. In the example shown in the figure, the system determines that the sensor data is inaccurate (e.g., because the data includes a gap 752 at the location where items 708a, 710a, 712a, and 714a are placed). For example, the system may determine that the field of view of the vision system for such a part of the stack is blocked. The system may determine that the sensor data is inconsistent because the sensor data (e.g., state 750 in FIG. 7B) includes 704b placed on top of the set of items without the instability resulting from the gap 752. Thus, the system determines to utilize the information of items 708a, 710a, 712a, and 714a in the geometric model in relation to modeling the positions of items 708c, 710c, 712c, and 714c in the estimated state. For example, the system determines to utilize only the geometric model (and not combine corresponding information such as sensor data).

[0177] Similar to the above example, the system determines the estimated state to include item 716c placed with a gap 754 between adjacent items, item 702c placed with a shift 756, and item 706c placed with a shift. The system determines 、706c which of the geometric model, the sensor data, or both to utilize for modeling the placement of items 702c interpolate the positions of items 706a and 706b to determine the position of 706c, and 716c. In some embodiments, the system interpolates the positions of items 702a and 702b to determine the position of 702c,

[0178] FIG. 8 is a flowchart showing one embodiment for determining an estimate of the state of a stack of pallets and / or items. In some embodiments, process 800 is performed, at least in part, by the robotic system 100 of FIG. 1 and / or the system 200 of FIG. 2.

[0179] In some embodiments, process 800 is performed by one or more of an application 802, a server 804, a state estimator 806, a vision system 808, and a placement determiner 810, running on a control system for a robotic arm.

[0180] In step 820, application 802 sends a request to server 804. The request may correspond to a placement request asking for a plan and / or strategy for placing an item.

[0181] In response to receiving the placement request, in step 822, server 804 invokes a state determination. For example, server 804 sends a request or command to state estimator 806 to determine (and provide) an estimated state. In some embodiments, state estimator 806 is a module operating on server 804. In some embodiments, state estimator 806 is a service queried by multiple different servers / robotic systems. For example, state estimator 806 may be a cloud service.

[0182] In response to invoking the state determination, state estimator 806 obtains a vision state. In some embodiments, state estimator 806 sends a request for the vision state to vision system 808.

[0183] In response to receiving the request for the vision state in step 824, in step 826, vision system 808 provides the vision state to state estimator 806. For example, in response to receiving the request for the vision state, vision system 808 captures a snapshot of the work space using one or more sensors in the work space.

[0184] In response to receiving the vision state, the state estimator 806 determines a pallet state (e.g., the estimated state of the pallet and / or stack of items). The state estimator 806 may determine the estimated state based on one or more of the geometric model and the vision state. In some embodiments, the state estimator 806 combines the geometric model and the vision state (at least with respect to a portion of the stack).

[0185] In step 828, the state estimator 806 provides the pallet state to the server 804.

[0186] In step 830, the server 804 sends a placement request including the pallet state to the placement determiner 810. In some embodiments, the placement determiner 810 is a module operating on the server 804. In some embodiments, the placement determiner 810 is a service queried by a plurality of different server / robot systems. For example, the placement determiner 810 may be a cloud service.

[0187] In step 832, the placement determiner 810 provides a set of one or more potential placements to the server 804. The set of one or more potential placements may be determined based at least in part on the item to be placed (e.g., attributes associated with the item) and the pallet state (e.g., available locations, and attributes of the items within the stack of items).

[0188] In some embodiments, the set of one or more potential placements is a part of all possible placements. For example, the placement determiner 810 uses a cost function to determine a set of one or more potential placements for providing to the server 804. The placement determiner 810 may determine potential placements that meet a cost criterion with respect to the cost function (e.g., having a cost less than a cost threshold).

[0189] In response to receiving one or more sets of potential placements, at step 834, server 804 selects a placement and transmits the selected placement to app 802. For example, the selected placement is provided as a response to the initial placement request in step 820.

[0190] At step 836, app 802 controls the robotic arm to place the item. In some embodiments, app 802 determines a plan for moving the item to the selected placement (e.g., based on the attributes of the item and the position corresponding to the selected placement (such as coordinates within the workspace)).

[0191] At step 838, app 802 provides an instruction to server 804 to perform an update regarding the geometric state. For example, app 802 provides confirmation that the placement of the item has been performed at step 836, and server 804 interprets such confirmation as an instruction to trigger an update to the geometric state (e.g., a geometric model).

[0192] At step 840, server 804 transmits a request to update the geometric state to state estimator 806. For example, server 804 requests that state estimator 806 update the geometric model to reflect the placement of the item according to the corresponding plan.

[0193] In response to receiving the request to update the geometric state, state estimator 806 performs the corresponding update. At step 842, state estimator 806 provides an indication to server 804 that the geometric state has been successfully updated.

[0194] At step 844, server 804 provides an indication to app 802 that the geometric state has been successfully updated to reflect the placement of the item.

[0195] Process 800 may be repeated for a set of items to be stacked.

[0196] Figure 9 is a flowchart showing the process of palletizing / depalletizing a set of items according to various embodiments. In some embodiments, process 900 is performed, at least in part, by the robot system 100 of FIG. 1 and / or the system 200 of FIG. 2.

[0197] In step 910, sensor data indicating the current state of the work space is acquired. In some embodiments, the system receives sensor data from a vision system provided within the work space.

[0198] In step 920, the robotic arm is controlled to move a first set of items. In some embodiments, the system determines a plan for moving the first set of items and controls the robotic arm to move the first set of items according to the plan (e.g., pick and place).

[0199] In step 930, the geometric model is updated based on the movement of the first set of items. In some embodiments, in response to moving the first set of items, the system updates the geometric model to reflect the movement (e.g., the placement of the first set of items). In the case of item palletization, the geometric model is updated to reflect the placement of the first set of items on the stack.

[0200] In step 940, an estimated state (e.g., the state of the item stack) is determined based at least in part on the geometric model and the sensor data. The sensor data may correspond to the sensor data acquired in step 910 or updated sensor data captured after / during the movement of the first set of items. In some embodiments, the system determines the estimated state based on a combination of the geometric model and the sensor data.

[0201] In operation 950, a plan for controlling the robotic arm to move the second set of items is determined based at least in part on the estimated state. In some embodiments, in response to determining a plan for moving the second set of items, the system controls the robotic arm to move the second set of items according to the corresponding plan.

[0202] According to various embodiments, the system determines the estimated state after moving each set of items (e.g., a set of items determined based on the number of items or based on the attributes of the current or previously placed items, etc.).

[0203] In operation 960, a determination is made as to whether process 900 is complete. In some embodiments, process 900 is determined to be complete in response to a determination that no further updates to the estimated state are performed, no further items are moved, the user has terminated the system, the administrator has indicated a pause or stop of process 900, etc. In response to a determination that process 900 is complete, process 900 ends. In response to a determination that process 900 is not complete, process 900 returns to operation 910.

[0204] The examples described above are in the context of palletizing or depalletizing a set of items, but various embodiments may be implemented in connection with singulating a set of items and / or kitting a set of items. For example, various embodiments are implemented to determine / estimate the state of a work space (e.g., a chute, conveyor, container, etc.) based at least in part on geometric data and sensor data (e.g., a combination of geometric data and sensor data such as interpolation between geometric data and sensor data).

[0205] Various examples of the embodiments described in this specification are described in relation to flowcharts. Those examples may include several steps that are executed in a specific order, but according to various embodiments, various steps may be executed in various orders and / or various steps may be integrated into a single step or executed in parallel.

[0206] The above embodiments have been described in some detail for ease of understanding, but the present invention is not limited to the details provided. There are many alternative ways to implement the present invention. The disclosed embodiments are illustrative and not intended to be limiting.

Claims

1. A robot system, comprising: a communication interface configured to receive sensor data indicating a current state of the work space from one or more sensors deployed in the work space, the work space including a pallet or other container and a plurality of items stacked on or in the container; one or more processors connected to the communication interface; wherein the one or more processors are configured to: control a robotic arm to place a first set of items on or in the pallet or other container, or to remove the first set of items from the pallet or other container; update a geometric model based on the first set of items placed on or in the pallet or other container, or removed from the pallet or other container; estimate the state of the pallet or other container and one or more items stacked on or in the container by using the geometric model and the sensor data in combination; generate or update a plan for controlling the robotic arm to place a second set of items on or in the pallet or other container, or to remove the second set of items from the pallet or other container, using the estimated state.

2. The robot system according to claim 1, wherein the geometric model reflects one or more respective attributes of the first set of items and the second set of items.

3. The robot system according to claim 2, wherein the geometric model reflects the respective weights of the items included in the one or more of the first set of items and the second set of items.

4. The robot system according to claim 1, wherein the geometric model reflects the respective rigidities of the items included in the one or more of the first set of items and the second set of items.

5. The robot system according to claim 1, wherein the geometric model reflects the respective package strengths of items included in one or more of the first set of items and the second set of items.

6. The robot system according to claim 1, wherein the geometric model reflects the distribution of weights across a plurality of items included in one or more of the first set of items and the second set of items.

7. The robot system according to claim 1, wherein the geometric model includes the predicted stability of the one or more items stacked on or in the container.

8. The robot system according to claim 7, wherein the predicted stability is at least partially based on a set of positions where the robot system considers each item to be placed and one or more attributes of each item stacked on or in the container.

9. The robot system according to claim 1, wherein the geometric model includes the predicted stability of the one or more items stacked on or in the container, as well as the one or more simulated items for which a simulation of the stacking of the one or more simulated items has been performed.

10. The robot system according to claim 1, wherein the one or more sensors include an image sensor.

11. The robot system according to claim 10, wherein the image sensor includes a 3D camera.

12. The robot system according to claim 1, wherein a point cloud is determined at least partially based on the sensor data.

13. The robot system according to claim 1, wherein the estimated state of the pallet or other container and the one or more items stacked on or in the container is determined at least partially based on an interpolation between the sensor data and the geometric model.

14. The robot system according to claim 13, wherein the position of each item is determined based on interpolation between (i) the position of each item according to the sensor data and (ii) the position of each item according to the geometric model.

15. The robot system according to claim 1, wherein the estimated state of the pallet or other container and the one or more items stacked on or in the container is determined based at least in part on a midpoint between the sensor data and the geometric model.

16. The robot system according to claim 1, wherein the one or more processors are further configured to update the current state of the work space according to the sensor data using the geometric model in response to a determination that the current state of the work space according to the sensor data includes an anomaly.

17. The robot system according to claim 16, wherein the anomaly is caused by one or more of (i) a sensor among the one or more sensors having its field of view blocked or obscured with respect to a particular point in the work space, and (ii) noise or gaps in the sensor data that create a gap in the current state of the work space.

18. The robot system according to claim 1, wherein the one or more processors are further configured to determine that a difference between a state predicted based on the geometric model and a state observed via the one or more sensors exceeds a threshold.

19. The robot system according to claim 18, wherein in response to a determination that the difference exceeds the threshold, a warning is communicated to the user to confirm that the stack of the one or more items on or in the container is okay.

20. The robot system according to claim 1, wherein the one or more processors are a first subset of the one or more processors, Generate or update the plan for controlling the robotic arm to place the second set of items on or in the pallet or other container, or to remove the second set of items from the pallet or other container. A first subset of the one or more processors, wherein the plan for controlling the robotic arm is configured to be generated or updated at least partially based on the estimated state. A second subset of the one or more processors, Update the geometric model and determine the estimation of the state of the pallet or other container and the one or more items stacked on or in the container at least partially based on a combination of the geometric model and the sensor data. A robotic system comprising the above.

21. The robotic system according to claim 20, wherein the second subset of the one or more processors is remote from the work space.

22. The robotic system according to claim 1, wherein the first set of items includes N items, and N is a positive integer determined dynamically.

23. The robotic system according to claim 22, wherein the state of the one or more items stacked on or in the pallet or other container is estimated after the N items are stacked.

24. The robotic system according to claim 23, wherein the state is estimated before the N items are stacked in response to a determination that an irregularly shaped item has been placed.

25. The robotic system according to claim 1, wherein the second set of items corresponds to the next M items on a source to be placed on the pallet or other container, and M is a positive integer.

26. The robotic system according to claim 25, wherein the source is a conveyor.

27. A method for controlling a robot, Receive sensor data indicating the current state of the work space from one or more sensors deployed in the work space, where the work space includes a pallet or other container and a plurality of items stacked on or in the container, Control a robotic arm by one or more processors to place a first set of items on or in the pallet or other container or to remove the first set of items from the pallet or other container, Update a geometric model based on the first set of items placed on or in the pallet or other container or removed from the pallet or other container, Use the geometric model and the sensor data to estimate the state of the pallet or other container and one or more items stacked on or in the container, Generate or update a plan for controlling the robotic arm to place a second set of items on or in the pallet or other container or to remove the second set of items from the pallet or other container using the estimated state, A method comprising the steps of.

28. A computer program product for controlling a robot, embodied in a non-transitory computer-readable medium, Computer instructions for receiving sensor data indicating the current state of the work space from one or more sensors deployed in the work space, where the work space includes a pallet or other container and a plurality of items stacked on or in the container, Computer instructions for controlling a robotic arm by one or more processors to place a first set of items on or in the pallet or other container or to remove the first set of items from the pallet or other container, Computer instructions for updating a geometric model based on the first set of items placed on or in the pallet or other container or removed from the pallet or other container, Computer instructions for estimating the state of the pallet or other container and one or more items stacked on or in the container by using the geometric model and the sensor data in combination; Computer instructions for generating or updating a plan for controlling the robotic arm to place a second set of items on or in the pallet or other container or to remove the second set of items from the pallet or other container using the estimated state; A computer program product comprising the same.

Images