Packing and planning for tight, sturdy loads

Abstract

The present application discloses a method, system, and computer system for determining an arrangement of a set of objects for loading into a shipping container, the method including (i) receiving a suggestion of a set of objects to be loaded into a shipping container for transportation from a source location to a destination location, (ii) determining an arrangement of the set of objects to be loaded into the shipping container according to a stable model based at least in part on object information corresponding to the set of objects, and (iii) providing the arrangement of the set of objects to a robotic system for implementation in connection with loading the set of objects into the shipping container.

Description

CROSS-REFERENCE TO OTHER APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 305,195, entitled "PACKING AND PLANNING FOR TIGHT INCOLLAPSIBLE LOADS," filed on January 31, 2022, which is incorporated by reference herein for all purposes. [Background technology]

[0002] Shipping and distribution centers, warehouses, loading docks, air cargo terminals, large retail stores, and other activities that ship and receive heterogeneous sets of objects utilize strategies such as packing and unpacking heterogeneous objects into boxes, crates, containers, conveyor belts, pallets, etc. Packing the heterogeneous objects into boxes, crates, pallets, etc. allows the resulting set of objects to be handled by lifting equipment (forklifts, cranes, etc.) and allows the objects to be packed more efficiently for storage (e.g., in a warehouse) and / or shipment (e.g., in a truck, cargo hold, etc.).

[0003] In some contexts, objects may differ so much in size, weight, density, bulk, stiffness, packaging strength, etc. that any given object or set of objects may or may not have attributes that allow them to support the size, weight, weight distribution, etc. of given other objects that they may need to package (e.g., in a box, container, pallet, etc.). When loading a heterogeneous set of objects into a shipping container, the objects must be carefully selected and loaded to ensure that the objects do not collapse, tip, or otherwise become unstable within the shipping container (and to avoid damage to the objects).

[0004] The use of robotics is made more difficult in many environments due to the variety of objects, the variability in the order, number, and combination of objects that are packaged, for example, on a given pallet, and the variety in the types and locations of containers and / or feeding mechanisms from which objects must be picked up in order to be placed on a pallet or other container. [Brief description of the drawings]

[0005] Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

[0006] [Figure 1] FIG. 1 illustrates a robotic system for loading objects into a shipping container in accordance with various embodiments.

[0007] [Figure 2A] 1 illustrates an arrangement of a set of objects within a shipping container in accordance with various embodiments.

[0008] [Figure 2B] 1 illustrates an arrangement of a set of objects within a shipping container in accordance with various embodiments.

[0009] [Figure 2C] 1 illustrates a "T" boundary between a subset of objects within a shipping container in accordance with various embodiments.

[0010] [Figure 3A] 1 illustrates an arrangement of a set of objects within a shipping container in accordance with various embodiments.

[0011] [Figure 3B] 1 illustrates an arrangement of a set of objects within a shipping container in accordance with various embodiments.

[0012] [Figure 3C] 1 illustrates an arrangement of a set of objects within a shipping container in accordance with various embodiments.

[0013] [Figure 3D] 1 illustrates an arrangement of a set of objects within a shipping container in accordance with various embodiments.

[0014] [Figure 4A] 1A-1C illustrate trees corresponding to different states or configurations of an object in accordance with various embodiments.

[0015] [Figure 4B] 1A-1C illustrate trees corresponding to different states or configurations of an object in accordance with various embodiments.

[0016] [Figure 5A] 1 illustrates a subset of objects loaded into a shipping container in accordance with various embodiments.

[0017] [Figure 5B] 1 illustrates a subset of objects loaded into a shipping container in accordance with various embodiments.

[0018] [Figure 5C] 1 illustrates a set of objects loaded into a shipping container according to various embodiments.

[0019] [Figure 6] 1 is a flow chart illustrating a process for arranging a set of objects to be loaded into a set of shipping containers in accordance with various embodiments.

[0020] [Figure 7] 1 is a flow chart illustrating a process for arranging a set of objects to be loaded into a set of shipping containers in accordance with various embodiments.

[0021] [Figure 8] 1 is a flow chart illustrating a process for arranging a set of objects to be loaded into a set of shipping containers in accordance with various embodiments.

[0022] [Figure 9] 1 is a flow chart illustrating a process for evaluating the alignment of a set of objects to be loaded into a shipping container in accordance with various embodiments.

[0023] [Figure 10] 1 is a flow chart illustrating a process for evaluating the alignment of a set of objects to be loaded into a shipping container in accordance with various embodiments.

[0024] [Figure 11] 1 is a flow chart illustrating a process for evaluating the alignment of a set of objects to be loaded into a shipping container in accordance with various embodiments.

[0025] [Figure 12] 1 is a flow chart illustrating a process for planning loading of a set of objects into a shipping container according to various embodiments.

[0026] [Figure 13] 1 is a flow chart illustrating a process for planning loading of a set of objects into a shipping container according to various embodiments.

[0027] [Figure 14] 1 is a flowchart illustrating a process for determining a plan for unloading a set of objects from a shipping container, according to various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments or any other form the present invention may take may be referred to herein as techniques. In general, the order of steps of a disclosed process may be altered within the scope of the present invention. Unless otherwise noted, components such as a processor or memory described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0029] A detailed description of one or more embodiments of the present invention will now be given with reference to the drawings illustrating the principles of the present invention. The present invention will be described in connection with such embodiments, but is not limited to any of them. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0030] As used herein, a vision system comprises one or more sensors that acquire sensor data (e.g., sensor data related to a workspace). The sensors may include one or more of a camera, a high-resolution camera, a 2D camera, a 3D (e.g., RGBD) camera, an infrared (IR) sensor, other sensors for generating a three-dimensional view of the workspace (or a portion of the workspace, such as a pallet and a stack of objects on the pallet), any combination of the above, and / or a sensor array including multiple sensors as described above. Examples of vision systems are further described in U.S. Patent Application Serial No. 16 / 667,661, which is incorporated herein by reference in its entirety for all purposes.

[0031] As used herein, a shipping container includes a shipping container, an intermodal container, a cargo hold within a vehicle (truck, truck bed, etc.), etc. In some embodiments, various other types of physically constrained spaces may be considered a shipping container for purposes herein.

[0032] Loading a set of objects into one or more shipping containers is a computationally expensive operation due to the extremely large number of permutations or combinations to assign objects to specific shipping containers and arrange a set of objects within a specific shipping container. A particular loading (e.g., arrangement) of objects into a specific shipping container may be more desirable than other loadings because the stability of various loadings may differ (e.g., may cause objects to fall, damage objects, cause unsafe transportation due to changes in weight distribution during transportation, etc.), because the density of the loading affects the number of shipping containers required to transport the set of objects, or by various other criteria for the desired / best loading. Pallet and truck loads (or loaded vehicles / containers) may be unstable unless carefully and purposefully packed. Object movement may make transportation difficult, damage objects, and / or complicate unloading. To operate autonomously, a robotic system used to load a set of objects must be configured to pack the set objects tightly and securely.

[0033] Various embodiments use one or more models to optimize the loading of the set of objects or otherwise select the best loading of the set of objects. In some embodiments, the system determines / selects the best loading of the set of objects within one or more predefined constraints. Examples of one or more predefined constraints include: (i) the time to calculate / determine a particular loading of the set of objects into one or more shipping containers; (ii) resource constraints of the robotic system (e.g., the number of robots available to load the set of objects, robotic motion constraints, etc.); (iii) constraints imposed by the dimensions of the shipping container into which the set of objects is to be loaded; (iv) weight distribution constraints (e.g., to eliminate loadings that are deemed unsafe or unstable); (v) scoring constraints based on predefined scoring functions (e.g., a function that measures the goodness of one or more characteristics of the loading (density, stability, etc.)); (vi) buffer zone constraints (e.g., constraining the extent to which objects may be buffered in relation to reordering the objects); and the like. Various other constraints may also be implemented.

[0034] Determining how to load a set of objects into one or more shipping containers generally takes significant time, such as in conjunction with simulating various possible loads. Various embodiments implement time constraints on the load determination / selection. In some embodiments, the system determines the load (e.g., an arrangement, etc.) based on a determination that the load's score according to a predefined scoring function exceeds a predefined score threshold. For example, the system selects a first evaluated load having a satisfactory score, such as to quickly determine a "good enough" load. In some embodiments, the system repeatedly evaluates potential loads, and after a predefined time threshold has expired, the system determines the load associated with the highest score among the set of evaluated loads and uses such load in conjunction with providing a loading plan as an output for loading the set of objects into one or more shipping containers. Various embodiments implement different time constraints in conjunction with determining the optimal / best arrangement. For example, a cloud service may run overnight (or for extended periods of time) to comprehensively evaluate various arrangements. As another example, a local computer system may perform rapid evaluation of sequences in connection with determining an optimal / best / ideal sequence in real-time or synchronously with the loading of a set of objects into a shipping container. For example, as the robotic system is loading a set of objects, the robotic system may query the system for an updated sequence, such as in response to determining that the current conditions of the workspace are different from those predicted during simulation and evaluation by a cloud service or remote server that evaluated various sequences over a longer period of time. The system may implement a lightweight model for sequence evaluation and sequence selection performed in real-time with the loading. The lightweight model may use a set of predefined heuristics to determine sequences predicted to be acceptable / feasible (e.g., predicted to satisfy one or more predefined criteria or constraints).

[0035] Various embodiments include a system, method, and / or apparatus for determining an arrangement for loading a set of objects into a transport container, the method comprising: (i) receiving an indication of a set of objects to be loaded into a transport container for transportation from a source location to a destination location, (ii) determining an arrangement of the set of objects to be loaded into the transport container according to a stable model based at least in part on object information corresponding to the set of objects, and (iii) providing the arrangement of the set of objects to a robotic system for performance in connection with loading the set of objects into the transport container.

[0036] Examples of challenges related to partitioning that are overcome by various embodiments include the following: Computational complexity: The number of distributions of objects in the input set is combinatorially large, which makes exact optimization computationally difficult. · Safety: Objects should be safely loaded into each shipping container and transported safely. Safety constraints may include personnel safety (e.g., loading assistants and truck drivers) and object safety (e.g., avoidance of damage during loading, transportation, and unloading). Time: The sequencing decision is performed within certain time constraints. As an example, the loading of a set of objects according to a sequence can be performed in a shipping facility within a certain time, subject to the constraints of storage, layout, and routing hardware. As another example, the loading of a set of objects according to a sequence can be performed within a certain time. Cost: The operational costs of carrying out the loading of a set of objects according to a sequence and the transportation of shipping containers should be minimized or below a predefined cost threshold. Personnel and Hardware: The personnel and hardware utilized to load objects into shipping containers may vary, such as between truck bays or shipping facilities.

[0037] In various embodiments, the robotic loading systems disclosed herein utilize knowledge of the arriving objects (e.g., boxes, etc.) as well as the scene or workspace (e.g., loading area, shipping vehicle, shipping container, interior of shipping container, packaging or arrangement as a result of previous object placement) to stack or otherwise arrange the objects in a stable configuration.

[0038] In various embodiments, the robotic loading system disclosed herein includes a planner / controller (e.g., a controller implementing a placement module to determine placement or arrangement) that does one or more of the following: Generally, you attempt to create a "T" boundary between items by placing one object on top of at least two other objects. · In general, avoid placing objects of the same (or similar) size directly on top of each other (which can lead to instability of stacks or columns in arrays such as in shipping containers). Using the walls of the shipping container (or the truck the object is being loaded into) to pack the object tightly from wall to wall to prevent the object from shifting during transport. · Rearrange objects as deemed necessary to improve load stability.

[0039] According to various embodiments, arranging a set of objects to include a "T" boundary between the objects is more stable than stacking the objects. In some embodiments, a robotic system (e.g., a robotic loading or unloading system) is configured to plan or re-plan the loading of a set of objects. For example, the robotic system plans / reschedules the arrangement of a set of objects required to prioritize a stable "T" boundary over a vertically aligned stack. The robotic system may be required to re-plan the arrangement if the order in which the objects are expected to arrive in the workspace, or the set of objects, differs from the predicted set of objects based on which the initial plan was determined.

[0040] In some embodiments, the system performs a simulation of loading or arranging a set of objects. For example, the simulation is performed in association with predicting an arrangement with low stability. The system utilizes computer vision and / or force-based sensing in association with detecting an arrangement with low stability. In response to detecting an arrangement with low stability (e.g., stability below a predefined stability threshold), the system initiates / performs corrections to address the instability issue. For example, the system moves one or more objects predicted to cause instability to an arrangement that meets a stability criterion (e.g., stability above a predefined stability threshold).

[0041] In some embodiments, the system loads objects into the shipping container such that the objects are pushed from wall to wall (e.g., side wall to side wall) in the shipping container. Tight packaging where objects are pushed from wall to wall increases stability. For example, by packing a set of objects such that the objects engage the walls of the shipping container for a particular layer of objects, the objects in the layer have less degrees of freedom that forces acting on the objects may cause them to fall or move. A robotic system (e.g., a robotic loading or unloading system) is configured to load the shipping container from the deepest (e.g., furthest) location first, and then back outward from the shipping container to place the objects in the next deeper location. For each subsequent layer of objects, the objects may be loaded along the wall first, and then the robotic system operates to place the objects inside the layer (e.g., between the objects along the wall). The robotic system loads the objects between the objects along the wall to ensure a tight fit pack against the wall. In some embodiments, a robotic arm is used to apply a force to one or more objects to push them against a wall as other objects are loaded inside (e.g., using another robotic arm or another robotic system that may be controlled in coordination with the robotic arm that is pushing the object against the wall). Vacuum or other side grippers may be used to push the objects against the wall to fit them snugly within the layer and / or to slide a last box into any remaining gaps in the layer of objects (e.g., near the centerline of the shipping container).

[0042] In some embodiments, the system unloads objects in a substantially reverse order from the order in which they were loaded into the shipping container. For example, the system begins removing objects from a layer of objects in the shipping container. Unloading objects from a particular layer includes first removing objects closer to the center (e.g., the centerline of the shipping container), followed by objects closer to the wall until the objects facing the wall are removed. The system then repeatedly performs the unloading sequence deeper within the shipping container until the set of objects being unloaded is complete. In some embodiments, the system implements computer vision to obtain a model of the loading of objects into the shipping container. The system creates a segmented (e.g., object-by-object) view of the scene / workspace (e.g., the shipping container) and determines a plan for unloading the set of objects. The system determines a plan for unloading the set of objects taking into account the need to avoid removing objects (e.g., boxes) in an order predicted to lead to a potentially unstable configuration for the remaining objects. The robotic system may use the robotic arms to prevent remaining objects from tipping over as other objects are retrieved (e.g., a central object is retrieved while an object to the side provides support to prevent tipping).

[0043] FIG 1 illustrates a robotic system for loading objects into a shipping container, in accordance with various embodiments. In some embodiments, system 100 is implemented, at least in part, by process 600 of FIG 6, process 700 of FIG 7, process 800 of FIG 8, process 900 of FIG 9, process 1000 of FIG 10, process 1100 of FIG 11, process 1100 of FIG 12, process 1300 of FIG 13, and process 1400 of FIG 14.

[0044] In the illustrated example, the system 100 includes a robotic arm 102. In this example, the robotic arm 102 is fixed, but in various alternative embodiments, the robotic arm 102 may be fully or partially movable, e.g., mounted on rails, fully movable on a motor-driven chassis, etc. In other examples, the system 100 may include multiple robotic arms having a workspace. As shown, the robotic arm 102 is used to pick various objects (e.g., any and / or heterogeneous objects) from one or more conveyors (or other sources) 110 and 105 in connection with loading objects into a shipping container 150, or to pick objects from the shipping container 150 and place the objects on one or more conveyors in connection with unloading the shipping container 150. In some embodiments, other robots not shown in FIG. 1 may be used to load pallets of objects to be loaded / unloaded from the transport container 150, to push pallets or trays of objects into the workspace of the robot arm 102 for loading / unloading from the transport container 150, etc.

[0045] As shown in FIG. 1, the system 100 may include one or more predefined zones. For example, pallet 112, pallet 11 are shown to be located within buffer zone 115. In some embodiments, one of the predefined zones is used as a buffer or staging area where objects are temporarily stored (e.g., until objects are loaded onto a shipping container 150 in connection with loading objects onto the shipping container 150 or until objects are loaded onto conveyors 105 and / or 110 in connection with unloading objects from the shipping container 150). As shown in FIG. 1, the buffer zone 115 may include one or more containers or pallets to hold objects for buffering / staging during loading / unloading. The predefined zone may be indicated by marking or labeling on the ground or structurally in another manner, such as with a frame as shown in the system 100. Each of the predefined zones (e.g., buffer zones) may be located within range of the robotic arm 102 (e.g., such that the robotic arm 102 can place objects within and / or pick objects from the buffer zone, etc.). For example, the predefined zones are located radially around the robotic arm 102.

[0046] One or more objects are provided (e.g., transported) to (or from) the workspace of the robotic arm 102, such as via conveyor 110 and / or conveyor 105. Various other mechanisms may be implemented to transport the objects to the workspace, such as utilizing various other robotic systems. The system 100 controls the operation of the conveyors 105, 110, such as by controlling the speed, direction of movement, and the like. By way of example, the system 100 is configured to control the speed of the conveyor 105 independently of the speed of the conveyor 110, or the system 100 controls the speed of both the conveyor 105 and / or conveyor 110. Controlling the conveyors 105, 110 may include pausing the conveyors (e.g., to allow sufficient time for the robotic arm 102 to pick and place the object). In some embodiments, conveyor 105 and / or conveyor 110 carry objects for one or more manifests (e.g., orders) or one or more sets of objects to be loaded into different shipping containers, respectively. For example, conveyor 105 and conveyor 110 may carry objects for the same manifest and / or different manifests.

[0047] The system 100 controls the robotic arm 102 to perform pick-and-place operations on objects within a workspace, such as loading objects into and unloading objects from a transport container 150, and moving objects to a buffer zone 115. For example, the system 100 controls the robotic arm 102 to pick an object from a conveyor (such as conveyor 110 or conveyor 105) and place the item o in a transport container 150. The system 100 controls (e.g., via control computer 122) the robotic arm 102 to pick an object and move the object to a corresponding destination location (e.g., a location within a transport container 150) based at least in part on a plan associated with the object or a plan associated with loading / unloading a set of objects into / from a particular transport container.

[0048] In some embodiments, determining a plan associated with loading / unloading a set of objects into / from one or more shipping containers includes one or more of: (i) determining an arrangement of objects to follow as the objects are loaded into the shipping container, (ii) simulating the loading of one or more objects into the shipping container, (iii) simulating the transportation of the shipping container (e.g., simulating forces that the shipping container is expected to be subjected to during transportation, such as in connection with assessing the stability or safety of loading the objects into the shipping container), and / or (iv) evaluating various possible arrangements (e.g., scoring based on a predefined scoring function or selecting an arrangement to implement), etc. System 100 may determine the arrangement based at least in part on one or more heuristics or based on querying a model, such as a machine learning model.

[0049] In some embodiments, the system 100 determines a plan for loading / unloading a set of objects into / from a particular shipping container offline or asynchronously from the loading / unloading of the shipping container. For example, the system 100 determines a plan for loading / unloading a set of objects into / from one or more shipping containers by querying one or more servers (e.g., querying a cloud service) based at least in part on the set of objects. The one or more servers may determine the plan based on running multiple simulations of loading various objects into the shipping container and evaluating the loading. The one or more servers may run the simulations and evaluation of various loadings and determine the plan for loading the set of objects offline, for example, overnight or otherwise before the set of objects is provided to the workspace of the robot (e.g., the robotic arm 102).

[0050] The system 100 may update a plan for loading / unloading the set of objects to / from one or more transport containers in real time while the set of objects is being transported or placed in the robot's workspace. For example, the system 100 updates a plan for loading / unloading the set of objects (e.g., a loading / unloading plan previously determined offline) using a lightweight model or one or more predefined heuristics. The system 100 updates the plan for loading / unloading the set of objects using the control computer 122. The update of the plan for loading / unloading the set of objects may take into account variations in one or more of (i) the objects transported to the workspace (e.g., a malfunction may cause more objects to be transported or the transport of a particular object to be withheld, etc.), (ii) the characteristics of the objects transported to the workspace (e.g., errors in detecting attributes of the objects), and / or (iii) the order in which the objects were transported to the workspace, etc.

[0051] In response to receiving the plan (e.g., from a server), the system 100 determines a strategy for performing a pick-and-place operation on an object to be loaded into the shipping container 150. For example, determining the strategy may include determining how to grasp the object, the trajectory of the object, the robot motion, how the object is released, etc.

[0052] In some embodiments, the system 100 determines updates to the plan related to the object while the object is on the conveyor 105, 110, after picking up the object (e.g., based on previously acquired attributes of the object (e.g., weight) or in response to information acquired by sensors in the workspace (e.g., indication of a predicted collision with another object or human, etc.)), and / or during the placement of the object (e.g., the system may use force sensors to detect instability, etc., before / after the release of the object).

[0053] The system 100 may obtain an identifier associated with the object (such as a barcode, QR code, or other identifier or information on the object). For example, the system 100 scans / obtains the identifier as the object is being transported on the conveyor. In response to obtaining the identifier, the system 100 utilizes the identifier in connection with determining the order in which the items are loaded into the shipping container 150, the arrangement of the objects (e.g., target location and / or orientation), a strategy for picking and placing the objects, whether to reorder the objects based on staging the objects in the buffer zone 115, etc. In response to determining the arrangement of the objects, the system 100 determines a strategy or determines whether to update a plan for loading the objects into the shipping container 150 based at least in part on a model or simulation of the loading of the objects into the shipping container 150 and / or the placing of the objects into the shipping container 150. In connection with determining the arrangement of the objects, the system 100 determines a particular location at which the objects are placed into the shipping container 150 or the buffer zone 115. Additionally, a plan may be determined for moving the object to the destination location, including a planned path or trajectory along which the object will be moved. In some embodiments, the plan is updated as the robotic arm 102 is moving the object, such as in connection with taking proactive measures to change or adapt to detected states or conditions associated with one or more objects in the workspace (e.g., to avoid anticipated collision events, to account for a measured weight of the object being greater than a predicted weight, to reduce shear forces on the object as it is moved, etc.).

[0054] According to various embodiments, the system 100 includes a vision system. The system 100 uses information captured by the vision system to determine a scene / workspace context, such as to generate a model of the workspace. The system 100 may include one or more sensors and / or sensor arrays, such as cameras 116, 118. For example, the system 100 includes one or more sensors proximate the conveyor 105 and / or the conveyor 110. The one or more sensors may obtain information related to objects on the conveyor, such as identifiers or information on the object's label or attributes of the object, such as the object's dimensions. In some embodiments, the system 100 includes one or more sensors and / or sensor arrays that obtain information about a predefined zone and / or pallets within the zone. For example, the system 100 includes sensors that obtain information related to the buffer zone 115 and / or the shipping container 150. The sensors may include one or more 2D cameras, 3D (e.g., RGBD) cameras, infrared, and other sensors for generating a three-dimensional view of the workspace (or a portion of the workspace, such as a pallet and a stack of objects on the pallet). Information about the buffer zone 115 and / or the shipping container 150 may be used in connection with determining a condition of the buffer zone and / or loading of objects into the shipping container 150. As an example, the system 100 generates a model of loading of objects into the shipping container 150 based at least in part on the information about the shipping container 150 and the information about at least a portion of the set of objects. The system 100 then utilizes the model in connection with determining / updating a plan for placing objects into the shipping container 150.

[0055] According to various embodiments, the system 100 determines a plan (or updates a plan) for picking and placing an object based at least in part on a determination of the stability of the object loaded into the transport container 150 or the predicted stability of the object to be loaded into the transport container 150. The system 100 determines a model of the loading of the object into the transport container 150, and the system 100 utilizes the model in connection with determining / updating the plan for placing the object. As an example, if the object to be moved next is relatively large (e.g., the surface area of ​​the object is large compared to the top surface of the object that the relatively large object will be placed under when placed into the transport container 150), the system 100 may determine that placing the object at the planned destination location may destabilize the stack of objects in the transport container 150 (e.g., due to a non-flat surface of the stack). In contrast, system 100 may determine that placing a relatively large (e.g., flat) object at the planned destination location may result in relatively heavy loading (e.g., alignment of the object in shipping container 150). System 100 may determine that the predicted stability of placing the object at the destination location is greater than a predefined stability threshold, or that alignment of the object at the destination location may result in an optimal alignment of the object (e.g., at least in terms of stability).

[0056] In some embodiments, the system 100 generates a simulated load or a model of a load of objects actually loaded into the shipping container 150 and determines whether the load is predicted to meet one or more predefined stability criteria. Examples of the one or more predefined stability criteria may include: (i) the number of "T" boundaries between objects in the load is less than a predefined "T" boundary count threshold; (ii) the number of "T" boundaries between objects in a load layer (e.g., a vertical layer of objects or a depth layer of objects) is less than a predefined layer "T" boundary count threshold; (iii) the number of objects stacked vertically in the load or load layer is less than a predefined vertical stacking threshold; (iv) the ratio of the number of "T" boundaries to the number of objects is less than a predefined "T" boundary ratio threshold; (v) the ratio of the number of "T" boundaries to the number of non-"T" boundaries is less than a predefined "T" boundary:non-"T" boundary ratio threshold; (vi) the predicted stability is less than a predefined stability threshold; (vii) the score value of a set of objects according to a predefined scoring function (or a predefined stability score threshold) is less than a predefined score threshold; (viii) the density of the objects is less than a predefined density threshold; (ix) the gaps between one or more objects are greater than a predefined gap threshold. A variety of other stability criteria may also be implemented.

[0057] The system 100 may communicate the status of the transport container 150 within a given zone and / or the status of the operation of the robotic arm 102. The status of the transport container 150 and / or the status of the operation of the robotic arm 102 may be communicated to a user or other human operator 120 (such as a human operator 120) (e.g., to provide the status of the loading / unloading of the transport container, to request human intervention to successfully complete a pick or place operation, etc.). For example, the system 100 may include a communication interface (not shown) through which information regarding the status of the system 100 (e.g., the status of the transport container 150, the given zone, the robotic arm, etc.) is communicated to a terminal (such as an on-demand remote operator 124 and / or a terminal utilized by a human operator, etc.).

[0058] According to various embodiments, the system 100 uses information obtained by one or more sensors in the workspace to determine an abnormal condition related to the transport container 150 (e.g., instability of objects loaded into the transport container 150 or identification of objects that have fallen from their respective destination locations). In response to detecting the abnormal condition, the system communicates an indication of the abnormal condition, such as to an on-demand remote control 124 or another terminal utilized by an operator. In addition to or instead of notifying the operator of the abnormal condition, the system 100 executes proactive measures in response to detecting the abnormal condition. The proactive measures may include controlling the robotic arm 102 to at least partially correct the abnormal condition (e.g., to re-place the dropped object, re-align the objects loaded into the transport container 150, re-arrange the objects, etc.). In some embodiments, the system executes proactive measures to partially correct the abnormal condition in response to determining that the degree of the abnormality is less than a threshold. Examples of determining that the degree of abnormality is less than a threshold include: (i) a determination that the object deviation is less than a threshold deviation value; (ii) a determination that the number of misaligned, misplaced, or dropped objects is less than a threshold number; (iii) a determination that the size of the misaligned, misplaced, or dropped objects meets a size threshold; and the like.

[0059] A human operator may communicate with the system 100 via a network (e.g., a wired network and / or a wireless network). For example, the system 100 may include a communication interface for connecting the system 100 to one or more networks. In some embodiments, a terminal connected to the system 100 via a network provides a user interface through which the human operator can provide instructions to the system 100 and / or obtain information regarding the status of the system 100 (e.g., the status of the robotic arm, the status of a particular shipping container 150, the status of a loading process for a particular manifest, etc.). The human operator may provide instructions to the system 100 by inputting to the user interface. For example, the human operator may use the user interface to pause the robotic arm, pause the loading / unloading process for a particular manifest, etc.

[0060] In various embodiments, elements of the system 100 may be added, removed, replaced, etc. In such an example, the control computer initializes and registers the new element, performs operational tests, and starts / resumes loading / unloading operations to incorporate the newly added element, for example. Examples of elements may include conveyors, robotic arms, buffer zones, etc.

[0061] According to various embodiments, the system 100 determines (e.g., calculates, maintains, stores, etc.) an estimated state of the transport container 150 and / or the buffer zone 115. For example, the system 100 determines a geometric model corresponding to the workspace and / or the loading of objects into the transport container 150.

[0062] According to various embodiments, the system 100 includes a vision system with one or more sensors (e.g., sensor 240, sensor 241, etc.). In various embodiments, the system 100 utilizes sensor data and geometric data (e.g., a geometric model) in connection with determining a location to place one or more objects in the shipping container 150 (or in connection with unloading one or more objects from the shipping container 150). The system 100 uses different data sources to model the state of the shipping container 150 (or the arrangement / loading of objects in the shipping container). For example, the system 100 estimates the location of one or more objects in the shipping container 150 and one or more characteristics (or attributes) associated with the one or more objects (e.g., object size). Examples of one or more characteristics associated with the one or more objects include object size (e.g., object dimensions), center of gravity, object stiffness, package type, deformability, fragility (e.g., as indicated by a fragile marker), identifier location, etc.

[0063] The system 100 determines the geometric model based at least in part on one or more attributes of one or more objects in the workspace. For example, the geometric model reflects attributes of each of a set of objects (e.g., one or more of a first set that have been palletized / stacked, as well as a second set of objects to be palletized / stacked, etc.). Examples of objects include object size (e.g., object dimensions), center of gravity, object rigidity, package type, identifier location, object deformability, object shape, object fragility, etc. Various other attributes of objects or objects in the workspace may be implemented.

[0064] In some embodiments, the system 100 generates a geometric model (e.g., a predicted geometric model) of the workspace (e.g., a model of the loading of objects in the shipping container 150) based at least in part on the simulated placement of the objects. For example, the system simulates the placement of a set of objects into the shipping container 150 (e.g., in-line or offline), generates a geometric model of the simulated placement, and evaluates the simulated loading (e.g., evaluates the stability of the loading). As another example, the system simulates the loading of a particular object or objects (e.g., in-line), generates a geometric model based on the actual conditions and the simulated placement of the workspace, and evaluates the simulated placement or predicted loading conditions in response to the simulated placement.

[0065] In some embodiments, the system 100 generates a geometric model based at least in part on the placed object(s) (e.g., objects that the system 100 has controlled the robotic arm 102 to place), one or more attributes respectively associated with at least a portion of the one or more objects, one or more objects in the workspace (e.g., predefined objects such as containers in the buffer zone 115, the robotic arm, a shelving system, a chute or other transport structure, or other infrastructure included in the workspace), etc. The geometric model may be determined at least in part based on executing a physics engine on the control computer to model the arrangement of the objects (e.g., model the state / stability of the arrangement of objects in the shipping container 150 or the buffer zone 115, etc.). The geometric model is determined based at least in part on predicted interactions of various components of the workspace, such as an object with another object, an object, simulated forces applied to a shipping container or an object within a shipping container (e.g., to model the use of a forklift or other equipment to lift / move a shipping container for purposes of modeling the transportation of a shipping container, such as a model of a vehicle accelerating / decelerating during transportation of the shipping container).

[0066] According to various embodiments, the system 100 uses the geometric model and the sensor data to determine a best estimate of the state of the workspace or the state of the shipping container 150. The system 100 can adjust for (e.g., cancel) noise in one or more of the geometric model and / or the sensor data. In some embodiments, the system 100 detects anomalies or differences between the state according to the geometric model and the state according to the sensor data (e.g., sensor data acquired by a vision system). In response to determining that there is an anomaly or difference between the geometric model and the sensor data, the system 100 can perform a best estimate of the state despite the anomaly or difference. For example, the system 100 determines whether to use the geometric model or the sensor data, or whether to use a combination of the geometric model and the sensor data (e.g., interpolation between them), etc. In some embodiments, the system 100 determines the estimated state on a segment-by-segment basis (e.g., for each voxel in the workspace, for each object, or for each object, etc.). For example, a first portion of the workspace may be estimated using only the geometric model, a second portion of the workspace may be estimated using only the sensor data (e.g., if there is an anomaly in the geometric model), and / or a third portion of the workspace may be estimated based on a combination of the geometric model and the sensor data. Using the example shown in FIG. 1, the system 100 may, in connection with determining the aggregate estimated state, utilize only the geometric model to determine individual estimated states for the objects loaded in the shipping container 150, utilize only the sensor data to determine individual estimated states for the workspace (e.g., the state of the shipping container 150 and / or the buffer zone 115), and utilize a combination of the respective geometric model and sensor data for the objects loaded in the shipping container.

[0067] The system 100 is configured to determine or update a plan for loading a set of objects into a transport container 150. Determining the plan for loading the set of objects includes one or more of: (i) determining whether to distribute the set of objects (e.g., for loading across multiple transport containers), (ii) distributing the set of objects in response to a decision to distribute the set of objects, (iii) determining an order in which the objects are transported to a workspace of the system for loading the objects into the transport container 150, (iv) determining an order in which the robotic arm 102 loads the objects into the transport container 150, and (v) determining an arrangement (e.g., an arrangement of the objects, such as position and orientation) of the set of objects.

[0068] In some embodiments, system 100 determines a solution for loading the set of objects using control computer 122. Control computer 122 is configured to provide a rapid solution in real-time or in synchronization with the control of robotic arm 102 for loading the set of objects into shipping container 150. For example, system 100 (e.g., control computer 122) stores a lightweight model (e.g., machine learning model) or a set of one or more heuristics that system 100 follows in determining a solution for loading the set of objects into the shipping container.

[0069] In some embodiments, the system 100 queries one or more servers (not shown), such as a cloud service, to determine a plan for loading a set of objects. The one or more servers are configured to provide a solution that is generally more optimal than a solution otherwise provided by a lightweight model used to determine the solution inline (e.g., because the one or more servers have more robust computational capabilities and are less time-constrained). For example, the one or more servers have greater computational resources and are less constrained with respect to time for determining a solution as the locally stored lightweight model. The one or more servers provide an offline service for determining a plan for loading a set of objects. For example, the one or more servers determine an asynchronous plan for loading objects into a shipping container 150. The one or more servers are configured to store / utilize a more robust model that provides a higher fidelity solution than a solution obtained by querying a locally stored lightweight model. The lightweight model is configured to sacrifice high fidelity for a faster computation of a solution for loading a set of objects.

[0070] In some embodiments, the system 100 retrieves a robust solution for loading a set of objects into a shipping container 150 from a cloud service, and the system 100 (e.g., on the control computer 122) queries the locally stored lightweight model based at least in part on a difference in the state of the workspace relative to a predicted state of the workspace used by the cloud service to model the workspace and determine the solution. Examples of differences in the state of the workspace include: (i) different hardware constraints, such as the absence of a particular robot, different robot system capabilities, or a different configuration of the workspace into which a particular shipping container is loaded; (ii) objects are delivered to the workspace in an order that is different from the predicted order used to determine the plan; and (iii) one or more attributes of one or more objects are different from the attributes used in modeling the workspace to determine the plan. Various other differences may occur, in response to which the system 100 uses the locally stored lightweight model to update the plan toward a more optimal plan for loading the set of items.

[0071] In some embodiments, the system 100 implements a model to select an optimal or best solution for arranging a set of objects. As an example, the optimal or best solution is the best solution within a predefined execution time. The optimal or best solution is selected based on the solution (e.g., arrangement) with the highest score value according to a predefined scoring function. For example, the solution with the highest score value is the solution with the highest stability. As another example, the solution with the score value that has the best balance of various factors (e.g., stability, time to complete loading or unloading, cost to complete loading, time to complete a combination of loading and unloading, etc.).

[0072] In some embodiments, various modules used in a model for evaluating the loading of a set of objects implement techniques for limiting a solution space of potential solutions (e.g., potential arrangements, etc.) (e.g., to provide a sufficiently good or best solution for loading the set of objects). For example, the modules implement techniques for narrowing the solution space to include feasible solutions and near-feasible potential solutions. The modules may implement one or more heuristics in connection with trimming the solution space or may be trained (e.g., using machine learning processes) to quickly identify solutions predicted to be considered feasible (e.g., quickly eliminate solutions predicted to be considered infeasible). For models implemented offline (e.g., to provide asynchronous evaluation of solutions for loading a set of objects), the modules are configured to trim the solution space to allow more likely solutions to be evaluated (e.g., edge case solutions are included in the potential solution space because the offline model is less constrained in terms of computational resources or execution time to provide feasible solutions). In contrast, for models that are implemented inline (e.g., to provide synchronous / real-time solutions), the modules are configured to trim the solution space to generate a relatively narrow space of potential solutions. For example, a model used for inline evaluation of solutions may trim the solution space to eliminate solutions that are predicted to violate one or more predefined heuristics (e.g., solutions that are predicted to be infeasible or that have been found to result in solutions having score values ​​below a score value threshold).As another example, a model used for in-line evaluation of solutions includes solutions in a solution space that satisfy one or more predefined heuristics (e.g., heuristics that have been found to result in solutions predicted to be feasible or having a score value above a score value threshold). In some embodiments, one or more heuristics used in connection with defining the potential solution space (e.g., trimming the solution space to eliminate solutions predicted to be infeasible or including solutions predicted to be feasible in the solution space) are determined based at least in part on a machine learning process.

[0073] In some embodiments, the models and / or modules used in conjunction with the models are configured to utilize combinatorial optimization and / or machine learning to efficiently find feasible solutions and iteratively refine / improve the solutions.

[0074] In some embodiments, the model is configured to provide a feasible solution within a predefined time period. For example, the model is configured to repeatedly evaluate potential solutions to identify a feasible solution within a predefined time period. As an example, for a model implemented in-line to provide a solution synchronously or in real time with the loading of an object, the predefined time period is 5 seconds. As another example, for a model implemented offline to provide an asynchronous solution, the predefined time period is in hours, such as 5 hours, 12 hours, etc. If the model completes the evaluation of the solution space before the end of the predefined time period, the model returns the best solution (e.g., the solution with the highest score value for evaluation using a predefined scoring function). If the model does not complete the evaluation of the solution space before the end of the predefined time period, and if the model has identified a set of one or more feasible solutions, the model returns the best solution among the one or more feasible solutions. If the model does not complete the evaluation of the solution space before the end of the predefined time period and the model has not yet identified a set of one or more feasible solutions, the model may implement one or more heuristics to quickly provide a solution that is deemed likely to be feasible.Example heuristics may include loading heavy objects first, loading large objects first, arranging heavy or large objects near the bottom of the shipping container (e.g., on the floor of the shipping container or in the first set of rows of objects), buffering fragile or deformable objects (e.g., whose fragility or deformability exceeds a corresponding predefined fragility or deformability threshold), determining to load an object such that it forms at least one "T" boundary with other objects (e.g., when the solution space for arranging the objects includes solutions where a "T" boundary is formed and solutions where an object rests vertically on another object) and otherwise loading such objects toward the end of loading the shipping container or toward the top row of objects in the shipping container, providing a bias to select configurations that result in the formation of a "T" boundary over configurations where a "T" boundary is not formed, etc. Various other heuristics may be implemented.

[0075] In some embodiments, the system (e.g., a distribution module) selects the sequence that has both (i) the highest score according to a predefined scoring function and (ii) meets one or more feasible loading criteria. Examples of feasible loading criteria include: (a) the longest time to load, unload, or collectively load and unload, (b) a minimum expected density of objects to be loaded into the shipping container, (c) a minimum number of total objects to be loaded into the shipping container, (d) a minimum stability of a subset of objects to be loaded into the shipping container, (e) a weight distribution that meets one or more criteria (e.g., weight is distributed to a location of the shipping container that corresponds to the wheels of a truck that is expected to carry the shipping container or is otherwise supported during transportation), etc. Various other feasible loading criteria may be implemented. The one or more feasible loading criteria may be a minimum requirement for loading a set of objects for the loading to be considered feasible.

[0076] In some embodiments, the model bounds the solution space based on resources available for loading / unloading a set of objects. Examples of boundaries include: (i) dimensional boundaries based on the dimensions of the shipping containers into which the set of objects is expected to be loaded (e.g., different shipping containers may have different dimensions or configurations (e.g., different types of doors or access capabilities for loading / unloading)), (ii) one or more characteristics (e.g., capabilities, limitations, etc.) of the loading / unloading bay to which the set of objects or shipping container is assigned, (iii) personnel assigned to loading / unloading the set of objects (e.g., characteristics of personnel assigned to a particular loading / unloading bay at a shipping facility), (iv) execution time to perform the loading / unloading of the set of objects, and (v) a time boundary based on the number of resources required to determine feasible solutions. (vi) the storage locations, layout, and routing hardware of the shipping facility, including an indication of whether the shipping facility has areas that may be implemented as buffer zones for staging items during loading / unloading to facilitate reordering of items, (vii) the available loading / unloading hardware (e.g., robotic arms, robotically operated forklifts or other means, etc.) and their characteristics (e.g., capabilities, limitations, etc., such as the extent to which a robotic arm may be configured to avoid / limit improper poses), and (viii) the set of objects to be loaded / unloaded (e.g., a set of objects to be transported from location A to location B). Various other types of boundaries may be implemented in connection with determining a solution for distributing a set of objects.

[0077] In connection with determining the arrangement of a set of objects (e.g., determining one or more partitions according to which the set of objects are loaded), various embodiments of the model implement one or more other modules to evaluate the feasibility of the arrangement. Examples of the one or more other modules include: (a) a modeling module that determines a model (e.g., a geometric model) of the workspace or shipping container; (b) a simulation module that simulates pick / place operations to simulate loading / unloading of one or more objects into / from the shipping container; and (c) a physics engine that simulates the physical behavior of the shipping container / load, such as simulating forces acting on or between the objects. The one or more modules called by the model are used to evaluate one or more of the following: density of objects in the shipping container, number of objects predicted to be loaded into the shipping container, stability of the load in the shipping container, weight distribution of the load in the shipping container, predicted execution time to perform loading / unloading of the shipping container. One or more modules may be implemented to evaluate the load along various other aspects.

[0078] In some embodiments, the model invokes the physics engine to simulate the loading / unloading of a set of objects into / from a shipping container to simulate cargo within the shipping container. The simulation of loading / unloading may include simulating forces acting on / between the objects during loading / unloading or in conjunction with the placement of the objects. As another example, the model may invoke the physics engine to simulate the transportation of cargo within a shipping container (e.g., to simulate forces that may act on the cargo during transportation).

[0079] The physics engine may model the shipping container and / or the load of objects to determine one or more characteristics associated with the loading of the objects into the shipping container (e.g., stability of the set of objects, weight distribution of the set of objects, etc.). In some embodiments, the physics engine may determine a value for the stability (or predicted stability) of the shipping container and / or the load of objects. The physics engine may determine that the shipping container and / or the load of objects is stable or unstable or predicted to be stable or unstable based on a comparison of the value for stability (or predicted stability) to a predefined stability threshold.

[0080] In some embodiments, the default stability threshold is set based on a confidence interval associated with stability. For example, the default stability threshold may be set such that there is a 95% probability that the shipping container / load is stable without objects falling off the load within the shipping container, etc. As another example, the default stability threshold may be set such that there is a 99% probability that the shipping container / load is stable. The default stability threshold may be set by a user (e.g., an administrator or other human operator).

[0081] In some embodiments, the physics engine is used in connection with determining the predicted stability of the shipping container / cargo, such as before an action is performed. For example, before performing a placement of an object in the shipping container (e.g., during loading), the system uses the physics engine to evaluate the predicted stability of placing the object at a particular location in the shipping container. As another example, before removing an object from the shipping container (e.g., during unloading), the system uses the physics engine to evaluate the predicted stability of removing the object from the shipping container. In response to determining that the predicted stability of the shipping container / cargo is stable after performing an action, the action may be performed or may be considered to be an action that is a possible / allowable action. For example, in response to determining that the predicted stability of the load exceeds a stability threshold after all objects to be loaded into the shipping container have been loaded, the system considers the load to be a feasible load. Conversely, upon a determination that the predicted stability of the load of the transport container / object is not stable (or is not sufficiently stable, such as being below a predefined stability threshold) after performance of an action (e.g., full or partial loading / unloading), the action may be performed or may be deemed to be an action that is not possible / not permissible.

[0082] In some embodiments, the physics engine is used in connection with determining the stability of a shipping container / load, such as after an action has been performed. The physics engine may be invoked by the system in connection with an in-line evaluation of the load, such as in connection with determining whether a plan for loading a set of objects should be updated (e.g., to account for differences in the offline simulated predictive geometric model and the actual characteristics of the load of objects). For example, after placing an object into a shipping container (e.g., during loading), the system may use the physics engine to evaluate stability in response to placing the object at a particular location within the shipping container. As another example, after removing an object from a shipping container (e.g., during unloading), the system may use the physics engine to evaluate the stability of the shipping container / remaining load after removing the object. In response to determining that the stability of the shipping container / load is stable (e.g., exceeds a predefined stability threshold), the system may consider the shipping container / load to be stable and plan a subsequent action (e.g., placing the next object or removing the next object, etc.). Conversely, in response to determining that the stability of the transport container / load is not stable (or is not sufficiently stable, such as being below a predefined stability threshold), the system may consider the transport container / load to be unstable, and the system may implement corrective action (e.g., to improve the stability of the transport container / load).

[0083] According to various embodiments, the system performs a corrective action in response to determining that the stability of the transport container / load is not stable (or is not stable enough, such as below a predefined stability threshold). Examples of corrective actions include: (i) providing a warning / suggestion to a user; (ii) requesting human intervention; (iii) determining and implementing a plan to remove an object causing instability from the transport container / load or adjust the placement of the object therein; (iv) determining and implementing a plan to add an object to the transport container / load that is predicted to improve the stability of the transport container / load (e.g., improve stability above a predefined stability threshold); and (v) deciding to insert a spacer that is predicted to improve the stability of the load. In some embodiments, one or more corrective actions are performed. The system may select one or more corrective actions to perform based at least in part on the likelihood (e.g., a measure of likelihood) that the one or more corrective actions may improve stability and / or the extent to which the one or more corrective actions are predicted to improve stability (e.g., a measure of extent).

[0084] In some embodiments, the physics engine is included in a module that is loaded and / or executed by a computer system that controls a robot that performs the loading / unloading of a set of items. In some embodiments, the physics engine is included in a module that is loaded and / or executed by a remote system (e.g., a server). For example, the physics engine may be hosted as a service that is called by one or more robotic systems to evaluate the stability or predicted stability, etc. of a shipping container / cargo.

[0085] The system determines stability or predicted stability based at least in part on sensor data related to a workspace, such as a shipping container, objects being or already being transported to the workspace, and attributes of objects to be loaded / unloaded from the shipping container in conjunction with the invocation of the physics engine. As an example, the sensor data is acquired by a vision system associated with the workspace in which a robotic arm performing the loading / unloading operates. In some embodiments, the physics engine determines or acquires a model of the shipping container / cargo. The model is generated based at least in part on the sensor data. The model of the shipping container / cargo may be generated by a modeling module that is invoked by the physics engine or the like. In some embodiments, the model includes one or more attributes related to one or more objects in the shipping container / cargo. Examples of item attributes include size (e.g., length, width, height, etc.), weight, center of gravity, type of packaging, degree of rigidity, an indication of whether the item is rigid or not, an identifier (e.g., barcode, label, etc.), degree of deformability, an indication of whether the object is considered fragile or not, an indication of whether the object is considered heavy (e.g., heavier than a predefined weight threshold), an indication of whether the object is prone to deformation or not. Other attributes may be included in or associated with the model. According to various embodiments, the physics engine determines the stability of the shipping container / cargo based at least in part on a model of the shipping container / cargo. As an example, the physics engine determines the stability of the shipping container / cargo based on the position (or relative position) of one or more items within the shipping container and at least one attribute of at least one item.

[0086] In some embodiments, the system finds a tentative solution and then iteratively tries to improve the solution with respect to one or more predefined objectives. For example, when determining an arrangement of a set of objects (or an arrangement of a subset of objects), the system uses an arrangement module to determine tentative feasible arrangements and then tries to improve the arrangements to improve loading / unloading execution time, improve predicted load stability, improve load weight distribution for distribution, etc. In some embodiments, the system uses one or more heuristics to make intelligent guesses on the tentative solution. The one or more heuristics may be empirically determined based on a training set of simulated loading / unloading history into / from shipping containers. For example, the one or more heuristics are determined based on performing a machine learning process to infer a heuristic associated with a solution whose corresponding score value (e.g., determined according to a predefined scoring function) exceeds a minimum score threshold.Examples of heuristics include loading heavy objects first (e.g., objects whose predicted weight exceeds a predefined weight threshold), loading large objects first (e.g., objects whose predicted volume or dimensions exceed a predefined size threshold), arranging heavy or large objects near the bottom of the shipping container (e.g., on the floor of the shipping container or in the row of the first set of objects), buffering fragile or deformable objects (e.g., whose fragility or deformability exceeds a corresponding predefined fragility or deformability threshold), and balancing such as selecting configurations that result in the formation of a "T" boundary over configurations that do not form a "T" boundary. providing an estimate for the number of objects that can be loaded, determining to load the object to form at least one "T" boundary with other objects (e.g., if the solution space for arranging the objects includes solutions where a "T" boundary is formed and solutions where the object sits vertically on another object), otherwise loading such objects toward the end of loading the shipping container or toward the top row of objects in the shipping container, loading such objects toward the end of loading the shipping container or toward the top row of objects in the shipping container, etc. Various other heuristics may be implemented.

[0087] An example of the system finding a tentative solution for distributing objects between two shipping containers is to determine a set of configurations to follow when placing objects along an exterior wall and subsequently placing objects between objects at the wall. For example, the system determines to place a first object to be engaged with the left wall and a second object to be engaged with the right wall. The system then determines to continue placing objects between the first and second objects while moving toward the centerline of the shipping container. A tentative solution may be formed based on the placement of a subset of objects along the walls, after which subsequent placements to fill gaps between the objects are optimized. In some embodiments, the system iteratively attempts to improve the solution. An example of iteratively attempting to improve the solution is by altering the placement by changing the order of items or by altering the position of the objects between the first and second objects placed against the wall. The system runs simulations in conjunction with identifying feasible solutions and invokes the model to evaluate various iteratively generated solutions. As another example, in response to determining a tentative or iteratively generated solution, the system analyzes the load (e.g., a model of a load of a set of objects in a shipping container) to determine configurations of objects predicted to cause instability or have adversely affected density. In response to identifying configurations predicted to have adversely affected density, stability, weight distribution, etc., the system determines alternative configurations of such objects to improve density, stability, weight distribution, etc. The system simulates the alternative configurations and evaluates the load to determine whether the arrangements correspond to feasible solutions.

[0088] In some embodiments, the system determines whether to reorder the objects as they are placed into the shipping container by determining whether to stage one or more objects in a buffer zone to allow the system to place objects that are to be delivered to the workspace after one or more objects first. The system selects the reordering of objects that has both (i) the highest score according to a predefined scoring function and (ii) meets one or more feasible loading criteria. Examples of feasible loading criteria include: (a) a maximum time to load, unload, or collectively load and unload, (b) a minimum expected density of objects to be loaded into the shipping container, (c) a minimum number of total objects to be loaded into the shipping container, (d) a minimum stability of a subset of objects to be loaded into the shipping container, (e) a weight distribution that meets one or more criteria (e.g., weight is distributed to a location of the shipping container that corresponds to the wheels of a truck that is expected to carry the shipping container or a location of the shipping container where the shipping container is otherwise supported during transportation), etc. Various other feasible loading criteria may be implemented. The one or more feasible loading criteria may be minimum requirements for loading a set of objects for the loading to be considered feasible.

[0089] Examples of machine learning processes that may be performed in connection with training the model include random forests, linear regression, support vector machines, naive Bayes, logistic regression, k-nearest neighbors, decision trees, gradient boosted decision trees, K-means clustering, hierarchical clustering, density-based spatial clustering of noisy applications (DBSCAN), principal component analysis, etc. In some embodiments, the system trains an XGBoost machine learning classifier model. As an example, the input to the classifier (e.g., XGBoost machine learning classifier model) is a combined feature vector or set of feature vectors, and based on the combined feature vector or set of feature vectors, the classifier model determines (i) a weight distribution of a load of objects, (ii) whether loading of a set of objects is feasible (e.g., safe, efficient), (iii) a partition of a set of objects into N subsets (e.g., N partitions) of objects, (iv) an arrangement / ordering of objects to be loaded into a particular shipping container, (v) an arrangement / orientation of objects to be loaded into a particular shipping container, etc.

[0090] Although the above examples are discussed in the context of the system loading a set of objects into one or more shipping containers, the robotic system may also be utilized in connection with unloading a set of objects from one or more shipping containers.

[0091] 2A and 2B are diagrams illustrating the arrangement of a set of objects within a shipping container according to various embodiments, for example, from a rear view of the shipping container.

[0092] In the example shown in FIG. 2A, the system obtains an array 200 of objects 211-239. The system determines the array 200 based on one or more boundary conditions (e.g., the position of the side walls 201-204 of the shipping container). In connection with determining the array 200, the system may generate / simulate a geometric model of the arrangement of the items, such as for optimizing the array (or selecting the best array within a predetermined computational execution time) according to a predefined score threshold. In response to determining the array 200, the system scores the array 200 based on a predefined score threshold. For example, the obtained score value indicates the quality (e.g., goodness) of the array. The array 200 has a relatively high density of objects (e.g., the system has determined a load with no space between the objects). Thus, the array 200 has a high score at least along the density dimension or object clearance (e.g., total clearance between the objects), but the array 200 may have other dimensions that are less high, such as weight distribution, time to load a set of objects, number of objects in the load, etc.

[0093] In the example shown in FIG. 2B, the system obtains an array 250 of objects 251-276. The system bounds a solution for the array of objects 251-276 based on the location of the side walls 201-204 of the shipping container. In response to determining the array 250, the system scores the array 250 based on a predefined score threshold. Although the array 250 may have a high score along various dimensions (such as weight distribution or execution time for loading / unloading), the array 250 has a relatively low density of objects (at least as viewed from the rear of the shipping container) and a relatively large number of gaps between the objects 251-276. For example, the array 250 has several gaps / voids between certain objects, such as a gap above object 252, a gap between objects 262 and 264, a gap between objects 265 and 266, and a gap between objects 253 and 257.

[0094] In some embodiments, the system evaluates the loading by depth layer of objects. For example, the system determines the loading of a layer of objects in the deepest part of a shipping container while the robot works to load objects into the shipping container (e.g., from the deepest point in the shipping container to a position corresponding to an opening at the back of the shipping container), and then determines the loading of subsequent layers of placed objects.

[0095] 2C is a diagram illustrating a "T" boundary between a subset of objects within a shipping container, according to various embodiments. In the illustrated example, the "T" boundary is formed in an array 290. For example, objects 291-293 are placed in the array such that a "T" boundary 294 is formed between objects 291-293. A "T" boundary is generally considered / predicted to be more stable than a vertical arrangement of objects (e.g., an arrangement in which one object is placed on top of another, etc.).

[0096] In some embodiments, the system is biased to select, from among a set of possible configurations of objects, configurations of objects that result in a "T" boundary with other objects. For example, the system utilizes a heuristic that configurations form "T" boundaries as a mechanism for trimming the solution space of possible configurations (e.g., solutions for which a "T" boundary does not form for a particular configuration are trimmed from the solution space of possible configurations).

[0097] 3A, 3B, 3C, and 3D are diagrams illustrating arrays of a set of objects within a shipping container, according to various embodiments. In the illustrated example, arrays 300 and 325 are shown in a side view within a shipping container carried on trailer 301, which is supported by wheels 302, 303, and arrays 350 and 375 are shown in a rear view of the shipping container.

[0098] As shown in FIG. 3A, array 300 includes objects that are loaded such that the density of objects is relatively high, there are few gaps between objects (e.g., no air spaces are provided within array 300), and several "T" boundaries are formed between the objects loaded into the shipping container. For example, array 300 includes a first "T" boundary between objects 306-308, a second "T" boundary between objects 307-309, and a third "T" boundary between objects 310-312. In some embodiments, the system is biased to select arrangements of objects that are predicted to form "T" boundaries with other objects. For example, the system implements heuristics related to prioritizing "T" boundaries over stacking objects vertically.

[0099] As shown in FIG. 3B, array 325 includes objects that are loaded such that the density of objects is lower than array 300, there are more gaps between objects (e.g., space exists, for example, between objects 328 and 329, between objects 331 and 333, etc.) than there were in array 300, and there are fewer "T" boundaries formed between objects loaded into the shipping container. The gaps between objects are indicated by hashed regions. For example, the gaps between various objects preclude the existence of a greater number of "T" boundaries. For example, array 325 includes a first "T" boundary between objects 305-307, a second "T" boundary between objects 305-309, and a third "T" boundary between objects 310-312. In some embodiments, the system is biased to select an arrangement of objects that is predicted to form a "T" boundary with other objects. For example, the system implements a heuristic related to prioritizing "T" boundaries over stacking objects vertically.

[0100] As shown in FIG. 3C, array 350 includes objects that are loaded such that the density of objects is relatively high, there are few gaps between objects (e.g., no empty spaces are provided within array 350), and several "T" boundaries are formed between the objects loaded into the shipping container. For example, array 350 includes a first "T" boundary between objects 353-354, a second "T" boundary between objects 354-356, a third "T" boundary between objects 351, 357, and 359, a fourth "T" boundary between objects 358-360, a fifth "T" boundary among objects 351, 352, and 359, etc. In some embodiments, the system is biased to select arrangements of objects that are predicted to form "T" boundaries with other objects. For example, the system implements a heuristic related to prioritizing "T" boundaries over stacking objects vertically.

[0101] As shown in FIG. 3D, array 375 includes objects that are loaded such that the density of objects is lower than array 350, there are more gaps between objects (e.g., spaces exist as shown by hashed regions, etc.) than in array 350, and there are fewer "T" boundaries formed between objects loaded into the shipping container. For example, the gaps between the various objects preclude the existence of a greater number of "T" boundaries. For example, the array includes a first "T" boundary between objects 384-386, a second "T" boundary between objects 385-387, and a third "T" boundary between objects 377-379, etc. In some embodiments, the system is biased to select an arrangement of objects that is predicted to form a "T" boundary with other objects. For example, the system implements a heuristic related to prioritizing "T" boundaries over stacking objects vertically. As an example, object 377 is stacked vertically on top of object 378, object 376, etc. The presence of vertical stacking of objects can lead to instability. In some embodiments, the system attempts to find a solution where the degree of vertical stacking is minimized, or where the degree of vertical stacking is less than a predefined number of vertical stacking configurations, or where the ratio of vertical stacking configurations to loaded objects is below a predefined vertical stacking threshold, etc.

[0102] In some embodiments, the system determines how to bound or trim a solution space of possible solutions (e.g., solutions for arranging a set of objects, etc.). As an example, the system generates a tree / graph where each node corresponds to a decision regarding an object (e.g., placement of an object, etc.). The system may then determine how to trim the tree to eliminate infeasible solutions or solutions that do not meet one or more predefined criteria. As another example, the system bounds the solution space based at least in part on one or more constraints. One example of a constraint is the execution time for the system to determine a solution. For inline determination of a solution, the execution time may be 5 seconds or less. For offline determination of a solution, the execution time may be 12 hours or less, etc.

[0103] The system uses heuristics to guess a solution (eg, distribution, ordering, ordering, etc.), then evaluates the guess and trims / bounds the search space accordingly.

[0104] 4A illustrates a tree that corresponds to different states or configurations of objects, according to various embodiments. In some embodiments, the system 100 of FIG. 1 determines or utilizes a search space 400 in connection with determining a configuration of one or more objects, such as a current object (e.g., a next object in a set of objects to be placed). In some embodiments, the system determines the configuration offline, such as by a server that simulates the arrangement of the objects into one or more shipping containers.

[0105] In the example shown in FIG. 4A, the search space 400 is represented in a tree structure. The search space 400 includes multiple nodes, each of which may correspond to an object placement scenario, and each branch may correspond to a placement of the current object at a particular location and a particular orientation. In some embodiments, the system determines whether a particular branch corresponds to a placement in which a "T" boundary is formed between the current object and other objects. For example, the system stores, for each branch in the search space 400, whether the corresponding placement forms a "T" boundary with other objects or forms a vertical stack with another object.

[0106] The search space 400 includes a root node 402. The root node 402 corresponds to the current state of a palette / stack of objects. For example, the system queries a state estimation service / module to obtain an estimated state and uses such estimated state as the state in the root node 402.

[0107] In response to determining the root node, the system iteratively determines placements for the next set of objects. For example, the system proceeds through various orders of the tree based on placements of various next objects. Nodes 404, 406, 408, and 410 correspond to placements of first-order objects (e.g., objects for which the system has determined the best placement). As an example, nodes 404, 406, 408, and 410 are primary nodes that respectively correspond to different placements of first-order objects (e.g., different positions and / or orientations, or different objects if the system is allowed to buffer some objects). As shown in Figures 4A and 4B, each of nodes 404, 406, 408, and 410 has secondary nodes branching off from it, respectively. For example, nodes 512, 514, 416, and 418 branch off from node 404. Each of the secondary nodes corresponds to a different configuration of the second-order objects (e.g., different positions and / or orientations, or different objects if the system is allowed to buffer some objects). Once the system has determined the configuration corresponding to each node, it determines nodes branching off from the node for the configurations of the remaining objects to be simulated. For example, in response to determining the configuration for node 412, the system determines nodes 420, 422, 424, and 426 (e.g., the system determines the estimated states and associated information for the nodes).

[0108] If the system were to analyze stability at each node in the search space 400, such an analysis could be computationally expensive, therefore, as shown in Figure 4B, various embodiments prune the search space to limit the number of nodes whose state / stability is determined in conjunction with determining the placement of objects (e.g., first-order objects). In some embodiments, the system prunes the search space to limit the number of vertical stacking placements between objects in a particular arrangement.

[0109] 4B illustrates a tree that corresponds to different states or configurations of objects, according to various embodiments. In some embodiments, the system 100 of FIG. 1 determines or utilizes a search space 430 in connection with determining a configuration of one or more objects, such as a current object (e.g., a next object in a set of objects to be placed). In some embodiments, the system determines the configuration offline, such as by a server that simulates the distribution, ordering, and / or arrangement of the objects in one or more shipping containers.

[0110] The system determines search space 430 based on determining whether to prune and / or how to prune search space 400. Partitioned regions 442, 444, 446, and 448 correspond to regions of the tree that the system has determined to be pruned from search space 400 (e.g., a tree) because they are unfavorable scenarios.

[0111] In some embodiments, the system determines whether a node corresponds to a favorable or unfavorable scenario based on one or more of: (i) simulating the state of the stack of objects for a configuration associated with the node; and (ii) one or more heuristics. In some embodiments, the system simulates the state of the stack of objects for a primary node (e.g., 404, 406, 408, and 410) in connection with determining the predicted stability of the stack of objects. In some embodiments, for all nodes having a higher order than the primary node (e.g., 412, 414, 416, 418, 420, 422, 424, 426, etc.), the system uses one or more heuristics to determine whether the corresponding configuration is predicted to be an unfavorable scenario (e.g., predicted to lead to a stable or unstable stack, predicted to have a significant cost (e.g., a cost higher than a cost threshold), predicted to place the robot in an improper pose, predicted to form a vertical stack of objects, etc.).

[0112] According to various embodiments, in response to determining the results of the analysis of the state, predicted stability, and / or favorability of the placement based on one or more heuristics, the system decides whether to prune the corresponding node. If the system decides to prune the node, it prunes the node and all potential nodes branching from the node. For example, in response to determining that node 426 is to be pruned (e.g., the node corresponds to an unfavorable scenario), the system correspondingly decides to prune nodes 432, 434, 436, etc., based on such nodes that directly or indirectly branch from node 426. In some embodiments, the system decides to prune nodes 432, 434, 436, etc., without further analysis using one or more heuristics. For example, the system decides to prune nodes 432, 434, 436, etc., based on the decision to prune node 426. In some embodiments, the system prunes all nodes (and nodes branching from such nodes) where the "T" boundary selection criteria are not met, such as the ratio of "T" boundaries to all objects being less than a predefined "T" boundary ratio threshold, or all nodes where the vertical stacking of objects corresponding to the array exceeds a predefined vertical stacking threshold (e.g., the maximum ratio of vertically stacked objects to the number of objects loaded).

[0113] In some embodiments, after determining the search space 430 (e.g., after pruning the search space 400), the system determines the best placement for the next object (e.g., the object relative to the primary node). The system can determine the best placement based on one or more of the predicted stability, the cost of the placement, the cost of placing the M objects after placing the next object and the next object, the predicted stability of the stack of objects after placing the M objects after placing the next object and the next object, and the desirability of the placement.

[0114] In some embodiments, the system determines that the best placement of the next object corresponds to a placement at node 408. For example, the system determines that the placement at node 408 is best based on determining that the state of the stack of objects remains favorable (e.g., cost-effective, stable, etc.) through several placement orders (e.g., the state remains favorable for the placement of the largest number of subsequent objects). In some embodiments, the system determines that nodes 438 and 440 remain favorable through the placement of nine different objects (e.g., the next object and M objects after the next object, where M is 8 in this example). In some embodiments, the system performs a beam search with respect to the search space 430 to determine the best placement.

[0115] In connection with determining whether to prune a node, the system can implement pruning criteria that can include one or more of: (i) the state is predicted to have a stability less than a state stability threshold, (ii) the state violates a heuristic (e.g., a deformable object is placed at or near the bottom of a stack of objects, etc.), (iii) the cost of the corresponding placement is predicted to exceed a cost threshold, (iv) the robot is predicted to be placed in an improper pose, (v) the predicted density of the stack of objects is less than a predefined density threshold, etc.

[0116] 5A-5C are diagrams illustrating a subset of objects loaded into a shipping container, according to various embodiments. As shown in FIGS. 5A-5C, an example framework for an order for loading objects 511-539 into a shipping container is provided. The solution space for an arrangement of objects 511-539 is bounded by walls 501-504 of the shipping container. According to various embodiments, the system selects an arrangement of objects 511-539 to optimize stability, such as based on the degree to which the objects form a "T" boundary.

[0117] In the example shown in FIG. 5A, array 500 corresponds to a state near the beginning of loading of objects into a shipping container. The system loads objects along sidewalls 501, 503 and then works to place objects toward the centerline until no further placement can be performed in the layer or until objects are placed from sidewall 501 to sidewall 503 with no space between them. As an example, one layer of objects may correspond to objects 538 and 539 forming a bottom layer of objects. In some examples, a layer corresponds to a predefined height range. For example, objects 537, 538, and 539 may correspond to a single bottom layer.

[0118] While FIG. 5A shows objects 519, 520-522, 531, and 532 being stacked along sidewall 503 before the bottom layer object is packed from sidewall 501 to sidewall 503, in various embodiments, a bias may be implemented to preferentially arrange objects in a threshold number of layers (e.g., one layer, two layers, etc.) before the system proceeds with stacking objects along the wall to form a new layer. For example, after objects 537, 538, and 539 have been placed, the system may select an object to place in the space between objects 537, 539 (e.g., above object 538). The system may then place objects 532 and 536 before proceeding with placing objects in the layer between objects 532 and 536 (e.g., placing object 533).

[0119] In the example shown in FIG. 5B, array 550 corresponds to a state about halfway through the process of loading objects into a shipping container. The system has sequentially packed objects between the objects loaded along sidewalls 501, 503. The system may sequentially load objects from an outer location corresponding to the sidewall toward the centerline. For example, the system may place object 535, then object 528, then object 534 to complete a layer such that the load has no gap between objects 532 and 536. After completing the layer between objects 532 and 536, the system may load object 530 to begin to fill the layer between object 531 and sidewall 501. In some embodiments, before placing object 530, the system may first place objects along sidewall 501 above objects 536 and / or 535.

[0120] In the example shown in FIG. 5C, array 575 corresponds to a completed loading of objects into the shipping container. The system sequentially packs objects between the objects loaded along sidewalls 501, 503, and sequentially loads another layer of objects between sidewalls 501, 503, working its way from the outside of the layer to the centerline. The system loads objects to limit gaps between objects in a particular layer. For example, the system selects object 529 as ideally placed between object 538 and objects 30, 533 because object 529 fits within the gap to complete the layer. The system may further select object 529 because its placement forms a level top surface with object 528 that may more stably support one or more other objects.

[0121] Although FIGS. 5A-5C are described in connection with loading a shipping container, various embodiments perform a substantially similar process in reverse in connection with unloading objects 511-539.

[0122] 6 is a flow chart illustrating a process for arranging a set of objects to be loaded into a set of shipping containers, according to various embodiments. In some embodiments, process 600 is performed by system 100 of FIG. 1 or by one or more servers associated with providing cloud services, etc. In some embodiments, process 600 is performed by a placement module. For example, a system (e.g., a model for planning the loading of objects) invokes the placement module to invoke process 600.

[0123] At step 605, an indication of a set of objects to be loaded into a shipping container is received. In some embodiments, the indication of the set of objects may be determined based at least in part on information obtained from one or more sensors in the workspace, such as a vision system that detects objects that have been delivered or are being delivered to a robotic arm configured to load the objects into the shipping container. In some embodiments, the system obtains the indication of the set of objects from a data structure, such as a set of objects that is mapped to a manifest or order or other list of items to be loaded into a shipping container or shipped to a particular destination.

[0124] At step 610, arranging the set of objects to be loaded into the transport container according to the stability model. In response to receiving a suggestion for the set of objects to be loaded into the transport container, determining an arrangement to follow when placing the objects. For example, the arrangement may include locations of various objects and / or an order in which the objects are to be loaded into the transport container. In some embodiments, the arrangement includes an indication of a plan for the robot to follow when loading the objects into the transport container.

[0125] In some embodiments, the system determines sequences based at least in part on determining particular sequences that meet one or more stability criteria or have a score value (e.g., a score value determined according to a predefined score threshold) that exceeds a predefined score threshold. The one or more stability criteria may include a bias or threshold for a minimum number or ratio of "T" boundaries that are met for a feasible sequence.

[0126] At step 615, the sequence is provided as an output for use in connection with loading the set of objects. In some embodiments, the system provides the sequence to a robotic system that is controlled to load the set of objects.

[0127] At step 620, a determination is made as to whether process 600 is complete. In some embodiments, process 600 is determined to be complete in response to a determination that there are no more objects to be loaded into the shipping container, the system receiving a confirmation that a robotic system configured to load objects into a shipping container has received a sequence, the system receiving an indication that the robotic system has completed loading a set of objects into a shipping container, an administrator indicating that process 600 is to be paused or stopped, etc. In response to a determination that process 600 is complete, process 600 ends. In response to a determination that process 600 is not complete, process 600 returns to step 605.

[0128] 7 is a flow chart illustrating a process for arranging a set of objects to be loaded into a set of shipping containers, according to various embodiments. In some embodiments, process 700 is performed by system 100 of FIG. 1 or by one or more servers associated with providing a cloud service, etc. In some embodiments, process 700 is performed by a placement module. For example, a system (e.g., a model for planning the loading of objects) calls the placement module to invoke process 700. Process 700 may be invoked by process 600 of FIG. 6, such as at step 610 of process 600.

[0129] An indication that a set of objects is to be loaded is received at step 705. In some embodiments, the system receives an indication that a set of objects is to be delivered to a destination location and provides a feasible solution for arranging the set of objects.

[0130] At step 710, a suggestion for a set of objects is received. In some embodiments, in connection with obtaining a suggestion to distribute the set of objects, the system obtains a set of objects to be loaded and shipped to a destination. As an example, the system obtains a manifest of the objects. Obtaining a suggestion for the set of objects may include obtaining one or more attributes respectively associated with the objects. As another example, the system obtains the set of objects based on detecting the set of objects within (or being transported to) a workspace of a robotic system controlled to load the objects into a shipping container.

[0131] At step 715, one or more objects are selected from the set of objects. In some embodiments, the system selects one or more of the objects based on one or more heuristics for arranging the load. As an example, the one or more heuristics are obtained using a machine learning process. The machine learning process determines a set of heuristics that are predicted to indicate feasible arrangements (e.g., arrangements in which placement of the objects is feasible or that otherwise meet one or more stability criteria or score thresholds). One example of a heuristic may be to select heavier or larger objects (e.g., objects having weights heavier than a weight threshold or dimensions or volumes larger than a size threshold) to be arranged toward the bottom of a shipping container (e.g., to avoid leaving heavy objects toward the end of the load that may cause instability if placed toward the top of a stack / layer of cargo). As another example, a heuristic may select objects that are not rigid or fragile (e.g., objects having rigid packaging, deformability below a deformability threshold, etc.) to be arranged toward the bottom of a shipping container. As another example of a heuristic, the system may select objects for a particular layer based on a heuristic that ensures that objects are placed along the sidewalls (if there are possible feasible arrangements) and then the spaces between the objects at the sidewalls are successfully filled until the layer is complete.

[0132] In some embodiments, the one or more objects are selected based on the order in which the objects are delivered to the workspace. For example, if the system does not have the ability to buffer objects, the system selects the objects based on the order in which the objects arrive at the workspace. As another example, if the system has the ability to buffer one or more objects, the system selects one or more objects that can be best placed / arranged. The one or more objects are selected from among the next N objects, taking into account the possibility of buffering M objects to reorder the objects. N and M are positive integers.

[0133] In some embodiments, the system determines the sequence based at least in part on a sequence / configuration model.

[0134] In some embodiments, the system determines the arrangement based at least in part on one or more heuristics. For example, the system uses one or more heuristics to make an educated guess of an arrangement that is deemed feasible. The system can then iteratively improve the arrangement based on an evaluation of the arrangement. An initial arrangement (e.g., an initial iteration of process 1500 for a set of objects) may be selected using one or more heuristics, and subsequent arrangements may be variations of the initial arrangement or a prior arrangement based on updates to the arrangement made at least in part based on the evaluation of the arrangement. The evaluation of the arrangement may identify one or more objects that are predicted to cause the arrangement to be infeasible or otherwise adversely affect the score of the arrangement to a degree greater than a predefined score threshold. For example, the evaluation of the arrangement identifies one or more objects that are predicted to cause instability. As another example, the evaluation of the arrangement identifies one or more objects that are predicted to cause violations of weight distribution criteria or stability criteria.

[0135] At step 720, the loading of one or more objects into the shipping container is simulated. In some embodiments, the system invokes a simulation module to simulate various loadings (e.g., arrangements, sequences, etc.) of the one or more objects. For example, the simulation module generates a geometric model of an arrangement of the set of objects. As another example, the system simulates the loading of the set of objects to determine whether feasible arrangements / configurations exist for the one or more objects or to select the best arrangement / configuration of the one or more objects.

[0136] At step 725, a score value for the loading of one or more objects is determined based at least in part on a predefined scoring function. In some embodiments, the system scores the sequence based on a predefined scoring function. For example, the system scores the sequence in connection with assessing whether the sequence is a feasible / possible sequence and / or determining a best sequence.

[0137] In some embodiments, the predefined scoring function takes into account cost, time, stability, density, weight distribution, etc. Various other characteristics of the load (e.g., sequence) can be implemented in the predefined scoring function.

[0138] In some embodiments, determining the score value of the loading includes comparing the loading with respect to one or more heuristics. For example, the system may use a set of heuristics to quickly determine whether a loading is predicted to be feasible or infeasible. The system may use a set of heuristics to evaluate the loading before performing a computationally expensive simulation of the loading, such as to avoid arrangements where one or more heuristics suggest that the loading is predicted to violate one or more loading criteria.

[0139] At step 730, the system determines whether the arrangement of one or more objects is a feasible arrangement. In some embodiments, the system determines whether the arrangement is a feasible arrangement based at least in part on determining whether a score value of the load (e.g., possible arrangement) exceeds a predefined score threshold.

[0140] In response to determining at step 730 that the arrangement of the one or more objects is not a feasible arrangement, process 700 returns to step 715 where another subset of the set of objects is selected, and loading of the next subset of the set of objects is simulated and evaluated. For example, process 700 repeats steps 715-730 until the arrangement of the one or more objects (e.g., a subset of the set of objects) is determined to be a feasible arrangement. Conversely, in response to determining at step 730 that the arrangement of the one or more objects is a feasible arrangement, process 700 proceeds to step 735.

[0141] At step 735, the system determines whether the set of objects includes another object for which a feasible arrangement is to be determined. In response to determining at step 735 that the set of objects includes another object for which a feasible arrangement is to be determined, process 700 returns to step 715, where a next subset of the set of objects is simulated and evaluated. For example, process 700 repeats steps 715-735 until no further arrangements are to be determined for objects in the set of objects. Conversely, in response to determining at step 735 that the set of objects does not include additional objects for which a feasible arrangement is to be determined, process 700 proceeds to step 740, where suggestions for arrangements of the set of objects are provided.

[0142] At step 745, a determination is made as to whether process 700 is complete. In some embodiments, process 700 is determined to be complete in response to a determination that there are no more objects to be loaded into the shipping container, the system receiving a confirmation that a robotic system configured to load objects into a shipping container has received a sequence, the system receiving an indication that the robotic system has completed loading a set of objects into a shipping container, an administrator indicating that process 700 is to be paused or stopped, etc. In response to a determination that process 700 is complete, process 700 ends. In response to a determination that process 700 is not complete, process 700 returns to step 705.

[0143] 8 is a flow chart illustrating a process for arranging a set of objects to be loaded into a set of shipping containers, according to various embodiments. In some embodiments, process 800 is performed by system 100 of FIG. 1 or by one or more servers associated with providing a cloud service, etc. In some embodiments, process 800 is performed by a placement module. For example, a system (e.g., a model for planning the loading of objects) calls the placement module to invoke process 800. Process 800 may be invoked by process 600 of FIG. 6, such as at step 610 of process 600.

[0144] In some embodiments, the system uses heuristics to determine the order of selection. For example, the system may select objects to be placed along the sides before the center, and the system may select objects based on one or more heuristics (such as object stiffness, object weight, object shape, etc.).

[0145] At step 805, a suggestion is received that a set of objects is to be loaded. At step 810, the suggestion of the objects is obtained. At step 815, an arrangement of the set of objects is selected. At step 820, loading of the one or more objects into the shipping container according to the selected arrangement is simulated. At step 825, a score value of the arrangement of the one or more objects is determined based at least in part on a predefined scoring function. At step 830, the system determines whether the arrangement of the one or more objects is a feasible arrangement. In response to determining at step 830 that the arrangement of the one or more objects is not a feasible arrangement, process 800 proceeds to step 835. Conversely, in response to determining at step 830 that the arrangement of the one or more objects is a feasible arrangement, process 800 proceeds to step 840. According to various embodiments, steps 805-830 correspond to or are similar to steps 705-730 of FIG. 7.

[0146] The current sequence is discarded at step 835. In some embodiments, in response to determining that a particular sequence is not feasible, the sequence is discarded from the set of potential sequences.

[0147] In step 840, information about the arrangement and / or an indication that the current arrangement is feasible is stored.

[0148] At step 845, the system determines whether additional arrangements are to be analyzed. In some embodiments, process 800 repeats steps 820-845 until there are no more configurations for the various sets of objects to be determined, or until there are no more potential arrangements for arranging / loading a particular set of objects to be evaluated. For example, process 800 may end after a first arrangement is deemed a feasible arrangement and the evaluation of arrangements has been performed for a predefined time threshold, or after a set of X possible / feasible arrangements have been determined for arranging / loading a set of objects to be loaded into a shipping container. X may be a positive integer. The best arrangement among the evaluated feasible arrangements may be selected as the arrangement to be implemented.

[0149] In response to determining at step 845 that additional sequences are to be analyzed, process 800 proceeds to step 850 where a next configuration is selected. Process 800 then returns to step 820, where process 800 repeats steps 820-845 until there are no additional sequences to be analyzed. Conversely, in response to determining at step 845 that there are no additional sequences to be analyzed, process 800 proceeds to step 855.

[0150] At step 855, a sequence is selected from the possible sequences. In some embodiments, the system selects the optimal or best sequence from among the set of possible sequences. As an example, the optimal or best sequence is selected based at least in part on the score value of the sequence. The score value is determined according to a predefined score threshold, and the sequence with the highest score value is considered to be the optimal or best sequence.

[0151] In some embodiments, determining sequences for a set of objects is limited by the run time required for the system to evaluate and select sequences. If the run time ends before the system has evaluated all sequences (e.g., all potentially feasible solutions), the system selects the best sequence from the set of feasible sequences that have been determined at the end of the run time.

[0152] At step 860, an arrangement suggestion is provided. In some embodiments, the system provides a suggestion that an arrangement be provided to another system or module that invoked process 800 (e.g., process 600) or to a robotic system used to load the set of objects into a shipping container. In some embodiments, providing the arrangement suggestion includes storing the arrangement in a set of feasible arrangements (e.g., from which a preferred or best arrangement may be selected for implementation).

[0153] At step 865, a determination is made as to whether process 800 is complete. In some embodiments, process 800 is determined to be complete in response to a determination that there are no more objects to be loaded into the shipping container, a determination that a sequence has been selected for loading a set of objects, the system receiving a confirmation that a robotic system configured to load objects into a shipping container has received a sequence, the system receiving an indication that the robotic system has completed loading a set of objects into a shipping container, an administrator indicating that process 800 is paused or stopped, etc. In response to a determination that process 800 is complete, process 800 ends. In response to a determination that process 800 is not complete, process 800 returns to step 805.

[0154] 9 is a flow chart illustrating a process for evaluating an alignment of a set of objects to be loaded into a shipping container, according to various embodiments. In some embodiments, process 900 is performed by system 100 of FIG. 1 or by one or more servers associated with providing a cloud service, or the like. In some embodiments, process 900 is performed by a placement module. For example, a system (e.g., a model for planning the loading of objects) calls the placement module to invoke process 900. As an example, process 900 is invoked by step 825 of process 800.

[0155] An indication to determine a score value for the sequence is obtained at step 905. In some embodiments, the system receives the indication to determine a score value in connection with determining whether the sequence is feasible. For example, the system simulates a load according to the sequence and evaluates the simulated load.

[0156] At step 910, a model of the arrangement is obtained. In some embodiments, the system obtains or generates a model of the arrangement according to a simulation. As an example, the model is a geometric representation of a set of objects placed according to a corresponding arrangement.

[0157] At step 915, the model is segmented. In response to obtaining the model, the system segments the model to identify the various objects loaded within the shipping container. In some embodiments, the model is already segmented (e.g., the simulation of the array preserves the segmentation of the objects), in such cases step 915 may be omitted.

[0158] The number of "T" boundaries in the array is determined in step 920. The system analyzes a model of the loading of a set of objects and determines the number of "T" boundaries formed between the objects.

[0159] At step 925, a score value for the sequence is determined based at least in part on the number of "T" boundaries in the sequence. In some embodiments, the system determines a score value for the sequence according to a predefined scoring function, where the score value is based at least in part on the number of "T" boundaries in the sequence.

[0160] In step 930, the score values ​​for the array are stored.

[0161] At step 935, a suggestion of a score value for the arrangement is provided. The suggestion of a score value may be provided to another system or module that invoked process 900 (e.g., process 800) or to a robotic system used to load the set of objects into a shipping container. In some embodiments, providing the suggestion of a score value includes storing a score value associated with the arrangement (e.g., from which a preferred or best arrangement may be selected for implementation), such as a mapping of arrangements to score values.

[0162] At step 940, a determination is made as to whether process 900 is complete. In some embodiments, process 900 is determined to be complete in response to a determination that there are no more sequences to be evaluated, a determination that there are no more objects to be loaded into the shipping container, the system receiving a confirmation that a robotic system configured to load objects into a shipping container has received a sequence, the system receiving an indication that the robotic system has completed loading a set of objects into the shipping container, an administrator indicating that process 900 is paused or stopped, etc. In response to a determination that process 900 is complete, process 900 ends. In response to a determination that process 900 is not complete, process 900 returns to step 905.

[0163] 10 is a flow chart illustrating a process for evaluating an alignment of a set of objects to be loaded into a shipping container, according to various embodiments. In some embodiments, process 1000 is performed by system 100 of FIG. 1 or by one or more servers associated with providing a cloud service or the like. In some embodiments, process 1000 is performed by a placement module. For example, a system (e.g., a model for planning the loading of objects) calls the placement module to invoke process 1000. As an example, process 1000 is invoked by step 830 of process 800.

[0164] At step 1005, an indication of the score value is obtained. At step 1010, the system determines whether the score value is greater than a predefined score threshold. In response to determining at step 1010 that the score value is greater than the predefined score threshold, process 1000 proceeds to step 1015 where the arrangement is deemed feasible. Conversely, in response to determining at step 1010 that the score value is not greater than the predefined score value, process 1000 proceeds to step 1020 where the arrangement is deemed not feasible. At step 1025, an indication of the arrangement's feasibility is provided. The indication of the feasibility may be provided to another system or module that invoked process 1000 (e.g., process 800) or to a robotic system used to load the set of objects into a shipping container. In some embodiments, providing an indication of the arrangement's feasibility includes storing the arrangement in a set of possible arrangements (e.g., from which a preferred or best arrangement may be selected for implementation), such as a list of possible arrangements.

[0165] At step 1030, a determination is made as to whether process 1000 is complete. In some embodiments, process 1000 is determined to be complete in response to a determination that there are no more sequences to be evaluated, a determination that there are no more objects to be loaded into the shipping container, the system receiving a confirmation that a robotic system configured to load objects into a shipping container has received a sequence, the system receiving an indication that the robotic system has completed loading a set of objects into the shipping container, an administrator indicating that process 1000 is paused or stopped, etc. In response to a determination that process 1000 is complete, process 1000 ends. In response to a determination that process 1000 is not complete, process 1000 returns to step 1005.

[0166] FIG. 11 is a flow chart illustrating a process for evaluating an alignment of a set of objects to be loaded into a shipping container, according to various embodiments. In some embodiments, process 1100 is performed by system 100 of FIG. 1 or by one or more servers associated with a cloud service offering or the like. In some embodiments, process 1100 is performed by a placement module. For example, a system (e.g., a model for planning the loading of objects) calls the placement module to invoke process 1100. As an example, process 1100 is invoked by step 830 of process 800.

[0167] An indication of the score value is obtained at step 1105. At step 1110, the system determines whether the value associated with the number of "T" boundaries in the array is greater than a predefined "T" boundary threshold.

[0168] In response to determining at step 1110 that the value associated with the number of "T" boundaries in the array is greater than the predefined "T" boundary threshold, process 1100 proceeds to step 1115 where the array is deemed feasible. Conversely, in response to determining at step 1110 that the value associated with the number of "T" boundaries in the array is not greater than the predefined "T" boundary threshold, process 1100 proceeds to step 1120 where the array is deemed not feasible.

[0169] At step 1125, a suggestion of the feasibility of the arrangement is provided. The suggestion of the feasibility may be provided to another system or module that invoked process 1000 (e.g., process 800) or to a robotic system used to load the set of objects into a shipping container. In some embodiments, providing the suggestion of the feasibility of the arrangement includes storing the arrangement in a set of feasible arrangements (e.g., from which a preferred or best arrangement may be selected for implementation), such as a list of feasible arrangements.

[0170] At step 1130, a determination is made as to whether process 1100 is complete. In some embodiments, process 1100 is determined to be complete in response to a determination that there are no more sequences to be evaluated, a determination that there are no more objects to be loaded into the shipping container, the system receiving a confirmation that a robotic system configured to load objects into a shipping container has received a sequence, the system receiving an indication that the robotic system has completed loading a set of objects into the shipping container, an administrator indicating that process 1100 is paused or stopped, etc. In response to a determination that process 1100 is complete, process 1100 ends. In response to a determination that process 1100 is not complete, process 1100 returns to step 1105.

[0171] 12 is a flow chart illustrating a process for planning the loading of a set of objects into a shipping container, according to various embodiments. In some embodiments, process 1200 is performed by system 100 of FIG. 1 or by one or more servers associated with providing a cloud service, etc. In some embodiments, process 1200 is performed by a placement module. For example, a system (e.g., a model for planning the loading of objects) invokes the placement module to invoke process 1200.

[0172] An indication is received that a plan is to be determined for loading the set of objects into the shipping container at step 1205. In some embodiments, in response to determining / selecting a sequence for loading the set of objects, the system determines a plan for the robotic arm to load the set of objects into the shipping container according to the sequence.

[0173] At step 1210, a plan for loading the set of objects into the shipping container is determined. The system may determine the plan for loading the set of objects based on building the load layer by layer (e.g., stacking layers from the bottom of the shipping container to the top of the shipping container) and / or based on loading the objects into the deepest available position / layer within the shipping container, followed by loading objects until positioned all the way to the back of the shipping container or until the set of objects is completely loaded.

[0174] At step 1215, the plan suggestions are provided as output for use in connection with loading the set of objects. In some embodiments, the system provides the plan to a robotic system that is controlled to load the set of objects.

[0175] At step 1220, a determination is made as to whether process 1200 is complete. In some embodiments, process 1200 is determined to be complete in response to a determination that there are no further plans for loading objects into a shipping container to be determined, the system receiving a confirmation that a robotic system configured to load objects into a shipping container has received a sequence, the system receiving an indication that the robotic system has completed loading a set of objects into a shipping container, an administrator indicating that process 1200 is paused or stopped, etc. In response to a determination that process 1200 is complete, process 1200 ends. In response to a determination that process 1200 is not complete, process 1200 returns to step 1205.

[0176] 13 is a flow chart illustrating a process for planning loading of a set of objects into a shipping container, according to various embodiments. In some embodiments, process 1300 is performed by system 100 of FIG. 1 or by one or more servers associated with providing a cloud service or the like. In some embodiments, process 1300 is performed by a placement module. For example, a system (e.g., a model for planning loading of objects) calls the placement module to invoke process 1300. As an example, process 1300 is invoked by step 1210 of process 1200.

[0177] At step 1304, a suggestion for an arrangement of the set of objects is obtained. For example, the system determines / obtains a best or optimal arrangement to follow when the set of objects are loaded into a shipping container. At step 1310, a current object to be placed is determined. The current object may be the next object to be placed. At step 1315, a configuration model is queried to determine an arrangement of the objects. At step 1320, loading of the objects into the shipping container is simulated based at least in part on the arrangement. At step 1325, a score value for the arrangement is determined based at least in part on a predefined scoring function. For example, the system evaluates the simulated loading. At step 1330, the system determines whether the arrangement is a feasible arrangement. For example, the system determines whether a score value associated with the arrangement exceeds a predefined score threshold. As another example, the system determines whether the arrangement leads to a violation of a stability criterion. In response to determining at step 1330 that the placement is not feasible, process 1300 proceeds to step 1335 where the placement for the current object is discarded and a next current object is determined. In some embodiments, selecting the next current object for which the placement is evaluated corresponds to selecting the same object and simulating another placement. In response to selecting the next current object, process 1300 returns to step 1315. Process 1300 then repeats steps 1315-1330 until a feasible placement is determined. In response to determining at step 1330 that the placement is feasible, process 1300 proceeds to step 1335 where a placement suggestion is stored. For example, information regarding the placement is stored in a list of feasible placements. In step 1340, the system determines whether additional objects are to be placed. In response to determining at step 1340 that additional objects in the set of objects are to be placed, process 1300 proceeds to step 1345 where a next current object is determined / selected. Then, the process 1300 returns to step 1315, and the process 1300 repeats steps 1315-1340 until there are no more objects in the set of objects for which an arrangement / configuration is to be determined. In step 1350, the system determines a plan according to the feasible configurations.For example, the system determines a plan to be followed by the robotic system in loading a set of objects according to the determined sequence. In step 1355, a plan suggestion is provided.

[0178] At step 1360, a determination is made as to whether process 1300 is complete. In some embodiments, process 1300 is determined to be complete in response to a determination that there are no further plans for loading objects into a shipping container to be determined, the system receiving a confirmation that a robotic system configured to load objects into a shipping container has received a sequence, the system receiving an indication that the robotic system has completed loading a set of objects into a shipping container, an administrator indicating that process 1300 is paused or stopped, etc. In response to a determination that process 1300 is complete, process 1300 ends. In response to a determination that process 1300 is not complete, process 1300 returns to step 1305.

[0179] Although the descriptions of the examples shown in Figures 6-13 are provided in the context of loading a set of objects into a shipping container, similar processes may be performed in the context of unloading a set of objects into a shipping container, according to various embodiments.

[0180] 14 is a flow chart illustrating a process for determining a plan for unloading a set of objects from a shipping container, according to various embodiments. In some embodiments, process 1400 is performed by system 100 of FIG. 1 or by one or more servers associated with providing a cloud service, etc. In some embodiments, process 1400 is performed by a placement module. For example, a system (e.g., a model for planning the loading of objects) invokes the placement module to invoke process 1400.

[0181] In some embodiments, unloading a set of objects from a shipping container is similar to loading a set of objects in reverse order, for example, the system determines which object to unload next or the order in which to unload the objects based on the effect of removing the object on the stability of the objects within the shipping container.

[0182] At step 1405, a suggestion of a set of objects to be unloaded from the shipping container is received. At step 1410, a plan for unloading the set of objects from the shipping container is determined. At step 1415, the suggestion of the plan is provided as an output for use in connection with unloading the set of objects. At step 1420, a determination is made as to whether the process 1400 is complete. In some embodiments, the process 1400 is determined to be complete in response to a determination that there is no further plan for unloading objects into the shipping container to be determined, the system receiving a confirmation that the robotic system configured to unload objects into the shipping container has received a plan, the system receiving an indication that the robotic system has completed unloading the set of objects into the shipping container, an administrator indicating that the process 1400 is paused or stopped, etc. In response to a determination that the process 1400 is complete, the process 1400 ends. In response to determining that the process 1400 is not complete, the process 1400 returns to step 1405 .

[0183] Various examples of the embodiments described herein are described with reference to flow charts. Although the examples may include some operations performed in a particular order, according to various embodiments, various operations may be performed in different orders and / or various operations may be combined into a single operation or performed in parallel.

[0184] Although the above embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and are not intended to be limiting.

Claims

1. 1. An object sequencing system for sequencing a set of objects for loading into a shipping container, comprising:

1. A processor comprising: receiving an indication that the set of objects is to be loaded into the shipping container for transportation from a source location to a destination location; determining an arrangement of the set of objects to be loaded into the shipping container according to a stability model based at least in part on object information corresponding to the set of objects; a processor configured to provide the array of the set of objects to a robotic system for performance in connection with loading the set of objects into the transport container; a memory configured to store the object information; A system comprising:

2. 2. The object ordering system of claim 1, further comprising: one or more sensors configured to obtain information indicative of one or more object attributes of the set of objects; The one or more object attributes are stored within the object information.

3. The object ordering system of claim 1 , wherein the object information includes one or more object attributes.

4. 4. The object ordering system of claim 3, wherein the one or more object attributes include one or more of: shape, mass, material, moment of inertia, stiffness, object size, object dimensions, object weight, packaging type, and object identifier.

5. 2. The object ordering system of claim 1, wherein the stability model optimizes a T-boundary formed at an interface between objects to be loaded into the shipping container.

6. 6. The object ordering system of claim 5, wherein when a first object is positioned such that it partially rests on a second object and partially rests on a third object, a T-boundary is formed at the interface between the three objects.

7. 6. The object ordering system of claim 5, wherein determining the arrangement according to the stability model includes selecting an arrangement based on a scoring function in which a T-boundary has a more favorable score value than a stack of vertically aligned objects.

8. 2. The sequencing system of claim 1, wherein determining the sequence according to the stability model includes simulating packing of the one or more objects according to a particular sequence and predicting stability of the objects packed according to the particular sequence.

9. 2. The object ordering system of claim 1, further comprising: A system comprising one or more sensors configured to obtain information regarding a state of a workspace, the one or more sensors including a vision system or a force-based sensor.

10. 10. The object ordering system of claim 9, wherein utilizing the array of the set of objects in connection with loading the set of objects into the shipping container includes iteratively performing: (i) an operation of a robotic arm packing a particular object in the set of objects according to the array; (ii) acquiring sensor information from the one or more sensors after placement of the particular object; (iii) determining whether a current configuration of the set of objects satisfies a stability constraint; and (iv) performing proactive measures to update the current configuration to correct for detected instability if the current configuration of the set of objects does not satisfy the stability constraint.

11. 2. The object ordering system of claim 1, wherein an ordering includes an indication of an order in which the set of objects are to be loaded into the shipping container.

12. 12. The object ordering system of claim 11 , wherein loading the set of objects according to the order includes (i) operating a robot to pack a layer of objects in an innermost portion of the shipping container, (ii) operating the robot to move toward an opening of the shipping container, and (iii) operating the robot to pack subsequent layers of objects, wherein in each subsequent layer, the robot loads objects along a first wall and a second wall of the shipping container and then loads objects between the objects loaded at the walls.

13. 13. The object ordering system of claim 12, wherein loading an object between the objects loaded at the wall includes applying a force with a robotic arm to press an object against the wall and loading an object inside between the objects at the wall using a robot.

14. 2. The object ordering system of claim 1, wherein the one or more processors further comprise: determining to unload the shipping container; capturing an image of the shipping container; creating a split view of a workspace including the shipping container; determining a plan for unloading a set of objects; The system is configured to control a robot to unload the set of objects according to the plan.

15. 15. The system of claim 14, wherein the plan is determined based on predicting stability of at least a subset of the objects within the shipping container during unloading.

16. 15. The object sequencing system of claim 14, wherein determining the plan for unloading the set of objects comprises performing an evaluation of one or more plans for unloading the set of objects, the evaluation of the one or more plans comprising: Simulate the removal of the first item, Calculating a predicted stability of a remaining set of objects within the shipping container; comparing the predicted stability to a predefined stability threshold; determining to retrieve another item before retrieving the first item if the predicted stability is determined to be less than the predefined stability threshold.

17. 17. The object ordering system of claim 16, wherein the evaluation of the one or more plans includes iteratively evaluating different plans for unloading the set of objects until a stability of unloading an object from the set of objects associated with a plan is determined to meet the predefined stability threshold.

18. 15. The object ordering system of claim 14, wherein unloading the set of objects includes applying a force with one or more robotic arms to one or more other objects in the shipping container while the robot is retrieving a particular object.

19. 1. A method for aligning a set of objects for loading into a shipping container, comprising: receiving, by one or more processors, an indication that the set of objects is to be loaded into the shipping container for transportation from a source location to a destination location; determining an arrangement of the set of objects to be loaded into the shipping container according to a stability model based at least in part on object information corresponding to the set of objects; providing the array of the set of objects to a robotic system for performance in connection with loading the set of objects into the shipping container; A method comprising:

20. A computer program product embodied in a non-transitory computer-readable medium, computer instructions for receiving, by one or more processors, an indication that a set of objects is to be loaded into a shipping container for transportation from a source location to a destination location; computer instructions for determining an arrangement of the set of objects to be loaded into the shipping container according to a stability model based at least in part on object information corresponding to the set of objects; computer instructions for providing the array of the set of objects to a robotic system for execution in connection with loading the set of objects into the shipping container; A computer program product comprising:

Images