Robotic system with error detection and dynamic packing mechanism

Abstract

To solve the problem in which robots lack the granularity of control and the flexibility in executed actions to take into account deviations, errors or uncertainties that may result from various real-world factors.SOLUTION: A method for operating a robotic system includes: determining a discretized object model based on source sensor data; comparing it to a packing plan or master data; determining a discretized platform model and height measures based on destination sensor data; comparing the discretized platform model and / or the height measures to an expected platform model and / or expected height measures; and determining one or more errors by (i) determining at least one source matching error by identifying one or more disparities between the discretized object model and the packing plan or master data, or (ii) determining at least one destination matching error by identifying one or more disparities between the discretized platform model or height measures and the expected platform model or expected height measures, respectively.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application includes subject matter related to a concurrently filed U.S. patent application entitled "A ROBOTIC SYSTEM WITH PACKING MECHANISM" by Rosen N. Diankov and Denys Kanunikov, which is assigned to Mujin, Inc. and identified by attorney docket number 131837-8005.US01, and which is incorporated herein by reference in its entirety.

[0002] This application includes subject matter related to a concurrently filed U.S. patent application entitled "A ROBOTIC SYSTEM WITH DYNAMIC PACKING MECHANISM" by ROSEN N. DIANKOV and DENYS KANUNIKOV, which is assigned to MUJIN, INC. and identified by attorney docket number 131837-8006.US01, and which is incorporated herein by reference in its entirety.

[0003] This application includes subject matter related to a concurrently filed U.S. patent application entitled "ROBOTIC SYSTEM FOR PROCESSING PACKAGES ARRIVING OUT OF SEQUENCE" by ROSEN N. DIANKOV and DENYS KANUNIKOV, which is assigned to MUJIN, INC. and identified by attorney docket number 131837-8008.US01, and which is incorporated herein by reference in its entirety.

[0004] This application includes subject matter related to a concurrently filed U.S. patent application entitled "ROBOTIC SYSTEM FOR PALLETIZING PACKAGES USING REAL-TIME PLACEMENT SIMULATION" by ROSEN N. DIANKOV and DENYS KANUNIKOV, which is assigned to MUJIN, INC. and identified by attorney docket number 131837-8009.US01, and which is incorporated herein by reference in its entirety.

[0005] The present technology is directed generally to robotic systems, and more particularly to systems, processes, and techniques for dynamically packing objects based on identified errors. [Background technology]

[0006] Due to ever-improving performance and decreasing costs, many robots (e.g., machines configured to automatically / autonomously perform physical actions) are now widely used in many fields. Robots can be used to perform various tasks (e.g., manipulating or transposing objects through space), for example, in manufacturing and / or assembly, packing and / or packaging, transportation and / or shipping, etc. In performing tasks, robots can replicate human actions, thereby replacing or reducing human involvement that would otherwise be required to perform dangerous or repetitive tasks.

[0007] However, despite technological advances, robots often lack the sophistication necessary to replicate the human acuity and / or adaptability needed to perform more complex tasks. For example, robots often lack the granularity and flexibility of control in the actions they perform to account for variability, error, or uncertainty that may arise from various real-world factors. Thus, there remains a need for improved techniques and systems for controlling and managing various aspects of robots to complete tasks despite various real-world factors. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram of an exemplary environment in which a robotic system having a dynamic packing mechanism may operate. [Figure 2] FIG. 1 is a block diagram illustrating a robotic system in accordance with one or more embodiments of the present technology. [Figure 3A] FIG. 1 is a diagram of a discretized object in accordance with one or more embodiments of the present technology. [Figure 3B] FIG. 1 is an illustration of a discretized packing table in accordance with one or more embodiments of the present technique; [Figure 4A] FIG. 10 is an illustration of a calculation of support in accordance with one or more embodiments of the present technology. [Figure 4B] FIG. 1 is a diagram of a support metric in accordance with one or more embodiments of the present technology. [Figure 5] 1 is a top view illustrating an exemplary placement performed by a robotic system in accordance with one or more embodiments of the present disclosure. [Figure 6A] 1 is a side view illustrating a first exemplary ingress according to one or more embodiments of the present disclosure. [Figure 6B] 10 is a side view illustrating a second exemplary ingress according to one or more embodiments of the present disclosure. [Figure 7] 2 is a flow diagram for operating the robotic system of FIG. 1 in accordance with one or more embodiments of the present technique. [Figure 8]2 is a flow diagram for operating the robotic system of FIG. 1 in accordance with one or more embodiments of the present technique. DETAILED DESCRIPTION OF THE INVENTION

[0009] Described herein are systems and methods for identifying various packaging errors and dynamically packing objects (e.g., packages and / or boxes). A robotic system (e.g., an integrated system of devices that perform one or more designated tasks) configured in accordance with some embodiments improves packing and storage efficiency by dynamically deriving storage locations for objects and stacking them accordingly.

[0010] Conventional systems use offline packing simulators to predetermine packing sequences / placements. Conventional packing simulators process object information (e.g., case shapes / sizes) for a given or estimated set of cases to generate a packing plan. Once determined, the packing plan specifies and / or requires specific placement positions / poses of objects at a destination (e.g., pallets, bins, cages, boxes, etc.), a predefined sequence of placements, and / or a predetermined motion plan. From the given packing plan, conventional packing simulators can derive source requirements (e.g., sequence and / or placement for objects) that match or enable the packing plan.

[0011] Because packing plans are developed offline in conventional systems, the plans are independent of actual packing operations / situations, object arrivals, and / or other system implementations. Therefore, the overall operation / implementation requires that received packages (e.g., at a start / pickup location) follow a fixed sequence that matches a predetermined packing plan. Therefore, conventional systems cannot adapt to real-time conditions and / or variations in received packages (e.g., different sequences, positions, and / or orientations), unexpected errors (e.g., collisions, losses, and / or different packaging situations), real-time packing requirements (e.g., orders), and / or other real-time factors. Furthermore, because conventional systems group and pack objects according to a strict, predetermined plan / sequence, they require all objects at a source location to (1) have the same expected dimensions / type and / or (2) arrive according to a known sequence. For example, conventional systems require that objects arrive at a pickup location (e.g., via a conveyor) according to a fixed sequence. Also, for example, conventional systems require that objects at a pickup location be placed in a specified position according to a predetermined pose. Thus, conventional systems require one or more operations to align and / or place objects at the source (i.e., prior to the packing operation) according to a predetermined sequence / position. Often, conventional systems require a sequence buffer to align and / or place objects at the source according to a predetermined sequence / pose, which can cost over $1 million.

[0012] In contrast to conventional systems, the robotic systems described herein can (i) identify real-time conditions and / or variations in received packages and / or other unexpected errors, and (ii) dynamically derive object placement locations during system operation (e.g., when one or more objects arrive or are identified and / or after initially initiating one or more operations, such as a packing operation). In some embodiments, the robotic system can initiate / perform dynamic placement derivation based on a trigger event, e.g., identification of one or more packaging / handling errors (e.g., a collision event or a lost event), an unrecognized object (e.g., at the source and / or destination), a change in the position / or orientation of a placed package, and / or the occurrence of other dynamic conditions. In dynamically deriving placement locations, the robotic system can utilize various real-time conditions (e.g., currently existing or ongoing conditions), including, for example, available / arriving objects, object characteristics and / or requirements, placement requirements, and / or other real-time factors.

[0013] The robotic system can derive the placement location based on a discretization mechanism (e.g., a process, circuit, function, and / or routine). For example, the robotic system can use the discretization mechanism to describe the physical size / shape of an object and / or a target location according to a discretization unit (i.e., one discrete area / space). The robotic system can generate a discretized object profile that describes the expected object using the discretization unit and / or a discretized destination profile that describes the target location (e.g., the top surface of a pallet and / or the space / bottom surface within a bin / case / box). Thus, the robotic system can convert continuous real-world spaces / areas into computer-readable digital information. Furthermore, the discretized data can reduce the computational complexity for describing package footprints and comparing various package placements. For example, package dimensions can correspond to integers in the discretization unit rather than real-world decimal numbers, facilitating mathematical calculations.

[0014] In some embodiments, the robotic system can check the discretized cells of the placement platform to determine the object placement probability. For example, the robotic system can use a depth measurement or height of the object placed on the placement platform. The robotic system can determine the depth dimension to determine the height in / accordance with the discretized cell. The robotic system can evaluate the depth dimension according to a group of discretized cells corresponding to the object to be placed. The robotic system can determine the maximum height within the group to evaluate the placement probability. In other words, the robotic system can determine whether the placement location being tested provides sufficient support such that the object to be placed can be placed relatively flat (e.g., according to a predetermined threshold and / or condition). More details regarding the derivation of dynamic placement are provided below.

[0015] Thus, robotic systems can improve the efficiency, speed, and accuracy of dynamically deriving object placement based on real-time conditions. For example, the systems described herein can derive placement locations when real-world conditions impose uncertainties associated with and / or deviations from expected conditions. Furthermore, robotic systems can reduce overall costs by eliminating one or more operations, machines (e.g., sequence buffers), and / or human assistance that would be required in conventional systems to queue or place objects at a source and / or for packing operations (e.g., for error handling). By dynamically deriving placement locations as objects become available (e.g., based on object arrival and / or trigger events), robotic systems eliminate the need to reorganize or sequence packages, along with associated machine / human actions.

[0016] After deriving the object placement, the robotic system can place the object at the derived placement location according to the derived approach plan. In some embodiments, the robotic system can verify the accuracy of the placement of the object at the derived placement location and / or dynamically derive placement locations for one or more other objects (e.g., based on previous placement of the object at the corresponding placement location).

[0017] In the following description, numerous specific details are set forth to provide a thorough understanding of the disclosed technology. In other embodiments, the technology presented herein may be practiced without these specific details. In other instances, well-known features, such as particular functions or routines, have not been described in detail so as not to unnecessarily obscure the present disclosure. References herein to "an embodiment," "one embodiment," or the like mean that the particular feature, structure, material, or characteristic being described is included in at least one embodiment of the present disclosure. Thus, appearances of such phrases herein do not necessarily all refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Furthermore, particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that the various embodiments shown in the figures are for illustrative purposes only and are not necessarily drawn to scale.

[0018] For the sake of clarity, some details describing structure or processing that are well-known and often associated with robotic systems and subsystems, but that may unnecessarily obscure some important aspects of the disclosed technology, are not included in the following description. Also, while the following disclosure describes several embodiments of different aspects of the technology, some other embodiments may have different configurations or different components than those described in this section. Thus, the disclosed technology may have other embodiments that have additional elements and that do not have some of the elements described below.

[0019] Many embodiments or aspects of the present disclosure described below may take the form of computer-executable or processor-executable instructions, including routines executed by a programmable computer or processor. Those skilled in the art will appreciate that the disclosed technology may be practiced on computer or processor systems other than those illustrated and described below. The technology described herein may be implemented in a special-purpose computer or data processor that is specially programmed, configured, or constructed to execute one or more of the computer-executable instructions described below. Thus, the terms "computer" and "processor," as used generally herein, refer to any data processor, including Internet appliances and handheld devices (e.g., palmtop computers, wearable computers, cellular or mobile phones, multiprocessor systems, processor-based or programmable consumer electronics, network computers, minicomputers, etc.). Information manipulated by these computers and processors may be presented on any suitable display medium, such as a liquid crystal display (LCD). Instructions for performing computer-executable or processor-executable tasks may be stored in or on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. The instructions may be contained in any suitable memory device, including, for example, a flash drive and / or other suitable medium.

[0020] The terms “coupled” and “connected,” as well as their derivatives, may be used herein to describe structural relationships between components. It should be understood that these terms are not intended as synonyms for each other. Rather, in particular embodiments, “connected” may be used to indicate that two or more elements are in direct contact with each other. Unless the context clearly dictates otherwise, the term “coupled” may be used to indicate that two or more elements are in direct or indirect contact with each other (through other intervening elements therebetween), or that two or more elements cooperate or interact with each other (e.g., as in a causal relationship due to sending or receiving signals or function calls, etc.), or both.

[0021] favorable environment 1 is a diagram of an example environment in which a robotic system 100 having a dynamic packing mechanism may operate. The robotic system 100 may include and / or communicate with one or more units (e.g., robots) configured to perform one or more tasks. Aspects of the dynamic packing mechanism may be practiced or implemented by various units.

[0022] In the example shown in FIG. 1 , the robotic system 100 may include an unloading unit 102, a transposition unit 104 (e.g., a palletizing robot and / or a piece picker robot), a transport unit 106, a loading unit 108, or a combination thereof, in a warehouse or distribution / shipping location. Each unit in the robotic system 100 may be configured to perform one or more tasks. Tasks may be combined in sequence to perform actions to achieve a goal, such as unloading an object from a truck or van and storing it in a warehouse, or unloading an object from a storage location and preparing it for shipment. In some embodiments, a task may include placing an object at a target location (e.g., on a pallet and / or in a bin / cage / box / case). As described in more detail below, the robotic system 100 may derive individual placement positions / or orientations, calculate corresponding motion plans, or a combination thereof, to place and / or stack the objects. Each unit may be configured to perform a sequence of actions (e.g., operate one or more components therein) to perform the task.

[0023] In some embodiments, a task may include manipulating (e.g., moving and / or reorienting) a target object 112 (e.g., one of a package, box, case, cage, pallet, etc. corresponding to the task being performed) from a start / source location 114 to a task / destination location 116 (e.g., based on information collected by one or more sensors, such as one or more three-dimensional (3D) vision cameras 122). For example, the unloading unit 102 (e.g., a devanning robot) may be configured to transpose the target object 112 from a position in a carrier (e.g., a truck) to a position on a conveyor belt. Also, the transposition unit 104 may be configured to transpose the target object 112 from one position (e.g., a conveyor belt, a pallet, or a bin) to another position (e.g., a pallet, a bin, etc.). In another example, the transposition unit 104 (e.g., a palletizing robot) may be configured to transpose the target object 112 from a source location (e.g., a pallet, a pick-up area, and / or a conveyor) to a destination pallet. In completing an operation, the transport unit 106 may transpose the target object 112 from an area associated with the transposition unit 104 to an area associated with the loading unit 108, and the loading unit 108 may transpose the target object 112 from the transposition unit 104 (e.g., by moving a pallet carrying the target object 112) to a storage location (e.g., a location on a shelf). More details regarding the tasks and associated actions are provided below.

[0024] For illustrative purposes, robotic system 100 is described in the context of a shipping center; however, it should be understood that robotic system 100 can be configured to perform tasks in other environments / purposes, such as for manufacturing, assembly, packaging, healthcare, and / or other types of automation. It should also be understood that robotic system 100 can include other units not shown in FIG. 1 , such as manipulators, service robots, modular robots, etc. For example, in some embodiments, robotic system 100 can include a depalletizing unit for transposing objects from a cage cart or pallet onto a conveyor or other pallet, a container switching unit for transposing objects from one container to another, a packaging unit for packaging objects, a sorting unit for grouping objects according to one or more characteristics thereof, a piece-picking unit for manipulating (e.g., sorting, grouping, and / or transposing) objects differently according to one or more characteristics thereof, or combinations thereof.

[0025] Suitable system 2 is a block diagram illustrating a robotic system 100 in accordance with one or more embodiments of the present technology. In some embodiments, for example, the robotic system 100 (e.g., in one or more of the units and / or robots described above) may include electronic / electrical devices, such as one or more processors 202, one or more storage devices 204, one or more communication devices 206, one or more input / output devices 208, one or more actuation devices 212, one or more transport motors 214, one or more sensors 216, or combinations thereof. The various devices may be coupled to one another via wired and / or wireless connections. For example, the robotic system 100 may include a bus, such as, for example, a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) Standard 1394 bus (also known as "Firewire"). Also, for example, the robotic system 100 may include bridges, adapters, processors, or other signal-related devices to provide wired connections between devices. Wireless connections may be based on, for example, cellular communication protocols (e.g., 3G, 4G, LTE, 5G, etc.), wireless local area network (LAN) protocols (e.g., Wireless Fidelity (Wi-Fi)), peer-to-peer or device-to-device communication protocols (e.g., Bluetooth, Near Field Communication (NFC), etc.), Internet of Things (IoT) protocols (e.g., NB-IoT, LTE-M, etc.), and / or other wireless communication protocols.

[0026] The processor 202 may include a data processor (e.g., a central processing unit (CPU), a special-purpose computer, and / or an on-board server) configured to execute instructions (e.g., software instructions) stored in a storage device 204 (e.g., computer memory). In some embodiments, the processor 202 may be included in an independent / standalone controller operably coupled to other electronic / electrical devices shown in FIG. 2 and / or the robotic unit shown in FIG. 1. The processor 202 may control / interface other devices to implement program instructions that cause the robotic system 100 to perform actions, tasks, and / or operations.

[0027] The storage device 204 may include a non-transitory computer-readable medium on which program instructions (e.g., software) are stored. Some examples of the storage device 204 may include volatile memory (e.g., cache and / or random access memory (RAM)) and / or non-volatile memory (e.g., flash memory and / or magnetic disk drives). Other examples of the storage device 204 may include portable memory and / or cloud storage devices.

[0028] In some embodiments, the storage device 204 can be used to further store and provide access to processing results and / or predetermined data / thresholds. For example, the storage device 204 can store master data 252 including descriptions of objects (e.g., boxes, cases, and / or products) that can be manipulated by the robotic system 100. In one or more embodiments, the master data 252 can include dimensions, shapes (e.g., templates for possible poses and / or computer-generated models for recognizing objects in different poses), color schemes, images, identification information (e.g., barcodes, Quick Response (QR) codes, logos, etc., and / or their expected locations), expected weights, other physical / visual characteristics, or combinations thereof, for the objects expected to be manipulated by the robotic system 100. In some embodiments, the master data 252 can include operation-related information about the objects, such as the center of gravity (CoM) location of each object, expected sensor measurements (e.g., related to force, torque, pressure, and / or contact measurements) corresponding to one or more actions / procedures, or combinations thereof. Also, for example, storage device 204 may store object tracking data 254. In some embodiments, object tracking data 254 may include a log of objects being scanned or manipulated. In some embodiments, object tracking data 254 may include imaging data (e.g., photographs, point clouds, live video feeds, etc.) of objects at one or more locations (e.g., designated pick-up or drop-off locations and / or conveyor belts). In some embodiments, object tracking data 254 may include the position and / or orientation of objects at one or more locations.

[0029] The communications device 206 may include circuitry configured to communicate with external or remote devices over a network. For example, the communications device 206 may include a receiver, a transmitter, a modulator / demodulator (modem), a signal detector, a signal coder / decoder, a connector port, a network card, etc. The communications device 206 may be configured to transmit, receive, and / or process electrical signals according to one or more communications protocols (e.g., Internet Protocol (IP), wireless communications protocols, etc.). In some embodiments, the robotic system 100 may use the communications device 206 to exchange information between units of the robotic system 100 and / or with systems or devices external to the robotic system 100 (e.g., for purposes of reporting, data collection, analysis, and / or troubleshooting).

[0030] The input / output devices 208 may include user interface devices configured to communicate and / or receive information from a human operator. For example, the input / output devices 208 may include a display 210 and / or other output devices (e.g., speakers, haptic circuitry, or tactile feedback devices, etc.) for communicating information to a human operator. The input / output devices 208 may also include control or receiving devices, such as a keyboard, mouse, touchscreen, microphone, user interface (UI) sensors (e.g., a camera for receiving motion commands), wearable input devices, etc. In some embodiments, the robotic system 100 may use the input / output devices 208 to interact with a human operator in performing actions, tasks, operations, or combinations thereof.

[0031] The robotic system 100 may include physical or structural members (e.g., robotic manipulator arms) connected at joints for movement (e.g., rotational and / or translational displacement). The structural members and joints may form a kinematic chain configured to manipulate an end effector (e.g., a gripper) configured to perform one or more tasks (e.g., grasping, rotating, joining, etc.) depending on the application / operation of the robotic system 100. The robotic system 100 may include actuation devices 212 (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or manipulate (e.g., displace and / or reorient) the structural members around or at the corresponding joints. In some embodiments, the robotic system 100 may include a transport motor 214 configured to transport the corresponding unit / chassis from one location to another.

[0032] The robotic system 100 may include sensors 216 configured to acquire information used to perform tasks, for example, to manipulate structural members and / or transport the robotic unit. The sensors 216 may include devices configured to detect or measure one or more physical characteristics of the robotic system 100 (e.g., the state, condition, and / or position of one or more structural members / joints thereof) and / or the surrounding environment. Some examples of the sensors 216 may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

[0033] In some embodiments, for example, the sensors 216 may include one or more imaging devices 222 (e.g., visual and / or infrared cameras, two-dimensional (2D) and / or 3D imaging cameras, distance measuring devices, e.g., lidar or radar, etc.) configured to detect the surrounding environment. The imaging devices 222 may generate a representation of the detected environment, such as a digital image and / or a point cloud, that may be processed via machine / computer vision (e.g., for automated inspection, robotic guidance, or other robotic applications). As described in more detail below, the robotic system 100 (e.g., via the processor 202) may process the digital image and / or point cloud to identify the target object 112 of FIG. 1, the start position 114 of FIG. 1, the task position 116 of FIG. 1, the pose of the target object 112, a confidence metric for the start position 114 and / or pose, or a combination thereof.

[0034] To manipulate the target object 112, the robotic system 100 (e.g., via the various circuits / devices described above) can capture and analyze image data of a designated area (e.g., a pick-up location, such as in a truck or on a conveyor belt) to identify the target object 112 and its start location 114. Similarly, the robotic system 100 can capture and analyze image data of other designated areas (e.g., a drop-off location for placing the object on a conveyor, a location for placing the object in a container, or a location on a pallet for stacking purposes) to identify a task location 116. For example, the imaging device 222 can include one or more cameras configured to generate image data of the pick-up area and / or one or more cameras configured to generate image data of the task area (e.g., a drop-off area). Based on the image data, as described below, the robotic system 100 can determine the start location 114, the task location 116, associated poses, packing / placing locations, and / or other processing results. More information regarding the dynamic packing algorithm is provided below.

[0035] In some embodiments, for example, sensors 216 may include position sensors 224 (e.g., position encoders, potentiometers, etc.) configured to detect the position of structural members (e.g., robotic arms and / or end effectors) and / or corresponding joints of robotic system 100. Robotic system 100 may use position sensors 224 to track the position and / or orientation of structural members and / or joints during the performance of a task.

[0036] Discretized Model 3A and 3B are diagrams of discretized data used to plan and pack objects in accordance with one or more embodiments of the present technology, with Fig. 3A showing a discretized object and Fig. 3B showing a discretized packing table for packing the object.

[0037] In some embodiments, the robotic system 100 of FIG. 1 may include a predetermined discretized model / representation of an expected object stored in the master data 252 of FIG. 2. In some embodiments, the robotic system 100 (e.g., via the processor 202 of FIG. 2) may dynamically generate the discretized model by mapping continuous faces / edges of real-world objects (e.g., packages, pallets, and / or other objects relevant to the task) to discrete counterparts (e.g., unit lengths and / or unit areas). For example, the robotic system 100 may discretize image data (e.g., top-view images and / or point cloud data) of the target object 112 and / or the pallet top surface captured by one or more imaging devices 222 of FIG. 2. In other words, the robotic system 100 may discretize image data of the start position 114 of FIG. 1, a position on the conveyor prior to the start position 114, and / or the task position 116 of FIG. 1. The robotic system 100 may discretize based on identifying the perimeter of the object / pallet in the image data and then dividing the area within the perimeter according to a unit dimension / area. In some embodiments, the unit dimension / area may be scaled or mapped to the image data based on the size and / or position of the object / pallet relative to the imaging device 222 according to a coordinate array and / or a predetermined adjustment factor / formula.

[0038] 3A , some embodiments of the robotic system 100 can use a discretized object model 302 to plan / derive a placement location for an object (e.g., target object 112). The discretized object model 302 (shown by dotted lines) can represent the physical dimensions, shapes, edges, faces, or combinations thereof (shown by dashed lines) of the appearance of an arriving or incoming object (e.g., a package, box, case, etc.) according to discretized units (e.g., unit lengths). The discretized object model 302 can represent expected / known objects and / or unexpected / unknown objects that have been imaged and discretized as described above.

[0039] As shown in FIG. 3B , some embodiments of the robotic system 100 can plan / derive stacking placement of objects using one or more discretized bed models 304 (e.g., discretized representations of the task location 116 in FIG. 1 ). The discretized bed model 304 can represent a placement area 340 (e.g., the physical dimensions, shape, or combination thereof, of the task location 116, such as the top surface of the task location 116, the top surface of a package placed thereon, or a combination thereof) according to a discretized unit. In one or more embodiments, the discretized bed model 304 can represent the real-time status of the placement area 340, such as through real-time updates. For example, with respect to a top view, the discretized bed model 304 can initially represent the top surface of a pallet, the interior bottom surface of a bin or box, etc., that receives and directly contacts the object. As the robotic system 100 places the object, the placement area 340 can change to include the top surface of the placed package (e.g., to stack the package), and the discretized bed model 304 can be updated to reflect the change.

[0040] In some embodiments, the discretized bed model 304 can be based on a top view of one or more standard-sized pallets (e.g., a 1.1 m by 1.1 m pallet). Thus, the discretized bed model 304 can correspond to a pixelated 2D representation of a placement area along a horizontal plane (e.g., an xy plane) according to a grid system utilized by the robotic system 100. In some embodiments, the discretized object model 302 can include a top view (e.g., an xy plane) of an expected or arriving object. Thus, the discretized object model 302 can correspond to a pixelated 2D representation of the object.

[0041] The discretization units used to generate the discretized model may include lengths set by a system operator, a system designer, predetermined inputs / settings, orders, or a combination thereof. In some embodiments, the robotic system 100 may use unit pixels 310 (e.g., polygons, e.g., rectangles, having one or more dimensions according to the discretization units) to describe the area / surface of the target object (e.g., via the discretized object model 302) and the platform / surface (e.g., via the discretized platform model 304). Thus, the robotic system 100 may pixelate the object and platform in 2D along the x- and y-axes. In some embodiments, the size of the unit pixels 310 (e.g., the discretization units) may vary depending on the dimensions of the object and / or the dimensions of the platform. Additionally, the size of the unit pixels 310 may be adjusted (e.g., via preset rules / formulas and / or operator selection) to balance required resources (e.g., computation time, required memory, etc.) with packing accuracy. For example, decreasing the size of the unit pixels 310 may increase computation time and packing accuracy. Therefore, by discretizing the packing task (e.g., target package and packing table) using adjustable unit pixels 310, flexibility for palletizing packages is increased. The robotic system 100 can control the balance between computational resources / time and packing accuracy depending on real-time demands, patterns, and / or environments.

[0042] In some embodiments, the robotic system 100 can ensure that the unit pixels 310 extend beyond the actual perimeter of the object by including instances of the unit pixels 310 that only partially overlap the object for the discretized object model 302. In other embodiments, the robotic system 100 can ensure that the unit pixels 310 in the discretized object model 302 are covered by and / or contained within the actual perimeter of the table surface by excluding overlapping instances of the unit pixels 310 from the discretized table model 304 that exceed the actual dimensions of the table surface.

[0043] As an illustrative example, FIG. 3A shows a first model orientation 332 and a second model orientation 334 of discretized object models representing the target object 112. In some embodiments, the robotic system 100 can rotate one of the discretized models (i.e., the one captured / stored as the first model orientation 332) a predetermined amount along the imaged plane. As shown in FIG. 3A , the robotic system 100 can rotate the discretized object model 302 90 degrees about a vertical axis (an axis extending inward / outward or perpendicular to the plane of the figure) and along a horizontal plane (e.g., a plane represented along the x- and y-axes) to obtain the second model orientation 334. The robotic system 100 can use the different orientations to test / evaluate the corresponding placement of the object.

[0044] Based on the discretized data / representation, the robotic system 100 can dynamically derive a placement location 350 for the target object 112. As shown in FIG. 3B , the robotic system 100 can dynamically derive the placement location 350 even after one or more objects (e.g., shown as the shaded objects in FIG. 3B ) have been placed in the placement area 340. Additionally, the dynamic derivation of the placement location 350 can occur after / while the target object 112 is being unloaded / de-shelfed, registered, scanned, imaged, or a combination thereof. For example, the robotic system 100 can dynamically derive the placement location 350 as the target object 112 is being transported (e.g., via a conveyor), after the imaging device 222 of FIG. 2 generates image data for the target object 112, or a combination thereof.

[0045] Dynamically deriving object placement positions 350 increases flexibility and reduces human labor associated with the shipping / packaging environment. Using discretized, real-time image / depth maps of the objects and pallets (i.e., including the placed objects), the robotic system 100 can test and evaluate various placement positions and / or orientations. Thus, the robotic system 100 can still pack objects without human operator intervention, even when the objects are not recognizable (e.g., in the case of new / unexpected objects and / or computer vision errors), when the object arrival sequence / order is unknown, and / or when unexpected events occur (e.g., lost and / or collision events).

[0046] For illustrative purposes, the placement location 350 is shown in FIG. 3B as being adjacent to the placed object (i.e., placed on the same horizontal layer / at the same height), e.g., directly above / contacting the pallet. However, it should be understood that the placement location 350 may also be above the placed object. In other words, the robotic system 100 can derive a placement location 350 for stacking the target object 112 above and / or on top of one or more objects already on the pallet. As described in more detail below, the robotic system 100 can evaluate the height of the placed object when deriving the placement location 350 to ensure that the object is adequately supported when stacked on top of the placed object.

[0047] In some embodiments, the robotic system 100 can identify object edges 362 when deriving the placement location 350. The object edges 362 can include lines in the image data that represent edges and / or sides of objects that have been placed on the pallet. In some embodiments, the object edges 362 can define the perimeter of one or a group of objects (e.g., a layer of objects) placed on the task location 116 by corresponding to edges that are exposed (e.g., not directly contacting / adjacent to other objects / edges).

[0048] As described in further detail below, the robotic system 100 can derive the placement location 350 according to a set of placement rules, conditions, parameters, requirements, etc. In some embodiments, the robotic system 100 can derive the placement location 350 based on evaluating / testing one or more candidate locations 360. The candidate locations 360 can correspond to the discretized object model 302 overlaid on the discretized base model 304 in various positions and / or orientations. Accordingly, the candidate locations 360 can include placing the target object 112 adjacent to one or more of the object edges 362 and / or stacking the target object 112 on one or more of the placed objects. The robotic system 100 can evaluate each candidate location 360 according to various parameters / conditions, such as support level / condition, support weight (e.g., maximum support weight for packages stacked on top, etc.) versus the fragility rating of the supporting object, space / packing concerns, or a combination thereof. The robotic system 100 can further evaluate the candidate locations 360 using one or more placement rules, such as collision-free requirements, stack stability, customer-specified rules / priorities, package spacing requirements or lack thereof, maximizing total loaded packages, or a combination thereof.

[0049] Real-time surface updates 4A and 4B illustrate various aspects of support calculations and support metrics according to one or more embodiments of the present technology. In some embodiments, as shown in FIG. 4A , the robotic system 100 of FIG. 1 can generate a candidate position 360 of FIG. 3B based on overlaying the discretized object model 302 of FIG. 3A of the target object 112 of FIG. 1 on the discretized mount model 304 of the task location 116 of FIG. 1. Furthermore, the robotic system 100 can iteratively move the discretized object model 302 throughout the discretized mount model 304 when generating the candidate position 360. For example, the robotic system 100 can generate an initial instance of the candidate position 360 by placing the discretized object model 302 corresponding to a predetermined initial position (e.g., a corner) of the discretized mount model 304 according to one or more orientations (e.g., the first model orientation 332 of FIG. 3A and / or the second model orientation 334 of FIG. 3A ). For the next instance of the candidate position 360, the robotic system 100 may move the discretized object model 302 corresponding to the other / next object by a predetermined distance (e.g., one or more unit pixels 310 in FIG. 3B) according to a predetermined direction / pattern.

[0050] If the candidate location 360 overlaps one or more objects already placed at the task location 116, the robotic system 100 can calculate and evaluate the degree of support provided by the placed objects. To calculate and evaluate the degree of support, the robotic system 100 can determine the height / contour for the placement area 340 of FIG. 3B in real time using one or more of the imaging devices 222 of FIG. 2. In some embodiments, the robotic system 100 can use depth measurements (e.g., point cloud values) from one or more of the imaging devices 222 positioned above the task location 116. Because the vertical position of the ground and / or platform (e.g., pallet) surface (e.g., the height of the platform surface above the facility ground) is known, the robotic system 100 can use the depth measurements to calculate the height / contour of the exposed top surface(s) of the platform, the placed object, or a combination thereof. In some embodiments, the robotic system 100 can image the task location 116 and update the height of the exposed top surface(s) in real time, for example, after transposing an object to the platform and / or placing the object on the platform.

[0051] 4A , the robotic system 100 can update the discretized bed model 304 to include a height dimension 402. The robotic system 100 can determine the height dimension 402 according to each discretized pixel (e.g., unit pixel 310) in the discretized bed model 304. For example, the robotic system 100 can determine the height dimension 402 as the maximum height for the surface portion of the mounting area 340 represented by the corresponding unit pixel 310.

[0052] For each candidate location 360 that overlaps one or more of the placed objects, the robotic system 100 can evaluate the placement probability based on the height dimension 402. In some embodiments, the robotic system 100 can evaluate the placement probability based on identifying the maximum overlapping height dimension 402 at each candidate location 360. The robotic system 100 can further identify other height dimensions 402 located at each candidate location 360 that have a height dimension 402 within a difference threshold limit relative to the maximum one of the height dimensions 402. Eligible cells / pixels can represent locations that can provide support for the stacked objects so that the stacked objects are essentially flat / level.

[0053] 4A, for the first of the candidate locations 360 (the upper left corner of the discretization platform model 304), the maximum height dimension may be 0.3 (i.e., a height of 300 millimeters (mm)). With a predetermined difference threshold of 0.02 (e.g., representing 20 mm), the robotic system 100 may identify the top four discretized cells / pixels as meeting the difference threshold. The robotic system 100 may use the identified / qualified cells / pixels to evaluate / express the degree of support.

[0054] 4B shows a further example of support calculation. Figure 4B shows one of the candidate locations 360 from Figure 3 with the discretized object model 302 (shown in solid, bold outline) overlaid on the top left corner of the discretized base model 304. The robotic system 100 can calculate / utilize various support parameters 410, which are parameters used to evaluate the candidate locations 360. For example, the support parameters 410 can include a discretization dimension 412, an overlap area 414, a height difference threshold 416, a support threshold 418, a maximum height 420, a lower height limit 422, an eligible number 424, a set of support area contours 426, a support area size 428, a support ratio 430, a center of gravity (CoM) location 432, or a combination thereof.

[0055] 3A according to the unit pixels 310. For example, the discretized dimensions 412 may include a quantity of unit pixels 310 that form the perimeter of the discretized object model 302. The overlap area 414 may describe the area occupied by the object of interest 112 (e.g., footprint size along a horizontal plane), which may similarly be expressed according to the unit pixels 310. In other words, the overlap area 414 may correspond to the quantity of unit pixels 310 in the discretized object model 302. In the example shown in FIG. 4B, the object of interest 112 may have a discretized dimension 412 of 6 pixels by 7 pixels, which corresponds to an overlap area 414 of 42 pixels.

[0056] The height difference threshold 416 and the support threshold 418 may correspond to limits used to process and / or validate the candidate locations 360. The height difference threshold 416, which may be predetermined and / or adjustable by an operator and / or order, may represent an acceptable deviation from another reference height (e.g., a maximum height 420 corresponding to the largest instance of the height dimension 402 within the overlap area of the discretized object models 302) to contact and / or support a package placed thereon. In other words, the height difference threshold 416 may be used to define a range of heights of surfaces that can contact and / or support a package placed thereon. Thus, relative to the maximum height 420, the lower height limit 422 may correspond to a lower limit of a height within the overlap area 414 that can provide support for a stacked package. In the example shown in FIG. 4B , the height difference threshold 416 may be 0.02. If the maximum height 420 is 0.2, the lower height limit 422 may be 0.18. Therefore, when placing the target object 112 at the candidate location 360, the robotic system 100 can estimate that faces / pixels with a height greater than 0.18 will contact and / or provide support to the target object 112.

[0057] Thus, in one or more embodiments, the robotic system 100 can categorize the unit pixels 310 in the overlap area 414 according to the height difference threshold 416. For example, the robotic system 100 can categorize the unit pixels 310 having heights that satisfy the height difference threshold 416 (i.e., values equal to or greater than the lower height limit 422) as support positions 422 (e.g., a group of unit pixels 310 representing surfaces on which an object can be stacked, represented by the shaded pixels in FIG. 4B). The robotic system 100 can categorize other unit pixels 310 as ineligible positions 444 (e.g., pixels having heights less than the lower height limit 422).

[0058] The support threshold 418 may represent a limit for evaluating the candidate location 360 based on the sufficiency of the support locations 442. For example, the support threshold 418 may be for evaluating a quantity, ratio, area, location, or combination thereof associated with the support locations 442. In some embodiments, the support threshold 418 may be used to determine whether the eligible number 424 for the candidate location 360 (e.g., the sum of the support locations 442) is sufficient to support the target object 112.

[0059] In one or more embodiments, the support threshold 418 can be used to evaluate the support area (e.g., unit pixels 310 that can provide support to an object stacked above, which can be determined by a height threshold) associated with the support location 442. For example, the robotic system 100 can determine the support area contour 426 based on connecting the corners of the outermost / perimeter instances of the support location 442 by extending edges and / or determining lines that span or extend around the ineligible locations 444. In this manner, the support area contour 426 can exclude the ineligible locations 444. Thus, the support area contour 426 can define the perimeter of the support area based on the perimeter instances of the support location 442. Because the support area contour 426 can span and / or include the ineligible locations 444, the support area size 428 (e.g., the number of unit pixels 310 within the support area) can be greater than the eligible number 424. Thus, the support area size 428 effectively represents the spacing between the outermost edges / corners where support is provided. Because wider support is preferred (e.g., when the portion of the support area contour 426 is larger than the object overlap area 414 to reduce protrusion and / or improve stability), the support threshold 418 can correspond to a minimum number of unit pixels 310 within the support area (e.g., for evaluating the support area contour 426), thereby effectively evaluating the spacing between the outermost edges / corners where support is provided.

[0060] In some embodiments, the support threshold 418 can be for evaluating a support ratio 430, which can be calculated based on comparing the eligible number 424 and / or the support area size 428 to the overlap area 414. For example, the support ratio 430 can include a ratio of the eligible number 424 to the overlap area 414 to represent horizontal stability, support weight concentration, or a combination thereof. The support ratio 430 can also include a ratio of the support area size 428 to the overlap area 414 to represent the relative width between support edges / corners under the target object 112.

[0061] Additionally, the robotic system 100 can further evaluate the candidate location 360 based on the CoM location 432 of the target object 112. In some embodiments, the robotic system 100 can access the CoM location 432 of the target object 112 from the master data 252 of FIG. 2 and / or dynamically estimate the CoM location 432 based on grasping and / or lifting the target object 112. Once accessed / estimated, the robotic system 100 can compare the CoM location 432 with the support area contour 426. The robotic system 100 can require the candidate location 360 to contain the CoM location 432 within the support area contour 426 and reject / disqualify any candidate location 360 that does not meet such requirements. In one or more embodiments, the robotic system 100 can calculate and evaluate a placement score based on the separation distance (e.g., along the x-axis and / or y-axis) between the CoM location 432 and the support area contour 426.

[0062] The robotic system 100 can use the support parameters 410 to evaluate the constraints / requirements. For example, the robotic system 100 can eliminate / disqualify candidate locations that do not satisfy the support threshold 418, the CoM position threshold (e.g., the requirement that the CoM position 432 be within the support area contour 426), and / or other stacking rules. The robotic system 100 can also use the support parameters 410 to calculate placement scores for candidate locations 360 (e.g., locations that satisfy the constraints) according to predetermined weights and / or formulas. As described in more detail below, the robotic system 100 can use the calculated placement scores to rank the candidate locations 360 according to predetermined settings (e.g., as reflected by the weights / formulas).

[0063] Object placement operation 5 is a top view illustrating an exemplary placement performed by a robotic system 100 in accordance with one or more embodiments of the present disclosure. In some embodiments, the robotic system 100 can include and / or communicate with a robotic arm 502 (e.g., part of the transposition unit 104 of FIG. 1 , such as a palletizing robot) configured to transpose the target object 112 from the start position 114 and place it at a derived placement location 350 at the task location 116. For example, the robotic system 100 can operate the robotic arm 502 to grasp and pick up the target object 112 from a designated location / portion on a conveyor and place the target object 112 on a pallet.

[0064] The robotic system 100 can dynamically derive the placement location 350 (e.g., when the target object 112 arrives at the facility and / or start location 114 and / or after initially starting one or more operations, such as a packing operation). The robotic system 100 can dynamically derive the placement location 350 based on or taking into account one or more error or uncertainty factors, such as the absence of a packing plan (e.g., a plan to represent the placement location 350 derived for a set of objects, including the target object 112, at the task location 116), an error in the arriving object (e.g., if the object does not match an expected / known object or sequence), or a combination thereof. The robotic system 100 may also dynamically derive the placement location 350 based on or taking into account one or more uncertainties or errors in the destination due to, for example, an unexpected and / or changed placement area 340 (e.g., if access to the task location 116 is partially obstructed, e.g., if a cage or cart track is not fully open), the placed object 508 (e.g., an unrecognized and / or unexpected package on a pallet and / or misalignment of one or more of the placed objects 508), and / or a collision event (e.g., between the robotic arm 502 and the placed object 508).

[0065] In some embodiments, the robotic system 100 can dynamically derive the placement location 350 based on data (e.g., image data and / or measurement data) dynamically collected via one or more of the sensors 216 of FIG. 2 (e.g., the imaging device 222 of FIG. 2). For example, the robotic system 100 can include and / or communicate with a source sensor 504 (e.g., one of the 3D cameras 122 of FIG. 1) positioned above the start location 114 and / or the incoming path (e.g., a conveyor). The robotic system 100 can use the data from the source sensor 504 to generate and / or access the discretized object model 302 of FIG. 3A. In one or more embodiments, the robotic system 100 can use the source sensor 504 to image an object and / or measure one or more dimensions of the object. The robotic system 100 can compare the image and / or measurements to the master data 252 of FIG. 2 to identify the incoming object. Based on the identification, the robotic system 100 can access a discretized object model 302 associated with the object. In one or more embodiments, the robotic system 100 can dynamically generate the discretized object model 302 based on dividing the image / dimension according to unit pixels 310 as described above.

[0066] The robotic system 100 may also include and / or communicate with a destination sensor 506 (e.g., one of the 3D cameras 122 of FIG. 1 ) positioned above the task location 116. The robotic system 100 may use data from the destination sensor 506 to determine and dynamically update the discretized bed model 304 of FIG. 3B . In one or more embodiments, the robotic system 100 may image and / or measure one or more dimensions of the mounting area 340 (e.g., the task location 116, such as a pallet, cage, and / or cart truck). The robotic system 100 may use the images and / or measurements to identify, access, and / or generate the discretized bed model 304 in a manner similar to that described above for the discretized object model 302. Additionally, the robotic system 100 may use data (e.g., a depth map) from the destination sensor 506 to determine the height dimension 402 of FIG. 4A . Thus, the robotic system 100 can use the height dimension 402 to update the placement area 340 and the discretized platform model 304 in real time. For example, the robotic system 100 can update the height dimension 402 in response to the placed object 508, such as after placing the target object 112 at the placement location 350.

[0067] The robotic system 100 can derive an approach path 510 for displacing the target object 112 to the placement location 350 and / or each candidate location 360 in Figure 3B. The approach path 510 can correspond to a motion plan for manipulating / displacing the target object 112 across space from the start location 114 to the corresponding candidate location 360. The approach path 510 can be 3D, extending horizontally and / or vertically.

[0068] When placing the target object 112, the robotic system 100 can verify that the placement of the target object 112 at the derived placement location is stable in order to place the target object 112 according to the metrics described above in FIGS. 3-6 (e.g., if the situational state at the task location 116 changes). After placing the object, the robotic system 100 can check and verify whether the placement of the object at the derived placement location was accurate and / or whether the situational state at the task location 116 has changed. Based at least in part on the verification information, the robotic system 100 can continue to package one or more other objects (e.g., further / subsequent instances of the target object 112) at the task location 116 (e.g., by dynamically deriving placement locations for one or more other objects).

[0069] Approach route evaluation 6A and 6B are side views illustrating an exemplary approach for placing the target object 112 of FIG. 1 in accordance with one or more embodiments of the present disclosure. 6A and 6B illustrate the approach path 510 of FIG. 5 for placing the target object 112 onto one or more of the placed objects 508 on the task location 116 (e.g., a pallet) at the corresponding candidate location 360 of FIG. 3B.

[0070] The robotic system 100 in FIG. 1 can derive an approach path 510 based on approach units 602, illustrated by dashed boxes F-1 through F-5. The approach units 602 may include sequential positions of the target object 112 in 3D space along the corresponding approach path 510. In other words, the approach units 602 may correspond to sample positions of the target object 112 for following the corresponding approach path 510. The approach units 602 may be positioned according to path segments 604 of the corresponding approach path 510. The path segments 604 may correspond to straight line segments / directions in the approach path 510. The path segments 604 may include a final segment 606 for placing the target object 112 at the corresponding candidate position 360. The final segment 606 may include a vertical (e.g., downward) direction.

[0071] To derive the approach path 510, the robotic system 100 can identify any of the placed objects 508 that may be obstacles 610 (e.g., potential obstacles when placing the target object 112 at the candidate location 360). In one or more embodiments, the robotic system 100 can identify the potential obstacle(s) 610 as instance(s) of the placed object 508 that overlap with a horizontal line 611 (e.g., a straight line along the x-y plane) connecting the start location 114 and the corresponding candidate location 360. The robotic system 100 can further identify the potential obstacle(s) 610 as instance(s) of the placed object 508 that overlap with a derived lane 613 near the horizontal line 611, for example, based on deriving a lane that is parallel to and overlaps the horizontal line, the lane having a width based on one or more dimensions (e.g., width, length, and / or height) of the target object 112. 6A and 6B, the starting location 114 may be to the right of the candidate location 360. Thus, the robotic system 100 may identify the placed object on the right as a potential obstacle 610.

[0072] 4A . For example, the robotic system 100 may verify / identify as a potential obstacle 610 a potential obstacle 610 having one or more of the height dimensions 402 equal to or greater than the candidate location 360. The robotic system 100 may eliminate a placed object 508 having a height dimension 402 less than the candidate location 360. In one or more embodiments, the robotic system 100 may identify / eliminate a potential obstacle 610 based on ambiguity associated with the height of the candidate location 360 and / or the height of the potential obstacle 610.

[0073] In some embodiments, the robotic system 100 can derive the entry path 510 in reverse order, e.g., starting from the candidate position 360 and ending at the starting position 114 in FIG. 5 . Thus, the robotic system 100 can derive the final segment 606 first (e.g., before the other segments) to avoid the potential obstacle 610. For example, the robotic system 100 can determine the entry unit 602 based on iteratively increasing the height of the entry unit 602 by a predetermined distance (e.g., first "F-1," then "F-2," etc.). For each iteration, the robotic system 100 can calculate and analyze the vector 612 between the determined entry unit 602 (e.g., bottom surface / its edge) and the potential obstacle 610 (e.g., top surface / its edge). The robotic system 100 may continue to increase the height of the entry unit 602 until the vector 612 indicates that the determined entry unit 602 is above the potential obstacle 610 and / or has jumped over the potential obstacle 610 by a gap threshold 614 (e.g., a requirement for a minimum vertical spacing for the target object 112 above the highest point of the potential obstacle 610 to avoid contact or collision between the target object 112 and the potential obstacle 610). If the determined entry unit meets the gap threshold 614, or for the next iteration, the robotic system 100 may adjust the corresponding entry unit 602 by a predetermined distance along the horizontal direction (e.g., toward the start position 114). Thus, the robotic system 100 may derive the final segment 606 and / or subsequent path segments 604 to derive the entry path 510 based on the candidate position 360 and the entry units 602 that met the gap threshold 614.

[0074] Once derived, the robotic system 100 can use the approach path 510 to evaluate the corresponding candidate location 360. In some embodiments, the robotic system 100 can calculate a placement score according to the approach path 510. For example, the robotic system 100 can calculate a placement score according to a setting for a shorter length / distance for the final / vertical segment 606 (e.g., according to one or more weights corresponding to a predetermined placement setting). Thus, when comparing the approach paths 510 of FIGS. 6A and 6B , the robotic system 100 can prioritize the path shown in FIG. 6B , in which the length of the final / vertical segment 606 is shorter. In one or more embodiments, the robotic system 100 can include constraints, such as upper limits, associated with the approach path 510 (e.g., for the final / vertical segment 606) that are used to eliminate or disqualify the candidate location 360.

[0075] In some embodiments, the robotic system 100 may further evaluate the corresponding candidate locations 360 according to other collision / obstacle-related parameters. For example, the robotic system 100 may evaluate the candidate locations 360 according to the horizontal spacing 616 between the candidate locations 360 and one or more of the placed objects 508. Each horizontal spacing 616 may be the distance (e.g., the shortest distance) along the horizontal direction (e.g., the x-y plane) between the corresponding candidate location 360 and an adjacent instance of the placed object 508. The robotic system 100 may calculate a placement score for the candidate locations 360 based on the horizontal spacing 616, similar to that described above for the entry path 510. The robotic system 100 may also eliminate or disqualify the candidate locations 360 based on the horizontal spacing 616, such as if the horizontal spacing 616 does not meet a minimum requirement. Details regarding the calculation of the placement score and / or constraints for eliminating the candidate locations 360 are described below.

[0076] Operation flow 7 is a flow diagram for a method 700 of operating the robotic system 100 of FIG. 1 in accordance with one or more embodiments of the present technology. The method 700 may be for placing packages (e.g., cases and / or boxes) on a platform (e.g., a pallet) and / or for generating a 2D / 3D packing plan for placing the packages accordingly. The method 700 may be performed based on execution by one or more of the processors 202 of FIG. 2 of instructions stored in one or more of the storage devices 204 of FIG. 2.

[0077] In block 702, the robotic system 100 may identify a package set (e.g., available packages) and a destination (e.g., task location 116 in FIG. 1 , such as a pallet and / or container for receiving the packages). For example, the robotic system 100 may identify a package set representing available packages, including packages available for packing, located at a source, designated for placement, and / or listed in an order / request / inventory. The robotic system 100 also identifies the size or dimensions of the area of the task location 116 where the packages can be placed (e.g., the top loading surface of a pallet, such as placement pallet 340 in FIG. 3 ). In some embodiments, the robotic system 100 may identify the size, dimensions, type, or a combination thereof, of the pallet.

[0078] At block 704, the robotic system 100 may generate and / or access a discretized model (e.g., the discretized object model 302 of FIG. 3A and / or the discretized platform model 304 of FIG. 3B ) corresponding to the package set and / or task location 116 representing the available packages. In some embodiments, the robotic system 100 may generate the discretized model (e.g., in real time, i.e., after receiving an order and / or before starting a packing operation, or offline) based on dividing up the physical dimensions of the object and / or platform area (e.g., the pallet top surface according to the unit pixel 310 of FIG. 3B ). The unit pixel 310 may be predetermined (e.g., by the manufacturer, the ordering customer, and / or the operator), such as, for example, 1 millimeter (mm) or 1 / 16 inch (in) or more (e.g., 5 mm or 20 mm).

[0079] In some embodiments, the robotic system 100 can access discretized models stored in the storage device 204 and / or other devices (e.g., a package supplier's storage device, database, and / or server accessed via the communication device 206 of FIG. 2). The robotic system 100 can access a predetermined discretized model representing an available package and / or task location 116. For example, the robotic system 100 can access the discretized object model 302 corresponding to an available package by searching for the available package and its corresponding model in the master data 252 of FIG. 2 (e.g., a predetermined table or lookup table). Similarly, the robotic system 100 can access the discretized platform model 304 representing an identified platform, such as a pallet, on which the available package will be placed.

[0080] At block 706, the robotic system 100 may determine package groups (e.g., subgroups of available packages). The robotic system 100 may determine package groups based on the available packages for placement on the identified platform (e.g., loading pallet 340). The robotic system 100 may determine package groups according to similarities and / or patterns in one or more characteristics of the available packages. In some embodiments, as shown in block 721, the robotic system 100 may determine package groups by grouping the available packages according to grouping criteria / requirements. Some examples of grouping criteria / requirements may include package priority (e.g., as specified by one or more customers), fragility rating (e.g., maximum weight that can be supported by a package), weight, package dimensions (e.g., package height), package type, or a combination thereof. In grouping the available packages, the robotic system 100 may search the master data 252 for various characteristics of the available packages that match the grouping criteria / requirements.

[0081] At block 708, the robotic system 100 may calculate a processing order (e.g., a sequence in which placement locations are considered / derived) for available packages and / or groups thereof (i.e., package groups). In some embodiments, as shown in block 722, the robotic system 100 may calculate the processing order according to one or more ordering conditions / requirements. For example, the robotic system 100 may prioritize placement plans for package groups according to the number of packages in each group, e.g., to process package groups with a larger number of packages earlier in the placement plan. As another example, the robotic system 100 may prioritize placement plans for package groups according to the fill rate of each group, e.g., to process package groups with fewer larger packages earlier in the placement plan than package groups with more smaller packages. In some embodiments, ordering conditions may overlap with grouping conditions, e.g., with respect to weight range, fragility rating, etc. For example, the robotic system 100 may prioritize processing of heavier and / or less fragile packages to process them earlier and / or place them on a lower layer.

[0082] In some embodiments, the robotic system 100 may prioritize placement plans according to total horizontal area. The robotic system 100 may use information specified in the master data 252 to calculate or access the surface area of the top faces of the packages in a group (e.g., by multiplying corresponding widths and lengths). In calculating the total horizontal area, the robotic system 100 may add up the surface areas of packages that are the same type and / or have heights within a threshold range. In some embodiments, the robotic system 100 may prioritize placement plans for groups with larger total horizontal areas to be processed faster and / or placed on a lower layer.

[0083] For one or more embodiments, the robotic system 100 can load identifiers and / or quantities of available packages into a buffer. The robotic system 100 can arrange the identifiers in the buffer according to group. Additionally, the robotic system 100 can arrange the identifiers in the buffer according to processing order. Thus, the arranged values in the buffer can correspond to available packages and / or remaining packages.

[0084] As shown in block 724, for example, robotic system 100 may calculate a processing order for an initial set of available packages (e.g., a package set) before implementing the corresponding stacking plan, e.g., before any of the packages in the package set are placed on the pedestal. In some embodiments, as shown in block 726, robotic system 100 may calculate a processing order for a remaining set of available packages after initiation or during implementation of the corresponding stacking plan. For example, as shown by the feedback loop from block 716, robotic system 100 may calculate a processing order for the remaining set (e.g., a portion of available packages that have not been transposed to the pedestal and / or remain at the source location) according to one or more trigger conditions. Exemplary trigger conditions may include a stacking error (e.g., a lost or dropped package), a collision event, a predetermined retrigger timing, or a combination thereof.

[0085] In block 710, the robotic system 100 may generate a 2D plan for placing the available packages along a horizontal plane. For example, the robotic system 100 may generate a placement plan to represent a 2D mapping of the available packages along a horizontal plane. The robotic system 100 may generate two or more placement plans based on the discretized model. For example, the robotic system 100 may generate a placement plan based on comparing the discretized object model 302 with the discretized bed model 304. The robotic system 100 may determine different placements / arrangements of the discretized object model 302, overlay / compare it with the discretized bed model 304, and, if overlaid, verify / preserve placement within the boundaries of the discretized bed model 304. The robotic system 100 may assign another layer (e.g., another instance of a placement plan) to packages that cannot be placed within the boundaries of the discretized bed model 304. Thus, the robotic system 100 can iteratively derive placement positions for the placement plan representing a 2D layer of the stacking plan until each package in the package set has been assigned a position in the placement plan.

[0086] In some embodiments, the robotic system 100 can generate a placement plan based on package groups. For example, the robotic system 100 can determine the placement of packages in one package group before considering the placement of packages in other groups. If the packages in a package group overflow a layer (i.e., if the packages do not fit into one layer or one instance of the discretized bed model 304) and / or after placing all the packages in one group, the robotic system 100 can assign locations for the packages in the next group to any remaining / unoccupied area in the discretized bed model 304. The robotic system 100 can iteratively repeat the assignment until there are no unassigned packages that do not fit into the remaining space in the discretized bed model 304.

[0087] Similarly, the robotic system 100 can generate a placement plan based on the processing order (e.g., based on package groups according to the processing order). For example, the robotic system 100 can determine a test placement based on assigning packages and / or groups according to the processing order. The robotic system 100 can assign the earliest sequenced package / group the first placement of the test placement, and then test / assign subsequent packages / groups according to the processing order. In some embodiments, the robotic system 100 can maintain a processing order for packages / groups across layers (e.g., across instances of the placement plan). In some embodiments, the robotic system 100 can recalculate and update the processing order after each layer is filled (illustrated by the dashed feedback line in FIG. 7).

[0088] In some embodiments, as an illustrative example of the above process, the robotic system 100 may generate a 2D plan by identifying different package types within a package set. In other words, at block 732, the robotic system 100 may identify unique packages (e.g., represented by package type) within each package group and / or package set.

[0089] At block 734, the robotic system 100 may (e.g., iteratively) derive a placement location for each available package. At block 736, the robotic system 100 may determine an initial placement location for the first unique package in the sequence according to the processing order. The robotic system 100 may determine the initial placement location according to a predetermined pattern, as described above. In some embodiments, the robotic system 100 may calculate an initial placement for each unique package. Each resulting initial placement may be developed into a unique placement combination (e.g., an instance of a search tree), for example, by tracking the placement plan 350 over the iterations. At block 738, the robotic system 100 may derive and track candidate placement locations for subsequent packages according to the processing order and / or the remaining packages described above. Thus, the robotic system 100 may iteratively derive placement combinations.

[0090] In deriving placement combinations (e.g., candidate placement locations), the robotic system 100 may test / evaluate positions of the discretized object model 302 of the corresponding package based on iteratively deriving and evaluating candidate stacking scenarios (e.g., possible combinations of unique placement locations for available packages). Each candidate stacking scenario may be derived based on identifying unique possible locations for the package according to the sequence described above (e.g., according to a predetermined sequence / rule for placement locations). The candidate stacking scenarios and / or unique placement locations may be evaluated according to one or more placement criteria (e.g., requirements, constraints, placement costs, and / or heuristic scores). For example, the placement criteria may require that the discretized object model 302, when placed in the selected location, fit entirely within the horizontal boundaries of the discretized platform model 304. The placement criteria may also require that the placement of the discretized object model 302 be within or beyond a threshold distance relative to an initial placement location and / or a previous placement location (e.g., along a horizontal direction, etc.) for adjacent placement or separation requirements, etc. Other examples of placement criteria may include a preference for adjacently placing packages with a minimum difference(s) in one or more package dimensions (e.g., height), fragility rating, package weight range, or combinations thereof. In some embodiments, the placement criteria may include a collision probability that may correspond to the location and / or characteristics (e.g., height) of an assigned package within a layer relative to a reference location (e.g., the location of a palletizing robot). Thus, the robotic system 100 may generate multiple unique placement combinations of package placement locations (i.e., candidate placement plans for each layer and / or candidate stacking scenarios each including multiple layers). In some embodiments, the robotic system 100 may track the placement of the combinations based on generating and updating a search tree over placement iterations.

[0091] In block 740, the robotic system 100 can calculate / update a placement score for each combination / package placement. The robotic system 100 can calculate the placement score according to one or more of the placement conditions / settings (e.g., package dimensions, collision probability, fragility rating, package weight range, spacing requirements, package quantity requirements). For example, the robotic system 100 can use preference factors (e.g., multiplier weights) and / or formulas to describe preferences for package-to-package spacing distances, package dimension / fragility rating / package weight differences for adjacent packages, collision probability, consecutive / adjacent surfaces at the same height, their statistical results (e.g., average, maximum, minimum, standard deviation, etc.), or combinations thereof. Each combination can be scored according to preference factors and / or formulas that can be predefined by the system manufacturer, order, and / or system operator. In some embodiments, the robotic system 100 can calculate the placement score at the end of an entire placement iteration.

[0092] In some embodiments, the robotic system 100 can update the sequence of placement combinations in the priority queue after each placement iteration. The robotic system 100 can update the sequence based on the placement score.

[0093] The robotic system 100 can stop a placement iteration based on determining an empty source status, a full layer status, or an invariant score status, for example, when one candidate placement plan is completed. An empty source status can indicate that all available packages have been placed. A full layer status can indicate that no other packages can be placed within the remaining area of the discretized bed model 304 under consideration. An invariant score status can indicate that the placement score for the combination remains constant over one or more consecutive placement iterations. In some embodiments, the robotic system 100 can derive other instances of a candidate stacking scenario by repeating the placement iteration using different initial placement positions and / or different processing orders (e.g., to reorder groups with identical array values / scores associated with the array conditions). In other words, the robotic system 100 can generate multiple 2D placement plans, where each 2D placement plan can represent a layer in a 3D stack (e.g., an instance of a candidate stacking scenario). In other embodiments, the robotic system 100 can iteratively consider 3D effects when a 2D placement plan is derived, and when the 2D placement plan is full, begin deriving the next layer as the next iteration.

[0094] At block 712, the robotic system 100 may generate a stacking plan. In some embodiments, the robotic system 100 may initiate generation of a stacking plan when the placement location of the package to be processed overlaps with one or more placed / processed packages.

[0095] In generating a stacking plan and / or assessing a 2D plan, the robotic system 100 can convert each of the placement combinations and / or placement plans into a 3D state, as shown in block 752. For example, the robotic system 100 can assign height values for the packages to the placement combination. In other words, the robotic system 100 can generate a contour map (depth map estimate) based on adding the package heights to the placement combination.

[0096] The 3D state enables the robotic system 100 to evaluate placement combinations according to one or more stacking rules (e.g., horizontal offset rules, support spacing rules, and / or vertical offset rules). As an illustrative example, if a package to be placed is stacked on top of / over one or more processed packages, the robotic system 100 can eliminate any placement combinations that violate an overlap requirement, an overhang requirement, a vertical offset rule, a CoM offset requirement, or a combination thereof. In one or more embodiments, the robotic system 100 can eliminate any placement combinations that violate the fragility rating of one or more packages below the processed package, for example, by estimating the support weight of the overlapping packages and comparing it with the corresponding fragility rating.

[0097] For the remaining placement combinations, the robotic system 100 can calculate or update a 3D placement score, for example, as shown in block 754. The robotic system 100 can use predetermined settings (e.g., weights and / or formulas) related to placement costs and / or heuristic values for 3D placement. The predetermined 3D settings can be similar to 2D settings, group settings, arrangement conditions, or a combination thereof. For example, the 3D settings can be configured to calculate a collision probability based on the 3D state and calculate a score that prioritizes placement combinations with a lower collision probability. The robotic system 100 can also calculate the score based on the remaining packages, the size of the support area having a common height, the number of packed items in the 3D state, the height difference of the processed packages, or a combination thereof. In some embodiments, the robotic system 100 can update the sequence of the placement combinations in the priority queue according to the scores.

[0098] After the 3D state is processed, the robotic system 100 may update the 2D plan by deriving a placement for the next package in the remaining packages, such as at block 710. The robotic system 100 may iterate the above process until a stopping condition, such as when all available packages have been processed (i.e., an empty value / set for the remaining packages) and / or when the placement combination cannot be improved (also referred to as an unimproved combination). Some examples of an unimproved combination include when the currently processed placement eliminates the last of the placement combinations in the priority queue due to one or more of the violations and / or when the placement score remains constant for a threshold number of iterations for a prioritized combination.

[0099] If a stop condition is detected, for example, in block 756, the robotic system 100 can select one of the derived placement combinations according to the placement scores (e.g., 2D and / or 3D association scores), and thus, the robotic system 100 can designate the selected placement combination as a stacking plan (e.g., a set of placement plans).

[0100] In some embodiments, as an illustrative example, the robotic system 100 can implement the functions of blocks 710 and 712 differently. For example, in block 710, the robotic system 100 can generate a 2D plan (e.g., an instance of the placement plan 350) for the bottom layer as described above. In doing so, the robotic system 100 can be configured to provide heavier settings (e.g., higher parameter weights) for matching package heights, heavier package weights, and / or larger supportable weights for packages when considering placement and / or processing orders. The robotic system 100 can derive a first 2D plan for the base layer as described above for block 710.

[0101] Once the first 2D layer is complete / filled as described above to form the base layer, the robotic system 100 can convert the placement plan to a 3D state as described for blocks 712 / 752. Using the 3D information, the robotic system 100 can identify one or more planar sections / areas of the base layer (e.g., placement surfaces 352-356 in FIG. 3B ) as described above. Using the planar sections, the robotic system 100 can iteratively / recursively derive package placements for the next layer above the base layer. The robotic system 100 can treat each planar section as a new instance of the discretized bed model 304 and test / evaluate different placements as described above for block 710. In some embodiments, the robotic system 100 can derive the 2D placement using the placement surfaces but can calculate a score across the entire placement pallet 340. Thus, the robotic system 100 can be configured to prioritize larger placement areas for subsequent layers without being limited by the preceding placement areas.

[0102] Once the iterative placement process stops for the second layer, the robotic system 100 can calculate a planar section (e.g., a top surface having a height within a threshold range) for the derived layer to generate a 2D placement of the remaining packages / groups for the next layer up. The iterative layering process can continue until a stopping condition is met, as described above.

[0103] In some embodiments, the robotic system 100 can independently generate the 2D plans (e.g., two or more of the mounting plans) in block 712. The robotic system 100 can generate a stacking plan based on vertically combining the 2D plans (e.g., arranging / stacking the 2D mounting plans along the vertical direction).

[0104] At block 714, the robotic system 100 may calculate a packing sequence (e.g., stacking sequence 530 in FIG. 5B ) based on the stacking plan. As an example, the packing sequence may be for specifying a placement order for available packages. In some embodiments, the robotic system 100 may calculate the packing sequence layer by layer, as shown in block 762. In other words, the robotic system 100 may calculate a packing sequence for each layer and then connect the sequences from bottom to top according to the order / position of the layers. When calculating the packing sequence, in some embodiments, the robotic system 100 may adjust the placement plan, as shown in block 772. For example, the robotic system 100 may adjust the placement plan by reallocating one or more of the packages (e.g., packages with heights that increase the collision probability for subsequent operations / transpositions) from a placement plan in a lower layer to a placement plan in a higher layer. Any packages supported by the reallocated package may also be reallocated to a higher layer. In other words, the reassigned package may remain in the same horizontal placement and be associated with a higher layer so that the package can be placed later as shown in Figure 5B. In block 774, the robotic system 100 may calculate a packing sequence (e.g., stacking sequence 530) based on the adjusted placement plan, for example, by packing / manipulating objects assigned to a higher layer after objects assigned to a lower layer.

[0105] In other embodiments, the robotic system 100 may calculate the packing sequence independently of the layer assignments, as shown in block 764. In other words, the robotic system 100 may calculate the packing sequence such that packages assigned to lower layers are placed after packages assigned to higher layers.

[0106] In calculating the packing sequence, both within and between layers, the robotic system 100 can analyze the location of packages within the stacking plan according to one or more package dimensions (e.g., height), relative placement location, or a combination thereof. For example, the robotic system 100 can sequence the placement of boxes farther from a unit / reference location (e.g., the location of a palletizing robot) before packages assigned closer. Additionally, the robotic system 100 can place taller / heavier packages earlier if their assigned locations are along the perimeter of the placement plan and away from the unit location.

[0107] In block 716, the robotic system 100 may implement the stacking plan to place the available packages on the platform. The robotic system 100 may implement the stacking plan based on transmitting one or more motion plans, actuator commands / settings, or a combination thereof to corresponding devices / units (e.g., the transposition unit 104 in FIG. 1 , the actuation device 212 in FIG. 2 , the sensor 216 in FIG. 2 , etc.) according to the stacking plan. The robotic system 100 may further implement the stacking plan based on executing the transmitted information in the devices / units to transpose the available packages from the source locations to the destination platform. Thus, the robotic system 100 may place the available packages according to 3D mapping, where one or more of the available packages are placed / stacked on top of other packages, for example, the available packages are placed layer by layer. Furthermore, the robotic system 100 may manipulate / transpose the packages according to a packing sequence. Thus, the robotic system 100 may place the packages layer by layer, as described above, or without such constraints.

[0108] FIG. 8 is a flowchart of a method 800 for operating the robotic system 100 of FIG. 1 in accordance with one or more embodiments of the present technology. The method 800 can be used to detect errors and potential collisions and determine appropriate responses. For example, the method 800 can be used to dynamically derive the placement location 350 of FIG. 3B for the target object 112 of FIG. 3 at the task location 116 of FIG. 1 and / or to adjust an existing packing plan (e.g., details regarding the specific placement positions / poses of the set of objects at the destination, the associated sequence and / or motion plan, or a combination thereof) due to a detected error. As described in more detail below, possible errors include differences or discrepancies between the expected packaging situation and the real-time packaging situation. In other words, the errors can correspond to unexpected events (e.g., collisions, dropped / lost objects, misalignment of placed objects, and / or occlusions at the task location 116) that may occur while placing the objects at the task location 116 according to the packing plan. In these and other embodiments, the expected packaging situation may be based at least in part on changes in the packaging situation (e.g., according to a packing plan) before, during, or after placing the target object and / or placed object 508. In these and still other embodiments, the method 800 may be implemented by execution by one or more processors 202 of FIG. 2 of instructions stored in one or more storage devices 204 of FIG.

[0109] As noted above, method 700 of FIG. 7 can be implemented to derive and / or implement a packing plan as described in more detail in a co-filed U.S. patent application entitled "A ROBOTIC SYSTEM WITH PACKING MECHANISM" by Rosen N. Kiankov and Denys Kanunikov, which is assigned to Mujin, Inc. and identified by attorney docket number 131837-8005.US01, and which is incorporated herein by reference in its entirety.

[0110] After the 3D stacking plan and / or packing sequence are generated, the robotic system 100 can begin executing the packing plan, as shown in block 801. To execute the packing plan, the robotic system 100 can operate one or more of the robotic units shown in FIG. 1 , such as the unloading unit 102, the transposition unit 104, the transport unit 106, the loading unit 108, etc. (e.g., by generating, sending, and implementing corresponding commands, settings, motion plans, etc.). The robotic system 100 can operate the robotic units to transport the object to the start location 114 of FIG. 1 and manipulate the object from the start location 114 (e.g., via the transposition unit 104) to place it at / on a task location 116 of FIG. 1 (e.g., a pallet, cage, cart truck, etc.) according to the 3D stacking plan. For example, the robotic system 100 can transport objects to the start location 114 according to the packing sequence (e.g., via a transport unit 106, such as a conveyor) and place the objects in positions on the placement area 340 of Figure 3B according to the 3D packing plan. The robotic system 100 can further track progress as it executes the packing plan to identify objects in the packing sequence and / or the 3D packing plan that have been placed in their assigned positions.

[0111] At block 802, the robotic system 100 can determine a real-time packaging situation. While executing the packing plan, the robotic system 100 can determine a real-time packaging situation at or near (e.g., within a predetermined distance of) the start location 114 and / or the task location 116. For example, the robotic system 100 can receive and analyze information (e.g., sensor data from the sensors 216 in FIG. 2 ) in real time regarding each of the incoming objects and / or objects at the task location 116. In some embodiments, the robotic system 100 can receive and analyze source sensor data (e.g., from the source sensors 504 in FIG. 5 ) representing one or more objects, including the target object 112, at or approaching the start location 114. In these and other embodiments, the robotic system 100 can receive and analyze destination sensor data (e.g., from the destination sensors 506 in FIG. 5 ) representing a placement area (e.g., placement area 340) associated with the task location 116 and / or the placed object 508 in FIG. 5 thereon.

[0112] In some embodiments, the robotic system 100 can analyze sensor data. In analyzing the sensor data, the robotic system 100 can process the sensor data (e.g., images and / or depth maps from the sensors 216) to identify / estimate edges. For example, the robotic system 100 can recognize edges of the target object 112, the task location 116, the placed object 508, or a combination thereof by processing the sensor data using a Sobel filter, or the like. The robotic system 100 can use the edges to identify areas representing separate objects and / or their dimensions. In these and other embodiments, the robotic system 100 can estimate one or more dimensions or lengths of the detected objects (e.g., of the incoming object, the target object 112, the pallet, the cage, etc.) based on the sensor data (e.g., the source sensor data). The robotic system 100 can further use the identified areas to identify the pose and / or position of the objects. For example, the robotic system 100 can map edges to an existing grid system to determine the orientation and / or position of the objects.

[0113] At block 804, the robotic system 100 may access and / or generate a discretized model (e.g., the discretized object model 302 of FIG. 3A and / or the discretized bed model 304 of FIG. 3B ) representing the incoming package (including, e.g., the target object 112) and / or the task location 116, e.g., a pallet and / or a cage. The robotic system 100 may determine (e.g., generate and / or access) the discretized model (e.g., the discretized object model 302 and / or the discretized bed model 304) based on real-time sensor data (e.g., source sensor data and / or destination sensor data). In some embodiments, the robotic system 100 may identify the object type (e.g., the identity or category of the incoming object) of the target object 112 based on the source sensor data. The robotic system 100 can find and access a matching discretized model by using the identification information (e.g., surface image and / or estimated dimensions) to search the master data 252 stored in the storage device of FIG. 2 and / or other devices (e.g., a package supplier's storage device, database, and / or server accessed via the communication device 206 of FIG. 2).

[0114] In some embodiments, the robotic system 100 can dynamically generate a discretized model of the target object in real time, e.g., immediately in response to receiving source sensor data. To dynamically generate the discretized model, the robotic system 100 can divide the sensor data and / or corresponding physical dimensions (e.g., of an incoming object, a pallet top, etc.) according to the unit pixels 310 of FIG. 3B . In other words, the robotic system 100 can generate the discretized model based on overlaying the unit pixels 310 on an area representing the target object 112 and / or task location 116 according to the corresponding sensor data. The unit pixels 310 can be predetermined (e.g., by the manufacturer, the ordering customer, and / or the operator), such as, for example, 1 mm or 1 / 16 inch (in) or more (e.g., 5 mm or 20 mm). In some embodiments, the unit pixels 310 can be based on the dimensions or size of one or more of the packages and / or platforms (e.g., a percentage or fraction).

[0115] In block 806, the robotic system 100 may detect or determine whether one or more errors have occurred. For example, the robotic system 100 may determine whether an error has occurred by identifying a difference or discrepancy between an expected packaging situation and a real-time packaging situation. In other words, the robotic system 100 may determine whether an error has occurred by comparing the sensor data with an expected state of the start location 114 and / or an expected state of the task location 116. Examples of possible errors identified by the robotic system 100 include a source matching error (e.g., master data error, unexpected object error, arrival sequence error, etc.), a destination matching error (e.g., placement accessibility error, unexpected placement error, placement area error, etc.), and / or an operation status error (e.g., collision error, transportation error, object misalignment error, etc.).

[0116] At block 832, the robotic system 100 may identify a source matching error. To identify a source matching error, the robotic system 100 may compare the source sensor data to data corresponding to the expected packaging situation at the start location 114. In one embodiment, the robotic system 100 may detect a master data error by comparing the source sensor data to master data (e.g., master data 252 in FIG. 2 ) that includes descriptions, such as characteristic information, of potential objects pre-registered in the robotic system 100. In these embodiments, the robotic system 100 may determine whether the characteristics (e.g., physical characteristics such as height, width, length, weight, and / or other characteristics) of the target object 112 captured in the source sensor data match the object characteristic information stored in the master data 252. If the characteristics of the target object 112 captured in the source sensor data do not match the object characteristic information stored in the master data 252, the robotic system 100 may determine that a master data error has occurred. Thus, the robotic system 100 may identify a source matching error associated with the arrival of an unregistered and / or unrecognized object.

[0117] In these and other embodiments, the robotic system 100 can detect unexpected object errors and / or arrival sequence errors by comparing source sensor data with data corresponding to the packing plan and / or packing sequence. Continuing with this example, the robotic system 100 can compare derived characteristics (e.g., physical characteristics) of the target object 112 with expected characteristics of an object scheduled to arrive at the start location 114 according to the packing sequence and tracked progress. If the characteristics of the target object 112 match one of the objects registered in the master data 252, the robotic system 100 can compare the identifier / type of the matched object with those of the expected object according to the tracked progress of the packing sequence. If the target object 112 does not match the expected object, the robotic system can determine that an unexpected object error has occurred (e.g., the target object 112 is not the expected object and / or arrived out of sequence). In some embodiments, the robotic system 100 may then compare the target object 112 (e.g., its characteristics and / or matching identifiers) to one or more other objects that are scheduled to arrive at the start location 114 according to the packing sequence (e.g., after and / or before the target object 112). For example, the robotic system 100 may compare the target object 112 to a predetermined number of objects that precede or follow the expected object in the packing sequence. If the characteristics of the target object 112 match the expected characteristics of the other objects that are scheduled to arrive at the start location 114, the robotic system 100 may determine that an arrival sequence error has occurred (e.g., the target object 112 arrived at the start location 114 out of order) and / or may store the target object 112 in preparation for its next occurrence and perform one or more real-time adjustments (e.g., store the target object 112 in a temporary holding area and then access it according to the next expected sequence / timing).

[0118] Additionally or alternatively, in block 834, the robotic system 100 can identify a destination matching error. To identify a destination matching error, the robotic system 100 can compare the destination sensor data to data corresponding to an expected packaging situation at the task location 116. For example, the robotic system 100 can track the current progress of the 3D packing plan based on identifying placed objects. The robotic system 100 can use one or more computer models to determine an expected shape and / or expected surface contour (e.g., a set of height estimates corresponding to an expected placement surface) corresponding to the tracked progress. The robotic system 100 can compare images captured using the sensors 216, depth maps, and / or other data representing the current state of the task location 116 (e.g., a discretized platform model and / or the current height dimension of the task location 116) to the expected shape and / or expected surface contour. The robotic system 100 can determine a destination matching error based at least in part on the difference between the current state of the task location 116 and the expected state of the task location 116.

[0119] As an illustrative example, the robotic system 100 can analyze captured images and / or other data of the task location 116 (e.g., of the placement area 340 and / or the placed object 508 at the task location 116) to determine current characteristics of the task location 116 and / or the placed object 508, e.g., location / position, pose / orientation, physical dimensions, shape, height measurements, and / or other characteristics. To determine a destination matching error, the robotic system 100 can compare one or more of the current characteristics of the task location 116 with one or more expected characteristics of the task location 116, the placed object 508, and / or the placement area 340 to identify any inconsistencies or discrepancies. Using the comparison of the current characteristics to the expected characteristics, the robotic system 100 can determine whether one or more errors (e.g., a placement accessibility error, an unexpected placement error, and / or a placement area error) have occurred. Some examples of a placement accessibility error (e.g., characteristics of the placement area 340 are different from expected) may be based on the placement area 308 having a different size or shape than expected, for example, because an incorrectly sized pallet was placed at the task location 116 or because the walls of the container (e.g., a cage or cart) at the task location 116 are not fully open. Some examples of an unexpected placement error may be based on one or more of the placed objects 508 having an unexpected position / pose, for example, because one or more of the placed objects 508 have moved, shifted, and / or fallen, and / or been placed in an incorrect position and / or orientation. Some examples of a placement area error may correspond to a height measurement of the placement area 340 being different from expected. In these and other embodiments, the robotic system 100 may use destination sensor data to determine that one or more of the placed objects 508 were previously placed incorrectly or are missing and / or an unexpected object is at the task location 116.

[0120] In block 836, the robotic system 100 can determine other types of errors. For example, the robotic system 100 can analyze real-time packaging conditions (e.g., feedback data from the robotic units and / or current conditions at the object source / destination) to identify operation errors, such as collision errors (e.g., the robotic units and / or the object collided) and / or object movement errors (e.g., the object shifted during or after placement). As another example, the robotic system 100 can identify transportation or manipulation errors, such as when the gripper of the transposition unit 104 does not have a sufficient grip on the object and / or when the object is dropped / lost during transportation / manipulation. While performing the packing operation (shown in block 801), the robotic system 100 can obtain various feedback data from the robotic units, such as position, velocity, status (e.g., external contact status and / or gripping status), force measurements (e.g., externally applied force, gripping force, and / or weight / torque measured at the gripper), or a combination thereof. The robotic system 100 can compare the acquired data to one or more predetermined thresholds / templates that characterize movement errors to identify their occurrence.

[0121] At block 838, the robotic system 100 may identify a packaging situation that poses a risk of collision. In some embodiments, the robotic system 100 may identify whether a packaging situation poses a risk of collision, independently of or in response to determining that one or more errors have occurred (at blocks 832-836). The robotic system 100 may analyze sensor data to identify a risk of collision of the robot units and / or objects if the robotic system 100 continues to package / palletize the target object 112. If the target object 112 is an expected object according to the packing sequence, the robotic system 100 may recalculate and / or access the approach path 510 of FIG. 5 to place the target object 112 according to the 3D packing plan. Because the current situation deviates from the expected situation due to the determined error, the robotic system 100 may compare the approach path 510 and the real-time situation to determine a risk of collision. In some embodiments, the robotic system 100 analyzes real-time sensor data, allowing the robotic system 100 to identify real-time packaging situations according to a specified priority (e.g., in order of decreasing risk of collision and / or error).With reference to task location 116, for example, robotic system 100 can analyze destination sensor data to enable robotic system 100 to identify real-time packaging situations that may impede entry path 510 in the following order: (a) a placement accessibility error situation, in which a container at task location 116 poses a collision risk (e.g., because the container is not fully open); (b) an unexpected placement error situation, in which one or more placed objects 508 that have moved, shifted, and / or fallen pose a collision risk; (c) a placement area error situation, in which one or more placed objects 508 that have been placed in an incorrect position and / or orientation pose a collision risk; and / or (d) a placement area error situation, in which the height of the palletized cases poses a collision risk (e.g., due to a difference between the expected and actual height measurements of placement area 340, as described above).

[0122] If, in blocks 832-838, the robotic system 100 determines that no errors occurred or does not identify any errors or packaging conditions that pose a risk of collision, the robotic system 100 may return to block 801 to continue executing the packing plan. Otherwise, in any of the error and potential collision scenarios outlined above, continued execution of the packing plan may introduce further errors. For example, stacked pallets may contain incorrect items, and / or the pallets or their objects may become unstable due to different characteristics than intended by the packing plan (e.g., different heights of one or more supporting objects in the stack). Additionally, the risk of collision of the robotic units and / or objects may increase due to discrepancies in object position and / or unexpected obstacles.

[0123] Thus, in response to determining one or more errors and / or identifying one or more packaging conditions that pose a risk of collision, the robotic system 100 may (at block 840) determine a response to the errors and / or potential collisions identified in blocks 832-838. Examples of suitable responses include (a) alerting an operator and / or other systems to the real-time condition that poses the error and / or risk of collision, (b) returning to block 804 to dynamically generate a discretized model of the target object 112, (c) updating the master data 252, (d) updating the master list of task locations 116 to include unexpected objects at the task location 116 and / or to exclude objects that are missing at the task location 116, (e) placing the target object 112 at a location other than the task location 116 (e.g., to later relocate the target object 112 to a new location other than the task location 116), (f) relocating the target object 112 to a new location other than the task location 116 (e.g., to a new location for later relocating the target object 112 to a new location), (g) relocating the target object 112 to a new location other than the task location 116 (e.g., to a new location for later relocating the target object 112 to a new location), (h) relocating the target object 112 to a new location other than the task location 116 (e.g., to a new location for later relocating the target object 112 to a new location), (i) relocating the target object 112 to a new location other than the task location 116 (e.g., to a new location for later relocating the target object 112 to a new location), (j) relocating the target object 112 to a new location other than the task location 116 (e.g., to a new location for later relocating the target object 112 to a new location), (k) re (f) setting aside target object 112 for transport / manipulation to packing location 116 and / or storing target object 112); (g) dynamically adjusting approach path 510 to implement packing plan; (h) dynamically deriving updated placement locations to modify or adjust packing plan; (i) creating a new packing plan; and / or (j) discarding the packing plan and dynamically deriving placement locations upon object arrival.

[0124] In some embodiments, the robotic system 100 can determine an appropriate response based on the type of error or potential collision identified. For example, in the event of a master data error, the robotic system 100 may be able to continue executing the packing plan. Thus, an appropriate response to a master data error may include placing the target object 112 (i.e., the object that does not match the master data 252) in a temporary location, placing subsequent objects in their designated or temporary locations (e.g., if the placement would obstruct the target object 112's originally intended path of entry 510). If the expected object (i.e., as opposed to the target object 112) arrives at the start location 114, the robotic system 100 may place the expected object in its originally intended location. Storing or placing the target object 112 in a temporary location is described in more detail in a concurrently filed U.S. patent application by Rosen N. Kiankov and Denys Kanunikov, entitled "ROBOTIC SYSTEM FOR PROCESSING PACKAGES ARRIVING OUT OF SEQUENCE," which is assigned to Mujin, Inc. and identified by attorney docket number 131837-8008.US01, and is incorporated herein by reference in its entirety. If one or more subsequent instances of the expected object (i.e., of the same type / identifier as the target object 112) also cause a master data error, in some embodiments, the robotic system 100 can compare the dimensions and / or discretized model of the target object 112 with those of the expected object. If the compared data matches or is within a predetermined threshold range, the robotic system 100 can notify an operator and / or continue executing the packing plan after updating the master data. For example, the flow may return to block 804 to dynamically generate a discretized model of the object of interest 112 in real time and update the master data 252 to include the generated discretized model.

[0125] If the dimensions and / or discretization data do not match or differ by a dimension that exceeds a threshold range, the robotic system 100 can discard the existing packing plan and re-derive a packing plan according to the updated master data and the current situation (e.g., the initial height of the remaining packages and / or the loading surface). Thus, the robotic system 100 can perform the above-described method 700 or one or more portions thereof. In some embodiments, for example, the robotic system 100 can re-identify the package set according to the remaining objects (block 706) and perform one or more of the following operations (e.g., as shown in block 704, etc.): In some embodiments, the robotic system 100 can access and adjust the determined package group according to the current situation (e.g., by removing placed objects), maintain the determined processing order, and re-derive the 2D and 3D loading plans accordingly. For example, the robotic system 100 can use the current situation at the task location 116 as the updated placement surface (e.g., instead of the discretized platform model 304), or can consider the current situation as an existing part of the plan, e.g., the result of a previous planning iteration as described above. Alternatively, if the remaining number of packages is below a threshold value, the robotic system 100 can dynamically derive a placement location, as described in more detail below.

[0126] As another example, in the case of a placement accessibility error in which a container at the task location 116 is not fully open, the robotic system 100 can alert an operator and / or other systems that the cage or car track is not fully open. Additionally or alternatively, the robotic system 100 can determine the extent to which the packing plan is affected by the container not being fully open. For example, the robotic system 100 can determine the extent (e.g., the number and / or relative positions of affected objects in the 3D packing plan) based on overlaying the 3D packing plan on destination data (e.g., images and / or depth maps) and identifying and / or counting objects in the 3D packing plan that overlap with a partially closed wall or cover of the cage / car track. The robotic system 100 can also determine the extent by comparing the entry paths 510 for one or more of the objects in the 3D packing plan with the walls / edges of the cage, car track, and identify paths that intersect with the walls / edges. The robotic system 100 can adjust the range based on identifying (e.g., according to a predetermined function, rule, model, etc.) and taking into account (e.g., by increasing the number of affected objects) other dependent objects supported by the affected object. In these embodiments, if the location / pattern of the affected objects matches one or more predetermined templates and / or the quantity of affected objects falls below a threshold quantity, the robotic system 100 can determine that the appropriate response is to implement the packing plan by dynamically modifying or adjusting the approach path 510 such that the robotic system 100 places the target object 112 in a position at the task location 116 specified by the packing plan while avoiding collisions with the container and / or placed objects 508.

[0127] Alternatively, the robotic system 100 may determine, for example, that the number of affected objects exceeds a threshold quantity, which corresponds to a situation in which most of the task locations 116 are inaccessible to the robotic system 100, but the container is fully or mostly closed. In some embodiments, the robotic system 100 may determine that an appropriate response to a placement accessibility error is to dynamically re-derive / adjust the placement locations 350 of the target object 112 and / or subsequently arriving objects (as described in more detail below with respect to blocks 808-816) when the objects arrive at the start location 114. In some embodiments, the robotic system 100 may determine that an appropriate response to a placement accessibility error is to create a new packing plan that utilizes only as many task locations 116 as are still accessible to the robotic system 100. For example, the robotic system 100 can stop the packing operation (block 801) and determine package groups for a new packing plan by grouping available packages (e.g., the target object 112, packages that have not yet arrived at the start location 114 but will arrive according to the previous packing plan, and / or the placed objects 508) according to a grouping condition. The robotic system 100 can then generate a new packing plan by (i) generating a 2D plan by identifying unique items and iteratively deriving placement positions, and (ii) converting the 2D plan to a 3D state, calculating the 3D score, and selecting a placement combination according to the placement score. In these and other embodiments, the robotic system 100 can (i) buffer one or more other packages arriving at the target object 112 and / or the start location 114 to adjust the packing sequence defined by the previous packing plan, and / or (ii) adjust the position of one or more placed objects 508.

[0128] In some embodiments, the robotic system 100 can determine that an appropriate response to the identified error and / or potential collision is to dynamically derive the placement location 350 of the target object 112 and subsequent objects. For example, the robotic system 100 can dynamically derive the placement location 350 when the identified error and / or potential collision meets or exceeds a predefined condition or threshold. As a specific example, the robotic system 100 can dynamically derive the placement location in response to an unexpected placement error as described above (e.g., if one or more placed objects 508 move, shift, etc.).

[0129] To dynamically derive the placement location 350 of the target object 112, in block 808, the robotic system 100 may derive a set of candidate locations (e.g., the candidate locations 360 in FIG. 3B ) for placing the target object 112 at / above the task location 116. The robotic system 100 may derive the candidate locations 360 based on overlaying the discretized object model 302 of the target object 112 onto the discretized bed model 304 of the current state of the task location 116 at corresponding locations within / above the task location 116. The candidate locations 360 may correspond to positions of the discretized object model 302 above / within the discretized bed model 304 along a horizontal plane. The robotic system 100 may derive the candidate locations 360 that overlap and / or are adjacent to the placed object 508.

[0130] In some embodiments, the robotic system 100 can iteratively determine the position of the discretized object model 302 based on determining an initial placement location (e.g., a predetermined location for an instance of the candidate location 360, e.g., a designated corner of the placement area). The robotic system 100 can determine the subsequent candidate locations 360 according to a predetermined direction for deriving the next candidate location 360, spacing requirements between candidate locations 360 across iterations, rules / conditions governing placement, a limit on the total number of candidate locations 360, one or more patterns thereof, or a combination thereof. Furthermore, the robotic system 100 can include a set of settings and / or rules for determining the candidate locations 360 for the placed object 508. For example, the robotic system 100 can be configured with a setting for determining candidate locations 360 where the discretized object model 302 is adjacent to or abutting one or more edges of the placed object 508 and / or the peripheral boundary / perimeter of the placement area 340 (e.g., to perform a function sooner than most other types / categories of candidate locations 360). The robotic system 100 can also be configured with settings to determine candidate positions 360 where the discretized object model 302 is above the placed objects 508, fits within one of the objects, and / or overlaps with one or more edges of the object.

[0131] The robotic system 100 can derive the candidate positions 360 according to predetermined rules, patterns, restrictions, and / or sequences for placing the discretized object model 302. For example, the robotic system 100 can derive the candidate positions 360 based on, for example, a setting for an object edge that is adjacent to and / or within a predetermined distance limit to the outermost edge of the placed object 508. The robotic system 100 can also derive the candidate positions 360 based on, for example, a setting for an outer edge / boundary of the placement area 340 where the discretized object model 302 is closest to or abuts the boundary / edge of a pallet, cage, etc. The robotic system 100 can also derive the candidate positions 360 that overlap the placed object 508.

[0132] In block 810, the robotic system 100 can determine / update the real-time status of the placement area 340 of FIG. 3B , such as for evaluating 3D stacking. For example, the robotic system 100 can use the destination sensor data to determine the height dimension 402 of FIG. 4 . The robotic system 100 can calculate the height of the top surface(s) at the task location 116 using the depth dimension derived from the destination sensor data and the known height of the task location 116 and / or sensors. The robotic system 100 can match the calculated height with the unit pixels 310 in the discretized base model 304 and assign the largest calculated height within the unit pixel 310 as the corresponding height dimension 402. In some embodiments, the robotic system 100 can determine the height dimension 402 for the unit pixel 310 that the discretized object model 302 overlaps within the candidate location 360.

[0133] In block 812, the robotic system 100 may evaluate the candidate locations 360. In some embodiments, the robotic system 100 may evaluate the candidate locations 360 according to real-time conditions, processing results, predetermined rules and / or parameters, or a combination thereof. For example, the robotic system 100 may evaluate the candidate locations 360 based on calculating a corresponding placement score, validating / qualifying the candidate locations 360, or a combination thereof.

[0134] In block 842, the robotic system 100 may calculate a placement score for each candidate location 360. The robotic system 100 may calculate the placement score according to one or more of the placement conditions. For example, the robotic system 100 may use placement settings and / or formulas (e.g., via multiplier weights) to describe settings for package-to-package separation distance, package dimension / fragility rating / package weight difference for horizontally adjacent packages, collision probability (e.g., based on approach path 510 in FIG. 5 or its characteristics and / or horizontal spacing 616 in FIG. 6), consecutive / adjacent surfaces at the same height, their statistical results (e.g., average, maximum, minimum, standard deviation, etc.), or combinations thereof. Other examples of placement settings may include resulting height, proximity, edge-placed status, maximum supportable weight, object type, support weight ratio, or combinations thereof. Thus, in some embodiments, the robotic system 100 may include processing weights / multipliers that represent settings for a lower maximum height, for placing the target object 112 near the boundary of an already placed object or the edge of a placement platform, for minimizing the difference in height and / or maximum supportable weight of adjacent objects, for reducing the ratio of support weight to maximum supportable weight for objects that the target object 112 overlaps, for matching object types of adjacent objects, or combinations thereof. Each placement location may be scored according to preference factors and / or formulas predefined by the system manufacturer, order, and / or system operator.

[0135] In some embodiments, for example, the robotic system 100 can calculate a placement score based on the degree of support for the candidate locations 360. The robotic system 100 can calculate the amount of support (e.g., when stacking objects) for one or more of the candidate locations 360 based at least in part on the height dimension 402. As an illustrative example, the robotic system 100 can calculate the amount of support based on identifying the maximum height 420 of FIG. 4B for each candidate location 360. Based on the maximum height 420 of FIG. 4B and the height difference threshold 416, the robotic system 100 can calculate the lower height limit 422 of FIG. 4B for each candidate location 360. The robotic system 100 can compare the height dimension 402 of the candidate location 360 with the corresponding lower height limit 422 to identify a support location 442 of FIG. 4B for each candidate location 360. The robotic system 100 can calculate a placement score for each candidate location 360 based on the eligible number 424 of FIG. 4B of the corresponding support location 442.

[0136] In one or more embodiments, the robotic system 100 can calculate a placement score based on deriving the support area contours 426 of FIG. 4B for the candidate locations 360. As described above, the robotic system 100 can derive a set of support area contours 426 for each candidate location 360 based on extending the outer edges and / or joining corners of the outermost / perimeter instances of support locations 442 at the corresponding locations. Based on the support locations 442, the robotic system 100 can determine the support area size 428 of FIG. 4B and / or the support ratio 430 of FIG. 4B to calculate the placement score. The robotic system 100 can also calculate a minimum separation distance between the CoM location 432 and the support area contours 426. The robotic system 100 can use the support area size 428, the support ratio 430, the minimum separation distance, the corresponding preference weight, or a combination thereof to calculate a placement score for the corresponding candidate location.

[0137] In one or more embodiments, the robotic system 100 can calculate a placement score based on deriving the entry path 510 for the candidate location 360 as described above. The robotic system 100 can calculate a placement score for each candidate location 360 depending on the final segment 606 in Figure 6 (e.g., its length), the quantity / length of one or more path segments 604 in Figure 6, or a combination thereof. In some embodiments, the robotic system 100 can calculate a placement score based on the horizontal spacing 616 in Figure 6 for the candidate location 360.

[0138] In some embodiments, as shown in block 844, the robotic system 100 can qualify the candidate locations 360. The robotic system 100 can qualify the candidate locations 360 based on dynamically deriving a validated set of the candidate locations 360 according to one or more placement constraints. In deriving the validated set, the robotic system 100 can eliminate or disqualify instances of the candidate locations 360 that violate or do not satisfy one or more of the placement constraints related at least in part to the height dimension 402. In one or more embodiments, the robotic system 100 can first derive the validated set and then calculate a placement score for the validated set. In one or more embodiments, the robotic system 100 can derive the validated set simultaneously with calculating the placement score.

[0139] In one or more embodiments, the placement constraints may involve comparing the eligible numbers 424, the set of support area contours 426, the support area size 428, the support ratio 430, the CoM location 432, the approach path 510, the horizontal spacing 616, or a combination thereof, to a threshold (e.g., the support threshold 418 in FIG. 4B ) or requirement. For example, the robotic system 100 may derive the verified set to include locations where the eligible numbers 424, the support area size 428, and / or the support ratio 430 meet / exceed the corresponding thresholds. The robotic system 100 may also derive the verified set to include locations where the CoM location 432 is within / enclosed by the support area contours 426 and / or meets a minimum separation distance from the support area contours 426. The robotic system 100 may also derive the verified set to include locations where the approach path 510 (e.g., the final segment 606 therein) meets a maximum length threshold and / or where the horizontal spacing 616 meets a minimum threshold.

[0140] In block 814, the robotic system 100 can dynamically derive a placement location 350 for placing the target object 112 above / at the task location 116. The robotic system 100 can dynamically derive the placement location 350 based on selecting one of the locations in the validated set or the candidate locations 360 according to the placement score. In some embodiments, the robotic system 100 can track the candidate locations 360 using a heap structure. Accordingly, the robotic system 100 can remove a location from the heap structure if the location violates the above-mentioned constraints. Furthermore, the robotic system 100 can order or rank the tracked locations according to the corresponding placement score. In some embodiments, the robotic system 100 can continuously order the tracked locations as the placement score or iterative updates of the score are being calculated. As a result, the robotic system 100 can select a location at a designated position (e.g., the first slot) in the heap structure as the placement location 350 when the score calculation is finished.

[0141] In block 816, the robotic system 100 may communicate information for placing the target object 112 at the derived placement location 350. In communicating information for placing the target object 112 at the placement location 350, one or more components / devices of the robotic system 100 may communicate with and / or operate other components / devices. For example, one or more of the processor 202 and / or a stand-alone controller (e.g., a control device of a warehouse / shipping center, etc.) may send information to the other components / devices, such as, for example, the placement location 350, a corresponding motion plan, a set of commands and / or settings for operating the actuation device 212 of FIG. 2 and / or the transport motor 214 of FIG. 2, or a combination thereof. Other components / devices, such as other instances of processor 202, and / or robotic arm 502, actuation device 212, transport motor 214, and / or other external devices / systems of FIG. 5, can receive the information and perform corresponding functions to manipulate (e.g., grasp and pick up, transpose and / or reorient through space, place at a destination, and / or release) target object 112 and place it at a placement location.

[0142] In block 818, the robotic system 100 may verify the placement of the target object 112 at the derived placement location. For example, in response to the information communicated in block 816, the robotic system 100 may determine whether the robotic system 100 placed the target object 112 at the derived placement location, and if so, the robotic system 100 may determine the accuracy of the placement of the target object 112 compared to the derived placement location by analyzing the destination sensor data. In some embodiments, the robotic system 100 may update or re-identify the real-time packaging situation after placing the target object 112. In other words, following block 818, the control flow may move to blocks 801 and / or 802. Accordingly, the robotic system 100 may update / identify the next incoming object as the target object 112. The robotic system 100 may also update information about the placement area 340 and / or the placed objects 508 thereon to include recently placed objects. In other embodiments, the robotic system 100 may recalculate or adjust the packing plan and / or resume following the packing plan after placing the target object 112 (block 801).

[0143] The above-described task discretization and 2D / 3D layering improves the efficiency, speed, and accuracy of packing objects. Therefore, the reduced operator input and increased accuracy can further reduce the human effort involved in the automated packing process. In some environments, the above-described robotic system 100 can eliminate the need for array buffers, which can cost approximately $1 million or more.

[0144] Furthermore, by dynamically calculating the placement location 350 in response to real-time conditions (e.g., as represented by sensor data and other status / data), operational errors are reduced. As described above, the robotic system can account for and resolve errors and / or potential collisions introduced by unexpected conditions / events without requiring human intervention. The above-described embodiments can also stack objects in 3D without a pre-existing packing plan or when unexpected conditions / events affect all or part of a pre-existing packing plan, such as by dynamically deriving the placement location 350 when the object arrives at the start location 114. Compared to conventional systems limited to 2D dynamic packing (i.e., placing objects directly on the platform as a single layer), by considering height, the above-described embodiments can stack objects on top of each other, improving packing density. In this manner, the robotic system 100 can identify errors and / or potential collisions and determine appropriate actions in response, whether that be continuing to implement a previous packing plan, recovering portions of a previous packing plan, generating a new packing plan, and / or dynamically deriving placement locations for the target object 112 and one or more packages that will subsequently arrive at the start location 114.

[0145] conclusion The above detailed description of embodiments of the disclosed technology is not intended to be exhaustive or to limit the disclosed technology to the precise form disclosed above. While specific examples of the disclosed technology have been described above for illustrative purposes, those skilled in the art will recognize that various equivalent modifications are possible within the scope of the disclosed technology. For example, while processes or blocks are presented in a given order, alternative embodiments may perform routines having steps in a different order or employ systems having blocks in a different order, or some processes or blocks may be removed, moved, added, subdivided, combined, and / or modified to provide alternative combinations or sub-combinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks may be shown as being performed in sequence, these processes or blocks may instead be performed or implemented in parallel, or may be performed at different times. Furthermore, any specific numbers described herein are merely examples, and alternative embodiments may employ different values or ranges.

[0146] These and other changes can be made to the disclosed technology in light of the above detailed description. While the detailed description sets forth the best mode contemplated as well as specific examples of the disclosed technology, no matter how detailed the above description may appear in text, the disclosed technology can be practiced in many ways. The details of the system may vary significantly in its particular embodiment and still be encompassed by the technology disclosed herein. As noted above, specific terms used in describing particular features or aspects of the disclosed technology should not be construed to mean that the terms are redefined herein to be limited to any particular characteristic, feature, or aspect of the disclosed technology to which they relate. Accordingly, the present invention is not limited except as by the appended claims. In general, the terms used in the following claims should not be construed to limit the disclosed technology to the specific examples disclosed herein unless the detailed description section above explicitly defines such terms. As used herein, the term "and / or," as in "A and / or B," refers to A only, B only, and both A and B.

[0147] Although certain aspects of the invention are presented below in particular claim forms, Applicant contemplates various aspects of the invention in any number of claim forms, and accordingly, Applicant reserves the right to pursue additional claims after filing this application to pursue such additional claim forms in this application or any continuing application.

Claims

1. 1. A method of operating a robotic system, comprising: (i) determining a source matching error by identifying differences between (a) an object model representing a target object, the object model being discretized by overlaying discretization units on an area representing the target object obtained from image data of the target object, and (b) an assumed discretized object model contained within a packing plan or master data; or (ii) determining a destination matching error by identifying differences between (a) a platform model representing a placement area representing the physical dimensions or shape of at least a portion of a task location, the platform model being discretized by overlaying discretization units on an area representing the placement area obtained from image data of the placement area, and (b) an assumed platform model; determining an error, including: adjusting a placement position of the target object at least partially in response to determining the error; and A method comprising:

2. determining the error includes identifying discrepancies between the object model and the master data; The method of claim 1 , wherein the error comprises a master data error indicating that the master data is incorrect.

3. the packing plan specifies a sequence in which objects, including the target object, arrive at a starting location; the start position is a position from which the robotic system transports the target object to the placement position; determining the error includes identifying a difference between the object model and the assumed discretized object model specified by the sequence; The method of claim 1 , wherein the errors include an arrival sequence error indicating that the target object arrived at the starting location out of the sequence specified by the packing plan.

4. determining the error includes identifying a difference between the platform model and the assumed platform model; 2. The method of claim 1, wherein the errors include a placement accessibility error indicating that the accessible portion of the placement area has a different size or shape than expected, and / or an unexpected placement error indicating that an object previously placed at the task location has an unexpected position or orientation at the task location.

5. The method of claim 4 , wherein determining the error includes identifying a loading accessibility error indicating that a wall of a container, cage, or cart truck at the task location is not fully open.

6. determining the error includes identifying a difference between (a) the platform model and (b) the assumed platform model; The method of claim 1 , wherein determining the error includes identifying an unexpected placement error indicative of an object being shifted, dropped, or displaced from among the placed objects on the task location.

7. determining the error includes identifying a difference between (a) the platform model and (b) the assumed platform model; The method of claim 1 , wherein determining the error includes identifying a placement area error indicating that an object among the placed objects on the task location was placed incorrectly.

8. the assumption vehicle model is specified by the packing plan; determining the error includes identifying a difference between (a) the platform model and (b) the assumed platform model; The method of claim 1 , wherein determining the error includes identifying a placement area error indicating that an object not included in the packing plan has been placed on the task location.

9. Determining the error is performed by: determining a loading accessibility error indicative of a wall of a container, cage, or cart truck at the task location not being fully open; determining an unexpected placement error indicative of a first one of the placed objects on the task location being shifted, dropped, or displaced; determining a placement area error indicating that a second object of the placed objects on the task location is incorrectly placed; determining a placement area error indicative of (a) a third one of the placed objects, (b) the container, cage, or cart truck, and / or (c) an object not included in the packing plan but present at the task location contributing to a difference between the height dimension and the expected height dimension; The method of claim 1 , comprising identifying the error by:

10. determining the error includes identifying a difference between (a) the platform model and (b) the assumed platform model; The method of claim 1 , wherein the error includes a placement area error indicating that an object among the placed objects on the task location is missing from the task location.

11. determining the error includes identifying a difference between (a) the platform model and (b) the assumed platform model; The method of claim 1 , wherein the error comprises a transportation or handling error representing an object being dropped while being transported to the task location.

12. determining the error includes identifying a difference between (a) the platform model and (b) the assumed platform model; 2. The method of claim 1, wherein the errors include collision errors representing collisions between (1) a robotic unit of the robotic system, (2) the target object, (3) a placed object at the task location, (4) a container at the task location, and / or (5) an object not included in the packing plan but present at the task location.

13. the packing plan specifies the placement location of the target object; The method of claim 1 , wherein the method is performed before placing the target object at the target object's placement location specified in the packing plan.

14. In response to determining the error, deriving candidate locations based at least in part on overlaying the discretized object model onto corresponding locations on the discretized base model; validating the candidate location according to one or more placement constraints related to a height dimension; calculating a placement score for the candidate location; dynamically deriving the placing location based on selecting the candidate location according to the placing score, the placing location being for placing the target object at the task location; The method of claim 1 further comprising:

15. The method of claim 14 , wherein validating the candidate location includes deriving an approach path for placing the target object at the candidate location.

16. adjusting the placement position of the target object includes generating information for placing the target object at the placement position on the placement area; The method of claim 14 , further comprising transmitting the generated information for placing the target object at the placement location on the placement area.

17. The method of claim 16 , further comprising verifying placement accuracy of the target object at the placement location.

18. The method of claim 14 , further comprising updating or replacing the packing plan based at least in part on the determined error or the placement location for placing the target object.

19. at least one processor; at least one memory device coupled to said at least one processor and storing instructions executable by said processor; The instruction: (i) determining a source matching error by identifying differences between (a) an object model representing a target object, the object model being discretized by overlaying discretization units on an area representing the target object obtained from image data of the target object, and (b) an assumed discretized object model contained within a packing plan or master data; or (ii) determining a destination matching error by identifying differences between (a) a platform model representing a placement area representing the physical dimensions or shape of at least a portion of a task location, the platform model being discretized by overlaying discretization units on an area representing the placement area obtained from image data of the placement area, and (b) an assumed platform model; determining an error, including: adjusting a placement position of the target object based at least in part on the determined error; and A robot system comprising:

20. 1. A tangible, non-transitory computer-readable medium having stored thereon processor instructions that, when executed by a robotic system via one or more processors thereof, cause the robotic system to perform a method, comprising: The processor instructions: (i) determining a source matching error by identifying differences between (a) an object model representing a target object, the object model being discretized by overlaying discretization units on an area representing the target object obtained from image data of the target object, and (b) an assumed discretized object model contained within a packing plan or master data; or (ii) determining a destination matching error by identifying differences between (a) a platform model representing a placement area representing the physical dimensions or shape of at least a portion of a task location, the platform model being discretized by overlaying discretization units on an area representing the placement area obtained from image data of the placement area, and (b) an assumed platform model; an instruction for determining an error, including: instructions to adjust the placement position of the target object based at least in part on the determined error; and 1. A tangible, non-transitory computer-readable medium, comprising:

Images