Robot system with depth-based processing mechanism and its operation method

Abstract

A system and method for estimating aspects of a target object and / or an associated task implementation are disclosed.

Description

[0001] This application is a divisional application of Chinese Application No. CN202210874140.6, with a filing date of July 21, 2022, and an invention title of "Robot System with Depth-Based Processing Mechanism and Its Operating Method".

[0002] Related Applications

[0003] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 224,292, filed on July 21, 2021, which is hereby incorporated by reference in its entirety.

[0004] This application incorporates by reference the subject matter of the following applications: U.S. Patent Application No. 16 / 290,741, filed on March 1, 2019, now U.S. Patent No. 10,369,701; U.S. Patent Application No. 16 / 443,743, filed on June 17, 2019, now U.S. Patent No. 10,562,188; U.S. Patent Application No. 16 / 443,757, filed on June 17, 2019, now U.S. Patent No. 10,562,189; U.S. Patent Application No. 16 / 736,667, filed on January 7, 2020, now U.S. Patent No. 11,034,025; U.S. Patent Application No. 17 / 313,921, filed on May 6, 2021; U.S. Patent Application No. 16 / 539,790, filed on August 13, 2019, now U.S. Patent No. 10,703,584; and U.S. Patent Application No. 16 / 888,376, filed on May 29, 2020. The subject matter of all these applications is hereby incorporated by reference in its entirety.

[0005] This application also incorporates by reference the subject matter of the U.S. patent application titled "ROBOTIC SYSTEM WITH IMAGE-BASED SIZING MECHANISM AND METHODS FOR OPERATING THE SAME", filed simultaneously herewith. Technical Field

[0006] This technology generally relates to robotic systems, and more particularly, to robotic systems with depth-based processing mechanisms. Background Art

[0007] Robots (e.g., machines configured to automatically / autonomously perform physical actions) are now widely used in many fields. For example, robots can be used to perform various tasks (e.g., manipulate or transport objects) in manufacturing, packaging, transportation, and / or delivery. When performing tasks, robots can replicate human actions, thereby replacing or reducing the human involvement required to perform dangerous or repetitive tasks. Robots generally lack the precision necessary to replicate the sensitivity, flexibility, and / or adaptability of humans required to analyze and perform more complex tasks. For example, robots often have difficulty inferring multiple conclusions and / or generalizations based on limited information. Therefore, there is still a need for improved robot systems and technologies for inferring conclusions and / or generalizations. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Illustrates an exemplary environment in which a robot system transports an object according to one or more embodiments of the present technology.

[0009] Figure 2 Is a block diagram of a robot system according to one or more embodiments of the present technology.

[0010] Figure 3 Illustrates a robot handling component according to one or more embodiments of the present technology.

[0011] Figure 4A Illustrates an exemplary first stack according to one or more embodiments of the present technology.

[0012] Figure 4B Illustrates exemplary image data depicting a first stack according to one or more embodiments of the present technology.

[0013] Figure 5A Illustrates an exemplary second stack according to one or more embodiments of the present technology.

[0014] Figure 5B Illustrates exemplary image data depicting a second stack according to one or more embodiments of the present technology.

[0015] Figure 6A Illustrates a first image of a starting position according to one or more embodiments of the present technology.

[0016] Figure 6B Illustrates a second image of a starting position according to one or more embodiments of the present technology.

[0017] Figure 7A Illustrates a first image of a task position according to one or more embodiments of the present technology.

[0018] Figure 7BShows a second image of the task location according to one or more embodiments of the present technology.

[0019] Figure 8 Is a flowchart for operating a robotic system according to one or more embodiments of the present technology. Detailed Description

[0020] Systems and methods are described herein for deriving estimates based on one or more measurement results (e.g., depth metrics) captured during task execution. In some implementations, a robotic system may be configured to move one or more objects (e.g., boxes, packages, objects, etc.) from a starting location (e.g., a pallet, cabinet, conveyor, etc.) to a task location (e.g., a different pallet, cabinet, conveyor, etc.). The robotic system may obtain a set or series of image data (e.g., two-dimensional (2D) and / or three-dimensional (3D) image data) depicting the starting location and / or the task location during the handling of the corresponding object. The robotic system may use the image data to estimate and / or derive various implementation conditions, such as the number of objects in a stack, verification of pick / place, detection of object interruptions, etc.

[0021] Numerous specific details are set forth below to provide a thorough understanding of the presently disclosed technology. In other embodiments, the technology presented herein may be practiced without these specific details. In other instances, well-known features such as specific functions or routines are not described in detail so as not to unnecessarily obscure aspects of the present disclosure. References to "embodiments", "one embodiment", etc. in this description mean that the particular feature, structure, material, or characteristic described is included in at least one embodiment of the present disclosure. Thus, the appearances of such phrases in this specification are not necessarily all referring to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Additionally, a particular feature, structure, material, or characteristic may be combined in any suitable manner in one or more embodiments. It should be understood that the various embodiments shown in the figures are merely illustrative representations and are not necessarily drawn to scale.

[0022] For clarity, several details of structures or processes that are well-known and often associated with robotic systems and subsystems but may unnecessarily obscure some important aspects of the disclosed technology are not stated in the following description. Additionally, although the following disclosure presents several embodiments of different aspects of the present technology, several other embodiments may have configurations or components different from those described in this section. Thus, the disclosed technology may have other embodiments with additional elements or without several of the elements described below.

[0023] Many embodiments or aspects of the present disclosure described below may take the form of computer or controller-executable instructions, including routines executed by a programmable computer or controller. Those skilled in the art will appreciate that the disclosed techniques may be practiced on computer or controller systems other than the computer or controller systems shown and described below. The techniques described herein may be embodied in a special-purpose computer or data processor that is specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described below. Thus, the terms "computer" and "controller" as generally used herein refer to any data processor and may include Internet appliances and handheld devices (including palm computers, wearable computers, cellular or mobile phones, multiprocessor systems, processor-based or programmable consumer electronics, network computers, minicomputers, etc.). Information processed by these computers and controllers may be presented at any suitable display medium, including a liquid crystal display (LCD). Instructions for performing tasks executable by a computer or controller may be stored in or on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. The instructions may be contained in any suitable memory device, including, for example, a flash drive, a USB device, and / or other suitable media, including tangible non-transitory computer-readable media.

[0024] The terms "coupled" and "connected" and their derivatives may be used herein to describe a structural relationship between components. It should be understood that these terms are not intended as synonyms for each other. Rather, in a particular embodiment, "connected" may be used to indicate that two or more elements are in direct contact with each other. Unless otherwise apparent from the context, the term "coupled" may be used to indicate that two or more elements are in direct or indirect contact with each other (with other intervening elements between them), or that two or more elements cooperate or interact with each other (e.g., interact in a causal relationship, such as for signal transmission / reception or for function calls), or both.

[0025] Suitable environment

[0026] Figure 1 is a diagram of an exemplary environment in which a robotic system 100 transports an object according to one or more embodiments of the present technology. The robotic system 100 may include one or more units (e.g., robots) configured to perform one or more tasks, and / or communicate with the one or more units. Aspects of object detection / update may be practiced or implemented by the various units.

[0027] For Figure 1In the example illustrated, the robotic system 100 may include an unloading unit 102, a handling unit 104 or handling components (e.g., a stacker robot and / or a picking robot), a transport unit 106, a loading unit 108, or a combination thereof, in a warehouse or a distribution / delivery center, and / or communicate with each of the foregoing. Each of the units in the robotic system 100 may be configured to perform one or more tasks. The tasks may be combined in sequence to perform an operation to achieve a goal, such as unloading objects from a truck or a van and storing them in a warehouse, or unloading objects from a storage location and preparing them for delivery. For another example, the task may include placing an object at a task location (e.g., on top of a pallet and / or inside a cabinet / cage / box / container). As described below, the robotic system may derive a plan for placing and / or stacking objects (e.g., a placement position / orientation, an order for handling the objects, and / or a corresponding motion plan). Each of the units may be configured to perform a series of actions (e.g., by operating one or more components therein) to perform the task according to one or more of the derived plans.

[0028] In some embodiments, the task may include manipulating (e.g., moving and / or reorienting) a target object 112 (e.g., one of a package, a box, a container, a cage, a pallet, etc. corresponding to the task being performed), such as moving the target object 112 from a start position 114 to a task position 116. For example, the unloading unit 102 (e.g., an unpacking robot) may be configured to move the target object 112 from a position in a carrier (e.g., a truck) to a position on a conveyor belt. Moreover, the handling unit 104 may be configured to move the target object 112 from one position (e.g., a conveyor belt, a pallet, or a cabinet) to another position (e.g., a pallet, a cabinet, etc.). For another example, the handling unit 104 (e.g., a stacker robot) may be configured to move the target object 112 from a source position (e.g., a pallet, a pick-up area, and / or a conveyor) to a destination pallet. Upon completion of the operation, the transport unit 106 may move the target object 112 from an area associated with the handling unit 104 to an area associated with the loading unit 108, and the loading unit 108 may move the target object 112 from the handling unit 104 to a storage location (e.g., a position on a shelf) (e.g., by moving the pallet carrying the target object 112). Details regarding the tasks and the associated actions are described below.

[0029] For illustrative purposes, the robotic system 100 is described in the context of a fulfillment center; however, it should be understood that the robotic system 100 can be configured to perform tasks in other environments / for other purposes (such as for manufacturing, assembly, packing, healthcare, and / or other types of automation). It should also be understood that the robotic system 100 can include other units not shown in Figure 1 such as manipulators, service robots, modular robots, etc., and / or communicate with such other units. For example, in some embodiments, the other units can include: a depalletizing unit configured to transfer an object from a cage cart or pallet to a conveyor or another pallet; a container switching unit configured to transfer an object from one container to another container; a packing unit configured to package an object; a sorting unit configured to group objects based on one or more characteristics of the objects; a picking unit configured to manipulate (e.g., for sorting, grouping, and / or transferring) an object in different ways based on one or more characteristics of the object; or a combination thereof.

[0030] The robotic system 100 can include and / or be coupled to physical or structural members (e.g., robotic manipulator arms) that are connected at joints for movement (e.g., rotational displacement and / or translational displacement). The structural members and joints can form a power chain configured to manipulate an end effector (e.g., a gripper) in accordance with the use / operation of the robotic system 100, the end effector being configured to perform one or more tasks (e.g., gripping, spinning, welding, etc.). The robotic system 100 can include actuation devices (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) and / or communicate with such actuation devices, the actuation devices being configured to drive or manipulate (e.g., displace and / or reorient) the structural members about or at corresponding joints. In some embodiments, the robotic unit can include a transport motor configured to transport the corresponding unit / chassis from one location to another location.

[0031] The robotic system 100 can include sensors and / or communicate with sensors configured to obtain information for performing tasks (such as for manipulating the structural members and / or for transporting the robotic unit). The sensors can include devices configured to detect or measure one or more physical properties of the robotic system 100 (e.g., the state, condition, and / or position of one or more structural members / their joints) and / or one or more physical properties of the surrounding environment. Some examples of sensors can include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

[0032] For example, in some embodiments, the sensor may include one or more imaging devices configured to detect the surrounding environment (e.g., visual and / or infrared cameras, 2D and / or 3D imaging cameras, distance measurement devices such as lidar or radar, etc.). The imaging device may generate a representation of the detected environment, such as a digital image and / or a point cloud, and the representation may be processed via machine / computer vision (e.g., for automated inspection, robot guidance, or other robotic applications). The robotic system 100 may process the digital image and / or the point cloud to identify the target object 112, the starting position 114, the task position 116, the pose of the target object 112, or a combination thereof.

[0033] To manipulate the target object 112, the robotic system 100 may capture and analyze images of a specified area (e.g., a pick-up location such as inside a truck or on a conveyor belt) to identify the target object 112 and its starting position 114. Similarly, the robotic system 100 may capture and analyze images of another specified area (e.g., a drop-off location for placing an object on a conveyor, a location for placing an object inside a container, or a location on a pallet for stacking purposes) to identify the task position 116. For example, the imaging device may include one or more cameras configured to generate images of the pick-up area and / or one or more cameras configured to generate images of the task area (e.g., the drop-off area). As described below, based on the captured images, the robotic system 100 may determine the starting position 114, the task position 116, the associated pose, the packing / placement plan, the handling / packing sequence, and / or other processing results.

[0034] In some embodiments, for example, the sensor may include position sensors (e.g., position encoders, potentiometers, etc.) configured to detect the position of structural members (e.g., robotic arms and / or end effectors) and / or corresponding joints of the robotic system 100. The robotic system 100 may use the position sensors to track the position and / or orientation of the structural members and / or joints during task execution.

[0035] Robot system

[0036] Figure 2is a block diagram showing components of a robotic system 100 in accordance with one or more embodiments of the present technology. In some embodiments, for example, the robotic system 100 (e.g., at one or more of the above units or components and / or robots) may include electronic / electrical devices such as one or more processors 202, one or more storage devices 204, one or more communication devices 206, one or more input-output devices 208, one or more actuation devices 212, one or more transport motors 214, one or more sensors 216, or combinations thereof. The various devices may be coupled to each other via wired connections and / or wireless connections. For example, one or more units / components and / or robots of the robotic system 100 may include a bus such as a system bus, a Peripheral Component Interconnect (PCI) bus or a PCI-Express bus, HyperTransport or an Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also known as “FireWire”). Moreover, for example, the robotic system 100 may include a bridge, an adapter, a controller, or other signal-related devices for providing wired connections between devices, and / or communicate with the foregoing. The wireless connection may be based on (e.g.) a cellular communication protocol (e.g., 3G, 4G, LTE, 5G, etc.), a Wireless Local Area Network (LAN) protocol (e.g., Wi-Fi (Wireless Fidelity)), a peer-to-peer or device-to-device communication protocol (e.g., Bluetooth, Near Field Communication (NFC), etc.), an Internet of Things (IoT) protocol (e.g., NB-IoT, Zigbee, Z-wave, LTE-M, etc.), and / or other wireless communication protocols.

[0037] The processor 202 may include a data processor (e.g., a central processing unit (CPU), a dedicated computer, and / or an on-board server) configured to execute instructions (e.g., software instructions) stored on the storage device 204 (e.g., a computer memory). The processor 202 may implement program instructions to control other devices / interface with other devices, thereby causing the robotic system 100 to perform actions, tasks, and / or operations.

[0038] The storage device 204 may include a non-transitory computer-readable medium having program instructions (e.g., software) stored thereon. Some examples of the storage device 204 may include volatile memory (e.g., a cache and / or random access memory (RAM)) and / or non-volatile memory (e.g., flash memory and / or a disk drive). Other examples of the storage device 204 may include a portable memory drive and / or a cloud storage device.

[0039] In some embodiments, the storage device 204 can be used to further store master data, processing results, and / or predetermined data / thresholds and provide access thereto. For example, the storage device 204 can store master data that includes descriptions of objects (e.g., boxes, containers, and / or products) that can be manipulated by the robotic system 100. In one or more embodiments, the master data can include the dimensions, shape (e.g., templates for potential poses and / or computer-generated models for identifying objects in different poses), mass / weight information, color schemes, images, identification information (e.g., barcodes, quick response (QR) codes, logos, etc., and / or their expected locations), expected mass or weight, or combinations thereof of the objects that are expected to be manipulated by the robotic system 100. In some embodiments, the master data can include manipulation-related information about the objects, such as the center of mass (CoM) positions on each of the objects, expected sensor measurements (e.g., force, torque, pressure, and / or contact measurements) corresponding to one or more actions / manipulations, or combinations thereof. The robotic system can look up pressure levels (e.g., vacuum levels, suction levels, etc.), gripping / picking areas (e.g., the areas or rows of vacuum grippers to be activated), and other stored master data for controlling the handling robot. The storage device 204 can also store object tracking data. In some embodiments, the object tracking data can include a log of the objects that have been scanned or manipulated. In some embodiments, the object tracking data can include image data (e.g., pictures, point clouds, real-time video feeds, etc.) of the objects at one or more locations (e.g., designated pick or drop locations and / or conveyor belts). In some embodiments, the object tracking data can include the position and / or orientation of the objects at one or more locations.

[0040] The communication device 206 can include circuitry configured to communicate with external or remote devices via a network. For example, the communication device 206 can include a receiver, a transmitter, a modulator / demodulator (modem), a signal detector, a signal encoder / decoder, a connector port, a network card, etc. The communication device 206 can be configured to send, receive, and / or process telecommunications signals according to one or more communication protocols (e.g., Internet Protocol (IP), wireless communication protocols, etc.). In some embodiments, the robotic system 100 can use the communication device 206 to exchange information between the units of the robotic system 100 and / or with systems or devices external to the robotic system 100 (e.g., for reporting, data collection, analysis, and / or troubleshooting purposes).

[0041] The input-output device 208 may include a user interface device configured to communicate information to and / or receive information from a human operator. For example, the input-output device 208 may include a display 210 for communicating information to the human operator and / or other output devices (e.g., speakers, haptic circuits, or haptic feedback devices, etc.). Moreover, the input-output device 208 may include control or receiving devices such as keyboards, mice, touchscreens, microphones, user interface (UI) sensors (e.g., cameras for receiving motion commands), wearable input devices, etc. In some embodiments, the robotic system 100 may use the input-output device 208 to interact with a human operator when performing actions, tasks, operations, or combinations thereof.

[0042] In some embodiments, the controller (e.g., Figure 2 controller 209) may include a processor 202, a storage device 204, a communication device 206, and / or an input-output device 208. The controller may be a stand-alone component or part of a unit / component. For example, each unloading unit, handling component, transport unit, and loading unit of the robotic system 100 may include one or more controllers. In some embodiments, a single controller may control multiple units or stand-alone components.

[0043] The robotic system 100 may include physical or structural members (e.g., robotic arms) connected at joints for movement (e.g., rotational displacement and / or translational displacement), and / or communicating with the physical or structural members. The structural members and joints may form a power chain configured to manipulate an end effector (e.g., a gripper) according to the use / operation of the robotic system 100, the end effector being configured to perform one or more tasks (e.g., gripping, spinning, welding, etc.). The power chain may include actuation devices 212 (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or manipulate (e.g., displace and / or reorient) the structural members around or at corresponding joints. In some embodiments, the power chain may include a transport motor 214 configured to transport a corresponding unit / chassis from one place to another. For example, the actuation devices 212 and the transport motor 214 may be connected to a robotic arm, a linear slider, or other robotic components or a part thereof.

[0044] The sensor 216 can be configured to obtain information for performing tasks such as manipulating structural members and / or transporting the robotic unit. The sensor 216 can include means configured to detect or measure one or more physical properties of the controller, the robotic unit (e.g., the state, condition, and / or position of one or more structural members / their joints), and / or one or more physical properties of the surrounding environment. Some examples of the sensor 216 can include contact sensors, proximity sensors, accelerometers, gyroscopes, force sensors, strain gauges, torque sensors, position encoders, pressure sensors, vacuum sensors, etc.

[0045] In some embodiments, for example, the sensor 216 can include one or more imaging devices 222 (e.g., two-dimensional imaging devices and / or three-dimensional imaging devices) configured to detect the surrounding environment. The imaging devices can include cameras (including visual and / or infrared cameras), lidar devices, radar devices, and / or other ranging or detection devices. The imaging device 222 can generate a representation of the detected environment (such as a digital image and / or a point cloud) for performing machine / computer vision (e.g., for automated inspection, robotic guidance, or other robotic applications).

[0046] Now referring Figure 1 and Figure 2 , the robotic system 100 (e.g., via the processor 202) can process the image data and / or the point cloud to identify Figure 1 the target object 112, Figure 1 the starting position 114, Figure 1 the task position 116, Figure 1 the pose of the target object 112, or a combination thereof. The robotic system 100 can use the image data from the imaging device 222 to determine how to approach and pick up the object. The image of the object can be analyzed to determine (e.g., via a planning system) a motion plan for setting the position of the vacuum gripper assembly to grip the target object. The robotic system 100 (e.g., via various units) can capture and analyze images of a specified area (e.g., the pick-up position of an object inside a truck, inside a container, or on a conveyor belt) to identify the target object 112 and its starting position 114. Similarly, the robotic system 100 can capture and analyze images of another specified area (e.g., the drop-off position for placing an object on a conveyor belt, the position for placing an object inside a container, or the position on a pallet for stacking purposes) to identify the task position 116.

[0047] Moreover, for example, Figure 2 the sensor 216 of Figure 2a position sensor 224 (e.g., a position encoder, a potentiometer, etc.), the position sensor being configured to detect the position of a structural member (e.g., a robotic arm and / or an end effector) and / or a corresponding joint of the robotic system 100. The robotic system 100 may use the position sensor 224 to track the position and / or orientation of the structural member and / or the joint during task execution. The offloading unit, the handling unit, the transport unit / component, and the loading unit disclosed herein may include the sensor 216.

[0048] In some embodiments, the sensor 216 may include one or more force sensors 226 (such as, a weight sensor, a strain gauge, a piezoresistive / piezoelectric sensor, a capacitance sensor, a piezoresistive sensor, and / or other tactile sensors), the one or more force sensors being configured to measure a force applied to a power train, such as at an end effector. For example, the sensor 216 may be used to determine a load on the robotic arm (e.g., an object being grasped). The force sensor 226 may be attached to or around the end effector and configured such that the resulting measurement represents the weight of the grasped object and / or a torque vector relative to a reference position. In one or more embodiments, the robotic system 100 may process the torque vector, weight, and / or other physical characteristics (e.g., dimensions) of the object to estimate the CoM of the grasped object.

[0049] Robot handling component

[0050] Figure 3 Illustrated is a handling unit 104 according to one or more embodiments of the present technology. The handling unit 104 may include an imaging system 160 and a robotic arm system 132. The imaging system 160 may provide image data captured from a target environment having a depalletizing platform 110. The robotic arm system 132 may include a robotic arm assembly 139 and an end effector 140 (e.g., a gripper assembly). The robotic arm assembly 139 may position the end effector 140 above a set of objects in a stack 165 at a pick-up environment 163.

[0051] Figure 3Shows an end effector 140 carrying a single target object 112 (e.g., a package) positioned above a conveyor 120 (e.g., a conveyor belt). The end effector 140 can release the target object 112 onto the conveyor 120, and the robotic arm system 132 can then retrieve packages 112a, 112b by positioning the unloaded end effector 140 directly above the package 112a, the package 112b, or both. The end effector 140 can then hold one or more of the two packages 112a, 112b via vacuum gripping force, and the robotic arm system 132 can transport the held packages 112a, 112b to a position directly above the conveyor 120. The end effector 140 can then release the packages 112a, 112b (e.g., simultaneously or sequentially) onto the conveyor 120. This process can be repeated any number of times to transport objects from the stack 165 to the conveyor 120.

[0052] Continuing to refer Figure 3 , the depalletizing platform 110 can include any platform, surface, and / or structure on which multiple target objects 112 (e.g., packages) can be stacked and / or piled when ready to be transported. The imaging system 160 can include one or more imaging devices 161 configured to capture image data of the packages 112a, 112b on the depalletizing platform 110. The imaging device 161 can capture distance data, position data, video, still images, lidar data, radar data, and / or motion at the pick-up environment 163 or pick-up area. It should be noted that although the terms "object" and "package" are used herein, these terms include any other item that can be gripped, lifted, transported, and delivered, such as but not limited to "boxes", "crates", "cartons", or any combination thereof. Additionally, although polygon boxes (e.g., rectangular boxes) are depicted in the figures disclosed herein, the shape of the boxes is not limited to this shape but includes any regular or irregular shape that can be gripped, lifted, transported, and delivered as discussed in detail below.

[0053] Similar to the unstacking platform 110, the receiving conveyor 120 can include any platform, surface, and / or structure designated to receive packages 112a, 112b for further tasks / operations. In some embodiments, the receiving conveyor 120 can include a conveyor system for transporting the target object 112 from one location (e.g., the release point) to another location for further operations (e.g., sorting and / or storage). In some embodiments, the robotic system 100 can include a second imaging system (not shown) configured to provide image data captured from a target environment having a target placement location (e.g., the conveyor 120). The second imaging system can capture image data of the packages 112a, 112b at the receiving / placement location (e.g., the receiving conveyor 120).

[0054] Task implementation process

[0055] Figure 4A An exemplary first stack 400 in accordance with one or more embodiments of the present technology is illustrated. Figure 4B Exemplary image data 406 depicting the first stack 400 in accordance with one or more embodiments of the present technology is illustrated. Now referring together Figure 4A and Figure 4B , the image data 406 can represent a top view of the first stack 400 at a starting position (e.g., Figure 3 the unstacking platform 110). The image data 406 can include 2D and / or 3D data from Figure 3 the imaging system 160.

[0056] As Figure 4A shown, the stack 400 (e.g., an object stack) includes objects 402, which include objects (e.g., objects 402-1, 402-2, and 402-3) arranged in an organized stack. In Figure 4A the stack 400 illustrated can correspond to an object stack positioned at a starting position (e.g., Figure 1 the starting position 114) before any of the objects 402 in the object stack 400 are transported to a task location (e.g., Figure 1 the task location 116). The image data 406 thus represents a top view of the stack 400 at the starting position at a point in time such as before any of the objects are picked up from the stack 400 (e.g., the image data 406 can be referred to as prior image data). The objects 402 can be arranged to satisfy optimization conditions such as minimizing the overall volume of the stack 400. In Figure 4AIn [description], the object 402 is arranged such that the object stack 400 has a coplanar or substantially coplanar top surface 400-A. The coplanar top surface 400-A is composed of the respective coplanar or substantially coplanar top surfaces 402-1A, 402-2A, and 402-3A of the objects 402-1, 402-2, and 402-3 arranged adjacent to each other. In Figure 4A [description], the object stack 400 has dimensions corresponding to the stack height (H S ), the stack width (W S ), and the stack length (L S ).

[0057] In some embodiments, the image data 406 may include a depth map that represents the distance between the imaging system 160 and the detected surfaces / points of the objects within the field of view of the imaging system 160. For example, as described above, the imaging device 222 may generate a representation of the environment detected in an image corresponding to the depth map and / or point cloud. The depth map may include depth measurements (e.g., along the Z direction) at discrete points along a transverse plane (e.g., at the positions 'x', 'y', and 'z' of the XY plane illustrated in Figure 4B [description]). For the example illustrated in Figure 4B [description], the image data 406 may depict the coplanar top surfaces (e.g., matching depth measurements) of nine objects forming the top surface of the stack 400 (e.g., the respective coplanar or substantially coplanar top surfaces 402-1A, 402-2A, and 402-3-A of the objects 402-1, 402-2, and 402-3). The matching depth may correspond to the stack height ( Figure 4A the H in s ). The image data 406 may also depict the depth measurements of the top surface of the placement platform 404 (e.g., the tray shown filled with a dashed line). In 3D image data, the adjacent or adjoining edges of the objects within the stack may or may not be easily detectable. The stack height H S may correspond to the vertical distance between the corresponding top surface of the object / stack (e.g., surface 400-A) and the top surface of the placement platform (e.g., the top surface of the placement platform 404).

[0058] The robot system 100 can use the image data 406 to detect the objects 402 in the stack 400. Object detection can include estimating the identity and / or location of the objects depicted in the image data 406. In some embodiments, the robot system 100 can process the image data 406 (e.g., 2D and / or 3D depictions) to identify the corners and / or edges / lines depicted therein (e.g., the peripheral edges of the stack or its top layer). Such identification can include identifying the corners and edges of the stack 400 and / or identifying the corners and edges of the objects 402 in the stack 400. The robot system 100 can process the corners and / or edges to estimate the surface or peripheral boundary of each of the depicted objects. The robot system 100 can use the estimated boundaries to estimate the bounded surface (e.g., the top surface) of each of the depicted objects. For example, the robot system can estimate the peripheral boundaries of the surfaces 402-1A, 402-2A, and 402-3A of the objects 402-1, 402-2, and 402-3, respectively, within the coplanar surface 400-A of the stack 400. For example, the identification can include analyzing the 3D data of the image data 406 to identify stack corners, identifying edges within the 2D visual representation of the stack 400 (e.g., via a Sobel filter), comparing portions of the 2D visual representation to templates of known objects within the main data, or a combination thereof. Moreover, for example, the identification can include applying other image detection methods, including, for example, algorithms for identifying the corners of boxes and packages. In addition, such image detection methods may be able to distinguish the corners and edges of an object from visual features on the object. For example, the robot system can distinguish a flap, tape, or other visual feature on the surface of an object from the actual edge of the object.

[0059] The robot system 100 can process the unrecognized / unmatched portions of the image data as corresponding to one or more unrecognized or unexpected objects. For example, the unrecognized portion of the image data 406 may correspond to an object with an irregular shape or a damaged object. The robot system 100 can automatically or autonomously register unexpected objects during manipulation or task execution. For example, the robot system 100 can deduce the minimum viable region (MVR) for gripping an unexpected object. The robot system 100 can use the MVR to grasp and lift the object and / or transport the object from a starting position to a task position. The robot system 100 can detect the actual edges, corresponding dimensions (e.g., lateral dimensions), and / or visual surface images (e.g., corresponding portions of the image data) of the object based on the movement of the unrecognized object. For example, the robot system 100 can compare images taken before and after removing / moving the unrecognized object to deduce the dimensions (e.g., lateral dimensions and / or height) of the object. The robot system 100 can further determine the height of the object during transportation, such as using cross / line sensors and / or side cameras. The robot system 100 can obtain other measurements or estimates during object transportation, such as weight, CoM position, etc.

[0060] The robot system 100 can use additional information describing the content of the stack, such as a shipping manifest, an order receipt, a task tracker (e.g., the history corresponding to the removed / transported object), etc., to process objects (such as recognized and / or unrecognized objects). For example, the robot system 100 can determine a preliminary list of expected objects based on the content description of the stack. During object detection, the robot system 100 can compare the image data with the registered descriptions of the objects on the preliminary list before other objects.

[0061] The robot system 100 can use object detection, the results from processing the image data, master data, stack descriptions, and / or additional descriptive data to infer additional information about the stack, the objects therein, and / or the status of task execution. For example, the robot system 100 can estimate the number of objects in the stack and / or the arrangement of the objects in the stack.

[0062] Quantity / layout estimation

[0063] As an illustrative example of quantity estimation, the first stack 400 can include a set of single or common stock keeping units (SKUs) (e.g., a stack of objects of the same / common type). Thus, the objects 402 within the first stack 400 have the same dimensions, the same surface characteristics, the same weight, etc.

[0064] The robotic system 100 may detect the individual SKU composition of the first stack 400 based on one or more factors such as supplier data, shipping manifests, the shape or appearance of the stack, etc. Once detected, the robotic system 100 may use the commonalities of the stacked objects 402 to derive additional information. For example, the robotic system 100 may estimate the number of objects 402 within the stack 400 based on one or more dimensions of the objects and / or the calculated volume of the stack. The robotic system 100 may use the depth map to determine the outer edges of the stack 400 and the height at various locations of the stack 400. The robotic system 100 may derive the number of objects 402 within the stack 400 that have the same depth measurement (e.g., height H within a threshold range). O ) of the region (e.g., length L O and width W O ). The robotic system 100 may use the lateral dimensions and the corresponding heights to calculate the volume of the corresponding regions. The robotic system 100 may combine the calculated volumes of the regions across the stack to calculate the total volume. The robotic system 100 may calculate the total volume based on dividing the total volume by the volume of one object (e.g., the object length (L o ), object height (H o ) and object width (W o ) to calculate the estimated number of objects. For example, in Figure 4A In the embodiment, the object stack 400 has a height corresponding to the stack height (H S )、Stacking width (W s ) and stacking length (L S ) and the volume (V S ) corresponds to V S =H S x W s xL S Note that the volume of a stack of objects is defined based on the outermost edges and surfaces and does not take into account any spacing or gaps within the stack. The volume of an individual object (e.g., object 402-1) in stack 400 has a volume corresponding to the object height (H O ), object width (W O ) and object length (L O ) and the volume of the object (V O ) corresponds to V O =H O x W O xL O .

[0065] The robot system 100 can also estimate the number of objects within a mixed SKU stack (e.g., a stack of various types of objects). The objects within the mixed SKU stack can have different sizes, surface features, weights, etc. The robot system 100 can estimate the stacking pattern based on the information obtained (such as stack description, regions of the same height (e.g., its lateral dimensions, shape, and / or height), detection of the topmost object, etc.). The robot system 100 can estimate the stacking pattern based on comparing the information obtained with a set of predefined templates and / or based on processing the information obtained according to a set of stacking rules. Additionally or alternatively, the robot system 100 can use the volume of the expected SKUs to calculate a combination of the number of SKUs having a total volume that matches the total volume of the stack.

[0066] As an illustrative example, Figure 5A an exemplary second stack 500 is depicted, and Figure 5B exemplary image data 502 depicting the second stack 500 according to one or more embodiments of the present technology is shown. Now referring together to Figure 5A and Figure 5B , the second stack 500 can be a mixed SKU stack that includes objects 504 of type 'A', type 'B', and type 'C'. As discussed above, the robot system 100 can use the image data, stack height measurements, object detection results, the overall shape of the stack, etc. to identify groupings of objects or corresponding regions and / or the stack configuration.

[0067] In some embodiments, a predefined stacking configuration can specify areas / columns on a pallet where one type of object is grouped (shown using dashed lines in Figure 5B ). For example, objects 504 of type A are grouped together in the first column, objects 504 of type B are grouped together in the second column, and objects 504 of type C are grouped together in the third column. Additionally or alternatively, a shipping manifest or stack description can indicate the stacking configuration of the objects according to one or more predefined patterns. The robot system 100 can use the height and lateral dimensions of the areas (e.g., the height and lateral dimensions of the first column, second column, and third column) and calculate the volume of the areas according to the stacking configuration. The robot system 100 can divide the volume of the area by the volume of the corresponding object. Alternatively, the robot system 100 can calculate different combinations of quantities for the corresponding objects within the area to find a combination of quantities having a combined volume that matches the combined volume of the area. The robot system 100 can add the quantities of each SKU across different areas to estimate the total number of objects of each type of SKU.

[0068] State estimation

[0069] In some embodiments, the robotic system 100 may use the acquired data to determine the task execution status, such as for verifying object pick-up and / or placement. For object pick-up / placement, the robotic system 100 may acquire a set or series of image data obtained at different times (e.g., before and / or after a set of pick-up / placements).

[0070] For context, the robotic system 100 may deduce the handling sequence and / or packing configuration (e.g., a set of placement positions for each object to be placed at the task location) and a motion plan for the target object. Each motion plan may include a set of commands and / or settings for operating the robotic unit (e.g., Figure 1 the handling unit 104, Figure 3 the robotic arm system 132, etc.) to move a corresponding set of objects from a starting position to the task location. Thus, the start portion of the motion plan may correspond to the pick-up operation, and / or the end portion of the motion plan may correspond to the placement operation. The robotic system 100 may deduce the motion plan such that the placement operation follows the packing configuration. Additionally, the motion plans may be sorted according to the handling sequence.

[0071] The robotic system 100 may use the pick-up / placement history that follows the said handling sequence to track the execution of the motion plan. The history may indicate which object was picked up from which location or which object was placed at which location over time. The robotic system 100 (e.g., using a module / process separate from the module / process that performs the handling) may acquire and process additional image data during and / or after the execution of the motion plan. The robotic system 100 may compare the tracked history with the image data to verify the pick-up / placement of the objects that occurred between the image times.

[0072] For an example of verification, Figure 6A a first image 602 of the starting position is shown, and Figure 6B a second image 604 of the starting position according to one or more embodiments of the present technology is shown. Similarly, Figure 7A a first image 702 of the task location is shown, and Figure 7B a second image 704 of the task location according to one or more embodiments of the present technology is shown. The first images 602 / 702 may depict the corresponding starting / task location before picking up or placing one or more objects, and the second images 604 / 704 may depict the location after picking up or placing one or more objects. In some embodiments, the first image and the second image may be consecutive and represent two back-to-back images taken during the task execution process.

[0073] For example, the first image 602 depicts a stack 606 (e.g., indicated as stack 606-1) corresponding to the stack at the starting position before picking up any object. As shown, in FIG. 6A, stack 606-1 includes objects 1, 2, 3, and 4. The second image 604 depicts the stack 606 (e.g., indicated as stack 606-2) after picking up a group of one or more objects. As shown, Figure 6B illustrates the state of stack 606-2 after objects 1 and 2 have been picked up from stack 606-2 (i.e., objects 1 and 2 are not shown in Figure 6B and object 4 has been displaced. The displacement of object 4 can be due to an accidental / unplanned event resulting from picking up object 1 and / or 2. The first image 702 depicts a stack 706 (e.g., indicated as stack 706-1) corresponding to the stack at the task position after placing one or more objects, and the second image 704 depicts the stack 706 (e.g., indicated as stack 706-2) after placing one or more additional objects. As shown, in Figure 7A stack 706-1 includes objects 1 and 2 (e.g., objects that have been placed on stack 706-1). Figure 7B It also includes object 3 that is subsequently placed in stack 706-2.

[0074] The robotic system 100 can compare depth metrics at and / or around the picked-up object (illustrated using different shadings in Figures 6A to 7B ) to verify object pick-up. The comparison of depth metrics can include the comparison of depth metrics at specific locations on the top surface of the object. For example, when the difference in depth metrics across the first and second images 602 and 604 is greater than the pick-up verification threshold, the robotic system 100 can determine that the target object has been successfully picked up. The pick-up verification threshold can correspond to the expected minimum height of the object. For example, the object at the expected pick-up position has a specific minimum height. If the difference in depth metrics across the first and second images 602 and 604 is equal to or greater than the specific minimum height, the robotic system 100 can determine that the target object has been successfully picked up. The pick-up verification threshold can also correspond to the expected minimum height of the corresponding target object. In some embodiments, the robotic system 100 can identify the comparison locations of one or more objects (e.g., in Figures 6A to 7BThe positions depicted using the symbols {a, b, c, ...} within the dashed circles in [description], including positions 608-a and 608-b). Instead of comparing depth measures across the entire depth map / image, the robotic system 100 can compare depth measures at comparison positions to improve processing efficiency. The robotic system 100 can identify comparison positions as positions (e.g., X-Y coordinates) at or within a threshold distance / orientation from an object's edge or corner. The robotic system 100 can identify comparison positions on one or both sides of the estimated outer perimeter of the picked-up object. In some embodiments, the robotic system 100 can identify comparison positions as including the estimated central portion of object detection and / or MVR. Additionally or alternatively, the robotic system 100 can compare the difference in depth measures with the height of the picked-up object (e.g., when the object matches the registration record in the master data) to verify that the picked-up object matches the detected object. When the difference in depth measures matches the height of the picked-up object, the robotic system 100 can further verify a successful pick-up and / or a pick-up without damage (e.g., indicating that no object has been crushed).

[0075] The robotic system 100 can compare heights at specific positions, such as comparison positions with respect to each estimated object corner, to verify the pick-up and / or changes of surrounding objects (e.g., object displacement or crushed objects). Using Figure 6A and Figure 6B to illustrate an example of pick-up verification, due to the difference in depth measures at the comparison positions {g, h, j, k} of object 2 and the constant depth measures at the surrounding positions {i, l, m, n, o}, the robotic system 100 can detect a valid pick-up of object 2. In contrast, as depicted for object 1, when the depth measures at the comparison positions (e.g., positions {a, b, c, d}) and / or the surrounding positions (e.g., {e, f, p, q}) are inconsistent with the pick-up, the robotic system 100 can detect a pick-up anomaly. In response to anomaly detection, the robotic system 100 can additionally examine the depth measures and / or visual images to determine potential failure modes.

[0076] In some embodiments, when the depth measure of an object position (e.g., position {a, b, c, d}) remains constant across an image (not shown), the robotic system 100 can determine that the robotic system 100 has failed to pick up an object (e.g., object 1). When eliminating the failed pick-up pattern, when the difference in the depth measure at the object position (e.g., position {a, b, c, d}) does not match the expected height of the removed object 1, the robotic system 100 can determine a potential misidentification or crushed object. When the depth measure of the removed object in the second image 604 (e.g., representing the top surface of a newly exposed object below the removed object) has a non-planar pattern, the robotic system 100 can determine a misidentification. Otherwise, the robotic system 100 can process the data to determine whether surrounding objects have shifted or whether underlying objects have been crushed. When the height of the surrounding objects matches the position (e.g., position {c}) corresponding to the removed object, the robotic system 100 can determine that the surrounding objects (e.g., Figure 6B object 4 in) have shifted, the difference in the depth measure indicates different positions of the surface, and / or other positions of the surrounding objects (e.g., positions {f, p, q}) correspond to unexpected differences. In some embodiments, the robotic system 100 can cache or store the object shift and adjust one or more corresponding motion plans accordingly, such as by adjusting the pick-up / gripping pose.

[0077] As an additional example of exception handling (in Figure 6A and Figure 6B(not shown in the figure), when the difference in depth measurements at {a, b, c, d} represents a plane and does not match the expected height of object 1 (e.g., according to object detection), the robot system 100 can determine a potential misdetection of object 1. When the difference in depth measurements at {a, b, c, d} or other points of the object to be picked up is abnormal or does not correspond to a plane pattern, the robot system 100 can determine a potential crushed object under the picked-up object. When the depth measurements of the surrounding unpicked objects (e.g., positions {i, l, e, f}) vary across the image and / or when the height of the surface of the surrounding objects in the second image 604 does not correspond to a planar surface, the robot system 100 can determine a potential crushed object around the picked-up object. When the difference in depth measurements at the start position across the pick-up operation does not match (e.g., is greater than) the difference in depth measurements at the task position across the corresponding place operation, the robot system 100 can further determine the carried object as a potential crushed object. When the difference in depth measurements of the surrounding objects (e.g., positions {i, l, e, f}) matches the height of the surrounding objects and / or the corresponding lateral shape / size matches the lateral shape / size of the surrounding objects, the robot system 100 can determine a double pick-up. Double pick-up corresponds to an instance when the end effector of the robotic arm grasps and carries one or more unplanned objects together with the target object. For example, the robotic arm inadvertently grasps and carries object 2 together with the expected object 1.

[0078] Similar to the pick-up verification process, the robot system 100 can use the difference in depth measurements at the task position to verify object placement and / or determine associated failures. Using Figure 7A Figures 7A and 7B for illustrative examples, when the change in depth measurements at the target placement position (e.g., position {y, z}) matches the expected height of object 2, the robot system 100 can determine the placement verification of object 2. When the depth measurements at other comparison positions remain constant across image 702 and image 704, the robot system 100 can further verify the placement and / or eliminate other failure modes. The robot system 100 can use an analysis similar to the pick-up anomalies described above to determine potential error modes, such as crushed objects, displacement of previously placed objects, mislocated target objects, misidentified target objects, etc.

[0079] The robot system 100 can use a defined failure mode to control / implement subsequent operations. For example, the robot system 100 can obtain and / or analyze additional data, such as the weight of the object being handled, additional images, updated object detection, etc. Moreover, the robot system 100 can determine and initiate new processes, such as abandoning the current set of plans (e.g., motion plans, packing configurations, handling sequences, etc.), notifying a human operator, determining / implementing error recovery (e.g., adjusting one or more objects at the start / task location), etc.

[0080] Operation process

[0081] Figure 8 is a flowchart of an exemplary method 800 for operating a robot system (e.g., Figure 1 the robot system 100) according to one or more embodiments of the present technology. The method 800 can be used to derive estimators (e.g., the number of objects and / or the implementation status) based on one or more measurements (e.g., depth metrics) captured during task execution. It can be implemented based on instructions executed by one or more of the processors 202 in Figure 2 and stored on one or more of the storage devices 204 in Figure 2 . When implementing the motion plan and / or the method 800, the processor 202 can send the motion plan or a related set / series of commands / settings to the robot unit (e.g., Figure 3 the handling unit 104 and / or Figure 3 the end effector 140). Thus, the handling unit 104 and / or the end effector 140 can execute the motion plan to grasp and handle the package.

[0082] At block 802, the robot system 100 can obtain a stack description, such as a shipping list, an order receipt, etc. At block 804, the robot system 100 can obtain initial image data depicting one or more corresponding locations (e.g., at t = x and the start location and / or task location before performing one or more tasks). The robot system 100 can use an imaging system, sensors, and / or cameras. The obtained image data can include 2D images and / or 3D images (e.g., depth maps) depicting the object stack and / or the corresponding platform (e.g., cabinets, pallets, conveyors, etc.). The stack description can identify whether the stack is a regular SKU (e.g., Figure 4A the stack 400 in Figure 5A ) or a mixed SKU (e.g., Figure 5A and Figure 5BAs shown, objects 504 of type A are grouped together in the first column, objects 504 of type B are grouped together in the second column, and objects 504 of type C are grouped together in the third column.

[0083] At block 806, the robotic system 100 may process the acquired image to detect the depicted objects. For example, the robotic system 100 may detect edges, identify surfaces, and / or compare the image of the surface to master data to detect objects. Detecting the objects may include identifying the type or SKU and estimating their real-world position based on image processing.

[0084] At block 808, the robotic system 100 may estimate the number of objects at least in part based on the initial image and / or the object detection results. The robotic system 100 may estimate the number of objects based on depth metrics, the arrangement of the detected objects, the estimated arrangement of the objects, the number of SKUs in the stack, etc., as described above with respect to Figures 4A to 5B In some embodiments, the robotic system 100 may use the estimated number to process subsequent processing steps, such as for coordinating storage locations, storage containers, corresponding transport mechanisms, etc. The robotic system 100 may further use the estimated number to verify the task plan (e.g., task sequence) and / or task execution / progress.

[0085] At block 810, the robotic system 100 may derive a plan for the objects in the stack (e.g., a motion plan, a handling sequence, a packing plan, etc.). The robotic system 100 may derive the plan based on a predetermined process, such as by deriving the placement location of each object that satisfies a set of predetermined rules; deriving an object handling sequence to achieve a packing plan; and / or by iterating through potential locations from the placement location / pose to the starting location to derive a motion plan.

[0086] At block 812, the robotic system 100 may handle the objects in the stack, such as by implementing the motion plan in the planned order. At block 814, the robotic system 100 may obtain subsequent image data during object handling (e.g., implementing the motion plan). The robotic system 100 may obtain images (e.g., 2D images and / or 3D depth maps) before and / or after handling one or more subsets of the objects in the stack (e.g., as described with respect to Figures 6A to 7B . The robotic system 100 may obtain a description of the starting location and the remaining objects and / or a description of the task location and the newly placed objects.

[0087] At block 816, the robotic system 100 may compare the acquired image data to the previous image. For example, the robotic system 100 will compare the second image 604 obtained after picking up one or more objects from the starting location to Figure 6B inFigure 6A Compare with the first image 602 in []. Similarly, after placing one or more objects from the starting position, the robot system 100 will obtain the Figure 7B Compare the second image 704 in [] with the Figure 7A First image 702 in []. For example, the robot system 100 can compare depth metrics, such as by calculating the difference in depth metrics across images as described above. Calculating the difference in depth metrics can include calculating the difference in depth metrics across the surfaces of the objects in the depicted stack. Alternatively, calculating the difference in depth metrics can include calculating the difference in depth metrics at specific locations (e.g., Figure 6A Positions {a, b, c, d,...} in []) on the surfaces of the objects in the depicted stack.

[0088] At decision block 818, the robot system 100 can analyze the comparison to determine whether the corresponding pick and / or place can be verified. For example, the robot system 100 can evaluate the change in depth metrics at one or more positions associated with and / or around the picked / placed object for verification as described above. When the pick / place is verified, the robot system 100 can continue with the originally planned handling without any adjustments. When it is determined that the pick / place is invalid, the robot system 100 can determine an error mode, as depicted at block 820. The robot system 100 can analyze depth metrics, images, pick history, or other data / results to determine an appropriate error mode, such as for crushed objects, displaced objects, object misdetection, etc. as described above.

[0089] As described with respect to Figure 6A And Figure 6B The robot system 100 can compare the heights at specific positions of surrounding objects (e.g., Figure 6A Comparison positions at the corners of the objects in []) to verify the pick and / or change of surrounding objects. For example, referring to Figure 6A And Figure 6B , due to the difference in depth metrics at the comparison positions {g, h, j, k} of object 2 and the constant depth metrics at the surrounding positions {i, l, m, n, o}, the robot system 100 can detect an effective pick of object 2. When the measured depth at the comparison position (e.g., position {a, b, c, d}) and / or the surrounding position (e.g., {e, f, p, q}) is inconsistent with the pick, the robot system 100 can detect a pick anomaly. For example, when the depth metric at the object position (e.g., position {a, b, c, d} of object 1) remains constant across images, the robot system 100 can determine that the robot system 100 failed to pick up the object (e.g., Figure 6A And Figure 6BThe object in 1). Another pick-up anomaly may include misidentification of the picked-up object. For example, when the depth measure of the object removed in the second image 604 (e.g., representing the top surface of the newly exposed object below the removed object) has a non-planar pattern, the robotic system 100 determines misidentification. The non-planar pattern may be an indication that surrounding objects have been displaced and / or objects below the object being carried have been damaged. Yet another pick-up anomaly may include false detection of the picked-up object. For example, when the difference in depth measures at the location of the object being carried (e.g., the location {a, b, c, d} at object 1) represents a flat plane that does not match the expected height of the object being carried (e.g., according to object detection), the robotic system 100 may determine false detection of the object being carried. When the difference in depth measures of surrounding objects (e.g., the location {i, l, e, f} in FIG. 6B) matches the height of the surrounding objects and / or the corresponding lateral shape / size matches the lateral shape / size of the surrounding objects, the robotic system 100 may also determine double pick-up. This indicates that the surrounding objects have been inadvertently removed together with the object being carried.

[0090] At block 822, the robotic system 100 may implement an error response based on the determined error pattern. The robotic system 100 may implement the error response according to a predetermined set of rules. The implemented response may include notifying a human operator, abandoning the handling plan, and / or implementing an error recovery process. Some exemplary error recovery processes may include re-detecting the object, re-planning the handling, re-gripping the object, re-stacking the displaced objects, removing obstacles or crushed objects, etc. When the recovery process is successful and / or when the detected error pattern indicates conditions for continued implementation, the robotic system 100 may continue to implement handling the object. In some embodiments, the robotic system may cache the differences, such as by calculating the distance and / or change in pose of the displaced objects. The robotic system may adjust the corresponding motion plan according to the calculated change measures, such as by changing the approach and gripping positions. The robotic system may continue to handle the object using the adjusted motion plan.

[0091] Implementation plan

[0092] According to some embodiments, a method of operating a robotic system (e.g., Figure 1 the robotic system 100 in) includes obtaining image data for the start position of the object stack at the start position for transporting an object from the object stack at the start position to the object stack at the task position (e.g., regarding Figure 6A and Figure 6B the images 602 and 604 described) and image data for the task position (e.g., regarding Figure 7A and Figure 7BThe described images 702 and 704). The image data includes a series of images depicting the movement of one or more objects from a starting position to a task position. The series of images includes a first set of images depicting the starting position and the task position before the movement of one or more objects (e.g., Figure 6A and FIG. 7A) and a second set of images depicting the starting position and the task position after the movement of one or more objects (e.g., Figure 6B and Figure 7B ). The image data includes depth measures determined from each image in the series of images. The depth measures represent the estimated surface height at the starting position and the estimated surface height at the task position (e.g., the estimated heights depicted using different fills in Figures 6A to 7B ). The method includes determining a first depth difference between depth measures representing the estimated surface height at the starting position in the first and second sets of images (e.g., the depth difference between the first image 602 and the second image 604 at positions {g, h, j, k} in FIGS. 6A and Figure 6B ). The method includes verifying that the target object among the one or more objects has been successfully picked up from the starting position based on determining that the first depth difference is greater than a pick verification threshold (e.g., object 2 has been picked up from the starting position). The method includes determining a second depth difference between depth measures representing the estimated surface height at the task position in the first and second sets of images (e.g., the depth difference between the first image 702 and the second image 704 at positions {y, z} in Figure 7A and Figure 7B ). The method includes verifying that the target object among the one or more objects has been placed at the task position based on determining that the second depth difference is greater than a place verification threshold (e.g., object 2 has been placed at the task position). For example, according to the motion plan, the target object among the one or more objects is placed at the task position of the target object. Similar to the pick verification threshold described above, the place verification threshold may correspond to the expected minimum height of the object. For example, the object expected to be moved has a specific minimum height. If the difference in depth measures across the first and second images 702 and 704 is equal to or greater than the specific minimum height, the robot system 100 may determine that the target object has been successfully placed at the task position. The place verification threshold may also correspond to the expected minimum height of the respective object being moved.

[0093] In some embodiments, the method further includes estimating the number of objects within the object stack at the starting position and / or at the task position. The estimation includes based on the image data (e.g., Figure 5Bthe image data 502) to determine the outer edge of the object stack (e.g., object stack 500) and the height of the object stack (e.g., height H S ). The method includes deriving the volume of the object stack based on the outer edge and height of the object stack. The method includes determining the lateral dimensions of different regions within the object stack (e.g., different regions corresponding to a first column including objects 504 of type A, a second column including objects 504 of type B, and a third column including objects 504 of type C). The surface heights of the different regions are the same or the height variations in one or more lateral directions have a linear pattern. The method includes determining the volume of the corresponding different regions within the object stack based on the lateral dimensions of the different regions (e.g., the lateral horizontal surfaces depicted by the dashed lines in Figure 5B ) and the corresponding estimated heights of the outer vertical surfaces. The method includes deriving an estimate of the number of objects within the object stack based on the derived volume of the object stack and the volumes corresponding to the respective different regions within the object stack.

[0094] In some embodiments, estimating the number of objects within the object stack further includes estimating the stacking pattern of the object stack. The estimation is accomplished by comparing the determined outer edge of the object stack, the volume of the object stack, and the lateral dimensions of the different regions within the object stack with a set of predefined templates and / or a set of stacking rules (e.g., the master data of the robot system 100 includes data regarding the predefined templates and / or the set of stacking rules). The method includes deriving an estimate of the number of objects within the object stack based on the stacking pattern and the volume of the object stack. The objects within the object stack include objects of different sizes (e.g., objects 504 of type A, type B, and type C have different shapes and / or sizes).

[0095] In some embodiments, the method further includes comparing the estimate of the number of objects within the object stack with tracking history data (e.g., the tracking history of the master data of the robot system 100) identifying one or more objects that have been transported from a starting position to a task position to verify whether the tracking history data is accurate.

[0096] In some embodiments, the method further includes determining that the target object of one or more objects has not been successfully placed at the task position based on determining that a second depth difference is less than a placement verification threshold.

[0097] In some embodiments, the method includes deriving the dimensions of a target object based on image data and comparing the derived dimensions of the target object with tracking historical data identifying a previously handled object. When the derived dimensions are different from the dimensions of the previously handled object, the method includes determining that the target object has been mis-identified based on determining that the derived dimensions of the target object do not correspond to one or more of the identified objects expected to be handled.

[0098] In some embodiments, the method includes estimating a perimeter edge of a target object that is verified to have been successfully picked up at a starting position (e.g., Figure 6A and Figure 6B illustrates the removal of an object 2 between corresponding stacks 606-1 and 606-2). The method includes identifying a comparison position adjacent to the perimeter edge of the target object at the starting position. The comparison position corresponds to an object positioned adjacent to the target object (e.g., the comparison position {m,n} corresponding to object 3 in Figure 6B . The method includes comparing an estimated height at the comparison position before and after the target object has been picked up from the starting position to identify whether the object corresponding to the comparison position has an anomaly.

[0099] In some embodiments, the method further includes determining that a difference between the estimated heights at the comparison position before and after the target object has been picked up corresponds to the height of an adjacent object that was initially positioned adjacent to the target object. Based on this determination, the method includes determining that the adjacent object has been inadvertently picked up with the target object.

[0100] In some embodiments, the method further includes determining a difference between the estimated heights at the comparison position before and after the target object has been picked up. Based on determining that this difference is greater than a minimum height change requirement and / or that the estimated heights at the comparison position before and after the target object has been picked up do not correspond to a flat surface, the method includes determining that an object adjacent to the target object has been damaged. The minimum height change requirement may correspond to a threshold for classifying a height difference across different images (e.g., different time points) as having processing significance. In other words, the robotic system 100 may use the minimum height change requirement as a filter to block out measurement noise or other less significant height changes. Thus, the minimum height change requirement may correspond to a value just above the typical measurement error (e.g., standard deviation or average deviation) of the difference in the estimated heights of an object determined from two different images. For example, when a first difference is within the minimum height change requirement, the system may determine that the first difference corresponds to measurement error. When the first difference is greater than the minimum height change requirement, the system may determine that the adjacent object has been displaced from its position.

[0101] In some embodiments, the comparison locations include a first subset of comparison locations (e.g., comparison locations {p,q} in FIG. 6B) and a second subset of comparison locations (e.g., comparison locations {e,f}). The first subset of comparison locations and the second subset of comparison locations are associated with a surface of a neighboring object that positions the location adjacent to the target object. The method further includes determining a first difference between estimated heights at the first subset of comparison locations before and after the target object has been picked up, and a second difference between estimated heights at the second subset of comparison locations before and after the target object has been picked up. The method includes determining that the neighboring object has shifted (e.g., object 4 has shifted during the removal of object 1) based on determining that the first difference is within a minimum height change requirement and the second difference is greater than the minimum height change requirement.

[0102] According to some embodiments, a robotic system (e.g., Figure 1 robotic system 100 in Figure 2 ) includes at least one processor (e.g.,

[0103] processor 202 in

[0104] Conclusion

[0105] The above detailed description of examples of the disclosed technology is not intended to be exhaustive or to limit the disclosed technology to the precise forms disclosed above. While the above description has described specific examples of the disclosed technology for illustrative purposes, those skilled in the art will recognize that various equivalent modifications are possible within the scope of the disclosed technology. For example, although a process or block is presented in a given order, alternative implementations may execute routines with steps in a different order, or systems with blocks in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternatives or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, although processes or blocks are sometimes shown as being executed serially, these processes or blocks may alternatively be executed or implemented in parallel, or at different times. Additionally, any specific numbers mentioned herein are merely examples; alternative implementations may employ different values or ranges.

[0106] These and other changes may be made to the disclosed technology in light of the above detailed embodiments. While the detailed embodiments describe certain examples of the disclosed technology and the contemplated best mode, the disclosed technology may be practiced in many ways regardless of how detailed the above description may appear in the text. Details of the system may vary significantly in its detailed embodiments while still being encompassed by the technology disclosed herein. As noted above, specific terms used in describing certain features or aspects of the disclosed technology should not be taken to imply that the term is redefined herein to be limited to any specific characteristic, feature, or aspect of the disclosed technology associated with that term. Thus, the invention is not limited except as by the appended claims. In general, unless the terms used in the appended claims are explicitly defined in the detailed embodiments section above, the terms should not be construed as limiting the disclosed technology to the specific examples disclosed in the specification.

[0107] While certain aspects of the invention are presented below in the form of certain claims, the applicant contemplates aspects of the invention in any number of claim forms. Accordingly, the applicant reserves the right to seek additional claims after filing this application to pursue such additional claim forms in this application or in a continuing application.

Claims

1. A method of operating a robotic system, the method comprising: Obtain image data of the start position and the task position of the object stack for transporting an object from the object stack at the start position to the object stack at the task position, where: The image data includes a series of images corresponding to transporting one or more objects from the start position to the task position, the series of images including a first image set depicting the start position and the task position before transporting the one or more objects, and a second image set depicting the start position and the task position after transporting the one or more objects; and The image data includes depth measures representing an estimated height of the surface at the start position and an estimated height of the surface at the task position; Determine a first depth difference between the depth measures representing the estimated height of the surface at the start position in the first image set and the second image set; Verify that the target object among the one or more objects is transported from the start position based on determining that the first depth difference is greater than a pick verification threshold; Determine a second depth difference between the depth measures representing the estimated height of the surface at the task position in the first image set and the second image set; and Verify that the target object among the one or more objects is placed at the task position based on determining that the second depth difference is greater than a place verification threshold.

2. The method according to claim 1, the method further comprising estimating the number of objects within the object stack at the start position and / or at the task position by the following operations: Determining the outer perimeter edge of the object stack and the height of the object stack based on the image data; Derive the volume of the object stack based on the peripheral edge and the height of the object stack; Determine the lateral dimensions of different regions within the object stack, where the surface heights of the different regions are the same or the height variations in one or more lateral directions have a linear pattern; Determine the volumes of the corresponding different regions within the object stack based on the lateral dimensions of the different regions and the corresponding estimated heights of the peripheral vertical surfaces; And Derive an estimate of the number of objects within the object stack based on the derived volume of the object stack and the volumes corresponding to the respective different regions within the object stack.

3. The method according to claim 2, wherein estimating the number of objects within the object stack further comprises: Estimate the stacking pattern by comparing the determined peripheral edge of the object stack, the volume of the object stack, and the lateral dimensions of the different regions within the object stack with a set of predetermined templates and / or a set of stacking rules; And Derive an estimate of the number of objects within the object stack based on the stacking pattern and the volume of the object stack, where the objects within the object stack include objects of different sizes.

4. The method according to claim 3, the method further comprising comparing the estimate of the number of objects within the object stack with tracking history data identifying one or more objects that have been transported from the start position to the task position to verify whether the tracking history data is accurate.

5. The method according to claim 1, the method further comprising: Determine that the target object among the one or more objects is not placed at the task position based on determining that the second depth difference is less than the place verification threshold.

6. The method according to claim 1, the method further comprising: Derive the size of the target object based on the image data; Compare the derived size of the target object with tracking history data identifying previously transported objects; And Determine that the target object has been misidentified when the derived size is different from the size of the previously transported object.

7. The method according to claim 1, the method further comprising: Estimate the peripheral edge of the target object verified to be transported from the start position from the image data; Identify a comparison position adjacent to the peripheral edge of the target object at the starting position, the comparison position corresponding to an object positioned adjacent to the target object; And Compare the estimated heights at the comparison position before and after the target object has been moved from the starting position to identify whether the object corresponding to the comparison position has an anomaly.

8. The method according to claim 7, wherein the method further comprises: Determine that the neighboring object has been inadvertently moved with the target object based on determining that the difference between the estimated heights at the comparison position before and after the target object has been moved corresponds to the height of the neighboring object initially positioned adjacent to the target object.

9. The method according to claim 8, wherein the method further comprises: Determine that an object adjacent to the target object has been damaged based on determining that the difference between the estimated heights at the comparison position before and after the target object has been moved is greater than a minimum height change requirement and / or the estimated heights at the comparison position before and after the target object has been moved do not correspond to a planar surface.

10. The method according to claim 7, wherein the comparison locations include a first subset of comparison locations and a second subset of comparison locations, the first subset of comparison locations and the second subset of comparison locations are associated with the surface of a neighboring object that positions the location adjacent to the target object, and the method further comprises: Determine a first difference between the estimated heights at a first subset of comparison positions before and after the target object has been moved, and a second difference between the estimated heights at a second subset of comparison positions before and after the target object has been moved; And Determine that the neighboring object has shifted based on determining that the first difference is within a minimum height change requirement and the second difference is greater than the minimum height change requirement.

11. A robot system, the robot system comprising: At least one processor; At least one memory including processor instructions that, when executed, cause the at least one processor to perform a method, the method including: Obtain image data of a starting position and a task position for moving an object from a stack of objects at the starting position to a stack of objects at the task position, where: The image data includes a series of images depicting moving one or more objects from the starting position to the task position, the series of images including a first set of images depicting the starting position and the task position before moving the one or more objects, and a second set of images depicting the starting position and the task position after moving the one or more objects; and The image data includes depth measures representing an estimated height of a surface at the starting position and an estimated height of a surface at the task position determined from each image in the series of images; Determine a first depth difference between the depth measures representing the estimated height of the surface at the starting position in the first set of images and the second set of images; Verify that a target object among the one or more objects has been moved from the starting position based on determining that the first depth difference is greater than a pick verification threshold; Determine a second depth difference between the depth measures representing the estimated height of the surface at the task position in the first set of images and the second set of images; and Verify that the target object among the one or more objects has been placed at the task position based on determining that the second depth difference is greater than a place verification threshold.

12. The robot system according to claim 11, wherein the method further comprises: Estimate the number of objects within the object stack at the start position and / or at the task position by performing the following operations: Determine the outer perimeter edge of the object stack and the height of the object stack based on the image data; Derive the volume of the object stack based on the outer perimeter edge and the height of the object stack; Determine the lateral dimensions of different regions within the object stack, where the surface heights of the different regions are the same or the height variations in one or more lateral directions have a linear pattern; Determine the volumes of the respective different regions within the object stack based on the lateral dimensions of the different regions and the corresponding estimated heights of the outer vertical surfaces; And Derive an estimate of the number of objects within the object stack based on the derived volume of the object stack and the volumes corresponding to the respective different regions within the object stack.

13. The robot system according to claim 12, wherein estimating the number of objects within the object stack further comprises: Estimate the stacking pattern by comparing the determined outer perimeter edge of the object stack, the volume of the object stack, and the lateral dimensions of the different regions within the object stack with a set of predefined templates and / or a set of stacking rules; And Derive an estimate of the number of objects within the object stack based on the stacking pattern and the volume of the object stack, where the objects within the object stack include objects of different sizes.

14. The robot system according to claim 13, wherein the method further comprises comparing the estimated number of objects within the object stack with tracking history data identifying one or more objects that have been transported from the starting position to the task position to verify the accuracy of the tracking history data.

15. The robot system according to claim 13, wherein the method further comprises determining that the target object among the one or more objects is not placed at the task position based on determining that the second depth difference is less than the placement verification threshold.

16. A non - transitory computer - readable medium, the non - transitory computer - readable medium comprising processor instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations: Obtain image data of the starting position and the task position of the object stack for transporting an object from an object stack at a starting position to an object stack at a task position, wherein: The image data includes a series of images depicting the transfer of one or more objects from the start position to the task position, the series of images including a first set of images depicting the start position and the task position before the transfer of the one or more objects, and a second set of images depicting the start position and the task position after the transfer of the one or more objects; and The image data includes depth measures representing the estimated height of the surface at the start position and the estimated height of the surface at the task position determined from each image in the series of images; Determine a first depth difference between the depth measures representing the estimated height of the surface at the start position in the first set of images and the second set of images; Verify that the target object among the one or more objects has been transferred from the start position based on determining that the first depth difference is greater than a pick verification threshold; Determine a second depth difference between the depth measures representing the estimated height of the surface at the task position in the first set of images and the second set of images; And Verify that the target object among the one or more objects has been placed at the task position based on determining that the second depth difference is greater than a place verification threshold.

17. The non - transitory computer - readable medium according to claim 16, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform the following operations: Estimate the number of objects within the object stack at the starting position and / or at the task position by: Determine the outer peripheral edge and the height of the object stack based on the image data; Derive the volume of the object stack based on the outer perimeter edge and the height of the object stack; Determine the lateral dimensions of different regions within the object stack, where the surface heights of the different regions are the same or the height variations in one or more lateral directions have a linear pattern; Determine the volumes of the respective different regions within the object stack based on the lateral dimensions of the different regions and the corresponding estimated heights of the outer vertical surfaces; And Derive an estimate of the number of objects within the object stack based on the derived volume of the object stack and the volumes corresponding to the respective different regions within the object stack.

18. The non - transitory computer - readable medium according to claim 17, wherein estimating the number of objects within the object stack further comprises: Estimate the stacking pattern by comparing the determined outer perimeter of the object stack, the volume of the object stack, and the lateral dimensions of the different regions within the object stack with a set of predetermined templates and / or a set of stacking rules; and Derive an estimate of the number of objects within the object stack based on the stacking pattern and the volume of the object stack, where the objects within the object stack include objects of different sizes.

19. The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform the following operations: Compare the estimate of the number of objects within the object stack with tracking history data identifying one or more objects that have been transported from the starting position to the task position to verify whether the tracking history data is accurate.

20. The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform the following operations: Determine that the target object among the one or more objects is not placed at the task position based on determining that the second depth difference is less than the placement verification threshold.

Images