Robotic system with automatic package registration and automatic discovery pipeline

Abstract

To prevent robotic system stoppage and package mishandling that are caused by failure to correctly identify physical characteristics of imaged objects.SOLUTION: The present disclosure relates to detecting and registering unrecognized or unregistered objects. A minimum viable range (MVR) may be derived based on inspecting image data that represents objects at a start location. The MVR may be determined to be a certain MVR or an uncertain MVR according to one or more features represented in the image data. The MVR may be used to register corresponding objects according to the certain or uncertain determination.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 752,756, filed October 30, 2018, which is incorporated herein by reference in its entirety. This application further claims the benefit of U.S. Provisional Patent Application No. 62 / 852,963, filed May 24, 2019, which is incorporated herein by reference in its entirety. This application is U.S. Patent Application No. 16 / 290,741, filed March 1, 2019, which is now also related to U.S. Patent No. 10,369,701, which is incorporated herein by reference in its entirety.

[0002] This application contains subject matter related to a concurrently filed U.S. patent application entitled "A ROBOTIC SYSTEM WITH AUTOMATED PACKAGE REGISTRATION MECHANISM AND MINIMUM VIABLE REGION DETECTION" by Jinze Yu, Jose Jeronimo Moreira Rodrigues, and Rose Nikolaev Diankov. The related application is assigned to Mujin, Inc. and is identified by docket number 131837-8003.US40. The subject matter is incorporated herein by reference.

[0003] The present technology is directed generally to robotic systems, and more particularly to systems, processes, and techniques for registering objects. [Background technology]

[0004] Packages are often placed on pallets (or "palletized") for shipment to a destination and then depalletized at the destination. Packages may be depalletized by human workers, which can be resource intensive and increase the risk of injury. In industrial environments, the depalletizing operation is performed by industrial robots, such as robotic arms that grasp, lift, carry, and transport packages to a release point. Imaging devices are also used to capture images of the palletized stack of packages. The system may process the images to ensure that the packages are handled efficiently by the robotic arms, such as by comparing the captured images with registered images stored in a registration data source.

[0005] The captured image of the package may match the registered image. As a result, the physical characteristics of the photographed object (e.g., measurements of the package's dimensions, weight, and / or center of mass) may be unknown. Failure to accurately identify the physical characteristics may lead to a variety of undesirable consequences. For example, such a failure may cause a stall, which may require the package to be manually registered. Such a failure may also result in mishandling of the package, especially if the package is relatively heavy and / or tilted.

[0006] Various features and characteristics of the technology will become apparent to those skilled in the art upon review of the detailed description together with the drawings. Embodiments of the technology are illustrated by way of example, and not by way of limitation, in the drawings, in which like reference numerals may indicate like elements. [Brief explanation of the drawings]

[0007] [Figure 1] 1 illustrates an example of an environment in which a robotic system may operate. [Figure 2] 1 illustrates a robotic system in accordance with one or more embodiments of the present technology. [Figure 3A]1 illustrates an example of a stack of objects being processed by a robotic system in accordance with one or more embodiments of the present technology. [Figure 3B] 1 illustrates a top view of an example stack, in accordance with one or more embodiments of the present technology. [Figure 3C] 10 illustrates sensor data corresponding to a top surface in accordance with one or more embodiments of the present technology. [Figure 4A] 10 shows sensor data corresponding to the top surface after a first set of movements in accordance with one or more embodiments of the present technology. [Figure 4B] 4B illustrates a portion of the sensor data shown in FIG. 4A in accordance with one or more embodiments of the present technology. [Figure 5] 1 shows a top view of a set of palletized packages in accordance with an embodiment of the present technology; [Figure 6] 1 shows a graphical representation of an exposed outer corner in accordance with an embodiment of the present technology; [Figure 7] FIG. 1 is a block diagram of a set of palletized packages in accordance with an embodiment of the present technology; [Figure 8] 1 illustrates an example of merged MVR regions, in accordance with an embodiment of the present technology. [Figure 9] FIG. 1 is a flow diagram of an object recognition process, in accordance with an embodiment of the present technology. [Figure 10] FIG. 2 is a block diagram of a method of operating the robotic system of FIG. 1 in accordance with an embodiment of the present technology. [Figure 11] FIG. 1 is a block diagram illustrating an example of a processing system capable of performing at least some of the operations described herein. DETAILED DESCRIPTION OF THE INVENTION

[0008] The drawings depict various embodiments for purposes of illustration only. Those skilled in the art will recognize that alternative embodiments may be employed without departing from the spirit of the technology. Thus, while specific embodiments are shown in the drawings, the technology may be susceptible to various modifications.

[0009] Systems and methods are described herein for a robotic system with automatic package registration. A robotic system (e.g., a system integrating devices that perform one or more designated tasks) configured in accordance with certain embodiments can improve usability and flexibility by manipulating and autonomously / automatically registering unknown or unrecognized objects (e.g., packages, boxes, cases, etc.) (e.g., with little or no input from a human operator).

[0010] To determine whether an object has been recognized, the robotic system can obtain data about the object at a start position (e.g., one or more images of the object's exposed surface) and compare it to registration data of known or expected objects. The robotic system can determine that the object has been recognized if the compared data (e.g., a portion of the compared image) matches the registration data of one of the objects (e.g., one of the registered surface images). The robotic system can determine that the object has not been recognized if the compared data does not match the registration data of the known or expected object.

[0011] The robotic system can manipulate the unrecognized object according to one or more estimates and can determine additional information about the unrecognized object (e.g., surface image and / or physical dimensions). For example, the robotic system can identify exposed edges and / or exposed outer corners of the unrecognized object that are separate from or not adjacent to other objects.

[0012] The estimation may include generating a minimum feasible region (MVR) representing the minimum and / or optimal area required to contact and lift the corresponding unrecognized object. Each MVR may further represent an estimate of the surface of one unrecognized object (e.g., a perimeter boundary of the surface). During MVR generation, exposed outer corners and exposed edges may be identified by inspecting two-dimensional (2D) and / or three-dimensional (3D) image data. Based on the identified exposed outer corners and exposed edges, an initial MVR may be generated by identifying edges opposite the exposed edges. The initial MVR may be further processed, such as by determining and testing enlarged and / or reduced regions, to generate a verified MVR. The initial MVR may be processed according to the certainty of the MVR determination (e.g., a status or level representing the accuracy of the initial MVR). For example, the robotic system may identify the initial MVR as a certain MVR (e.g., an instance of a verified MVR that is likely to be accurate) if the initial MVR includes three or more exposed corners and / or is derived using three or more exposed corners. Otherwise (e.g., if the initial MVR includes two or fewer exposed corners and / or is derived using two or fewer exposed corners), the robotic system may identify the initial MVR as an uncertain MVR (e.g., an instance of a verified MVR that is unlikely to be accurate).

[0013] The robotic system can use the verified MVR to register an unrecognized object, such as by storing the verified MVR and / or other processing results derived using the verified MVR. For example, the robotic system can begin registering a corresponding unrecognized object with a certain MVR. The robotic system can further manipulate the certain MVR object (e.g., by grasping it and moving it to perform a task), thereby increasing the likelihood of exposing additional corners of the object remaining in the starting position. After moving the object, the robotic system can repeat the detection and analysis to increasingly identify previously uncertain MVRs as certain MVRs based on the newly exposed corners.

[0014] Thus, the embodiments described herein increase the likelihood of accurately generating an MVR for an unrecognized object. Increased accuracy can lead to fewer errors, such as dropped and / or collided packages, caused by inaccurate data about the moved object. Furthermore, increased MVR accuracy can lead to accurate enrollment data, which can provide increased efficiency resulting from reliably identifying subsequent processing of the same type of object. Furthermore, the embodiments described herein can be used to autonomously initiate and populate master data (e.g., enrollment data collection) without the need for initial data. In other words, a robotic system can autonomously enroll an object without existing enrollment data and without the interaction / input of a human operator.

[0015] In the following description, numerous specific details are set forth to provide a thorough understanding of the technology of the present disclosure. In other embodiments, the technology introduced herein may be practiced without these specific details. In other instances, well-known features, such as particular functions or routines, are not described in detail so as not to unnecessarily obscure the present disclosure. In this specification, references to "an embodiment," "one embodiment," etc., mean that the particular feature, structure, material, or characteristic being described is included in at least one embodiment of the present disclosure. Thus, appearances of such phrases in this specification do not necessarily all refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Furthermore, particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that the various embodiments shown in the figures are for illustrative purposes only and are not necessarily drawn to scale.

[0016] For clarity, some well-known details describing structures or processes often associated with robotic systems and subsystems, but which may unnecessarily obscure some important aspects of the disclosed technology, are not described below. Furthermore, although the following disclosure describes some embodiments of different aspects of the technology, some other embodiments may have different configurations or components than those described in this section. Thus, the disclosed technology may have other embodiments that have additional elements or that do not include some of the elements described below.

[0017] Many embodiments or aspects of the present disclosure described below may take the form of computer- or processor-executable instructions, including routines executed by a programmable computer or processor. Those skilled in the art will recognize that the disclosed technology can be practiced on computer or processor systems other than those shown and described below. The technology described herein can be implemented on a special-purpose computer or data processor that is specially programmed, configured, or constructed to execute one or more of the computer-executable instructions described below. Accordingly, the terms "computer" and "processor," as used generally herein, refer to any data processor and may include Internet appliances and handheld devices (including palmtop computers, wearable computers, cellular phones, mobile phones, multiprocessor systems, processor-based or programmable consumer electronics, network computers, minicomputers, etc.). Information handled by these computers and processors can be presented on any suitable display medium, including a liquid crystal display (LCD). Instructions for performing computer- or processor-executable tasks can be stored on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. The instructions may be contained in any suitable memory device, including, for example, a flash drive and / or other suitable medium.

[0018] The terms "coupled" and "connected," as well as derivatives thereof, may be used herein to describe a structural relationship between components. It should be understood that these terms are not intended as synonyms for each other. Rather, in particular embodiments, "connected" may be used to indicate that two or more elements are in direct contact with each other. Unless the context clearly indicates otherwise, the term "coupled" may be used to indicate that two or more elements are in direct or indirect contact with each other (there are other intervening elements between the elements), that two or more elements cooperate or interact with each other (e.g., as in a causal relationship such as signal transmission / reception or function call), or both.

[0019] favorable environment 1 is an example of an environment in which a robotic system 100 may operate. The robotic system 100 may include and / or communicate with one or more units (e.g., robots) configured to perform one or more tasks. Aspects of the packaging mechanism may be practiced or performed by various units.

[0020] For the example shown in FIG. 1 , the robotic system 100 may include an unloading unit 102, a mobile unit 104 (e.g., a palletizing robot and / or a piece-picking robot), a transport unit 106, a loading unit 108, or a combination thereof, in a warehouse or distribution / shipping location. Each unit in the robotic system 100 may be configured to perform one or more tasks. Tasks may be sequentially combined to perform actions to achieve a goal, such as unloading objects from a truck or van and storing the objects in a warehouse, or unloading objects from a storage location and preparing them for shipment. For another example, a task may include placing objects in a target location (e.g., on a pallet and / or in a bin / cage / box / case). As described below, the robotic system may derive a plan for placing and / or stacking the objects (e.g., placement location / or orientation, order in which to move the objects, and / or corresponding motion plan). Each unit may be configured to perform a series of actions to perform a task (e.g., by operating one or more components of the unit) according to one or more of the derived plans.

[0021] In some embodiments, a task may involve manipulating (e.g., moving and / or reorienting) a target object 112 (e.g., one of a package, box, case, cage, pallet, etc. corresponding to the task to be performed) from a start location 114 to a task location 116. For example, an unloading unit 102 (e.g., a de-bunking robot) may be configured to move a target object 112 from a position in a transport device (e.g., a truck) to a position on a conveyor belt. Also, a moving unit 104 may be configured to move a target object 112 from one position (e.g., a conveyor belt, a pallet, or a bin) to another position (e.g., a pallet, a bin, etc.). For another example, a moving unit 104 (e.g., a palletizing robot) may be configured to move a target object 112 from an original position (e.g., a pallet, a pickup area, and / or a conveyor) to a destination pallet. Upon completing the operation, the transport unit 106 can move the target object 112 from the area associated with the mobile unit 104 to the area associated with the loading unit 108, and the loading unit 108 can move the target object 112 (e.g., by moving a pallet carrying the target object 112) from the mobile unit 104 to a storage location (e.g., a location on a shelf). More details regarding this task and its associated operations are provided below.

[0022] For illustrative purposes, robotic system 100 is described in the context of a shipping center; however, it should be understood that robotic system 100 may be configured to perform tasks in other environments / for other purposes, such as manufacturing, assembly, packaging, healthcare, and / or other types of automation. It should also be understood that robotic system 100 may include other units not shown in FIG. 1 , such as manipulators, service robots, modular robots, etc. For example, in some embodiments, robotic system 100 may include a depalletizing unit that moves objects from a cage cart or pallet onto a conveyor or other pallet, a container switching unit that moves objects from one container to another, a packaging unit that packages objects, a sorting unit that groups objects according to one or more characteristics of the objects, a piece-picking unit that manipulates (e.g., sorts, group, and / or moves) objects differently according to one or more characteristics of the objects, or a combination thereof.

[0023] The robotic system 100 may include physical or structural members (e.g., robotic manipulator arms) connected at joints for movement (e.g., rotational and / or translational displacement). The structural members and joints can form kinematic chains configured to manipulate end effectors (e.g., grippers) configured to perform one or more tasks (e.g., grasping, rotating, welding, etc.) depending on the use / operation of the robotic system 100. The robotic system 100 may include drives (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or manipulate (e.g., displace and / or reorient) the structural members about or at the corresponding joints. In some embodiments, the robotic system 100 may include transport motors configured to transport the corresponding units / chassis from location to location.

[0024] The robotic system 100 may include sensors configured to acquire information used to perform a task, such as to manipulate a structural member and / or transport a robotic unit. The sensors may include devices configured to detect or measure one or more physical characteristics of the robotic system 100 (e.g., the status, state, and / or position of one or more structural members / joints of the robotic system) and / or one or more physical characteristics of the surrounding environment. Some examples of sensors may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

[0025] In one embodiment, for example, the sensors may include one or more imaging devices (e.g., visual and / or infrared cameras, two-dimensional (2D) and / or three-dimensional (3D) imaging cameras, distance measurement devices such as lidar or radar, etc.) configured to detect the surrounding environment. The imaging devices may generate a representation of the detected environment, such as a digital image and / or a point cloud, that may be processed via machine / computer vision (e.g., for automated inspection, robotic guidance, or other robotic applications). As described in further detail below, the robotic system 100 may process the digital image and / or point cloud to identify the target object 112, the start position 114, the task position 116, the pose of the target object 112, a confidence metric for the start position 114 and / or pose, or a combination thereof.

[0026] To manipulate a target object 112, the robotic system 100 can capture and analyze images of a designated area (e.g., a pick-up location, such as in a truck or on a conveyor belt) to identify the target object 112 and a start location 114 for the target object 112. Similarly, the robotic system 100 can capture and analyze images of another designated area (e.g., a drop location for placing the object on a conveyor, a location for placing the object in a container, or a location on a pallet for stacking) to identify a task location 116. For example, the imaging device may include one or more cameras configured to generate images of the pick-up area and / or one or more cameras configured to generate images of the task area (e.g., a drop area). Based on the captured images, the robotic system 100 can determine the start location 114, the task location 116, associated poses, packing / placing plans, transport / packing sequences, and / or other processing results, as described below.

[0027] In some embodiments, for example, the sensors may include position sensors (e.g., position encoders, potentiometers, etc.) configured to detect the position of structural members (e.g., robotic arms and / or end effectors) and / or corresponding joints of the robotic system 100. The robotic system 100 can use the position sensors to track the position and / or orientation of the structural members and / or joints during task execution.

[0028] Object Movement and Registration Using Destination-Based Sensors 2 is a diagram of the robotic system 100 of FIG. 1 in accordance with one or more embodiments of the present technology. The robotic system 100 may include a robotic arm 202 (e.g., an instance of the mobile unit 104 of FIG. 1 ) including an end effector 204 (e.g., a gripper). The robotic arm 202 may be configured to move a target object 112 between a start location 114 of FIG. 1 and a task location 116 of FIG. 1 . As shown in FIG. 2 , the start location 114 may have a pallet 208 carrying a target stack 210 (e.g., a group of objects). The task location 116 of the robotic arm 202 may be a loading location (e.g., a start / release point) of a conveyor 206 (e.g., an instance of the transport unit 106 of FIG. 1 ). For example, the robotic arm 202 may be configured to pick up an object from the target stack 210 and place the object on the conveyor 206 for transport to another destination / task.

[0029] The robotic system 100 can use one or more sensors when performing movement operations using the robotic arm 202. In some embodiments, the robotic system 100 can include a first image sensor 212 and / or a second image sensor 214. The first image sensor 212 can include one or more 2D and / or 3D sensors, such as a camera and / or depth sensor, configured to image and / or analyze the start location 114. The second image sensor 214 can include one or more 2D and / or 3D sensors, such as a camera and / or depth sensor, configured to image and / or analyze the task location 116. For example, the first image sensor 212 can include one or more cameras and / or depth sensors positioned at known positions above and facing toward the start location 114. The first image sensor 212 can generate image data (e.g., a 3D point cloud and / or visual or 2D images) corresponding to one or more top views of the start location 114, such as a top view of the object stack 210. As described in further detail below, the robotic system 100 can use image data from the first image sensor 212 to derive a minimum feasible region (MVR) for an unrecognized (e.g., unregistered) object in the object stack 210. The robotic system 100 can use the MVR to grasp (e.g., via the end effector 204) and manipulate (e.g., via the robot arm 202) the unrecognized object, such as when moving the unrecognized object from the start position 114 to the task position 116. The second image sensor 214 can also include one or more cameras and / or depth sensors positioned at one or more known locations above / beside and facing toward the task position 116, or in a related space. Thus, the second image sensor 214 can generate image data corresponding to one or more top and / or side views of the target object 112 at or within a threshold distance from the task position 116.

[0030] object recognition In accordance with one or more embodiments of the present technology, FIG. 3A is a diagram of an example stack of objects (e.g., object stack 210 of FIG. 2 ) processed by robotic system 100 of FIG. 1 , FIG. 3B is a diagram of a top view of the example stack (e.g., actual top view 310 of object stack 210), and FIG. 3C is a diagram of sensor data corresponding to the top view (e.g., top view data 320). Referring together to FIGS. 3A , 3B, and 3C , robotic system 100 may be configured to move objects in object stack 210 to other locations, such as task position 116 of FIG. 1 (e.g., conveyor 206 of FIG. 2 ), as described above. In connection with object movement, robotic system 100 can use image data (e.g., top view data 320) from first image sensor 212 of FIG. 2 positioned above object stack 210. For example, top view data 320 can include one or more visual images and / or one or more depth maps that show or represent actual top view 310. Additionally, the robotic system 100 can analyze the top view data 320 to identify edges that may correspond to boundaries of an object. For example, the robotic system 100 can identify edges and / or continuous surfaces represented in the image data based on differences in depth measurements and / or image characteristics (e.g., different colors, linear patterns, shadows, differences in clarity, etc.). The robotic system 100 can identify exposed edges 322 (e.g., edges of the top surface of an object that are not horizontally abutting other objects / surfaces of approximately the same height), based on differences in depth measurements, etc.

[0031] The object stack 210 may include objects registered in master data, which includes registration records of objects from predicted or previous processes, and / or unpredicted objects not registered in the master data. Thus, the robotic system 100 can recognize or identify objects in the object stack 210 using image data of the object surface 316. In some embodiments, the robotic system 100 can recognize objects in the object stack 210 by comparing the image data, or one or more portions of the image data, with the master data. For example, the robotic system 100 can identify a known object (e.g., a recognized object 312) in the object stack 210 when a portion of the top view data 320 matches one or more images of the object surface 316 in the registration data. The remaining portion of the actual top view 310 (e.g., the portion that does not match the registration data) may correspond to an unrecognized object 314. The edges of the unrecognized object 314 are indicated in FIG. 3C using dashed lines.

[0032] Based on the image data match, the robotic system 100 can locate the recognized object 312 in the corresponding image data, which may be further translated to a real-world position of the object stack 210 (e.g., via a pre-calibration table and / or equations that map pixel locations to a coordinate system). Additionally, the robotic system 100 can estimate the positions of the unexposed edges of the recognized object 312 based on the match. For example, the robotic system 100 can obtain the dimensions of the recognized object 312 from the master data. The robotic system 100 can measure a portion of the image data that is separated by a known dimension from the exposed edge 322 of the recognized object 312. According to the mapping, the robotic system 100 can determine one or more registration-based edges 324 of the recognized object 312 and / or similarly map the registration-based edges 324 to real-world locations, as described above.

[0033] In some embodiments, the robotic system 100 can identify exposed outer corners 326 of the object stack 210 represented in the image data (e.g., point cloud data). For example, the robotic system 100 can identify the exposed outer corners 326 based on detecting intersections / junctions between two or more sets of exposed edges 322 (e.g., edges identified in the 3D image data, also referred to as 3D edges) that have different orientations (e.g., extend at different angles). In one or more embodiments, the robotic system 100 can identify the exposed outer corners 326 when the exposed edges 322 form angles that fall within a predetermined range (also referred to as an angle range), such as a threshold corner angle range greater than and / or less than 90 degrees. As described in more detail below, the robotic system 100 can use the exposed outer corners 326 and the corresponding exposed edges 322 to process and / or manipulate the unrecognized object 314.

[0034] Handling Unrecognized Objects In one embodiment, the robotic system 100 of FIG. 1 can process (e.g., identify and / or move) an object according to its recognition status and / or relative position within the object stack 210 of FIG. 2. For example, the robotic system 100 can first grasp and move the recognized object, and then generate another image data set from a sensor (e.g., the first image sensor 212 of FIG. 2). FIG. 4A is an illustration of sensor data 401 corresponding to a top surface after a first set of actions (e.g., grasping and moving the recognized object 312) in accordance with one or more embodiments of the present technology. The shaded portion of FIG. 4A corresponds to the change in depth measurements after removing the recognized object 312 shown in FIGS. 3B and 3C.

[0035] If the robotic system 100 does not identify any of the recognized objects 312 in the image data (e.g., 2D image and / or 3D point cloud), the robotic system 100 can process the image data to identify any exposed corners 326 and / or exposed edges 322 to locate the unrecognized object 314 in FIG. 3A . For example, the robotic system 100 can process the sensor data 401 (e.g., 3D point cloud) to identify exposed outer corners 326 and / or exposed edges 322, similar to the above. Thus, the robotic system 100 can further identify and / or locate exposed corners / edges after removing the recognized objects 312. In one embodiment, the robotic system 100 can further identify exposed inner corners 402 as intersections / junctions between two or more of the exposed edges 322 according to corresponding thresholds. For example, the robotic system 100 can identify an exposed interior corner 402 as a junction between two or more exposed edges 322 that form an angle greater than 180 degrees relative to a corresponding continuous surface. In one embodiment, the robotic system 100 can identify an exposed interior corner 402 when the exposed edges 322 form an angle that is within a threshold angle range greater than and / or less than 270 degrees.

[0036] In one embodiment, if no recognized objects 312 remain, the robotic system 100 can identify a registered object 406 in the object stack 210 (e.g., from among the unrecognized objects 314) based on exposed corners and / or exposed edges. For example, the robotic system 100 can evaluate the exposed corners / edges according to a set of preferences and / or a scoring mechanism. In one embodiment, the robotic system 100 can be configured to select the exposed outer corner 326 closest to the robot arm 202 of FIG. 2 . The robotic system 100 can select a corresponding region of the sensor data 401 and a corresponding unrecognized object 314 associated with the selected exposed outer corner 326. In other words, after processing the recognized object 312, the robotic system 100 can process the unrecognized object 314 that forms / constitutes the exposed outer corner 326 of the object stack 210. Based on analyzing the corresponding portion of the sensor data 401 (e.g., by deriving an MVR), the robotic system 100 can grasp the unrecognized object 314, lift and / or move the grasped object horizontally, and / or photograph the grasped object for registration. Furthermore, after photographing the grasped object, the robotic system 100 can move the grasped object to a destination (e.g., conveyor 206 in FIG. 2).

[0037] To further describe the analysis of the sensor data, FIG. 4B is a detailed view of a portion of the sensor data 401 of FIG. 4A in accordance with one or more embodiments of the present technology. The robotic system 100 can analyze the sensor data 401 to derive an MVR 412. The MVR 412 can represent a minimum size area used to contact, grasp, and / or lift an object. The MVR 412 can also represent an estimate of the surface and / or surrounding boundary of an unrecognized object. For example, the MVR 412 can be associated with the attachment area of ​​the end effector 204 or a larger area with an additional / buffer region surrounding the attachment area. For example, the MVR 412 can also be associated with the estimated location of the horizontal boundary / edge of an unrecognized object (e.g., a selected enrollment target). In one embodiment, the MVR 412 may correspond to a minimum and / or maximum candidate size, which may correspond to the physical dimensions (e.g., length, width, height, diameter, circumference, etc.) of the smallest / largest possible instances of predicted objects in the object stack 210, respectively. In other words, the robotic system 100 may determine that there are no objects in the object stack 210 having dimensions smaller than the minimum candidate size or larger than the maximum candidate size. The minimum and maximum candidate sizes may be predetermined values ​​(i.e., values ​​given or known before processing the object stack 210). Details regarding the derivation of the MVR 412 are provided below.

[0038] The robotic system 100 can use the MVR 412 to determine the gripping location 420. The gripping location 420 can correspond to an area on the object / stack that is directly beneath and / or in contact with the end effector 204 for an initial operation. In other words, the robotic system 100 can position a gripper over the gripping location 420 to grasp the corresponding object for a subsequent operation (e.g., a lifting process, a horizontal movement process, and / or a data collection process for registration). In one embodiment, the robotic system 100 can select the gripping location 420 from a set of possible gripping locations. For example, the robotic system 100 can select from the set of gripping locations according to the relative orientation of the arms (e.g., with a preference that the robot arms extend across the exposed edge 322 and do not overlap other portions).

[0039] In one embodiment, the robotic system 100 can derive the MVR 412 based on the estimated edge 424. For example, the robotic system 100 can select or identify an instance of an exposed outer corner 326 (e.g., a 3D corner) of an upper layer of the target stack 210. The robotic system 100 can move away from the selected exposed outer corner 326 along the associated exposed edge 322 (e.g., meeting / forming at the selected corner in 3D). While moving along the edge, the robotic system 100 can identify the estimated edge 424 based on 3D depth measurements in the sensor data 401 and / or differences in 2D image characteristics (e.g., brightness, color, etc.). The robotic system 100 can identify edges or lines in the 2D / 3D image data that intersect with the traversed edge and / or are within a threshold separation distance from the traversed edge. The robotic system 100 can test the identified edges when determining the estimated edge 424. The robotic system 100 may test / verify the inferred edge 424 based on comparing the orientation of the identified edge with the orientation of the exposed edges 322. For example, the robotic system 100 can verify the identified edge as the inferred edge 424 if the identified edge is parallel to one of the exposed edges 322. In some embodiments, the robotic system 100 can test for parallel orientation based on verifying that the distances between two or more corresponding points on the tested pair of edges (e.g., the identified edge and the non-crossed instance of the exposed edge 322) are equal. In some embodiments, the robotic system 100 can identify parallel orientation if the tested pair of edges and / or their extensions intersect with a common edge at the same angle, such as when both edges and / or their extensions intersect with the crossed instance of the exposed edge 322 at an angle between 80 degrees and 100 degrees.

[0040] Thus, the robotic system 100 can derive a grasp position 420 that does not overlap with detected lines (e.g., 2D edges and / or incomplete edges) and / or estimated edges 424. The robotic system 100 can derive the grasp position 420 based on balancing the ratio between the distances between the edges of the MVR 412 and the nearest detected lines and / or estimated edges 424. Because the robotic system 100 grasps an object at or near a corner based on the MVR 412, the robotic system 100 can derive a grasp position 420 that reduces the maximum potential torque along any one particular direction based on balancing the ratio. Additionally, the robotic system 100 may further derive or adjust the MVR 412 to coincide with or extend outward from the estimated edges 424.

[0041] The robotic system 100 can use the derived gripping position 420 to operate the robot arm 202 and end effector 204 of FIG. 2 . The robotic system 100 can grasp an object (e.g., registered target 406) at a corner of the stack at the gripping position 420. In one embodiment, the robotic system 100 can lift and / or horizontally move the grasped object to clearly distinguish previously unexposed edges. For example, the robotic system 100 can lift and / or horizontally move the object a predetermined height corresponding to a minimum distance to accurately distinguish edges. Also, for example, the robotic system 100 can lift and / or horizontally move the object while monitoring and / or analyzing height changes and / or tilt of the end effector, thereby enabling recognition of additional edges opposite the exposed edge 322. The robotic system 100 can acquire and process data during and / or after the initial lifting to further describe the unrecognized object 314.

[0042] MVR Detection Overview This embodiment may relate to generating an accurate minimum feasible region (MVR) of an object. Exposed outer corners and exposed edges may be identified by inspecting 2D and / or 3D image data (e.g., point cloud data). Based on the identified exposed outer corners and exposed edges, an initial MVR may be generated by identifying an edge opposite the exposed edge. In one embodiment, the robotic system 100 can generate the MVR based on identifying an opposite edge (e.g., estimated edge 424 in FIG. 4B ) that corresponds to (e.g., is parallel to) the exposed edge. The initial MVR may extend from the exposed outer corner along the exposed edge to the opposite edge.

[0043] After determining the initial MVR, a possible MVR region (e.g., a set of laterally adjacent locations having depth measurements within a threshold range of each other) may be identified extending from the initial MVR defined by the point cloud to the edge of the surface or layer. A merged MVR of the object may include the initial MVR and the possible MVR. A verified MVR may be generated by inspecting / testing the merged MVR. The verified MVR may represent the precise region containing the unrecognized object. Based on the verified MVR, the robotic system 100 described herein may register the object and perform tasks related to the object, such as grasping and / or moving the object.

[0044] In many cases, an edge of an object (e.g., an outer or exposed edge) can be identified. For example, an outer edge of an object located along the perimeter of object stack 210 in FIG. 2 (e.g., exposed edge 322 in FIG. 3C ) may be free of surrounding objects and / or may be separated from any surrounding objects. Thus, the imaging results (e.g., 3D point cloud) may include abrupt changes in displayed values ​​(e.g., height measurements and / or image values ​​such as color or brightness) representing the outer edge. However, in many cases, other edges may be invisible or difficult to precisely define (e.g., based on a threshold confidence value). For example, a design / image of a box surface may lead to an inaccurate edge being identified on the object. Thus, the inaccurate edges may make it difficult to accurately isolate portions of the 2D / 3D image data.

[0045] The robotic system 100 may identify exposed outer corners 326 and / or exposed edges 322 of an object by inspecting image data (e.g., point clouds and / or 2D images) and determining one or more layers. For example, the robotic system 100 can identify an upper layer (e.g., unrecognized object 314 in FIG. 3B ) of the object(s) in the target stack 210 based on the imaging results. Within the upper layer, the robotic system 100 can select portions of the object and / or surface (e.g., regions having height values ​​within a threshold continuous range of each other) for MVR derivation. The robotic system 100 can infer that exposed edges forming / defining a surface (e.g., one of the layers or a portion thereof) correspond to the lateral / perimeter boundaries of the surface and / or corresponding object. An initial MVR may be generated by identifying an opposing edge opposite an exposed edge, where the initial MVR may extend from an exposed outer corner (e.g., an intersection formed by a set of exposed edges and / or a perimeter boundary) along the exposed edge to the opposing edge.

[0046] The robotic system 100 can further process (e.g., adjust) the initial MVR by scaling the initial estimate based on markers (e.g., incomplete edges) in the image data. The adjusted MVR can be examined to determine a final MVR that will be used to determine the grasp location 420 and / or register the unrecognized object.

[0047] FIG. 5 illustrates a top view of a set 500 of palletized packages (e.g., target stack 210 of FIG. 2 ) in accordance with an embodiment of the present technology. As shown in FIG. 5 , a first object 510 may include sides 512a-512d. The sides may represent the boundaries of the object from a given view (e.g., a top view). In certain circumstances, at least some of the sides 512a-512d (e.g., sides 512c-512d) may be closed and abut other adjacent objects / sides. Conversely, some of the sides (e.g., sides 512a-512b) may be open and correspond to exposed side 322 of FIG. 3C .

[0048] The first box 510 may include one or more exposed outer corners 514 that are spaced apart from or free of horizontally adjacent objects. The exposed outer corners 514 may correspond to the exposed outer corners 326 of the stack in FIG. 3. As described below, the image data (e.g., point clouds and / or 2D images) may be inspected to identify the exposed edges and exposed outer corners of the package.

[0049] Point Cloud Segmentation FIG. 6 illustrates a graphical representation 600 of an exposed outer corner, according to an embodiment of the present technology. As illustrated in FIG. 6, a point cloud (e.g., the graphical representation 600) may be processed to identify an exposed outer corner 614 (e.g., one of the exposed outer corners 326 of FIG. 3). For example, the point cloud may correspond to a top view generated by a 3D camera of the first object 510 of FIG. 5, or a portion thereof. The point cloud may include a three-dimensional point cloud having multiple layers indicating depth. Each layer and / or surface may correspond to a set of horizontally adjacent depth values ​​that are within a threshold continuous range from each other. For example, the threshold continuous range may require that horizontally adjacent locations have depth measurements that are within a threshold distance (e.g., less than one centimeter) from each other, or that are within a threshold distance according to slope. The threshold continuous range may also define a floor and / or slope relative to a reference height (e.g., the highest / closest point of the target stack 310). The threshold contiguous range can similarly define limits for 2D image characteristics (e.g., color and / or brightness) to identify continuity between sets of horizontally adjacent locations. The depth of a point cloud layer can correspond to a separation along a direction perpendicular to the surface of the corresponding object (e.g., vertical separation).

[0050] Thus, the point cloud may be analyzed and processed to separate layers and / or identify open 3D edges / corners. In one embodiment, the robotic system 100 (e.g., one or more processors within the robotic system 100) can identify layers based on grouping depth values ​​of the point cloud according to one or more predetermined continuity rules / thresholds. For example, the robotic system 100 can group sets of horizontally adjacent / connected depth values ​​if the depth values ​​are within a threshold continuity range of each other and / or if the depth values ​​follow a certain slope representing a flat or continuous surface. The robotic system 100 can identify exposed edges (e.g., exposed edges 512a and 512b in FIG. 5) as boundaries of the identified layer. In other words, the robotic system 100 can identify exposed edges 512a and 512b as horizontal perimeter locations of the layer / surface where qualifying depth changes occur. In general, the depth measurements of objects / edges forming an upper layer of the object stack 210 may have a smaller magnitude (e.g., representing a closer distance to the first image device 312) than objects / edges forming a layer below the upper layer.

[0051] In one embodiment, the robotic system 100 can determine exposed edges based on identifying visible lines in a 2D visual image. For example, a pallet and / or floor may correspond to a known color, brightness, etc. Thus, the robotic system 100 can identify lines that delimit such known patterns as exposed edges of the object(s). Additionally, the robotic system 100 can verify the 3D identification of exposed edges using the 2D analysis.

[0052] Based on the exposed edges, the robotic system 100 can identify open 3D corners (e.g., exposed exterior corners 514). For example, the robotic system 100 can identify shapes / angles associated with the exposed edges. The robotic system 100 can be configured to determine the exposed exterior corners 514 as points where exposed edges (e.g., edges 512a-512b) intersect / form an angle within a threshold angle range (e.g., 80-100 degrees).

[0053] As an illustrative example, the robotic system 100 can identify an open 3D corner 614 by identifying a first region 612 and adjacent regions 616a-616c. The robotic system 100 can identify the first region 612 when a set of adjacent horizontal locations in the scanned region layer have depth values ​​that are within a threshold continuous range from each other. The robotic system 100 can identify the adjacent regions 616a-616c as other horizontal locations that have depth values ​​that are outside the threshold continuous range from the depth value of the first region 612. In one embodiment, the robotic system 100 can identify a side of the first region 612 and / or the beginning of an adjacent region 616a-616c when the depth value changes and falls outside the threshold continuous range and / or when the location of the depth value change matches a shape template (e.g., a straight line and / or a minimum separation width between objects). More specifically, the adjacent regions 616a-616c may have depth values ​​that represent distances farther from the first image sensor 212 than the depth value of the surface of the object stack 210 (i.e., the first region 612). The resulting edges between the first region 612 and the adjacent regions 616a-616c may correspond to exposed edges. In one embodiment, identifying the open 3D corner 614 may include verifying that the first region 612 forms a quadrant and that the adjacent regions 616a-616c correspond to remaining quadrants, such as locations outside the stack of objects, and / or empty space. Empty space may indicate space detected with a very sparse point cloud, which may be considered point cloud noise.

[0054] Other 3D corners may be determined using the 3D point cloud. In one embodiment, the exposed outer corners may be a contour shape, and the corners of an "L" shape may not include valid corners. Thus, the robotic system 100 can identify edge segments that meet one of the requirements (e.g., minimum continuous straight line length) based on extending such edge segments by a predetermined length. If the extended edge segment intersects with another segment or the extended segment at an angle, the robotic system 100 can identify a point on the contour shape (e.g., the midpoint of the arc between the intersecting edge segments) as an exposed outer corner.

[0055] In some embodiments, the 3D corners may be ranked. For example, a 3D corner surrounded by empty space (e.g., of an object at the top corner of a stack) may be ranked higher than other objects. The open 3D corner 614 may be ranked based on other factors, such as the size of the first region 612, the position of the open 3D corner 614 relative to the shape of the first region 612, the difference in depth values ​​between the surrounding regions (e.g., between the first region 612 and adjacent regions 616 a and 616 c), and / or the horizontal distance between the first region 612 and other regions (e.g., other surfaces / objects) having depth values ​​within a threshold continuous range from the depth value of the first region 616 a.

[0056] In one embodiment, the robotic system 100 may identify incomplete edges. Incomplete edges may be edges identified in 2D and / or 3D analysis, which may or may not be actual edges. Some incomplete edges may correspond to actual edges of boxes / gaps between boxes that may not be identified due to the presence of noise from other objects and / or the capabilities / position of the imaging device (e.g., camera). Incomplete edges may also be visual patterns or marks on the object surface detected from 2D image analysis, such as a picture or mark on the surface or a divider / seam between the flaps of a box taped together. Conversely, a box without a pattern may not have any 2D lines that can be identified as an incomplete edge. The robotic system 100 can identify incomplete edges where the sensor output location exceeds the noise variance but does not fully satisfy the rules / thresholds for edge identification. In one embodiment, the robotic system 100 can use the 3D sensor output to identify exposed outer edges (e.g., edges around the perimeter of the first region 612) and the 2D sensor output to identify incomplete edges. The robotic system 100 can also identify incomplete edges as 2D or 3D edges that do not intersect with other edges at angles that are within an angle threshold range. More details regarding incomplete edges are provided below.

[0057] Generating the First MVR FIG. 7 illustrates a top view of a set of palletized packages in accordance with an embodiment of the present technology. FIG. 7 further illustrates an initial MVR 710 associated with the top view (e.g., 3D and / or 2D sensor image of target stack 210 in FIG. 2 ). The initial MVR 710 may represent an initial estimate of the MVR's boundary and / or corresponding object edges. In other words, the initial MVR 710 may represent an initial estimate of the surface of a single unrecognized object. The robotic system 100 can derive the initial MVR 710 using open 3D corners and / or exposed edges. Deriving the initial MVR 710 may include inspecting each of the identified layers from the point cloud to identify objects, edges, corners, etc. For example, the robotic system 100 can derive the initial MVR 710 as regions (e.g., first region 612 in FIG. 6 ) of the point cloud data (e.g., 3D sensor output) having depth values ​​within a threshold continuous range from each other. In other words, the robotic system 100 can derive the initial MVR as a group of horizontally adjacent locations having sufficiently consistent or linearly patterned depth values ​​corresponding to a continuous surface. Accordingly, the robotic system 100 can derive the initial MVR as an area at least partially bounded by exposed exterior corners 514 (e.g., first corner 714a and / or second corner 714b) and corresponding exposed edges. For example, the initial MVR 710 can extend from the first corner 714a along exposed edges 722 and 726 to the opposing edges 724 and 728. In one embodiment, the robotic system 100 can derive the initial MVR 710 by starting from an open 3D corner (e.g., first corner 714a) and tracing the exposed edges (e.g., away from the open 3D corner). The initial MVR 710 may be extended until a traced exposed edge intersects or meets another exposed edge (e.g., an edge opposite an untraced edge and / or an edge parallel to an untraced edge, such as opposing edges 724 and / or 728).

[0058] In some cases, the initial MVR 710 may correspond to the surface of multiple objects due to various reasons (e.g., spacing between objects, sensor granularity, etc.). Therefore, the robotic system 100 may validate one or more dimensions of the derived initial MVR 710. The robotic system 100 may validate that one or more dimensions of the MVR 710 are greater than a minimum candidate size and less than a maximum candidate size. The threshold dimension may represent a minimum and / or maximum object dimension acceptable / expected to the robotic system 100. The threshold dimension may also represent the horizontal footprint of the end effector 204 of FIG. 2, such as to represent a minimum size of the grasping / contact area.

[0059] If one or more dimensions of the initial MVR 710 fall outside a threshold (e.g., by exceeding a maximum dimension or by falling below a minimum dimension), the robotic system 100 can adjust the initial MVR 710, such as by further segmenting the initial MVR 710 (e.g., the top layer) according to the incomplete edges 712 (e.g., 2D / 3D edges that do not coincide with or intersect with other edges of one or more edges). In other words, the robotic system 100 can adjust / shrink the initial MVR according to the incomplete edges 712 and test the corresponding results. In one embodiment, the robotic system 100 can determine the incomplete edges 712 as 2D and / or 3D edges that do not intersect with exposed edges of one or more edges. The robotic system 100 can also determine the incomplete edges 712 as 2D and / or 3D edges that are parallel to one of the exposed edges. In one embodiment, the robotic system 100 can calculate a confidence value associated with the incomplete edges 712. The confidence value may represent the likelihood that the incomplete edge 712 corresponds to a surface edge and / or a separation between adjacent objects. By way of example, the robotic system 100 can calculate the confidence value based on the total length of the incomplete edge 712, the shape of the incomplete edge 712, and / or the difference (e.g., depth, color, brightness, etc.) between the incomplete edge 712 and the portion surrounding the incomplete edge 712.

[0060] As described in more detail below, the robotic system 100 may derive the verified MVR 720 based on shrinking the initial MVR 710 according to or up to the incomplete edges 712. In other words, the robotic system 100 may identify a reduced candidate MVR as an area within the initial MVR 710 bounded by one or more of the incomplete edges 712 instead of the opposing parallel edges 724 and / or 728. The robotic system 100 may reduce the initial MVR 710 by following the opposing parallel edges (e.g., the opposing parallel edges 724, 728, which may be 2D and / or 3D edges, such as the estimated edge 424 in FIG. 4B ) toward the corresponding exposed edges (e.g., the corresponding exposed edges 722, 726, respectively) until the incomplete edge 712 is reached. In other words, the robotic system 100 can identify the second corner 714b associated with the first MVR 710 and then move away from the second corner 714 along the associated edge (e.g., the opposing parallel edge 724 and / or 728) while searching for other edges (e.g., intersecting and / or parallel edges, such as the incomplete edge 712). The robotic system 100 can test the orientation of the identified edge, as described above. For example, the robotic system 100 may verify the identified edge if the angle between the traversed edge and the identified edge is within a threshold angle range (e.g., 80 to 100 degrees and / or other range corresponding to a right angle). The robotic system 100 may also verify that the identified edge is parallel to the corresponding exposed edge, such as when a set of distances between multiple sets of corresponding points along the identified, untraversed exposed edge is within a threshold range from each other. This may be done iteratively or incrementally. If the MVR cannot be reduced any further, the robotic system 100 may conclude that the resulting area is the verified MVR region 720.

[0061] The robotic system 100 can verify the reduced candidate MVR based on comparing the reduced dimension to a threshold value as described above. For example, the robotic system 100 can derive the reduced candidate MVR as a verified MVR 720 if the reduced area defined by the incomplete edge 712 meets a minimum / maximum threshold. The robotic system 100 can also verify the reduced candidate MVR if the incomplete edge 712 corresponds to a confidence value that exceeds a predetermined threshold. Furthermore, the robotic system 100 can extend the incomplete edge 712 a threshold distance in one or more directions. For example, the robotic system 100 can verify the reduced candidate MVR if the extended incomplete edge intersects with another edge to form an angle that meets a threshold angle range.

[0062] As an example of expanding the initial MVR 710, FIG. 8 shows an example of a merged MVR region according to one embodiment of the present technology. In some cases, multiple adjacent surfaces may have essentially the same height and may have edges / horizontal separations between the surfaces. As an illustrative example, the object represented by the point cloud may be the top of a box with two separated rectangular flaps / segments connected by a piece of tape. Due to one or more of the limitations noted above, the robotic system 100 may derive the initial MVR 710 to include one flap and extend to the edge corresponding to the tape or separation between the flaps / segments. For such an example, the robotic system 100 may derive additional regions and merge / test those regions with the initial MVR 710 upon deriving the verified MVR 820.

[0063] 8, additional valid MVR regions 812a-812b may be determined based on the initial MVR 710. The additional valid MVR regions 812a-812b may include regions that correlate to the initial MVR 710. For example, the robotic system 100 can determine the additional valid MVR regions 812a-812b based on analyzing portions of 2D and / or 3D images. The robotic system 100 can determine the additional valid MVR regions 812a-812b as locations that have matching visual characteristics (e.g., color, brightness, and / or image pattern) and / or matching depth values ​​that are within a threshold continuous range relative to the initial MVR 710. The robotic system 100 can derive candidate areas as portions of the images determined to be associated with the initial MVR 710.

[0064] As an illustrative example, the robotic system 100 can process MVRs (e.g., initial MVR and expanded MVR) based on tracing a first exposed edge 722 and a second exposed edge 726 (e.g., edges depicted in the 3D image data) away from the exposed outer corner 714a. The robotic system 100 can identify an initial set of opposing edges including a first initial opposing edge 822 and a second initial opposing edge 826. The robotic system 100 can verify the initial set of opposing edges when the first exposed edge 722 is parallel to the first initial opposing edge 822 and / or when the second exposed edge 726 is parallel to the second opposing edge 826. The robotic system 100 can derive the initial MVR 710 using the verified opposing edges.

[0065] The robotic system 100 can further determine additional plausible MVR regions 812a-b based on tracing the first exposed edge 722 and the second exposed edge 726 beyond the initial set of opposing edges (e.g., away from the exposed outer corner 714a). The robotic system 100 can identify one or more additional opposing edges (e.g., the first edge 832 and / or the second edge 836) that intersect with or are within a threshold separation distance from the traced edges (e.g., the first exposed edge 722 and / or the second exposed edge 726). The robot system 100 can verify additional opposing sides in a similar manner as described above, such as when the first side 832 is parallel to the first exposed side 722 and / or the first initial opposing side 822, and / or when the second side 836 is parallel to the second exposed side 726 and / or the second initial opposing side 826.

[0066] Once one or more additional opposing sides have been verified, the robotic system 100 can identify additional valid MVR regions. For example, the robotic system 100 can identify a first additional valid MVR region 812a as the area between the first initial opposing side 822 and a first of the additional opposing sides (e.g., first side 832). The robotic system 100 can also identify a second additional valid MVR region 812b as the area between the second initial opposing side 826 and a second of the additional opposing sides (e.g., second side 836).

[0067] The robotic system 100 can determine additional valid MVR regions 812a-812b based on verifying / testing the candidate areas (e.g., the combination of the initial MVR 710 with the first additional valid MVR region 812a and / or the second additional valid MVR region 812b). For example, the robotic system 100 can verify that the separation distance between the candidate area (e.g., the portion of the image determined to be associated with the initial MVR 710) and the initial MVR 710 is less than a predetermined threshold. The robotic system 100 can further test the candidate area by comparing one or more dimensions of the candidate area with the minimum / maximum dimension thresholds described above. The robotic system 100 can determine the candidate area as an additional valid MVR region 812a-812b if the candidate area is less than a minimum threshold (e.g., a dimension of the minimum candidate size). In one embodiment, the robotic system 100 can calculate a confidence level using size comparison, separation distance, and / or relatedness / similarity between the candidate area and the initial MVR 710. The confidence level may represent the likelihood that the candidate area corresponds to the same object as the portion corresponding to the initial MVR 710. The robotic system 100 can compare the confidence level to a predetermined threshold to determine whether the candidate area should be classified as an additional valid MVR region 812a-812b or as a new instance of the initial MVR 710 (e.g., corresponding to a different object).

[0068] Generate a validated MVR The robotic system 100 can derive a verified MVR 820 based on combining the initial MVR 710 with additional valid MVRs 812a-812b. Thus, the robotic system 100 can derive candidate MVRs by expanding the initial MVR 710 to include other nearby regions. Thus, the robotic system 100 can increase the likelihood of accurately estimating the complete surface of an unregistered object via the verified MVR 820.

[0069] In one embodiment, the robotic system 100 can derive both a verified MVR 820 and a verified MVR 720 (e.g., a result of scaling down the initial MVR 710). According to one or more predetermined processes / equations, the robotic system 100 can calculate a confidence value for each of the verified MVRs using one or more of the processing parameters described above. The robotic system 100 can select the verified MVR with the larger confidence value as the final MVR.

[0070] Alternatively, the robotic system 100 can derive the initial MVR 710 as the final MVR if testing of smaller and / or larger candidate areas is unsuccessful. For example, if the merged MVR is larger than the maximum candidate size, the merged MVR can be rejected, and the verified MVR 820 can include the initial MVR 710 without any additional valid MVRs. Also, if the reduced MVR described in FIG. 7 is smaller than the minimum candidate size, the verified MVR 710 can include the initial MVR 710. In one embodiment, the robotic system 100 can derive an increased MVR by first enlarging the initial MVR 710. The robotic system 100 can then iteratively reduce the increased MVR to derive the final MVR. The robotic system 100 can use the final verified MVR to register and / or manipulate the unregistered object.

[0071] Generating a Minimum Feasible Region (MVR) Auto-Register Pipeline FIG. 9 shows a flow diagram of an object recognition process 900 for recognizing an object, according to an embodiment of the present technology. The process may be based on information available to and / or received by the robotic system 100 of FIG. 1. For example, the robotic system 100 may receive / access (e.g., via the first image sensor 212 of FIG. 2) 2D and / or 3D image data representing the object stack 210 of FIG. 2 and / or the objects therein (e.g., registered objects and / or unrecognized objects). The robotic system 100 may access predetermined and stored data, such as the minimum possible dimensions (also referred to as minimum candidate size) of the package 112 of FIG. 1, the maximum possible dimensions (also referred to as maximum candidate size) of the package 112, or a combination thereof. The robotic system 100 may further access and utilize master data (also referred to as an object set) containing known or predetermined data / descriptions regarding receivable or predicted objects. For example, the object set may include descriptions of characteristics associated with various objects, such as object dimensions, surface image / appearance, pattern, color, etc. When making the comparison, the dimensions of the individual objects (i.e., packages 112, etc.) in the target stack 210 may or may not be known by the robotic system 100.

[0072] The object recognition process 900 may include a descriptor-based detection process (block 902). In one embodiment, the descriptor-based detection process may include a visual or computer vision analysis of the 2D image data. Based on the visual analysis, a detection estimate may be generated based on the object set (block 904). The detection estimate may include a hypothesis that a portion of the image data (e.g., the 2D image data) matches or corresponds to an object represented in the object set. As an example, the robotic system 100 can generate the detection estimate based on matching image portions / features, descriptors, and / or other appearance-related data between the object set and the 2D image data.

[0073] The descriptor-based detection process may include verifying a hypothesis (block 906). In other words, the robotic system 100 can verify the detection estimate based on the characteristics of registered objects in the object set and / or other sensor data (e.g., 3D image data, such as point cloud data). The robotic system 100 can verify the hypothesis by inspecting / matching other features, local feature descriptors, and / or 3D image data from a 3D depth sensor. For example, the robotic system 100 can generate a hypothesis based on matching a set of important visual features (e.g., logos, names, largest / brightest features, etc.) with the object set and verify the hypothesis based on comparing other remaining visual features with the object set. The robotic system 100 can also verify a hypothesis based on identifying the location of 2D image data corresponding to the surface corresponding to the hypothesis (i.e., its boundary / edge). For example, the robotic system 100 can calculate the characteristics (e.g., edge dimensions and positions) of the relevant 2D edges corresponding to the identified portion of the 2D image data. The robotic system 100 can verify a hypothesis based on comparing the 2D edge positions / lengths of the hypothesized portion of the 2D image data with the edge positions / lengths represented in the 3D image data (e.g., point cloud). The robotic system 100 can verify a hypothesis if the 3D edge characteristics (e.g., length) corresponding to the hypothesized area match the edge characteristics of the hypothesized object represented in the object set. In other words, the robotic system 100 can verify that the hypothesized portion of the 2D image data matches the object represented in the object set if additional aspects of the hypothesized portion, such as the corresponding 3D edge positions / lengths, match additional aspects of the object represented in the object set.

[0074] The object registration process 900 may include point cloud detection (block 908). In some embodiments, the point cloud detection process may include 3D analysis of the 3D image data from the 3D depth sensor. Point cloud detection may include segmenting the point cloud data to identify edge and / or individual layer features (block 910). For example, the robotic system 100 may segment the point cloud data according to depth measurements. As described above, the robotic system 100 may identify groups of adjacent locations having depth measurements within a threshold range from each other. Thus, the robotic system 100 may identify groups corresponding to layers and / or surfaces of the target stack 210. In some embodiments, the robotic system 100 may identify edges / boundaries, surface orientations, and / or discontinuities from the 3D image data. The robotic system 100 may use the identified edges / boundaries, surface orientations, and / or discontinuities to identify layers separated by the edge, having the same surface orientation, and / or on opposite sides of the discontinuity. Additionally or alternatively, the robotic system 100 can use edges identified in the 2D image data to identify edges and / or authenticate 3D edges.

[0075] A detection estimate may be generated based on the measured dimensions (block 912). The detection estimate may include a hypothesis that a portion of the 3D image data corresponds to or belongs to an object represented in the object set. For example, the robotic system 100 can generate a detection estimate based on the measured dimensions of each individual layer. The robotic system 100 can assume that the identified edges / boundaries of the layer / surface correspond to edges on the perimeter of one or more objects represented in the object set. For example, the robotic system 100 measures or calculates edge lengths as described above. The robotic system 100 can generate a detection estimate that includes portions of the image and / or corresponding regions of the target stack 210 that correspond to or match known / registered objects in the object set.

[0076] The robotic system 100 may verify the detection hypothesis (block 914). The robotic system 100 can verify the hypothesis by comparing other characteristics of the hypothesized area with other characteristics of the registered objects in the object set. For example, if the measured dimensions of an edge forming a layer match the corresponding dimensions of one of the registered objects, the robotic system 100 can verify that the object bounded by the compared edge is likely the matching registered object. Alternatively, or additionally, the robotic system 100 can verify the hypothesis based on analyzing portions of the 2D image data that correspond to the hypothesized portion of the 3D image data. For example, the robotic system 100 can determine the coordinates / locations of the identified 3D edge points used for length comparison. The robotic system 100 can identify corresponding locations / portions in the 2D image data of the object stack 210. The robotic system 100 can verify the hypothesis by comparing the identified portions of the 2D image data with corresponding representations of surfaces (which may include surface images) stored in the object set. If the 2D characteristics (e.g., appearance, such as brightness, color pattern, etc.) of the identified portion of the 2D image match the surface representation of the hypothesized registered object, the robotic system 100 can verify the hypothesis and detect that the corresponding object is present at the location of the object stack 210. Thus, the robotic system 100 can identify the known object (e.g., recognized object 312 in FIG. 3B ) in the object stack 210.

[0077] The object recognition process 900 may begin with a non-empty object set. In one embodiment, the object recognition process 900 may begin with either descriptor-based detection and / or point cloud-based detection, as described above with respect to Figure 9. Alternatively, or additionally, the object recognition process 900 may perform descriptor-based detection or point cloud detection without using other detection methods or merging different detection methods.

[0078] 10 is a block diagram of a method 1000 for operating the robotic system 100 of FIG. 1 in accordance with an embodiment of the present technology. The robotic system 100 can perform the method 1000 to process 2D / 3D image data generated by and / or received from the first image sensor 212 of FIG. 2. For example, the robotic system 100 can perform the method 1000 by executing processor instructions stored in a memory device using one or more processors. Thus, the one or more processors can perform the process(es) described herein. The one or more processors can receive data, analyze the data, generate results / commands / settings, and / or communicate commands / settings to operate one or more robotic units.

[0079] The method 1000 may include recognizing known / registered objects (e.g., recognized objects 312 of FIG. 3B) in the object stack 210 of FIG. 2. According to the method 1000, the robotic system 100 may further process and register unrecognized objects 314 of FIG. 3B that correspond to unresolved regions of the image data. Thus, the method 1000 may include the object recognition process 900 of FIG. 9.

[0080] The robotic system 100 may identify (block 1002) that one or more unresolved regions remain in the image data after the object recognition process 900. The robotic system 100 may identify an unresolved region as any portion of the image data for the start location 112 in Figure 1 (e.g., within the horizontal boundaries of a pallet, cart, container, etc.) that does not return a match (or correspond to) an object represented in the object set (e.g., enrollment data in the master data). The robotic system 100 may determine that the unresolved region(s) correspond to one or more unrecognized objects 314 in Figure 3B.

[0081] The robotic system 100 can test the unresolved region(s) (block 1004). The robotic system 100 can test the unresolved region(s) according to their shape and / or their dimensions. Accordingly, the robotic system 100 can identify edges in the image data (e.g., 2D visual images and / or 3D point clouds) (block 1020). For example, the robotic system 100 can identify layers / surfaces in the unresolved region(s) of the image data (e.g., 2D images and / or 3D point clouds) based on identifying adjacent locations having depth measurements that are within a threshold range from each other. The robotic system 100 can identify exposed edges as boundaries of the identified layers where depth measurements across adjacent horizontal locations deviate outside a threshold range. The robotic system 100 can also identify edges based on analyzing the 2D image according to changes in brightness, color, etc. In one embodiment, the robotic system 100 can identify edges using a Sobel filter.

[0082] The robotic system 100 can test the unsolved region(s) by comparing the identified edges of the unsolved region(s) to predetermined shape templates and / or lengths. For example, the robotic system 100 can compare the shape of the unsolved region (i.e., defined by a set of connected edges) to a shape template (e.g., a box and / or a cylindrical shape) representing the predicted object. The robotic system 100 can adjust the dimensions of the shape template to test the shape of the unsolved region. Additionally or alternatively, the robotic system 100 can calculate the length of the identified edges (e.g., edges shown in the 2D and / or 3D image data) and compare the length to predetermined minimum / maximum thresholds.

[0083] If the shape and / or length meet a predetermined threshold / condition, the robotic system 100 may perform object detection using MVR analysis as described above and generate an MVR (block 1006). The robotic system 100 may use the identified edges / corners to derive an initial MVR (block 1022). For example, the robotic system 100 can select one of the open corners and trace the corresponding open edge to find other eligible 2D / 3D edges (e.g., the estimated edge 424 in FIG. 4B and / or the opposing parallel edges 724, 728 in FIG. 7). The robotic system 100 can derive an initial MVR if the open edge and the other eligible edges meet a predetermined condition / threshold (e.g., minimum / maximum size requirements).

[0084] In one embodiment, the robotic system 100 can further process the initial MVR, such as by expanding the initial MVR (block 1024) and / or by shrinking the initial MVR (block 1026). The robotic system 100 can expand the initial MVR based on determining additional valid MVR regions 812a-812b in FIG. 8 as described above. The robotic system 100 can shrink the MVR (e.g., the initial MVR and / or the expanded MVR) based on finding edges starting from a different corner (e.g., a corner diagonally opposite the originally selected corner). For example, the robotic system 100 can consider other edges (e.g., the incomplete edge 712 in FIG. 7 and / or other 2D / 3D edges) to shrink the MVR.

[0085] The robotic system 100 may validate the adjusted and / or initial MVR (block 1028). The robotic system 100 may derive a validated MVR 820 of FIG. 8 as a combination of the initial MVR and the valid MVR region if the valid MVR region and / or combination meets a predetermined condition / threshold (e.g., corresponding minimum / maximum size requirements). The robotic system 100 may derive a validated MVR 720 of FIG. 7 if the area bounded by the considered edge(s) meets a predetermined condition / threshold (e.g., corresponding minimum / maximum size requirements). If the considered edge(s) do not meet a predetermined condition / threshold, the robotic system 100 may derive a validated MVR based on the last valid result. For example, the robotic system 100 may set the initial MVR as the validated MVR if the enlarged and reduced results do not meet the test condition. The robotic system 100 may also generate a validated MVR 820 if the subsequent reduction does not meet the test condition. In one embodiment, the robotic system 100 can first shrink and then expand the MVR, thus allowing the robotic system 100 to generate a verified MVR 720 if the subsequent expansion does not meet the test conditions.

[0086] Each verified MVR may represent an estimate of one surface (e.g., the top surface) of the unrecognized object 314 in the unresolved region. In other words, each verified MVR may represent an estimated detection representing one object in the unresolved region.

[0087] The robotic system 100 can further analyze the MVR to determine whether the MVR is certain or uncertain (block 1008). For example, the robotic system 100 can determine the number of open corners (e.g., 3D corners) included in and / or utilized in deriving the MVR and classify the MVR accordingly. The robotic system 100 can classify / determine the classification of the MVR based on comparing the number of associated open corners to a predetermined threshold. As an illustrative example, the robotic system 100 can determine an initial and / or verified MVR that includes or is derived using two or fewer open 3D corners as a certain MVR. Additionally, the robotic system 100 can determine an initial and / or verified MVR that includes or is derived using three or more open 3D corners as a certain MVR.

[0088] In one embodiment, the robotic system 100 can determine the authenticity of the initial MVR (e.g., before blocks 1024-1028). If the initial MVR is a reliable MVR, the robotic system 100 can bypass the above operations of blocks 1024-1028. If the initial MVR is an uncertain MVR, the robotic system 100 can perform the above processes of blocks 1024-1028. In other embodiments, the robotic system 100 can scale, reduce, and / or verify the processed MVR before determining the authenticity of the verified MVR.

[0089] The robotic system 100 may update the object set if the MVR is certain (block 1030). In one embodiment, the robotic system may update the object set with the MVR according to the classification of the MVR. For example, using the certain MVR, the robotic system 100 may detect objects, such as by determining / concluding that each certain MVR corresponds to one detected object (e.g., one instance of the unrecognized object 314). Thus, the robotic system 100 may detect and / or directly register the certain MVR (e.g., without further tuning, testing, and / or verifying the MVR), such as by storing the certain MVR (e.g., visual image and / or corresponding dimensions) in the object set. Using the updated object set (i.e., the object set updated with the certain MVR), the robotic system 100 may perform the object recognition process 900 again, as represented by a feedback loop.

[0090] If the MVR is uncertain, the robotic system 100 may perform the task according to the verified MVR (block 1010). The robotic system 100 can use the verified MVR to derive the grasp position 420 of FIG. 4B. For example, the robotic system 100 can derive the grasp position 420 that is aligned with or abuts an exposed corner of the verified MVR. Alternatively, the robotic system 100 can derive the grasp position 420 that overlaps a middle portion of the verified MVR. Based on the grasp position 420, the robotic system 100 can derive commands, settings, and / or motion plans to operate the robot arm 202 of FIG. 2 and / or the end effector 204 of FIG. 2 to grasp and move the object. For the task, the robotic system 100 can move the object to the task position 116 of FIG. 1. Additionally, the robotic system 100 may further position the object at one or more specific locations during movement, such as presenting the object to one or more sensors and / or positioning the object relative to one or more lines / planes.

[0091] The robotic system 100 may acquire additional data about the object during its movement (block 1032). For example, the robotic system 100 can grasp the object and perform an initial movement (e.g., lift it a predetermined distance and / or move it horizontally). After the initial movement, the robotic system 100 can generate one or more updated 2D / 3D image data (e.g., via the first image sensor 212). Based on the increasing separation between the grasped object and surrounding objects, the robotic system 100 can analyze the updated 2D / 3D image data to re-derive the edges, MVR, and / or object dimensions. The robotic system 100 can also calculate the height of the object according to one or more intersection sensors (not shown). The robotic system 100 can determine the height of the end effector 204 when the bottom of the object being moved enters / exits a laterally oriented detection line / plane associated with an intersection sensor (e.g., a sensor configured to detect an interruption in an optical signal). The robotic system 100 can calculate the height based on the height of the end effector and the known height of the detection line / plane, and can acquire images of the surroundings, identifier values / positions, weights, and / or other physical descriptions of the object being moved.

[0092] The robotic system 100 may register the moved unrecognized object (block 1034). The robotic system 100 may register the object based on storing additional data and / or the verified MVR in the object set as described above. The robotic system 100 may further recognize other objects using the updated object set, as indicated by the feedback loop to the object recognition process 900.

[0093] When performing the process corresponding to block 1010, the robotic system 100 may return the uncertain MVR along with the detected object (e.g., resulting from the object recognition process 900) and / or the newly registered detected object (e.g., the certain MVR) to a planning module (not shown). The planning module can derive a motion plan to manipulate / move the recognized object, the detected object (e.g., the unrecognized object corresponding to the certain MVR), and / or other unrecognized objects. The robotic system 100 can further acquire additional information (e.g., object height, identifier data, profile image, etc.) during the movement of the unrecognized object and can use the acquired additional information to update the registration data of the object set. The robotic system 100 can use the acquired additional information to register the unrecognized object and / or derive a final verified MVR.

[0094] In one embodiment, the robotic system 100 can process an unresolved region and generate all initial / verified MVRs for that region. Following the iterative implementation of method 1000, the robotic system 100 can compare the unresolved region with the object set to initially determine and register a certain MVR among the generated MVRs. Using the updated object set (e.g., certain MVR), the robotic system 100 can re-perform the object recognition process 900 to detect objects in the unresolved region that correspond to the certain MVR. The robotic system 100 can update the unresolved region by identifying areas in the unresolved region that match the certain MVR used to update the object set, thereby detecting the corresponding objects. The robotic system 100 can finish updating the unresolved region by removing portions of the unresolved region that correspond / match to certain MVRs. The robotic system 100 can repeat this iterative process until the resulting unresolved region no longer contains certain MVRs. With respect to the uncertain MVR, the robotic system 100 can derive / use the corresponding verified MVR from the unresolved region to manipulate the unrecognized object within that region. The robotic system 100 may further acquire additional information while manipulating the unrecognized object, as described above. The robotic system 100 may update the object set with the verified MVR and / or additional information, continue the object recognition process 900, recognize other objects that match the moved object, and repeat the iterative process. The robotic system 100 updates the unresolved region by identifying regions within the unresolved region that match the verified MVR of the moved object, thereby detecting the corresponding object. The robotic system 100 may finish updating the unresolved region by removing the portion of the unresolved region that matches the verified MVR of the moved object. The robotic system 100 can process the updated unresolved region to derive and verify the next MVR corresponding to the next type / instance of the unrecognized object.

[0095] Additionally or alternatively, the robotic system 100 may determine recognized objects, derive all MVRs, and then plan the object movements. In other embodiments, the robotic system 100 may derive or update MVRs based on first moving recognized objects and / or objects corresponding to certain MVRs. Thus, the robotic system 100 may increase the likelihood of exposing additional 3D edges / corners of unrecognized objects. Increasing the likelihood of exposing additional 3D edges / corners can increase the likelihood of identifying certain MVRs, which can improve the accuracy of detecting objects and derived MVRs. Furthermore, improved accuracy can improve efficiency by reducing / eliminating processing steps and / or reducing object loss and / or collisions due to inaccurate object detection.

[0096] The robotic system 100 can autonomously populate the object set using the embodiments described herein. In other words, the robotic system 100 can start with an empty object set that does not contain enrollment data. Under such conditions, the robotic system 100 can perform the method 1000, starting at block 1002, and identify the entire received image data as unresolved territory. Because no enrollment data exists, the robotic system 100 can enroll objects with little or no human operator input as the objects are processed (e.g., moved from the start location 114 to the task location 116). Thus, the robotic system 100 can begin deriving MVRs without performing the object recognition process 900 and store a subset of the derived MVRs (e.g., reliable MVRs) in the object set. The robotic system 100 can iteratively perform the method 1000 as described above to enroll objects. For example, the robotic system 100 can detect a first set of objects represented in the unresolved region based on matching corresponding portions in the unresolved region with a subset of derived MVRs stored in the object set. The robotic system 100 can update the unresolved region by removing matching portions from the unresolved region according to the detection of the first set of objects. The robotic system 100 can process the updated unresolved region to register other objects in the updated unresolved region and derive a new / next set of MVRs (e.g., another set of confirmed MVRs and / or other verified MVRs).

[0097] Processing system example 11 is a block diagram illustrating an example of a processing system 1100 that may perform at least some of the operations described herein. As shown in FIG. 11, the processing system 1100 may include one or more central processing units (“processors”) 1102, a main memory 1106, a non-volatile memory 1110, a network adapter 1112 (e.g., a network interface), a video display 1118, input / output devices 1120, a controller 1122 (e.g., a keyboard and pointing device), a drive unit 1124 including a storage medium 1126, and a signal generator 1130, which are communicatively coupled to a bus 1116. The bus 1116 is shown as an abstraction that may represent any one or more separate physical buses, point-to-point connections, or both, connected by appropriate bridges, adapters, or controllers. The bus 1116 may thus include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) Standard 1394 bus, also known as "Firewire."

[0098] In various embodiments, the processing system 1100 operates as part of a user device, although the processing system 1100 may be connected (e.g., wired or wirelessly) to the user device. In a network deployment, the processing system 1100 may operate as a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0099] The processing system 1100 may be a server computer, a client computer, a personal computer, a tablet, a laptop computer, a personal digital assistant (PDA), a cellular phone, a processor, a web appliance, a network router, switch or bridge, a console, a handheld console, a game console, a music player, a network-connected ("smart") television, a television-connected device, or any portable device or machine capable of executing (sequentially or otherwise) a set of instructions that specify actions to be taken by the processing system 1100.

[0100] Although the main memory 1106, the non-volatile memory 1110, and the storage medium 1126 (also referred to as a "machine-readable medium") are shown as one medium, the terms "machine-readable medium" and "storage medium" should be taken to include one medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions 1128. The terms "machine-readable medium" and "storage medium" should be taken to include any medium that can store, encode, or convey a set of instructions for execution by a computer system, causing the computer system to perform any one or more of the methodologies of the embodiments of the present disclosure.

[0101] Generally, the routines executed to implement the disclosed embodiments may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions referred to as a “computer program.” A computer program typically includes one or more instructions (e.g., instructions 1104, 1108, 1128) stored at various times in various memories and storage devices of the computer, which, when read and executed by one or more processing units or processors 1102, cause the processing system 1100 to perform operations that implement elements associated with various aspects of the disclosure.

[0102] Furthermore, while embodiments have been described in the context of fully functional computers and computer systems, those skilled in the art will appreciate that various embodiments may be distributed as program products in a variety of forms, and that the disclosure applies equally regardless of the particular type of machine or computer-readable medium used to actually achieve the distribution. For example, the techniques described herein may be implemented using virtual machines or cloud computing services.

[0103] Further examples of machine-readable storage media, machine-readable media, or computer-readable (storage) media include, but are not limited to, recordable type media such as volatile and non-volatile memory devices 1110, floppy and other removable disks, hard disk drives, optical disks (e.g., compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs)), and transmission media such as digital and analog communication links.

[0104] The network adapter 1112 enables the processing system 1100 to broker data from the network 1114 to entities external to the processing system 1100 through any known and / or convenient communication protocol supported by the processing system 1100 and the external entities. The network adapter 1112 may include one or more of a network adapter card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multi-layer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, and / or a repeater.

[0105] Network adapter 1112 may include a firewall, which in some embodiments governs and / or manages permissions to access / proxy data on a computer network and tracks varying levels of trust between different machines and / or applications. A firewall may be any number of modules having any combination of hardware and / or software components that can enforce predetermined sets of access rights between machines and applications, machines and / or particular sets of applications, for example, to regulate traffic flow and resource sharing between these various entities. A firewall may also manage and / or have access to access control lists that detail permissions, including, for example, rights by individuals, machines, and / or applications to access and act on objects and the circumstances under which the permissions are valid.

[0106] In one embodiment, a method of operating a robotic system includes receiving two-dimensional (2D) image data and three-dimensional (3D) image data representing an object stack including one or more unrecognized objects; identifying unresolved regions of the 2D and / or 3D image data based on a comparison with an object set representing one or more physical aspects of registered objects; identifying one or more layers within the unresolved regions of the 2D image data and / or the 3D image data, each layer including a set of laterally adjacent points in the 3D image data having depth measurements within a threshold range from each other; and identifying boundaries of one or more layers where depth measurements across horizontally adjacent locations deviate outside the threshold range. identifying exposed edges of the unresolved region representing a surface of the unregistered object based on the exposed edges, each of the exposed edges representing one or more exposed corners, deriving a minimum feasible region (MVR) representing an estimate of the surface of the unregistered object represented in the unresolved region based on the identified exposed edges and exposed corners, determining a number of exposed corners associated with the derived MVR, detecting the unrecognized object according to the MVR and the number of exposed corners, and registering the unrecognized object based on the MVR and / or updating the object set to include the MVR and / or processed results of the MVR as a representation of the unrecognized object. The method may further include iteratively updating and analyzing the unresolved region according to the updated object set based on comparing the unresolved region with the updated object set, identifying portions in the unresolved region that match the MVR and / or MVR processing results used to update the object set, detecting objects according to the portions that match the MVR and / or MVR processing results, and updating the unresolved region by removing the portions that match the MVR and / or MVR processing results. The object set may be an empty set for the initial iteration, and the unresolved region may include 2D image data, 3D image data, a portion thereof, or a combination thereof for the initial iteration.For the first iteration, registering the unrecognized object may include autonomously populating the object set while performing a task involving the unrecognized object in the target stack.

[0107] In one embodiment, detecting an unrecognized object may include directly detecting the unrecognized object if the number of exposed corners exceeds a predetermined threshold, and registering a recognized object may include directly storing the MVR in an object set if the number of exposed corners exceeds a predetermined threshold. In one embodiment, the predetermined threshold may be 2.

[0108] In one embodiment, the robotic system may include at least one processor and at least one memory device coupled to the at least one processor and storing instructions executable by the processor, the instructions including: receiving one or more image data representing one or more unrecognized objects at a start location; deriving a minimum feasible region (MVR) based on identifying one or more exposed corners in the one or more image data representing junctions between exposed edges that are not horizontally adjacent to other surfaces or objects, respectively; updating an object set representing one or more physical aspects of previously registered objects with an MVR representing one or more physical attributes of one of the unrecognized objects at the start location and / or results of processing the MVR based on the number of exposed corners associated with the MVR; and recognizing other objects at the start location based on the updated object set. The at least one memory device may include instructions for determining the MVR as a certain MVR based on comparing the number of exposed corners to a predetermined threshold, such as when the MVR is associated with three or more exposed corners, and / or instructions for updating the object set directly with the MVR and / or results of processing the MVR without further updating or adjusting the MVR based on determining the MVR as a certain MVR. The at least one memory device may further include instructions for identifying unresolved regions of the image data that do not return a match with an object represented in the object set, instructions for deriving a certain MVR within the unresolved region and an MVR based on the unresolved region, and instructions for removing portions of the unresolved region that match the certain MVR for further object recognition and / or manipulation. In one or more embodiments, the at least one memory device may further include instructions for performing an operation to move an unrecognized object corresponding to the certain MVR away from the start position, instructions for deriving a next MVR based on the updated unresolved area after performing the operation to move the unrecognized object, the next MVR representing a further unrecognized object at the start position and represented in the updated unresolved area, and instructions for updating the object set with a second MVR and / or a result of processing the second MVR.

[0109] In one embodiment, the robotic system may include at least one memory device having instructions for determining an MVR as an uncertain MVR if the MVR is associated with two or fewer exposed corners, and for updating the MVR and / or the object set with the MVR and / or the results of processing the MVR based on further analyzing the uncertain MVR and / or manipulating one or more of the unrecognized objects at the start location. The at least one memory device may include instructions for detecting one or more recognized objects based on comparing the image data to the object set, instructions for identifying unresolved regions of the image data that do not return a match with the object set, and instructions for deriving an MVR within the unresolved regions representing an estimate of the surface of one of the unrecognized objects at the start location based on identifying one or more exposed corners in the unresolved regions of the image data. The at least one memory device may further include instructions for performing an operation to move one or more recognized objects away from the start location, instructions for updating the MVR after performing the operation to move the one or more recognized objects, and instructions for updating the object set with the updated MVR and / or the results of processing the MVR.

[0110] In one embodiment, a tangible, non-transitory computer-readable medium having processor instructions stored thereon, when executed by a processor, causes the processor to perform a method including: iteratively analyzing one or more received image data representing registered and / or unrecognized objects at a start location by deriving one or more minimum feasible regions (MVRs) based on identifying one or more exposed corners in the received image data that represent junctions between exposed edges that are not horizontally adjacent to other surfaces or objects, respectively; determining a classification of each MVR based on the number of exposed corners associated with the corresponding MVR; updating an object set representing one or more physical aspects of a previously registered object with an MVR and / or processed results of the MVR, respectively, that represent one or more physical attributes of one of the unrecognized objects at the start location according to the corresponding classification; and recognizing another object at the start location using the updated object set. In an embodiment, the one or more received image data include two-dimensional (2D) image data and three-dimensional (3D) image data both representing a stack of objects from one imaging location, and the method may further include generating a detection estimate based on comparing the 2D image data to the object set and verifying the detection estimate based on the 3D image data. Generating the detection estimate may further include generating a detection estimate based on matching a portion of the 2D image data with a surface image of an object represented in the object set, and verifying the detection estimate may further include verifying the detection estimate based on identifying a portion of the 3D image data corresponding to the detection estimate, calculating a 3D edge length of the identified portion of the 3D image data, and verifying the detection estimate if the 3D edge length matches the edge length of the object represented in the object set.

[0111] In an embodiment, the non-transitory computer-readable medium may include a method, wherein the one or more received image data include two-dimensional (2D) image data and three-dimensional (3D) image data, both representing a stack of objects from one imaging location, and the method may further include generating a detection estimate based on comparing the 3D image data to an object set and validating the detection estimate based on the 2D image data. Generating the detection estimate may include generating the detection estimate based on identifying a layer in the 3D image data including a set of horizontally adjacent locations having depth measurements within a threshold range of each other, identifying edges of the layer that represent locations across the horizontally adjacent locations where the depth measurements deviate outside the threshold range, calculating edge lengths, and matching the edge lengths with dimensions of objects represented in the object set. Verifying the detection estimate may include verifying the detection estimate based on identifying a portion of the 2D image data that corresponds to the detection estimate, identifying 2D characteristics of the identified portion of the 2D image data, and validating the detection estimate if the 2D characteristics match corresponding characteristics of an image of one of the registered objects represented in the object set. In one embodiment, the non-transitory computer-readable medium may include a method further including iteratively constructing an object set from an empty set based on analyzing one or more received image data, where the analysis of the one or more received image data includes identifying one or more received image data or portions thereof as unresolved areas if the object set does not include registration data for any objects; deriving one or more MVRs from the unresolved areas; storing at least a subset of the one or more MVRs in the object set according to the corresponding classification; detecting a first set of objects represented in the received image data based on matching portions of the received image data with the one or more MVRs stored in the object set; updating the unresolved areas by removing the matched portions; and deriving a new set of MVRs based on the updated unresolved areas to register a second set of objects represented in the received image data.

[0112] As noted above, the techniques presented herein may be implemented, for example, in programmable circuitry (e.g., one or more microprocessors) programmed using software and / or firmware, in all-dedicated hardwired (i.e., non-programmable) circuitry, or in a combination of such forms. The dedicated circuitry may be in the form of, for example, one or more application-specific integrated circuits (ASICs), programmable logic circuits (PLDs), field-programmable gate arrays (FPGAs), etc.

[0113] From the foregoing, it will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications may be made without departing from the scope of the invention. Accordingly, the invention is not limited except as by the appended claims.

Claims

1. 1. A method of using a robotic system including at least one processor and at least one memory device coupled to the at least one processor and storing instructions executable by the processor, comprising: storing, in the at least one memory device, image data representative of the received one or more unrecognized objects; the at least one processor: identifying unresolved regions in the image data that do not correspond to registered objects based on a comparison with a set of objects representative of the registered objects; identifying one or more exposed corners associated with the unresolved region based on features depicted in the image data; deriving a region corresponding to an estimate of an unrecognized object based on the one or more exposed corners; generating a plan for moving the unrecognized object based on the region corresponding to the estimate of the unrecognized object; A method comprising:

2. the at least one processor: comparing the unresolved region with an updated object set; identifying a portion within the unresolved region that corresponds to an area corresponding to an estimate of the unrecognized object used to update the object set; detecting an object according to the portion corresponding to a region corresponding to the estimate of the unrecognized object; updating the unresolved region by removing the portion that corresponds to the region corresponding to the estimate of the unrecognized object; The method of claim 1 , further comprising iteratively updating and analyzing the unsolved region according to the updated object set based on:

3. the object set is an empty set for the first iteration, the unresolved region includes the image data, a portion thereof, or a combination thereof for the first iteration; The method of claim 2.

4. the at least one processor: determining a number of the one or more exposed corners associated with a region corresponding to the estimate of the unrecognized object; further comprising: The method of claim 1.

5. at least one processor; at least one memory device coupled to said at least one processor and storing instructions executable by said processor; wherein the instruction comprises: storing, in the at least one memory device, image data representative of the received one or more unrecognized objects; the at least one processor: identifying unresolved regions in the image data that do not correspond to registered objects based on a comparison with a set of objects representative of the registered objects; identifying one or more exposed corners associated with the unresolved region based on features depicted in the image data; deriving a region corresponding to an estimate of an unrecognized object based on the one or more exposed corners; generating a plan for moving the unrecognized object based on the region corresponding to the estimate of the unrecognized object; The robot system.

6. 6. The robotic system of claim 5, wherein the at least one memory device includes instructions for determining a region corresponding to the unrecognized object estimate as a certain region based on comparing the number of the one or more exposed corners to a predetermined threshold.

7. 7. The robotic system of claim 6, wherein the at least one memory device includes instructions for determining a region corresponding to the estimation of the unrecognized object as the certain region if the region corresponding to the estimation of the unrecognized object is associated with three or more of the one or more exposed corners.

8. 8. The robot system of claim 7, wherein the at least one memory device includes instructions for directly updating the object set with the region corresponding to the estimate of the unrecognized object based on the determination of the region corresponding to the estimate of the unrecognized object as the certain region without further updating or adjusting the region corresponding to the estimate of the unrecognized object.

9. the at least one memory device instructions to identify unresolved regions of the image data that do not return a match with an object represented in the object set; the certain region is within an unresolved region, and instructions for deriving a region corresponding to an estimate of the unrecognized object based on the unresolved region; instructions to remove portions of the unknown region that match the certain region for further object recognition and / or manipulation; The robotic system of claim 8 , comprising:

10. the at least one memory device instructions to perform an action of moving the unrecognized object corresponding to the certain region away from a start position; instructions for, after performing the action of moving the unrecognized object, deriving a next region based on the updated unresolved region that represents a further unrecognized object at the start position and that is represented in the updated unresolved region; instructions to update the object set in the next region; The robotic system of claim 9 , comprising:

11. the at least one memory device instructions for determining a region corresponding to an estimate of an unrecognized object as an uncertain region if the region corresponding to the estimate of an unrecognized object is associated with no more than two of the one or more exposed corners; instructions for updating the object set with regions corresponding to estimates of the unrecognized objects based on further analyzing the uncertainty regions and / or manipulating one or more of the unrecognized objects; The robotic system of claim 5 , comprising:

12. the at least one memory device instructions for detecting one or more recognized objects based on comparing the image data to the set of objects; instructions to identify unresolved regions of the image data that do not return a match with the object set; Including, the region corresponding to the unrecognized object estimate represents an estimate of a surface of one of the unrecognized objects. The robot system according to claim 5 .

13. the at least one memory device instructions to perform an action to move the one or more recognized objects away from a start position; instructions for updating a region corresponding to an estimate of the unrecognized object after performing the action of moving the one or more recognized objects; instructions to update the object set with the updated region; The robotic system of claim 12 , comprising:

14. A tangible, non-transitory computer-readable medium having stored thereon processor instructions that, when executed by a processor, cause the processor to perform a method including iteratively analyzing one or more received image data representing registered and / or unrecognized objects; The analysis identifying unresolved regions in the image data that do not correspond to registered objects based on a comparison with a set of objects representative of the registered objects; identifying one or more exposed corners associated with the unresolved region based on features depicted in the image data; deriving a region corresponding to an estimate of an unrecognized object based on the one or more exposed corners; generating a plan for moving the unrecognized object based on the region corresponding to the estimate of the unrecognized object; by, A tangible, non-transitory computer-readable medium.

15. the one or more received image data both include two-dimensional (2D) image data and three-dimensional (3D) image data representing the registered and / or unrecognized images from one imaging position; The method comprises: generating a detection estimate based on comparing the 2D image data to the object set; verifying the detection estimate based on the 3D image data; 15. The non-transitory computer-readable medium of claim 14, further comprising:

16. generating the detection estimate includes generating the detection estimate based on matching a portion of the 2D image data with surface images of objects represented in the object set; Validating the detection estimation includes: identifying a portion of the 3D image data corresponding to the detection estimate; calculating 3D edge lengths of the identified portion of the 3D image data; validating the detection estimate if the length of the 3D edge matches the length of an edge of the object represented in the object set; validating the detection estimate based on 16. The non-transitory computer-readable medium of claim 15.

17. the one or more received image data both include two-dimensional (2D) image data and three-dimensional (3D) image data representing the registered and / or unrecognized images from one imaging position; The method comprises: generating a detection estimate based on comparing the 3D image data to the object set; verifying the detection estimation based on the 2D image data; 15. The non-transitory computer-readable medium of claim 14, further comprising:

18. generating the detection estimate identifying a layer within the 3D image data comprising a set of horizontally adjacent locations having depth measurements within a threshold range of each other; identifying edges of the layer that represent locations where the depth measurements fall outside the threshold range across adjacent horizontal locations; calculating the length of the side; matching the edge lengths to dimensions of objects represented in the object set; generating the detection estimate based on Validating the detection estimation includes: identifying a portion of the 2D image data corresponding to the detection estimate; identifying 2D characteristics of the identified portion of the 2D image data; validating the detection estimation if the 2D characteristics match corresponding characteristics of an image of one of the registered objects represented in the object set; validating the detection estimate based on 20. The non-transitory computer-readable medium of claim 17.

19. The method further includes iteratively constructing the object set from an empty set based on analyzing the one or more received image data, wherein analyzing the image data includes: identifying the one or more received image data or portions of the image data as an unresolved area if the object set does not include any object registration data; deriving a region from the unresolved region corresponding to an estimate of the unrecognized object; storing at least a subset of the regions corresponding to the unrecognized object estimates in the object set according to the corresponding classifications; detecting a first set of objects depicted in the received image data based on portions of the received image data that match regions corresponding to estimates of the unrecognized objects stored in the object set; updating the unresolved region by removing the matched portion; deriving a new set of regions corresponding to estimates of the unrecognized objects based on the updated unresolved regions to register a second set of objects represented in the received image data; and 15. The non-transitory computer-readable medium of claim 14.

20. 10. The method of claim 1, further comprising: determining a number of the one or more exposed corners; and detecting the unrecognized object based on the number of the one or more exposed corners.

Images