Robotic system with automated package registration mechanism and auto-detection pipeline
The robotic system autonomously registers and manipulates unrecognized objects by generating a minimum viable region (MVR) using 2D and 3D image data, addressing the challenge of handling unknown objects and enhancing operational efficiency and safety.
Patent Information
- Application Number
- JP2024135810
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-05-24
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2039-10-29
AI Technical Summary
Existing robotic systems face challenges in accurately identifying and handling unknown or unrecognized objects during depalletization, leading to potential mishandling and increased risk of injury due to resource-intensive manual intervention.
A robotic system with automatic package registration mechanisms that autonomously registers and manipulates unrecognized objects by generating a minimum viable region (MVR) using 2D and 3D image data to determine the optimal area for contact and lift, reducing errors and improving accuracy through iterative MVR generation and verification.
Enhances the accuracy of object registration and handling, reducing errors and collisions, and enabling efficient autonomous operation without initial registration data, thereby improving operational efficiency and safety.
Smart Images

Figure 2024166204000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 752,756, filed October 30, 2018, which is incorporated herein by reference in its entirety. This application further claims the benefit of U.S. Provisional Patent Application No. 62 / 852,963, filed May 24, 2019, which is incorporated herein by reference in its entirety. This application is U.S. Patent Application No. 16 / 290,741, filed March 1, 2019, which is now also related to U.S. Patent No. 10,369,701, which is incorporated herein by reference in its entirety.
[0002] This application contains subject matter related to concurrently filed U.S. patent application "A ROBOTIC SYSTEM WITH AUTOMATED PACKAGE REGISTRATION MECHANISM AND MINIMUM VIABLE REGION DETECTION" by Jinze Yu, Jose Jeronimo Moreira Rodrigues, and Rose Nikolaev Diankov. The related application is assigned to Mujin, Inc. and is identified by docket number 131837-8003.US40. The subject matter therein is incorporated by reference.
[0003] The present technology is directed generally to robotic systems, and more specifically to systems, processes, and techniques for registering objects. [Background technology]
[0004] Often, packages are placed on pallets (or "palletized") for shipment to a destination, where they are then depalletized. Packages may be depalletized by human workers, which can be resource intensive and increase the risk of injury. In industrial environments, the depalletizing operation is performed by industrial robots, such as a robotic arm that grasps, lifts, carries, and delivers packages to a release point. Also, imaging devices are utilized to capture images of the palletized stack of packages. The system may process the images to ensure that the packages are efficiently handled by the robotic arm, such as by comparing the captured images to registered images stored in a registered data source.
[0005] The captured image of the package may match the registered image. As a result, the physical characteristics of the photographed object (e.g., measurements of the package's dimensions, weight, and / or center of mass) may be unknown. Failure to accurately identify the physical characteristics may lead to a variety of undesirable consequences. For example, such a failure may cause a stop, which may require the package to be manually registered. Such a failure may also result in mishandling of the package, especially if the package is relatively heavy and / or tilted.
[0006] Various features and characteristics of the technology will become apparent to those skilled in the art upon review of the detailed description in conjunction with the drawings, in which: Embodiments of the technology are illustrated by way of example and not limitation in the drawings, in which like reference numbers may indicate similar elements. [Brief description of the drawings]
[0007] [Figure 1] 1 illustrates an example of an environment in which a robotic system may operate. [Diagram 2] 1 illustrates a robotic system in accordance with one or more embodiments of the present technology. [Figure 3A]1 illustrates an example of a stack of objects to be processed by a robotic system in accordance with one or more embodiments of the present technology. [Figure 3B] 1 illustrates a top view of an example stack, in accordance with one or more embodiments of the present technology. [Figure 3C] 1 illustrates sensor data corresponding to a top surface in accordance with one or more embodiments of the present technology. [Figure 4A] 13 illustrates sensor data corresponding to the top surface after a first set of movements in accordance with one or more embodiments of the present technology. [Figure 4B] 4B illustrates a portion of the sensor data shown in FIG. 4A in accordance with one or more embodiments of the present technology. [Diagram 5] FIG. 1 shows a top view of a set of palletized packages according to an embodiment of the present technology. [Figure 6] 1 shows a graphical representation of an exposed outer corner in accordance with an embodiment of the present technology. [Figure 7] FIG. 1 is a block diagram of a set of palletized packages in accordance with an embodiment of the present technology; [Figure 8] 1 illustrates an example of merged MVR regions, according to an embodiment of the present technology. [Figure 9] FIG. 2 is a flow diagram of an object recognition process, according to an embodiment of the present technology. [Figure 10] FIG. 2 is a block diagram of a method of operating the robotic system of FIG. 1 in accordance with an embodiment of the present technology. [Figure 11] FIG. 1 is a block diagram illustrating an example of a processing system capable of performing at least some of the operations described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] The drawings depict various embodiments for purposes of illustration only. Those skilled in the art will recognize that alternative embodiments may be employed without departing from the spirit of the technology. Thus, while specific embodiments are shown in the drawings, the technology may be susceptible to various modifications.
[0009] Described herein are systems and methods for a robotic system with an automatic package registration mechanism. A robotic system (e.g., a system incorporating devices that perform one or more designated tasks) configured according to an embodiment increases usability and flexibility by manipulating and autonomously / automatically registering unknown or unrecognized objects (e.g., packages, boxes, cases, etc.) (e.g., with little or no input from a human operator).
[0010] To determine whether an object has been recognized, the robotic system can obtain data about the object at a start position (e.g., one or more images of the exposed surface of the object) and compare it to enrollment data of known or expected objects. The robotic system can determine that the object has been recognized if the compared data (e.g., a portion of the compared image) matches the enrollment data of one of the objects (e.g., one of the registered surface images). The robotic system can determine that the object has not been recognized if the compared data does not match the enrollment data of the known or expected object.
[0011] The robotic system can manipulate the unrecognized object according to one or more estimates and can determine additional information about the unrecognized object (e.g., a surface image and / or physical dimensions. For example, the robotic system can identify exposed edges and / or exposed exterior corners of the unrecognized object that are separate from or not adjacent to other objects.
[0012] The estimation may include generating a minimum viable region (MVR) that represents a minimum and / or optimal area required to contact and lift a corresponding unrecognized object. Each MVR may further represent an estimation of the surface of one unrecognized object (e.g., a boundary around the surface). During MVR generation, exposed outer corners and exposed edges may be identified by inspecting two-dimensional (2D) and / or three-dimensional (3D) image data. Based on the identified exposed outer corners and exposed edges, an initial MVR may be generated by identifying edges opposite the exposed edges. The initial MVR may be further processed, such as by determining and testing expanded and / or contracted areas, to generate a verified MVR. The initial MVR may be processed according to the certainty of the MVR determination (e.g., a status or level representing the accuracy of the initial MVR). For example, the robotic system may identify the initial MVR as a solid MVR (e.g., an instance of a verified MVR that is likely to be accurate) if the initial MVR includes three or more exposed corners and / or is derived with three or more exposed corners, otherwise (e.g., if the initial MVR includes two or fewer exposed corners and / or is derived with two or fewer exposed corners), the robotic system may identify the initial MVR as an uncertain MVR (e.g., an instance of a verified MVR that is unlikely to be accurate).
[0013] The robotic system can use the verified MVR to register an unrecognized object, such as by storing the verified MVR and / or other processing results derived using the verified MVR. For example, the robotic system can begin registering a corresponding unrecognized object with a certain MVR. The robotic system can further manipulate the certain MVR object (e.g., by grasping and moving to perform a task), thereby increasing the likelihood of exposing additional corners of the object remaining in the starting position. After moving the object, the robotic system can repeat the detection and analysis to increasingly identify previously uncertain MVRs as certain MVRs based on the newly exposed corners.
[0014] Thus, the embodiments described herein increase the likelihood of accurately generating an MVR for an unrecognized object. The increased accuracy may lead to fewer errors, such as dropped and / or collided packages, caused by inaccurate data about the moved object. Furthermore, the increased accuracy of the MVR may lead to accurate enrollment data, which may provide increased efficiency resulting from reliably identifying subsequent processing of the same type of object. Furthermore, the embodiments described herein may be used to autonomously initiate and populate master data (e.g., collection of enrollment data) without initial data. In other words, a robotic system may autonomously enroll an object without existing enrollment data and without human operator interaction / input.
[0015] In the following description, numerous specific details are set forth to provide a thorough understanding of the technology of the present disclosure. In other embodiments, the technology introduced herein can be practiced without these specific details. In other instances, well-known features, such as specific functions or routines, are not described in detail so as not to unnecessarily obscure the present disclosure. In this specification, references to "an embodiment," "one embodiment," and the like, mean that the particular feature, structure, material, or characteristic being described is included in at least one embodiment of the present disclosure. Thus, appearances of such phrases in this specification do not necessarily all refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Moreover, particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that the various embodiments shown in the figures are merely representations for purposes of illustration and are not necessarily drawn to scale.
[0016] For clarity, some well-known details describing structures or processes that are often associated with robotic systems and subsystems, but that may unnecessarily obscure some important aspects of the disclosed technology, are not described below. Additionally, the following disclosure describes some embodiments of different aspects of the technology, but some other embodiments may have different configurations or components than those described in this section. Thus, the disclosed technology may have other embodiments that have additional elements or that do not include some of the elements described below.
[0017] Many embodiments or aspects of the disclosure described below may take the form of computer or processor executable instructions, including routines executed by a programmable computer or processor. Those skilled in the art will appreciate that the disclosed technology may be practiced in computer or processor systems other than those shown and described below. The technology described herein may be implemented in a special purpose computer or data processor that is specifically programmed, configured, or constructed to execute one or more of the computer executable instructions described below. Thus, the terms "computer" and "processor" as used generally herein refer to any data processor, and may include Internet appliances and handheld devices (including palmtop computers, wearable computers, cellular phones, mobile phones, multiprocessor systems, processor-based or programmable consumer electronics, network computers, minicomputers, etc.). Information handled by these computers and processors may be presented on any suitable display medium, including a liquid crystal display (LCD). Instructions for performing computer or processor executable tasks may be stored in any suitable computer readable medium, including hardware, firmware, or a combination of hardware and firmware. The instructions may be included in any suitable memory device, including, for example, a flash drive and / or other suitable medium.
[0018] The terms "coupled" and "connected," as well as derivatives thereof, may be used herein to describe a structural relationship between components. It should be understood that these terms are not intended as synonyms for each other. Rather, in certain embodiments, "connected" may be used to indicate that two or more elements are in direct contact with each other. Unless otherwise clear from the context, the term "coupled" may be used to indicate that two or more elements are in direct or indirect contact with each other (there are other intervening elements between the elements), that two or more elements cooperate or interact with each other (e.g., as in a causal relationship such as signal transmission / reception or function invocation), or both.
[0019] Favourable environment 1 is an example of an environment in which a robotic system 100 may operate. The robotic system 100 may include and / or communicate with one or more units (e.g., robots) configured to perform one or more tasks. Aspects of the packaging mechanism may be practiced or performed by various units.
[0020] For the example shown in FIG. 1 , the robotic system 100 may include an unloading unit 102, a moving unit 104 (e.g., a palletizing robot and / or a piece-picking robot), a transporting unit 106, a loading unit 108, or a combination thereof, in a warehouse or distribution / shipping point. Each unit of the robotic system 100 may be configured to perform one or more tasks. The tasks may be sequentially combined to perform actions to achieve a goal, such as unloading an object from a truck or van and storing the object in a warehouse, or unloading an object from a storage location and preparing it for shipment. For another example, a task may include placing an object in a target location (e.g., on a pallet and / or in a bin / cage / box / case). As described below, the robotic system may derive a plan for placing and / or stacking the objects (e.g., placement location / orientation, order in which to move the objects, and / or corresponding motion plan). Each unit may be configured to perform a sequence of actions (e.g., by operating one or more components of the unit) in accordance with one or more of the derived plans to perform a task.
[0021] In some embodiments, a task may include manipulating (e.g., moving and / or reorienting) a target object 112 (e.g., one of a package, box, case, cage, pallet, etc. corresponding to the task to be performed) from a start location 114 to a task location 116. For example, the unloading unit 102 (e.g., a devanning robot) may be configured to move the target object 112 from a position in a transport device (e.g., a truck) to a position on a conveyor belt. Also, the moving unit 104 may be configured to move the target object 112 from one position (e.g., a conveyor belt, a pallet, or a bin) to another position (e.g., a pallet, a bin, etc.). For another example, the moving unit 104 (e.g., a palletizing robot) may be configured to move the target object 112 from an original position (e.g., a pallet, a pick-up area, and / or a conveyor) to a destination pallet. Upon completing the operation, the transport unit 106 can move the target object 112 from an area associated with the mobile unit 104 to an area associated with the loading unit 108, and the loading unit 108 can move the target object 112 (e.g., by moving a pallet carrying the target object 112) from the mobile unit 104 to a storage location (e.g., a location on a shelf). More details regarding this task and its associated operations are provided below.
[0022] For illustrative purposes, the robotic system 100 is described in the context of a shipping center, however, it is understood that the robotic system 100 may be configured to perform tasks in other environments / for other purposes, such as manufacturing, assembly, packaging, healthcare, and / or other types of automation. It is also understood that the robotic system 100 may include other units, such as manipulators, service robots, modular robots, etc., that are not shown in FIG. 1 . For example, in one embodiment, the robotic system 100 may include a depalletizing unit that moves objects from a cage cart or pallet onto a conveyor or other pallet, a container switching unit that moves objects from one container to another, a packaging unit that packages objects, a sorting unit that groups objects according to one or more characteristics of the objects, a piece-picking unit that manipulates (e.g., sorts, groups, and / or moves) objects differently according to one or more characteristics of the objects, or a combination thereof.
[0023] The robotic system 100 may include physical or structural members (e.g., robotic manipulator arms) connected to joints for movement (e.g., rotational and / or translational displacement). The structural members and joints may form a kinematic chain configured to manipulate an end effector (e.g., a gripper) configured to perform one or more tasks (e.g., gripping, rotating, welding, etc.) depending on the use / operation of the robotic system 100. The robotic system 100 may include drives (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or manipulate (e.g., displace and / or reorient) the structural members about or at the corresponding joints. In an embodiment, the robotic system 100 may include a transport motor configured to transport the corresponding unit / chassis from location to location.
[0024] The robotic system 100 may include sensors configured to obtain information used to perform a task, such as to manipulate a structural member and / or transport a robotic unit. The sensors may include devices configured to detect or measure one or more physical characteristics of the robotic system 100 (e.g., the status, state, and / or position of one or more structural members / joints of the robotic system) and / or one or more physical characteristics of the surrounding environment. Some examples of sensors may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.
[0025] In one embodiment, for example, the sensor may include one or more imaging devices (e.g., visual and / or infrared cameras, two-dimensional (2D) and / or three-dimensional (3D) imaging cameras, distance measuring devices such as lidar or radar, etc.) configured to detect the surrounding environment. The imaging devices may generate a representation of the detected environment, such as a digital image and / or a point cloud, that may be processed via machine / computer vision (e.g., for automated inspection, robotic guidance, or other robotic applications). As described in more detail below, the robotic system 100 may process the digital image and / or point cloud to identify the target object 112, the start position 114, the task position 116, the pose of the target object 112, a confidence metric for the start position 114 and / or pose, or a combination thereof.
[0026] To manipulate a target object 112, the robotic system 100 can capture and analyze images of a designated area (e.g., a pick-up location, such as in a truck or on a conveyor belt) to identify the target object 112 and a start location 114 for the target object 112. Similarly, the robotic system 100 can capture and analyze images of another designated area (e.g., a drop location for placing the object on a conveyor, a location for placing the object in a container, or a location on a pallet for stacking) to identify a task location 116. For example, the imaging device can include one or more cameras configured to generate images of the pick-up area and / or one or more cameras configured to generate images of the task area (e.g., a drop area). Based on the captured images, the robotic system 100 can determine the start location 114, the task location 116, associated poses, packing / placing plans, transport / packing sequences, and / or other processing results, as described below.
[0027] In one embodiment, for example, the sensors may include position sensors (e.g., position encoders, potentiometers, etc.) configured to detect the position of structural members (e.g., robotic arms and / or end effectors) and / or corresponding joints of the robotic system 100. The robotic system 100 can use the position sensors to track the positions and / or orientations of the structural members and / or joints during task execution.
[0028] Object Movement and Registration Using Destination-Based Sensors 2 is a diagram of the robotic system 100 of FIG. 1 in accordance with one or more embodiments of the present technology. The robotic system 100 may include a robotic arm 202 (e.g., an instance of the mobile unit 104 of FIG. 1) including an end effector 204 (e.g., a gripper). The robotic arm 202 may be configured to move a target object 112 between a start location 114 of FIG. 1 and a task location 116 of FIG. 1. As shown in FIG. 2, the start location 114 may have a pallet 208 with a target stack 210 (e.g., a group of objects) on it. The task location 116 of the robotic arm 202 may be a loading location (e.g., a start / release point) of a conveyor 206 (e.g., an instance of the transport unit 106 of FIG. 1). For example, the robotic arm 202 may be configured to pick an object from the target stack 210 and place the object on the conveyor 206 for transport to another destination / task.
[0029] The robotic system 100 can use one or more sensors when performing movement operations with the robotic arm 202. In an embodiment, the robotic system 100 can include a first image sensor 212 and / or a second image sensor 214. The first image sensor 212 can include one or more 2D and / or 3D sensors, such as a camera and / or depth sensor, configured to image and / or analyze the start location 114. The second image sensor 214 can include one or more 2D and / or 3D sensors, such as a camera and / or depth sensor, configured to image and / or analyze the task location 116. For example, the first image sensor 212 can include one or more cameras and / or depth sensors positioned at a known position above and facing toward the start location 114. The first image sensor 212 can generate image data (e.g., a 3D point cloud and / or visual or 2D images) corresponding to one or more top views of the start location 114, such as a top view of the object stack 210. As described in further detail below, the robotic system 100 can use image data from the first image sensor 212 to derive a minimum viable region (MVR) of an unrecognized (e.g., unregistered) object in the target stack 210. The robotic system 100 can use the MVR to grasp (e.g., via the end effector 204) and manipulate (e.g., via the robot arm 202) the unrecognized object, such as when moving the unrecognized object from the start position 114 to the task position 116. The second image sensor 214 can also include one or more cameras and / or depth sensors positioned at one or more known positions above / beside and facing toward the task position 116, or in a related space. Thus, the second image sensor 214 can generate image data corresponding to one or more top and / or side views of the target object 112 at or within a threshold distance from the task position 116.
[0030] object recognition In accordance with one or more embodiments of the present technology, FIG. 3A is a diagram of an example stack of objects (e.g., target stack 210 of FIG. 2) processed by robotic system 100 of FIG. 1, FIG. 3B is a diagram of a top view of the example stack (e.g., actual top view 310 of target stack 210), and FIG. 3C is a diagram of sensor data (e.g., top view data 320) corresponding to the top view. With reference to FIGS. 3A, 3B, and 3C together, robotic system 100 may be configured to move objects in target stack 210 to other locations, such as task location 116 of FIG. 1 (e.g., conveyor 206 of FIG. 2), as described above. For object movement, robotic system 100 may use image data (e.g., top view data 320) from first image sensor 212 of FIG. 2 positioned above target stack 210. For example, top view data 320 may include one or more visual images and / or one or more depth maps that show or represent actual top view 310. Additionally, the robotic system 100 can analyze the top view data 320 to identify edges that may correspond to boundaries of an object. For example, the robotic system 100 can identify edges and / or continuous surfaces represented in the image data based on differences in depth measurements and / or image characteristics (e.g., different colors, linear patterns, shadows, differences in clarity, etc.). The robotic system 100 can identify exposed edges 322 (e.g., edges of the top surface of an object that are not horizontally abutting other objects / surfaces of approximately the same height) based on differences in depth measurements, etc.
[0031] The object stack 210 may include objects registered in the master data, which includes registration records of objects of predicted or previous processes, and / or unpredicted objects not registered in the master data. Thus, the robotic system 100 can recognize or identify objects in the object stack 210 using image data of the object surface 316. In some embodiments, the robotic system 100 can compare the image data, or one or more portions of the image data, to the master data to recognize objects in the object stack 210. For example, the robotic system 100 can identify a known object (e.g., a recognized object 312) in the object stack 210 when a portion of the top view data 320 matches one or more images of the object surface 316 in the registration data. The remaining portions of the actual top view 310 (e.g., portions that do not match the registration data) may correspond to an unrecognized object 314. The edges of the unrecognized object 314 are indicated in FIG. 3C using dashed lines.
[0032] Based on the image data matching, the robotic system 100 can locate the recognized object 312 in the corresponding image data, which may be further translated (e.g., via a pre-calibration table and / or equations that map pixel locations to a coordinate system) to a real-world location of the target stack 210. Additionally, the robotic system 100 can estimate the location of an unexposed edge of the recognized object 312 based on the matching. For example, the robotic system 100 can obtain the dimensions of the recognized object 312 from the master data. The robotic system 100 can measure a portion of the image data that is separated by a known dimension from an exposed edge 322 of the recognized object 312. According to the mapping, the robotic system 100 can determine one or more registration-based edges 324 of the recognized object 312 and / or similarly map the registration-based edges 324 to real-world locations, as described above.
[0033] In some embodiments, the robotic system 100 can identify exposed outer corners 326 of the object stack 210 represented in the image data (e.g., point cloud data). For example, the robotic system 100 can identify exposed outer corners 326 based on detecting intersections / junctions between two or more sets of exposed edges 322 (e.g., edges identified in the 3D image data, also referred to as 3D edges) having different orientations (e.g., extending at different angles). In one or more embodiments, the robotic system 100 can identify exposed outer corners 326 when the exposed edges 322 form angles that are within a predetermined range (also referred to as an angle range), such as a threshold corner angle range greater than and / or less than 90 degrees. As described in more detail below, the robotic system 100 can use the exposed outer corners 326 and the corresponding exposed edges 322 to process and / or manipulate the unrecognized object 314.
[0034] Handling Unrecognized Objects In one embodiment, the robotic system 100 of FIG. 1 can process (e.g., identify and / or move) an object according to its recognition status and / or relative position within the object stack 210 of FIG. 2. For example, the robotic system 100 can first grasp and move the recognized object, and then generate another image data set from a sensor (e.g., the first image sensor 212 of FIG. 2). FIG. 4A is an illustration of sensor data 401 corresponding to a top surface after a first set of actions (e.g., grasping and moving the recognized object 312) in accordance with one or more embodiments of the present technology. The shaded portion of FIG. 4A corresponds to the change in depth measurements after removing the recognized object 312 shown in FIG. 3B and FIG. 3C.
[0035] In the event that the robotic system 100 does not identify any of the recognized objects 312 in the image data (e.g., 2D image and / or 3D point cloud), the robotic system 100 can process the image data to identify any exposed corners 326 and / or exposed edges 322 to locate the unrecognized object 314 of FIG. 3A. For example, the robotic system 100 can process the sensor data 401 (e.g., 3D point cloud) to identify exposed outer corners 326 and / or exposed edges 322 in a similar manner as described above. Thus, the robotic system 100 can further identify and / or locate exposed corners / edges after removing the recognized object 312. In an embodiment, the robotic system 100 can further identify exposed inner corners 402 as intersections / junctions between two or more of the exposed edges 322 according to corresponding thresholds. For example, the robotic system 100 can identify an exposed interior corner 402 as a junction between two or more of the exposed edges 322 that are greater than 180 degrees relative to a corresponding contiguous surface. In one embodiment, the robotic system 100 can identify an exposed interior corner 402 when the exposed edges 322 form an angle that is within a threshold angle range greater than and / or less than 270 degrees.
[0036] In one embodiment, when no recognized objects 312 remain, the robotic system 100 can identify a registered object 406 of the object stack 210 (e.g., among the unrecognized objects 314) based on an exposed corner and / or an exposed edge. For example, the robotic system 100 can evaluate the exposed corner / edge according to a set of preferences and / or a scoring mechanism. In one embodiment, the robotic system 100 can be configured to select the exposed outer corner 326 closest to the robot arm 202 of FIG. 2. The robotic system 100 can select a corresponding region of the sensor data 401 and a corresponding unrecognized object 314 associated with the selected exposed outer corner 326. In other words, after processing the recognized object 312, the robotic system 100 can process the unrecognized object 314 that forms / constitutes the exposed outer corner 326 of the object stack 210. Based on analyzing the corresponding portion of the sensor data 401 (e.g., by deriving an MVR), the robotic system 100 can grasp the unrecognized object 314, lift and / or move the grasped object horizontally, and / or photograph the grasped object for registration. Furthermore, after photographing the grasped object, the robotic system 100 can move the grasped object to a destination (e.g., the conveyor 206 in FIG. 2).
[0037] To further describe the analysis of the sensor data, FIG. 4B is a detailed view of a portion of the sensor data 401 of FIG. 4A, according to one or more embodiments of the present technology. The robotic system 100 can analyze the sensor data 401 to derive the MVR 412. The MVR 412 can represent a minimum size area used to contact, grasp, and / or lift an object. The MVR 412 can also represent an estimate of the surface and / or surrounding boundary of one unrecognized object. For example, the MVR 412 can be associated with the attachment area of the end effector 204, or a larger area with additional / buffer areas surrounding the attachment area. For example, the MVR 412 can be associated with an estimated location of the horizontal boundary / side of the unrecognized object (e.g., the selected enrollment target). In one embodiment, the MVR 412 may correspond to a minimum and / or maximum candidate size, which may correspond to the physical dimensions (e.g., length, width, height, diameter, circumference, etc.) of the smallest / largest possible instances of predicted objects in the target stack 210, respectively. In other words, the robotic system 100 may determine that there are no objects in the target stack 210 having dimensions smaller than the minimum candidate size or larger than the maximum candidate size. The minimum and maximum candidate sizes may be predefined values (i.e., values given or known prior to processing the target stack 210). More details regarding the derivation of the MVR 412 are provided below.
[0038] The robotic system 100 can use the MVR 412 to determine the gripping location 420. The gripping location 420 can correspond to an area on the object / stack that is directly under and / or in contact with the end effector 204 for an initial operation. In other words, the robotic system 100 can place a gripper on the gripping location 420 to grip the corresponding object for a next operation (e.g., a lifting process, a horizontal movement process, and / or a data collection process for registration). In one embodiment, the robotic system 100 can select the gripping location 420 from a set of possible gripping locations. For example, the robotic system 100 can select from the set of gripping locations according to the relative orientation of the arms (e.g., with a preference that the robotic arms extend across the exposed edge 322 and do not overlap other portions).
[0039] In one embodiment, the robotic system 100 can derive the MVR 412 based on the estimated edge 424. For example, the robotic system 100 can select or identify an instance of an exposed outer corner 326 (e.g., a 3D corner) of a top layer of the target stack 210. The robotic system 100 can move away from the selected exposed outer corner 326 along an associated exposed edge 322 (e.g., meeting / forming at the selected corner in 3D). While moving along the edge, the robotic system 100 can identify the estimated edge 424 based on 3D depth measurements of the sensor data 401 and / or differences in 2D image characteristics (e.g., brightness, color, etc.). The robotic system 100 can identify edges or lines of the 2D / 3D image data that intersect with the traversed edge and / or are within a threshold separation distance from the traversed edge. The robotic system 100 can test the identified edges when determining the estimated edge 424. The robotic system 100 may test / verify the inferred edge 424 based on comparing the orientation of the identified edge to the orientation of the exposed edges 322. For example, the robotic system 100 may verify the identified edge as the inferred edge 424 if the identified edge is parallel to one of the exposed edges 322. In one embodiment, the robotic system 100 may test for parallel orientation based on verifying that the distances between two or more corresponding points on the tested pair of edges (e.g., the identified edge and the non-crossed instance of the exposed edge 322) are equal. In one embodiment, the robotic system 100 may identify parallel orientation if the tested pair of edges and / or their extensions intersect a common edge at the same angle, such as when both edges and / or their extensions intersect the exposed edge 322 crossed instance at an angle between 80 degrees and 100 degrees.
[0040] Thus, the robotic system 100 can derive a grip position 420 that does not overlap with the detected lines (e.g., 2D edges and / or incomplete edges) and / or estimated edges 424. The robotic system 100 can derive the grip position 420 based on balancing the ratio between the distance between the edge of the MVR 412 and the nearest detected line and / or estimated edge 424. Because the robotic system 100 grasps an object at or near a corner based on the MVR 412, the robotic system 100 can derive a grip position 420 that reduces the maximum potential torque along any one particular direction based on balancing the ratio. Additionally, the robotic system 100 can further derive or adjust the MVR 412 to coincide with or extend outward from the estimated edge 424.
[0041] The robotic system 100 can use the derived gripping position 420 to manipulate the robot arm 202 and end effector 204 of FIG. 2. The robotic system 100 can grip an object (e.g., registered target 406) at a corner of the stack at the gripping position 420. In one embodiment, the robotic system 100 can lift and / or move the gripped object horizontally to clearly distinguish the previously unexposed side. For example, the robotic system 100 can lift and / or move the object horizontally a predetermined height corresponding to a minimum distance to accurately distinguish the side. Also, for example, the robotic system 100 can lift and / or move the object horizontally while monitoring and / or analyzing the change in height and / or tilt of the end effector, thereby allowing additional sides opposite the exposed side 322 to be recognized. The robotic system 100 can acquire and process data during and / or after the initial lifting to further describe the unrecognized object 314.
[0042] MVR Detection Overview The present embodiment may relate to generating an accurate minimum feasible region (MVR) of an object. Exposed outer corners and exposed edges may be identified by inspecting 2D and / or 3D image data (e.g., point cloud data). Based on the identified exposed outer corners and exposed edges, an initial MVR may be generated by identifying an opposite edge to the exposed edge. In one embodiment, the robotic system 100 may generate the MVR based on identifying an opposite edge (e.g., estimated edge 424 in FIG. 4B) that corresponds to (e.g., parallel to) the exposed edge. The initial MVR may extend from the exposed outer corner along the exposed edge to the opposite edge.
[0043] After determining the initial MVR, a possible MVR region (e.g., a set of laterally adjacent locations having depth measurements within a threshold range of each other) may be identified that extends from the initial MVR defined by the point cloud to the edge of the surface or layer. A merged MVR of the object may include the initial MVR and the possible MVR. A verified MVR may be generated by inspecting / testing the merged MVR. The verified MVR may represent the exact region that contains the unrecognized object. Based on the verified MVR, the robotic system 100 described herein may register the object and perform tasks on the object, such as grasping and / or moving the object.
[0044] In many cases, an edge (e.g., an outer or exposed edge) of an object may be identified. For example, an outer edge (e.g., exposed edge 322 in FIG. 3C) of an object located along the perimeter of the object stack 210 in FIG. 2 may be free of surrounding objects and / or may be distant from any surrounding objects. Thus, the imaging result (e.g., 3D point cloud) may include a sudden change in the indicated value (e.g., height measurement and / or image value such as color or brightness) representing the outer edge. However, in many cases, other edges may not be visible or may be difficult to precisely define (e.g., based on a threshold confidence value). For example, a design / image of a box surface may lead to an inaccurate edge being identified on the object. Thus, due to the inaccurate edges, it may be difficult to precisely isolate portions of the 2D / 3D image data.
[0045] The robotic system 100 may identify exposed outer corners 326 and / or exposed edges 322 of an object by inspecting image data (e.g., point clouds and / or 2D images) and determining one or more layers. For example, the robotic system 100 may identify a top layer (e.g., unrecognized object 314 in FIG. 3B ) of the object(s) of the target stack 210 based on the imaging results. Within the top layer, the robotic system 100 may select portions of the object and / or surface (e.g., regions having height values that are within a threshold continuous range of each other) for MVR derivation. The robotic system 100 may infer that exposed edges forming / defining a surface (e.g., one of the layers or a portion therein) correspond to the lateral / perimeter boundary of the surface and / or corresponding object. An initial MVR may be generated by identifying an opposing edge opposite an exposed edge, where the initial MVR may extend from an exposed outer corner (e.g., an intersection formed by a set of exposed edges and / or a perimeter boundary) along the exposed edge to the opposing edge.
[0046] The robotic system 100 can further process (e.g., refine) the initial MVR by scaling up and / or scaling down the initial estimate based on markers (e.g., incomplete edges) in the image data. The refined MVR can be inspected to determine a final MVR that is used to determine the grasp location 420 and / or register an unrecognized object.
[0047] FIG. 5 illustrates a top view of a set 500 of palletized packages (e.g., target stack 210 of FIG. 2) according to an embodiment of the present technology. As shown in FIG. 5, a first object 510 may include sides 512a-512d. The sides may represent the boundaries of the object from a given view (e.g., a top view). In certain circumstances, at least some of the sides 512a-512d (e.g., sides 512c-512d) may be closed and abut other adjacent objects / sides. Conversely, some of the sides (e.g., sides 512a-512b) may be open and correspond to exposed side 322 of FIG. 3C.
[0048] The first box 510 may include one or more exposed outer corners 514 that are spaced apart from or free of horizontally adjacent objects. The exposed outer corners 514 may correspond to the exposed outer corners 326 of the stack in FIG. 3. As described below, the image data (e.g., point clouds and / or 2D images) may be inspected to identify exposed edges and exposed outer corners of the package.
[0049] Point Cloud Segmentation FIG. 6 illustrates a graphical representation 600 of an exposed outer corner, according to an embodiment of the present technology. As illustrated in FIG. 6, a point cloud (e.g., the graphical representation 600) may be processed to identify an exposed outer corner 614 (e.g., one of the exposed outer corners 326 of FIG. 3). For example, the point cloud may correspond to a top view generated by a 3D camera of the first object 510 of FIG. 5, or a portion thereof. The point cloud may include a three-dimensional point cloud having multiple layers indicating depth. Each layer and / or surface may correspond to a set of horizontally adjacent depth values that are within a threshold continuous range from each other. For example, the threshold continuous range may require that horizontally adjacent locations have depth measurements that are within a threshold distance (e.g., less than one centimeter) from each other, or within a threshold distance according to a slope. The threshold continuous range may also define a floor and / or slope relative to a reference height (e.g., the highest / closest point of the target stack 310). A threshold contiguous range can similarly define limits of 2D image characteristics (e.g., color and / or brightness) to identify continuity of a set of horizontally adjacent locations. The depth of a layer of point clouds can correspond to a separation along a direction perpendicular to the surface of the corresponding object (e.g., vertical separation).
[0050] Thus, the point cloud may be analyzed and processed to separate layers and / or identify open 3D edges / corners. In one embodiment, the robotic system 100 (e.g., one or more processors in the robotic system 100) may identify layers based on grouping depth values of the point cloud according to one or more predefined continuity rules / thresholds. For example, the robotic system 100 may group a set of horizontally adjacent / connected depth values if the depth values are within a threshold continuity range of each other and / or if the depth values follow a certain slope that represents a flat or continuous surface. The robotic system 100 may identify exposed edges (e.g., exposed edges 512a and 512b in FIG. 5) as boundaries of the identified layers. In other words, the robotic system 100 may identify exposed edges 512a and 512b as horizontal peripheral locations of the layer / surface where qualifying depth changes occur. In general, the depth measurements of objects / edges forming an upper layer of the object stack 210 may have a smaller magnitude (e.g., representing a closer distance to the first image device 312) than objects / edges forming a layer below the upper layer.
[0051] In one embodiment, the robotic system 100 can determine exposed edges based on identifying visible lines in a 2D visual image. For example, a pallet and / or floor may correspond to a known color, brightness, etc. Thus, the robotic system 100 can identify lines that delimit such known patterns as exposed edges of the object(s). Additionally, the robotic system 100 can use the 2D analysis to verify the 3D identification of exposed edges.
[0052] Based on the exposed edges, the robotic system 100 can identify open 3D corners (e.g., exposed exterior corners 514). For example, the robotic system 100 can identify shapes / angles associated with the exposed edges. The robotic system 100 can be configured to determine exposed exterior corners 514 as locations of points where exposed edges (e.g., edges 512a-512b) intersect / form an angle within a threshold angle range (e.g., 80 degrees-100 degrees).
[0053] As an illustrative example, the robotic system 100 can identify an open 3D corner 614 by identifying a first region 612 and adjacent regions 616a-616c. The robotic system 100 can identify the first region 612 when a set of adjacent horizontal locations in the scanned region layer have depth values that are within a threshold continuous range from each other. The robotic system 100 can identify the adjacent regions 616a-616c as other horizontal locations that have depth values that are outside the threshold continuous range from the depth value of the first region 612. In one embodiment, the robotic system 100 can identify a side of the first region 612 and / or the beginning of an adjacent region 616a-616c when the depth value changes to fall outside the threshold continuous range and / or when the location of the depth value change matches a shape template (e.g., a straight line and / or a minimum separation width between objects). More specifically, the adjacent regions 616a-616c may have depth values that represent distances further from the first image sensor 212 than the depth value of the surface of the target stack 210 (i.e., the first region 612). The resulting edges between the first region 612 and the adjacent regions 616a and 616c may correspond to exposed edges. In an embodiment, identifying the open 3D corner 614 may include verifying that the first region 612 forms a quadrant and the adjacent regions 616a-616c correspond to remaining quadrants, such as locations outside the stack of objects, and / or empty space. Empty space may indicate space detected with a very sparse point cloud, which may be considered as point cloud noise.
[0054] Other 3D corners may be determined using the 3D point cloud. In an embodiment, the exposed exterior corners may be contour shapes, and the "L" shaped corners may not include valid corners. Thus, the robotic system 100 may identify edge segments that meet one of the requirements (e.g., minimum continuous straight line length) based on extending such edge segments by a predetermined length. If the extended edge segment intersects with another segment or the extended segment at an angle, the robotic system 100 may identify a point on the contour shape (e.g., the midpoint of the arc between the intersecting edge segments) as an exposed exterior corner.
[0055] In some embodiments, the 3D corners may be ranked. For example, a 3D corner surrounded by empty space (e.g., of an object at the top corner of a stack) may be ranked higher than other objects. The open 3D corner 614 may be ranked based on other factors, such as the size of the first region 612, the position of the open 3D corner 614 relative to the shape of the first region 612, the difference in depth values between the surrounding regions (e.g., between the first region 612 and adjacent regions 616a and 616c), and / or the horizontal distance between the first region 612 and other regions (e.g., other surfaces / objects) having depth values within a threshold continuous range from the depth value of the first region 616a.
[0056] In an embodiment, the robotic system 100 may identify incomplete edges. Incomplete edges may be edges identified in 2D and / or 3D analysis that may or may not be real edges. Some of the incomplete edges may correspond to real edges of the box / gaps between boxes that may not be identified due to the presence of noise of other objects and / or the capabilities / position of the imaging device (e.g., camera). Incomplete edges may be visual patterns or marks on the object surface detected from 2D image analysis, such as pictures or marks on the surface, or dividers / seams between flaps of a box that are taped together. Conversely, a box without a pattern may not have even one 2D line that can be identified as an incomplete edge. The robotic system 100 can identify incomplete edges for locations of sensor outputs that exceed the noise variance but do not fully meet the rules / thresholds for edge identification. In one embodiment, the robotic system 100 can use the 3D sensor output to identify exposed outer edges (e.g., edges around the perimeter of the first region 612) and the 2D sensor output to identify incomplete edges. The robotic system 100 can also identify incomplete edges as 2D or 3D edges that do not intersect with other edges at angles that are within an angle threshold range. More details regarding incomplete edges are provided below.
[0057] Generate the first MVR FIG. 7 illustrates a top view of a set of palletized packages according to an embodiment of the present technology. FIG. 7 further illustrates an initial MVR 710 associated with the top view (e.g., 3D and / or 2D sensor image of the target stack 210 of FIG. 2). The initial MVR 710 may represent an initial estimate of the boundary of the MVR and / or the edges of the corresponding objects. In other words, the initial MVR 710 may represent an initial estimate of the surface of one unrecognized object. The robotic system 100 can derive the initial MVR 710 using the open 3D corners and / or exposed edges. Deriving the initial MVR 710 can include inspecting each of the layers identified from the point cloud to identify objects, edges, corners, etc. For example, the robotic system 100 can derive the initial MVR 710 as regions of the point cloud data (e.g., 3D sensor output) that have depth values within a threshold continuous range from each other (e.g., the first region 612 of FIG. 6). In other words, the robotic system 100 can derive the initial MVR as a group of horizontally adjacent locations having sufficiently consistent or linearly patterned depth values corresponding to a continuous surface. Thus, the robotic system 100 can derive the initial MVR as an area bounded at least in part by exposed exterior corners 514 (e.g., first corner 714a and / or second corner 714b) and corresponding exposed edges. For example, the initial MVR 710 can extend from the first corner 714a along exposed edges 722 and 726 to the opposing edges 724 and 728. In one embodiment, the robotic system 100 can derive the initial MVR 710 by starting from an open 3D corner (e.g., first corner 714a) and following the exposed edges (e.g., away from the open 3D corner). The initial MVR 710 may be extended until the traced exposed edge intersects or encounters another exposed edge (e.g., an edge opposite an untraced edge and / or an edge parallel to an untraced edge, such as opposing edges 724 and / or 728).
[0058] In some cases, the initial MVR 710 may correspond to the surface of multiple objects for various reasons (e.g., space between objects, sensor granularity, etc.). Thus, the robotic system 100 may validate one or more dimensions of the derived initial MVR 710. The robotic system 100 may validate that one or more dimensions of the MVR 710 are greater than a minimum candidate size and less than a maximum candidate size. The threshold dimension may represent a minimum and / or maximum dimension of an object that is acceptable / expected to the robotic system 100. Also, the threshold dimension may represent a horizontal footprint of the end effector 204 of FIG. 2, such as to represent a minimum size of a gripping / contact area.
[0059] If one or more dimensions of the initial MVR 710 fall outside the threshold (e.g., by exceeding a maximum dimension or by falling below a minimum dimension), the robotic system 100 can adjust the initial MVR 710, such as by further segmenting the initial MVR 710 (e.g., the top layer) according to the incomplete edges 712 (e.g., 2D / 3D edges that do not match or intersect with other edges of one or more edges). In other words, the robotic system 100 can adjust / shrink the initial MVR according to the incomplete edges 712 and test the corresponding results. In one embodiment, the robotic system 100 can determine the incomplete edges 712 as 2D and / or 3D edges that do not intersect with the exposed edges of one or more edges. The robotic system 100 can also determine the incomplete edges 712 as 2D and / or 3D edges that are parallel to one of the exposed edges. In one embodiment, the robotic system 100 can calculate a confidence value associated with the incomplete edges 712. The confidence value may represent the likelihood that the incomplete edge 712 corresponds to an edge of a surface and / or a separation between adjacent objects. By way of example, the robotic system 100 may calculate the confidence value based on the total length of the incomplete edge 712, the shape of the incomplete edge 712, and / or the difference (e.g., in depth, color, brightness, etc.) between the incomplete edge 712 and the portions surrounding the incomplete edge 712.
[0060] As described in more detail below, the robotic system 100 may derive the verified MVR 720 based on shrinking the initial MVR 710 according to or up to the incomplete edges 712. In other words, the robotic system 100 may identify the reduced candidate MVR as an area within the initial MVR 710 bounded by one or more of the incomplete edges 712 instead of the opposing parallel edges 724 and / or 728. The robotic system 100 may reduce the initial MVR 710 by following the opposing parallel edges (e.g., opposing parallel edges 724, 728, which may be 2D and / or 3D edges, such as the estimated edge 424 in FIG. 4B ) towards the corresponding exposed edges (e.g., corresponding exposed edges 722, 726, respectively) until the incomplete edge 712 is reached. In other words, the robotic system 100 may identify the second corner 714b associated with the first MVR 710 and then move away from the second corner 714 along the associated edge (e.g., the opposing parallel edges 724 and / or 728) while searching for other edges (e.g., intersecting and / or parallel edges, such as the incomplete edge 712). The robotic system 100 may test the orientation of the identified edge, as described above. For example, the robotic system 100 may verify the identified edge if the angle between the traversed edge and the identified edge is within a threshold angle range (e.g., 80 degrees to 100 degrees, and / or other ranges corresponding to right angles). The robotic system 100 may also verify that the identified edge is parallel to the corresponding exposed edge, such as when a set of distances between multiple sets of corresponding points along the identified, untraversed exposed edge are within a threshold range from each other. This may be done iteratively or incrementally. If the MVR cannot be reduced any further, the robotic system 100 may conclude that the resulting area is the verified MVR region 720.
[0061] The robotic system 100 can verify the reduced candidate MVR based on comparing the reduced dimension to a threshold as described above. For example, the robotic system 100 can derive the reduced candidate MVR as a verified MVR 720 if the reduced area defined by the incomplete edge 712 meets a minimum / maximum threshold. The robotic system 100 can also verify the reduced candidate MVR if the incomplete edge 712 corresponds to a confidence value that exceeds a predefined threshold. Additionally, the robotic system 100 can extend the incomplete edge 712 a threshold distance in one or more directions. For example, the robotic system 100 can verify the reduced candidate MVR if the extended incomplete edge intersects with another edge to form an angle that meets a threshold angle range.
[0062] As an example of expanding the initial MVR 710, FIG. 8 shows an example of a merged MVR region according to an embodiment of the present technology. In some cases, multiple adjacent surfaces may have essentially the same height and may have a side / horizontal separation between the surfaces. As an illustrative example, the object represented in the point cloud may be the top of a box with two separated rectangular flaps / segments connected by a piece of tape. Due to one or more of the limitations noted above, the robotic system 100 may derive the initial MVR 710 to include one flap and extend to a side corresponding to the tape or separation between the flaps / segments. For such an example, the robotic system 100 may derive an additional region and merge / test that region with the initial MVR 710 upon derivation of the verified MVR 820.
[0063] As shown in FIG. 8, additional valid MVR regions 812a-812b may be determined based on the initial MVR 710. The additional valid MVR regions 812a-812b may include regions that correlate to the initial MVR 710. For example, the robotic system 100 can determine the additional valid MVR regions 812a-812b based on analyzing portions of 2D and / or 3D images. The robotic system 100 can determine the additional valid MVR regions 812a-812b as locations that have matching visual characteristics (e.g., color, brightness, and / or image patterns) and / or matching depth values that are within a threshold continuous range relative to the initial MVR 710. The robotic system 100 can derive candidate areas as portions of the images determined to be associated with the initial MVR 710.
[0064] As an illustrative example, the robotic system 100 can process the MVRs (e.g., the initial MVR and the expanded MVR) based on following a first exposed edge 722 and a second exposed edge 726 (e.g., edges depicted in the 3D image data) away from the exposed outer corner 714a. The robotic system 100 can identify an initial set of opposing edges including a first initial opposing edge 822 and a second initial opposing edge 826. The robotic system 100 can verify the initial set of opposing edges when the first exposed edge 722 is parallel to the first initial opposing edge 822 and / or when the second exposed edge 726 is parallel to the second opposing edge 826. The robotic system 100 can derive the initial MVR 710 using the verified opposing edges.
[0065] The robotic system 100 can further determine additional plausible MVR regions 812a-b based on tracing the first exposed edge 722 and the second exposed edge 726 beyond the initial set of opposing edges (e.g., away from the exposed outer corner 714a). The robotic system 100 can identify one or more additional opposing edges (e.g., the first exposed edge 832 and / or the second exposed edge 836) that intersect with or are within a threshold separation distance from the traced edges (e.g., the first exposed edge 722 and / or the second exposed edge 726). The robot system 100 can verify additional opposing sides in a similar manner as described above, such as when the first side 832 is parallel to the first exposed side 722 and / or the first initial opposing side 822, and / or when the second side 836 is parallel to the second exposed side 726 and / or the second initial opposing side 826.
[0066] Once one or more additional opposing sides have been verified, the robotic system 100 can identify additional valid MVR regions. For example, the robotic system 100 can identify a first additional valid MVR region 812a as the area between the first initial opposing side 822 and a first of the additional opposing sides (e.g., first side 832). The robotic system 100 can also identify a second additional valid MVR region 812b as the area between the second initial opposing side 826 and a second of the additional opposing sides (e.g., second side 836).
[0067] The robotic system 100 can determine the additional valid MVR regions 812a-812b based on verifying / testing the candidate areas (e.g., the combination of the initial MVR 710 with the first additional valid MVR region 812a and / or the second additional valid MVR region 812b). For example, the robotic system 100 can verify that the separation distance between the candidate area (e.g., the portion of the image determined to be associated with the initial MVR 710) and the initial MVR 710 is less than a predetermined threshold. The robotic system 100 can further test the candidate area by comparing one or more dimensions of the candidate area to the minimum / maximum dimension thresholds described above. The robotic system 100 can determine the candidate area as an additional valid MVR region 812a-812b if the candidate area is less than a minimum threshold (e.g., a dimension of the minimum candidate size). In an embodiment, the robotic system 100 can calculate a confidence level using the size comparison, separation distance, and / or relatedness / similarity between the candidate area and the initial MVR 710. The confidence level may represent the likelihood that the candidate area corresponds to the same object as the portion corresponding to the initial MVR 710. The robotic system 100 can compare the confidence level to a predefined threshold to determine whether the candidate area should be classified as an additional valid MVR region 812a-812b or as a new instance of the initial MVR 710 (e.g., corresponding to a different object).
[0068] Generate a Validated MVR The robotic system 100 can derive a verified MVR 820 based on combining the initial MVR 710 with the additional valid MVRs 812a-812b. Thus, the robotic system 100 can derive candidate MVRs by expanding the initial MVR 710 to include other nearby regions. Thus, the robotic system 100 can increase the likelihood of accurately estimating the complete surface of the unregistered object via the verified MVR 820.
[0069] In one embodiment, the robotic system 100 can derive both a verified MVR 820 and a verified MVR 720 (e.g., a result of shrinking the initial MVR 710). According to one or more predefined processes / equations, the robotic system 100 can calculate a confidence value for each of the verified MVRs using one or more of the processing parameters listed above. The robotic system 100 can select the verified MVR with the larger confidence value as the final MVR.
[0070] Alternatively, the robotic system 100 can derive the initial MVR 710 as the final MVR if the testing of smaller and / or larger candidate areas is unsuccessful. For example, if the merged MVR is larger than the maximum candidate size, the merged MVR can be rejected and the verified MVR 820 can include the initial MVR 710 without any additional valid MVRs. Also, if the reduced MVR of FIG. 7 is smaller than the minimum candidate size, the verified MVR 710 can include the initial MVR 710. In one embodiment, the robotic system 100 can derive the increased MVR by first enlarging the initial MVR 710. The robotic system 100 can then iteratively reduce the increased MVR to derive the final MVR. The robotic system 100 can use the final verified MVR to register and / or manipulate the unregistered object.
[0071] Minimum Feasible Region (MVR) Auto-Register Pipeline Generation FIG. 9 shows a flow diagram of an object recognition process 900 for recognizing an object, according to an embodiment of the present technology. The process may be based on information available and / or received by the robotic system 100 of FIG. 1. For example, the robotic system 100 may receive / access (e.g., via the first image sensor 212 of FIG. 2) 2D and / or 3D image data representing the object stack 210 of FIG. 2 and / or the objects therein (e.g., registered objects and / or unrecognized objects). The robotic system 100 may access predefined and stored data, such as the minimum possible dimensions of the package 112 of FIG. 1 (also referred to as the minimum candidate size), the maximum possible dimensions of the package 112 (also referred to as the maximum candidate size), or a combination thereof. The robotic system 100 may further access and utilize master data (also referred to as an object set) that includes known or predefined data / descriptions regarding receivable or predicted objects. For example, the object set may include descriptions of characteristics associated with various objects, such as object dimensions, surface image / appearance, patterns, colors, etc. When making the comparison, the dimensions of the individual objects (i.e., packages 112, etc.) in the target stack 210 may or may not be known by the robotic system 100.
[0072] The object recognition process 900 may include a descriptor-based detection process (block 902). In an embodiment, the descriptor-based detection process may include a visual or computer vision analysis of the 2D image data. Based on the visual analysis, a detection estimate may be generated based on the object set (block 904). The detection estimate may include a hypothesis that a portion of the image data (e.g., the 2D image data) matches or corresponds to an object represented in the object set. As an example, the robotic system 100 can generate the detection estimate based on matching image portions / features, descriptors, and / or other appearance-related data between the object set and the 2D image data.
[0073] The descriptor-based detection process may include validating the hypothesis (block 906). In other words, the robotic system 100 may validate the detection estimate based on characteristics of registered objects in the object set and / or other sensor data (e.g., 3D image data, such as point cloud data). The robotic system 100 may validate the hypothesis by inspecting / matching other features, local feature descriptors, and / or 3D image data from a 3D depth sensor. For example, the robotic system 100 may generate a hypothesis based on matching a set of significant visual features (e.g., logos, names, largest / brightest features, etc.) with the object set and may validate the hypothesis based on comparing other remaining visual features with the object set. The robotic system 100 may also validate the hypothesis based on identifying the location of the 2D image data corresponding to the surface (i.e., its boundary / edges) corresponding to the hypothesis. For example, the robotic system 100 may calculate the characteristics (e.g., edge dimensions and positions) of the relevant 2D edges corresponding to the identified portions of the 2D image data. The robotic system 100 can verify the hypothesis based on comparing the 2D edge positions / lengths of the hypothesized portion of the 2D image data with the edge positions / lengths represented in the 3D image data (e.g., point cloud). The robotic system 100 can verify the hypothesis if the characteristics (e.g., length) of the 3D edges corresponding to the hypothesized area match the characteristics of the edges of the hypothesized object represented in the object set. In other words, the robotic system 100 can verify that the hypothesized portion of the 2D image data matches the object represented in the object set if additional aspects such as the corresponding 3D edge positions / lengths of the hypothesized portion match additional aspects of the object represented in the object set.
[0074] The object registration process 900 may include point cloud detection (block 908). In an embodiment, the point cloud detection process may include 3D analysis of the 3D image data from the 3D depth sensor. The point cloud detection may include partitioning the point cloud data to identify edges and / or individual layer features (block 910). For example, the robotic system 100 may partition the point cloud data according to depth measurements. As described above, the robotic system 100 may identify groups of adjacent locations having depth measurements within a threshold range from each other. Thus, the robotic system 100 may identify groups that correspond to layers and / or surfaces of the target stack 210. In an embodiment, the robotic system 100 may identify edges / boundaries, surface orientations, and / or discontinuities from the 3D image data. The robotic system 100 may use the identified edges / boundaries, surface orientations, and / or discontinuities to identify layers that are separated by edges, have the same surface orientation, and / or are on opposite sides of the discontinuity. Additionally or alternatively, the robotic system 100 can use edges identified in the 2D image data to identify edges and / or authenticate 3D edges.
[0075] A detection estimate may be generated based on the measured dimensions (block 912). The detection estimate may include a hypothesis that a portion of the 3D image data corresponds to or belongs to an object represented in the object set. For example, the robotic system 100 can generate a detection estimate based on the measured dimensions of each individual layer. The robotic system 100 can assume that an identified edge / boundary of a layer / surface corresponds to an edge of a perimeter of one or more objects represented in the object set. For example, the robotic system 100 measures or calculates the length of the edge as described above. The robotic system 100 can generate a detection estimate that includes portions of the image and / or corresponding regions of the target stack 210 that correspond or match known / registered objects in the object set.
[0076] The robotic system 100 may verify the detection hypothesis (block 914). The robotic system 100 may verify the hypothesis by comparing other characteristics of the hypothesized area with other characteristics of the registered objects of the object set. For example, if the measured dimensions of the edges forming the layer match the corresponding dimensions of one of the registered objects, the robotic system 100 may verify that the object bounded by the compared edges is likely to be the matching registered object. Alternatively, or additionally, the robotic system 100 may verify the hypothesis based on analyzing a portion of the 2D image data that corresponds to the hypothesized portion of the 3D image data. For example, the robotic system 100 may determine the coordinates / locations of the identified 3D edge points used for the length comparison. The robotic system 100 may identify the corresponding location / portion in the 2D image data of the object stack 210. The robotic system 100 may verify the hypothesis by comparing the identified portion of the 2D image data with the corresponding representation of the surface (which may include a surface image) stored in the object set. If the 2D characteristics (e.g., appearance, such as brightness, color pattern, etc.) of the identified portion of the 2D image match the representation of the surface of the hypothesized registered object, the robotic system 100 can verify the hypothesis and detect that the corresponding object is present at the location of the object stack 210. Thus, the robotic system 100 can identify the known object (e.g., recognized object 312 of FIG. 3B) in the object stack 210.
[0077] The object recognition process 900 may begin with a non-empty object set. In an embodiment, the object recognition process 900 may begin with either descriptor-based detection and / or point cloud-based detection, as described above with respect to Figure 9. Alternatively, or in addition, the object recognition process 900 may perform descriptor-based detection or point cloud detection without other detection methods or merging different detection methods.
[0078] 10 is a block diagram of a method 1000 for operating the robotic system 100 of FIG. 1 according to an embodiment of the present technology. The robotic system 100 can perform the method 1000 to process 2D / 3D image data generated by and / or received from the first image sensor 212 of FIG. 2. For example, the robotic system 100 can perform the method 1000 by executing processor instructions stored in a memory device with one or more processors. Thus, the one or more processors can perform the process(es) described herein. The one or more processors can receive data, analyze data, generate results / commands / settings, and / or communicate commands / settings to operate one or more robotic units.
[0079] The method 1000 may include recognizing a known / registered object (e.g., recognized object 312 of FIG. 3B) in the object stack 210 of FIG. 2. According to the method 1000, the robotic system 100 may further process and register an unrecognized object 314 of FIG. 3B that corresponds to an unresolved region of the image data. Thus, the method 1000 may include the object recognition process 900 of FIG. 9.
[0080] The robotic system 100 can identify (block 1002) that one or more unresolved regions remain in the image data after the object recognition process 900. The robotic system 100 can identify an unresolved region as any portion of the image data of the start location 112 of FIG. 1 (e.g., within the horizontal boundaries of a pallet, cart, container, etc.) that does not return a match (or correspond to) with an object represented in the object set (e.g., enrollment data in the master data). The robotic system 100 can determine that the unresolved region(s) correspond to one or more unrecognized objects 314 of FIG. 3B.
[0081] The robotic system 100 can test the unresolved region(s) (block 1004). The robotic system 100 can test the unresolved region(s) according to its shape and / or its dimensions. Thus, the robotic system 100 can identify edges of the image data (e.g., 2D visual images and / or 3D point clouds) (block 1020). For example, the robotic system 100 can identify layers / surfaces of the unresolved region(s) of the image data (e.g., 2D images and / or 3D point clouds) based on identifying adjacent locations having depth measurements that are within a threshold range from each other. The robotic system 100 can identify exposed edges as boundaries of identified layers where depth measurements fall outside a threshold range across horizontally adjacent locations. The robotic system 100 can also identify edges based on analyzing the 2D image according to changes in brightness, color, etc. In one embodiment, the robotic system 100 can identify edges using a Sobel filter.
[0082] The robotic system 100 can test the unsolved region(s) by comparing the identified edges of the unsolved region(s) to a predefined shape template and / or length. For example, the robotic system 100 can compare the shape of the unsolved region (i.e., defined by a set of connected edges) to a shape template (e.g., a box and / or a cylindrical shape) representing a predicted object. The robotic system 100 can adjust the dimensions of the shape template to test the shape of the unsolved region. Additionally or alternatively, the robotic system 100 can calculate the length of the identified edges (e.g., edges shown in the 2D and / or 3D image data) and compare the length to a predefined minimum / maximum threshold.
[0083] If the shape and / or length meet the predefined thresholds / conditions, the robotic system 100 may perform object detection using MVR analysis as described above and generate an MVR (block 1006). The robotic system 100 may use the identified edges / corners to derive an initial MVR (block 1022). For example, the robotic system 100 may select one of the open corners and follow the corresponding open edge to find other eligible 2D / 3D edges (e.g., the estimated edge 424 in FIG. 4B and / or the opposing parallel edges 724, 728 in FIG. 7). The robotic system 100 may derive an initial MVR if the open edge and the other eligible edges meet predefined conditions / thresholds (e.g., minimum / maximum size requirements).
[0084] In one embodiment, the robotic system 100 can further process the initial MVR, such as by expanding the initial MVR (block 1024) and / or by shrinking the initial MVR (block 1026). The robotic system 100 can expand the initial MVR based on determining additional valid MVR regions 812a-812b of FIG. 8 as described above. The robotic system 100 can shrink the MVR (e.g., the initial MVR and / or the expanded MVR) based on finding edges starting from a different corner (e.g., a corner diagonally opposite the originally selected corner). For example, the robotic system 100 can consider other edges (e.g., the incomplete edge 712 of FIG. 7 and / or other 2D / 3D edges) to shrink the MVR.
[0085] The robotic system 100 may validate the adjusted and / or initial MVR (block 1028). The robotic system 100 may derive the verified MVR 820 of FIG. 8 as a combination of the initial MVR and the valid MVR region if the valid MVR region and / or combination meets the predefined conditions / thresholds (e.g., corresponding minimum / maximum size requirements). The robotic system 100 may derive the verified MVR 720 of FIG. 7 if the area bounded by the considered edge(s) meets the predefined conditions / thresholds (e.g., corresponding minimum / maximum size requirements). If the considered edges do not meet the predefined conditions / thresholds, the robotic system 100 may derive the verified MVR based on the last valid result. For example, the robotic system 100 may set the initial MVR as the verified MVR if the upscaled and downscaled results do not meet the test conditions. The robotic system 100 may also generate the verified MVR 820 if the subsequent downscales do not meet the test conditions. In one embodiment, the robotic system 100 can first shrink and then expand the MVR, so that the robotic system 100 can generate a verified MVR 720 if the subsequent expansion does not meet the test conditions.
[0086] Each verified MVR may represent an estimate of a surface (e.g., a top surface) of one of the unrecognized objects 314 in the unresolved region. In other words, each verified MVR may represent an estimated detection representing one object in the unresolved region.
[0087] The robotic system 100 can further analyze the MVR to determine whether the MVR is certain or uncertain (block 1008). For example, the robotic system 100 can determine the number of open corners (e.g., 3D corners) included in and / or utilized in deriving the MVR and classify the MVR accordingly. The robotic system 100 can classify / determine the classification of the MVR based on comparing the number of associated open corners to a predefined threshold. As an illustrative example, the robotic system 100 can determine an initial and / or verified MVR that includes or is derived with two or fewer open 3D corners as a certain MVR. Additionally, the robotic system 100 can determine an initial and / or verified MVR that includes or is derived with three or more open 3D corners as a certain MVR.
[0088] In some embodiments, the robotic system 100 can determine the authenticity of the initial MVR (e.g., before blocks 1024-1028). If the initial MVR is a confirmed MVR, the robotic system 100 can bypass the above operations of blocks 1024-1028. If the initial MVR is an uncertain MVR, the robotic system 100 can perform the above processes of blocks 1024-1028. In other embodiments, the robotic system 100 can determine the authenticity of the verified MVR after scaling, reducing, and / or verifying the processed MVR.
[0089] The robotic system 100 may update the object set if the MVR is certain (block 1030). In one embodiment, the robotic system may update the object set in the MVR according to the classification of the MVR. For example, using the certain MVR, the robotic system 100 may detect the objects, such as by determining / concluding that each certain MVR corresponds to one detected object (e.g., one instance of the unrecognized object 314). Thus, the robotic system 100 may detect and / or directly register the certain MVR (e.g., without further tuning, testing, and / or verifying the MVR), such as by storing the certain MVR (e.g., visual image and / or corresponding dimensions) in the object set. With the updated object set (i.e., the object set updated with the certain MVR), the robotic system 100 may perform the object recognition process 900 again, as represented by the feedback loop.
[0090] If the MVR is uncertain, the robotic system 100 may perform the task according to the verified MVR (block 1010). The robotic system 100 can use the verified MVR to derive the gripping position 420 of FIG. 4B. For example, the robotic system 100 can derive the gripping position 420 that is aligned or abutting with an exposed corner of the verified MVR. Also, the robotic system 100 can derive the gripping position 420 that overlaps with a middle portion of the verified MVR. Based on the gripping position 420, the robotic system 100 can derive a command, setting, and / or motion plan to operate the robot arm 202 of FIG. 2 and / or the end effector 204 of FIG. 2 to grasp and move the object. For the task, the robotic system 100 can move the object to the task position 116 of FIG. 1. Additionally, during locomotion, the robotic system 100 may further position the object at one or more specific locations, such as presenting the object to one or more sensors and / or positioning the object relative to one or more lines / planes.
[0091] The robotic system 100 may obtain additional data regarding the object during the movement of the object (block 1032). For example, the robotic system 100 may grasp the object and perform an initial movement (e.g., lift it a predetermined distance and / or move it horizontally). After the initial movement, the robotic system 100 may generate one or more updated 2D / 3D image data (e.g., via the first image sensor 212). Based on the increasing separation between the grasped object and the surrounding objects, the robotic system 100 may analyze the updated 2D / 3D image data to re-derive the edges, MVR, and / or object dimensions. The robotic system 100 may also calculate the height of the object according to one or more intersection sensors (not shown). The robotic system 100 may determine the height of the end effector 204 when the bottom of the object being moved enters / exits a laterally oriented detection line / plane associated with the intersection sensor (e.g., a sensor configured to detect an interruption in the optical signal). The robotic system 100 can calculate the height based on the height of the end effector and the known height of the detection line / plane, and can acquire images of the surroundings, identifier values / positions, weights, and / or other physical descriptions of the object being moved.
[0092] The robotic system 100 may register the moved unrecognized object (block 1034). The robotic system 100 may register the object based on storing the additional data and / or the verified MVR in the object set as described above. The robotic system 100 may further recognize other objects using the updated object set, as indicated by the feedback loop to the object recognition process 900.
[0093] When performing the process corresponding to block 1010, the robotic system 100 may transmit the uncertain MVR along with the detected objects (e.g., resulting from the object recognition process 900) and / or the newly registered detected objects (e.g., the certain MVR) to a planning module (not shown). The planning module may derive a motion plan to manipulate / move the recognized objects, the detected objects (e.g., the unrecognized objects corresponding to the certain MVR), and / or other unrecognized objects. The robotic system 100 may further obtain additional information (e.g., object height, identifier data, profile image, etc.) during the movement of the unrecognized objects, and may use the obtained additional information to update the registration data of the object set. The robotic system 100 may use the obtained additional information to register the unrecognized objects and / or derive a final verified MVR.
[0094] In one embodiment, the robotic system 100 can process the unresolved region and generate all initial / verified MVRs for the region. According to the iterative implementation of the method 1000, the robotic system 100 can compare the unresolved region with the object set to initially determine and register a certain MVR among the generated MVRs. Using the updated object set (e.g., the certain MVR), the robotic system 100 can perform the object recognition process 900 again to detect objects in the unresolved region that correspond to the certain MVR. The robotic system 100 can update the unresolved region by identifying areas in the unresolved region that match the certain MVR used to update the object set, thereby detecting the corresponding objects. The robotic system 100 can finish updating the unresolved region by removing portions from the unresolved region that correspond / match the certain MVR. The robotic system 100 can repeat this iterative process until the resulting unresolved region does not contain a certain MVR. With respect to the uncertain MVR, the robotic system 100 may derive / use the corresponding verified MVR from the unresolved region to manipulate the unrecognized object in that region. The robotic system 100 may further acquire additional information while manipulating the unrecognized object as described above. The robotic system 100 may update the object set with the verified MVR and / or the additional information and may continue the object recognition process 900 to recognize other objects that match the moved object and repeat the iterative process. The robotic system 100 may update the unresolved region by identifying a region in the unresolved region that matches the verified MVR of the moved object, thereby detecting the corresponding object. The robotic system 100 may finish updating the unresolved region by removing the portion of the unresolved region that matches the verified MVR of the moved object. The robotic system 100 may process the updated unresolved region to derive and verify the next MVR corresponding to the next type / instance of the unrecognized object.
[0095] Additionally or alternatively, the robotic system 100 may determine the recognized objects, derive all MVRs, and then plan the movement of the objects. In other embodiments, the robotic system 100 may derive or update the MVR based on first moving the recognized objects and / or objects corresponding to the certain MVRs. Thus, the robotic system 100 may increase the likelihood of exposing additional 3D edges / corners of the unrecognized object. Increasing the likelihood of exposing additional 3D edges / corners can increase the likelihood of identifying a certain MVR, which can improve the accuracy of detecting the object and the derived MVR. Furthermore, the increased accuracy can improve efficiency by reducing / eliminating processing steps and / or reducing object loss and / or collisions due to inaccurate object detection.
[0096] The robotic system 100 can autonomously populate the object set using the embodiments described herein. In other words, the robotic system 100 can start with an empty object set that does not include enrollment data. Under such conditions, the robotic system 100 can perform the method 1000 beginning at block 1002 and identify the entirety of the received image data as an unresolved region. Since no enrollment data exists, the robotic system 100 can enroll the object with little or no human operator input as the object is processed (e.g., moved from the start location 114 to the task location 116). Thus, the robotic system 100 can begin deriving MVRs without performing the object recognition process 900 and can store a subset of the derived MVRs (e.g., the hard MVRs) in the object set. The robotic system 100 can iteratively perform the method 1000 as described above to enroll the object. For example, the robotic system 100 can detect a first set of objects represented in the unresolved region based on matching corresponding portions in the unresolved region with a subset of derived MVRs stored in the object set. The robotic system 100 can update the unresolved region according to the detection of the first set of objects by removing the matching portions from the unresolved region. The robotic system 100 can process the updated unresolved region to register other objects in the updated unresolved region and derive a new / next set of MVRs (e.g., another set of confirmed MVRs and / or other verified MVRs).
[0097] Processing system example 11 is a block diagram illustrating an example of a processing system 1100 that may perform at least some of the operations described herein. As shown in FIG. 11, the processing system 1100 may include one or more central processing units ("processors") 1102, a main memory 1106, a non-volatile memory 1110, a network adapter 1112 (e.g., a network interface), a video display 1118, input / output devices 1120, a controller 1122 (e.g., a keyboard and pointing device), a drive unit 1124 including a storage medium 1126, and a signal generator 1130, which are communicatively connected to a bus 1116. The bus 1116 is illustrated as an abstraction that represents any one or more separate physical buses, point-to-point connections, or both, connected by appropriate bridges, adapters, or controllers. The bus 1116 may thus include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus or a PCI-Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or the Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus, also known as "Firewire."
[0098] In various embodiments, the processing system 1100 operates as part of a user device, although the processing system 1100 may be connected (e.g., wired or wirelessly) to the user device. In a networked deployment, the processing system 1100 may operate as a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.
[0099] The processing system 1100 may be a server computer, a client computer, a personal computer, a tablet, a laptop computer, a personal digital assistant (PDA), a cellular phone, a processor, a web appliance, a network router, switch or bridge, a console, a handheld console, a game console, a music player, a networked ("smart") television, a television connection device, or any portable device or machine capable of executing (sequentially or otherwise) a set of instructions that specify actions that the processing system 1100 is to take.
[0100] Although the main memory 1106, the non-volatile memory 1110, and the storage medium 1126 (also referred to as a "machine-readable medium") are shown as one medium, the terms "machine-readable medium" and "storage medium" should be taken to include one medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions 1128. The terms "machine-readable medium" and "storage medium" should be taken to include any medium that can store, encode, or convey a set of instructions for execution by a computer system, causing a computer system to perform any one or more of the methodologies of the embodiments of the present disclosure.
[0101] Generally, the routines executed to implement the disclosed embodiments may be implemented as parts of an operating system or a specific application, component, program, object, module, or sequence of instructions referred to as a “computer program.” A computer program typically includes one or more instructions (e.g., instructions 1104, 1108, 1128) stored at various times in various memories and storage devices of the computer that, when read and executed by one or more processing units or processors 1102, cause the processing system 1100 to perform operations that implement elements associated with various aspects of the disclosure.
[0102] Furthermore, while embodiments have been described in the context of fully functional computers and computer systems, those skilled in the art will appreciate that various embodiments may be distributed as program products in a variety of forms, and that the disclosure applies equally regardless of the particular type of machine or computer-readable medium used to actually accomplish the distribution. For example, the techniques described herein may be implemented using virtual machines or cloud computing services.
[0103] Further examples of machine-readable storage media, machine-readable media, or computer-readable (storage) media include, but are not limited to, recordable type media such as volatile and non-volatile memory devices 1110, floppy and other removable disks, hard disk drives, optical disks (e.g., compact disk read only memories (CD ROMS), digital versatile disks (DVD)), and transmission media such as digital and analog communication links.
[0104] The network adapter 1112 enables the processing system 1100 to broker data on the network 1114 to entities external to the processing system 1100 through any known and / or convenient communications protocol supported by the processing system 1100 and the external entities. The network adapter 1112 may include one or more of a network adapter card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multi-layer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, and / or a repeater.
[0105] The network adapter 1112 may include a firewall, which in some embodiments governs and / or manages permissions to access / proxy data on a computer network and tracks changing trust levels between different machines and / or applications. The firewall may be any number of modules having any combination of hardware and / or software components that can enforce a set of predefined access rights between specific sets of machines and applications, machines and / or applications, for example, to regulate traffic flow and resource sharing between these various entities. The firewall may further manage and / or have access to access control lists that detail permissions, including, for example, rights by individuals, machines and / or applications to access and act on objects, and the circumstances under which the authorization rights are valid.
[0106] In one embodiment, a method of operating a robotic system includes receiving two-dimensional (2D) and three-dimensional (3D) image data representative of a target stack including one or more unrecognized objects; identifying unresolved regions of the 2D and / or 3D image data based on a comparison to an object set representative of one or more physical aspects of registered objects; identifying one or more layers within the unresolved regions of the 2D and / or 3D image data, each layer including a set of laterally adjacent points in the 3D image data having depth measurements within a threshold range from each other; and identifying boundaries of one or more layers where depth measurements across horizontally adjacent locations deviate outside the threshold range. the unrecognized object based on the identified exposed edges; identifying one or more exposed corners, each corresponding to a junction between a set of exposed edges; deriving a minimum feasible region (MVR) representing an estimate of a surface of the unregistered object represented in the unrecognized region based on the identified exposed edges and exposed corners; determining a number of exposed corners associated with the derived MVR; detecting the unrecognized object according to the MVR and the number of exposed corners; and registering the unrecognized object based on the object set to include the MVR and / or a processed result of the MVR as a representation of the unrecognized object. The method may further include iteratively updating and analyzing the unresolved region according to the updated object set based on comparing the unresolved region with the updated object set, identifying portions in the unresolved region that match the MVR and / or MVR processing results used to update the object set, detecting objects according to the portions that match the MVR and / or MVR processing results, and updating the unresolved region by removing the portions that match the MVR and / or MVR processing results. The object set may be an empty set for a first iteration, and the unresolved region may include 2D image data, 3D image data, a portion thereof, or a combination thereof for the first iteration.For the first iteration, registering the unrecognized objects may include autonomously populating the object set while performing a task involving the unrecognized objects in the target stack.
[0107] In one embodiment, detecting unrecognized objects may include directly detecting unrecognized objects if the number of exposed corners exceeds a predetermined threshold, and registering recognized objects may include directly storing the MVR in an object set if the number of exposed corners exceeds a predetermined threshold. In one embodiment, the predetermined threshold may be 2.
[0108] In one embodiment, the robotic system may include at least one processor and at least one memory device coupled to the at least one processor having stored thereon instructions executable by the processor, the instructions including: receiving one or more image data representing one or more unrecognized objects at a start location; deriving a minimum feasible region (MVR) based on identifying one or more exposed corners in the one or more image data representing junctions between exposed edges that are not horizontally adjacent to other surfaces or objects, respectively; updating an object set representing one or more physical aspects of a previously registered object with an MVR representing one or more physical attributes of one of the unrecognized objects at the start location and / or a result of processing the MVR based on a number of exposed corners associated with the MVR; and recognizing another object at the start location based on the updated object set. The at least one memory device may include instructions for determining the MVR as a certain MVR based on comparing the number of exposed corners to a predefined threshold, such as when the MVR is associated with three or more exposed corners, and / or instructions for updating the object set directly with the MVR and / or results of processing the MVR without further updating or adjusting the MVR based on determining the MVR as a certain MVR. The at least one memory device may further include instructions for identifying unresolved regions of the image data that do not return a match with an object represented in the object set, instructions for deriving a certain MVR within the unresolved region based on the unresolved region, and instructions for removing portions of the unresolved region that match the certain MVR for further object recognition and / or manipulation. In one or more embodiments, the at least one memory device may further include instructions for performing an operation to move an unrecognized object corresponding to the certain MVR away from the start position, instructions for deriving a next MVR based on the updated unresolved area after performing the operation to move the unrecognized object, the next MVR representing a further unrecognized object at the start position and represented in the updated unresolved area, and instructions for updating the object set with a second MVR and / or a result of processing the second MVR.
[0109] In one embodiment, the robotic system may include at least one memory device having instructions for determining an MVR as an uncertain MVR if the MVR is associated with two or less exposed corners, and instructions for further analyzing the uncertain MVR and / or updating the object set with the MVR and / or a result of processing the MVR based on manipulating one or more of the unrecognized objects of the start location. The at least one memory device may include instructions for detecting one or more recognized objects based on comparing the image data to the object set, instructions for identifying unresolved regions of the image data that do not return a match with the object set, and instructions for deriving an MVR in the unresolved regions that represents an estimate of a surface of one of the unrecognized objects of the start location based on identifying one or more exposed corners of the unresolved regions of the image data. The at least one memory device may further include instructions for performing an operation to move one or more recognized objects away from the start location, instructions for updating the MVR after performing an operation to move one or more recognized objects, and instructions for updating the object set with the updated MVR and / or a result of processing the MVR.
[0110] In one embodiment, a tangible, non-transitory computer-readable medium having processor instructions stored thereon, when executed by a processor, causes the processor to perform a method including: iteratively analyzing one or more received image data representing registered and / or unrecognized objects at a start location by deriving one or more minimum feasible regions (MVRs) based on identifying one or more exposed corners of the received image data that represent junctions between exposed edges that are not horizontally adjacent to other surfaces or objects, respectively; determining a classification of each MVR based on the number of exposed corners associated with the corresponding MVR; updating an object set representing one or more physical aspects of a previously registered object with an MVR and / or a result of processing the MVR, respectively, that represents one or more physical attributes of one of the unrecognized objects at the start location in accordance with the corresponding classification; and recognizing another object at the start location using the updated object set. In an embodiment, the one or more received image data include two-dimensional (2D) image data and three-dimensional (3D) image data both representing a stack of objects from one imaging location, and the method may further include generating a detection estimate based on comparing the 2D image data to the object set and verifying the detection estimate based on the 3D image data. Generating the detection estimate may further include generating a detection estimate based on matching a portion of the 2D image data with a surface image of an object represented in the object set, and verifying the detection estimate may further include verifying the detection estimate based on identifying a portion of the 3D image data that corresponds to the detection estimate, calculating a 3D edge length of the identified portion of the 3D image data, and verifying the detection estimate if the 3D edge length matches a edge length of the object represented in the object set.
[0111] In an embodiment, the non-transitory computer-readable medium may include a method, where the one or more received image data include two-dimensional (2D) image data and three-dimensional (3D) image data both representing a stack of objects from one imaging location, and the method may further include generating a detection estimate based on comparing the 3D image data to an object set, and validating the detection estimate based on the 2D image data. Generating the detection estimate may include generating the detection estimate based on identifying a layer in the 3D image data including a set of horizontally adjacent locations having depth measurements within a threshold range of each other, identifying an edge of the layer that represents a location where the depth measurements fall outside the threshold range over the horizontally adjacent locations, calculating an edge length, and matching the edge length with a dimension of an object represented in the object set, and validating the detection estimate may include validating the detection estimate based on identifying a portion of the 2D image data that corresponds to the detection estimate, identifying 2D characteristics of the identified portion of the 2D image data, and validating the detection estimate if the 2D characteristics match a corresponding characteristic of an image of one of the registered objects represented in the object set. In one embodiment, the non-transitory computer-readable medium may include a method further including iteratively constructing an object set from an empty set based on analyzing one or more received image data, where the analysis of the one or more received image data includes identifying the one or more received image data or a portion thereof as an unresolved area if the object set does not include registration data of any objects, deriving one or more MVRs from the unresolved area, storing at least a subset of the one or more MVRs in the object set according to the corresponding classification, detecting a first set of objects represented in the received image data based on matching the portions of the received image data with the one or more MVRs stored in the object set, updating the unresolved area by removing the matched portions, and deriving a new set of MVRs based on the updated unresolved area to register a second set of objects represented in the received image data.
[0112] As noted above, the techniques presented herein may be implemented, for example, in programmable circuitry (e.g., one or more microprocessors) programmed with software and / or firmware, in all-dedicated hardwired (i.e., non-programmable) circuitry, or in a combination of such forms. The dedicated circuitry may be in the form of, for example, one or more application specific integrated circuits (ASICs), programmable logic circuits (PLDs), field programmable gate arrays (FPGAs), etc.
[0113] From the foregoing it will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications may be made without departing from the scope of the invention. Accordingly, the invention is not limited except as by the appended claims.
Claims
1. 1. A method of operating a robotic system, comprising: Receiving two-dimensional (2D) image data and three-dimensional (3D) image data representing an object stack including one or more unrecognized objects; identifying unresolved regions of the 2D image data and / or the 3D image data based on a comparison with a set of objects representative of one or more physical aspects of a registered object; identifying one or more layers within the unresolved region of the 2D image data and / or the 3D image data, each layer comprising a set of laterally adjacent points in the 3D image data having depth measurements within a threshold range of each other; identifying exposed edges representing boundaries of the one or more layers where the depth measurements fall outside the threshold range across adjacent horizontal locations; identifying one or more exposed corners, each of which corresponds to a junction between a set of exposed edges, based on the exposed edges; deriving a minimum feasible region (MVR) representing an estimate of a surface of an unregistered object represented in the unresolved region based on the identified exposed edges and exposed corners; determining the number of exposed corners associated with the derived MVR; and detecting an unrecognized object according to the MVR and the number of exposed corners; registering the unrecognized object based on the MVR and / or updating the object set to include a result of processing the MVR as a representation of the unrecognized object; The method comprising:
2. comparing the unsolved region to the updated object set; identifying portions within the unsolved region that match the MVR and / or the processing results of the MVR used to update the object set; detecting an object according to the MVR and / or the portion that matches the processing result of the MVR; updating the unresolved region by removing the portion that matches the MVR and / or the processing result of the MVR; The method of claim 1 , further comprising iteratively updating and analyzing the unsolved region according to the updated object set based on:
3. the object set is an empty set for the first iteration, the unresolved region includes, for the first iteration, the 2D image data, the 3D image data, a portion thereof, or a combination thereof; and registering the unrecognized object includes autonomously populating the object set while performing a task involving the unrecognized object in the object stack. The method of claim 2.
4. Detecting the unrecognized object includes directly detecting the unrecognized object if the number of exposed corners exceeds a predetermined threshold; registering the recognized object includes storing the MVR directly in the object set if the exposed corner exceeds the predetermined threshold. The method of claim 1.
5. The method of claim 4 , wherein the predetermined threshold is two.
6. At least one processor; at least one memory device coupled to said at least one processor and storing instructions executable by said processor; wherein the instructions include: receiving one or more image data representative of one or more unrecognized objects at a start location; instructions for deriving a minimum feasible region (MVR) based on identifying one or more exposed corners of the one or more image data that represent junctions between exposed edges that are not horizontally adjacent to other surfaces or objects, respectively; instructions for updating the MVR and / or a set of objects representing one or more physical aspects of a previously registered object with the MVR representing one or more physical attributes of one of the unrecognized objects at the start location based on the number of exposed corners associated with the MVR; instructions for recognizing other objects at the starting position based on the updated object set; The robot system.
7. The robotic system of claim 6 , wherein the at least one memory device includes instructions for determining the MVR as a certain MVR based on comparing the number of exposed corners to a predetermined threshold.
8. The robotic system of claim 7 , wherein the at least one memory device includes instructions for determining the MVR as the certain MVR if the MVR is associated with three or more of the exposed corners.
9. 10. The robotic system of claim 8, wherein the at least one memory device includes instructions for directly updating the object set with the MVR and / or a result of processing the MVR without further updating or adjusting the MVR based on the determination of the MVR as the certain MVR.
10. The at least one memory device instructions for identifying unresolved regions of the image data that do not return a match with an object represented in the object set; the certain MVR is within an unknown region, and instructions for deriving the MVR based on the unknown region; instructions to remove portions of the unresolved region that match the certain MVR for further object recognition and / or manipulation; The robotic system of claim 9 .
11. The at least one memory device instructions to perform an action to move the unrecognized object corresponding to the certain MVR away from the start location; instructions for deriving, after performing the action of moving the unrecognized object, a next MVR representing a further unrecognized object at the start position and represented in an updated unresolved area based on the updated unresolved area; instructions for updating the object set with the second MVR and / or results of processing the second MVR; The robotic system of claim 10 .
12. The at least one memory device instructions for determining the MVR as a questionable MVR if the MVR is associated with two or less of the exposed corners; instructions for updating the MVR and / or the set of objects with the results of processing the MVR based on further analyzing the uncertain MVR and / or manipulating one or more of the unrecognized objects at the start location; The robotic system of claim 6 .
13. The at least one memory device detecting one or more recognized objects based on comparing the image data to the set of objects; instructions for identifying unresolved regions of the image data that do not return a match with the object set; instructions for deriving the MVR within the unresolved region based on identifying the one or more exposed corners of the unresolved region of the image data; Including, the MVR represents an estimate of a surface of the one of the unrecognized objects at the start position; The robot system according to claim 6.
14. The at least one memory device instructions for performing an action to move the one or more recognized objects away from the start position; instructions for updating the MVR after performing the action of moving the one or more recognized objects; instructions for updating the set of objects with the updated MVR and / or results of processing the updated MVR; The robotic system of claim 13 .
15. A tangible, non-transitory computer readable medium having stored thereon processor instructions that, when executed by a processor, cause the processor to perform a method including iteratively analyzing one or more received image data representative of a registered and / or unrecognized object at a start position; The analysis deriving one or more minimum feasible regions (MVRs) based on identifying one or more exposed corners of the received image data that represent junctions between exposed edges that are not horizontally adjacent to other surfaces or objects, respectively; determining a classification for each of the MVRs based on the number of exposed corners associated with the corresponding MVR; updating an object set representing one or more physical aspects of a previously registered object with the MVR and / or a processing result of the MVR, each representing one or more physical attributes of one of the unrecognized objects at the start position according to the corresponding classification; updating the object set; recognizing another object at the starting position using the updated object set; and A tangible, non-transitory computer readable medium by
16. the one or more received image data both include two-dimensional (2D) image data and three-dimensional (3D) image data representative of the object stack from one imaging position; The method comprises: generating a detection estimate based on comparing the 2D image data to the object set; verifying the detection estimate based on the 3D image data; and 20. The non-transitory computer readable medium of claim 15, further comprising:
17. generating the detection estimate includes generating the detection estimate based on matching a portion of the 2D image data with surface images of objects represented in the object set; Verifying the detection estimation includes: identifying a portion of the 3D image data that corresponds to the detection estimate; calculating a 3D edge length of the identified portion of the 3D image data; validating the detection estimate if the lengths of the 3D edges match the lengths of the edges of the objects represented in the object set; validating the detection estimate based on 20. The non-transitory computer readable medium of claim 16.
18. the one or more received image data both include two-dimensional (2D) image data and three-dimensional (3D) image data representative of the object stack from one imaging position; The method comprises: generating a detection estimate based on comparing the 3D image data to the set of objects; verifying the detection estimation based on the 2D image data; 20. The non-transitory computer readable medium of claim 15, further comprising:
19. Generating the detection estimate includes: identifying a layer within the 3D image data comprising a set of horizontally adjacent locations having depth measurements within a threshold range of each other; identifying edges of the layer that represent locations where the depth measurements fall outside the threshold range across horizontally adjacent locations; calculating the length of said side; checking said edge lengths against dimensions of objects represented in said object set; generating the detection estimate based on Verifying the detection estimation includes: identifying a portion of the 2D image data corresponding to the detection estimate; identifying 2D characteristics of the identified portion of the 2D image data; validating the detection estimation if the 2D characteristics match corresponding characteristics of an image of the one of the registered objects represented in the object set; validating the detection estimate based on 20. The non-transitory computer readable medium of claim 18.
20. The method further includes iteratively constructing the object set from an empty set based on analyzing the one or more received image data, where analyzing the image data includes: identifying the one or more received image data or portions of the image data as an unresolved region if the object set does not include enrollment data of any object; deriving said one or more MVRs from said uncharacterized region; storing at least a subset of the one or more MVRs in the object set according to the corresponding classification; detecting a first set of objects depicted in the received image data based on portions of the received image data that match the one or more MVRs stored in the object set; updating the unresolved region by removing the matched portion; deriving a new set of MVRs based on the updated unresolved regions for registering a second set of objects represented in the received image data; and 16. The non-transitory computer readable medium of claim 15.
Citation Information
Patent Citations
Image processor and object transfer device
JP1998031742A
Work stacking state recognition device
JP2011247819A
Article recognition device and article recognition facility
JP2015043175A
Robot and method for controlling the same
US20100168913A1
