Systems and methods for object detection

Abstract

To provide technical improvement for a robotic system configured to be used for identifying, detecting, and extracting an object.SOLUTION: A method for refining a detection hypothesis includes: receiving object image information on an object in a scene; receiving a detection hypothesis including a corresponding object recognition template representing a template object; performing processing of identifying a mismatch between the template object and the object image information; performing processing of identifying a set of template locations in the template object corresponding to a set of object locations of the object image information; performing processing of adjusting the set of template locations to converge to the set of object locations; and outputting, to a robotic system, an adjusted detection hypothesis including an adjusted corresponding object recognition template according to the set of template locations after adjustment.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 230,931, filed August 9, 2021, entitled "A ROBOTIC SYSTEM FOR FACILITATING TEMPLATE MATCHING AND DETECTION FOR OBJECT PICKING," which is incorporated by reference in its entirety.

[0002] The present technology is directed generally to robotic systems, and more particularly to systems, processes, and techniques for identifying and detecting objects. More particularly, the present technology can be used to identify and detect objects in a container. [Background technology]

[0003] With ever-increasing capabilities and decreasing costs, many robots (e.g., machines configured to automatically / autonomously perform physical actions) are now widely used in a variety of different fields. Robots can be used to perform a variety of tasks (e.g., manipulating or transporting objects through space), for example, in manufacturing and / or assembly, packing and / or packaging, transporting and / or shipping, etc. In performing a task, a robot can replicate human actions, thereby replacing or reducing human involvement required to perform an otherwise dangerous or repetitive task.

[0004] However, despite advances in technology, robots often lack the sophistication necessary to replicate the human interactions required to perform larger and / or more complex tasks. Thus, a need remains for improved techniques and systems for managing the actions and / or interactions between robots. Summary of the Invention

[0005] In one embodiment, a computing system configured to generate a set of object recognition templates for identifying an object in a scene is provided, the computing system including at least one processing circuit configured to: acquire registration data for the object including an object model representing the object, determine a plurality of views of the object model in three-dimensional space, estimate a plurality of appearances of the object model at each of the plurality of views, generate a plurality of object recognition templates according to the plurality of appearances, each corresponding to a respective one of the plurality of appearances, and communicate the plurality of object recognition templates to a robotic control system as the object recognition template set, each of the plurality of object recognition templates representing a possible pose of the object relative to an optical axis of a camera generating image information of the object in the scene.

[0006] In another embodiment, a method for generating an object recognition template set for identifying an object in a scene is provided. The method includes obtaining enrollment data for an object including an object model representing the object, determining a plurality of views of the object model in three-dimensional space, estimating a plurality of appearances of the object model at each of the plurality of views, generating a plurality of object recognition templates according to the plurality of appearances, each object recognition template corresponding to a respective one of the plurality of appearances, and communicating the plurality of object recognition templates to a robotic control system as the object recognition set. Each of the plurality of object recognition templates represents a pose that the object may have relative to an optical axis of a camera that generates image information for the object in the scene.

[0007] In another embodiment, a non-transitory computer-readable medium operable by at least one processing circuit via a communication interface configured to communicate with a robotic system is provided, the non-transitory computer-readable medium configured with executable instructions for performing a method for generating object recognition templates for identifying an object in a scene, the method including receiving enrollment data for the object including an object model representing the object, performing processing for generating a plurality of views of the object model in three-dimensional space, performing processing for estimating a plurality of appearances of the object model at each of the plurality of views, performing processing for generating a plurality of object recognition templates according to the plurality of appearances, each of the plurality of object recognition templates corresponding to a respective one of the plurality of appearances, and outputting the plurality of object recognition templates as an object recognition template set to the robotic system, each of the plurality of object recognition templates representing a pose that the object may have relative to an optical axis of a camera that generates image information for the object in the scene.

[0008] In another embodiment, a computing system configured to generate an object recognition template for identifying an object in a scene is provided, the computing system including at least one processing circuitry that includes the steps of obtaining object information including a digitally represented object, extracting two-dimensional measurement information from the object information, extracting three-dimensional measurement information from the object information, and generating the object recognition template according to the two-dimensional measurement information and the three-dimensional measurement information.

[0009] In another embodiment, a method for generating an object recognition template for identifying an object in a scene is provided, the method including obtaining object information including a digitally represented object, extracting two-dimensional measurement information from the object information, extracting three-dimensional measurement information from the object information, and generating the object recognition template according to the two-dimensional measurement information and the three-dimensional measurement information.

[0010] In another embodiment, a non-transitory computer-readable medium operable by at least one processing circuit via a communication interface configured to communicate with a robotic system, the non-transitory computer-readable medium having executable instructions for performing a method for generating an object recognition template for identifying an object in a scene, the method including receiving object information including a digitally represented object, performing operations to extract two-dimensional measurement information from the object information, performing operations to extract three-dimensional measurement information from the object information, and outputting the object recognition template to the robotic system according to the two-dimensional measurement information and the three-dimensional measurement information.

[0011] In another embodiment, a computing system is provided. The computing system includes at least one processing circuit in communication with a robot having an arm and an end effector connected to the arm, and in communication with a camera having a field of view. The at least one processing circuit is configured to execute instructions stored on a non-transitory computer-readable medium when one or more objects are or were within the field of view. The executed instructions include acquiring object image information of an object in a scene, acquiring detection hypotheses including corresponding object recognition templates representing a template object, identifying mismatches between the template object and the object image information, identifying a set of template locations in the template object that correspond to a set of object locations in the object image information, adjusting the set of template locations to converge to the set of object locations, and generating adjusted detection hypotheses including adjusted corresponding object recognition templates according to the adjusted set of template locations.

[0012] In another embodiment, a method is provided, the method including: acquiring object image information for an object in a scene, acquiring detection hypotheses including corresponding object recognition templates representing a template object, identifying mismatches between the template object and the object image information, identifying a set of template locations in the template object corresponding to a set of object locations in the object image information, adjusting the set of template locations to converge to the set of object locations, and generating adjusted detection hypotheses including the adjusted corresponding object recognition templates according to the adjusted set of template locations.

[0013] In another embodiment, a non-transitory computer-readable medium operable by at least one processing circuit via a communication interface configured to communicate with a robotic system, the non-transitory computer-readable medium having executable instructions for performing a method for refining a detection hypothesis, the method including: receiving object image information of an object in a scene, receiving a detection hypothesis including a corresponding object recognition template representing a template object, performing operations of identifying a mismatch between the template object and the object image information, identifying a set of template locations in the template object corresponding to a set of object locations in the object image information, adjusting the set of template locations to converge to the set of object locations, and outputting an adjusted detection hypothesis to the robotic system including an adjusted corresponding object recognition template according to the adjusted set of template locations.

[0014] In another embodiment, a computing system is provided that includes at least one processing circuit in communication with a robot having an arm and an end effector connected to the arm and in communication with a camera having a field of view, the at least one processing circuit configured to execute instructions stored on a non-transitory computer readable medium when one or more objects are or were within the field of view, the instructions that are executed include obtaining object image information for an object in a scene, obtaining a set of detection hypotheses each including a corresponding object recognition template representing a template object, and testing each detection hypothesis in the set of detection hypotheses. The verifying is performed by generating a plurality of three-dimensional validation scores including at least one of an occlusion validator score, a point cloud validator score, a hole match validator score, and a normal vector validator score based on a comparison of the three-dimensional information of the object recognition template of the detection hypothesis with the three-dimensional information of the object image information corresponding to the object; generating a plurality of two-dimensional validation scores including at least one of a rendered match validator score and a template matching validator score based on a comparison of the two-dimensional information of the corresponding object recognition template of the detection hypothesis with the three-dimensional information of the object image information; filtering detection hypotheses from the set of detection hypotheses according to the plurality of three-dimensional validation scores and the plurality of two-dimensional validation scores; and detecting an object in the scene according to the unfiltered detection hypotheses remaining in the set of detection hypotheses after verification.

[0015] In another embodiment, a method is provided, the method including: acquiring object image information of an object in a scene; acquiring a set of detection hypotheses, each of which includes a corresponding object recognition template representing a template object; and validating each detection hypothesis in the set of detection hypotheses by: generating a plurality of three-dimensional validation scores, including at least one of an occlusion validator score, a point cloud validator score, a hole match validator score, and a normal vector validator score, based on a comparison of three-dimensional information of the object recognition template of the detection hypothesis with three-dimensional information of the object image information corresponding to the object; generating a plurality of two-dimensional validation scores, including at least one of a rendered match validator score and a template matching validator score, based on a comparison of two-dimensional information of the corresponding object recognition template of the detection hypothesis with three-dimensional information of the object image information; filtering detection hypotheses from the set of detection hypotheses according to the plurality of three-dimensional validation scores and the plurality of two-dimensional validation scores; and detecting an object in the scene according to unfiltered detection hypotheses remaining in the set of detection hypotheses after validation.

[0016] In another embodiment, a non-transitory computer readable medium operable by at least one processing circuit via a communications interface configured to communicate with a robotic system, the non-transitory computer readable medium having executable instructions for performing a method for validating a detection hypothesis is provided. The method includes receiving object image information of an object in the scene; receiving a set of detection hypotheses, each detection hypothesis including a corresponding object recognition template representing a template object; performing a process of generating a plurality of three-dimensional validation scores including at least one of an occlusion validator score, a point cloud validator score, a hole match validator score, and a normal vector validator score based on a comparison of three-dimensional information of the object recognition template of the detection hypothesis with three-dimensional information of the object image information corresponding to the object; performing a process of generating a plurality of two-dimensional validation scores including at least one of a rendered match validator score and a template matching validator score based on a comparison of two-dimensional information of the corresponding object recognition template of the detection hypothesis with three-dimensional information of the object image information; performing a process of filtering detection hypotheses from the set of detection hypotheses according to the plurality of three-dimensional validation scores and the plurality of two-dimensional validation scores; detecting an object in the scene according to unfiltered detection hypotheses remaining in the set of detection hypotheses after verification; and outputting the detected object in the scene to the robotic system. [Brief description of the drawings]

[0017] [Figure 1A] 1 illustrates a system for performing or facilitating object detection, identification, and retrieval, according to embodiments herein.

[0018] [Figure 1B] 1 illustrates one embodiment of a system for performing or facilitating object detection, identification, and retrieval in accordance with embodiments herein.

[0019] [Figure 1C]1 illustrates another embodiment of a system for performing or facilitating object detection, identification, and retrieval in accordance with embodiments herein.

[0020] [Figure 1D] 1 illustrates yet another embodiment of a system for performing or facilitating object detection, identification, and retrieval in accordance with embodiments herein.

[0021] [Figure 2A] FIG. 1 is a block diagram illustrating a computing system configured to perform or facilitate object detection, identification and retrieval consistent with embodiments herein.

[0022] [Figure 2B] FIG. 1 is a block diagram illustrating one embodiment of a computing system configured to perform or facilitate object detection, identification, and retrieval consistent with embodiments herein.

[0023] [Figure 2C] FIG. 2 is a block diagram illustrating another embodiment of a computing system configured to perform or facilitate object detection, identification, and retrieval consistent with embodiments herein.

[0024] [Figure 2D] FIG. 2 is a block diagram illustrating yet another embodiment of a computing system configured to perform or facilitate object detection, identification, and retrieval consistent with embodiments herein.

[0025] [Figure 2E] 1 is an example of image information processed by the system and consistent with embodiments herein.

[0026] [Figure 2F] 4 is another example of image information processed by the system and consistent with embodiments herein.

[0027] [Figure 3A] 1 illustrates an exemplary environment for operating a robotic system, according to embodiments herein.

[0028] [Figure 3B] 1 illustrates an exemplary environment for object detection, identification, and retrieval by a robotic system consistent with embodiments herein.

[0029] [Figure 4] A flow diagram is provided illustrating the overall flow of a method and process for object detection, identification, and extraction according to embodiments herein.

[0030] [Diagram 5] 1 illustrates an example of object registration data, consistent with embodiments herein.

[0031] [Figure 6] 1 illustrates a method for generating an object recognition template consistent with embodiments herein.

[0032] [Figure 7A] 1 illustrates aspects of a method for generating object recognition templates consistent with embodiments herein. [Figure 7B] 1 illustrates aspects of a method for generating object recognition templates consistent with embodiments herein.

[0033] [Figure 8] 1 illustrates a method for generating an object recognition template consistent with embodiments herein.

[0034] [Figure 9A] 1 illustrates aspects of a method for generating an object recognition template consistent with embodiments herein. [Figure 9B] 1 illustrates aspects of a method for generating an object recognition template consistent with embodiments herein. [Figure 9C]1 illustrates aspects of a method for generating an object recognition template consistent with embodiments herein. [Figure 9D] 1 illustrates aspects of a method for generating an object recognition template consistent with embodiments herein.

[0035] [Figure 10A] 1 illustrates a method for object identification and hypothesis generation via template matching consistent with embodiments herein. [Figure 10B] 1 illustrates a method for object identification and hypothesis generation via template matching consistent with embodiments herein.

[0036] [Figure 11] FIG. 11 illustrates a method for refining a detection hypothesis consistent with embodiments herein.

[0037] [Figure 12A] 1 illustrates aspects of a method for refining a detection hypothesis consistent with embodiments herein. [Figure 12B] 1 illustrates aspects of a method for refining a detection hypothesis consistent with embodiments herein. [Figure 12C] 1 illustrates aspects of a method for refining a detection hypothesis consistent with embodiments herein.

[0038] [Figure 13] 1 illustrates a method for validating a detection hypothesis consistent with embodiments herein.

[0039] [Figure 14] 1 illustrates aspects of a method for refining a detection hypothesis consistent with embodiments herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] Systems and methods related to object detection, identification, and retrieval are described herein. In particular, the systems and methods of the present disclosure may facilitate object detection, identification, and retrieval where the object is located in a container. As discussed herein, the object may be metal or other material and may be located in a container such as a box, bin, crate, etc. The object may be located in the container in an unorganized or irregular manner, such as, for example, a box filled with screws. Although object detection and identification in such situations may be difficult due to the irregular placement of the objects, the systems and methods discussed herein may similarly improve object detection, identification, and object retrieval for objects that are placed in a regular or semi-regular manner. Thus, the systems and methods described herein are designed to identify individual objects among multiple objects, where the individual objects may be located at different positions, different angles, etc. The systems and methods discussed herein may include robotic systems. A robotic system configured according to embodiments herein may autonomously perform integrated tasks by coordinating the operation of multiple robots. A robotic system may include any suitable combination of robotic devices, actuators, sensors, cameras, and a computing system configured to control, issue commands, receive information from the robotic devices and sensors, access, analyze, and process data generated by the robotic devices, sensors, and cameras, generate data or information usable for control of the robotic system, and plan actions for the robotic devices, sensors, and cameras, as described herein. As used herein, a robotic system does not necessarily have immediate access to or control of the robotic actuators, sensors, or other devices. A robotic system may also be a computing system configured to improve the performance of such robotic actuators, sensors, and other devices through receiving, analyzing, and processing information, as described herein.

[0041] The technology described herein provides technical improvements to robotic systems configured for use in identifying, detecting, and retrieving objects. The technical improvements described herein increase the speed, precision, and accuracy of these tasks, making it easier to detect, identify, and retrieve objects from containers. The robotic and computing systems described herein address the technical problem of identifying, detecting, and retrieving objects from containers, where the objects may be irregularly positioned. Addressing this technical problem improves the technology of identifying, detecting, and retrieving objects.

[0042] This application refers to systems and robotic systems. A robotic system, as discussed herein, may include robotic actuator components (e.g., a robotic arm, a robotic gripper, etc.), various sensors (e.g., a camera, etc.), and various computational or control systems. As discussed herein, a computational or control system may be referred to as "controlling" various robotic components, such as the robotic arm, the robotic gripper, the camera, etc. Such "control" may refer to direct control and interaction of various actuators, sensors, and other functional aspects of the robotic components. For example, a computational system may control a robotic arm by issuing or providing all of the necessary signals to various motors, actuators, and sensors to cause robotic movements. Such "control" may also refer to the issuance of abstract or indirect commands to a further robotic control system that converts such commands into the necessary signals to cause robotic movements. For example, a computational system may control a robotic arm by issuing commands describing a trajectory or destination location along which the robotic arm should move, and a further robotic control system associated with the robotic arm may receive and interpret such commands and then provide the necessary direct signals to the various actuators and sensors of the robotic arm to cause the necessary movements.

[0043] Specifically, the technology described herein assists a robotic system in interacting with a target object among multiple objects in a container. Detecting, identifying, and removing an object from a container requires several steps, including generating an appropriate object recognition template, extracting features that can be used for identification, and generating, refining, and validating a detection hypothesis. For example, due to possible irregular placement of objects, it may be necessary to recognize and identify objects in multiple different poses (e.g., angles and positions) and when potentially obscured by parts of other objects.

[0044] In the following, specific details are described to provide an understanding of the technology of the present disclosure. In an embodiment, the technology introduced herein may be implemented without including each specific detail disclosed herein. In other instances, well-known features, such as specific functions or routines, are not described in detail to avoid unnecessarily obscuring the present disclosure. References herein to "embodiments," "one embodiment," and the like, mean that the particular feature, structure, material, or characteristic described is included in at least one embodiment of the present disclosure. Thus, appearances of such phrases herein do not necessarily all refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Moreover, particular features, structures, materials, or characteristics described with respect to any one embodiment can be combined in any suitable manner with those of any other embodiment, unless such items are mutually exclusive. It should be understood that the various embodiments shown in the figures are merely illustrative representations and are not necessarily drawn to scale.

[0045] For purposes of clarity, certain details describing structures or processes that are well known and often associated with robotic systems and subsystems, but that may unnecessarily obscure certain important aspects of the technology of the present disclosure, are not included in the following description. Additionally, although the following disclosure describes several embodiments of different aspects of the technology, some other embodiments may have different configurations or different components than those described in this section. Thus, the disclosed technology may have other embodiments that have additional elements or that do not have some of the elements described below.

[0046] Many embodiments or aspects of the disclosure described below may take the form of computer or controller executable instructions, including routines executed by a programmable computer or controller. Those skilled in the relevant art will appreciate that the disclosed technology may be practiced on or with computer or controller systems other than those shown and described below. The technology described herein may be embodied in a special purpose computer or data processor that is specially programmed, configured, or constructed to execute one or more of the computer executable instructions described below. Thus, the terms "computer" and "controller" as used generally herein refer to any data processor, and may include Internet appliances and handheld devices, including palmtop computers, wearable computers, cellular or mobile phones, multiprocessor systems, processor-based or programmable appliances, network computers, minicomputers, and the like. Information processed by these computers and controllers may be presented on any suitable display medium, including a liquid crystal display (LCD). Instructions for performing computer or controller executable tasks may be stored in or on any suitable computer readable medium, including hardware, firmware, or a combination of hardware and firmware. The instructions may be contained in any suitable memory device, including, for example, a flash drive, a USB device, and / or other suitable medium.

[0047] The terms "coupled" and "connected," along with their derivatives, may be used herein to describe a structural relationship between components. It should be understood that these terms are not intended as synonyms for each other. Rather, in certain embodiments, "connected" may be used to indicate that two or more elements are in direct contact with each other. Unless otherwise clear in the context, the term "coupled" may be used to indicate that two or more elements are in direct or indirect (with other intervening elements therebetween) contact with each other, or that two or more elements cooperate or interact with each other (e.g., in a causal relationship, such as for signal transmission / reception or for function calls), or both.

[0048] Any reference herein to image analysis by a computing system may be performed in accordance with or using spatial structure information, which may include depth information describing respective depth values ​​of various locations relative to a selected point. The depth information may be used to identify an object or estimate how an object is spatially arranged. In some instances, the spatial structure information may include, or be used to generate, a point cloud describing locations on one or more surfaces of an object. Spatial structure information is only one form of possible image analysis, and other forms known to those skilled in the art may be used in accordance with the methods described herein.

[0049] FIG. 1A illustrates a system 1000 for performing object detection, or more specifically, object recognition. More specifically, the system 1000 may include a computing system 1100 and a camera 1200. In this example, the camera 1200 may be configured to generate image information that describes or otherwise represents an environment in which the camera 1200 is located, or more specifically, represents an environment within a field of view of the camera 1200 (also referred to as a camera field of view). The environment may be, for example, a warehouse, a manufacturing plant, a retail space, or other facility. In such an instance, the image information may represent objects located in such a facility, such as boxes, bins, cases, crates, pallets, or other containers. The system 1000 may be configured to generate, receive, and / or process the image information, such as using the image information to distinguish between individual objects within the camera field of view, perform object recognition or object registration based on the image information, and / or perform a robot interaction plan based on the image information, as discussed in more detail below (the terms "and / or" and "or" are used interchangeably in this disclosure). The robot interaction plan may be used to control a robot at a facility, for example, to facilitate robot interaction between the robot and a container or other object. The computing system 1100 and the camera 1200 may be located at the same facility or may be located remotely from one another. For example, the computing system 1100 may be part of a cloud computing platform hosted at a data center remote from a warehouse or retail space and may communicate with the camera 1200 via a network connection.

[0050] In one embodiment, the camera 1200 (which may also be referred to as an image sensing device) may be a 2D camera and / or a 3D camera. For example, FIG. 1B shows a computing system 1100 and a system 1500A, which may be an embodiment of the system 1000, including a camera 1200A and a camera 1200B, both of which may be embodiments of the camera 1200. In this example, the camera 1200A may be a 2D camera configured to generate 2D image information that includes or forms a 2D image describing the visual appearance of the environment in the camera's field of view. The camera 1200B may be a 3D camera (also referred to as a spatial structure sensing camera or a spatial structure sensing device) configured to generate 3D image information that includes or forms spatial structure information about the environment in the camera's field of view. The spatial structure information may include depth information (e.g., a depth map) that describes the respective depth values ​​of various locations relative to the camera 1200B, such as locations on the surface of various objects in the field of view of the camera 1200. These locations on the field of view of the camera or the surface of the object may also be referred to as physical locations. The depth information in this example can be used to estimate how the object is spatially positioned in three-dimensional (3D) space. In some instances, the spatial structure information may include, or be used to generate, a point cloud that describes locations on one or more surfaces of an object within the field of view of camera 1200B. More specifically, the spatial structure information can describe various positions on the structure of the object (also referred to as the object structure).

[0051] In one embodiment, the system 1000 may be a robotic manipulation system for facilitating robotic interaction between a robot and various objects in the environment of the camera 1200. For example, FIG. 1C illustrates a robotic manipulation system 1500B, which may be an embodiment of the system 1000 / 1500A of FIGS. 1A and 1B. The robotic manipulation system 1500B may include a computing system 1100, a camera 1200, and a robot 1300. As described above, the robot 1300 may be used to interact with one or more objects, such as boxes, crates, bins, pallets, or other containers, in the environment of the camera 1200. For example, the robot 1300 may be configured to pick up containers from one location and move them to another location. In some cases, the robot 1300 may be used to perform an unpalletizing operation, in which a group of containers or other objects are unloaded and moved, for example, to a conveyor belt. In some embodiments, the camera 1200 may be attached to the robot 1300 or the robot 3300, discussed below. This is also known as a handheld camera or handheld camera solution. The camera 1200 may be attached to a robot arm 3320 of the robot 1300. The robot arm 3320 may then move to various lifting areas to generate image information for those areas. In some embodiments, the camera 1200 may be separate from the robot 1300. For example, the camera 1200 may be mounted on the ceiling of a warehouse or other structure and remain stationary relative to the structure. In some embodiments, multiple cameras 1200 may be used, including multiple cameras 1200 separate from the robot 1300 used with a handheld camera 1200 and / or cameras 1200 separate from the robot 1300. In some embodiments, the camera 1200 or multiple cameras 1200 may be mounted or fixed to a dedicated robotic system separate from the robot 1300 used for object manipulation, such as a robotic arm, gantry, or other automated system configured for camera movement. Throughout this specification, we may discuss “control” or “controlling” the camera 1200 .For a handheld camera solution, control of the camera 1200 also includes control of the robot 1300 to which the camera 1200 is attached or mounted.

[0052] In one embodiment, the computing system 1100 of FIGS. 1A-1C may form or be incorporated into the robot 1300, which may be referred to as a robot controller. A robot control system may be included in the system 1500B and configured to generate commands for the robot 1300, such as, for example, robot interaction movement commands for controlling a robot interaction between the robot 1300 and a container or other object. In such an embodiment, the computing system 1100 may be configured to generate such commands based on, for example, image information generated by the camera 1200. For example, the computing system 1100 may be configured to determine a motion plan based on the image information, where the motion plan may, for example, be intended to grasp or otherwise pick up an object. The computing system 1100 may generate one or more robot interaction movement commands to execute the motion plan.

[0053] In an embodiment, the computing system 1100 may form or be part of a vision system. The vision system may be, for example, a system that generates visual information describing the environment in which the robot 1300 is located, or alternatively or additionally describing the environment in which the camera 1200 is located. The visual information may include the 3D image information discussed above, and / or the 2D image information, or some other image information. In some scenarios, if the computing system 1100 forms a vision system, the vision system may be part of the robot control system discussed above, or may be separate from the robot control system. If the vision system is separate from the robot control system, the vision system may be configured to output information describing the environment in which the robot 1300 is located. The information may be output to a robot control system, which may receive such information from the vision system, and based on the information, execute a motion plan and / or generate a robot interaction movement command. More information regarding the vision system is described in detail below.

[0054] In one embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 through a direct connection, such as a connection provided through a dedicated wired communication interface, such as an RS-232 interface, a universal serial bus (USB) interface, and / or a local computer bus, such as a peripheral component interconnect (PCI) bus. In one embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 through a network. The network may be any type and / or form of network, such as a personal area network (PAN), a local area network (LAN), e.g., an intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The network may utilize different technologies and layers or stacks of protocols, including, for example, Ethernet protocols, Internet Protocol Suite (TCP / IP), Asynchronous Transfer Mode (ATM) technologies, Synchronous Optical Networking (SONET) protocols, or Synchronous Digital Hierarchy (SDH) protocols.

[0055] In an embodiment, the computing system 1100 may communicate directly with the camera 1200 and / or the robot 1300, or may communicate via an intermediate storage device, or more broadly, an intermediate non-transitory computer-readable medium. For example, FIG. 1D illustrates a system 1500C, which may be an embodiment of the systems 1000 / 1500A / 1500B, including a non-transitory computer-readable medium 1400 that may be external to the computing system 1100, and may act as an external buffer or repository for storing image information generated by the camera 1200, for example. In one such example, the computing system 1100 may retrieve or otherwise receive image information from the non-transitory computer-readable medium 1400. Examples of the non-transitory computer-readable medium 1400 include electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. The non-transitory computer readable medium may form, for example, a computer diskette, a hard disk drive (HDD), a solid state drive (SDD), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read only memory (CD-ROM), a digital versatile disc (DVD), and / or a memory stick.

[0056] As mentioned above, the camera 1200 may be a 3D camera and / or a 2D camera. The 2D camera may be configured to generate a 2D image, such as a color image or a grayscale image. The 3D camera may be a depth-sensing camera, such as a time-of-flight (TOF) camera or a structured light camera, or any other type of 3D camera. In some cases, the 2D camera and / or the 3D camera may include an image sensor, such as a charge-coupled device (CCD) sensor and / or a complementary metal-oxide semiconductor (CMOS) sensor. In an embodiment, the 3D camera may include a laser, a LIDAR device, an infrared device, a light / dark sensor, a motion sensor, a microwave detector, an ultrasonic detector, a radar detector, or any other device configured to capture depth information or other spatial structure information.

[0057] As described above, image information may be processed by computing system 1100. In one embodiment, computing system 1100 may include or be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop computer, etc.), a smartphone, a tablet computing device, and / or any other computing system. In one embodiment, any or all of the functionality of computing system 1100 may be performed as part of a cloud computing platform. Computing system 1100 may be a single computing device (e.g., a desktop computer) or may include multiple computing devices.

[0058] 2A provides a block diagram illustrating one embodiment of a computing system 1100. The computing system 1100 in this embodiment includes at least one processing circuit 1110 and non-transitory computer-readable medium(s) 1120. In some instances, the processing circuit 1110 may include a processor (e.g., a central processing unit (CPU), a special-purpose computer, and / or an on-board server) configured to execute instructions (e.g., software instructions) stored on the non-transitory computer-readable medium 1120 (e.g., computer memory). In some embodiments, the processor may be included in a separate / standalone controller operably coupled to other electronic / electrical devices. The processor may execute program instructions to control / interface with other devices, thereby causing the computing system 1100 to perform actions, tasks, and / or processes. In one embodiment, the processing circuitry 1110 includes one or more processors, one or more processing cores, a programmable logic controller ("PLC"), an application specific integrated circuit ("ASIC"), a programmable gate array ("PGA"), a field programmable gate array ("FPGA"), any combination thereof, or any other processing circuitry.

[0059] In one embodiment, the non-transitory computer readable medium 1120 that is part of the computing system 1100 may be a replacement for or in addition to the intermediate non-transitory computer readable medium 1400 discussed above. The non-transitory computer readable medium 1120 may be a storage device such as an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, such as, for example, a computer diskette, a hard disk drive (HDD), a solid state drive (SSD), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, any combination thereof, or any other storage device. In some instances, the non-transitory computer readable medium 1120 may include multiple storage devices. In certain embodiments, the non-transitory computer readable medium 1120 is configured to store image information generated by the camera 1200 and received by the computing system 1100. In some instances, the non-transitory computer-readable medium 1120 may store one or more object recognition templates used to perform the methods and processes discussed herein. The non-transitory computer-readable medium 1120 may alternatively or additionally store computer-readable program instructions that, when executed by the processing circuit 1110, cause the processing circuit 1110 to perform one or more methodologies described herein.

[0060] FIG. 2B illustrates an embodiment of a computing system 1100, including a communications interface 1131 1 shows a computing system 1100A including a communications interface. 11311A-1D . The image information may be received via an intermediate non-transitory computer-readable medium 1400 or a network as discussed above, or via a more direct connection between the camera 1200 and the computing system 1100 / 1100A. In one embodiment, the communications interface 1131 may be configured to communicate with the robot 1300 of FIG. 1C. If the computing system 1100 is external to the robot control system, the communication interface of the computing system 1100 1131 may be configured to communicate with the robot control system. 1131 may also be referred to as a communications component or communications circuitry, and may include, for example, communications circuitry configured to perform communications over a wired or wireless protocol. By way of example, the communications circuitry may include an RS-232 port controller, a USB controller, an Ethernet controller, a Bluetooth controller, a PCI bus controller, any other communications circuitry, or a combination thereof.

[0061] 2C, the non-transitory computer-readable medium 1120 may include a storage space 1125 configured to store one or more data objects discussed herein. For example, the storage space may store object recognition templates, detection hypotheses, image information, object image information, robotic arm movement commands, and any additional data objects that the computing system discussed herein may need to access.

[0062] In one embodiment, the processing circuit 1110 may be programmed by one or more computer readable program instructions stored on a non-transitory computer readable medium 1120. For example, FIG. 2D illustrates a computing system 1100C, an embodiment of the computing system 1100 / 1100A / 1100B, where the processing circuit 1110 is programmed by one or more modules, including an object recognition module 1121, a motion planning module 1129, and an object manipulation planning module 1126. The processing circuit 1110 may be further programmed with a hypothesis generation module 1128, an object registration module 1130, a template generation module 1132, a feature extraction module 1134, a hypothesis refinement module 1136, and a hypothesis verification module 1138. Each of the above modules may represent computer readable program instructions configured to perform a particular task when instantiated on one or more of the processors, processing circuits, computing systems, etc. described herein. Each of the above modules may cooperate with each other to accomplish the functions described herein. Various aspects of the functionality described herein may be performed by one or more of the above-mentioned software modules, and the software modules and their descriptions are not to be understood as limiting the computational structure of the systems disclosed herein. For example, a particular task or function may be described with respect to a particular module, but that task or function may be performed by different modules as appropriate. Furthermore, the system functions described herein may be performed by different sets of software modules configured with different functional divisions or allocations.

[0063] In one embodiment, the object recognition module 1121 may be configured to acquire and analyze image information as discussed throughout this disclosure. The methods, systems, and techniques discussed herein with respect to image information may use the object recognition module 1121. The object recognition module may be further configured for object recognition tasks related to object identification as discussed herein.

[0064] The motion planning module 1129 may be configured to plan and execute robot movements. For example, the motion planning module 1129 may interact with other modules described herein to plan the movements of the robot 3300 for object picking and camera placement operations. The methods, systems, and techniques discussed herein with respect to robot arm movements and trajectories may be implemented by the motion planning module 1129.

[0065] The object manipulation planning module 1126 may be configured to plan and execute object manipulation activities for the robotic arm, for example, to grasp and release objects, and to execute robotic arm commands to assist and facilitate such grasping and releasing.

[0066] The hypothesis generation module 1128 may be configured to perform template matching and recognition tasks to generate detection hypotheses, for example, as described with respect to Figures 10A-10B. The hypothesis generation module 1128 may be configured to interact or communicate with any other necessary modules.

[0067] The object registration module 1130 may be configured to acquire, store, generate, and otherwise process object registration information that may be required for various tasks discussed herein. The object registration module 1130 may be configured to interact or communicate with any other necessary modules.

[0068] The template generation module 1132 may be configured to complete the object recognition template generation task as discussed herein, for example, in connection with Figures 6-9D. The template generation module 1132 may be configured to interact with the object registration module 1130, the feature extraction module 1134, and any other necessary modules.

[0069] The feature extraction module 1134 may be configured to complete the feature extraction and generation tasks discussed herein, for example, in connection with Figures 8-9D. The feature extraction module 1134 may be configured to interact with the object registration module 1130, the template generation module 1132, the hypothesis generation module 1128, and any other necessary modules.

[0070] The hypothesis refinement module 1136 may be configured to complete a hypothesis refinement task, for example, as discussed herein in connection with Figures 11-12C. The hypothesis refinement module 1136 may be configured to interact with the object recognition module 1121 and the hypothesis generation module 1128, as well as any other necessary modules.

[0071] The hypothesis verification module 1138 may be configured to complete hypothesis verification tasks as discussed herein, for example, in connection with Figures 13-14. The hypothesis verification module 1138 may be configured to interact with the object registration module 1130, the feature extraction module 1134, the hypothesis generation module 1128, the hypothesis refinement module 1136, and any other necessary modules.

[0072] With reference to Figures 2E, 2F, 3A and 3B, methods related to the object recognition module 1121 that may be implemented for image analysis are described. Figures 2E and 2F show example image information associated with the image analysis method, while Figures 3A and 3B show an example robot environment associated with the image analysis method. References herein related to image analysis by a computing system may be performed according to or using spatial structure information, which may include depth information describing respective depth values ​​of various locations relative to a selected point. The depth information may be used to identify an object or estimate how an object is spatially arranged. In some instances, the spatial structure information may include or be used to generate a point cloud describing locations on one or more surfaces of an object. Spatial structure information is just one form of possible image analysis, and other forms known to those skilled in the art may be used according to the methods described herein.

[0073] In an embodiment, the computing system 1100 may acquire image information representative of objects within a camera field of view (e.g., 3200) of the camera 1200. The steps and techniques described below for acquiring image information are hereinafter referred to as image information. capture The process 3001 may be referred to as process 3001. In some instances, the object may be one object 5012 from a plurality of objects 5012 in a scene 5013 of the field of view 3200 of the camera 1200. The image information 2600, 2700 may be generated by the camera (e.g., 1200) when the object 5012 is (or was) in the camera field of view 3200 and may describe one or more of the individual objects 5012 or the scene 5013. The object appearance describes the appearance of the object 5012 from the viewpoint of the camera 1200. If there are multiple objects 5012 in the camera field of view, the camera may generate image information representing the multiple objects or a single object (such image information for a single object may be referred to as object image information), as appropriate. The image information may be generated by the camera (e.g., 1200) when a group of objects is (or was) in the camera field of view and may include, for example, 2D image information and / or 3D image information.

[0074] As an example, FIG. 2E illustrates a first set of image information, more specifically, 2D image information 2600, which is generated by camera 1200 as described above and which depicts objects 3410A / 3410B / 3410C / 3410D of FIG. D More specifically, the 2D image information 2600 may be a grayscale or color image and represents an image of the object 3410A / 3410B / 3410C / 3410D from the viewpoint of the camera 1200. D In one embodiment, the 2D image information 2600 may correspond to a single color channel (e.g., red, green, or blue channel) of a color image. D When placed above, the 2D image information 2600 includes the objects 3410A / 3410B / 3410C / 3410D. DIn the example of FIG. 2E, the 2D image information 2600 may represent the appearance of the upper surface of each of the objects 3410A / 3410B / 3410C. 3410C / 3410 D Each of the 2D image information 2600 may include a respective portion 2000A / 2000B / 2000C / 2000D / 2550, also referred to as an image portion or object image information, representing a respective surface. In FIG. 2E, each of the image portions 2000A / 2000B / 2000C / 2000D / 2550 of the 2D image information 2600 may be an image region, or more specifically, a pixel region (where an image is formed by pixels). Each pixel in the pixel region of the 2D image information 2600 may be characterized as having a location described by a set of coordinates [U, V], and may have a value relative to the camera coordinate system or some other coordinate system, as shown in FIG. 2E and FIG. 2F. Each of the pixels may also have an intensity value, such as a value between 0 and 255 or between 0 and 1023. In further embodiments, each of the pixels may include any additional information associated with the pixel in various formats (e.g., hue, saturation, intensity, CMYK, RGB, etc.).

[0075] As mentioned above, the image information may be all or a portion of an image, such as the 2D image information 2600, in some embodiments. In an example, the computing system 1100 may be configured to extract the image portion 2000A from the 2D image information 2600 to obtain only image information associated with the corresponding object 3410A. If an image portion (such as the image portion 2000A) is directed at a single object, it may be referred to as object image information. The object image information need not include only information about the object at which it is directed. For example, the object at which it is directed may be located near, under, on, or otherwise in the vicinity of one or more other objects. In such a case, the object image information may include information about the object at which it is directed, as well as one or more adjacent objects. For example, the computing system 1100 may extract the image portion 2000A by performing an image segmentation or other analysis or processing operation based on the 2D image information 2600 and / or the 3D image information 2700 shown in FIG. 2F. In some embodiments, image segmentation or other operations may include detecting image locations where physical edges of objects (e.g., edges of objects) appear in the 2D image information 2600, and using these image locations to identify object image information limited to representing individual objects within the camera field of view (e.g., 3200) and substantially excluding other objects. By "substantially excluding," it is meant that the image segmentation or other processing techniques are designed and configured to exclude non-target objects from the object image information, while it is understood that errors may occur, noise may be present, and various other factors may result in the inclusion of portions of other objects.

[0076] 2F illustrates an example where the image information is 3D image information 2700. More specifically, the 3D image information 2700 may include, for example, 3D images of objects 3410A / 3410B / 3410C / 3410D. DIt may include a depth map or point cloud indicating respective depth values ​​for various locations on one or more surfaces (e.g., a top surface, or other exterior surfaces). In some embodiments, an image segmentation operation for extracting image information may include detecting image locations where physical edges of objects (e.g., edges of a box) appear in the 3D image information 2700, and using these image locations to identify image portions (e.g., 2730) that are restricted to representing individual objects within the camera field of view (e.g., 3410A).

[0077] Each depth value may be relative to the camera 1200 generating the 3D image information 2700, or may be relative to some other reference point. In some embodiments, the 3D image information 2700 may include a point cloud that includes respective coordinates for various locations on the structure of objects that are within the camera field of view (e.g., 3200). In the example of FIG. 2F, the point cloud includes the coordinates of objects 3410A / 3410B / 3410C / 3410D. D The 3D image information 2700 may include a respective set of coordinates describing a location of each surface. The coordinates may be 3D coordinates, such as [XYZ] coordinates, and may have values ​​relative to the camera coordinate system or some other coordinate system. For example, the 3D image information 2700 may include locations 27101-2710, also referred to as physical locations on the surface of the object 3410D. n 27. In addition, the 3D image information 2700 may further include a second portion, a third portion, a fourth portion, and a fifth portion 2720, 2730, 2740, and 2750. These portions are then respectively denoted as 27201 to 2720. n , 27301~2730 n , 27401~2740 n , and 27501 to 2750 nThese figures are merely examples and any number of objects having corresponding image portions may be used. As mentioned above, the captured 3D image information 2700 may in some instances be a portion of a first set of 3D image information 2700 generated by the camera. F In this example, if the acquired 3D image information 2700 represents the object 3410A of Figure 3A, the 3D image information 2700 may be narrowed to refer only to the image portion 2710. Similar to the discussion of the 2D image information 2600, the identified image portion 2710 may relate to an individual object and may be referred to as object image information. Thus, object image information, as used herein, may include 2D and / or 3D image information.

[0078] In one embodiment, an image normalization operation may be performed by the computing system 1100 as part of acquiring image information. The image normalization operation may involve transforming an image or image portion generated by the camera 1200 to generate a transformed image or a transformed image portion. For example, the acquired image information, which may include 2D image information 2600, 3D image information 2700, or a combination of the two, may be subjected to an image normalization operation to attempt to alter the image information in terms of viewpoint, object pose, and lighting conditions associated with the visual description information. Such normalization may be performed to facilitate a more accurate comparison between the image information and the model (e.g., template) information. The viewpoint may refer to the pose of the object relative to the camera 1200 and / or the angle at which the camera 1200 is viewing the object when the camera 1200 generates an image representing the object.

[0079] For example, image information may be generated during an object recognition operation in which a target object is within the camera field of view 3200. The camera 1200 may generate image information representing a target object when the target object has a particular pose relative to the camera. For example, the target object may have a pose that makes its top surface perpendicular to the optical axis of the camera 1200. In such an example, the image information generated by the camera 1200 may represent a particular perspective, such as a top view of the target object. In some instances, when the camera 1200 is generating image information during an object recognition operation, the image information may be generated at particular lighting conditions, such as lighting intensity. In such an example, the image information may represent a particular lighting intensity, lighting color, or other lighting condition.

[0080] In one embodiment, the image normalization operation may involve adjusting an image or image portion of a scene generated by a camera to better match the image or image portion to viewpoint and / or lighting conditions associated with the object recognition template information. The adjustment may involve transforming the image or image portion to generate a transformed image that matches at least one of the object pose or lighting conditions associated with the visual description information of the object recognition template.

[0081] Viewpoint adjustment may involve processing, warping, and / or shifting an image of a scene so that the image represents the same perspective as the visual description information that may be included in the object recognition template. Processing may include, for example, changing the color, contrast, or lighting of the image, warping the scene may include changing the size, dimensions, or ratio of the image, and shifting the image may include changing the position, orientation, or rotation of the image. In an exemplary embodiment, processing, warping, and / or shifting may be used to change objects in the image of a scene to have an orientation and / or size that matches or better corresponds to the visual description information of the object recognition template. If the object recognition template describes a front view (e.g., a top view) of some objects, the image of the scene may be warped to also represent the front view of the objects in the scene.

[0082] Further aspects of the object recognition methods embodied in this specification are described in more detail in U.S. patent application Ser. No. 16 / 991,510, filed Aug. 12, 2020, and U.S. patent application Ser. No. 16 / 991,466, filed Aug. 12, 2020, each of which is incorporated by reference herein.

[0083] In various embodiments, the terms "computer readable instructions" and "computer readable program instructions" are used to describe software instructions or computer code configured to perform various tasks and operations. In various embodiments, the term "module" refers broadly to a collection of software instructions or code configured to cause the processing circuitry 1110 to perform one or more functional tasks. The modules and computer readable instructions may be described as performing various operations or tasks when a processing circuitry or other hardware component executes the modules or computer readable instructions.

[0084] 3A-3B illustrate an exemplary environment in which computer readable program instructions stored on a non-transitory computer readable medium 1120 are utilized via a computing system 1100 to increase the efficiency of object identification, detection, and retrieval operations and methods. Image information acquired by the computing system 1100 and illustrated in FIG. 3A influences the system's decision-making procedures and command output to a robot 3300 present within the object environment.

[0085] 3A-3B illustrate an exemplary environment in which the processes and methods described herein may be implemented. FIG. 3A illustrates an environment having a system 3000 (which may be an embodiment of systems 1000 / 1500A / 1500B / 1500C of FIGS. 1A-1D ) including at least a computing system 1100, a robot 3300, and a camera 1200. The camera 1200 may be an embodiment of the camera 1200, and may represent a scene 5013, or more specifically, an object 5020, within a camera field of view 3200 of the camera 1200. 3410 A. 3410 B. 3410 C, and3410 The camera may be configured to generate image information representing an object (e.g., a box) within the camera field of view 3200, such as D. 3410 A~ 3410 Each of D may be, for example, a container such as a box or crate, while the object 3400 may be, for example, a pallet on which the container is disposed. 3410 A~ 3410 Each of objects 5012 may be a container that contains an individual object 5012. Each object 5012 may be, for example, a rod, a bar, a gear, a bolt, a nut, a screw, a nail, a rivet, a spring, a linkage, a cogwheel, or any other type of physical object, as well as an assembly of multiple objects. Figure 3A shows an embodiment that includes multiple containers of objects 5012, while Figure 3B shows an embodiment that includes a single container of objects 5012.

[0086] In one embodiment, the system 3000 of FIG. 3A can include one or more light sources. The light sources can be, for example, light emitting diodes (LEDs), halogen lamps, or any other light source that can emit visible light, infrared light, or object light. 3410 A~ 3410 D. In some embodiments, the computing system 1100 may be configured to communicate with the light sources to control when the light sources are activated. In other embodiments, the light sources may operate independently of the computing system 1100.

[0087] In one embodiment, the system 3000 may include a camera 1200 or multiple cameras 1200, including a 2D camera configured to generate 2D image information 2600 and a 3D camera configured to generate 3D image information 2700. The camera 1200 or multiple cameras 1200 may be mounted or attached to the robot 3300, may be stationary in the environment, and / or may be fixed to a dedicated robotic system separate from the robot 3300 used for object manipulation, such as a robotic arm, gantry, or other automated system configured for camera movement. FIG. 3A shows an example with a stationary camera 1200 and a handheld camera 1200, while FIG. 3B shows an example with only a stationary camera 1200. The 2D image information 2600 (e.g., a color image or a grayscale image) may include an image of an object in the camera field of view 3200. 3410 A / 3410 B / 3410 C / 3410 For example, the 2D image information 2600 may describe the appearance of one or more objects, such as the object 5012 or the object D. 3410 A / 3410 B / 3410 C / 3410 D and 5012. In one embodiment, the 3D image information 2700 may capture or otherwise represent visual details located on the exterior surfaces (e.g., top surfaces) of the objects D and 5012, and / or the contours of those exterior surfaces. 3410 A / 3410 B / 3410 C / 3410 D / 3400 and 5012, where the structure for an object may also be referred to as the structure of the object or the physical structure of the object. For example, the 3D image information 2700 may include a depth map, or more generally, depth information that may describe respective depth values ​​of various locations in the camera field of view 3200 relative to the camera 1200 or relative to some other reference point. The locations corresponding to each depth value may be a depth map of the object. 3410 A / 3410 B / 3410 C / 3410 D / 34005012 and 5013. In some instances, the 3D image information 2700 may be various surface locations (also referred to as physical locations) of the camera field of view 3200, such as locations on the top surfaces of the respective cameras 5012 and 5013. 3410 A / 3410 B / 3410 C / 3410 D / 3400 5012, or some other object within the camera field of view 3200. The point cloud may include a point cloud, which may include a plurality of 3D coordinates describing various locations on one or more exterior surfaces of the objects within the camera field of view 3200. The point cloud is shown in FIG.

[0088] 3A and 3B, a robot 3300 (which may be an embodiment of robot 1300) may include a robot arm 3320 having one end attached to a robot base 3310 and having another end attached to or formed by an end effector device 3330, such as a robot gripper. The robot base 3310 may be used to mount the robot arm 3320, while the robot arm 3320, and more specifically the end effector device 3330, may be used to interact with one or more objects in the environment of the robot 3300. The interaction (also referred to as robot interaction) may include, for example, the interaction of the robot with an object, 3410 A~ 3410 The robotic interaction may include grasping or otherwise picking up at least one of the object 5012 and the object 5012. For example, the robotic interaction may be part of an object picking operation to identify, detect, and remove the object 5012 from the container. The end effector device 3330 may have a suction cup or other component for grasping or gripping the object 5012. The end effector device 3330 may be configured to grasp or grip the object using the suction cup or other gripping component, for example, through contact with a single face or surface of the object via the top surface.

[0089] The robot 3300 may further include additional sensors configured to obtain information used to perform tasks, such as to manipulate structural members and / or to transport the robotic unit. The sensors may include devices configured to detect or measure one or more physical characteristics of the robot 3300 and / or the surrounding environment (e.g., its state, condition, and / or location of one or more structural members / joints). Some examples of sensors may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

[0090] 4 provides a flow diagram illustrating the overall flow of a method and operation for object detection, identification, and extraction according to an embodiment of the present disclosure. The object detection, identification, and extraction method 4000 may include any combination of features of the sub-methods and operations described herein. The method 4000 includes an object registration process 5000, an object recognition template generation method 6000, a feature generation method 8000, an image information generation method 8000, and an object recognition template generation method 9000. capture The method may include any or all of the process 3001, the hypothesis generation process 10000, the hypothesis refinement method 11000, the hypothesis verification method 13000, and the robot control process 15000, including obstacle detection, motion planning, and motion execution. In an embodiment, the object registration process 5000, the object recognition template generation method 6000, and the feature generation method 8000 may be performed in a pre-processing or offline environment, outside the context of the robot operation. Thus, these processes and methods may be performed in advance to facilitate later actions by the robot. Image information capture Process 3001, hypothesis generation process 10000, hypothesis refinement method 11000, hypothesis testing method 13000, and robot control process 15000 may each be performed in the context of robotic operations to detect, identify, and remove objects from a container.

[0091] 5 illustrates object registration data related to a type of object that may be generated, acquired, received, or otherwise obtained during an object registration process 5000. As described above, the methods and systems described herein are configured to acquire and use object registration data 5001, e.g., known, previously stored information related to an object 5011, to generate an object recognition template for use in identifying and recognizing similar objects in a physical scene. The object registration data 5001 may include any type of computer-readable information that identifies, associates with, and / or describes an object model 4200. The object registration data 5001 for an object model 4200 may represent an object 5011, the object model 4200 being in a two- or three-dimensional format that may or may not be interactive, providing measurements and dimensions of the represented object 5011. The object registration data 5001 may include, for example, CAD (i.e., computer-aided design) data or other modeling data that describes the object model 4200 and is stored in any suitable format. The registration data may be a solid CAD model, a wireframe CAD model, or a surface CAD model. In one embodiment, the registration data may be any type of three-dimensional file format, such as FBX, OBJ, USD, STL, STEP, COLLADE, etc. The object model 4200 represents one or more physical objects. The object model 4200 is a modeled (i.e., computer stored) version of one or more corresponding objects 5011 that physically exist in the world. As shown in FIG. 5, the object 5011 is a physical object that exists in the physical world, and the object model 4200 is a digital representation of the object 5011 described by the object registration data 5001. The depicted object 5011 may be any object, including, for example, a rod, a bar, a gear, a bolt, a nut, a screw, a nail, a rivet, a spring, a chain, a cogwheel, or any other type of physical object, as well as an assembly of multiple objects. In an embodiment, object 5011 may refer to an object accessible from a container (e.g., a bin, box, bucket, etc.) having, for example, a mass in the range of a few grams to a few kilograms, and a size in the range of, for example, 5 mm to 500 mm.The object model 4200 may be specific to a precise version of a real-world object 5011, such as, for example, a screw having a particular length, number of threads, thread size, head size, etc. For example, and for illustrative purposes, this description refers to a screw-shaped object as the object 5011. This is presented for convenience only and is not intended to limit the scope of the description in any way.

[0092] In some embodiments, the disclosure relates to generating a set of object recognition templates for identifying an object 5012 in a scene 5013. The object enrollment data 5001 may be based on a physical object 5011 and may be used to facilitate recognition of other physical objects 5012 that are similar to (and may be copies or versions of) the physical object 5011. Identifying the object 5012 in the scene may include identifying the object model 4200 to which the object 5012 corresponds (e.g., identifying what the object 5012 is), identifying the pose of the object 5012 (e.g., identifying the position, angle, and orientation of the object 5012).

[0093] 6 illustrates a flow diagram of an exemplary object recognition template generation method 6000 for generating a set of object recognition templates. In one embodiment, the object recognition template generation method 6000 is implemented, for example, by the computing system 1100 (or 1100A / 1100B / 1100C) of FIGS. 2A-2D or the computing system 1100 (or 1100A / 1100B / 1100C) of FIGS. 3B2D , which may perform the object recognition template generation method 6000. For example, the instructions may cause the computing system 1100 to execute one or more of the modules illustrated in FIG. 2D , which may perform the object recognition template generation method 6000. For example, in an embodiment, the steps of the object recognition template generation method 6000 may be performed by the object registration module 1130 and the template generation module 1132, which may cooperate to generate the object recognition template.

[0094] The steps of the object recognition template generation method 6000 may be utilized to effect object recognition template generation, which may then be used in conjunction with a particular continuous robot trajectory to perform a particular task. As a general overview, the object recognition template generation method 6000 may process to cause the computing system 1100 to generate a set of object recognition templates for the computing system to use in identifying objects within a scene for processing related to picking the object. The object recognition template generation method 6000 is described below with further reference to Figures 7A and 7B.

[0095] At least one processing circuit 1110 may perform certain steps of an object recognition template generation method 6000 to generate an object recognition template set 4301, which may include a plurality of object recognition templates 4300. The object recognition template generation method 6000 may begin with or include an operation 6001, which includes obtaining object enrollment data 5001 for an object model 4200 representing an object 5011.

[0096] In process 6001, the object recognition template generation method 6000 may include obtaining object registration data 5001 representing an object 5011, where the object registration data 5001 may include an object model 4200 representing the object 5011. The at least one processing circuit 1110 may determine a plurality of viewpoints 4120 of the object model in the three-dimensional space 4100. The at least one processing circuit 1110 may further estimate a plurality of appearances 4140 of the object model 4200 at each of the plurality of viewpoints 4120. The robotic system may further generate a plurality of object recognition templates 4300 (e.g., 4300A / 4300B / 4300C / 4300D) according to the plurality of appearances, where each of the plurality of object recognition templates 4300 corresponds to a respective one of the plurality of appearances 4140. The at least one processing circuit 1110 may then communicate the plurality of object recognition templates 4300 to the robotic system or a storage system as an object recognition template set 4301 for later use. Each of the multiple object recognition templates 4300 may represent a pose that the object model 4200 may have relative to the optical axis 4130 of the virtual camera 4110. Each object recognition template 4300 represents a view of the object 5011 corresponding to the object model 4200 from the line of sight of a camera 1200 that has a line of sight corresponding to the line of sight of the virtual camera 4110 during generation of the object recognition template 4300.

[0097] The at least one processing circuit 1110 may obtain the object registration data 5001 from within its own hardware storage component (i.e., HDD, SSD, USB, CD, RAID, etc.) or software storage component (i.e., cloud, VSP, etc.). In one embodiment, the at least one processing circuit 1110 may obtain the registration data from an external processor (i.e., an external laptop, desktop, mobile phone, or any other separate device with its own processing system).

[0098] The object recognition template generation method 6000 may further include a process 6003 that may include determining multiple viewpoints 4120 of the object model 4200 in the three-dimensional space 4100. This may be referred to as a spatial subsampling procedure. The three-dimensional space 4100 surrounding the object model 4200 may be surrounded by a surface 4101. The three-dimensional space 4100 and the surface 4101 are virtual entities surrounding the object model 4200, which is also a virtual entity. Each of the multiple viewpoints 4120 determined in the process 6003 may correspond to or represent a position of a virtual camera 4110 on the surface 4101 surrounding the three-dimensional space 4100 and a rotation angle of the virtual camera 4110 around an optical axis 4130 of the virtual camera 4110. Thus, each position on the surface 4101 may correspond to multiple viewpoints 4120.

[0099] The virtual cameras 4110 used in the spatial subsampling procedure may capture the appearance of the object from the viewpoint 4120 at which the virtual camera 4110 is located. For example, as shown in FIG. 7A, the virtual cameras 4110 located at the respective viewpoints 4120A may capture the appearance 4140 of the object model 4200. The appearance 4140 includes information describing the appearance of the object model 4200 to the virtual camera 4110 based on the angle of view and rotation of the virtual camera 4110 about the optical axis of the virtual camera 4110. The object model 4200 may be fixed within this three-dimensional space 4100. In one embodiment, the three-dimensional space may be substantially spherical. The object model 4200 may further be fixed at or near the center of the substantially spherical three-dimensional space. In another embodiment, the three-dimensional space may be any other suitable three-dimensional shape, such as an ellipsoid, or a parallelepiped. The object model 4200 may be fixed at either a central or non-central point within the three-dimensional space. Each of the individual object recognition templates 4300A / 4300B / 4300C / 4300D, etc., generated (e.g., via process 6007 as discussed further below) may correspond to one captured appearance 4140 of the object model 4200 from one of the multiple viewpoints 4120. Each object recognition template 4300 may include the appearance 4140 of the object model 4200 from a viewpoint 4120 that captures the pose of the object, i.e., the orientation and visible surfaces of the object, etc. In one embodiment, each of the multiple viewpoints 4120 may further correspond to a rotation angle of the virtual camera 4110 in the three-dimensional space 4100, i.e., a rotation angle of the camera from 1 to 360 degrees relative to its optical axis 4130.

[0100] The process 6003 may include a spatial subsampling procedure performed to select viewpoints 4120 whose corresponding object recognition templates 4300 are included in the object recognition template set 4301. The efficiency of the object recognition template generation method 6000 may be increased or maximized by reducing or otherwise optimizing the space (e.g., the number of viewpoints 4120 and appearances 4140) in which the object recognition templates 4300 are generated. In an embodiment, the excess viewpoints 4120 may be eliminated after the initial capture of the object appearances 4140 at those viewpoints 4120. For example, the excess viewpoints 4120 may be eliminated if they are determined to contain substantially similar information as other viewpoints 4120 (e.g., due to symmetry). In an embodiment, the excess viewpoints 4120 may be eliminated prior to the capture of the object appearances 4140 based on predetermined decisions regarding pose, spacing, etc., as discussed below. In an embodiment, the number of viewpoints 4120 selected and the spacing distance between adjacent viewpoints 4120 may depend on the number of object recognition templates 4300 required, based, for example, on the complexity and / or symmetry of the object model 4200 in question.

[0101] The multiple viewpoints 4120 may be selected or determined according to a number of different methods. For example, the at least one processing circuit 1110 may: Circle of longitude 4170 and Circle of latitude According to the intersection of 4180, the viewpoint 4120 may be determined. The viewpoint 4120 spans the surface 4101 of the three-dimensional space 4100. Circle of longitude 4170 and Circle of latitude In such a selection scheme, a high density of viewpoints 4120 may be clustered at or near the poles of the surface 4101, and a low density of viewpoints may be clustered at or near the intersections of the poles 4120 and the intersections of the poles 4180. Circle of longitude and Circle of latitude, further away from the poles (e.g., closer to the equator of the surface 4101). Such an uneven distribution of sample locations may cause the multiple object recognition templates 4300 to over-represent one range or set of ranges of relative pose / orientation between the virtual camera 4110 and the object model 4200 and under-represent another range or set of ranges. Such a selection may be advantageous in some scenarios with some object models 4200 and less advantageous in others.

[0102] In a further embodiment, the multiple viewpoints 4120 may be selected according to an even distribution across the surface 4101 surrounding the three-dimensional space 4100. An even distribution may refer to the viewpoints 4120 being distributed at equal distances from one another across the surface 4101. An even distribution may provide more consistent template generation than a non-uniform distribution and may be preferable for objects that lack symmetry.

[0103] In some embodiments, the multiple viewpoints 4120 may be selected to reduce the total number of viewpoints 4120 and / or to weight or bias the viewpoint distribution in favor of particular viewpoints.

[0104] In one embodiment, the plurality of viewpoints 4120 may be determined based on a range of predictable poses that are expected to be observed for the plurality of objects 5011 in a physical context. For example, in a container holding several tapered bottles, the bottles may be expected to be posed such that the wider or base end faces downwards. Thus, the distribution of viewpoints may be biased or weighted to have more viewpoints 4120 in the upper half of the surface 4101.

[0105] In another embodiment, the multiple viewpoints 4120 may be determined based on the symmetry (or lack thereof) of the object model 4200. Symmetry of the object model 4200 may be determined based on whether or not the appearance 4140 of the object model 4200 changes after a rotation of the object model 4200. 4200The appearance of substantially the same object may be determined based on whether the appearance of the object changes with degrees around the axis of rotation. For example, an object model 4200 that looks substantially the same after a 180 degree rotation has two-way symmetry. An object model 4200 that looks substantially the same after a 120 degree rotation has three-way symmetry. An object model 4200 that looks substantially the same after a 90 degree rotation has four-way symmetry. An object model 4200 that looks substantially the same after a 60 degree rotation has six-way symmetry. Other symmetries may be possible for different objects. The substantially same appearance may be determined according to a similarity threshold.

[0106] The object recognition template generation method 6000 may further include an operation 6005 including estimating or capturing a plurality of appearances 4140 of the object model 4200 at each of the plurality of viewpoints 4120. The estimation of the plurality of appearances 4140 may be performed at each viewpoint 4120 of the plurality of viewpoints 4120. Each appearance 4140 includes a pose or orientation of the object model 4200 as seen from the respective viewpoint 4120. Each of the object recognition templates 4300 corresponds to a respective viewpoint 4120 of the plurality of viewpoints 4120 and includes information representing an appearance 4140 of the object model 4200 from the respective viewpoint 4120. For example, the object recognition templates 4300 may correspond to or represent a respective appearance 4140 of the object model 4200 from a respective viewpoint 4120 corresponding to a virtual camera 4110 positioned directly above the object model (i.e., along the Y-axis of the three-dimensional plane). In another example, the object recognition template 4300 may correspond to a respective appearance 4140 of the object model from a respective viewpoint 4120 corresponding to a virtual camera 4110 positioned directly to the left of the object model 4200 (i.e., along the x-axis of a three-dimensional plane). In one embodiment, each of the object recognition templates 4300 of the object recognition template set 4301 may correspond to or represent a respective appearance 4140 of a plurality of appearances 4140 of the object model 4200 from a respective viewpoint 4120 of a plurality of viewpoints 4120 corresponding to a virtual camera 4110 positioned at a number of different positions and orientations (i.e., a number of positions in a three-dimensional plane) around the object model 4200. Thus, estimating the plurality of appearances 4140 may include determining or estimating how the object model 4200 looks when viewed at a particular orientation from a particular viewpoint. For example, viewpoints may include a direct downward look, upward look, left look, right look of the object model 4200, or any angle / position between the principal axes X, Y, and Z of and on the surface 4101 enclosing the three-dimensional space 4100. As discussed above, each viewpoint 4120 may also include a rotation angle of the virtual camera 4110 relative to the camera's optical axis 4130, which may range from 1 to 360 degrees.Thus, each camera position may correspond to a set of viewpoints 4120, and each viewpoint 4120 of the set of viewpoints may further correspond to a different rotation angle of the virtual camera 4110. For example, two distinct viewpoints 4120 of the set of viewpoints 4120 may be estimated or captured from the same angle / location between the principal axes X, Y, and Z of the surface 4101, but the rotation angle of the first viewpoint 4120 is rotated by 45° relative to the rotation angle of the second viewpoint 4120.

[0107] The object recognition template generation method 6000 may further include an operation 6007 in which a plurality of object recognition templates 4300 are generated based on the plurality of appearances 4140. Each of the plurality of object recognition templates 4300 corresponds to a respective one of the plurality of appearances 4140. Thus, the generated object recognition template 4300 may include information representing the object model 4200 at a particular pose with respect to the object model 4200, as well as at a particular angle and / or rotation of the virtual camera 4110. Thus, each of the plurality of object recognition templates 4300 may be different from others of the plurality of object recognition templates 4300 (although in some scenarios, two different object recognition templates 4300 may include substantially the same information due to symmetries of the object model 4200 that are not accounted for by the selection of the viewpoint 4120).

[0108] Each object recognition template 4300 may include 2D appearances 4302 and 3D appearances 4303 generated according to the respective captured or estimated appearances 4140. The 2D appearances 4302 may include rendered two-dimensional images, which may be rendered according to ray tracing and discontinuity detection techniques, for example. The 3D appearances 4303 may include rendered 3D point clouds, similar to the 3D image information 2700 described in connection with FIG. 2F.

[0109] In some embodiments, the 2D appearance 4302 and / or the 3D appearance 4303can be generated via ray tracing techniques. The ray tracing process can simulate various light rays from the eye of the virtual camera 4110 hitting the surface of the object model 4200. This can further determine the angle at which the light rays hit the surface of the object model 4200, the distance the light rays travel on the surface of the object model 4200, and / or the effects of diffuse reflection (wherein light rays deflected at multiple angles are so implemented) or specular reflection (wherein light rays deflected at a single angle are so implemented). The angle of the deflected light rays reflected from the surface of the object model 4200 can indicate a change in the angle of the surface normal of the object. Such a change in the angle of the surface normal of the object can occur at the edges of the object.

[0110] The total number of object recognition templates 4300 generated for an object model 4200 may range from approximately 100 templates to 3200 templates, with a larger number of templates correlating to the complexity of the object model 4200 for which the object recognition templates 4300 are generated. The cited numbers are common to some applications and some object types, but more or fewer templates may be used without departing from the scope of the present invention. For example, an object model 4200 that exhibits a substantially symmetrical appearance (e.g., a threaded nut) will generate a large number of redundant templates (i.e., matching templates) or substantially identical templates. Thus, such a simple object model 4200 may generate as few as 100 templates, or any number in the lower half of the range of 100 to 3200 templates. Conversely, an object model 4200 that lacks symmetry may require more object recognition templates 4300 to adequately represent the object model 4200 at a larger number of feasible angles.

[0111] The object recognition template generation method 6000 may further include an operation 6009 that includes communicating the plurality of object recognition templates 4300 to a robotic control system as an object recognition template set 4301. The object recognition template set 4301 may be communicated to a robotic control system, such as the computing system 1100, any other type of robotic control system, and / or any other system that may employ the object recognition templates 4300. In an embodiment, communicating the object recognition template set 4301 may include direct communication to a memory or other storage device via any suitable networking protocol and / or storage device for any time for later access by the robotic control system or other system that may employ the object recognition templates. Each of the plurality of object recognition templates 4300 in the object recognition template set 4301 represents a pose that the object model 4200 may have relative to the optical axis 4130 of the virtual camera 4110 when located at a particular viewpoint 4120. As previously described, the pose may include any position angle and rotation angle.

[0112] As described above, the object recognition template generation method 6000 of the present invention includes generating an object recognition template set 4301 from object enrollment data 5001. The object recognition template set 4301 can be used to identify one or more objects 5011 in a scene that can be grabbed, lifted, or interacted with during a physical process. The object enrollment data 5001 of an object model 4200 representing the object 5011 is acquired. A plurality of viewpoints 4120 of the object model 4200 in the three-dimensional space 4100 is determined. An appearance 4140 of the object model at each of the plurality of viewpoints 4120 is estimated or captured. A plurality of object recognition templates 4300 are generated according to the plurality of appearances 4140, each of the plurality of object recognition templates 4300 corresponding to a respective one of the plurality of appearances 4140. The plurality of object recognition templates 4300 are communicated to a robot control system as the object recognition template set 4301. Each of the plurality of object recognition templates 4300 represents a pose that the object model 4200 may have relative to the optical axis 4130 of the virtual camera 4110. Thus, each of the plurality of object recognition templates 4300 may correspond to a potential pose of the object 5011 in the physical scene relative to the optical axis of a camera (such as camera 1200) that generates image information (e.g., image information 2600 / 2700) of the object 5011 in the physical scene.

[0113] In further embodiments, additional or alternative methods may be used to generate object recognition template set 4301 from object enrollment data 5001, and object recognition template 4300 may include information in addition to or different from 2D appearance 4302 and 3D appearance 4303. Specifically, object recognition template 4300 may include two-dimensional (2D) measurement information 4304 and three-dimensional (3D) measurement information 4305.

[0114] The 2D measurement information 4304 may refer to a gradient feature map. The gradient feature map may include gradient information 9100 captured or extracted from the digital representation of the object at one or more gradient extraction locations 5100 on the surface of the digital object, as described below. The 3D measurement information 4305 may refer to a surface normal feature map. The surface normal feature map may include surface normal vectors 9101 captured or extracted from the digital representation of the object at one or more surface normal locations 5101 on the surface of the digital object, as described below. The generation and / or extraction of the 2D measurement information 4304 and 3D measurement information 4305 is described in more detail below with respect to Figures 8-9C.

[0115] FIG. 8 illustrates a flow diagram of an exemplary feature generation method 8000. In an embodiment, the feature generation method 8000 may be used to generate an object recognition template set and / or a plurality of object recognition templates. In a further embodiment, the feature generation method 8000 may be used to extract features from object image information in hypothesis generation, refinement, and validation methods, as discussed in more detail below. In an embodiment, the feature generation method 8000 may be performed, for example, by the computing system 1100 (or 1100A / 1100B / 1100C) of FIGS. 2A-2D or the computing system 1100 of FIGS. 3A-3B, or more specifically, by at least one processing circuit 1110 of the computing system 1100. In some scenarios, the computing system 1100 may perform the feature generation method 8000 by executing instructions stored on a non-transitory computer-readable medium (e.g., 1120). 2D, which may perform the feature generation method 8000. For example, in an embodiment, the steps of the feature generation method 8000 may be performed by the object registration module 1130, the object recognition module 1121, the feature extraction module 1134, and the template generation module 1132 operating together.

[0116] In an embodiment, the steps of feature generation method 8000 may be utilized to effect object recognition template generation, for example through feature generation and / or extraction methods, which may be subsequently used in conjunction with a particular continuous robot trajectory for performing a particular task. In an embodiment, the steps of feature generation method 8000 may be applied to extract or generate features from object image information for use in hypothesis generation, refinement, and validation. As a general overview, feature generation method 8000 may be processed to cause computing system 1100 to generate an object recognition template, feature map, and / or a set of extracted / generated features for use in identifying an object in a scene for processing related to lifting an object. Feature generation method 8000 is described below with further reference to Figures 7A and 7B and Figures 9A-9C.

[0117] The feature generation method 8000 may include 2D measurement information 4304 and 3D measurement information 4305 that may be used to generate an object recognition template 4300 and / or characterize an object 5012 (see, e.g., FIG. 3B) in a physical scene 5013. The at least one processing circuit 1110 may obtain object information 9121. As shown in FIG. 9A, the object information 9121 may include a digitally represented object 9200, e.g., object enrollment data 5001 of an object model 4200, an appearance 4140 of the object model 4200, an object recognition template 4300, and / or scene information 9131. The scene information 9131 may include captured 2D or 3D image information of a physical scene 5013 including a plurality of objects 5012, e.g., similar to the 2D image information 2600 and / or the 3D image information 2700. The scene information 9131 may also include image information 12001, which is discussed below with respect to hypothesis generation, validation, and refinement methods and processes. The at least one processing circuit 1110 may further extract or generate 2D measurement information 4304 and / or 3D measurement information 4305 from the object information 9121. In an embodiment, the at least one processing circuit 1110 may further generate an object recognition template 4300 according to the 2D measurement information 4304 and the 3D measurement information 4305. In an embodiment, the 2D measurement information 4304 and the 3D measurement information 4305 may be used or utilized for alternative purposes, such as hypothesis generation, validation, and refinement. The at least one processing circuit 1110 may perform certain steps of the feature generation method 8000 for generating the object recognition template set 4301 and / or for use in hypothesis refinement and validation.

[0118] In process 8001, feature generation method 8000 may include obtaining object information 9121. Object information 9121 may include digitally represented object 9200. Object information 9121 and digitally represented object 9200 may represent objects 5015 that physically exist in the world. Object 5015 may include, for example, object 5011 (e.g., a physical object represented by object model 4200) and / or object 5012 (e.g., a physical object represented by captured image information of physical scene 5013). In an embodiment, object information 9121 may include one or more of object recognition template 4300, object enrollment data 5001, object appearance 4140, and / or scene information 9131. The at least one processing circuit 1110 may obtain the object information 9121 from within a hardware storage component (i.e., HDD, SSD, USB, CD, RAID, etc.) or software storage component (i.e., cloud, VSP, etc.) of the computing system 1100. The at least one processing circuit 1110 may obtain the object information 9121 as part of internal processing, for example, as an object recognition template 4300. The at least one processing circuit 1110 may obtain the object information 9121 from a camera 1200 associated with the computing system 1100. The at least one processing circuit 1110 may obtain the object information 9121 for an object from an external processor (i.e., an external laptop, desktop, mobile phone, or any other separate device with its own processing system) or from an external storage device.

[0119] In operation 8003, the feature generation method 8000 may further include selecting feature locations including gradient extraction locations 5100 (shown in FIG. 9B ) and surface normal locations 5101 (shown in FIG. 9C ). The gradient extraction locations 5100 are locations selected for extraction or generation of 2D measurement information 4304. The surface normal locations 5101 are locations selected for extraction or generation of 3D measurement information 4305. The gradient extraction locations 5100 and surface normal locations 5101 are each locations on a surface 9122 of the digitally represented object 9200.

[0120] In an embodiment, the gradient extraction locations 5100 and the surface normal locations 5101 may correspond to one another. In an embodiment, some gradient extraction locations 5100 may correspond to some surface normal locations 5101, while other gradient extraction locations 5100 do not correspond to surface normal locations 5101. In a further embodiment, the gradient extraction locations 5100 and the surface normal locations 5101 may be selected such that they do not overlap one another. Thus, the gradient extraction locations 5100 and the surface normal locations 5101 may have any amount of overlap, including complete overlap and no overlap.

[0121] In an embodiment, the locations of the gradient extraction locations 5100 and the surface normal locations 5101 on the surface 9122 of the digitally represented object 9200 may be selected as a limited set to limit the amount of memory required to store the extracted or generated 2D measurement information 4304 and 3D measurement information 4305. This memory saving practice may be referred to as a linear modality, which may refer to a fixed number of total features (such as gradient information 9100 and / or surface normal vectors 9101, as described below) extracted and / or analyzed, regardless of the size (in bytes) of the object information 9121 of the digitally represented object 9200. The number of features captured for the 2D measurement information 4304 may be the same as or different from the number of features captured for the 3D measurement information 4305.

[0122] In an embodiment, a limited number of gradient extraction locations 5100 and surface normal locations 5101 may be located to produce efficient results. For example, the gradient extraction locations 5100 may be located along an identified edge of the digitally represented object 9200, as shown in FIG. 9B, while the surface normal locations 5101 may be located away from the edge of the digitally represented object 9200. In an embodiment, the edge of the digitally represented object 9200 may be identified, for example, according to ray tracing, pixel intensity discontinuities, or other analysis techniques. This may prove efficient because, as described below, the gradient information 9100 may be more meaningful in hypothesis generation, validation, and refinement when captured near the object edge, while the surface normal vectors 9101 may be more meaningful when captured away from the edge. The number of gradient extraction locations 5100 and surface normal locations 5101 selected may range from 100-1000, 50-5000, and / or 10-1000, although more or less may be appropriate as well. In certain embodiments, the number of gradient extraction locations 5100 and surface normal locations 5101 may be 256 each or 256 in total.

[0123] At operation 8005, the feature generation method 8000 may further include extracting 2D measurement information 4304 from the object information 9121. The 2D measurement information 4304 may represent a smaller set of information (e.g., compared to the 2D appearance 4302) to represent the object 5015 in order to conserve memory or other resources and / or to improve the speed at which the object recognition template set 4301 is generated or hypothesis testing and refinement is performed. As described above, the object recognition template 4300 from the object recognition template set 4301 may include 2D measurement information 4304 (and / or 3D measurement information 4305) that describes the object 5015.

[0124] The 2D measurement information 4304 may include two-dimensional features extracted or generated from the object information 9121. In one embodiment, extracting or generating the 2D measurement information 4304 may include extracting gradient information 9100 from the object information 9121. Thus, the 2D measurement information 4304 may include a gradient feature map including gradient information 9100 as described herein. The gradient information 9100 indicates the direction or orientation of the edges 5110 of the digitally represented object 9200. The gradient information 9100 may be extracted at multiple gradient extraction locations 5100 of the digitally represented object 9200. The gradient extraction locations 5100 may represent any or all of the internal and external edges identified within the digitally represented object 9200.

[0125] Extracting gradient information 9100 may include analyzing pixel intensities of the two-dimensional image information of the object information 9121 to determine the direction (e.g., represented by arrows 9150) along which pixel intensities of the two-dimensional image information at each gradient extraction location change in a process called gradient extraction. The changes in pixel intensities may represent the contours and orientation of the surfaces and edges of the digitally represented object 9200, and thus provide information that may be useful in comparing two digitally represented objects 9200. Locations near each other along an edge 5110 are likely to have similar gradient information 9100, e.g., pixel intensities near such adjacent locations change in a similar manner with increasing distance from the edge 5110. In some embodiments, portions of the digitally represented object 9200 that exhibit higher than average pixel intensities may indicate an edge 5110 or other identifiable feature. As described above, in some embodiments, the gradient extraction locations 5100 may be located along an edge 5110 of the digitally represented object 9200.

[0126] In one embodiment, the extracted gradient information 9100 may be used to improve a template matching process, hypothesis generation, or hypothesis testing process, which may determine whether an object recognition template 4300 from the object recognition template set 4301 matches an object 5012 in a scene. For example, if the 2D appearance 4302 has a particular portion that overlaps or intersects with a digitally represented object 9200 from the scene, the at least one processing circuit 1110 may determine whether the matching portion also matches or exhibits a similar gradient (e.g., whether a portion of the 2D measurement information 4304 matches). If the gradients are different or do not match, the at least one processing circuit 1110 may determine that the different particular portion is the result of a mismatch or is coincidental. The mismatch may be the result of the 2D appearance 4302 overlapping a portion of the scene by only a small amount.

[0127] For example, referring now to FIG. 9D, the 2D appearance 4302A of the object recognition template 4300A is represented by a rectangle and the 2D measurement information 4304A (e.g., gradient information 9100) of the object recognition template 4300A is represented by an arrow. The 2D image information 2600B of the object 5012 in the scene (physical object not shown) is represented by an L-shaped solid. The object 5012 is further represented by the 2D measurement information 4304B, which is represented by an arrow. A portion of the 2D appearance 4302A may be compared and overlaid with the patterned portion 9309 of the 2D image information 2600B (physical object not shown) representing the object 5012 in the scene. However, the gradients represented by the 2D measurement information 4304A and 4304B do not match and therefore the object recognition template 4300A may be determined by the at least one processing circuit 1110 to be a mismatch for the object 5012 in the scene 5013.

[0128] In an operation 8007, the feature generation method 8000 may further include extracting or generating 3D measurement information 4305 from the object information 9121. Referring now to Figure 9C, the operation 8007 may include determining a surface normal vector 9101 at the surface normal location 5101. The 3D measurement information 4305 may include a surface normal feature map that includes the extracted or generated surface normal vector 9101.

[0129] The extracted 3D measurement information 4305 may include surface normal vector information, e.g. measurements describing surface normal vectors 9101, which may be normal vectors (vectors perpendicular to a surface) taken at surface normal positions 5101 on a surface 9122 of the digitally represented object 9200. In one embodiment, extracting or generating the 3D measurement information 4305 includes extracting or generating surface normal vectors 9101 and / or surface normal vector information from the object information 9121. The surface normal vectors 9101 describe multiple vectors of normals to the surface 9122 of the digitally represented object 9200. The surface normal vectors 9101 may be extracted or generated at multiple surface normal positions 5101 of the digitally represented object 9200. Extracting the surface normal vectors 9101 may include identifying multiple surface normal vectors 9101 of the digitally represented object 9200 at each of the surface normal vector positions 5101.

[0130] At operation 8009, the feature generation method 8000 may include generating an object recognition template set 4301 or a plurality of object recognition templates 4300. At least one processing circuit 1110 may generate one or more object recognition templates 4300, including the 2D measurement information 4304 and the 3D measurement information 4305 described above. The one or more object recognition templates 4300 may form the object recognition template set 4301. As described above, the object recognition template 4300 may include one or more of the 2D measurement information 4304, the 3D measurement information 4305, the 2D appearance 4302, and the 3D appearance 4303. Thus, in some embodiments, the feature generation method 8000 may augment or further develop previously established object recognition templates 4300 and object recognition template set 4301. The extracted or generated 3D measurement information 4305 and 2D measurement information 4304 may be used to identify objects 5012 in a scene during a real-time or near real-time lifting process, as discussed below. The feature generation method 8000 may operate or process in parallel with or subsequent to the object recognition template generation method 6000 described above in generating an object recognition template set 4301 for later performing matching (hypothesis refinement and validation) processes against the scene (or objects in the scene). The feature generation method 8000 may serve as a final step towards creating an object recognition template set 4301 for use in subsequent hypothesis processes (such as, for example, methods 11000 and 13000, described in more detail below).

[0131] 10A and 10B illustrate aspects of a template matching and hypothesis generation method 10000 consistent with embodiments herein. The hypothesis generation techniques discussed herein may be generally consistent with the lineMod technique.

[0132] In process 10001, the template matching and hypothesis generation method 10000 may include acquiring image information. In one embodiment, acquiring image information 12001 may include capturing an image of a scene 5013 and one or more objects 5012 within the scene. In such an instance, the image information 12001 may represent the object 5012 located within a box, bin, case, crate, pallet, or other container. The image information 12001 may be acquired by a camera 1200 as discussed herein.

[0133] At least one processing circuit 1110 may be configured to generate, receive, and / or process image information 12001, such as by using image information 12001 to distinguish between individual objects in the field of view of camera 1200 and performing object recognition based on image information 12001. In one embodiment, image information 12001 may include two-dimensional image information (e.g., similar to 2D image information 2600) describing the visual appearance of an environment or scene 5013 in the field of view of camera 1200. In one embodiment, image information 12001 may include three-dimensional image information (e.g., similar to 3D image information 2700) providing a point cloud, spatial structure information, depth map, or other three-dimensional image of the scene 5013 in the field of view of camera 1200. The three-dimensional image information in this example may be used to estimate how object 5012 is spatially positioned in three-dimensional space (e.g., scene 5013). Acquiring image information 12001 may include generating or acquiring image information 12001 representing the scene 5013, and optionally may include generating or acquiring object image information 12002 representing an individual object 5012 or multiple objects 5012 within the scene 5013. Image information 12001 may be generated by camera 1200 when object 5012 is (or was) in the field of view of camera 1200, and may include, for example, two-dimensional image information and / or three-dimensional image information.

[0134] In one embodiment, the image information 12001 may include a two-dimensional grayscale or color image and may describe the appearance of the scene 5013 (and / or the object 5012 in the scene) from the viewpoint of the camera 1200. In one embodiment, the image information 12001 may correspond to a single color channel (e.g., red, green, or blue channel) of a color image. If the camera 1200 is positioned above the object 5012, the two-dimensional image information may represent the appearance of the top surface of each of the objects 5012. Furthermore, the image information 12001 may include three-dimensional image information, which may include, for example, a depth map or point cloud indicating respective depth values ​​of various object positions 6220 on one or more surfaces (e.g., top surface, or other outer surface) of the object 5012 or along one or more edges of the object 5012. The two-dimensional image information and the three-dimensional image information of the object image information 12002 may be referred to as 2D image information 12600 and 3D image information 12700, respectively. In some embodiments, the object positions 6220 representing the physical edges of the objects 5012 may be used to identify object image information 12002 that is limited to representing individual objects 5012 .

[0135] The object image information 12002 may include image information relating to a particular physical object 5012 in the scene 5013. The object image information 12002 may include 2D image information 12600 representing the object 5012, similar to image information 2600. The object image information 12002 may include 3D image information 12700 representing the object 5012, similar to image information 2700. The object image information 12002 may include object positions 6220, which may further include gradient extraction positions 8100 and surface normal positions 8101 representing positions from which respective gradient information 8102 and surface normal vectors 8103 are obtained, for example via the feature generation method 8000. The gradient extraction location 8100, surface normal location 8101, gradient information 8102, and surface normal vector 8103 may be similar to the gradient extraction location 5100, surface normal location 5101, gradient information 9100, and surface normal vector 9101, except that, as described above, the gradient extraction location 8100, surface normal location 8101, gradient information 8102, and surface normal vector 8103 are obtained from image information obtained from a physical object.

[0136] The template matching and hypothesis generation processes discussed below may be performed by comparing object recognition templates to image information 12001 and / or object image information 12002. In an embodiment, object image information 12002 may be generated from image information 12001, for example, based on image segmentation or other techniques, as well as feature generation method 8000, as described above.

[0137] In process 10003, the template matching and hypothesis generation method 10000 may include matching a template to object image information. The type of object 5012 present in the scene 5013 (whether it is a single type or multiple types) may be known. Thus, the object recognition template set 4301 corresponding to the known object type may be obtained, for example, via any method described herein. Information of each object recognition template 4300 of the object recognition template set 4301 representing information about how the object 5012 should look in various poses may be compared with the object image information 12002 representing the object 5012 to determine whether each object recognition template 4300 is a candidate for a match. Good candidates for a match may then be selected for generation of detection hypotheses.

[0138] Any relevant information of the object recognition template 4300 may be compared to corresponding information of the object image information 12002. For example, gradient information 8102 and gradient extraction location 8100 of the object image information 12002 may be compared to gradient information 9100 and gradient extraction location 5100 of the object recognition template 4300. Surface normal vector 8103 and surface normal location 8101 of the object image information 12002 may be compared to surface normal vector 9101 and surface normal location 5101 of the object recognition template 4300. The 2D information 12600 and 3D information 12700 may be compared to the 2D appearance 4302 and 3D appearance 4303, respectively.

[0139] The above information from the object recognition template 4300 and from the object image information 12002 may be understood as maps in that the information may be attributed to a series of two-dimensional positions. The template map (representing any of the information in the object recognition template 4300) may be slid laterally against the object map (representing any of the object image information 12002) until a match is found that exceeds a threshold. Template matching may involve comparison of the respective gradient information, the respective 2D image information, the respective 3D information, and / or the respective surface normal vector information.

[0140] A threshold may be used and tolerance may be allowed to account for potential changes in pose between the object recognition template 4300 and the object image information 12002. It may be understood and accounted for that the spatial subsampling procedure described above does not capture all possible poses in the object recognition template 4300 and therefore some variation is tolerated. Such tolerance techniques may include, for example, spreading, whereby the gradient information 9100 spreads between adjacent gradient extraction locations 5100 of the object recognition template 4300 to increase the likelihood of a match. Another tolerance technique may include finding a match based on a threshold level of match, for example, where the gradient information or surface normal vectors are close to each other but do not match perfectly. Template matching may generate a template matching score to indicate the quality of the match.

[0141] In process 10005, the template matching and hypothesis generation method 10000 may include clustering and grouping matching templates to reduce the total number of matches. The template matching process may find multiple object recognition templates 4300 that match the object represented by the object image information 12002. In some embodiments, the template matching process may be limited in terms of how many matches can be identified by time or computational resources. In such a situation, the process 10005 may avoid concentrating matches on a single part or set of parts in the scene 5013. Thus, matching templates with good quality matches (e.g., above a threshold) may be clustered, grouped, and filtered to maintain good scene coverage. Object recognition templates 4300 identified as corresponding to the same object image information 12002 may be clustered or grouped. Within each cluster or group, the best match may be selected and the rest may be removed. Thus, the remaining matches may represent the object 5012 throughout the scene 5013, rather than clustering in a single region. In one embodiment, if an object 5012 in the scene 5013 is near the top of a container and is very easily recognizable, it may generate more matches than an object that is partially occluded. By selecting only the best match for each object image information 12002, more objects can be identified.

[0142] In process 10007, the template matching and hypothesis generation method 10000 may include generating a set of one or more detection hypotheses. The object recognition templates 4300 remaining after clustering and grouping may be selected as detection hypotheses. These object recognition templates 4300 may be stored with pose information 6301 indicating information about where each object recognition template 4300 should be located in the scene 5013 to match the corresponding object image information. The pose information 6301 may further include information associating each object recognition template 4300 with the corresponding object image information 12002. The detection hypotheses 6300 may be combined in groups and / or sets. For example, a detection hypothesis set 8309 may include multiple detection hypotheses 8300 associated with object image information 12002 representing a single object 5012, while a group of detection hypotheses 8300 may include multiple detection hypotheses 8300 associated with object image information 12002 representing multiple different objects 5012 in the scene 5013.

[0143] FIG. 11 illustrates a flow diagram of an exemplary hypothesis refinement method 11000 for refining a detection hypothesis. In an embodiment, the hypothesis refinement method 11000 may be performed, for example, by the computing system 1100 (i.e., 1100A / 1100B / 1100C) of FIGS. 2A-2D or the computing system 1100 of FIGS. 3A-3B, or more specifically, by at least one processing circuit 1110 of the computing system 1100. In some scenarios, the computing system 1100 may perform the hypothesis refinement method 11000 by executing instructions stored on a non-transitory computer-readable medium (e.g., 1120). For example, the instructions may cause the computing system 1100 to execute one or more of the modules illustrated in FIG. 2D, which may perform the method 11000. For example, in an embodiment, the steps of the method 11000 may be performed by the hypothesis generation module 1128 and the hypothesis refinement module 1136 operating together.

[0144] The hypothesis refinement method 11000 may be used to refine one or more detection hypotheses 6300 (e.g., as described above) generated to identify an object 5012 physically located within a scene 5013. The hypothesis refinement method 11000 may operate on image information 12001 obtained from the scene 5013. The image information 12001 may be similar to the 2D image information 2600 and the 3D image information 2700. Within the image information 12001 there may be one or more object image information 12002 representing an object 5012 within the scene 5013. Identifying the object 5012 may include identifying the type of object or identifying the dimensions of the object from the corresponding object image information 12002 and / or matching the object image information 12002 to an object recognition template 4300. Thus, the detection hypotheses 6300 may be hypotheses regarding which of one or more object recognition templates 4300 may match object image information 12002 of image information 12001 representing scene 5013. The object image information 12002 may include 2D image information 12600 representing object 5012. The 2D image information 12600 may be similar to image information 2600 and / or may include rendered 2D image information generated according to rendering techniques such as ray tracing and discontinuity detection. The object image information 12002 may include 3D image information 12700 representing object 5012, similar to image information 2700. The detection hypotheses 6300 may be generated according to a template matching procedure, as described above. For example, in one embodiment, the detection hypotheses 6300 may be generated via the lineMod algorithm and / or procedure described above. The hypothesis refinement method 11000 may proceed to refine the match between the object recognition template 4300 and the object image information 12002 even in scenarios where the object recognition template 4300 does not match the object image information 12002 exactly.

[0145] In the hypothesis refinement method 11000, at least one processing circuit 1110 may be in communication with a robot 3300 having a robot arm 3320 and an end effector device 3330 connected to the arm, and a camera 1200 having a field of view and configured to execute instructions stored on a non-transitory computer readable medium when one or more objects 5012 are or were within the field of view. In an embodiment, the at least one processing circuit 1110 may not communicate directly with the robot 3300, but may send and receive information to and from the robot 3300 via a network and / or storage device. In an embodiment, the at least one processing circuit 1110 may communicate directly with the robot 3300. The at least one processing circuit 1110 may obtain image information 12001 of one or more objects 5012 in the scene 5013. The at least one processing circuit 1110 may also obtain detection hypotheses 6300. The detection hypotheses 6300 may include information associating the object recognition template 4300 (e.g., a corresponding object recognition template 4300B selected from the plurality of object recognition templates 4300) with the object image information 12002, and may include pose information 6301 of the object 5012 represented by the object image information 12002. The pose information 6301 of the object 5012 may refer to the position and orientation of the object 5012. In an embodiment, the detection hypotheses 6300 may include the corresponding object recognition template 4300B or may include a reference to the corresponding object recognition template 4300B. The at least one processing circuit 1110 may process to identify a mismatch between the corresponding object recognition template 4300B and the corresponding object image information 12002. The at least one processing circuit 1110 may process to identify a set of template positions 6210 of the corresponding object recognition template 4300B that correspond to a set of object positions 6220 of the object image information 12002. The at least one processing circuit 1110 may further process the set of template positions 6210 to converge to the set of object positions 6220.At least one processing circuit 1110 may process to generate an adjusted detection hypothesis 6300′ or multiple iteratively adjusted detection hypotheses 6300′ including an object recognition template adjusted according to a set of adjusted template positions 6210.

[0146] At least one processing circuit 1110 may perform certain steps of a hypothesis refinement method 11000 to refine a detection hypothesis 6300. In process 11001, the hypothesis refinement method 11000 may include acquiring image information 12001 of one or more objects 5012 in a scene 5013. In one embodiment, acquiring image information 12001 may include capturing an image of the scene 5013. In such an instance, the image information 12001 may represent an object 5012 located within a box, bin, case, crate, pallet, or other container. The image information 12001 may be acquired by a camera 1200 as discussed herein.

[0147] At least one processing circuit 1110 may be configured to generate, receive, and / or process image information 12001, such as using image information 12001 to distinguish between individual objects in the field of view of camera 1200 and perform object recognition or object registration based on image information 12001. In one embodiment, image information 12001 may include two-dimensional image information (e.g., similar to 2D image information 2600) describing the visual appearance of an environment or scene 5013 in the field of view of camera 1200. In one embodiment, image information 12001 may include three-dimensional image information (e.g., similar to 3D image information 2700) providing a point cloud, spatial structure information, depth map, or other three-dimensional image of scene 5013 in the field of view of camera 1200. The three-dimensional image information in this example may be used to estimate how object 5012 is spatially positioned in three-dimensional space (e.g., scene 5013). With respect to process 11001, obtaining image information 12001 may include generating or obtaining image information 12001 representing the scene 5013, and optionally generating or obtaining one or more object image information 12002 representing an individual object 5012 or multiple objects 5012 within the scene 5013. Image information 12001 may be generated by camera 1200 when object 5012 is (or was) in the field of view of camera 1200, and may include, for example, two-dimensional image information and / or three-dimensional image information.

[0148] In one embodiment, image information 12001 may include a two-dimensional grayscale or color image and may describe the appearance of the scene 5013 (and / or objects 5012 within the scene) from the perspective of camera 1200. In one embodiment, image information 12001 may correspond to a single color channel (e.g., a red, green, or blue channel) of a color image. When camera 1200 is positioned above objects 5012, the two-dimensional image information may represent the appearance of the top surfaces of each of objects 5012.

[0149] The object image information 12002 may include image information relating to a particular physical object 5012 in the scene 5013. The object image information 12002 may include 2D image information 12600 representing the object 5012, similar to image information 2600. The object image information 12002 may include 3D image information 12700 representing the object 5012, similar to image information 2700. The object image information 12002 may include object positions 6220, which may further include gradient extraction positions 8100 and surface normal positions 8101 representing positions from which respective gradient information 8102 and surface normal vectors 8103 are obtained, for example via the feature generation method 8000. The gradient extraction location 8100, surface normal location 8101, gradient information 8102, and surface normal vector 8103 may be similar to the gradient extraction location 5100, surface normal location 5101, gradient information 9100, and surface normal vector 9101, except that, as described above, the gradient extraction location 8100, surface normal location 8101, gradient information 8102, and surface normal vector 8103 are obtained from image information obtained from a physical object.

[0150] In operation 11003, the hypothesis refinement method 11000 may further include obtaining a detection hypothesis 6300. The detection hypothesis 6300 may include a number of pieces of information. For example, the detection hypothesis 6300 may include a corresponding object recognition template 4300B and object pose information 6301 indicating a position and orientation of the corresponding object recognition template 4300B required to overlay the corresponding object image information 12002 in the image information 12001. The corresponding object recognition template 4300B may include one or more of a 2D appearance 4302B, a 3D appearance 4303B, a 2D measurement information 4304B, and a 3D measurement information 4305B. As described above, the 2D measurement information 4304B may include gradient information 9100B and gradient extraction position 5100B, while the 3D measurement information 4305B may include a surface normal vector 9101B and a surface normal position 5101B. The corresponding object recognition template 4300B may further include template locations 6210, which may include gradient extraction locations 5100B and surface normal locations 5101B, or a subset thereof.

[0151] In operation 11005, the hypothesis refinement method 11000 may further include identifying a mismatch between the corresponding object recognition template 4300B and the object image information 12002 to which the template was matched according to the detection hypothesis 6300. The two-dimensional information of the corresponding object recognition template 4300B (e.g., 2D appearance 4302B) may be compared to the object image information 12002 to identify the mismatch. The mismatch may be identified or quantified according to areas of non-alignment or other inconsistency between the 2D appearance 4302B and the object image information 12002.

[0152] Upon identification of a mismatch or inconsistency between the corresponding object recognition templates 4300B, the two-dimensional information of the corresponding object recognition templates 4300B (e.g., 2D appearance 4302B) may be transformed from two-dimensional space to three-dimensional space for comparison and alignment with the object image information 12002. In some instances, the 3D appearance 4303B or the 3D transformation of the 2D appearance 4302B may be used for comparison with the object image information 12002 to identify the mismatch. In some embodiments, the mismatch may be identified or quantified according to the mismatch between the object location 6220 and the template location 6210. The object location 6220 represents a point on a digital representation of the object 5012 (e.g., object image information 12002), while the template location 6210 represents a point on the template object 6290 (as discussed below).

[0153] The transformation from two-dimensional space to three-dimensional space may be based on calibration parameters or other parameters of the camera 1200 or other image sensor, which may be determined during a camera calibration process or may be predefined. As described above, the corresponding object recognition template 4300B may be derived from the object enrollment data 5001 and have a coordinate system associated with it. In the transformation to three-dimensional space, the coordinate system of the corresponding object recognition template 4300B may be mapped to the coordinate system of the scene 5013 as captured in the image information 12001. Thus, the transformation may utilize calibration parameters or other parameters of the camera 1200 that captured the image information 12001. The information of the detection hypothesis 6300 may define a digital representation of the object, referred to herein as the template object 6290. The three-dimensional transformation may be referred to as the template object 6290 and may represent the information of the detection hypothesis 6300 in three-dimensional space in the coordinate system of the image information 12001 for comparison with the object image information 12002.

[0154] In operation 11007, the hypothesis refinement method 11000 generates a set of corresponding object positions on the corresponding object. recognition The method may further include identifying a set of template locations within the template. The object locations 6220 represent points on a digital representation of the object 5012 (e.g., object image information 12002), while the template locations 6210 represent points on the template object 6290. Thus, aligning the object locations 6220 with the template locations 6210 may help refine the detection hypotheses 6300.

[0155] As discussed above, the template positions 6210 may correspond to the gradient extraction positions 5100B and the surface normal positions 5101B, or a subset thereof. In further embodiments, the template positions 6210 may include additional or different positions used for alignment with the object positions 6220 of the object image information 12002.

[0156] The template positions 6210 (and corresponding object positions 6220) may be selected according to positions that have a high impact on hypothesis refinement (e.g., alignment between the object image information 12002 and the template object 6290). In some instances, the template positions 6210 and object positions 6220 may be selected as positions around the edges of the respective template object 6290 and object image information 12002. Such positions may be more useful for performing hypothesis refinement as they are less susceptible to noise and may provide an outline of the object's shape.

[0157] At operation 11009, the hypothesis refinement method 11000 may further include adjusting the set of template locations 6210 to converge to the set of object locations 6220. The at least one processing circuit 1110 may be further configured to adjust the set of template locations 6210. If a mismatch is identified, the at least one processing circuit 1110 may make the adjustment to improve an alignment value between the template locations 6210 of the template object 6290 to the corresponding object locations 6220 of the object image information 12002.

[0158] The alignment procedure may be performed using an iterative closest point (ICP) technique, shown in Figures 12B and 12C. The ICP technique may include adjusting the template positions 6210 to converge to a set of object positions 6220. A set of vectors 6215 between the template positions 6210 and their corresponding object positions 6220 may be determined. Each vector 6215 may represent a direction and a magnitude. In one embodiment, the vector direction and magnitude may be used to adjust the template positions 6210 to converge to the object positions 6220. The vectors 6215 extending from the template positions 6210 to the object positions 6220 have a direction and a magnitude. If the set of vectors 6215 have a vector direction and magnitude and are mathematically understood as forces acting on the template object 6290 at the template positions 6210, the template object 6290 may be adjusted or moved according to the direction and magnitude of the vectors 6215 applied or acting at the respective template positions 6210. Thus, vectors 6215 representing template positions 6210 and object positions 6220 that have a larger magnitude and are further apart (e.g., have a larger delta or offset) may be understood to apply a larger "force" to the template adjustment. For example, referring to FIG. 12B, the template object 6290 may overlay the object image information 12002. The vectors 6215 extend between the template position 6210 and the object position 6220. When the vectors 6215 are collectively applied as a "force" to the template object 6290 based on their direction and magnitude, the template object 6290 (shown in FIG. 12B) tends to rotate clockwise and come into closer alignment with the object image information 12002. After applying the vectors 6215, a new set of vectors 6215 may be generated and iteratively applied. In another example, as shown in FIG. 12C, applying the vectors 6215 may cause a translational movement of the template object 6290 resulting in alignment with the object image information 12002.In some embodiments, through iterative generation and application of vectors 6215, the template object 6290 is moved into better alignment with the object image information 12002 until the remaining vectors 6215 cancel each other out and no further movements can be generated. If no further movements can be generated, the alignment quality can be evaluated. In some embodiments, the iterative adjustments may be made until the alignment quality exceeds a threshold.

[0159] The quality of the alignment (or the level of misalignment) may be evaluated or determined in a number of different ways. For example, the quality of the alignment may be evaluated or determined according to the level of misalignment defined by the direction and magnitude of the vector 6215. The quality of the alignment may also be evaluated or determined according to a distance measurement between the new, updated, or adjusted set of template positions 6210 and the set of object positions 6220. The quality of the alignment may also be evaluated or determined according to a convergence rate. In an embodiment, any combination of these alignment quality measurements may be used.

[0160] The quality of the alignment may be determined based on the level of misalignment defined by the direction and magnitude of each new or updated vector 6215. As mentioned above, the vector 6215 may be mathematically interpreted as a force acting on the template object 6290 according to its direction and magnitude. When at rest and subjected to a force, the object will experience stress. In an embodiment, the level of misalignment may be calculated according to mathematically treating the vector 6215 as a force that generates internal stress in the template object 6290. Thus, for example, an equal vector and an opposite vector 6215 do not cancel each other (as would be the case if the vectors 6215 were simply added together), but generate "stress" in the template object 6290. If the level of alignment quality is good (and the level of misalignment is low), the vector 6215 will be of relatively small magnitude, thereby corresponding to low internal stress. If the alignment quality is poor (and the level of misalignment is high), the vector 6215 will be large, thereby corresponding to more significant internal stress. This internal stress calculation can be taken as an indication of the alignment quality.

[0161] In one embodiment, the quality of the alignment may be determined based on a distance measure between the new or updated set of template positions 6210 and the set of object positions 6220. The distance measure may be a Euclidean distance measure, or the length of a line segment between two points in Euclidean space. The Euclidean distance (or Pythagorean distance) may be expressed via the following formula:

[0162]

number

[0163] d=distance

[0164] p = the first point with 3D coordinates p1, p2, p3

[0165] q = second point with 3D coordinates q1, q2, q3

[0166] The distance measurements generated via the above equations output a distance value (typically equal to or greater than zero), with output values ​​closer to zero representing closer distances between points p, q (zero representing no distance or representing identical / overlapping points). The distance measurements between each of the new or updated set of template positions 6210 and the set of object positions 6220 may be combined, for example by taking the arithmetic or geometric mean. The combined distance values ​​may then be compared to a predefined threshold, with distance values ​​equal to or below the predefined threshold (i.e., zero and the predefined threshold) indicating a good match (i.e., between the template object 6290 and the object image information 12002) and distance output values ​​greater than the predefined threshold indicating a mismatch.

[0167] In one embodiment, the distance measure may be the cosine distance between the surface normal vector associated with the new set of template positions 6210 (i.e., template vector 6260) and the set of object positions 6220 (i.e., object vector 6270). The template vector 6260 may include some or all of the previously determined surface normal vector 9101 associated with the corresponding object recognition template 4300B. The object vector 6270 may include the surface normal vector 8101 associated with the object image information 12002. The measured cosine distance may indicate the angle between the surface normal vectors (e.g., template vector 6260 and object vector 6270), and the indicated angle directly correlates to the degree or quality of alignment between the surface normal vectors (e.g., template vector 6260 and object vector 6270). The cosine distance may be represented by the following equation:

[0168] Cosine distance = 1-cosine similarity,

[0169] Here, the cosine similarity is expressed by the following formula:

[0170]

number

[0171] Or, in the alternative:

number

[0172] The distance measure generated by the above equation outputs a value indicative of the distance between the two surface normal vectors (i.e., as a cosine distance). This output value may further indicate the angle between the template vector 6260 and the object vector 6270, or more specifically, the angle between a planar portion of the template object 6290 and a planar portion of the object image information 12002. A planar portion refers to a surface along which the surface normal vectors extend and are parallel. An output providing a small angle may be used to indicate the angle between the planar portion of the template object 6290 and the object image information 12002. Image information 12002. The cosine distances between each corresponding pair of template vector 6260 and object vector 6270 may be combined to generate a distance measurement, for example, by taking the arithmetic or geometric mean.

[0173] In another embodiment, the distance measurements may be planar distance measurements measured from one of the template locations 6210 to a plane containing the corresponding point from the object location 6220, or vice versa. The planar distances between each corresponding pair of template vectors 6260 and object vectors 6270 may be combined to generate the distance measurement, for example, by taking the arithmetic or geometric mean.

[0174] The quality of the alignment between the template object 6290 and the object image information 12002 may further be determined according to a profile showing a decrease in distance over successive iterations of the ICP technique. As described above, the ICP technique may be used to align the template position 6210 with the object position6220 The template object 6290 and the object image information 12002 may be aligned by converging with . During successive iterations, distance measurements (e.g., cosine distance, Euclidean distance, planar distance, etc.) between the template object 6290 and the object image information 12002 may be obtained. The profile may show the changes in such distances over successive iterations.

[0175] For example, a profile showing a consistent decrease in distance over successive iterations of the ICP may indicate high alignment quality in terms of convergence between the template object 6290 and the object image information 12002. Conversely, if the profile indicates that there are successive iterations of the ICP in which the distance increases or otherwise indicates that the distance is not decreasing very rapidly between successive iterations, the profile may indicate that the template object 6290 and the object image information 12002 do not exhibit high quality convergence and that the final alignment between the template object 6290 and the object image information 12002 may be of low quality.

[0176] In operation 11011, the hypothesis refinement method 11000 may include generating an adjusted detection hypothesis. The adjusted detection hypothesis 6300' may be generated according to the adjustments made to the template location 6210, as described above. The adjustments may represent adjusted versions of the various pieces of information stored in the detection hypothesis 6300. For example, the adjusted detection hypothesis 6300' may include information relating the object image information 12002 to the adjusted corresponding object recognition template 4300B', and may include adjusted pose information 6301'. The adjusted corresponding object recognition template 4300B' may include one or more of the adjusted 2D appearance 4302B', the adjusted 3D appearance 4303B', the adjusted 2D measurement information 4304B', and the adjusted 3D measurement information 4305B'. The adjusted 2D measurement information 4304B' may include adjusted gradient information 9100B' and adjusted gradient extraction positions 5100B', while the adjusted 3D measurement information 4305B' may include adjusted surface normal vectors 9101B' and adjusted surface normal positions 5101B'. The adjusted object recognition template 4300B' may further include adjusted template positions 6210', which may include adjusted gradient extraction positions 5100B' and adjusted surface normal positions 5101B' or a subset thereof. It is not required that all of the "adjusted" versions of the information included in the adjusted detection hypotheses 6300' differ from the corresponding information in the detection hypotheses 6300. For example, in an embodiment, a position may be adjusted while the information associated with that position (gradient and surface normal) may remain the same. In an embodiment, the adjusted information may be captured by storing information about the adjustment in conjunction with storing the original detection hypotheses 6300.

[0177] The present disclosure further relates to validation of detection hypotheses. FIG. 13 shows a flow diagram of an exemplary detection hypothesis validation method 13000 for validating detection hypotheses. The following description of detection hypothesis validation refers to FIG. 14. The detection hypothesis validation method 13000 may operate on one or more previously obtained detection hypotheses to validate a particular detection hypothesis as corresponding to a specifically detected physical object in a scene. As described above, through template matching and detection hypothesis generation and refinement, multiple detection hypotheses may be proposed as relating to or describing a particular physical object in a scene. The detection hypothesis validation method 13000 may receive object image information of a particular physical object as well as a set of detection hypotheses associated therewith, and validate the multiple detection hypotheses to determine an optimal or best-fit detection hypothesis. The set of detection hypotheses may be initial detection hypotheses (such as detection hypothesis 6300) and / or adjusted detection hypotheses (such as adjusted detection hypothesis 6300′), or a combination thereof. At least one processing circuit 1110 may perform certain steps of a detection hypothesis verification method 13000 to verify a detection hypothesis 8300, as described below.

[0178] In one embodiment, the detection hypothesis verification method 13000 may be implemented, for example, using the computing system 1100 (or 1100A / 1100B / 1100C) of FIGS. 2A-2D or the computing system 1100 (or 1100A / 1100B / 1100C) of FIGS. 3B 2D , which may perform the detection hypothesis verification method 13000. For example, the instructions may cause the computing system 1100 to execute one or more of the modules illustrated in FIG. 2D , which may perform the detection hypothesis verification method 13000. For example, in an embodiment, the steps of the method 13000 may be performed by the hypothesis generation module 1128, the hypothesis refinement module 1136, and the hypothesis verification module 1138 operating together.

[0179] The detection hypothesis verification method 13000 may be used to verify one or more detection hypotheses 8300 of the generated detection hypothesis set 8309 to identify one or more objects 5012 physically located within the scene 5013. The detection hypothesis verification method 13000 may operate on image information 12001 acquired from the scene 5013. The image information 12001 may be similar to the 2D image information 2600 and the 3D image information 2700. Within the image information 12001 there may be one or more object image information 12002 representing the objects 5012 within the scene 5013.

[0180] In the following discussion, the detection hypothesis verification method 13000 is discussed according to the use of a detection hypothesis set 8309 for a single object 5012 to be identified. As discussed below, the detection hypothesis verification method 13000 processes to identify a best detection hypothesis 8300 corresponding to a single object 5012. In other embodiments, the detection hypothesis set 8309 may include detection hypotheses 8300 for multiple objects 5012 in the scene 5013. Each individual object 5012 may have a corresponding group of detection hypotheses 8300 from the detection hypothesis set 8309, which may be verified according to the method described for the corresponding individual object 5012. In this manner, the detection hypothesis verification method 13000 may be utilized to verify and identify a best detection hypothesis 8300 for a single object 5012, or to identify multiple best detection hypotheses 8300, each corresponding to a different individual object 5012. By verifying multiple detection hypotheses 8300, complex lifting operations, including successive lifting of multiple objects 5012, may be planned and executed.

[0181] In the detection hypothesis verification method 13000, at least one processing circuit 1110 may be in communication with a robot 3300 having a robot arm 3320 and an end effector device 3330 connected to the arm, and a camera 1200 having a field of view and configured to execute instructions stored on a non-transitory computer readable medium when one or more objects 5012 are or were within the field of view. In an embodiment, the at least one processing circuit 1110 may not directly communicate with the robot 3300, but may send and receive information to and from the robot 3300 via a network and / or storage device. In an embodiment, the at least one processing circuit 1110 may directly communicate with the robot 3300. The at least one processing circuit 1110 may obtain image information 12001 of one or more objects 5012 in the scene 5013. The at least one processing circuit 1110 may also obtain one or more detection hypotheses 8300 and / or detection hypothesis sets 8309.

[0182] Each detection hypothesis 8300 may include information associating an object recognition template 4300 (e.g., a corresponding object recognition template 4300C selected from the plurality of object recognition templates 4300) with object image information 12002, and may include pose information 6301 of an object 5012 represented by the object image information 12002. The pose information 6301 of an object 5012 may refer to a position and orientation of the object 5012. In an embodiment, the detection hypothesis 8300 may include the corresponding object recognition template 4300C or may include a reference to the corresponding object recognition template 4300C.

[0183] In process 13001, the detection hypothesis verification method 13000 includes acquiring image information of one or more objects in a scene. Process 13001 may be similar to process 11001 discussed above. Acquiring image information 12001 may include capturing an image of a scene 5013. In such an instance, image information 12001 may represent an object 5012 located in a box, bin, case, crate, pallet, or other container. Image information 12001 may be acquired by camera 1200 as discussed herein. At least one processing circuit 1110 may be configured to generate, receive, and / or process image information 12001, such as using image information 12001 to distinguish between individual objects in a field of view of camera 1200 and perform object recognition or object registration based on image information 12001. In one embodiment, the image information 12001 may include two-dimensional image information (e.g., similar to the 2D image information 2600) that describes the visual appearance of the environment or scene 5013 in the field of view of the camera 1200. In one embodiment, the image information 12001 may include three-dimensional image information (e.g., similar to the 3D image information 2700) that provides a point cloud, spatial structure information, depth map, or other three-dimensional image of the scene 5013 in the field of view of the camera 1200. The three-dimensional image information in this example may be used to estimate how the objects 5012 are spatially arranged in a three-dimensional space (e.g., the scene 5013). The acquisition of the image information 12001 may include the generation or acquisition of the image information 12001 representing the scene 5013, and may optionally include the generation or acquisition of one or more object image information 12002 representing an individual object 5012 or multiple objects 5012 in the scene 5013. The object image information 12002 may include 2D image information 12600 representing the object 5012. The 2D image information 12600 may be similar to the image information 2600 and / or may include rendered 2D image information generated according to rendering techniques such as ray tracing and discontinuity detection. The object image information 12002 may include 3D image information 12700 representing the object 5012, similar to the image information 2700.Image information 12001 may be generated by camera 1200 when object 5012 is (or was) in the field of view of camera 1200, and may include, for example, two-dimensional image information and / or three-dimensional image information.

[0184] The object image information 12002 may include image information relating to a particular physical object 5012 in the scene 5013. The object image information 12002 may include 2D image information 12600 representing the object 5012, similar to image information 2600. The object image information 12002 may include 3D image information 12700 representing the object 5012, similar to image information 2700. The object image information 12002 may include object positions 6220, which may further include gradient extraction positions 8100 and surface normal positions 8101 representing positions from which respective gradient information 8102 and surface normal vectors 8103 are obtained, for example via the feature generation method 8000. The gradient extraction location 8100, surface normal location 8101, gradient information 8102, and surface normal vector 8103 may be similar to the gradient extraction location 5100, surface normal location 5101, gradient information 9100, and surface normal vector 9101, except that, as described above, the gradient extraction location 8100, surface normal location 8101, gradient information 8102, and surface normal vector 8103 are obtained from image information obtained from a physical object.

[0185] The image information 12001 may be the same image information 12001 obtained during performance of the hypothesis refinement method 11000. Thus, the computing system 1100 may obtain the image information 12001 for performance of the hypothesis refinement method 11000, store the image information 12001, and access the image information 12001 for performance of the detection hypothesis testing method 13000. In an embodiment, the image information 12001 may be obtained freshly, specifically for performance of the detection hypothesis testing method 13000.

[0186] As discussed above, the image information 12001 may include a two-dimensional grayscale and / or color image and may describe the appearance of the scene 5013 (and / or the object 5012 in the scene) from the viewpoint of the camera 1200. In one embodiment, the image information 12001 may correspond to a single color channel (e.g., red, green, or blue channel) of a color image. If the camera 1200 is positioned above the object 5012, the two-dimensional image information may represent the appearance of the top surface of each of the objects 5012. Furthermore, the image information 12001 may include three-dimensional image information, which may include, for example, a depth map or point cloud indicating respective depth values ​​of various object positions 6220 on one or more surfaces (e.g., top surface, or other outer surface) of the object 5012 or along one or more edges of the object 5012. The two-dimensional image information and the three-dimensional image information of the object image information 12002 may be referred to as 2D image information 12600 and 3D image information 12700, respectively. In some embodiments, the object positions 6220 representing the physical edges of the objects 5012 may be used to identify object image information 12002 that is limited to representing individual objects 5012 .

[0187] In process 13003, the detection hypothesis verification method 13000 may further include obtaining one or more detection hypotheses 8300 and / or a detection hypothesis set 8309. For ease of explanation, unless otherwise stated, the described attributes and qualities of a particular detection hypothesis 8300 may be understood to apply to each of the detection hypotheses 8300 in the detection hypothesis set 8309. The detection hypotheses 8300 may be obtained as adjusted detection hypotheses 6300' following performance of the hypothesis refinement method 11000. The detection hypotheses 8300 may be obtained as initial detection hypotheses 8300 from a template matching process, as described above.

[0188] The detection hypotheses 8300 may include a corresponding object recognition template 4300C and object pose information 8301 indicating the position and orientation of the corresponding object recognition template 4300C required to overlay the corresponding object image information 12002 in the image information 12001. The corresponding object recognition template 4300C may include one or more of the 2D appearance 4302C, the 3D appearance 4303C, the 2D measurement information 4304C, and the 3D measurement information 4305C. As described above, the 2D measurement information 4304C may include gradient information 9100C and gradient extraction positions 5100C, while the 3D measurement information 4305C may include the surface normal vector 9101C and the surface normal positions 5101C. The corresponding object recognition template 4300C may further include a template position 8210, which may include the gradient extraction positions 5100B and the surface normal positions 5101B, or a subset thereof. The information of the detection hypothesis 8300 may define a digital representation of the object, referred to herein as a template object 8290. The template object 8290 represents the information of the detection hypothesis 6300 in three-dimensional space in the coordinate system of the image information 12001 for comparison with the object image information 12002.

[0189] The detection hypothesis set 8309, and in particular the size of the set, may be selected or determined to balance proportion and completeness. Selecting more detection hypotheses 8300 may result in a higher chance of achieving a good match, but may also take longer to process. As discussed above with respect to the hypothesis refinement method 11000, the quality of the alignment may be measured or determined during the steps associated with refinement. Exceeding a quality threshold may be a marker that causes the hypothesis refinement method 11000 to be determined as complete. Similarly, exceeding a quality threshold may be considered as a marker that allows for the inclusion of an adjusted detection hypothesis 6300' in the detection hypothesis set 8309. Failure to exceed the quality threshold may result in the adjusted detection hypothesis 6300' being excluded. Thus, the size of the detection hypothesis set 8309 may be determined by how stringent the quality threshold is. In some embodiments, the size of the detection hypothesis set 8309 may be limited and only the highest quality alignment adjusted detection hypotheses 6300' are included. In embodiments, both a quality threshold and a ranked order may be used. In embodiments, a large number of detection hypotheses may be selected with the understanding that many false positives will be generated. Detection hypothesis It may be beneficial to use template matching and hypothesis refinement techniques to generate a set 8309 (e.g., more than 500, 1000, or 10,000 total detection hypotheses). Such an embodiment may rely on detection hypothesis validation methods 13000 to filter out false positives, as discussed below.

[0190] In process 13005, the detection hypothesis validation method 13000 includes validating each detection hypothesis of a set of detection hypotheses. A plurality of detection hypotheses 8300 are obtained and compared to the object image information 12002 of the image information 12001 to identify which detection hypothesis 8300 is the best estimate or best fit for describing the physical object 5012 represented by the object image information 12002. Selecting the best detection hypothesis from the detection hypothesis set 8309 includes validating each of the detection hypotheses according to processes 13007-13011 described below. Hypothesis validation may include generating three-dimensional and two-dimensional validation scores and filtering the detection hypothesis set 8309 accordingly.

[0191] In process 13007, Detection hypothesis verification method 13000 includes generating a plurality of three-dimensional validation scores. Each three-dimensional validation score may be based on a comparison of the three-dimensional information of the detection hypothesis 8300 and corresponding three-dimensional information of image information corresponding to an object from the scene (e.g., object image information 12002). The plurality of three-dimensional validation scores may include at least one of an occlusion validator score, a point cloud validator score, a hole match validator score, and a normal vector validator score. The three-dimensional information of the detection hypothesis 8300 may include 3D measurement information 4305C, including 3D appearance 4303C and surface normal vector 9101C, and surface normal position 5101C. The three-dimensional information of the object image information 12002 may include 3D image information 12700, surface normal position 8101, and surface normal vector 8103.

[0192] The validator score, as discussed herein, may be a score or number that represents how well a particular detection hypothesis corresponds to or aligns with the object image information 12002. The validator score, as discussed herein, may be a penalty score applied to the hypothesis confidence score with a lower value representing a better match. Alternatively, the validator score may be a bonus score with a higher value representing a better match. For ease of explanation, the validator scores discussed herein may be penalty scores, but it will be understood that all of the same concepts and techniques may be applied using bonus scores.

[0193] The occlusion validator score and the point cloud validator score may be obtained by comparing the object position 6220 of the object image information 12002 with the surface of the template object 8290 represented by the detection hypothesis 8300 and identifying the mismatch between the object position 6220 and the surface of the template object 8290 to obtain the occlusion validator score and the point cloud validator score, respectively. The three-dimensional information of the detection hypothesis may indicate the location of the surface of the template object 8290. If the three-dimensional information of the detection hypothesis 8300 actually represents the object 5012 of the scene 5013, the object position 6220 associated with the 3D image information 12700 should be on or near the surface. If not near the surface, the match determined by the template matching process may be a false positive. Comparing the object position 6220 with the surface of the template object 8290 may identify two types of mismatch: valid points as well as occlusions and invalid points. A mismatch that places the object position 6220 above or otherwise outside the surface of the template object 8290 may be referred to as an occlusion and used to calculate an occlusion validator score. A mismatch that places the object position 6220 below the surface of the template object 8290 may be referred to as an invalid point and used to calculate a point cloud validator score. An object position 6220 that is above or close to (within a threshold distance, also referred to as the skin depth parameter) the surface of the template object 8290 may be referred to as a valid point. Some deviation between the object position 6220 and the surface of the template object 8290 is expected. Such deviation may be accounted for by the skin depth parameter, the size of which determines the amount of allowable deviation.

[0194] The occlusion validator score is obtained by identifying discrepancies that place the object location 6220 above or outside the surface of the template object 8290. These discrepancies are called occlusions. The occlusion validator score results in a weighted penalty to the hypothesis confidence score, where the weight depends on the distance of the object location 6220 from the surface of the template object 6290. The occlusion validator score may be calculated as a function of the distance from the surface of the template object 6290. The function may be, for example, a log-normal function, where the peak of the curve of the log-normal function represents the distance from the surface that corresponds to a 3D point near the surface of the template object 8290, but is unlikely to be part of the template object 8290. In an embodiment, the function with the peak may be selected just beyond the point where the sensor or camera capturing the image information 12001 loses accuracy. For example, an object location 6220 that is beyond the surface of the template object 6290 at a very large distance may have a lower penalty applied to it due to the possibility that such an object location 6220 results from an occlusion from another object 5012 between the matching portion of the scene 5013 and the matching portion of the camera 1200, or from noise in the image information 12001, rather than being an actual point on the object 5012 represented by the object image information 12002. Thus, the penalty in the occlusion validator score for a particular object location 6220 may initially increase with distance, reducing the reliability of the detection hypothesis. After the distance increases beyond a peak, it becomes increasingly likely that the particular object location 6220 is not generated by the object 5012 represented by the object image information 12002, and the penalty decreases.

[0195] In an embodiment, an occlusion confidence score may be determined for the occlusion validator score. The occlusion confidence score represents a confidence level that the occlusion validator score provides good information about which decisions can be made. The object locations 6220 may represent points or locations that there is confidence that they belong to the object. However, the object image information 12002 may include additional points that are not confidently identified as belonging to the object 5012. The occlusion confidence score may be based on a ratio of the object locations 6220 to the total number of visible points of the object image information 12002. Thus, if the object locations 6220 that there is confidence that they belong to the object are a lower percentage of the total visible points, the confidence that the occlusion validator score based on the object locations 6220 provides accurate information decreases, and the associated occlusion confidence score decreases as well. In some embodiments, the final occlusion validator score may be represented by the initial occlusion validator score modified according to the occlusion confidence score.

[0196] The point cloud validator score is obtained by identifying discrepancies that place the object location 6220 inside or below the surface of the template object 8290. These discrepancies are called invalid points. The point cloud validator score is a penalty to the hypothesis confidence score. An object location 6220 identified as an invalid point, e.g., below the surface of the template object 8290, may be a strong indicator that the detection hypothesis 8300 is inaccurate and may result in a correspondingly high penalty score. In an embodiment, the point cloud validator score may be based on the number of invalid points or the ratio of invalid points to a cutoff value for invalid points.

[0197] The point cloud validator score may have a point cloud confidence score determined in the same manner as described above for the occlusion confidence score, for example, according to the ratio of object positions 6220 to the total number of visible points in the object image information 12002. In an embodiment, the final point cloud validator score may be represented by the point cloud validator score modified according to the point cloud confidence score.

[0198] In an embodiment, the point cloud validator score and the occlusion validator score may be combined into a single surface validator score. The surface validator score may be determined as a combination of the point cloud validator score and the occlusion validator score, for example by adding, averaging, or performing another mathematical operation to combine the two.

[0199] The normal vector validator score may be obtained by determining whether a valid point identified according to an object position 6220 on or near the surface of the template object 6290 has a surface normal vector 8103 that matches the orientation of the surface of the template object 6290. Such a determination may be made by comparing the surface normal vector 8103 associated with the object position 6220 with the corresponding surface normal vector 9101C associated with the corresponding surface normal position 5101C of the corresponding object recognition template 4300C. If the surface normal vector 8103 does not align or match the orientation of the corresponding surface normal vector 9101C, the normal vector validator score may be applied as a penalty to the detection hypothesis confidence score. In an embodiment, the amount of inconsistency or misalignment may affect the size of the penalty applied.

[0200] In an embodiment, some tolerance may be given to situations where the surface normal vector 8103 is not expected to align or match the orientation of the corresponding surface normal vector 9101C even if the detection hypothesis is accurate. For example, an object such as a gear with many teeth may have edges and parts that exhibit abrupt changes in the surface normal vector. Such object structures may cause large deviations in the surface normal vector 8103 and the corresponding surface normal vector 9101C even if there is only a slight misalignment between the object image information 12002 and the template object 8290 overlaid on the scene. To account for such scenarios, the at least one processing circuit 1110 may check whether the corresponding object recognition template 4300C or image information 12001 has areas with high variations in the surface normal vector 9101C / 8103. If the result is positive, at least one processing circuit 1110 may apply a higher amount of tolerance by lowering the normal vector verification score for differences in high change areas between the corresponding surface normal vector 9101C of the corresponding object recognition template 4300C and the surface normal vector 8103 of the object image information 12002.

[0201] In an embodiment, the surface normal validator score may have a surface normal confidence level associated with it. The surface normal confidence level may represent a confidence level of the information provided by the surface normal validator score. In one embodiment, the surface normal confidence level may be determined according to the quality of the extracted edge. In one embodiment, the surface normal validator score may be adjusted according to the surface normal confidence level.

[0202] The hole match validator score is obtained by comparing the object position 6220 obtained from the object image information 12002 with the structure of the template object 8290 as represented by the corresponding object recognition template 4300C, identifying discrepancies between the object position 6220 and the structure, and identifying invalid holes or empty coordinates (called hole invalidity) according to the object position 6220 corresponding to the empty volume of the structure of the template object or location where the structure is absent. Since the object position 6220 of the object image information 12002 represents a position on the surface of the physical structure of the object 5012, the object 5012 in the scene does not have a structure in the space that the corresponding object recognition template 4300C indicates to be empty. The presence of the object position 6220 in the part that the corresponding object recognition template 4300C indicates to be empty may be due to noise, but may also indicate an inaccurate detection hypothesis. Thus, the hole match validator score may be determined as a penalty score for the detection hypothesis confidence level for all identified hole invalidities.

[0203] In an embodiment, the hole match validator score may have a hole match confidence level associated with it. The hole match confidence level may represent a confidence level of the information provided by the hole match validator score. In one embodiment, the hole match confidence level may be determined according to the quality of the extracted edges. In one embodiment, the hole match validator score may be adjusted according to the hole match confidence level.

[0204] In an embodiment, a tolerance may be provided to account for other circumstances that may result in noise or hole invalidity even with a correct detection hypothesis. For example, if the object image information 12002 includes an object location 6220 that corresponds to an empty space (e.g., a hole or opening in the object) of the template object 8290, that object location 6220 may correspond to a portion of another object that is coincidentally located in the space. Such a scenario may be consistent with a correct detection hypothesis 8300 for an object 5012 in a scene 5013 when the object location 6220 of the empty space presumably does not belong to the object 5012 represented by the corresponding object recognition template 4300C, but instead belongs to another object. In an embodiment, the hole match validator score may provide a larger tolerance when the size of the hole, opening, or empty coordinate of the template object 8290 is relatively large, because of the increased likelihood of irregularities affecting the measurement of that space (e.g., an object intersecting or protruding through the hole or opening) and the likelihood of another object extending into that space.

[0205] In an embodiment, the point cloud validator score, the occlusion validator score, the hole match validator score, and the surface normal validator score may be combined into a single 3D validator score. The 3D validator score may be determined as any combination of the point cloud validator score, the occlusion validator score (or the combined surface validator score), the hole match validator score, and the surface normal validator score, for example, by adding, averaging, or performing another mathematical operation to combine the two.

[0206] In process 13009, Detection hypothesis verification method 13000 includes generating a plurality of two-dimensional validation scores, which may include at least one of a rendered match validator score and a template matching validator score.

[0207] The rendered match validator score is obtained by comparing the rendered 2D image information 12600 of the image information 12001 with the corresponding 2D appearance 4302C. The rendered match validator score may be further processed to extract edge information from both the rendered 2D image information 12600 and the corresponding 2D appearance 4302C. The rendered match validator score may be based on a determination of whether the edges extracted from the 2D image information 12600 align with the edges extracted from the corresponding 2D appearance 4302C. The rendered match validator score may be based on the amount of overlap between the regions defined by the extracted edges, the average distance between the extracted edges, or any other suitable metric. The rendered match validator score may be used as a penalty score applied to the detection hypothesis confidence score. In some instances, using rendering (e.g., ray tracing) to generate and extract edge information may compensate for noise and other conditions that may cause artifacts, such as glare of light reflecting off metal objects or shadows. In some instances, process 13009 may also process re-rendering information from the corresponding object recognition template 4300C to extract edges from the corresponding object recognition template 4300C.

[0208] In an embodiment, the rendered match validator score may have a rendered match confidence level associated with it. The rendered match confidence level may represent a confidence level in the information provided by the rendered match validator score. In one embodiment, the rendered match confidence level may be determined according to the quality of the extracted edges. In one embodiment, the rendered match validator score may be adjusted according to the rendered match confidence level.

[0209] The template matching validator score is obtained by comparing edges extracted from the object image information 12002 with an object image derived from the corresponding object recognition template 4300C (e.g., template object 8290 or 2D appearance 4302C, etc.). An edge detection algorithm, e.g., a Canny edge detector, may be utilized to identify object edges directly from the object image information 12002 and template edges from image information stored in the corresponding object recognition template 4300C. The template matching validator score may be determined according to whether there is an offset between the object edge and the template edge by sliding the template edge against the object edge and determining how much sliding (if any) results in a peak response or overlap. The template matching validator score may be based on the amount of sliding, movement, offset, or adjustment required to achieve the peak response or overlap. The more amount of movement or sliding required, the higher the template matching validator score and the larger the penalty applied. In other words, more required movement indicates a poorer match.

[0210] In an embodiment, the template matching validator score may have a template matching confidence level associated with it. The template matching confidence level may represent a confidence level in the information provided by the template matching validator score. In an embodiment, the template matching confidence level may be determined according to the quality of the extracted edges. In an embodiment, the template matching validator score may be adjusted according to the template matching confidence level.

[0211] The three-dimensional validator scores and the two-dimensional validator scores may be combined to determine an overall validation score that may be used in further processing to determine an overall confidence level in the detection hypotheses. The total validation score may be based on a combination of each of the three-dimensional and two-dimensional validator scores and a confidence value associated with each validator score. For example, validator scores with higher confidence values ​​and / or higher score weights may have a greater impact on the total validation score, while validator scores with lower confidence values ​​and / or lower score weights may have a lesser impact on the total validation score.

[0212] In an embodiment, the process 13005 may further include an additional validation step that determines whether the corresponding object recognition template 4300C has a global consistency with other structures or objects in the image information 12001 corresponding to the scene 5013. For example, such other structures and objects may include a container in which the workpiece or other object is located. For example, the process 13005 may further determine whether the template object 8290 fits completely within such a container (e.g., based on the position of the template object 8290 determined by the pose information 6301) or whether the template object 8290 extends or protrudes outside the surface of the container. If the template object 8290 or a portion thereof is outside the container, such a situation may be an indication of an incorrect detection hypothesis. In such a situation, the total validation score may be adjusted accordingly, with a penalty weighted according to how far the template object 8290 is outside the container. In an embodiment, if the template object 8290 or a portion thereof is outside the container more than a threshold amount, the total validation score may be adjusted to indicate an incorrect detection hypothesis. Some tolerance may be provided to account for situations where an accurate detection hypothesis may still match a template object 8290 that extends outside the container or extends beyond the plane defining the container's inner surface. Such situations may arise, for example, when the container is a mesh container or when the object is a metal object that is sufficiently rigid to cause the container's inner surface to become uneven or otherwise deform.

[0213] In process 13011, Detection hypothesis verification method 13000 further includes filtering detection hypotheses from the set of detection hypotheses according to the plurality of three-dimensional validation scores and the plurality of two-dimensional validation scores.

[0214] In an embodiment, multiple validator scores may be combined to generate an overall validator score that may be used to determine a detection hypothesis confidence level. The overall validator score and detection hypothesis confidence level may indicate how well a corresponding object recognition template 4300C matches the object image information 12002 obtained from the scene 5013. Detection hypothesis The confidence level may be used to decide whether to exclude a detection hypothesis or to use the detection hypothesis to plan a robot action to lift the object 5012 in the scene 5013. 。

[0215] In an embodiment, the filtering of the detection hypotheses may be performed according to a successive filtering technique, where each of the validator scores is compared to a corresponding threshold to determine whether to retain or filter a given detection hypothesis 8300 from the detection hypothesis set 8309. Each successive validator score may be compared to a threshold, and if the validator score exceeds the threshold, the detection hypothesis 8300 may be eliminated. In one example, the filtering of the detection hypotheses 8300 from the detection hypothesis set 8309 may include comparing the occlusion validator score, the point cloud validator score, the hole match validator score, the normal vector validator score, the rendered match validator score, and the template matching validator score to corresponding thresholds, and eliminating any detection hypotheses 8300 with validator scores exceeding the corresponding thresholds. The above comparisons may be performed sequentially. The above order is for illustrative purposes only, and any order may be used. When successive filtering is utilized, the efficiency of the process may be increased by not calculating additional validator scores for eliminated detection hypotheses.

[0216] In embodiments, comparing a validator score to a corresponding threshold may take into account a confidence level associated with the validator score. For example, an associated threshold may be adjusted according to the validator score confidence level and / or the validator score may be adjusted according to the confidence level. Thus, low-confidence validator scores indicating a bad match may be eliminated as detection hypotheses, while high-confidence validator scores may have more influence.

[0217] Thus, if one or more of the three-dimensional validation scores or two-dimensional validation scores exceed a corresponding threshold (taking into account confidence levels, if necessary), the detection hypothesis 8300 may be removed or filtered from the detection hypothesis set 8309. The detection hypothesis 8300 may remain in the detection hypothesis set 8309 if none of the three-dimensional validation scores and none of the two-dimensional validation scores exceed all of the corresponding thresholds (taking into account confidence levels, if necessary).

[0218] The filtering process may continue until a single detection hypothesis 8300 remains for each particular object 5012 corresponding to the object image information 12002. This may occur by selecting the detection hypothesis with the highest detection hypothesis confidence level (and lowest overall validation score) and / or by repeating the filtering process with increasingly lowered filter thresholds until only a single detection hypothesis 8300 is successful for each object 5012. The single detection hypothesis 8300 may be an unfiltered detection hypothesis 8300. In an embodiment, a minimum confidence level may be set for the detection hypothesis 8300. In such an embodiment, if the best-fit detection hypothesis 8300 for an object 5012 fails to exceed the confidence threshold, the system may not return a detection hypothesis 8300 for that object.

[0219] In operation 13013, the hypothesis testing method 13000 includes detecting one or more objects in the scene according to the unfiltered detection hypotheses remaining in the set of detection hypotheses after testing. After filtering, a best detection hypothesis 8300 corresponding to the object 5012 associated with the object image information 12002 is identified to detect the object 5012 in the scene. As mentioned above, the hypothesis testing method 13000 may also be utilized to identify multiple different objects 5012 according to multiple different associated detection hypotheses 8300.

[0220] In some embodiments, the hypothesis verification method 13000 may further include an overlap detection process, whereby one or more detection hypotheses 8300 are compared to each other to determine whether their corresponding template objects 8290 have overlaps. Such overlaps may indicate that one or both of the overlapping detection hypotheses 8300 are inaccurate. The detection hypotheses 8300 may be compared for overlaps after the filtering process 13011. Before the filtering process 13011, multiple detection hypotheses 8300 may remain for each object 5012, and therefore overlaps are expected. After the filtering process 13011, the remaining detection hypotheses 8300 represent the best matches for the individual objects 5012, and overlaps are not expected. In response to detecting overlaps, the system may be configured to discard one or both of the overlap detection hypotheses 8300, for example based on their confidence scores, or may be configured to perform additional analysis or processing on the overlap detection hypotheses 8300. The decision to discard, keep, or reanalyze the overlap detection hypotheses 8300 may further take into account the extent of the overlap.

[0221] Following detection of one or more objects 5012 in the scene 5013, the at least one processing circuit 1110 may process to execute a robot control process 15000 for retrieval of the one or more objects 5012 and output commands to cause the movement of the robot 3300 to retrieve the one or more objects 5012. The robot control process 15000 may include obstacle detection, motion planning, and motion execution.

[0222] Obstacle detection may include detecting and accounting for obstacles in the vicinity of the object 5012 to be retrieved. As discussed herein, the object 5012 may be in a container with other items and objects. Thus, the other items and objects, as well as the container itself, may represent obstacles to the robotic operations of the robot 3300. Such obstacles may be captured in the image information 12001 and / or object image information 12002, which may be processed to determine the location of the obstacles in the vicinity of the object.

[0223] Motion planning may include planning a robot motion, such as, for example, plotting a trajectory for the robot 3300 to execute to retrieve the object 5012. The trajectory may be plotted to account for and avoid identified obstacles. Motion execution may include sending commands related to the motion plan to the robot or a robot control system to cause the robot 3300 to execute the planned motion.

[0224] The methods discussed herein, for example, methods 6000, 8000, 10000, 11000, and 13000, may be processed in concert to create object recognition templates and utilize the object recognition templates to generate, refine, and validate detection hypotheses for objects within a scene. Thus, methods 6000, 8000, 10000, 11000, and 13000 may be utilized to facilitate a robotic process for detecting, identifying, and removing multiple objects from within a container.

[0225] It will be apparent to those skilled in the relevant art that other suitable modifications and adaptations to the methods and applications described herein can be made without departing from the scope of any of the embodiments. The embodiments described above are illustrative examples, and the disclosure should not be construed as being limited to these particular embodiments. It should be understood that the various embodiments disclosed herein may be combined in different combinations than those specifically presented in the description and accompanying figures. It should also be understood that, by way of example, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, or may be added, integrated, or omitted entirely (e.g., not all acts or events described may be required to perform a method or process). In addition, although certain features of the embodiments herein are described for clarity as being performed by a single component, module, or unit, it should be understood that the features and functions described herein may be performed by any combination of components, units, or modules. Thus, various changes and modifications may be effected by those skilled in the art without departing from the spirit or scope of the invention, as defined in the appended claims.

[0226] Further embodiments include the following:

[0227] Embodiment 1 is a computing system configured to generate a set of object recognition templates for identifying an object in a scene, the computing system comprising at least one processing circuit configured to: acquire object registration data including an object model representing the object; determine a plurality of viewpoints of the object model in three-dimensional space; estimate a plurality of appearances of the object model at each of the plurality of viewpoints; generate a plurality of object recognition templates according to the plurality of appearances, each corresponding to a respective one of the plurality of appearances; and communicate the plurality of object recognition templates to a robot control system as the object recognition template set, wherein each of the plurality of object recognition templates represents a pose the object may have relative to an optical axis of a camera that generates image information of the object in the scene.

[0228] Embodiment 2 is a computing system as described in embodiment 1, wherein the three-dimensional space is bounded by a surface, each of the multiple viewpoints corresponds to a camera position on the surface, and each of the object recognition templates corresponds to one of the multiple viewpoints and includes an appearance of the object from the one viewpoint.

[0229] Embodiment 3 is the computing system according to embodiment 1, wherein each of the multiple viewpoints further corresponds to a camera rotation angle.

[0230] Example 4 is the computing system according to Example 2, in which the object model is fixed in three-dimensional space.

[0231] Embodiment 5 is the computing system according to embodiment 2, wherein the three-dimensional space is substantially spherical and the object model is fixed to the center of the three-dimensional space.

[0232] Example 6 is the computing system of example 2, wherein the multiple viewpoints are selected according to an even distribution across the surface.

[0233] Embodiment 7 is the computing system of embodiment 3, wherein each camera position corresponds to a set of viewpoints, each viewpoint of the set of viewpoints corresponding to a different camera rotation angle.

[0234] Example 8 is the computing system of example 3, wherein the subset of the set of object recognition templates includes object recognition templates corresponding to different positions and viewpoints corresponding to different camera rotation angles.

[0235] Example 9 is the computing system of example 2, further comprising determining the multiple viewpoints based on a predicted range of poses observed for the multiple object recognition templates.

[0236] Example 10 is the computing system of example 2, further comprising determining the multiple viewpoints based on symmetry of the object.

[0237] Example 11 is the computing system of example 10, further comprising determining symmetry of the object according to at least one of determining that the object's appearance changes after rotation and identifying an axis of the object.

[0238] Embodiment 12 is a method for generating an object recognition template set for identifying an object in a scene, the method comprising: acquiring object registration data including an object model representing the object; determining a plurality of viewpoints of the object model in three-dimensional space; estimating a plurality of appearances of the object model at each of the plurality of viewpoints; generating a plurality of object recognition templates according to the plurality of appearances, each object recognition template corresponding to a respective one of the plurality of appearances; and communicating the plurality of object recognition templates to a robot control system as the object recognition template set, each of the plurality of object recognition templates representing a pose the object may have relative to an optical axis of a camera that generates image information for the object in the scene.

[0239] Example 13 is a method as described in example 12, wherein the three-dimensional space is enclosed by a surface, and further comprising: corresponding a plurality of viewpoints to camera positions on the surface; and corresponding each of the object recognition templates to one of the plurality of viewpoints.

[0240]

[0031] Example 14 is the method of example 13, further comprising: associating each of the multiple viewpoints with a camera rotation angle.

[0241] Example 15 is the method of example 13, further comprising fixing the object model in three-dimensional space.

[0242] Example 16 is the method of example 13, further comprising selecting the multiple viewpoints according to a uniform distribution across the surface.

[0243]

[0023] Example 17 is the method of example 13, further comprising determining the multiple viewpoints based on a predicted range of poses observed for the multiple object recognition templates.

[0244]

[0023] Example 18 is the method of example 13, further comprising determining the multiple viewpoints based on symmetry of the object.

[0245] Example 19 is the method of example 18, further comprising determining symmetry of the object according to at least one of determining that the object appearance of the object changes after rotation and identifying an axis of the object.

[0246]

[0023] Embodiment 20 is a non-transitory computer-readable medium operable by at least one processing circuit via a communication interface configured to communicate with a robotic system, the non-transitory computer-readable medium having executable instructions for performing a method for generating object recognition templates for identifying an object in a scene, the method including receiving enrollment data for an object including an object model representing the object, performing operations for generating a plurality of views of the object model in three-dimensional space, performing operations for estimating a plurality of appearances of the object model at each of the plurality of views, performing operations for generating a plurality of object recognition templates according to the plurality of appearances, each of the plurality of object recognition templates corresponding to a respective one of the plurality of appearances, and outputting the plurality of object recognition templates to the robotic system as an object recognition template set, each of the plurality of object recognition templates representing a possible pose of the object with respect to an optical axis of a camera generating image information of the object in the scene.

[0247] Embodiment 21 is a computing system configured to generate an object recognition template for identifying an object in a scene, the computing system including at least one processing circuit configured to acquire object information including a digitally represented object, extract two-dimensional measurement information from the object information, extract three-dimensional measurement information from the object information, and generate the object recognition template according to the two-dimensional measurement information and the three-dimensional measurement information.

[0248] Embodiment 22 is a computational system as described in embodiment 21, in which the digitally represented object is an object model, and extracting two-dimensional measurement information and three-dimensional measurements is performed to generate a feature map of the object model at a selected viewpoint.

[0249] Embodiment 23 is a computing system as described in embodiment 21, wherein at least one processing circuit is further configured to acquire image information of the scene, access an object recognition template, and compare the two-dimensional measurement information and the three-dimensional measurement information with the image information to identify an object corresponding to the digitally represented object.

[0250] Embodiment 24 is a computing system as described in embodiment 21, wherein extracting two-dimensional measurement information includes extracting gradient information from the object information, the gradient information indicating a direction or orientation of a candidate edge of the digitally represented object, and extracting three-dimensional measurement information includes extracting surface normal vector information from the object information, the surface normal vector information describing a plurality of vectors normal to a surface of the digitally represented object.

[0251] Embodiment 25 is the computing system of embodiment 21, wherein the object information includes registration data for the object and the digitally represented object includes an object model.

[0252] Embodiment 26 is the computing system of embodiment 21, wherein the object information includes at least one of two-dimensional image information and three-dimensional image information.

[0253] Embodiment 27 is a computational system as described in embodiment 24, in which gradient information is extracted at multiple gradient extraction locations of a digitally represented object, and extracting the gradient information includes analyzing pixel intensities of two-dimensional image information of the object information to measure a direction in which pixel intensities of the two-dimensional image information at each gradient extraction location change.

[0254] Embodiment 28 is a computing system as described in embodiment 24, in which surface normal vector information is extracted at multiple surface normal positions of the digitally represented object, and extracting the surface normal vector information includes identifying multiple vectors normal to the surface of the digitally represented object at each surface normal position.

[0255] Embodiment 29 is a computational system as described in embodiment 24, in which gradient information is extracted at multiple gradient extraction locations of the digitally represented object, and surface normal vector information is extracted at multiple surface normal locations of the digitally represented object, the multiple gradient extraction locations being different from the multiple surface normal locations.

[0256] Embodiment 30 is the computing system described in embodiment 29, wherein the multiple gradient extraction locations do not overlap with the multiple surface normal locations.

[0257] Embodiment 31 is a computing system as described in embodiment 29, in which the multiple gradient extraction locations are positioned at the edge of the digitally represented object, and the multiple surface normal locations are positioned away from the edge of the digitally represented object.

[0258] Embodiment 32 is a method for generating an object recognition template for identifying an object in a scene, the method including obtaining object information including a digitally represented object, extracting two-dimensional measurement information from the object information, extracting three-dimensional measurement information from the object information, and generating the object recognition template according to the two-dimensional measurement information and the three-dimensional measurement information.

[0259]

[0036] Embodiment 33 is the method of embodiment 32, further comprising generating a feature map of the object model at the selected viewpoint.

[0260] Embodiment 34 is a method as described in embodiment 32, further comprising acquiring image information of the scene, accessing an object recognition template, and comparing the two-dimensional measurement information and the three-dimensional measurement information with the image information to identify the object as corresponding to the digitally represented object.

[0261] Embodiment 35 is the method of embodiment 32, wherein extracting the two-dimensional measurement information further includes extracting gradient information from the object information, the gradient information indicating a direction or orientation of a candidate edge of the digitally represented object.

[0262] Embodiment 36 is the method of embodiment 32, in which extracting the three-dimensional measurement information further includes extracting surface normal vector information from the object information, the surface normal vector information describing a plurality of vectors of normals to a surface of the digitally represented object.

[0263] Embodiment 37 is a method as described in embodiment 35, further comprising extracting gradient information at a plurality of gradient extraction locations of a digitally represented object, and analyzing pixel intensities of two-dimensional image information of the object information to measure a direction in which pixel intensities of the two-dimensional image information at each gradient extraction location change.

[0264] Example 38 is a method as described in example 36, further comprising extracting surface normal vector information at multiple surface normal positions of the digitally represented object, and identifying multiple vectors of normals to the surface of the digitally represented object at each surface normal position.

[0265]

[0023] Example 39 is a non-transitory computer-readable medium operable by at least one processing circuit via a communication interface configured to communicate with a robotic system, the non-transitory computer-readable medium being configured with executable instructions for performing a method for generating an object recognition template for identifying an object in a scene, the method including receiving object information including a digitally represented object, performing operations to extract two-dimensional measurement information from the object information, performing operations to extract three-dimensional measurement information from the object information, and outputting the object recognition template to the robotic system according to the two-dimensional measurement information and the three-dimensional measurement information.

[0266] Embodiment 40 is an embodiment described in embodiment 39, further including receiving image information of the scene, accessing an object recognition template, and outputting a comparison of two-dimensional measurement information and three-dimensional measurement information for the image information to a robotic system to identify the object as corresponding to a digitally represented object.

[0267] Embodiment 41 is a computing system comprising at least one processing circuit in communication with a robot having an arm and an end effector connected to the arm, and in communication with a camera having a field of view, the at least one processing circuit being configured to execute instructions stored on a non-transitory computer-readable medium when one or more objects are or were within the field of view, the instructions including: acquiring object image information of an object in a scene; acquiring a detection hypothesis including a corresponding object recognition template representing a template object; identifying a mismatch between the template object and the object image information; identifying a set of template positions in the template object corresponding to the set of object positions in the object image information; adjusting the set of template positions to converge to the set of object positions; and generating an adjusted detection hypothesis including an adjusted corresponding object recognition template according to the adjusted set of template positions.

[0268] Embodiment 42 is a computing system as described in embodiment 41, further comprising: identifying respective vectors extending between the set of template positions and corresponding ones of the set of object positions; and adjusting the set of template positions by iteratively adjusting the set of template positions according to the respective vectors.

[0269] Embodiment 43 is a computing system as described in embodiment 42, in which iteratively adjusting the set of template positions includes iteratively generating an adjusted set of template positions according to a magnitude and direction of each vector acting on the template object, adjusting each vector according to the adjusted set of template positions, and identifying new each vector according to the adjusted set of template positions until the alignment quality exceeds a threshold.

[0270] Embodiment 44 is the computing system of embodiment 43, wherein the quality of the alignment is determined based on the level of misalignment defined by each new vector.

[0271] Example 45 is the computing system of example 43, wherein the quality of the alignment is determined based on distance measurements between the set of adjusted template positions and the set of object positions.

[0272] Example 46 is the computing system of example 45, wherein the distance measure comprises a Euclidean distance measure.

[0273] Example 47 is the computing system of example 45, wherein the distance measurement comprises a cosine distance between surface normal vectors associated with the set of adjusted template positions and the set of object positions.

[0274] Embodiment 48 is the computational system of embodiment 47, wherein the cosine distance indicates the angle between the surface normal vectors, the size of the angle being correlated with the quality of the alignment.

[0275] Example 49 is the computing system of example 45, wherein the distance measurement is a measurement from a first position in the set of adjusted template positions to a plane of a second position in the set of object positions.

[0276] Embodiment 50 is the computing system of embodiment 43, wherein the quality of the alignment is determined by a rate of convergence between the set of adjusted template positions and the set of object positions.

[0277] Embodiment 51 is a system as described in embodiment 41, further including obtaining a detection hypothesis by overlaying image information of the scene on an object recognition template and identifying object image information based on a comparison of template gradient information and template surface normal vector information of the object recognition template with object gradient information and object surface normal vector information extracted from the image information.

[0278] Embodiment 52 is a method including: obtaining object image information of an object in a scene; obtaining a detection hypothesis including a corresponding object recognition template representing a template object; identifying a mismatch between the template object and the object image information; identifying a set of template positions in the template object corresponding to a set of object positions in the object image information; adjusting the set of template positions to converge to the set of object positions; and generating an adjusted detection hypothesis including an adjusted corresponding object recognition template according to the adjusted set of template positions.

[0279] Embodiment 53 is a method according to embodiment 52, in which adjusting the set of template positions further includes identifying respective vectors extending between the set of template positions and corresponding ones of the set of object positions, and iteratively adjusting the set of template positions according to the respective vectors.

[0280] Embodiment 54 is a method according to embodiment 53, further comprising iteratively generating an adjusted set of template positions according to the magnitude and direction of each vector acting on the template object, adjusting each vector according to the set of adjusted template positions, and identifying new each vector according to the set of adjusted template positions until the alignment quality exceeds a threshold.

[0281] Embodiment 55 is the method of embodiment 54, further comprising determining a quality of the alignment based on a level of misalignment defined by each new vector.

[0282] Example 56 is the method of example 54, further comprising determining a quality of the alignment based on distance measurements between the set of adjusted template positions and the set of object positions.

[0283] Example 57 is the method of example 54, further comprising determining the quality of the alignment by a rate of convergence between the set of adjusted template positions and the set of object positions.

[0284] Embodiment 58 is a method according to embodiment 52, in which obtaining a detection hypothesis further includes overlaying image information of the scene on an object recognition template and identifying object image information based on a comparison of template gradient information and template surface normal vector information of the object recognition template with object gradient information and object surface normal vector information extracted from the image information.

[0285] Embodiment 59 is a non-transitory computer-readable medium operable by at least one processing circuit via a communications interface configured to communicate with a robotic system, the non-transitory computer-readable medium having executable instructions for performing a method for refining a detection hypothesis, the method including: receiving object image information of an object in a scene; receiving a detection hypothesis including a corresponding object recognition template representing a template object; performing a process for identifying a mismatch between the template object and the object image information; performing a process for identifying a set of template positions in the template object corresponding to a set of object positions in the object image information; performing a process for adjusting the set of template positions to converge to the set of object positions; and outputting an adjusted detection hypothesis to the robotic system including an adjusted corresponding object recognition template according to the adjusted set of template positions.

[0286] Embodiment 60 is a method according to embodiment 59, in which the process for adjusting the set of template positions includes performing a process of identifying respective vectors extending between the set of template positions and corresponding ones of the set of object positions, and performing a process of adjusting the set of template positions after iteratively adjusting the set of template positions according to the respective vectors.

[0287]

[0036] Embodiment 61 includes at least one processing circuit in communication with a robot having an arm and an end effector connected to the arm, and in communication with a camera having a field of view, the at least one processing circuit being configured to execute instructions stored on a non-transitory computer readable medium when one or more objects are or were within the field of view, the instructions including: acquiring object image information of an object in a scene; acquiring a set of detection hypotheses each including a corresponding object recognition template representing a template object; and verifying each detection hypothesis of the set of detection hypotheses, the verifying including generating a plurality of three-dimensional verification scores based on a comparison of three-dimensional information of the object recognition templates of the detection hypotheses to three-dimensional information of the object image information corresponding to the object, the plurality of three-dimensional verification scores being generated based on a comparison of three-dimensional information of the object recognition templates of the detection hypotheses to three-dimensional information of the object image information corresponding to the object, a verification score including at least one of an occlusion validator score, a point cloud validator score, a hole match validator score, and a normal vector validator score; generating a plurality of two-dimensional verification scores based on a comparison of two-dimensional information of a corresponding object recognition template of the detection hypotheses and three-dimensional information of the object image information, wherein the plurality of two-dimensional verification scores include at least one of a rendered match validator score and a template matching validator score; filtering detection hypotheses from the set of detection hypotheses according to the plurality of three-dimensional verification scores and the plurality of two-dimensional verification scores; and detecting an object in the scene according to unfiltered detection hypotheses remaining in the set of detection hypotheses after verification.

[0288] Embodiment 62 is a computing system as described in embodiment 61, wherein the instructions further include executing a robot motion planning procedure for removing the object from the scene and outputting commands for moving the robot to remove the object.

[0289] Embodiment 63 is a computational system as described in embodiment 61, wherein the multiple three-dimensional validation scores include a point cloud validator score, and the point cloud validator score is obtained by comparing an object position obtained from the object image information with a surface of a template object, identifying discrepancies between the object position and the surface, and obtaining a point cloud validator score.

[0290] Embodiment 64 is a computing system as described in embodiment 63, in which invalid object positions are identified according to discrepancies that place the object positions below the surface of the template object, and the point cloud validator score is based on the invalid object positions.

[0291] Embodiment 65 is a computational system as described in embodiment 61, wherein the multiple three-dimensional verification scores include an occlusion validator score, and the occlusion validator score is obtained by comparing an object position obtained from the object image information with a surface of a template object, identifying discrepancies between the object position and the surface, and obtaining an occlusion validator score.

[0292]

[0036] Example 66 is the computing system of example 65, in which occlusion is identified according to a discrepancy that places the corresponding object position above or outside the surface of the template object, and the occlusion validator score is based on the occlusion.

[0293] Embodiment 67 is a computational system as described in embodiment 61, wherein the multiple three-dimensional verification scores include a normal vector validator score, and the normal vector validator score is obtained by comparing a surface normal vector obtained from the object image information with a corresponding surface normal vector of a template object, identifying discrepancies between the surface normal vector and the corresponding surface normal vector to obtain a normal vector validator score.

[0294] Embodiment 68 is a computing system of embodiment 61, wherein the plurality of three-dimensional verification scores include hole match validator scores, and the hole match validator scores are obtained by comparing an object position obtained from the object image information with a structure of a template object, and identifying a discrepancy between the object position and the structure to identify invalidity of a hole according to an object position at a position corresponding to an empty volume within the structure of the template object.

[0295] Embodiment 69 is a computing system as described in embodiment 61, in which the rendered match validator score is obtained by generating a two-dimensional rendering of an object in the scene and comparing the rendered edges of the two-dimensional rendering of the object with extracted edges of a template object to identify invalid edges.

[0296] Embodiment 70 is a computing system as described in embodiment 61, in which verifying each detection hypothesis in the set of detection hypotheses further includes comparing a corresponding object recognition template with scene elements other than the object that correspond to the template object.

[0297] Embodiment 71 is a computing system as described in embodiment 70, in which comparing a corresponding object recognition template representing the estimated object with the scene elements includes determining whether an object corresponding to the template object is within the container.

[0298] Embodiment 72 is a computing system as described in embodiment 71, in which filtering detection hypotheses from the set of detection hypotheses includes comparing the occlusion validator score, the point cloud validator score, the hole match validator score, the normal vector validator score, the rendered match validator score, and the template matching validator score with corresponding thresholds, and if any of the three-dimensional validation score or the two-dimensional validation score fails to exceed the corresponding threshold, the detection hypothesis is removed from the set of detection hypotheses, and if the three-dimensional validation score and the two-dimensional validation score exceed all of the corresponding thresholds, the detection hypothesis remains in the set of detection hypotheses.

[0299] Embodiment 73 is a method including: obtaining object image information of an object in a scene; obtaining a set of detection hypotheses each including a corresponding object recognition template representing a template object; generating a plurality of three-dimensional validation scores based on a comparison of three-dimensional information of the object recognition template of the detection hypothesis with three-dimensional information of the object image information corresponding to the object, the plurality of three-dimensional validation scores including at least one of an occlusion validator score, a point cloud validator score, a hole match validator score, and a normal vector validator score; generating a plurality of two-dimensional validation scores based on a comparison of two-dimensional information of the corresponding object recognition template of the detection hypothesis with three-dimensional information of the object image information, the plurality of two-dimensional validation scores including at least one of a rendered match validator score and a template matching validator score; filtering detection hypotheses from the set of detection hypotheses according to the plurality of three-dimensional validation scores and the plurality of two-dimensional validation scores; and detecting an object in the scene according to unfiltered detection hypotheses remaining in the set of detection hypotheses after verification.

[0300] Embodiment 74 is a method according to embodiment 73, further comprising executing a robot motion planning procedure for removing the object from the scene and outputting commands for moving the robot to remove the object.

[0301] Embodiment 75 is a method according to embodiment 73, further comprising: generating a plurality of three-dimensional validation scores by obtaining a normal vector validator score; and the normal vector validator score being obtained by comparing a surface normal vector obtained from the object image information with a corresponding surface normal vector of a template object; and identifying a discrepancy between the surface normal vector and the corresponding surface normal vector to obtain a normal vector validator score.

[0302] Embodiment 76 is a method as described in embodiment 73, in which obtaining a hole match validator score includes comparing an object position obtained from the object image information with a structure of the template object, and identifying a discrepancy between the object position and the structure to identify invalidity of a hole due to an object position at a position corresponding to an empty volume in the structure of the template object.

[0303] Embodiment 77 is a method as described in embodiment 73, in which obtaining a rendered match validator score includes generating a two-dimensional rendering of an object in the scene, and comparing rendered edges of the two-dimensional rendering of the object with extracted edges of a template object to identify invalid edges.

[0304] Embodiment 78 is the method of embodiment 73, in which verifying each detection hypothesis in the set of detection hypotheses further includes comparing a corresponding object recognition template with scene elements other than the object that correspond to the template object.

[0305] Embodiment 79 is a method according to embodiment 73, in which filtering a detection hypothesis from the set of detection hypotheses includes comparing the occlusion validator score, the point cloud validator score, the hole match validator score, the normal vector validator score, the rendered match validator score, and the template matching validator score with corresponding thresholds, removing a detection hypothesis from the set of detection hypotheses if any of the three-dimensional validation score or the two-dimensional validation score cannot exceed the corresponding thresholds, and keeping the detection hypothesis in the set of detection hypotheses if the three-dimensional validation score and the two-dimensional validation score exceed all of the corresponding thresholds.

[0306] Embodiment 80 is a non-transitory computer-readable medium operable by at least one processing circuit via a communication interface configured to communicate with a robotic system, the non-transitory computer-readable medium having executable instructions for performing a method for validating detection hypotheses, the method including receiving object image information of an object in a scene, receiving a set of detection hypotheses each including a corresponding object recognition template representing a template object, and determining at least one of an occlusion validator score, a point cloud validator score, a hole match validator score, and a normal vector validator score based on a comparison of three-dimensional information of the object recognition templates of the detection hypotheses to three-dimensional information of the object image information corresponding to the object. A non-transitory computer-readable medium, comprising: performing a process for generating a plurality of three-dimensional validation scores; performing a process for generating a plurality of two-dimensional validation scores, including at least one of a rendered match validator score and a template matching validator score, based on comparing two-dimensional information of a corresponding object recognition template of the detection hypotheses and three-dimensional information of the object image information; performing a process for filtering detection hypotheses from a set of detection hypotheses according to the plurality of three-dimensional validation scores and the plurality of two-dimensional validation scores; detecting objects in the scene according to unfiltered detection hypotheses remaining in the set of detection hypotheses after verification; and outputting the detected objects in the scene to a robotic system.

[0307] Embodiment 81 is the method of embodiment 13, in which the three-dimensional space is substantially spherical and the object model is fixed to the center of the three-dimensional space.

[0308] Embodiment 82 is the method of embodiment 14, in which each camera position corresponds to a set of viewpoints, and each viewpoint of the set of viewpoints corresponds to a different camera rotation angle.

[0309]

[0046] Embodiment 83 is the method of embodiment 14, wherein the subset of the set of object recognition templates includes object recognition templates corresponding to different positions and viewpoints corresponding to different camera rotation angles.

[0310] Embodiment 84 is a method as described in embodiment 32, in which the digitally represented object is an object model and extracting two-dimensional measurement information and three-dimensional measurements is performed to generate a feature map of the object model at a selected viewpoint.

[0311] Embodiment 85 is the method of embodiment 32, in which the object information includes registration data of the object and the digitally represented object includes an object model.

[0312] Embodiment 86 is the method of embodiment 32, in which the object information includes at least one of two-dimensional image information and three-dimensional image information.

[0313] Embodiment 87 is a method as described in embodiment 36, in which gradient information is extracted at a plurality of gradient extraction locations of a digitally represented object, and surface normal vector information is extracted at a plurality of surface normal locations of the digitally represented object, and the plurality of gradient extraction locations are different from the plurality of surface normal locations.

[0314] Embodiment 88 is the method of embodiment 87, in which the multiple gradient extraction positions do not overlap with the multiple surface normal positions.

[0315] Embodiment 89 is a method according to embodiment 87, in which a plurality of gradient extraction locations are positioned at the edge of the digitally represented object, and a plurality of surface normal locations are positioned away from the edge of the digitally represented object.

[0316] Embodiment 90 is the method of embodiment 56, wherein the distance measure comprises a Euclidean distance measure.

[0317] Embodiment 91 is the method of embodiment 56, in which the distance measurement comprises a cosine distance between surface normal vectors associated with the set of adjusted template positions and the set of object positions.

[0318] Embodiment 92 is the method of embodiment 91, in which the cosine distance indicates the angle between the surface normal vectors, and the size of the angle correlates with the quality of the alignment.

[0319] Embodiment 93 is the method of embodiment 56, in which the distance measurement is a measurement from a first position in the set of adjusted template positions to a plane of a second position in the set of object positions.

[0320] Embodiment 94 is a method according to embodiment 73, in which the multiple three-dimensional validation scores include a point cloud validator score, and the point cloud validator score is obtained by comparing an object position obtained from the object image information with a surface of a template object, identifying discrepancies between the object position and the surface, and obtaining a point cloud validator score.

[0321]

[0046] Embodiment 95 is the method of embodiment 94, in which invalid object positions are identified according to discrepancies that place the object positions below the surface of the template object, and the point cloud validator score is based on the invalid object positions.

[0322] Embodiment 96 is a method according to embodiment 73, in which the multiple three-dimensional verification scores include an occlusion validator score, and the occlusion validator score is obtained by comparing an object position obtained from the object image information with a surface of a template object, identifying a discrepancy between the object position and the surface, and obtaining an occlusion validator score.

[0323]

[0081] Embodiment 97 is the method of embodiment 96, in which occlusion is identified according to a discrepancy that places the corresponding object position above or outside the surface of the template object, and the occlusion validator score is based on the occlusion.

[0324] Embodiment 98 is a method according to embodiment 78, in which comparing a corresponding object recognition template representing the estimated object with the scene elements includes determining whether an object corresponding to the template object is within the container.

Claims

1. at least one processing circuit configured to communicate with the robot and the camera; the at least one processing circuit is configured to execute instructions stored on a non-transitory computer-readable medium when one or more objects are or were within the field of view; The instruction: generating one or more detection hypotheses from object image information representing one or more objects in the scene based on an object recognition template representing a template object; and validating the one or more detection hypotheses based on the three-dimensional validation scores and / or the two-dimensional validation scores; the 3D verification score is generated based on comparing 3D information of the object recognition template and 3D information of the object image information; The two-dimensional verification score is generated based on comparing two-dimensional information of the corresponding object recognition template.

2. The computing system of claim 1 , wherein the three-dimensional validation score is based on at least one of an occlusion validator score, a three-dimensional information validator score, a hole matching validator score, and a normal vector validator score.

3. The three-dimensional information validator score is comparing object locations derived from the object image information with a surface of the template object; identifying discrepancies between the object location and the surface to obtain the three-dimensional information validator score; The computing system of claim 2 , wherein the computation is performed by:

4. The shielded validator score is: comparing object locations derived from the object image information with a surface of the template object; identifying discrepancies between the object positions and the surface to obtain the occlusion validator score; The computing system of claim 2 , wherein the computation is performed by:

5. The computing system of claim 1 , wherein the two-dimensional validation score is based on at least one of a rendered match validator score and a template matching validator score.

6. The computing system of claim 5 , wherein the rendered match validator score is calculated based on comparing edges of a two-dimensional rendering of the one or more objects to extracted edges of the template object.

7. Generating the one or more detection hypotheses comprises: identifying a set of template locations within a template object that correspond to a set of object locations in the object image information; adjusting the set of template positions to converge to the set of object positions; The computing system of claim 1 , comprising:

8. Obtaining the one or more detection hypotheses comprises: The computing system of claim 1 , further comprising identifying the object image information based on a comparison of template gradient information and template surface normal vector information of an object recognition template with object gradient information and object surface normal vector information extracted from the object image information.

9. generating one or more detection hypotheses from object image information representing one or more objects in the scene based on an object recognition template representing a template object; validating the one or more detection hypotheses based on the three-dimensional validation scores and / or the two-dimensional validation scores; Including, the 3D verification score is generated based on comparing 3D information of the object recognition template and 3D information of the object image information; The method, wherein the two-dimensional verification score is generated based on comparing two-dimensional information of the corresponding object recognition template.

10. The method of claim 9 , wherein the three-dimensional validation score is based on at least one of an occlusion validator score, a three-dimensional information validator score, a hole matching validator score, and a normal vector validator score.

11. The three-dimensional information validator score is comparing object locations derived from the object image information with a surface of the template object; identifying discrepancies between the object location and the surface to obtain the three-dimensional information validator score; The method of claim 10, wherein the calculated value is calculated by:

12. The shielded validator score is: comparing object locations derived from the object image information with a surface of the template object; identifying discrepancies between the object positions and the surface to obtain the occlusion validator score; The method of claim 10, wherein the calculated value is calculated by:

13. The method of claim 9 , wherein the two-dimensional validation score is based on at least one of a rendered match validator score and a template matching validator score.

14. The method of claim 13 , wherein the rendered match validator score is calculated based on comparing edges of a two-dimensional rendering of the one or more objects to extracted edges of the template object.

15. Generating the one or more detection hypotheses comprises: identifying a set of template locations within a template object that correspond to a set of object locations in the object image information; adjusting the set of template positions to converge to the set of object positions; 10. The method of claim 9, comprising:

16. Obtaining the one or more detection hypotheses comprises:

10. The method of claim 9, comprising identifying the object image information based on a comparison of template gradient information and template surface normal vector information of an object recognition template with object gradient information and object surface normal vector information extracted from the object image information.

17. A non-transitory computer-readable medium operable by at least one processing circuit via a communication interface configured to communicate with a robotic system, the non-transitory computer-readable medium being configured with executable instructions for performing a method for refining and validating a detection hypothesis, the method comprising: The method comprises: generating one or more detection hypotheses from object image information representing one or more objects in the scene based on an object recognition template representing a template object; validating the one or more detection hypotheses based on the three-dimensional validation scores and / or the two-dimensional validation scores; Including, the 3D verification score is generated based on comparing 3D information of the object recognition template and 3D information of the object image information; The two-dimensional verification score is generated based on comparing two-dimensional information of the corresponding object recognition template.

18. the three-dimensional validation score is based on at least one of an occlusion validator score, a three-dimensional information validator score, a hole matching validator score, and a normal vector validator score; The method of claim 17 , wherein the two-dimensional validation score is based on at least one of a rendered match validator score and a template matching validator score.

19. Generating the one or more detection hypotheses comprises: identifying a set of template locations within a template object that correspond to a set of object locations in the object image information; adjusting the set of template positions to converge to the set of object positions; 20. The method of claim 17, comprising:

20. Obtaining the one or more detection hypotheses comprises:

20. The method of claim 17, comprising identifying the object image information based on a comparison of template gradient information and template surface normal vector information of an object recognition template with object gradient information and object surface normal vector information extracted from the object image information.

Images