Method and computing system for object detection or robot interaction planning based on image information generated by a camera
The computing system enhances robotic object detection and interaction by using a camera to generate detailed structural information and refine object recognition templates, enabling accurate object grasping and movement in complex environments.
Patent Information
- Application Number
- JP2022003304
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-12-12
- Filing Date
- 2022-01-12
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-12-09
AI Technical Summary
Existing robotic systems struggle to accurately detect and interact with objects in complex environments, such as warehouses and retail spaces, due to limitations in object recognition and interaction planning based on image information.
A computing system that utilizes a camera mounted on a robot arm to generate multiple sets of image information, perform object detection by identifying object corners and types, and plan robot interactions based on refined structural information, using object recognition templates and pose refinement operations to enhance accuracy.
Enables precise object detection and interaction planning, allowing robots to grasp and move objects with high accuracy, even in cluttered environments, by leveraging detailed structural information and refined pose adjustments.
Smart Images

Figure 0007586436000001 
Figure 0007586436000002 
Figure 0007586436000003
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 946,973, filed December 12, 2019, entitled "ROBOTIC SYSTEM WITH GRIPPING MECHANISM," the entire contents of which are incorporated herein by reference.
[0002] The present disclosure relates to methods and computing systems for object detection or robot interaction planning based on image information generated by a camera. [Background technology]
[0003] As automation becomes more commonplace, robots are being used in more environments, such as warehousing and retail environments. For example, robots may be used to interact with merchandise or other objects in a warehouse. The robot's movements may be fixed or may be based on inputs, such as information generated by sensors in the warehouse. Summary of the Invention
[0004] One aspect of the disclosure relates to a method implemented by a computing system for performing object detection. In some instances, the computing system may include a non-transitory computer-readable medium having instructions that cause the computing system to perform the method. In an embodiment, the computing system may include a communications interface and at least one processing circuit. The communications interface is configured to communicate with (i) a robot having a robot arm and an end effector device disposed on or forming one end of the robot arm, and (ii) a camera mounted on the robot arm and having a camera field of view. The at least one processing circuit performs the following when the object is within the camera field of view: receiving first image information representative of at least a first portion of an object structure associated with the object, the first image information being generated by the camera when the camera is in a first camera pose in which the camera is directed toward the first portion of the object structure; generating or updating sensed structure information representative of the object structure associated with the object based on the first image information; identifying an object corner associated with the object structure based on the sensed structure information; outputting one or more camera positioning operation commands, when executed by the robot, to the robot arm to move the camera to a second camera pose in which the camera is directed toward the object corner; and generating or updating the second image information representative of the object structure based on the sensed structure information. receiving image information, where second image information is generated by the camera while the camera is in a second camera pose; updating the sensed structural information based on the second image information to generate updated sensed structural information; determining an object type associated with the object based on the updated sensed structural information; determining one or more robot interaction positions, which are one or more positions for interaction between the end effector device and the object, based on the object type; and outputting one or more robot interaction operation commands that cause an interaction at the one or more robot interaction positions, where the one or more robot interaction operation commands are generated based on the one or more robot interaction positions. [Brief description of the drawings]
[0005] [Figure 1A] 1 illustrates a system for processing image information consistent with embodiments herein. [Figure 1B] 1 illustrates a system for processing image information consistent with embodiments herein. [Figure 1C] 1 illustrates a system for processing image information consistent with embodiments herein. [Figure 1D] 1 illustrates a system for processing image information consistent with embodiments herein.
[0006] [Figure 2A] A block diagram illustrating a computing system configured to receive and process image information and / or perform object detection consistent with embodiments herein is provided. [Figure 2B] A block diagram illustrating a computing system configured to receive and process image information and / or perform object detection consistent with embodiments herein is provided. [Figure 2C] A block diagram illustrating a computing system configured to receive and process image information and / or perform object detection consistent with embodiments herein is provided. [Figure 2D] A block diagram illustrating a computing system configured to receive and process image information and / or perform object detection consistent with embodiments herein is provided.
[0007] [Figure 3A] 1 illustrates an environment having a robotic arm and end effector device for performing robotic interaction consistent with embodiments herein. [Figure 3B] 1 illustrates an environment having a robotic arm and end effector device for performing robotic interaction consistent with embodiments herein.
[0008] [Figure 4]1 shows a flow diagram illustrating an example method for generating a motion plan, according to embodiments herein.
[0009] [Figure 5A] 1 illustrates various aspects of generating image information representative of objects within a camera field of view, according to embodiments herein. [Figure 5B] 1 illustrates various aspects of generating image information representative of objects within a camera field of view, according to embodiments herein. [Figure 5C] 1 illustrates various aspects of generating image information representative of objects within a camera field of view, according to embodiments herein.
[0010] [Figure 6] 1 illustrates sensed structural information based on image information, according to embodiments herein.
[0011] [Figure 7] 1 illustrates an aspect of identifying object corners according to embodiments herein.
[0012] [Figure 8A] 1 illustrates various aspects of generating image information representative of objects within a camera field of view, according to embodiments herein. [Figure 8B] 1 illustrates various aspects of generating image information representative of objects within a camera field of view, according to embodiments herein. [Figure 8C] 1 illustrates various aspects of generating image information representative of objects within a camera field of view, according to embodiments herein.
[0013] [Figure 9] 1 illustrates sensed structural information based on image information, according to embodiments herein.
[0014] [Figure 10A] 1 illustrates an object recognition template according to embodiments herein. [Figure 10B] 1 illustrates an object recognition template according to embodiments herein.
[0015] [Figure 11A] 1 illustrates a comparison of sensed structural information with a set of object recognition templates according to embodiments herein. [Figure 11B] 1 illustrates a comparison of sensed structural information with a set of object recognition templates according to embodiments herein.
[0016] [Figure 12A] 1 illustrates a comparison of sensed structural information with a set of object recognition templates according to embodiments herein. [Figure 12B] 1 illustrates a comparison of sensed structural information with a set of object recognition templates according to embodiments herein.
[0017] [Figure 13A] 1 illustrates various aspects of filtering operations on a candidate set of object recognition templates or model-orientation combinations according to embodiments herein. [Figure 13B] 1 illustrates various aspects of filtering operations on a candidate set of object recognition templates or model-orientation combinations according to embodiments herein. [Figure 13C] 1 illustrates various aspects of filtering operations on a candidate set of object recognition templates or model-orientation combinations according to embodiments herein.
[0018] [Figure 14] 1 illustrates various aspects of a pose refinement operation, according to embodiments herein.
[0019] [Figure 15A] 1 illustrates various aspects of determining an error value according to embodiments herein. [Figure 15B] 1 illustrates various aspects of determining an error value according to embodiments herein. [Figure 15C] 1 illustrates various aspects of determining an error value according to embodiments herein.
[0020] [Figure 16] 1 illustrates how an error value may be determined according to an embodiment of the present disclosure.
[0021] [Figure 17A] 1 illustrates various aspects of determining a grip position of a robot, according to embodiments herein. [Figure 17B] 1 illustrates various aspects of determining a grip position of a robot, according to embodiments herein. [Figure 17C] 1 illustrates various aspects of determining a grip position of a robot, according to embodiments herein. [Figure 17D] 1 illustrates various aspects of determining a grip position of a robot, according to embodiments herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0022] One aspect of the disclosure relates to performing object detection on objects within a camera's field of view (also referred to as a camera field of view). For example, the objects may be boxes, boxes, crates, or other containers within a warehouse, retail space, or other facility. In embodiments, performing object detection may involve determining attributes of the objects, such as an object type associated with the object. One aspect of the disclosure relates to planning robotic interactions based on information obtained by performing object detection. Robotic interactions may involve a robot engaging an object within the camera field of view, such as an interaction in which the robot grasps or otherwise picks up the object, and moving the object to a destination location (e.g., as part of an unpalletizing operation).
[0023] In an embodiment, object detection may be performed based on multiple sets of image information generated by the camera, where the multiple sets of image information may represent multiple views or viewpoints from which the camera senses or otherwise generates image information representing an object in the camera's field of view. For example, the multiple sets of image information may include a first set of image information representing a top view of the object, and a second set of image information representing a perspective view of the object. In some implementations, the top view of the object may be used to perform general detection, which may involve obtaining image information having sufficient detail to identify an object corner of the object. The camera may be moved or otherwise positioned to point at the identified object corner, and a second set of image information representing the perspective view may be generated when the camera is pointed at the object corner. In some instances, the second image information may include a higher level of detail about the structure of the object compared to the first image information. In such an example, the second image information may be used to refine a description of an estimate of the structure of the object. In some implementations, the first image information and the second image information may be used to sense and generate structural information, such as a global point cloud, that represents the structure of the object.
[0024] In an embodiment, performing object detection may involve comparing the sensed structural information with a set of object recognition templates, or more specifically, a set of respective object structure models (e.g., CAD models) described by the set of object recognition templates. The comparison may be used, for example, to select one of the object recognition templates, which may be associated with an object type for the object. In some instances, the comparison may take into account different orientations of the object structure models. In such an example, the structural information may be compared with a set of model-orientation combinations, each of which may include an object structure model and an orientation of the object structure model.
[0025] In embodiments, the selection of an object recognition template or model-orientation combination may involve the calculation of a set of error values. Each of the error values may indicate a respective degree of deviation between the sensed structural information and the object structure model in one of the object recognition template or model-orientation combinations. More specifically, each of the error values may indicate how well or poorly a particular object structure model is explained or supported by the sensed structural information (e.g., a global point cloud). In some instances, the selected object recognition template may have the lowest error value in the set of error values.
[0026] In an embodiment, a filtering operation may be performed to remove certain object recognition templates or model-orientation combinations from being considered potential matches with the sensed structural information. For example, the sensed structural information may define an estimated region of space occupied by an object in the camera field of view. In such an implementation, the filtering operation may involve determining whether any of the object recognition templates or model-orientation combinations have an object structure model that does not substantially fall within the estimated region. If such an object structure model exists, the object recognition template or model-orientation combination associated with the object structure model may be filtered out.
[0027] In an embodiment, a pose refinement operation may be performed to adjust the object structure model to more closely match the sensed structure information. In some instances, the object structure model may describe various physical features of the object structure, and more specifically, may include pose information that describes the pose of the physical features. In such an example, the pose refinement operation may involve adjusting the pose information, which may change the orientation and / or position of the various physical features described by the object structure model.
[0028] In an embodiment, an object type associated with an object may be used to plan a robot interaction with the object. For example, an object type may be associated with a particular object design, which may include a physical design and / or a visual design, for a type or class of object. In some implementations, a physical design, such as a physical shape or size of an object structure, may be used to plan the robot interaction. In an embodiment, if the robot interaction involves a robot grasping an object, planning the robot interaction may involve determining one or more grasp locations on the object at which the robot should grasp the object. In some instances, if the determination of the object type is based on a selection of an object recognition template associated with the object type, one or more robot grasp locations may be determined based on the object recognition template, or more specifically, based on an object structure model described by the object recognition template.
[0029] FIG. 1A illustrates a system 1000 for performing object detection and / or creating a robot interaction plan based on image information. More specifically, the system 1000 may include a computing system 1100 and a camera 1200. In this example, the camera 1200 may be configured to generate image information that describes or otherwise represents an environment in which the camera 1200 is located, i.e., more specifically, represents an environment in a field of view (also referred to as a camera field of view) of the camera 1200. The environment may be, for example, a warehouse, a manufacturing plant, a retail space, or some other facility (the term "or" is used interchangeably with "and / or" in this disclosure). In such a case, the image information may represent objects located in such a facility, such as containers (e.g., boxes) that hold various items. The system 1100 may be configured to receive and process the image information, such as by performing object detection based on the image information. Object detection may involve, for example, determining a type of object (also referred to as an object type) for an object in the camera field of view. In some instances, the computing system may plan the robot's interaction based on the object type. The robot's interaction may involve, for example, the robot gripping, seizing, or otherwise picking up or engaging the object. For example, if the object is a container, the robot's interaction may involve the robot picking up the container by gripping or grabbing it and moving the container to a destination location. The computing system 1100 and the camera 1200 may be located in the same facility or may be located remotely from each other. For example, the computing system 1100 may be part of a cloud computing platform hosted in a data center remote from a warehouse or retail space and may be in communication with the camera 1200 via a network connection.
[0030] In an embodiment, the camera 1200 may be a 3D camera (also referred to as a spatial structure sensing camera or device) configured to generate 3D image information (also referred to as spatial structure information) regarding the environment in the field of view of the camera. In an embodiment, the camera 1200 may be a 2D camera configured to generate 2D image information, or more specifically, a 2D image describing the visual appearance of the environment in the field of view of the camera. In some instances, the camera 1200 may be a combination of a 3D camera and a 2D camera configured to generate 3D image information and 2D image information. The 3D image information may include depth information describing the respective depth values of various positions relative to the camera 1200, such as the positions on the surfaces of various objects in the field of view of the camera 1200, or more specifically, the positions on the surfaces of the objects. The depth information in this example may be used to estimate how the objects are spatially arranged in three-dimensional (3D) space. In some instances, the 3D image information may include a point cloud describing the positions on one or more surfaces of objects in the field of view of the camera. More specifically, the 3D image information may describe various locations on a structure of an object (also referred to as an object structure).
[0031] As mentioned above, the camera 1200 may be a 3D camera and / or a 2D camera. The 2D camera may be configured to generate a 2D image, such as a color image or a grayscale image, or other 2D image information. The 3D camera may be a depth-sensing camera, such as a time-of-flight (TOF) camera or a structured light camera, or any other type of 3D camera. In some cases, the 2D camera and / or the 3D camera may each include an image sensor, such as a charge-coupled device (CCD) sensor and / or a complementary metal-oxide semiconductor (CMOS) sensor. In an embodiment, the 3D camera may include a laser, a LIDAR device, an infrared device, a light / dark sensor, a motion sensor, a microwave detector, an ultrasonic detector, a radar detector, or any other device configured to capture or otherwise generate 3D image information.
[0032] In an embodiment, the system 1000 may be a robotic handling system for interacting with various objects in the environment of the camera 1200. For example, FIG. 1B illustrates a robotic handling system 1000A, which may be an embodiment of the system 1000 of FIG. 1A. The robotic handling system 1000A may include a computing system 1100, a camera 1200, and a robot 1300. In an embodiment, the robot 1300 may be used to interact with one or more objects, such as boxes, crates, cases, bins, or containers, in the environment of the camera 1200. For example, the robot 1300 may be configured to pick up containers from one location and move them to another location. In some scenarios, the robot 1300 may be used to perform an unpalletizing operation, where stacked containers are taken down and moved, for example, to a conveyor belt, or a palletizing operation, where containers are stacked on a pallet to prepare for transportation.
[0033] In some instances, the camera 1200 may be separate from the robot 1300. For example, the camera 1200 in such instances may be a fixed camera mounted on a ceiling or some other location of a warehouse or other facility. In some instances, the camera 1200 may be part of the robot 1300 or otherwise attached to the robot 1300, providing the robot 1300 with the ability to move the camera 1200. For example, FIG. 1C depicts a system 1000B (which may be an embodiment of the system 1000) including the computing system 1100, camera 1200, and robot 1300 of FIG. 1B, where the robot 1300 has a robot arm 1400 and an end effector device 1500. The end effector device 1500 may be attached to, disposed on, or form an end of the robot arm 1400. In the embodiment of FIG. 1C, the end effector device 1500 may be movable by the movement of the robot arm 1400. The camera 1200 in this example may be mounted on the end effector device 1500 or may otherwise be attached to the end effector device 1500. If the end effector device 1500 is a robot hand (e.g., a gripper device), the camera 1200 may be referred to as a handheld camera. By attaching the camera 1200 to the end effector device 1500, the robot 1300 may be able to move the camera 1200 to different poses (also referred to as camera poses) by movement of the robot arm 1400 and / or the end effector device 1500. For example, the camera 1200 may be positioned by the end effector device 1500 to have various camera poses, as discussed in more detail below. The camera 1200 may generate respective sets of image information at these camera poses. In such examples, each set of image information may represent a different perspective or vantage point from which the camera 1200 and / or robot 1300's environment is sensed, and such image information may facilitate accurate object detection and interaction planning for the robot.
[0034] 1A-1C may form or be part of a robot control system (also referred to as a robot controller) that is part of the robot manipulation system 1000A / 1000B. The robot control system may be, for example, a system configured to generate movement or other commands for the robot 1300. In such an embodiment, the computing system 1100 may be configured to generate such commands based on image information generated by the camera 1200, for example.
[0035] In an embodiment, the computing system 1100 may form or be part of a vision system. The vision system may be, for example, a system that generates visual information describing an environment in which the robot 1300 is located, or more specifically, describing an environment in which the camera 1200 is located. In some implementations, the visual information may include image information as discussed above. In some implementations, the visual information may describe object types or other attributes of objects in the environment of the camera 1200 and / or the robot 1300. In such implementations, the computing system 1100 may generate such visual information based on the image information. If the computing system 1100 forms a vision system, the vision system may be part of the robot control system as discussed above, or may be separate from the robot control system. If the vision system is separate from the robot control system, the vision system may be configured to output information describing the environment in which the robot 1300 is located, such as information describing object types or other attributes of objects in the environment of the camera 1200 and / or the robot 1300. Information determined by the vision system may be output to a robot control system, which may receive such information from the vision system and control the movement of the robot 1300 based on the information.
[0036] In an embodiment, when the computing system 1100 is configured to generate one or more operation commands, the operation commands may include, for example, a camera placement operation command and / or a robot interaction operation command. In this embodiment, the camera placement operation command controls the placement of the camera 1200, and more specifically, is an operation command used to move or otherwise position the camera 1200 in a particular camera pose on the robot 1300. The camera pose may include a combination of a particular camera position and a particular camera orientation. The robot interaction operation command (also referred to as an object interaction operation command) may be used to control the interaction between the robot 1300, more specifically, its end effector device, and one or more objects, such as stacked containers in a warehouse. For example, the robot interaction operation command may cause the end effector device 1500 on the robot arm 1400 of the robot 1300 in FIG. 1C to move closer to one of the containers, and cause the end effector device 1500 to grasp or otherwise pick up the container, and then cause the robot arm 1400 to move the container to a specific or calculated target position. When the end effector device 1500 has a gripper member, the robot interaction operation command may include, in some implementations, a gripper member positioning command that causes the gripper member to be placed or otherwise positioned at a position where it grasps a portion (e.g., an edge) of the container, causing the gripper member to move relative to the rest of the end effector device.
[0037] In an embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 through a dedicated wired communication interface, such as an RS-232 interface, a universal serial bus (USB) interface, and / or a direct wired connection, such as a connection provided through a local computer bus, such as a peripheral component interconnect (PCI) bus. In some implementations, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 through a wireless communication interface. In an embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 through a network. The network may be any type and / or form of network, such as a personal area network (PAN), a local area network (LAN), e.g., an intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The network may utilize different technologies and layers or stacks of protocols including, for example, the Ethernet protocol, the Internet Protocol suite (TCP / IP), Asynchronous Transfer Mode (ATM) technology, Synchronous Optical Networking (SONET) protocol, or Synchronous Digital Hierarchy (SDH) protocol.
[0038] In an embodiment, the computing system 1100 may communicate directly with the camera 1200 and / or the robot 1300, or may communicate via an intermediate storage device, or more broadly, an intermediate non-transitory computer-readable medium. For example, FIG. 1D depicts a system 1000C (which may be an embodiment of systems 1000 / 1000A / 1000B) that includes an intermediate non-transitory computer-readable medium 1600 for storing information generated by the camera 1200, the robot 1300, and / or the computing system 1100. Such an intermediate non-transitory computer-readable medium 1600 may be external to the computing system 1100, and may, for example, serve as an external buffer or repository for storing image information generated by the camera 1200, commands generated by the computing system 1100, and / or other information (e.g., sensor information generated by the robot 1300). For example, if the intermediate non-transitory computer readable medium 1600 is used to store image information generated by the camera 1200, the computing system 1100 may read or otherwise receive the image information from the intermediate non-transitory computer readable medium 1600. Examples of the non-transitory computer readable medium 1600 include electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. The non-transitory computer readable medium may form, for example, a computer diskette, a hard disk drive (HDD), a solid state drive (SDD), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read only memory (CD-ROM), a digital versatile disk (DVD), and / or a memory stick.
[0039] As described above, image information generated by camera 1200 may be processed by computing system 1100. In an embodiment, computing system 1100 may include or be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop, etc.), a smartphone, a tablet computing device, and / or any other computing system. In an embodiment, all of the functionality of computing system 1100 may be performed as part of a cloud computing platform. Computing system 1100 may be a single computing device (e.g., a desktop computer) or may include multiple computing devices.
[0040] 2A provides a block diagram illustrating an embodiment of a computing system 1100. The computing system 1100 includes at least one processing circuit 1110 and a non-transitory computer-readable medium (or media) 1120. In an embodiment, the processing circuit 1110 includes one or more processors, one or more processing cores, a programmable logic controller ("PLC"), an application specific integrated circuit ("ASIC"), a programmable gate array ("PGA"), a field programmable gate array ("FPGA"), any combination thereof, or any other processing circuit. In an embodiment, the non-transitory computer-readable medium 1120 that is part of the computing system 1100 may be in place of or in addition to the intermediate non-transitory computer-readable medium 1600 discussed above. The non-transitory computer-readable medium 1120 may be a storage device such as an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, such as a computer diskette, a hard disk drive (HDD), a solid state drive (SSD), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, any combination thereof, or any other storage device. In some instances, the non-transitory computer-readable medium 1120 may include multiple storage devices. In certain cases, the non-transitory computer-readable medium 1120 is configured to store image information generated by the camera 1200. The non-transitory computer-readable medium 1120 may alternatively or additionally store computer-readable program instructions that, when executed by the processing circuit 1110, cause the processing circuit 1110 to perform one or more of the techniques described herein, such as the operations described with respect to FIG.
[0041] FIG. 2B depicts an embodiment of the computing system 1100, the computing system 1100A, including a communication interface 1130. The communication interface 1130 (also referred to as a communication component or communication device) may be configured to receive image information generated by, for example, the camera 1200 of FIGS. 1A-1D. The image information may be received via an intermediate non-transitory computer-readable medium 1600 or a network as discussed above, or via a more direct connection between the camera 1200 and the computing system 1100 / 1100A. In an embodiment, the communication interface 1130 may be configured to communicate with the robot 1300 of FIGS. 1B and 1C. If the computing system 1100 is not part of a robot control system, the communication interface 1130 of the computing system 1100 may be configured to provide communication between the computing system 1100 and the robot control system. The communication interface 1130 may include or be a communication circuit configured to communicate via, for example, a wired or wireless protocol. By way of example, the communications circuitry may include an RS-232 port controller, a USB controller, an Ethernet controller, a Bluetooth controller, a PCI bus controller, any other communications circuitry, or a combination thereof.
[0042] In an embodiment, the processing circuit 110 may be programmed with one or more computer-readable program instructions stored on a non-transitory computer-readable medium 1120. For example, FIG. 2C illustrates an embodiment of computing system 1100 / 1100A, computing system 1100B, in which the processing circuit 1110 is programmed with one or more modules, including an object detection module 1121 and a robot interaction planning module 1122.
[0043] In an embodiment, the object detection module 1121 may be configured to determine information associated with an object (e.g., a container) that is currently within or was within the camera field of view of the cameras 1200 of FIGS. 1A-1D. The information may describe characteristics of the object, such as the type or class of object to which the object belongs (also referred to as the object type associated with the object), the size of the object, the shape of the object (also referred to as the object size and object shape, respectively), and / or any other characteristics of the object. In some implementations, the object detection module 1121 may be configured to perform object recognition by comparing image information representing the object to an object recognition template, as discussed in more detail below.
[0044] In an embodiment, the robot interaction planning module 1122 may be configured to determine how the robot 1300 of FIGS. 1B and 1C should interact with objects in the environment of the robot 1300 and / or the camera 1200, i.e., more specifically, objects that are or were in the camera field of view. The interaction may involve, for example, the robot 1300 grasping or otherwise picking up the object and moving the object to a destination location. In some instances, the robot interaction planning module 1122 may be configured to generate a motion plan for performing or executing the interaction. The motion plan for interacting with the object may be generated based on information determined by the object detection module 1121, such as, for example, an object type associated with the object. In an embodiment, the motion plan may identify one or more gripping locations or gripping portions of the object where the robot 1300 should grasp the object. The motion plan may further move at least a portion of the robot 1300 (e.g., the end effector device 1500) to the one or more gripping locations. In some instances, if the robot 1300, or more specifically, the end effector unit 1500, includes one or more grippers, the robot interaction planning module 1122 may be configured to plan the manipulation of the one or more grippers. More specifically, if the one or more grippers can transition from an open state to a closed state to grasp or otherwise engage an object, and transition from a closed state to an open state to release the object, the robot interaction planning module 1122 may be configured to control or otherwise determine when the one or more grippers transition between the open state and the closed state. In some implementations, the motion plan may include or describe a trajectory that the robot 1300, or a portion thereof (e.g., the end effector unit 1500), follows after the robot 1300 grasps or otherwise picks up an object. The trajectory may cause the robot 1300 to move the object to a desired destination location. It will be understood that the functionality of the modules discussed herein is representative and not limiting.
[0045] In various embodiments, the terms "computer readable instructions" and "computer readable program instructions" are used to describe software instructions or computer code configured to perform various tasks and operations. In various embodiments, the term "module" refers broadly to a collection of software instructions or code configured to cause the processing circuitry 1110 to perform one or more functional tasks. The modules and computer readable instructions may be described as performing various operations or tasks when the processing circuitry or other hardware components execute the modules or computer readable instructions.
[0046] In an embodiment, as shown in FIG. 2D, the non-transitory computer-readable medium 1120 may store or otherwise include object detection information 1126, which may be generated by the computing system 1100C (which may be an embodiment of the computing system 1100 / 1100A / 1100B). In an embodiment, the object detection information 1126 may describe one or more objects in the camera field of view of the camera 1200, or more specifically, one or more objects in the environment of the camera 1200 and / or the robot 1300. For example, the object detection information 1126 may include sensed structure information and / or object type information. The sensed structure information (also referred to as measured structure information) may be or include information (e.g., a comprehensive point cloud) describing the structure of one or more objects, which may also be referred to as a physical structure or an object structure. The sensed structure information may be based on depth information or other image information sensed by the camera 1200 or another sensing device. In other words, the sensed structural information may be structural information generated based on values (e.g., depth values) sensed or measured by camera 1200. In an embodiment, the object type information may describe an object type associated with an object in an environment of camera 1200 and / or robot 1300. In some instances, the object type may be associated with an object recognition template, discussed below, and the object type information may include or identify the object recognition template.
[0047] In an embodiment, computing system 1100 may have access to one or more object recognition templates (also referred to as object templates), which may be stored as part of object recognition template information 1128 on non-transitory computer-readable medium 1120, as shown in FIG. 2D, may be stored on non-transitory computer-readable medium 1600, and / or may be stored on another device. In some implementations, one or more object recognition templates may have been manually generated or received (e.g., downloaded) by computing system 1100 / 1100A / 1100B / 1100C via communication interface 1300 or in some other manner. In some implementations, one or more object recognition templates may have been generated as part of an object registration process performed by computing system 1100 / 1100A / 1100B / 1100C or by another device. Templates are discussed in more detail in U.S. Patent Application No. 16 / 991,466 (Attorney Docket No. MJ0054-US / 0077-0012US1) and U.S. Patent Application No. 16 / 991,510 (Attorney Docket No. MJ0051-US / 0077-0011US1), the entire contents of which are incorporated herein by reference.
[0048] In an embodiment, one or more object recognition templates (also referred to as one or more object templates) may each describe an object design associated with a type or class of object. The object design may include, for example, a visual design that describes or defines the appearance of an object associated with the object type or class (also referred to as an object type) and / or a physical design that describes or defines the structure of an object associated with the object type. For example, if the object design described by the object recognition template is, more specifically, a container design associated with a particular container type, the object recognition template may be, for example, a container template that describes the visual design and / or the physical design associated with the container type. In some implementations, the object recognition template may include visual description information (also referred to as an object appearance description) that describes the visual design and / or may include an object structure description (also referred to as structure description information) that describes the physical design.
[0049] In some instances, the visual description information may include or describe one or more feature descriptors that may represent visual features, visual patterns, or other visual details (e.g., logos or pictures) that form a visual design. In some instances, the object structure description may describe an object size, object shape or contour, and / or some other aspect of an object's structure associated with a particular object type. For example, the object structure description may include values describing an object size (e.g., a combination of length, width, and / or height) associated with the object type, a computer-aided design (CAD) file describing the object structure associated with the object type, and / or a point cloud describing the contours of the object structure. More specifically, the point cloud may include a plurality of coordinates describing a plurality of respective locations, e.g., on one or more surfaces of the object structure. In some implementations, one or more object recognition templates described by the object recognition template information 1128 may be compared to the sensed structure information, e.g., as discussed above, to determine which object recognition template best matches the sensed structure information. Such a comparison may be part of an object recognition operation. As discussed in more detail below, object recognition operations may be used to determine an object type associated with an object within the camera field of view of camera 1200 of Figures 1A-1D. Computing system 1100 / 1100A / 1100B / 1100C or another computing system may be configured to use the object's object type to plan a robot interaction with the object.
[0050] 3A and 3B show an example of an environment in which object detection and / or robot interaction may occur. More specifically, the environment includes a computing system 1100, a camera 3200 (which may be an embodiment of camera 1200 of FIGS. 1A-1D), and a robot 3300 (which may be an embodiment of robot 1300). In this embodiment, the robot 3300 may include a robot arm 3400 and an end effector device 3500. In an embodiment, the end effector device 3500 may form one end of the robot arm 3400 or be attached to one end of the robot arm 3400.
[0051] In the example of FIG. 3A, the robot 3300 may operate to move an end effector device 3500, via the robot arm 3400, towards one or more objects, such as cases, crates, or other containers, stacked and disposed on a pallet. The end effector device 3500 may further be capable of engaging (e.g., grasping) at least one of the one or more objects and moving the object from the pallet to another location (e.g., as part of an unpalletizing operation). More specifically, FIGS. 3A and 3B depict an environment having a stack 3710 of objects, or more specifically, stacked containers. In some scenarios, some or all of the containers may hold smaller objects (which may be referred to as smaller items), as shown in FIG. 3B. Stack 3710 in FIGS. 3A and 3B may include at least objects 3711-3719 and 3731-3733, while end effector device 3500 may be used to grasp or otherwise pick up one of the objects in stack 3710, such as object 3711, and move the object from stack 3710 to a destination location, such as a location on conveyor 3800 in FIG. 3A. To pick up object 3711, end effector device 3500 may move and tilt to align with object 3711. In the environment depicted in FIGS. 3A and 3B, the objects on the pallet may have physical structures (also more simply referred to as structures) that form a 3D pattern on at least one of their outer side surfaces. For example, the 3D pattern may be a pattern of ridges (also referred to as a ridge pattern) protruding from the outer side surface. By way of example, FIG. 3A depicts ridge pattern 3711A on the outer side surface of object 3711. In some scenarios, an object on a pallet may have visual details, such as a logo or other visual pattern, that form a 2D pattern on its outer side surface. In some instances, when the object is a container, the object may include a container edge. As discussed in more detail below, the ridge pattern and / or the container edge may be used to determine a robot interaction location, such as a location where an end effector device (e.g., 3500) of the robot should grasp the container.
[0052] In an embodiment, the end effector device 3500 may include one or more gripper members. For example, the end effector device 3500 may include a mounting plate or other mounting structure and include multiple gripper members mounted on or otherwise attached to a surface (e.g., a bottom surface) of the mounting structure. In some implementations, the camera 3200 may be mounted on or otherwise attached to an opposing surface (e.g., a top surface) of the mounting structure. In some instances, the multiple gripper members may include at least a first gripper member movable (e.g., slidable) along a first axis and a second gripper member movable along a second axis perpendicular to the first axis. The first axis may be, for example, parallel to a first edge of the mounting structure, while the second axis may be, for example, parallel to a second edge of the mounting structure. In some instances, the multiple gripper members may further include a third gripper member disposed at a location where the first axis and the second axis intersect. Such a location may be, for example, near a corner of the mounting structure.
[0053] In some implementations, each of the gripper members may have a respective gripper body formed by or attached to a respective gripper finger assembly. The gripper finger assemblies may be used to grip an object by clamping or pinching a portion of the object (e.g., a container) around the periphery, such as a portion of a lip forming an outer edge of the container. In one example, the gripper finger assembly may include two components, also referred to as gripper fingers, that are movable relative to each other. The two gripper fingers may form a chuck or clamp that is movable toward each other to transition to a closed state in which the two gripper fingers grip a portion of the object or to tighten a grip around the object. The two gripper fingers may also be movable away from each other to transition to an open state in which the gripper fingers loosen or release their grip. End effector devices and gripper members are discussed in more detail in US application Ser. No. 17 / 084,272 (Attorney Docket No. MJ0058-US / 0077-0014US1), the entire contents of which are incorporated herein by reference.
[0054] As mentioned above, one aspect of the present application relates to performing object detection, which may involve determining an object type for an object in a camera field of view. The object type may be used to plan a robot interaction with the object, such as an interaction in which the robot grasps the object and moves the object from a current position to a destination position. FIG. 4 depicts a flow diagram of an example method 4000 of performing object detection and / or planning a robot interaction. The method 4000 may be performed, for example, by the computing system 1100 of FIGS. 2A-2D or 3A, or more specifically, by at least one processing circuit 1110 of the computing system 1100. In some scenarios, the at least one processing circuit 1100 may perform the method 4000 by executing instructions stored on a non-transitory computer readable medium, such as the non-transitory computer readable medium 1120. For example, the instructions may cause the processing circuit 1110 to execute an object detection module 1121 and a robot interaction planning module 1122, which may perform some or all of the steps of the method 4000. In an embodiment, method 4000 may be performed in an environment in which computing system 1100 is currently in communication with a robot and camera, such as robot 3300 and camera 3200 of Figures 3A and 3B, or with any other robot discussed in this disclosure. For example, computing system 1100 may perform method 4000 when an object is currently in or was in the camera field of view of camera 3200. In some scenarios, the camera (e.g., 3200) may be mounted on an end effector device (e.g., 3500) of the robot (e.g., 3300), as shown in Figure 3A. In other scenarios, the camera may be mounted elsewhere and / or may be fixed.
[0055] In an embodiment, the method 4000 of FIG. 4 may begin or otherwise include step 4002 in which the computing system 1100 receives (e.g., via the object detection module 1121) first image information (also referred to as a first set of image information) representing at least a first portion of an object structure associated with an object in the field of view of the camera (also referred to as the camera field of view). For example, FIG. 5A depicts a scenario in which a group 3720 of objects 3721, 3722 is in the camera field of view 3202 of the camera 3200. Each of the objects 3721, 3722 may be, for example, a box, crate, case, bin, or other container. The group 3720 of objects 3721, 3722 may be located on another object 3728, such as a pallet, which may also be at least partially located within the camera field of view 3202. In some scenarios, the pallets 3728 may be used with a wide variety of stacking or arrangement configurations to stack or otherwise arrange containers or other objects, which may have a wide variety of sizes (e.g., a wide variety of length, width, and height values).
[0056] In an embodiment, the first image information received by the computing system 1100 may be generated by the camera (e.g., 3200) when the camera is in or has a first camera pose, such as the camera pose shown in FIG. 5A. The camera pose may refer to the position and orientation of the camera (e.g., 3200). In some scenarios, the camera pose may affect the view or viewpoint of the camera (e.g., 3200). For example, the first camera pose depicted in FIG. 5A may involve the camera 3200 having a position above a group 3720 of objects 3721, 3722 and an orientation in which the camera 3200 faces a first portion, or more specifically, the top (e.g., top surface) of the objects 3721, 3722. In some circumstances, the orientation of the first camera pose may be associated with the camera 3200 having a top view of the objects. For example, the first camera pose may involve the camera 3200 having an orientation in which an image sensor of the camera 3200 faces directly toward the top of the object 3721, 3722 and / or a focal axis of one or more lenses of the camera 3200 is vertical or substantially vertical. In some scenarios, the camera 3200 may be positioned directly above the object 3721, 3722 and may be pointed directly toward a first portion (e.g., the top) of the object 3721, 3722.
[0057] In an embodiment, the first image information of step 4002 may be used for a rough detection phase in which the computing system 1100 may determine a relatively incomplete or simpler description or estimate of the object structure. For example, the description or estimate of the object structure may be incomplete because the first image information may describe a first portion of the object structure (e.g., a top portion) but may not be able to describe or may only limitedly describe other portions of the object structure (e.g., side portions). In some instances, the rough detection phase may further involve positioning the camera (e.g., 3200) far enough away from the object structure to fit the entire object structure within the camera field of view. In such an example, the first image information generated by the camera may be less detailed than image information generated when the camera is closer to the object structure. Thus, the estimate or description of the object structure obtained based on the first image information may be simpler in its level of detail. As discussed in more detail below, this estimate or description may be, for example, a wide-area point cloud or some other sensed structural information. The sensed structure information generated using the rough detection phase may be used to identify object corners of the object structure and receive second image information associated with the object corners. In some implementations, the second image information may be more detailed and / or may supplement the first image information. Thus, the second image information may be used to refine a description or estimate of the object structure.
[0058] As discussed in more detail below, steps of method 4000 may be performed to facilitate robot interaction with individual objects of group 3720 of objects 3721, 3722, such as object 3722. In such examples, a particular object that is the target of robot interaction may be referred to as a target object. In some scenarios, steps of method 4000 (e.g., steps 4004-4016) may be performed multiple times or in multiple iterations to facilitate robot interaction with multiple target objects.
[0059] As discussed above, the first image information may represent a particular view of the group 3720 of objects 3721, 3722, or, more specifically, a particular view of each of the objects 3721, 3722. In the example of Figure 5A, the first image information may represent a top view of the objects 3721, 3722 because the first image information may be generated when the camera 3200 has a first camera pose as shown in Figure 5A, in which the camera 3200 is above the objects 3721, 3722 and points toward the top (e.g., top end or top side) of each of the objects 3721, 3722.
[0060] In the example of FIG. 5A, the objects 3721, 3722 may each be a crate or other open-top container having one or more walls surrounding a bottom interior surface of the container. The one or more walls may form a rim at the top end of the container. In such an example, the top view of the objects 3721 / 3722 may include a view of the rim surface (also referred to as the rim surface) for the objects 3721 / 3722. For example, FIG. 5B depicts an example where the first image information includes 3D image information 5720 (also referred to as spatial structure information) that describes the structure of the group 3720 of objects 3721, 3722. In such an embodiment, the camera 3200 that generated the 3D image information 5720 may be a 3D camera. The 3D image information 5720 in this example describes the structure (also referred to as the object structure) of object 3721 and may describe the object structure for object 3722, and more specifically may represent a top view of the object structures for objects 3721, 3722.
[0061] In an embodiment, the 3D image information 5720 may include depth information, such as a depth map, describing respective depth values of a portion or more of the object structures for the objects 3721, 3722 relative to a reference point, such as a point where the camera (e.g., 3200) is located when the camera generates the 3D image information 5720 or other image information used in step 4002. More specifically, the depth information may describe respective depth values for a plurality of positions (also referred to as a plurality of points) on one or more surfaces of the object structure for the object 3721 and / or the object structure for the object 3722. In the example of FIG. 5B, the 3D image information 5720 may include image portions 5721, 5722, and 5728 describing depth values of the objects 3721, 3722, and 3728, respectively. More specifically, the image portion 5728 describes the depth values of the positions 37281 to 3728 on the top surface of the object 3728 (e.g., a pallet). n Further in this example, object 3721 may be a container having a rim and a bottom interior surface. Image portion 5721 is located at positions 3721A1 to 3721A2 on the rim surface (also referred to as the rim surface) of object 3721. n 3721B1 to 3721B2 on the bottom inner surface of the object 3721, including their respective depth values. n Similarly, image portion 5722 may include respective depth values for positions 3722A1 to 3722A2 on the edge surface of object 3722. n 3722B1 to 3722B2 of the bottom inner surface of the object 3722, including their respective depth values. n may include respective depth values for each.
[0062] In some circumstances, when the objects 3721 / 3722 are containers that contain one or more other items, such items may also be represented in the 3D image information or other image information. For example, the 3D image information 5720 of FIG. 5B may include image portions 5723, 5724 that describe respective depth values for locations on two respective items or items contained within the object 3722. More specifically, the image portion 5723 describes the positions 37231 to 37232 on one of the objects.n , and image portion 5724 may include respective depth values for positions 37241 to 37244 on another one of the objects. n may include respective depth values for each.
[0063] In some scenarios, the first image information may describe respective depth values with a depth map, which may include, for example, an array of pixels corresponding to a grid of locations on one or more object surfaces in the camera field of view (e.g., 3202). In such scenarios, some or all of the pixels may each include a respective depth value for a respective location corresponding to the pixel, each location on one or more object surfaces in the camera field of view.
[0064] In some circumstances, the first image information may describe respective depth values through multiple 3D coordinates that may describe various locations on one or more object surfaces. For example, the 3D coordinates may represent locations 37281 to 37288 in FIG. 5B. n , positions 3721A1 to 3721A n , positions 3721B1 to 3721B n , positions 3722A1 to 3722A n , positions 3722B1 to 3722B n , positions 37231 to 3723 n , and positions 37241 to 3724 n A plurality of 3D coordinates may form a point cloud or part of a point cloud describing at least a portion of the object structure, i.e., such vertices of the object structure for objects 3721, 3722, 3723, 3724, and 3728, for example. The 3D coordinates may be expressed in the camera coordinate system or in some other coordinate system. In some instances, a depth value for a particular location may be represented by or based on components of the 3D coordinate for that location. As an example, if the 3D coordinate for a location is an [XYZ] coordinate, then the depth value for that location may be equal to or based on the Z component of the 3D coordinate.
[0065] In the example of FIG. 5B, the first image information may represent the bottom inner surface of the object structure. More specifically, the first image information depicted in FIG. 5B represents positions 3721B1 to 3721B2 on the bottom inner surface of the object 3721. n , and positions 3722B1 to 3722B2 on the bottom inner surface of the object 3722. n 3D image information 5720 including respective depth values or coordinates for each of the objects. In another example, a bottom interior surface of an object structure for an object (e.g., 3721 / 3722) may not be represented by the first image information because the bottom interior surface may be completely covered or otherwise obstructed from view. In such an example, if the object (e.g., 3721 / 3722) is a container, the bottom interior surface of the container may be completely covered by the contents of the container, such as materials or items disposed within the container, and / or may be completely covered by a lid, flap, or other component used to enclose the container. In such an example, the first image information may describe respective depth values or coordinates for the location on one or more surfaces of the materials or items within the container, or the location on a lid or flap.
[0066] In an embodiment, the first image information received in step 4002 may describe the visual appearance of group 3720 of objects 3721, 3722. For example, Figure 5C provides an example in which the first image information includes or forms a 2D image 5730 (e.g., a grayscale or color image) that includes image portion 5731 (e.g., a region of pixels) describing the appearance of object 3721 of Figure 5A, image portion 5732 describing the appearance of object 3722, and image portion 5728 describing the appearance of object 3728. More specifically, image 5730 may describe the appearance of objects 3721, 3722, and object 3728 from the perspective of camera 3200 of Figure 5A, and more specifically may represent a top view of objects 3721, 3722. As described above, the 2D image 5730 may be generated by the camera 3200 when the camera 3200 has a first camera pose depicted in FIG. 5A. More specifically, the 2D image 5730 may represent visual detail(s) on one or more surfaces of the object 3721 / 3722. For example, the image portion 5721 of the 2D image 5730 may more specifically include an image portion 5721A representing a first surface (e.g., an edge surface) of the object 3721 and an image portion 5721B representing a second surface (e.g., a bottom inner surface) of the object 3721. Similarly, the image portion 5722 may include an image portion 5722A representing a first surface (e.g., an edge surface) of the object 3722 of FIG. 5A and an image portion 5722B representing a second surface (e.g., a bottom inner surface) of the object 3722. In another example, if the objects 3721, 3722 are containers filled with items or materials, as discussed above, the image portions 5721, 5722 may describe the appearance of the items or materials disposed within the containers.
[0067] Returning to FIG. 4, the method 4000 may, in an embodiment, include step 4004 in which the computing system 1100 (e.g., via the object detection module 1121) generates or updates sensed structure information representing an object structure associated with an object (e.g., 3721 / 3722) in the camera field of view (e.g., 3202) based on the first image information. As described above, the sensed structure information (also referred to as measured structure information) may be or include information describing or otherwise representing an object structure associated with an object, such as an object structure for the object 3721 / 3722. For example, the sensed structure information may be a comprehensive point cloud including a plurality of coordinates describing a location on one or more surfaces of the object 3721 and / or a plurality of coordinates for a location on one or more surfaces of the object 3722. In some implementations, the computing system 1100 may generate the sensed structure information by incorporating the first image information or an image portion(s) thereof into the sensed structure information such that the sensed structure information includes values from the first image information. As an example, FIG. 6 depicts sensed structure information 6720 generated by a computing system. The sensed structure information 6720 may include values, such as depth values or coordinates, in the first image information 5720. More specifically, the sensed structure information 6720 may be a point cloud including coordinates of positions represented in image portions 5721, 5722, 5723, and 5724 of the first image information 5720. In other words, the sensed structure information 6720 may directly incorporate image portions 5721-5724 of the first image portion 5720. These image portions may describe at least a portion of the object structure for objects 3721, 3722, 3723, and 3724. For example, image portion 5722 may describe an edge of the object structure for object 3722 and at least a portion of a bottom inner surface of the object structure for object 3722. As discussed above, the computing system 1100 may store the sensed structural information (eg, 6720) in a non-transitory computer-readable medium 1120 as part of the object detection information 1126.
[0068] In an embodiment, if the sensed structure information already includes values describing the first portion of the object structure at or before the start of step 4004, the computing system 1100 may update the sensed structure information based on values of the first image information (e.g., 5720). For example, the sensed structure information may be generated based on a plurality of sets of image information all representing top views of one or more objects within the camera field of view (e.g., 3202). The plurality of sets of image information may have been generated by the camera (e.g., 3200) at different respective positions (as the camera moves laterally), but with the same or similar orientation relative to the camera, such as an orientation in which the camera's image sensor faces directly toward respective regions on top of the one or more objects. In this example, the first image information may be one of the plurality of sets of image information. If, at the start of step 4004, the sensed structure information already includes coordinates obtained from another one of the plurality of sets of image information, the computing system 1100 may update the sensed structure information to incorporate coordinates obtained from the first image information (e.g., 5720). Thus, the computing system 1100 may include the set of coordinates obtained from the first image information as a new part of the sensed structural information. In this manner, the sensed structural information may function as a composite set of image information that combines the sets of image information discussed above. In some instances, the computing system 1100 may generate a composite set of image information by merging the sets of image information discussed above, such as in situations where the sets of image information represent overlapping regions on top of one or more objects. Such a merging operation may involve, for example, adjusting one or more existing values (e.g., depth values or coordinates) of the sensed structural information based on values of the first image information. In some instances, the merging operation may involve discarding overlapping values (e.g., coordinates or depth values) described in two or more of the sets of image information discussed above.
[0069] As discussed above, the sensed structure information may be generated or updated based on image information representing a particular viewpoint, such as a top view of one or more objects. As discussed in more detail below with respect to step 4012, the sensed structure information may be updated based on image information representing another viewpoint, such as a perspective viewpoint. Because the sensed structure information may be updated to incorporate or reflect values from multiple sets of image information, the sensed structure information may function as global structure information that combines multiple sets of image information that may be associated with multiple camera poses, acting as a composite set of image information. Thus, if the sensed structure information is or includes a point cloud or depth map, the point cloud or depth map may be updated during method 4000 to be a global point cloud or global depth map that incorporates values from multiple sets of image information.
[0070] Returning to FIG. 4, the method 4000 may, in an embodiment, include step 4006, in which the computing system 1100 identifies an object corner associated with the object structure. For example, the object corner may be an exterior corner of the object structure for object 3722 of FIG. 5A. In some implementations, the computing system 1100 may identify the object corner based on the sensed structural information (e.g., 6720). For example, the computing system 1100 may identify a plurality of exterior edges described by the sensed structural information or a plurality of edge regions described by the sensed structural information. In such an example, the computing system 1100 may identify the object corner as a location at or near the intersection of a plurality of exterior edges and / or as a location within a region where a plurality of edge regions intersect.
[0071] In one example, the computing system 1100 may identify an edge of an object structure, for example, by identifying a set of outermost locations from among the locations described by the sensed structure information, where the set of locations may approximate a portion of a contour of the object structure (e.g., for object 3722). In some instances, the computing system 1100 may estimate or otherwise identify an edge as a line passing through the set of outermost locations. In some implementations, the computing system 1100 may identify an edge region of the object structure as a region of locations that includes the set of outermost locations.
[0072] 7 illustrates the computing system 1100's identification of a first edge region 7001 and a second edge region 7002, as described or otherwise represented by the sensed structure information 6720. The first edge region 7001 may be, for example, a strip or band of locations representing a portion of an object edge for the object 3722, where the object edge forms a set of edges for the object structure of the object 3722. Similarly, the second edge region 7002 may be, for example, another strip or band of locations on another portion of the object edge. In this example, the computing system 1100 may identify an object corner 3722C1 for the object 3722 based on the intersection of the edge regions 7001, 7002. More specifically, the computing system 1100 may determine the object corner 3722C1 as a location that is within an intersection region, which may be a region where the edge regions 7001, 7002 overlap or otherwise intersect. In some implementations, each of the edge regions 7001, 7002 may be identified as a respective set of locations described in the sensed structure information 6720 and having respective depth values that are the same or substantially the same. In such implementations, the computing system 1100 may determine each of the edge regions 7001, 7002 as a respective 2D plane passing through a corresponding set of locations that have substantially the same depth value or the same Z component in 3D coordinates. In some scenarios, the computing system 1100 may identify a convex corner of the object structure as the object corner in step 4004. A convex corner may be, for example, a corner where two orthogonal edges of the object structure intersect. Convex corners are discussed in detail in U.S. Application No. 16 / 578,900 (Attorney Docket No. MJ0037-US / 0077-0006US1), the entire contents of which are incorporated herein by reference.
[0073] In an embodiment, the computing system 1100 may identify a particular object corner in step 4004 based on its reachability or visibility. In such an embodiment, the sensed structure information may describe a plurality of object corners for the object structure. For example, the sensed structure information 6720 of FIG. 7 may describe a plurality of object corners 3722C1 to 3722C4 for the object structure of the object 3722. More specifically, the object corners 3722C1 to 3722C4 may be edge corners of the object 3722. In such a scenario, the computing system 1100 may be configured to select one of the object corners (e.g., 3722C1) from among the plurality of object corners 3722C1 to 3722C4. The selection may be based on at least one of (i) the reachability of each of the plurality of object corners 3722C1 to 3722C4, or (ii) the occlusion level of each of the plurality of object corners 3722C1.
[0074] In an embodiment, reachability of an object corner may refer to how reachable the object corner is for robot interaction with a robot arm (e.g., 3400), or more specifically, an end effector device (e.g., 3500) that forms or is disposed at one end of the robot arm. For example, if the robot interaction involves an end effector device (e.g., 3500) reaching a particular object corner of an object (e.g., 3721 / 3722) and grasping the object at that object corner, the reachability of that object corner may be affected, for example, by the presence of other objects in the robot's (e.g., 3300) environment that would physically block the end effector device (e.g., 3500) from reaching the object corner. Such blocking objects may include, for example, another object (e.g., another container) that may be disposed directly above the object corner.
[0075] In an embodiment, the level of occlusion of an object corner may refer to how well the object corner can be sensed by a camera (e.g., 3200), and more specifically, may refer to the level of visibility of the object corner to the camera (e.g., 3200). The level of visibility may be affected by whether the line of sight from the object corner to the camera (e.g., 3200) is blocked or otherwise occluded by another object. Occlusion may occur when the camera (e.g., 3200) is in a first camera pose discussed above and / or when the camera is in a second camera pose discussed below. In an embodiment, the computing system 1100 may select an object corner associated with the highest reachability and / or the lowest level of occlusion among the multiple object corners in step 4004. If multiple object corners are associated with the same or substantially the same reachability or occlusion level, the computing system 1100 may randomly select one of the object corners from among the multiple object corners.
[0076] Referring back to FIG. 4, the method 4000 may, in an embodiment, include a step 4008 in which the computing system 1100 positions the camera (e.g., by one or more camera positioning movement commands) such that the camera (e.g., 3200) has a second camera pose, in which the camera (e.g., 3200) faces the object corner identified in step 4006. In this embodiment, the second camera pose of step 4008 may be different from the first camera pose associated with step 4002. As discussed above, a camera pose may be a combination of the position and orientation of the camera (e.g., 3200). FIG. 8A depicts an example in which the camera 3200 has a second camera pose, in which the camera 3200 faces the object corner 3722C1 of the object 3722. In an embodiment, the second camera pose may be a camera pose in which the camera (e.g., 3200) has a perspective view of the object (e.g., 3722). 5A, at least a portion of the object 3721 / 3722, such as a side portion (e.g., an outer side surface), may not be within the line of sight of the camera 3200, and more specifically, may not be within the line of sight of an image sensor in the camera 3200. When the camera 3200 has a second camera pose, that portion (e.g., a side portion) of the object 3721 / 3722, along with the object corner 3722C1, may come within the camera field of view 3202 of the camera 3200 and may be within the line of sight of the image sensor of the camera 3200.
[0077] In an embodiment, when a camera (e.g., 3200) has a first camera pose associated with the first image information, the first camera may have a first distance from an object (e.g., 3722) that is in the camera field of view (e.g., 3202). For example, the computing system 1100 may generate a first set of one or more camera positioning movement commands as part of step 4002 to cause the robot arm 3400 of FIG. 5A to move or otherwise position the camera 3200 to a camera pose in which the camera is positioned directly above the object (e.g., 3722) and has a predefined first distance from the object (e.g., 3722), or more specifically, from a top (e.g., edge surface) of the object. In some instances, the first distance may be sufficiently far from the object (e.g., 3722) to enable the camera field of view (e.g., 3202) to encompass the entire top of the object, or more specifically, encompass multiple object corners of the object. As a result, when the camera (e.g., 3200) generates the first image information while having the first camera pose, the first image information may represent the entire top of the object, including multiple object corners (e.g., 3722C1-3722C4) of the object (e.g., 3722C1-3722C4). Such image information may facilitate the ability of the computing system 1100 to identify the object corners in step 4006. However, if the first distance has a large value, the resulting image information may not be as detailed as image information associated with a closer distance. Thus, in one example, the second camera pose associated with the second image information in step 4010 may involve positioning the camera (e.g., 3200) closer to the object (e.g., 3722), or more specifically, closer to the object corner (e.g., 3722C1) of the object (e.g., 3722). More specifically, the computing system 1100 may generate a second set of one or more camera positioning movement commands for causing a robotic arm (e.g., 3400) to move a camera (e.g., 3200) to a second camera pose.A second set of one or more camera positioning movement commands may cause the robot arm (e.g., 3400) to position the camera (e.g., 3200) to have a second distance away from the object corner (e.g., 3722C1), which may be less than the first distance. The shorter distance may enable the second image information to capture or otherwise represent the object structure of the object (e.g., 3722) with a greater degree of detail than the first image information. Thus, the second image information may be used to refine a description or estimate of the object structure.
[0078] Returning to FIG. 4, in an embodiment, method 4000 may include step 4010 in which computing system 1100 receives second image information (also referred to as a second set of image information) representing an object structure, such as the object structure for object 3722 / 3721 in FIG. 8A. The second image information may be generated by a camera (e.g., 3200) when the camera has a second camera pose, such as the camera pose shown in FIG. 8A. As described above, the second camera pose may provide the camera (e.g., 3200) with a perspective view of the object (e.g., 3722) in some circumstances. In such circumstances, the second image information may represent a perspective view of the object.
[0079] In some implementations, the second image information may be or include 3D image information. As an example, FIG. 8B depicts 3D image information 8720, which may form or be part of the second image information. The 3D image information 8720 may be or include a point cloud generated by the camera 3200, for example, when the camera 3200 has the second camera pose shown in FIG. 8A. As with the first image information, the second image information may include values, such as depth values or coordinates, for positions on one or more surfaces of various object structures. More specifically, the second image information may include image portions 8721, 8722, 8723, and 8724, which may represent object structures for objects 3721, 3722, 3723, and 3724 of FIG. 8A. In one example, if the object represented by the second image information has a ridge pattern, or more broadly, a plurality of physical protrusions protruding from one or more outer side surfaces of the object, the 3D image information 8720 may describe or otherwise represent the plurality of protrusions.
[0080] In some implementations, the second image information may be or include 2D image information. For example, FIG. 8C shows a 2D image 8730 that may form part of the second camera information. In this example, the 2D image 8730 may include at least image portions 8731, 8732 that represent the appearance of objects 3721, 3722, respectively, from a perspective or viewpoint of the camera 3200 when the camera 3200 has a second camera pose.
[0081] In an embodiment, as described above, the first image information received in step 4002 may be associated with a camera (e.g., 3200) having a first distance from the object (e.g., 3722), and the second image information received in step 4010 may be associated with a camera (e.g., 3200) having a second distance from the object (e.g., 3722), the second distance being less than the first distance. In this embodiment, the first camera pose may be associated with a first distance between the camera and the object or a portion thereof (e.g., the top of the object 3722), while the second camera pose may be associated with a second distance between the camera and the object or a portion thereof (e.g., a corner of the object 3722), the second distance being less than the first distance. As discussed above, a larger value of the first distance may cause the first image information to have a lower level of detail, which may be adequate for performing a rough detection phase involving identification of object corners, but may not be adequate for determining an object type associated with the object (e.g., 3722). A greater degree of detail may be provided by the second image information because the second image information is associated with a closer distance between the camera and the object (e.g., 3722). Additionally, as discussed above, the second image information may represent a portion of the object structure that is not represented, or only partially represented, in the first image information. Thus, the second image information may enhance the ability of the computing system 1100 to accurately determine an object type for the object (e.g., 3722), as discussed below.
[0082] Returning to FIG. 4, the method 4000 may include, in an embodiment, step 4012, updating the sensed structure information based on the second image information by the computing system 1100. After the sensed structure information is updated, it may be referred to as updated sensed structure information. In some implementations, step 4012 may involve incorporating values, such as depth values or coordinates, from the second image information into the sensed structure information (e.g., 6720). If the sensed structure information includes values from the first image information after being generated or updated in step 4004, step 4012 may generate updated sensed structure information that combines the first image information and the second image information, such as by including both the first image information and the second image information. For example, FIG. 6 shows sensed structure information 6720, which may be a global point cloud that incorporates or otherwise includes the coordinates described by the first image information 5720. The sensed structure information 6720 in this example may represent a portion of the object structure for the object 3722 and / or 3721, i.e., more specifically, the edge and bottom inner surface for the object structure of the object 3722 and / or 3721. In this example, the computing system 1100 may update the sensed structure information 6720 by updating the global point cloud to insert or otherwise incorporate coordinates described by the second image information 8720 of FIG. 8B. The sensed structure information 6720 may be updated to obtain the updated sensed structure information 9720 of FIG. 9, which may be, for example, an updated version of the global point cloud including a plurality of coordinates representing the object structure associated with the object (e.g., 3721 / 3722). The plurality of coordinates of the updated version of the global point cloud may combine or otherwise merge the coordinates described by the first image information 5720 and the coordinates described by the second image information 8720. As described above, the second image information may, in some circumstances, represent a perspective view of the object (e.g., 3721 / 3722). The perspective view may enable the second image information to represent at least a side portion(s) of the object structure for the object (e.g., 3721 / 3722).Because the updated sensed structure information (e.g., 9720) incorporates the second image information, the updated sensed structure information (e.g., 9720) may also represent a side portion(s) of the object structure. If an object (e.g., 3721 / 3722) within the camera field of view (e.g., 3202) has a ridge pattern on one or more outer side surfaces of the object, the updated sensed structure information (e.g., 9720) may describe or otherwise represent the ridge pattern.
[0083] Returning to FIG. 4, the method 4000 may, in an embodiment, include a step 4014 in which the computing system 1100 determines an object type associated with an object (e.g., 3722) in the camera field of view, which may be based on the updated sensed structure information (e.g., 9720). For example, if the objects 3721, 3722 are containers, step 4014 may involve determining a container type associated with the object 3721 and / or a container type associated with the object 3722. In an embodiment, the object type may be associated with a particular object design, which may include a physical design and / or a visual design. In this embodiment, the physical design may refer to, for example, the physical structure (also referred to as the object structure) of an object that belongs to or is otherwise associated with the object type. The physical structure may be characterized by an object shape, an object size, and / or physical features (e.g., ridge patterns) disposed on the surface of the object associated with the object type.
[0084] In an embodiment, an object type may be associated with an object recognition template, such as the template described by object recognition template information 1128 of FIG. 2D. In one example, the object recognition template may be a container template that describes a container design, i.e., more specifically, a visual design and / or physical design for a container type. When object recognition template information 1128 describes multiple object recognition templates, the multiple object recognition templates may each be associated with a different object type. For example, FIG. 10A depicts object recognition template information 9128 (which may be an embodiment of object recognition template information 1128) that describes object recognition templates 9128A1, 9128A2, and 9128A3. In this example, object recognition templates 9128A1, 9128A2, and 9128A3 may be associated with three different respective object types, i.e., container type 1, container type 2, and container type 3, respectively. The object recognition templates 9128A1, 9128A2, and 9128A3 stored or otherwise described by the template information 9128 may be used to populate a candidate set, or more specifically, a template candidate set. The template candidate set may represent a set of candidate object recognition templates that may potentially match an object in the camera field of view (e.g., 3722), or more specifically, may be candidates that match the updated sensed structural information. As discussed below, the computing system 1100 may compare the updated sensed structural information (e.g., a global point cloud) to these candidate templates to determine which object recognition templates match the updated structural information and / or which object recognition template provides the best match.
[0085] In some implementations, some or all of the object recognition templates (e.g., 9128A1, 9128A2, and 9128A3) may each include a respective object structure description (also referred to as structure description information). The object structure description of the object recognition template may describe the physical design, or more specifically, the object structure, for the object type associated with the object recognition template. In some instances, the object structure description may include a CAD file describing the object structure. In some instances, the object structure description may include a point cloud (also referred to as a template point cloud) describing the contours of the object structure, such as by describing the edges, surfaces, ridge patterns, or other physical features that form the object structure. In an embodiment, the set of object recognition templates (e.g., 9128A1, 9128A2, and 9128A3) may describe a set of object structure models associated with the respective container types, which may describe the respective object shapes, physical designs, or generally the object structures. For example, if the object structure description in the object recognition template includes a CAD file, the object structure model associated with the object recognition template may be the CAD model described by the CAD file. Figure 10A provides an example in which a set of object recognition templates 9128A1 through 9128A3 each describe a set of three object structure models in their object structure description.
[0086] In an embodiment, the object recognition template object structure description may include a direct description of a portion of the object structure and may omit a direct description of the remaining portion of the object structure because the remaining portion of the object structure may have structural details that are the same or substantially the same as the portion of the object structure directly described by the object recognition description. For example, FIG. 10B shows object recognition template information 9128 describing object recognition templates 9128B1, 9128B2, and 9128B3, which may also be associated with container type 1, container type 2, and container type 3, respectively, and may describe the respective container structures associated with container types 1 through 3. In the example of FIG. 10B, each of object recognition templates 9128B1, 9128B2, and 9128B3 may have an object structure description that directly describes structural details for two vertical sides of the respective container structure, while omitting a direct description of the remaining two vertical sides of the respective container structure. A direct description of the remaining two vertical sides may be omitted since their structural details may be the same or substantially the same as the structural details described by the object structure description, in other words, the object structure description may already indirectly describe the remaining two vertical sides of each container structure.
[0087] As described above, object recognition templates (e.g., 9128B1 through 9128B3), stored in computing system 1100 or elsewhere, may be used to populate a template candidate set, which may be a set of object recognition templates, where the object recognition templates in the set may describe object structures associated with different object types (e.g., container types 1, 2, and 3). In an embodiment, determining the object type associated with the object (e.g., 3722) may involve comparing the updated sensed structure information (e.g., the global point cloud) of step 4012 to the object recognition templates in the template candidate set. By way of example, FIG. 11A illustrates a comparison of updated sensed structure information 9720, which may represent object structures for object 3721 and object 3722 of FIG. 8A, to a template candidate set, which includes object recognition templates 9728A1 through 9728A3. Similarly, FIG. 11B illustrates a comparison of the updated sensed structural information 9720 with a template candidate set including object recognition templates 9728B1 through 9728B3.
[0088] In an embodiment, the comparison discussed above may be used to determine the respective degree to which each object recognition template (e.g., 9728A1 through 9728A3 or 9728B1 through 9728B3) in the template candidate set matches the updated sensed structural information. The comparison may indicate how well each of the object recognition templates is supported or explained by the updated sensed structural information (e.g., the global point cloud). In one example, the computing system 1100 may select an object recognition template (e.g., 9728A3 or 9728B3) from the template candidate set based on the comparison. The selected object recognition template may represent an object type (e.g., container type 3) associated with the object (e.g., 3722). More specifically, the selected object recognition template may be associated with the object type. Thus, in this example, determining the object type for an object in the camera field of view may involve selecting an object recognition template associated with the object type.
[0089] In an embodiment, the selection of the object recognition template may be based, for example, on which object recognition template of the template candidate set most closely matches the updated sensed structural information. As discussed in more detail below, the comparison may involve determining an error value, each describing a respective amount of deviation between the object recognition template and the updated sensed structural information. In such a scenario, the selection of the object recognition template may be based on the error value, as discussed in more detail below. In an embodiment, the computing system 1100 may be configured to use the description of the object structure in the object recognition template selected in step 4016 to determine one or more robot interaction locations. If the description of the object structure includes an object structure model, the computing system may be configured to use the object structure model of the selected object recognition template to determine one or more robot interaction locations.
[0090] As described above, the computing system 1100 may compare the updated sensed structure information with the set of candidate object recognition templates, or more specifically, with the set of descriptions of the respective object structures in the object recognition templates. For example, if the updated sensed structure information describes multiple ridges protruding from a side surface of the object structure (e.g., a container structure), the computing system 1100 may detect the multiple ridges based on the updated sensed structure information and / or the second image information, and compare the detected ridges with the ridges or other physical features described by the descriptions of the object structure in the object recognition templates. In such an example, an object type (e.g., a container type) for an object in the camera field of view may be determined based on a selection of which object recognition template best matches the ridges detected on the outer side surface of the object. Thus, the object type in this example may be determined based on the ridges detected on the outer side surface of the container structure. In some instances, the set of object structure descriptions may describe a set of respective object structure models. In some instances, the comparison may take into account the orientation of the object structure models. Thus, the computing system 1100 may more specifically compare the updated sensed structural information with candidate combinations of object structure models and orientations of the object structure models. In this example, the template candidate set may more specifically be a model-orientation candidate set, which may be a set including model-orientation combinations. Each of the model-orientation combinations in the candidate set may be a combination of (i) an object structure model that is one of the set of object structure models discussed above, and (ii) an orientation of the object structure model. In such an example, the computing system 1100 may compare the updated sensed structural information with the model-orientation combinations in the model-orientation candidate set.
[0091] In an embodiment, if an object structure model represents or describes multiple outer side surfaces (also referred to as outer lateral surfaces) of an object structure for a particular object type, the orientation of the object structure model may refer to the respective direction in which each of the multiple outer side surfaces faces. In an embodiment, the orientation of the object structure model may refer to how the computing system 1100 attempts to align the object structure model to a point cloud or other sensed structural information. In one example, the point cloud may represent at least a first outer side surface and a second perpendicular outer side surface of, for example, a container or other object within the camera field of view (e.g., 3202). In this example, the object structure model may also represent or describe at least a first outer side surface and a second outer side surface for the object type associated with the object structure model, and the second outer side surface of the object structure model may be perpendicular to the first outer side surface. The first exterior side surface and the second exterior side surface described by the point cloud and / or by the object structure model may represent, for example, two vertical side walls of a container or container structure.
[0092] In some instances, the first orientation of the object structure model may refer to the computational system 1100's determination of how well the first and second outer side surfaces of the object structure model align with the first and second outer side surfaces represented by the point cloud, respectively. More specifically, when the object structure model has the first orientation, the computational system 1100 may compare the physical features (e.g., protuberance) or other characteristics (e.g., size) of the first outer side surface of the object structure model with the physical features or other characteristics of the first outer side surface described by the point cloud, and may compare the physical features or other characteristics of the second outer side surface of the object structure model with the physical features of the second outer side surface described by the point cloud. Further, in this example, the second orientation of the object structure model may involve a 90 degree rotation of the first and second outer side surfaces of the object structure model relative to the first orientation. When the object structure model has a second orientation, the computing system 1100 may determine how well the first and second outer side surfaces of the object structure model align with the second and first outer side surfaces described by the point cloud, respectively. More specifically, when the object structure model has a second orientation, the computing system 1100 may compare a physical feature or other characteristic of the first outer side surface of the object structure model with a physical feature of the second outer side surface described by the point cloud, and may compare a physical feature or other characteristic of the second outer side surface of the object structure model with a physical feature of the first outer side surface described by the point cloud.
[0093] In an embodiment, the alignment between the object structure model and the point cloud may be better when the object structure model has one of the first orientation or the second orientation than when the object structure model has the other of the first orientation or the second orientation. Such an embodiment may occur because the first outer side surface and the second outer side surface described by the object structure model may have different physical features, such as different ridge patterns and / or other different characteristics (e.g., different sizes). As an example, if the first outer side surface of the object structure model corresponds to the first outer side surface sensed by the point cloud, the level of alignment between the physical features described by the object structure model (e.g., ridge patterns) and the physical features described by the point cloud (e.g., ridge patterns) may be better when the object structure model has the first orientation than when the object structure model has the second orientation because the first orientation may result in a comparison between the first outer side surface of the object structure model and the first outer side surface of the point cloud.
[0094] For example, FIGS. 12A and 12B show a comparison of the updated sensed structure information 9720 with a model-orientation candidate set including model-orientation combinations A through F (as shown in FIG. 12B) or U through Z (as shown in FIG. 12A). In FIG. 12A, each model-orientation combination may be a combination of (i) an object structure model described by one of the object recognition templates 9128A1 through 9128A3 and (ii) an orientation of that object structure model. Similarly, each model-orientation combination in FIG. 12B may be a combination of (i) an object structure model described by one of the object recognition templates 9128B1 through 9128B3 and (ii) an orientation of that object structure model. As an example, the model-orientation combination Y in FIG. 12A may be a combination of the object structure model described by the object recognition template 9128A3 with a first orientation of the object structure model, while the model-orientation combination Z may be a combination of the same object structure model with a second orientation of the object structure model. In this embodiment, determining an object type for an object in the camera field of view may more specifically involve selecting a particular model-orientation combination, and the object structure model of the selected combination is associated with the object type. As discussed in more detail below, the object structure model and orientation of the selected model-orientation combination may be used by the computing system 1100 to determine a robot interaction position. If the selection involves determining an error value by the computing system 1100, such an error value in this embodiment may be associated with the model-orientation combination in the model-orientation candidate set.
[0095] In an embodiment, the computing system 1100 may determine whether to filter out object recognition template(s) from the template candidate set or filter out model-orientation combination(s) from the model-orientation candidate set as part of step 4014. Filtering out templates or combinations may remove them from being considered as potential matches with the updated sensed structural information (e.g., 9720). In some instances, when the computing system 1100 determines an error value based on a template candidate set or a model-orientation candidate set, it may determine the error value after filtering these candidate sets, which may reduce the total number of error values that need to be calculated and therefore save computational resources. In other words, filtering may generate a filtered candidate set, and error values may be generated based on the object recognition template or model-orientation combinations in the filtered candidate set.
[0096] In an embodiment, the decision to filter out an object recognition template or model-orientation combination from a candidate set (e.g., a template candidate set of a model-orientation candidate set) may involve determining whether the corresponding object structure model has at least a portion that is outside of a region occupied by the updated sensed structure information (e.g., 9720). More specifically, the updated sensed structure information may estimate a region of space occupied by an object structure for an object (e.g., 3722) in the camera field of view (e.g., 3202). If a particular object recognition template or model-orientation combination includes an object structure model that is outside of a region of space, the computing system 1100 may determine that the object structure model likely does not represent the object, and therefore, it is unnecessary to even determine an error value for the object recognition template or model-orientation combination. Thus, the computing system 1100 may remove the template or combination from the candidate set.
[0097] As an example, the computing system 1100 may filter the template candidate set of FIG. 11A or 11B by identifying one or more object recognition templates including one or more respective object structure models that do not fall or do not substantially fall within the estimated region and removing the one or more object recognition templates from the template candidate set. In other words, the computing system 1100 may determine whether to filter out a particular object recognition template from the candidate set by determining whether the object structure model described by the object recognition template is sufficiently supported or explained by the updated sensed structure information. Such a determination may relate to whether the object structure model substantially falls within the estimated region of the space occupied by the object structure. There may be a substantial match when the object structure model falls entirely within the estimated region or when the percentage of the object structure model that is outside the estimated region is less than a predefined threshold. If the object structure model does not substantially fall within the estimation region, the computing system 1100 may determine that the object structure model is not sufficiently supported or explained by the updated sensed structural information associated with the object structure (e.g., 9702). Thus, the computing system 1100 may filter out the object recognition template by removing it from the template candidate set.
[0098] In an embodiment, the computing system 1100 may perform a filtering operation for an object recognition template in the template candidate set by determining whether an object structure model described by the object recognition template substantially falls within the estimation region for at least one orientation of the object structure model. If there is at least one orientation of the object structure model that would cause the object structure model to substantially fall within the estimation region, the computing system may decide not to filter out the associated object recognition template from the template candidate set.
[0099] In an embodiment, the computing system 1100 may filter out candidate orientations, or more specifically, candidate combinations of object structure models and their orientations. In such an embodiment, the computing system 1100 may more specifically determine whether to filter out model-orientation combinations from the model-orientation candidate set. As an example, the model-orientation candidate set may include model-orientation combinations A to F in FIG. 12B. In this example, the computing system 1100 may be configured to perform the filtering operation by determining, for each of the model-orientation combinations in the candidate set, whether to remove the model-orientation combination from the candidate set. More specifically, the computing system may determine whether the object structure model included in or associated with the model-orientation combination substantially falls within the estimation region discussed above when the object structure model has an orientation associated with the model-orientation combination. For example, FIG. 13A depicts an example with a determination of whether to filter out model-orientation combination C from the model-orientation candidate set. As shown in FIG. 13A, this model-orientation combination may involve an object structure model described in object recognition template 9128B2, and may involve an object structure model having orientation 1. The computing system 1100 may determine that the object structure model included in model-orientation combination C does not substantially fall within the estimated region defined by the updated sensed structure information 9720 when it has an orientation shown in or associated with model-orientation combination C. In response to such a determination, the computing system 1100 may remove model-orientation combination C from the model-orientation candidate set or may generate an indication that model-orientation combination C should be removed from the candidate set. FIG. 13B depicts the model-orientation candidate set after model-orientation combination C has been removed. The model-orientation candidate set may represent a filtered candidate set.
[0100] FIG. 13C illustrates another example of determining whether a model-orientation combination (model-orientation combination D) should be removed from the model-orientation candidate set. More specifically, the computing system 1100 of this example may determine whether an object structure model associated with the model-orientation combination D substantially fits within an estimated region defined by the updated sensed structure information when the object structure model has an orientation (orientation 2) associated with the model-orientation combination D. This object structure model may be the same as the object structure model of the model-orientation combination C, but may have an orientation different from that of the model-orientation combination C. In this example, the computing system 1100 may determine that the object structure model associated with the model-orientation combination D substantially fits within the estimated region when the object structure model has an orientation associated with the model-orientation combination D. As a result, the computing system 1100 may determine not to remove the model-orientation combination from the candidate set.
[0101] As discussed above, the computing system 1100 may determine a set of error values for model-orientation combinations in the model-orientation candidate set after the candidate set has been filtered. For example, the computing system 1100 may determine to filter out model-orientation combination C of FIG. 12B from the candidate set, and to not filter out model-orientation combinations A, B, D, E, and F. In this example, the computing system 1100 may determine error values for model-orientation combinations A, B, D, E, and F that remain in the candidate set after the candidate set has been filtered.
[0102] In an embodiment, the computing system 1100 may perform a refinement operation (e.g., a pose refinement operation) that adjusts the object structure description in the object recognition template, i.e., more specifically, the pose information associated with the physical features described by the object recognition template, to more closely match the object structure description to the updated sensed structure information (compared to the level of match before the adjustment). In some instances, the pose refinement operation may be performed on the object structure model associated with the object recognition template, more specifically, on the object structure model associated with the model-orientation combination. After being adjusted by the pose refinement operation, the object recognition template, the object structure description, the pose information, and the object structure model may be referred to as the refined object recognition template, the refined object structure description, the refined pose information, and the refined object structure model, respectively.
[0103] In some implementations, a pose refinement operation, discussed below, may be performed in parallel with the comparison of the object recognition template to the sensed structural information. For example, when performing the filtering operation discussed above, the pose refinement operation may be performed in parallel with the filtering operation and / or in parallel with the calculation of the error value, discussed in more detail below. In some implementations, the pose refinement operation may be performed before the comparison of the object recognition template to the sensed structural information. For example, the pose refinement operation may be performed before the filtering operation and / or before the calculation of the error value. In some implementations, the pose refinement operation may be performed after the filtering operation and / or before the calculation of the error value. In such implementations, the pose refinement operation may be performed on the object recognition templates in the template candidate set or the model-orientation candidate set after the candidate sets have been filtered.
[0104] For example, FIG. 14 illustrates a pose refinement operation that involves adjusting an object structure model associated with object recognition template 9128B2, i.e., more specifically, the object structure model associated with model-orientation combination D of FIGS. 12B and 13B, to generate a refined object recognition template, i.e., a refined object structure model. In some implementations, the object structure model being adjusted may describe at least one physical feature (e.g., an edge, a corner, a ridge, an outer surface) for the object structure associated with object recognition template 9128B2 and associated with the object structure model. In the example of FIG. 14 ... 9128B2, which may be a respective edge of the object structure associated with or represented by object recognition template 9128B2. 2-1 , 9128B 2-2 More specifically, the object structure model being adjusted may describe the physical features 9128B 2-1 , 9128B 2-2 and physical characteristics 9128B, which may include posture information describing the posture of each of the 2-1 , 9128B 2-2 As described above, the computing system 1100 may adjust the pose information in the object structure model based on the updated sensed structure information to generate refined pose information and / or a refined object structure model. For example, as shown in FIG. 14 , the computing system 1100 may adjust the pose information to refine the position and orientation of the physical feature 9128B. 2-1 Adjustments to and / or physical characteristics 9128B 2-2 The adjustment may indicate, for example, an adjustment to the physical feature 9128B to more closely approximate one group of coordinates described by the updated sensed structure information 9720. 2-1 Rotating the physical features 9128B 2-1 The adjustment may further involve changing the orientation of the physical feature 9128B to more closely approximate another group of coordinates described by the updated sensed structure information 9720. 2-2To move the physical features 9128B 2-2 The adjustment may involve shifting the position of the physical feature 9128B, for example, by 2 to 5 mm, as described by the refined object structure model generated in FIG. 2-1 , 9128B 2-2 may be made to more closely match the updated sensed structural information 9720.
[0105] In some instances, pose refinement may generate refined pose information, refined object structure model, and / or refined object recognition template that provide enhanced accuracy for comparison with sensed structure information (e.g., updated sensed structure information of step 4012). As discussed in more detail below, when a refined object recognition template is used to determine a robot interaction position, enhancing the accuracy of the refined pose information in the refined object recognition template may make the robot interaction position more optimal. In some instances, enhancing the accuracy of the refined pose information in the object recognition template may facilitate object type determination, such as by facilitating a comparison between the refined object recognition template and the updated sensed structure information. The comparison may involve determining an error value indicating the respective degree of deviation between the refined object recognition template and the updated sensed structure information. In such instances, a pose refinement operation may be performed to adjust the object recognition template before determining the error value. In some scenarios, adjusting the object recognition template may make the error value more reliable or more useful for determining which object recognition template most closely matches the updated sensed structural information after being adjusted by the pose refinement operation.
[0106] In some implementations, pose refinement may facilitate robust determination of object types in real-world, non-ideal environments that may be affected by manufacturing tolerances, physical damage, or other sources of deviation between the object structure model and the actual object associated with the model. For example, manufacturing tolerances may result in slight structural variations in objects of the same object type, and therefore may cause at least some of those objects to exhibit differences when compared to the object recognition template, or more specifically, the object structure model, associated with the object type. As another example, some of those objects may experience minor physical damage or other forms of structural changes during use as a result of interaction with the environment. In these examples, pose refinement may be used to account for slight structural variations that may naturally exist between the actual object in the camera's physical environment and the object structure model associated with that object. More specifically, the pose refinement operation may adjust the object structure model to bring the refined object structure model closer to the structural information sensed about the object in order to reduce the deviations discussed above.
[0107] In an embodiment, the computing system may perform a comparison between the object recognition templates in the template candidate set and the updated sensed structure information by calculating or otherwise determining at least one respective error value for each object recognition template in the candidate set. For example, as discussed above, if the updated sensed structure information includes a global point cloud having a plurality of coordinates, the at least one respective error value may be calculated, for example, based on how closely coordinates from the multiple coordinates of the global point cloud match the corresponding object recognition template, i.e., more specifically, how closely coordinates of the global point cloud match one or more physical features (e.g., edges, corners, ridges, surfaces, etc.) described by the object structure description information included in the object recognition template. In some implementations, if the object structure description of the object recognition template includes a point cloud (also referred to as a template point cloud), the error value associated with the object recognition template may be based on a respective distance between coordinates in the global point cloud and coordinates in the template point cloud. In some implementations, the template point cloud and the object recognition template may be a refined template point cloud or a refined object recognition template generated from a pose refinement operation, as discussed above. In some implementations, the error value may indicate the degree of deviation between the physical features (e.g., ridges, edges, and / or corners) described by the object structure model of the object recognition template and the physical features (e.g., ridges, edges, and / or corners) described by the global point cloud.
[0108] 15A-15C depict an example in which the computing system 1100 performs a comparison between the object recognition templates 9128B1-9128B3 in the template candidate set of FIG. 11B (which may be refined object recognition templates adjusted by a pose refinement operation, for example) and the updated sensed structural information 9720. More specifically, FIG. 15A illustrates the computing system 1100 determining at least one error value for the object recognition template 9128B1 in the template candidate set, while FIG. 15B illustrates the computing system 1100 determining at least one error value for the object recognition template 9128B2 in the template candidate set, and FIG. 15C illustrates the computing system 1100 determining at least one error value for the object recognition template 9128B3 in the template candidate set. In some implementations, the computing system 1100 may determine multiple error values (e.g., two error values) for the object recognition templates. The multiple error values may correspond respectively to multiple orientations of an object structure model (e.g., a refined object structure model generated by a pose refinement operation) described by the object recognition template. For example, as discussed in more detail below, FIG. 15C may show computing system 1100 determining one error value (corresponding to one orientation) for object recognition template 9128B3, while FIG. 16 shows computing system 1100 determining another error value (corresponding to another orientation) for object recognition template 9128B3.
[0109] Returning to the example of FIG. 15A , which pertains to error values associated with object recognition template 9128B1, the error values may indicate a respective degree of discrepancy between the object structure description (e.g., object structure model) in object recognition template 9128B1 and updated sensed structure information 9720. In some implementations, computing system 1100 of FIG. 15A may detect or otherwise determine whether the object structure model of object recognition template 9128B1 has any portions that are not fully explained by updated sensed structure information 9720. As an example, computing system 1100 may determine whether a distance between a particular portion of the object structure model and a corresponding (e.g., closest) portion of updated sensed structure information 9720 is greater than a predefined distance threshold. For example, if the object structure model and the updated sensed structure information 9720 are both point clouds each including multiple coordinates (also referred to as points), the computing system may determine whether the object structure model has any coordinates that are separated from the corresponding (e.g., closest) coordinates of the updated sensed structure information 9720 by more than a predefined distance threshold. If the distance separating the two corresponding coordinates is greater than a predefined distance threshold, the computing system 1100 may determine that a particular coordinate, or more broadly, a particular portion, of the object structure model is not adequately explained by the updated sensed structure information 9720. Such a portion may be referred to as an unexplained portion, or more specifically, an unexplained coordinate or an unexplained point. FIG. 15A illustrates an example in which the computing system 1100 determines that a portion 140311-140317 of the object structure model has coordinates that are not adequately explained by the coordinates of the updated sensed structure information 9720 (e.g., a global point cloud). These unexplained coordinates (also referred to as unexplained points) may form approximately 11% of the total number of coordinates or points in the object structure model and may have an average distance of approximately 3.05 mm with corresponding coordinates in the updated sensed structure information 9720. In Figure 15A, the computing system 1100 may determine an error value associated with the object recognition template 9128B1 to be equal to or based on the average distance or number of unexplained points in Figure 15A.
[0110] As shown in FIGS. 15B and 15C, the computing system 1100 may determine an error value associated with object recognition template 9128B2 and an error value associated with object recognition template 9128B3. In the example of FIG. 15B, the computing system 1100 may determine that the object structure model of object recognition template 9128B2 has portions 140321-140325 that have coordinates that are not fully explained by the updated sensed structure information 9720. As a specific example, these unexplained coordinates may form about 13% of the total number of coordinates in the object structure model of object recognition template 9128B2 and may have an average distance of about 3.85 mm from the corresponding coordinates in the updated sensed structure information 9720. In the example of FIG. 15C, the computing system 1100 may determine that the object structure model of object recognition template 9128B3 has portions 140331 that include coordinates that are not fully explained by the updated sensed structure information 9720. These unexplained coordinates in portion 140331 may form approximately 0.09% of the total number of coordinates in the object structure model and may have an average distance of approximately 1.31 mm with the corresponding coordinates in the updated sensed structure information 9720.
[0111] As mentioned above, in an embodiment, the computing system may determine a set of error values associated with each model-orientation combination in a model-orientation candidate set, such as the candidate sets of FIG. 12A, 12B, or 13B. As further discussed above, each model-orientation combination in the candidate set may be a combination of an object structure model and an orientation of the object structure model. In this example, each error value in the set of error values may indicate a respective degree of deviation between (i) the updated sensed structure information (e.g., 9720) and (ii) the object structure model of the respective model-orientation combination associated with the error value when the object structure model has an orientation associated with the respective model-orientation combination. For example, FIGS. 15A-15C show that the computing system 1100 determines three error values associated with three respective model-orientation combinations, namely, model-orientation combination A (as depicted in FIG. 15A), model-orientation combination D (as depicted in FIG. 15B), and model-orientation combination F (as depicted in FIG. 15C). FIG. 16 further illustrates the computing system 1100 determining an error value associated with model-orientation combination E. In the example of FIG. 16, the computing system 1100 may determine that when the object structure model of the object recognition template 9128B3 has the orientation of the model-orientation combination E (orientation 2), the object structure model has a portion 150331-150334 that includes coordinates that are not fully explained by the updated sensed structure information 9720. Furthermore, the error value of FIG. 15C may indicate a deviation between the updated sensed structure information 9720 and the object structure model of the object recognition template 9128B3 when the object structure model has orientation 1, which is the orientation of the model-orientation combination F, while the error value of FIG. 16 may indicate a deviation between the updated sensed structure information 9720 and the same object structure model when the object structure model has orientation 2, which is the orientation associated with the model-orientation combination E.
[0112] In an embodiment, the computing system 1100 may be configured to determine an object type by determining a set of error values discussed above based on the updated sensed structural information and by selecting an object recognition template and / or model-orientation combination based on the set of error values, and the object structural model of the selected object recognition template and / or model-orientation combination is associated with the object type. In an example, the computing system 1100 may select an object recognition template (e.g., 9128B3) from among a set of template candidates (e.g., 9128B1 to 9128B3). The selected object recognition template may have the lowest error value among the set of error values. For example, if the set of error values includes percentage values of 11.35%, 12.97%, and 0.09% associated with the object recognition templates 9128B1 to 9128B3 of FIGS. 15A-15C, the computing system 1100 may select the object recognition template 9128B3 associated with the lowest percentage value among the set of percentage values, i.e., 0.09%.
[0113] In one example, the computing system 1100 may select a model-orientation combination from a model-orientation candidate set, such as the candidate set of FIG. 12A or 12B, or 13B. For example, if the candidate set includes at least model-orientation combinations A, D, F, and E shown in FIG. 15A-15C and 16, the set of error values may include a set of percentage values of 11.35%, 12.97%, 0.09%, and 3.74% in those figures, in one example. In this example, the computing system 1100 may be configured to select the model-orientation combination F, which is associated with the lowest percentage value in the set of error values. As described above, the selected object recognition template and / or the selected model-orientation combination may be associated with an object type of an object (e.g., 3722) that is in the camera field of view (e.g., 3202). Thus, the computing system 1100 may determine an object type associated with the object (e.g., 3722) in step 4014 by selecting an object recognition template or a model-orientation combination in an embodiment.
[0114] Returning to FIG. 4, the method 4000 may, in an embodiment, include step 4016 in which the computing system 1100 may determine (e.g., by the robot interaction planning module 1122) one or more robot interaction locations based on the object type determined in step 4014. As described above, the object type may be associated with an object design for a type or class of object, or more specifically, a physical design (e.g., physical shape) and / or visual design for a type or class of object. In such an example, the computing system 1100 may determine the one or more robot interaction locations based on the physical design associated with the object type determined in step 4014. As an example, the object (e.g., 3722) may be a container, and the one or more robot interaction locations may be a number of grip locations where the container should be grasped, picked up, or otherwise engaged by the robot (e.g., 3300), or more specifically, by an end effector device (e.g., 3500). In such an example, the multiple grip positions may be determined based on a physical design (eg, physical shape) associated with the object type.
[0115] In an embodiment, one or more robot interaction positions may be determined based on a selected object recognition template and / or based on a selected model-orientation combination, discussed above, such as object recognition template 9128B3 of Figure 15C and / or model-orientation combination F. More specifically, one or more robot interaction positions may be determined based on an object structural model included in the object recognition template and / or model-orientation combination.
[0116] In an embodiment, the object structure model may already include or otherwise identify one or more robot grip positions. For example, FIG. 17A shows an object recognition template 16128 with an object structure model that identifies robot grip positions 161291 and 161292. In this example, the object structure model may be a container structure model that may describe a physical structure associated with the container type. More specifically, the container structure model of FIG. 17A may be associated with a type or class of container having a physical feature 161281 that is a container rim. In other words, the container structure model may describe the edge structure of the container. In this example, the robot grip positions 161291 and 161292 may be positions along the edge structure of the container. The container structure model of FIG. 17A may further describe other physical features 161282, 161283, and 161284, which may be a first ridge or other protrusion, a second ridge or other protrusion, and a corner, respectively.
[0117] In an embodiment, the object structure model of the selected object recognition template or selected model-orientation combination may identify regions that may have physical features that may interfere with the robot's grasp. For example, FIG. 17B shows an example in which the object structure model of the object recognition template 16128 identifies a first region 161301, which represents the space around the first ridge (161282) of FIG. 17A, and identifies a second region 161302, which represents the space around the second ridge (161283). In this example, if grasping the container represented by the object recognition template 16128 involves moving the gripper fingers of the end effector device (e.g., 3500) toward the edge of the container so that the gripper fingers can be secured around the edge, the first and second ridges may block the gripper fingers from an inward direction and therefore interfere with that movement. Thus, if the gripping location is near the first ridge or the second ridge, the gripper fingers may not be able to achieve a grip at those locations, or may only be able to achieve a shallow grip at those locations. Thus, the container structure model of FIG. 17B may identify a first region 161301 and a second region 161302 that surround the first ridge and the second ridge, such that the computing system 1100 may avoid determining a gripping location within the regions 161301, 161302.
[0118] In such a scenario, the computing system 1100 may determine a plurality of grip positions (also referred to as grasp positions) based on the container's edge structure, such as the container's edge structure described by the object recognition template 16128. In some implementations, this determination may involve determining overhang distances at different positions along the container edge, where a large overhang distance at a particular position may indicate that a deep or stable grip may be achievable at that position, while a small overhang distance at a particular position may indicate that only a shallow grip is possible at that position. More specifically, the computing system 1100 may determine a plurality of overhang distances associated with a plurality of respective positions along the container's edge structure, such as the edge structure about the container edge 16128 in FIGS. 17A and 17B. In this example, each overhang distance of the plurality of overhang distances may be a distance that the robot arm (e.g., 3400), or more specifically, the end effector device (e.g., 3500) or its gripper fingers, may extend inwardly toward the container structure and under the container's edge structure. For example, an overhang distance associated with a particular location along the edge structure may indicate a distance that a lower gripper finger of the end effector device may extend inwardly toward the container when the gripper finger is at that location along the edge structure. In some instances, the determination of the overhang distance may be part of a simulation in which the computing system 1100 simulates a robotic arm (e.g., 3400) sliding an end effector device (e.g., 3500) or a portion thereof along the edge structure to different positions. In this embodiment, the computing system may select multiple grip positions from among multiple respective positions along the edge structure of the container based on the multiple overhang distances. For example, the multiple grip positions may be positions having a maximum overhang distance, or more broadly, a portion of the maximum overhang distance. In some scenarios, a higher overhang distance for a particular location along the edge structure may indicate that a wider portion of the edge structure may be engaged by the end effector device (e.g., 3500) at that location, which may facilitate a deeper or more stable grip at that location.
[0119] In an embodiment, the computing system 1100 may determine whether an object (e.g., 3722) in the camera field of view (e.g., 3202) has a container lid and may determine one or more robot interaction positions (e.g., grip positions) based on whether the object has a container lid. For example, the computing system 1100 may determine whether the first image information of step 4002, the second image information of step 4010, and / or the updated sensed structural information indicate the presence of a container lid. The computing system 1100 may determine to detect the presence of a container lid by the multiple grip positions based thereon.
[0120] In an embodiment, where the selected object recognition template may include a container structure model describing at least the container lid structure, the computing system 1100 may determine a plurality of grip positions based on the container lid structure. For example, FIG. 17C shows an object recognition template 17128 that may include a container structure model that identifies the presence of a lid 171281. The container structure model in this example may further identify features in the container lid structure, such as gaps 171282 and 171283, that may interfere with grasping by the grip positions, as being near the gaps 171282 and 171283. Thus, in an embodiment, the computing system 1100 may use the object recognition template 17128 to avoid determining grip positions near the gaps 171282 and 171283. FIG. 17D provides an example in which the gaps 171282 and 171283 of the lid structure may be represented by regions 161303 and 161304. More specifically, regions 161303 and 161304 may encompass gaps 171282 and 171283. In this example, computing system 1100 may determine a grip position such that the grip position is not within regions 161303 and 161304.
[0121] In an embodiment, if the selected object recognition template or model-orientation combination includes an object structure model representing a container structure that is rotationally symmetric, i.e., more specifically, has four-fold rotational symmetry, the computing system 1100 may use the symmetry to simplify the determination of multiple grip positions. For example, the computing system may determine a first grip position based on the object structure model. Because the container structure is rotationally symmetric, the computing system 1100 may determine a second grip position based on the first grip position. For example, the computing system 1100 may determine that the first grip position has a first distance from a corner of the container structure, and the first grip position is on a first side of the container structure. The computing system 1100 in this example may determine the second grip position as a position that is on a second side of the container structure and has the same distance from the corner of the container structure. In the above example, the four-fold rotational symmetry or other rotational symmetry of the container structure may refer to rotational symmetry with respect to a vertical rotation axis that passes through the center of the container structure, and the vertical rotation axis may be an axis perpendicular to the floor or ground.
[0122] 4 may further include triggering, by the computing system 1000, one or more robot interaction commands that may be based on the grip positions or other robot interaction positions discussed above. For example, the computing system 1100 may output one or more robot interaction motion commands that trigger interactions at the one or more robot interaction positions, and the one or more robot interaction motion commands may be generated based on the one or more robot interaction positions.
[0123] In an embodiment, an object structure that is rotationally symmetric, or more specifically, has four-fold rotational symmetry, may affect how object registration is performed. Object registration may involve, for example, the generation of a new object recognition template for an object or object structure that does not match any existing object recognition template. For example, the computing system 1100 may perform object registration when the additional object (e.g., a new container) is in the camera field of view if the additional object does not match any existing object recognition template stored on the non-transitory computer-readable medium 1120 or 1600. Object registration may involve the generation of an additional object recognition template based on image information of the additional object. If the additional object has a rotationally symmetric object structure, the computing system 1100 may generate an additional object recognition template based on one corner of the object structure for the additional object, but not based on the remaining corners of the object structure for the additional object. More specifically, if the object structure of the additional object is rotationally symmetric, or more specifically, has four-fold rotational symmetry, the object structure may have corners that have substantially the same structure and / or sides that have substantially the same structure. Thus, while the computing system 1100 may determine an object recognition template that directly describes one corner and one side of an object structure, the additional object recognition template may not need to also directly describe the remaining corners or sides of the object structure, because that one corner and one side may have substantially the same structure as the corner or side described by the additional object recognition template.
[0124] Additional considerations regarding various embodiments:
[0125] Embodiment 1 relates to a computing system, or a method that may be performed by a computing system, for performing object detection, such as when the computing system executes instructions on a non-transitory computer-readable medium. In this embodiment, the computing system includes a communications interface and at least one processing circuit. The communications interface is configured to communicate with (i) a robot having a robot arm and an end effector device disposed on or forming one end of the robot arm, and (ii) a camera mounted on the robot arm and having a camera field of view. The at least one processing circuit performs the following actions when the object is within the camera field of view: receiving first image information representing at least a first portion of an object structure associated with the object, the first image information being generated by the camera when the camera is in a first camera pose in which the camera is directed toward the first portion of the object structure; generating or updating sensed structure information representing the object structure associated with the object based on the first image information; identifying an object corner associated with the object structure based on the sensed structure information; outputting one or more camera positioning operation commands, when executed by the robot, to the robot arm to move the camera to a second camera pose in which the camera is directed toward the object corner; and receiving second image information representing the object structure. the second image information is generated by the camera while the camera is in a second camera pose; updating the sensed structural information based on the second image information to generate updated sensed structural information; determining an object type associated with the object based on the updated sensed structural information; determining one or more robot interaction positions based on the object type, the one or more robot interaction positions being one or more positions for interaction between the end effector device and the object; and outputting one or more robot interaction action commands that cause interaction at the one or more robot interaction positions, the one or more robot interaction action commands being generated based on the one or more robot interaction positions.In some instances, a computing system may omit output of one or more robot interaction move commands that may be performed by another computing system.
[0126] Example 2 includes the computing system of Example 1. In this embodiment, the at least one processing circuit is configured to determine the object type by comparing the updated sensed structural information to a template candidate set, the template candidate set being a set including object recognition templates describing object structures associated with different object types, and selecting an object recognition template from the template candidate set based on the comparison, the object recognition template being a selected object recognition template, the selected object recognition template representing an object type associated with the object. In this embodiment, the at least one processing circuit is configured to determine one or more robot interaction locations based on the object structure description associated with the selected object recognition template.
[0127] Example 3 includes the computing system of Example 2. In this embodiment, the at least one processing circuit is configured to compare the updated sensed structure information to the set of candidate templates by calculating a set of error values associated with object recognition templates in the candidate template set, each error value in the set of error values indicating a respective degree of deviation between (i) the updated sensed structure information and (ii) a description of the object structure included in the object recognition template associated with the error value, and the selected object recognition template is associated with the lowest error value in the set of error values.
[0128] Example 4 includes the computing system of Example 3. In this embodiment, the updated sensed structure information is a point cloud including a plurality of coordinates for describing an object structure associated with the object, and the at least one processing circuit is configured to calculate, for each object recognition template in the candidate template set, at least one error value based on how closely a coordinate from the plurality of coordinates of the point cloud matches one or more physical features described by a description of the respective object structure included in the object recognition template.
[0129] A fifth embodiment includes the computing system of the third or fourth embodiment. In this embodiment, each of the object recognition templates in the template candidate set describes a set of object structure models. In this embodiment, the template candidate set is a model-orientation candidate set that is a set including model-orientation combinations, and each model-orientation combination in the model-orientation candidate set is a combination of (i) an object structure model that is one of the set of object structure models, and (ii) an orientation of the object structure model. In this embodiment, a set of error values is associated with each of the model-orientation combinations in the model-orientation candidate set, and each error value in the set of error values indicates a respective degree of deviation between (i) the updated sensed structure information and (ii) the object structure model of each model-orientation combination associated with the error value, and the error value is further associated with the object structure model having the orientation of each model-orientation combination.
[0130] Example 6 includes the computing system of Example 5. In this embodiment, the at least one processing circuit is configured to select an object recognition template by selecting a model-orientation combination from a model-orientation candidate set that includes an object structure model described by the selected object recognition template, the selected model-orientation combination being associated with a lowest error value in a set of error values, and the at least one processing circuit is configured to determine one or more robot interaction positions based on the object structure model of the selected model-orientation combination and based on an orientation of the selected model-orientation combination.
[0131] Example 7 includes the computing system of Example 6. In this embodiment, the updated sensed structure information defines an estimated region occupied by a structure of the object for an object in the camera field of view, and the at least one processing circuit is configured to filter the model-orientation candidate set by performing, for each model-orientation combination in the model-orientation candidate set, the following before calculating a set of error values associated with the model-orientation candidate set: determining whether an object structure model of the model-orientation combination substantially falls within the estimated region when the object structure model has an orientation of the model-orientation combination; and removing the model-orientation combination from the model-orientation candidate set in response to determining that the object structure model does not substantially fall within the estimated region when the object structure model has an orientation of the model-orientation combination. After the model-orientation candidate set is filtered, a set of error values is calculated based on the model-orientation combinations remaining in the model-orientation candidate set.
[0132] An eighth embodiment includes the computing system of any one of the third to seventh embodiments. In this embodiment, the updated sensed structure information defines an estimated region occupied by the object structure, and the object recognition templates in the template candidate set each describe a set of object structure models. In this embodiment, the at least one processing circuit is configured to filter the template candidate set by identifying one or more object recognition templates including one or more respective object structure models that do not substantially fall within the estimated region and removing one or more object recognition templates from the template candidate set before calculating a set of error values associated with the object recognition templates in the template candidate set. After the template candidate set is filtered, a set of error values is calculated based on the object recognition templates remaining in the template candidate set.
[0133] Example 9 includes the computing system of any one of Examples 2 to 8. In this embodiment, the at least one processing circuit is configured to, for at least one object recognition template in the template candidate set, adjust a respective object structure description included in the object recognition template based on the updated sensed structure information.
[0134] Example 10 includes the computing system of Example 9. In this embodiment, the object structure description of each of the at least one object recognition template describes physical features of each of the object structures described by the at least one object recognition template, and each object structure description further includes pose information describing a pose of the physical features. The at least one processing circuit is configured to adjust the pose information based on the updated sensed structure information to increase a degree to which the physical features described by the at least one object recognition template match the updated sensed structure information.
[0135] Example 11 includes the computing system of any one of Examples 1 to 10. In this embodiment, when the object is a container and when the object structure is a container structure, the at least one processing circuit is configured to determine the one or more robot interaction positions as a plurality of grip positions associated with gripping the container, the plurality of grip positions being determined based on an object type, the plurality of grip positions being a container type associated with the container.
[0136] Example 12 includes the computing system of Example 11. In this embodiment, when the container structure includes a plurality of ridges protruding from a side surface of the container structure, the at least one processing circuit is configured to detect the plurality of ridges based on the second image information or the updated sensed structure information, the plurality of ridges being ridges detected on the side surface of the container structure, and a container type associated with the container is determined based on the ridges detected on the side surface of the container structure.
[0137] Example 13 includes the computing system of example 11 or 12. In this embodiment, the at least one processing circuit is configured to determine a container type by comparing the updated sensed structural information to a template candidate set, the template candidate set being a set including object recognition templates describing container structures associated with different container types, and selecting an object recognition template from the template candidate set based on the comparison, such that the object recognition template is a selected object recognition template, the selected object recognition template representing a container type associated with the container. When the selected object recognition template includes a container structural model describing at least a rim structure of the container, the at least one processing circuit is further configured to determine a plurality of grip positions based on the rim structure of the container.
[0138] Example 14 includes the computing system of any one of Examples 11 to 13. In this embodiment, the at least one processing circuit is configured to determine a plurality of overhang distances associated with a plurality of respective positions along a lip structure of the container, each overhang distance of the plurality of overhang distances being a distance that the end effector device may extend inwardly under the lip structure of the container toward the container structure when the end effector device is at a respective one of the plurality of positions, and the at least one processing circuit is configured to select a plurality of grip positions from among the plurality of respective positions along the lip structure of the container based on the plurality of overhang distances.
[0139] Example 15 includes the computing system of any one of Examples 11 to 14. In this embodiment, the at least one processing circuit is configured to determine whether the first image information or the second image information indicates a presence of a container lid, and the plurality of grip positions are determined further based on whether the first image information or the second image information indicates a presence of a container lid.
[0140] Example 16 includes the computing system of example 15. In this example, when the selected object recognition template includes a container structure model describing at least the container lid structure, the at least one processing circuit is configured to determine a plurality of grip positions based on the container lid structure.
[0141] Example 17 includes the computing system of any one of Examples 1 to 16. In this embodiment, when an additional object is in the camera field of view and the additional object is rotationally symmetric, at least one processing circuit receives additional image information representing an object structure for the additional object, and generates an additional object recognition template for the set of object recognition templates based on the additional image information. The additional object recognition template is generated based on one corner of the object structure for the additional object and not based on the remaining corners of the object structure for the additional object.
[0142] Example 18 includes any one of the computing systems of embodiments 1 to 17, wherein the first image information is associated with a camera having a first distance from the object, and the second image information is associated with a camera having a second distance from the object that is smaller than the first distance.
[0143] Example 19 includes the computing system of any one of Examples 1 to 18, wherein the sensed structural information based on the first image information describes a plurality of object corners for the object structure, and the at least one processing circuit is configured to identify the object corner by selecting an object corner from among the plurality of object corners, such selection being based on at least one of (i) a respective reachability of the plurality of object corners for robot interaction with the robot arm, or (ii) a respective occlusion level for sensing the plurality of object corners by the camera.
[0144] It will be apparent to those skilled in the relevant art that other suitable modifications and adaptations to the methods and applications described herein can be made without departing from the scope of any of the embodiments. The above-described embodiments are illustrative examples, and the present invention should not be construed as being limited to these particular embodiments. It should be understood that the various embodiments disclosed herein may be combined in different combinations than those specifically presented in the description and accompanying figures. It should also be understood that, by way of example, certain acts or events of any of the processes or methods described herein may be performed in a different order, or may be added, integrated, or omitted entirely (e.g., not all acts or events described may be necessary to perform a method or process). In addition, although certain features of the embodiments herein are described as being performed by a single component, module, or unit for clarity, it should be understood that the features and functions described herein may be performed by any combination of components, modules, or units. Thus, various changes and modifications may be effected by those skilled in the art without departing from the spirit or scope of the invention, as defined in the appended claims.
Claims
1. 1. A computing system comprising: a communication interface configured to communicate with a robot and a camera mounted on the robot, the camera having a camera field of view; at least one processing circuit; The at least one processing circuit comprises: identifying an object corner associated with an object structure based on object structure information representative of at least a portion of the object structure associated with the object; outputting one or more camera placement operation commands based on the object structure information, the one or more camera placement operation commands, when executed by the robot, causing the robot to move the camera to a camera pose such that the camera faces the object corner; updating the object structure information based on image information generated by the camera at the camera pose to generate sensed structure information; determining an object type associated with the object based on the sensed structural information; and A computing system configured to:
2. The at least one processing circuit further comprises: comparing the sensed structural information to a set of template candidates, the set of template candidates including object recognition templates describing object structures associated with different object types; selecting an object recognition template from the set of candidate templates based on the comparison, such that the object recognition template is a selected object recognition template and represents the object type associated with the object; and determining one or more robot interaction locations based on the object type including a description of an object structure associated with the selected object recognition template.
3. the at least one processing circuit is further configured to perform the comparison between the sensed structural information and the set of candidate templates by calculating a set of error values associated with the object recognition templates in the set of candidate templates; each error value in the set of error values indicates a respective degree of discrepancy between (i) the sensed structural information and (ii) a description of the object structure included in a respective object recognition template associated with the respective error value; The computing system of claim 2 , wherein the selected object recognition template is associated with a lowest error value in the set of error values.
4. the sensed structure information is a point cloud including a plurality of coordinates representing the object structure associated with the object; 4. The computing system of claim 3, wherein the at least one processing circuitry is further configured to calculate, for each object recognition template in the candidate template set, at least one error value based on a match between a coordinate from the plurality of coordinates of the point cloud and one or more physical features described by a respective object structure description included in the object recognition template.
5. each of the object recognition templates in the set of candidate templates describes a set of object structure models; The template candidate set is a model-orientation candidate set that is a set including model-orientation combinations, and each model-orientation combination of the model-orientation candidate set is a combination of (i) an object structure model that is one of the set of object structure models, and (ii) an orientation of the object structure model; the set of error values being associated with each of the model-orientation combinations in the set of model-orientation candidates; 4. The computing system of claim 3, wherein each error value in the set of error values indicates a respective degree of discrepancy between (i) the sensed structural information and (ii) an object structural model for a respective model-orientation combination associated with the error value, and the error value is further associated with the object structural model having an orientation of the respective model-orientation combination.
6. the at least one processing circuit is further configured to select the object recognition template by selecting a model-orientation combination from the model-orientation candidate set that includes an object structure model described by the selected object recognition template, the selected model-orientation combination being associated with a lowest error value in the set of error values; 6. The computing system of claim 5, wherein the at least one processing circuitry is further configured to determine the one or more robot interaction positions based on the object structure model of the selected model-orientation combination and based on an orientation of the selected model-orientation combination.
7. the sensed structure information defines, for the object within the camera field of view, an estimated area occupied by the object structure; The at least one processing circuitry may further, prior to calculating the set of error values associated with the model-orientation candidate set, for each model-orientation combination in the model-orientation candidate set: determining whether the object structure model of the model-orientation combination substantially falls within the estimation region when the object structure model has an orientation of the model-orientation combination; removing the model-orientation combination from the model-orientation candidate set in response to determining that the object structure model does not substantially fall within the estimation region when the object structure model has the orientation of the model-orientation combination; configured to filter the model-orientation candidate set by performing The computing system of claim 6 , wherein after the model-orientation candidate set is filtered, the set of error values is calculated based on the model-orientation combinations remaining in the model-orientation candidate set.
8. the sensed structure information defines an estimated area occupied by the object structure; each of the object recognition templates in the set of candidate templates describes a set of object structure models; The at least one processing circuit may further, prior to calculating the set of error values associated with the object recognition templates in the candidate template set: identifying one or more object recognition templates including one or more respective object structure models that do not substantially fall within the estimation region; removing the one or more object recognition templates from the candidate template set; and and configured to filter the set of template candidates by performing The computing system of claim 3 , wherein after the template candidate set is filtered, the set of associated error values is calculated based on the object recognition templates remaining in the template candidate set.
9. 3. The computing system of claim 2, wherein the at least one processing circuit is further configured to, for at least one object recognition template in the template candidate set, adjust a description of a respective object structure included in the object recognition template based on the sensed structural information.
10. the description of each of the object structures of the at least one object recognition template describes physical features of each of the object structures described by the at least one object recognition template, the description of each of the object structures further including pose information describing a pose of the physical features; 10. The computing system of claim 9, wherein the at least one processing circuit is further configured to adjust the pose information based on the sensed structural information so as to increase the degree to which the physical features described by the at least one object recognition template match the sensed structural information.
11. 2. The computing system of claim 1, wherein when the object is a container and when the object structure is a container structure, the at least one processing circuit is further configured to determine the one or more robot interaction positions as a plurality of grip positions associated with grasping the container, the plurality of grip positions being determined based on the object type being a container type associated with the container.
12. When the container structure includes a plurality of ridges protruding from a side surface of the container structure, the at least one processing circuit is further configured to detect the plurality of ridges based on the image information or the sensed structure information, the plurality of ridges being ridges detected on the side surface of the container structure; The computing system of claim 11 , wherein the container type associated with the container is determined based on the detected ridge.
13. The at least one processing circuit further comprises: comparing the sensed structural information to a set of template candidates, the set of template candidates including object recognition templates describing container structures associated with different container types; selecting an object recognition template from the set of template candidates based on the comparison such that the object recognition template is a selected object recognition template, the selected object recognition template representing the container type associated with the container; and and configured to determine the container type by The computing system of claim 11 , wherein when the selected object recognition template includes a container structure model describing at least an edge structure of a container, the at least one processing circuitry is further configured to determine the plurality of grip positions based on an edge structure of the container.
14. The at least one processing circuit is further configured to determine a plurality of overhang distances associated with a plurality of respective positions along a lip structure of the container, each overhang distance of the plurality of overhang distances being a distance an end effector device of the robot can extend inwardly under the lip structure of the container toward the container structure when the end effector device is at a respective one of the plurality of positions; 14. The computing system of claim 13, wherein the at least one processing circuitry is further configured to select the plurality of grip positions from among the plurality of respective positions along the edge structure of the container based on the plurality of overhang distances.
15. The at least one processing circuit is further configured to determine whether the image information indicates a presence of a container lid; The computing system of claim 11 , wherein the plurality of grip positions are determined further based on whether the image information indicates a presence of the container lid.
16. The computing system of claim 15, wherein when the selected object recognition template includes a container structure model describing at least a container lid structure, the at least one processing circuit is further configured to determine the multiple grip positions based on the container lid structure.
17. When an additional object is within the camera field of view and the additional object is rotationally symmetric, the at least one processing circuitry further comprises: receiving additional image information representative of an object structure for the additional object; generating additional object recognition templates for the set of object recognition templates based on the additional image information; The computing system of claim 2 , wherein the additional object recognition template is generated based on one corner of the object structure for the additional object without being based on remaining corners of the object structure for the additional object.
18. A non-transitory computer-readable medium having instructions, The instructions, when executed by at least one processing circuit of a computing system, cause the at least one processing circuit to: identifying an object corner associated with an object structure based on object structure information representative of at least a portion of the object structure associated with the object; outputting one or more camera placement operation commands based on the object structure information, the one or more camera placement operation commands, when executed by a robot having a camera attached thereto, causing the robot to move the camera to a camera pose such that the camera faces the object corner; updating the object structure information based on image information generated by the camera at the camera pose to generate sensed structure information; determining an object type associated with the object based on the sensed structural information; and A non-transitory computer-readable medium configured to:
19. The at least one processing circuit further comprising: determining one or more robot interaction locations based on the object type; and outputting one or more robot interaction action commands for an interaction between the one or more robot interaction locations and the robot.
20. 1. A method performed by a computing system, comprising: identifying, at the computing system, an object corner associated with an object structure based on object structure information, the object structure information representing at least a portion of the object structure associated with an object, the computing system configured to communicate with (i) a robot and (ii) a camera mounted on the robot; outputting one or more camera placement operation commands based on the object structure information, the one or more camera placement operation commands, when executed by the robot, causing the robot to move the camera to a camera pose such that the camera faces the object corner; updating the object structure information based on image information generated by the camera at the camera pose to generate sensed structure information; determining an object type associated with the object based on the sensed structural information; and A method comprising:
Citation Information
Patent Citations
Robot control device and robot control method
JP2017087326A
Object recognition processor and method, and object picking device and method
JP2019185239A
Sample container recognition
WO2019084468A1
Control device and method for a robot system for locating objects and calculating appropriate grasp points for each object
WO2019097004A1
