Method and computing system for performing motion planning based on image information generated by a camera - Patents.com

Abstract

To determine a grip point with a sufficiently high level of reliability and accuracy in a system and method for motion planning. The system is configured to receive first image information generated when an object is or was within a camera field of view of a camera and the camera has a first camera pose. The system is further configured to determine a first estimate of the object structure based on the first image information and identify object corners based on the first estimate of the object structure or based on the first image information. The system is further configured to cause the end effector device to move the camera to a second camera pose and receive second image information representing the object structure. The system is configured to determine a second estimate of the object structure based on the second image information and generate a motion plan based on at least the second estimate.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 946,973, filed December 12, 2019, entitled "ROBOTIC SYSTEM WITH GRIPPING MECHANISM," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to a method and computing system for performing motion planning based on image information generated by a camera. [Background technology]

[0003] As automation becomes more commonplace, robots are being used in more environments, such as warehousing and retail environments. For example, robots may be used to interact with merchandise or other objects in a warehouse. The robot's movements may be fixed or may be based on inputs, such as information generated by sensors in the warehouse. Summary of the Invention

[0004] One aspect of the present disclosure relates to a computing system, method, and non-transitory computer-readable medium that facilitates motion planning and / or estimates structure of an object. In an embodiment, the method may be performed by the computing system, such as by executing instructions on the non-transitory computer-readable medium. The computing system includes a communications interface and at least one processing circuit. The communications interface is configured to communicate with (i) a robot having an end effector device, and (ii) a camera mounted on the end effector device and having a camera field of view. The at least one processing circuit receives first image information representing at least a first exterior surface of an object structure associated with the object when the object is or was within the camera field of view, the first image information being generated by the camera when the camera has a first camera pose in which the camera is oriented toward the first exterior surface such that the camera field of view encompasses the first exterior surface; determining a first estimate of the object structure based on the first image information; identifying a corner of the object structure based on the first estimate of the object structure or based on the first image information; and determining a second camera pose that, when adopted by the camera, orients the camera toward the corner of the object structure such that the camera field of view encompasses the corner and at least a portion of the second exterior surface of the object structure. and, when executed by the robot, outputting to the end effector device one or more camera positioning action commands that move the camera to a second camera pose; receiving second image information representing the object structure, the second image information being generated by the camera while the camera has the second camera pose; determining a second estimate of the object structure based on the second image information; generating a motion plan based on at least the second estimate of the object structure, the motion plan being for causing a robotic interaction between the robot and the object; and outputting one or more object interaction action commands for causing the robotic interaction, the one or more object interaction action commands being generated based on the motion plan. [Brief description of the drawings]

[0005] [Figure 1A] 1 illustrates a system for processing image information consistent with embodiments herein. [Figure 1B] 1 illustrates a system for processing image information consistent with embodiments herein. [Figure 1C] 1 illustrates a system for processing image information consistent with embodiments herein. [Figure 1D] 1 illustrates a system for processing image information consistent with embodiments herein.

[0006] [Figure 2A] A block diagram illustrating a computing system configured to receive and process image information and / or execute a motion plan consistent with embodiments herein is provided. [Figure 2B] A block diagram illustrating a computing system configured to receive and process image information and / or execute a motion plan consistent with embodiments herein is provided. [Figure 2C] A block diagram illustrating a computing system configured to receive and process image information and / or execute a motion plan consistent with embodiments herein is provided. [Figure 2D] A block diagram illustrating a computing system configured to receive and process image information and / or execute a motion plan consistent with embodiments herein is provided.

[0007] [Figure 3A] 1 illustrates an environment having a robotic arm and end effector device for executing motion planning consistent with embodiments herein. [Figure 3B] 1 illustrates an environment having a robotic arm and end effector device for executing motion planning consistent with embodiments herein.

[0008] [Figure 4A]1 illustrates various end effector devices consistent with embodiments herein. [Figure 4B] 1 illustrates various end effector devices consistent with embodiments herein. [Figure 4C] 1 illustrates various end effector devices consistent with embodiments herein. [Figure 4D] 1 illustrates various end effector devices consistent with embodiments herein.

[0009] [Diagram 5] 1 shows a flow diagram illustrating an exemplary method for generating an action plan, according to an embodiment herein.

[0010] [Figure 6A] 1 illustrates various aspects of generating image information representing an object or stack of objects according to embodiments herein. [Figure 6B] 1 illustrates various aspects of generating image information representing an object or stack of objects according to embodiments herein. [Figure 6C] 1 illustrates various aspects of generating image information representing an object or stack of objects according to embodiments herein.

[0011] [Figure 7A] 1 illustrates various aspects of generating image information representing an object or stack of objects according to embodiments herein. [Figure 7B] 1 illustrates various aspects of generating image information representing an object or stack of objects according to embodiments herein. [Figure 7C] 1 illustrates various aspects of generating image information representing an object or stack of objects according to embodiments herein. [Figure 7D] 1 illustrates various aspects of generating image information representing an object or stack of objects according to embodiments herein.

[0012] [Figure 8A]1 illustrates an end effector apparatus at various stages of motion planning, according to embodiments herein. [Figure 8B] 1 illustrates an end effector apparatus at various stages of motion planning, according to embodiments herein. [Figure 8C] 1 illustrates an end effector apparatus at various stages of motion planning, according to embodiments herein. [Figure 8D] 1 illustrates an end effector apparatus at various stages of motion planning, according to embodiments herein. [Figure 8E] 1 illustrates an end effector apparatus at various stages of motion planning, according to embodiments herein. [Figure 8F] 1 illustrates an end effector apparatus at various stages of motion planning, according to embodiments herein. [Figure 8G] 1 illustrates an end effector apparatus at various stages of motion planning, according to embodiments herein. [Figure 8H] 1 illustrates an end effector apparatus at various stages of motion planning, according to embodiments herein. [Figure 8I] 1 illustrates an end effector apparatus at various stages of motion planning, according to embodiments herein.

[0013] [Figure 9A] 1 illustrates various aspects of updating an estimate for a stacked structure representing a stack of objects, according to embodiments herein. [Figure 9B] 1 illustrates various aspects of updating an estimate for a stacked structure representing a stack of objects, according to embodiments herein. [Figure 9C] 1 illustrates various aspects of updating an estimate for a stacked structure representing a stack of objects, according to embodiments herein.

[0014] [Figure 10] A flow diagram is provided illustrating an exemplary method for controlling a robot to engage an object and move the object to a destination location according to embodiments herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] One aspect of the disclosure relates to using multiple sets of image information, representing multiple views or viewpoints, to execute a motion plan. The motion plan may involve, for example, determining a trajectory to be followed by an end effector device (e.g., a robot gripper or a robot hand) located at one end of a robotic arm of a robot. The trajectory may be part of a robotic interaction between the robotic arm and an object, such as a box or crate holding goods in a warehouse or retail space. For example, the robotic interaction may perform an operation in which the robotic arm picks up an object and moves the object to a desired destination location. In some cases, the object may be part of a stack of objects placed on a pallet, and the robotic arm may be used to move all of the objects from the pallet to another location.

[0016] In an embodiment, the multiple viewpoints may refer to viewpoints of a camera configured to generate 2D or 3D image information describing the camera and / or the robot's environment. In some cases, the camera may be mounted on or otherwise attached to the end effector device. In such cases, the computing system may move the camera by moving the end effector device. More specifically, the computing system may move the camera to different camera poses by the end effector device. In a first camera pose, the camera may be positioned, for example, directly above the object and may generate first image information representing a top view of the object. In such an example, the first image information may represent a top surface (also referred to as a top surface) of the object. In some cases, the computing system may use the first image information to determine a first estimate of the structure of the object (also referred to as an object structure) and / or generate an initial motion plan for causing an interaction between the robot and the object.

[0017] In an embodiment, the computing system may generate an updated motion plan based on second image information representing another view of the object. More specifically, the first estimate of the object structure or the initial motion plan generated based on the first image information may lack a high level of accuracy or certainty. For example, if the first image information represents a plan view of the object, the plan view may provide some information on the object dimensions, such as object length or object width, but may provide no information or limited information on the object dimensions, such as object height. Therefore, using only the first image information to execute the motion plan may lead to unreliable results. Therefore, the computing system may cause the camera to generate second image information, which may represent another view of the object.

[0018] In an embodiment, the computing system may cause the camera to generate second image information by using the first image information to identify a corner of the object (also referred to as an object corner). In this embodiment, the computing system may output a motion command that causes the end effector device to move the camera to a second camera pose, in which the camera faces the object corner. The second image information may be generated by the camera while the camera has the second camera pose. In one scenario, the second image information may represent a perspective view of the object, in which one or more outer side surfaces (also referred to as sides) of the object are represented by the second image information. Thus, the second image information may provide additional information regarding the structure of the object, such as information that can be used to estimate the object height. In some cases, the computing system may use the second image information (either alone or in conjunction with the first image information) to determine a second estimate of the object structure and / or to determine an updated motion plan. As a result, the second estimate of the object structure and / or the updated motion plan may have a higher degree of reliability or certainty relative to a first estimate or an initial motion plan generated based only on the first image information.

[0019] In an embodiment, the computing system may be configured to estimate the structure of the stack after the object is removed. More specifically, the computing system may use the estimate of the object structure to determine an estimate of the stack structure. For example, the computing system may use the estimated dimensions of the removed object structure to determine which portion of the estimate of the stack structure corresponds to the removed object and remove (e.g., mask out) that portion from the estimate of the stack structure. As a result, the computing system may generate an updated estimate of the stack structure. The updated estimate may represent the stack after the object is removed. In some cases, the computing system may use the updated estimate of the stack structure to identify a remaining object corner (e.g., a convex corner) of the stack, which may correspond to an object corner (e.g., a convex corner) of the objects remaining in the stack. The computing system may select one of the object corners that may belong to one of the remaining objects, and further move the camera to a camera pose where the camera faces the selected object corner. The camera may generate image information while in that camera pose, and the image information may be used by a computing system to generate a motion plan for moving the remaining object.

[0020] FIG. 1A illustrates a system 1000 for executing a motion plan based on image information. More specifically, the system 1000 may include a computing system 1100 and a camera 1200. In this example, the camera 1200 may be configured to generate image information that depicts or otherwise represents an environment in which the camera 1200 is located, or more specifically, represents an environment in a field of view (also referred to as a camera field of view) of the camera 1200. The environment may be, for example, a warehouse, a manufacturing plant, a retail space, or some other facility. In such cases, the image information may represent objects located in such a facility, such as containers (e.g., boxes) that hold goods or other items. The system 1100 may be configured to receive and process the image information, such as by executing a motion plan based on the image information, as discussed in more detail below. The motion plan may be used to control a robot, for example, to facilitate robotic interaction between the robot and the container or other object. The computing system 1100 and the camera 1200 may be located in the same facility or may be located remotely from each other. For example, the computing system 1100 may be part of a cloud computing platform hosted in a data center remote from the warehouse or retail space and may be in communication with the camera 1200 via a network connection.

[0021] In an embodiment, the camera 1200 may be a 3D camera (also referred to as a spatial structure sensing camera or device) configured to generate spatial structure information about an environment in the field of view of the camera, and / or may be a 2D camera configured to generate a 2D image describing the visual appearance of the environment in the field of view of the camera. The spatial structure information may include depth information describing the respective depth values ​​of various positions relative to the camera 1200, such as positions on the surface of various objects in the field of view of the camera 1200. The depth information in this example may be used to estimate how the objects are spatially arranged in three-dimensional (3D) space. In some instances, the spatial structure information may include a point cloud describing positions on one or more surfaces of objects in the field of view of the camera. More specifically, the spatial structure information may describe various positions on the structure of the object (also referred to as the object structure).

[0022] In an embodiment, the system 1000 may be a robotic handling system for interacting with various objects in the environment of the camera 1200. For example, FIG. 1B illustrates a robotic handling system 1000A, which may be an embodiment of the system 1000 of FIG. 1A. The robotic handling system 1000A may include a computing system 1100, a camera 1200, and a robot 1300. In an embodiment, the robot 1300 may be used to interact with one or more objects in the environment of the camera 1200, such as boxes, crates, bins, or other containers that hold goods in a warehouse. For example, the robot 1300 may be configured to pick up containers from one location and move them to another location. In some cases, the robot 1300 may be used to perform an unpalletizing operation, where stacked containers are taken down and moved, for example, to a conveyor belt, or a palletizing operation, where containers are stacked on a pallet to prepare for transportation.

[0023] In an embodiment, the camera 1200 may be part of or otherwise attached to the robot 1300, as depicted in FIG. 1B. In some cases, the camera 1200 may be attached to a movable portion of the robot 1300, which may provide the robot 1300 with the ability to move the camera 1200. For example, FIG. 1C depicts an example in which the robot 1300 includes a robot arm 1400 and an end effector device 1500 that forms one end of the robot arm 1400 or is attached to one end of the robot arm 1400. The end effector device 1500 may be movable by the movement of the robot arm 1400. In the example of FIG. 1C, the camera 1200 may be mounted on or otherwise attached to the end effector device 1500. When the end effector device 1500 is a robot hand (e.g., a gripper device), the camera 1200 may be referred to as a handheld camera. By attaching the camera 1200 to the end effector device 1500, the robot 1300 may be able to move the camera 1300 to different poses (also referred to as camera poses) by the movement of the robot arm 1400 and / or the end effector device 1500. For example, as discussed in more detail below, the end effector device 1500 may move the camera 1200 to a camera pose that is optimal or particularly useful for sensing information about objects in the environment of the camera 1200 or the environment of the robot 1300. In another embodiment, the camera 1200 may be separate from the robot 1200, as shown in FIG. 1D. For example, the camera 1200 in such an embodiment may be a fixed camera mounted on a ceiling or some other location in a warehouse or other facility.

[0024] In an embodiment, the computing system 1100 of FIGS. 1A-1D may form or be part of a robot control system (also referred to as a robot controller) that is part of the robot manipulation system 1000A. The robot control system may be, for example, a system configured to generate movement commands or other commands for the robot 1300. In such an embodiment, the computing system 1100 may be configured to generate such commands based on spatial structure information generated, for example, by the camera 1200. In an embodiment, the computing system 1100 may form or be part of a vision system. The vision system may be, for example, a system that generates visual information describing an environment in which the robot 1300 is located, or more specifically, an environment in which the camera 1200 is located. The visual information may include 3D or 2D images as discussed above, or some other image information. In some cases, if the computing system 1100 forms a vision system, the vision system may be part of the robot control system discussed above, or may be separate from the robot control system. When the vision system is separate from the robot control system, it may be configured to output information describing the environment in which the robot 1300 is located. The information may be output to the robot control system, which may receive such information from the vision system and control the movement of the robot 1300 based on the information.

[0025] In an embodiment, when the computing system 1100 is configured to generate one or more motion commands, the motion commands may include, for example, a camera placement motion command, an object interaction motion command, and / or a gripper member placement command. In this embodiment, the camera placement motion command may be a motion command used to control the placement of the camera 1200, and more specifically, to cause the robot 1300 to move the camera 1200 to a particular camera pose, which may include a combination of a particular camera position and a particular camera orientation. The object interaction motion command may be used to control the interaction between the robot 1300 and one or more objects, such as a stack of containers in a warehouse. For example, the object interaction motion command may cause the robot arm 1400 of the robot 1300 to move the end effector device 1500 close to one of the containers, cause the end effector device 1500 at one end of the robot arm 1400 to pick up the container, and then cause the robot arm 1400 to move the container to a desired destination location (e.g., a conveyor belt). If the end effector device 1500 has at least one gripper member, the gripper member position command may cause the gripper member to be moved relative to the rest of the end effector device to position or otherwise position the gripper member in position to grip a portion of a container.

[0026] In an embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 through a direct connection, such as a connection provided through a dedicated wired communication interface, such as an RS-232 interface, a universal serial bus (USB) interface, and / or a local computer bus, such as a peripheral component interconnect (PCI) bus. In an embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 through a network. The network may be any type and / or form of network, such as a personal area network (PAN), a local area network (LAN), e.g., an intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The network may utilize different technologies and layers or stacks of protocols, including, for example, Ethernet protocols, Internet Protocol Suite (TCP / IP), Asynchronous Transfer Mode (ATM) technologies, Synchronous Optical Networking (SONET) protocols, or Synchronous Digital Hierarchy (SDH) protocols.

[0027] In an embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 directly or through an intermediate storage device, or more broadly, an intermediate non-transitory computer readable medium. Such an intermediate non-transitory computer readable medium may be external to the computing system 1100 and may, for example, serve as an external buffer or repository for storing image information generated by the camera 1200, for storing sensor information generated by the robot 1300, and / or for storing commands generated by the computing system 1100. For example, if an intermediate non-transitory computer readable medium is used to store image information generated by the camera 1200, the computing system 1100 may read or otherwise receive image information from the intermediate non-transitory computer readable medium. Non-transitory computer readable media may include electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. The non-transitory computer readable medium may form, for example, a computer diskette, a hard disk drive (HDD), a solid state drive (SDD), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read only memory (CD-ROM), a digital versatile disc (DVD), and / or a memory stick.

[0028] As mentioned above, the camera 1200 may be a 3D camera and / or a 2D camera. The 2D camera may be configured to generate a 2D image, such as a color image or a grayscale image. The 3D camera may be a depth-sensing camera, such as a time-of-flight (TOF) camera or a structured light camera, or any other type of 3D camera. In some cases, the 3D camera may include an image sensor, such as a charge-coupled device (CCD) sensor and / or a complementary metal-oxide semiconductor (CMOS) sensor. In an embodiment, the 3D camera may include a laser, a LIDAR device, an infrared device, a light / dark sensor, a motion sensor, a microwave detector, an ultrasonic detector, a radar detector, or any other device configured to capture spatial structure information.

[0029] As described above, the image information may be processed by computing system 1100. In an embodiment, computing system 1100 may include or be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop, etc.), a smartphone, a tablet computing device, and / or any other computing system. In an embodiment, all of the functionality of computing system 1100 may be performed as part of a cloud computing platform. Computing system 1100 may be a single computing device (e.g., a desktop computer) or may include multiple computing devices.

[0030] 2A provides a block diagram illustrating an embodiment of a computing system 1100. The computing system 1100 includes at least one processing circuit 1110 and a non-transitory computer-readable medium (or media) 1120. In an embodiment, the processing circuit 1110 includes one or more processors, one or more processing cores, a programmable logic controller ("PLC"), an application specific integrated circuit ("ASIC"), a programmable gate array ("PGA"), a field programmable gate array ("FPGA"), any combination thereof, or any other processing circuit. In an embodiment, the non-transitory computer-readable medium 1120 that is part of the computing system 1100 may be in place of or in addition to the intermediate non-transitory computer-readable media discussed above. The non-transitory computer-readable medium 1120 is a storage device, such as an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, such as a computer diskette, a hard disk drive (HDD), a solid state drive (SSD), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, any combination thereof, or any other storage device. In some instances, the non-transitory computer-readable medium 1120 may include multiple storage devices. In certain cases, the non-transitory computer-readable medium 1120 is configured to store image information generated by the camera 1200. The non-transitory computer-readable medium 1120 may alternatively or additionally store computer-readable program instructions that, when executed by the processing circuit 1110, cause the processing circuit 1110 to perform one or more of the techniques described herein, such as the operations described with respect to FIG. 5.

[0031] FIG. 2B depicts an embodiment of the computing system 1100, the computing system 1100A, including a communication interface 1130. The communication interface 1130 may be configured to receive image information generated by, for example, the camera 1200 of FIGS. 1A-1D. The image information may be received via an intermediate non-transitory computer-readable medium or network as discussed above, or via a more direct connection between the camera 1200 and the computing system 1100 / 1100A. In an embodiment, the communication interface 1130 may be configured to communicate with the robot 1300 of FIGS. 1B-1D. If the computing system 1100 is not part of a robot control system, the communication interface 1130 of the computing system 1100 may be configured to communicate with the robot control system. The communication interface 1130 may include, for example, a communication circuit configured to communicate via a wired or wireless protocol. By way of example, the communications circuitry may include an RS-232 port controller, a USB controller, an Ethernet controller, a Bluetooth controller, a PCI bus controller, any other communications circuitry, or a combination thereof.

[0032] In an embodiment, the processing circuit 1110 may be programmed by one or more computer readable program instructions stored on a non-transitory computer readable medium 1120. For example, Figure 2C illustrates an embodiment of computing system 1100 / 1100A, computing system 1100B, in which the processing circuit 1110 is programmed by one or more modules, including a motion planning module 1122 and a grip control module 1124, discussed in more detail below.

[0033] In an embodiment, the motion planning module 1122 may be configured to determine robot motions for interacting with a container, such as a robot motion for an unpalletizing operation, in which the motion planning module 1122 generates object interaction motion commands to control the robot arm 1400 and / or end effector device 1500 of FIG. 1C or FIG. 1D to pick up a container from a pallet and move the container to a desired destination location. In some cases, the motion planning module 1122 may be configured to generate a motion plan for the robot 1300, or more specifically, for the robot arm 1400 and / or end effector device 1500, to accomplish the unpalletizing operation or other interaction. In some cases, the motion plan may include a trajectory for the end effector device 1500 to follow. The trajectory may cause the end effector device 1500 to approach a container or other object, engage the container (e.g., by picking up the container), and move the container to a desired destination location.

[0034] In an embodiment, where the end effector device 1500 of FIGS. 1C and 1D has one or more grippers, the computing system 1100B of FIG. 2C may be configured to execute the grip control module 1124 to control the one or more grippers. As discussed in more detail below, the one or more grippers may be movable to different positions and may transition from an open state to a closed state to pick up or otherwise engage an object, and may transition from a closed state to an open state to release an object. In this embodiment, the grip control module 1124 may be configured to control the movement of the one or more grippers to different positions and / or control whether the one or more grippers are in an open or closed state. It will be understood that the functionality of the modules discussed herein is representative and not limiting.

[0035] In various embodiments, the terms "computer readable instructions" and "computer readable program instructions" are used to describe software instructions or computer code configured to perform various tasks and operations. In various embodiments, the term "module" refers broadly to a collection of software instructions or code configured to cause the processing circuitry 1110 to perform one or more functional tasks. The modules and computer readable instructions may be described as causing a processing circuitry or other hardware component to perform various operations or tasks when executing the modules or computer readable instructions.

[0036] In an embodiment, the non-transitory computer-readable medium 1120 may store or otherwise include one or more object templates 1126 (e.g., container templates) used to describe a particular visual design, physical design, or other aspect of an object design for an object or object type (also referred to as a class of objects). For example, if the object is a container, the object templates 1126 may each describe a particular container design, which may include a visual design for the container or container type (also referred to as a class of containers) and / or a physical design for the container or container type. In some implementations, each of the object templates 1126 may include an object appearance description (also referred to as visual description information) that describes the visual design, and / or an object structure description (also referred to as structure description information) that describes the physical design. In some instances, the object appearance description may include one or more visual descriptors that represent patterns or other visual details (e.g., logos or pictures) that form the visual design. In some instances, the description of the object structure may include values ​​describing the size (e.g., a dimension such as length or width) of the object or object type, or a size associated with the object or object type, or a shape of the object or object type, or a shape associated with the object or object type, and / or a computer-aided design (CAD) file describing the structure of the object or object type. In some cases, the object templates 1126 may be used to perform object recognition, which may involve determining whether an object in the environment of the camera 1200 and / or robot 1300 of FIGS. 1A-1D matches any of the object templates 1126, and a match may indicate that the object is associated with the object type described by the matching template. The object templates 1126 may be generated, for example, as part of an object registration process, and / or may be received (e.g., downloaded) from a resource such as a server.Templates are discussed in more detail in U.S. Patent Application No. 16 / 991,466 (Attorney Docket No. MJ0054-US / 0077-0012US1) and U.S. Patent Application No. 16 / 991,510 (Attorney Docket No. MJ0051-US / 0077-0011US1), the entire contents of which are incorporated herein by reference.

[0037] 3A and 3B show an example of an environment in which motion planning for robot interaction may take place. More specifically, the environment includes a computing system 1100, a camera 3200 (which may be an embodiment of camera 1200 of FIGS. 1A-1D), and a robot 3300 (which may be an embodiment of robot 1300). In this embodiment, the robot 3300 may include a robot arm 3400 and an end effector device 3500. In an embodiment, the end effector device 3500 may form one end of the robot arm 3400 or be attached to one end of the robot arm 3400.

[0038] In the example of Figure 3A, the robot 3300 may operate by the robotic arm 3400 to move the end effector device 3500 toward one or more objects, such as a stack of crates or other containers on a pallet, engage the one or more objects, and move the one or more objects from the pallet to another location (e.g., as part of an unpalletizing operation). More specifically, Figures 3A and 3B depict an environment having a stack 3710 of objects, or more specifically, a stack of crates or other containers. In some scenarios, as shown in Figure 3B, some or all of the containers may hold smaller objects (which may be referred to as items within the container) within the container. Stack 3710 of Figures 3A and 3B may include at least objects 3711-3719 and 3731-3733, while end effector device 3500 may be used to pick up one of the objects in stack 3710, such as object 3711 (e.g., pick up a container in the stack) and move the object from stack 3710 to a destination location, such as a location on conveyor 3800. To pick up object 3711, end effector device 3500 may move and tilt to align with object 3711. Movement of end effector device 3500 may involve movement of robot arm 3400, such as one or more joints of robot arm 3400 rotating relative to one another. In the environment depicted in Figures 3A and 3B, the objects on the pallet may have a 3D pattern on at least one of their outer side surfaces. For example, the 3D pattern may be a pattern of ridges (also referred to as a ridge pattern) protruding from an outer side surface. As an example, Figure 3A depicts ridge pattern 3711A on an outer side surface of object 3711. In some cases, objects on a pallet may have visual details, such as a logo or other visual pattern, that form a 2D pattern on their outer side surface.

[0039] 4A and 4B depict end effector device 3500A, which may be an embodiment of end effector device 3500. In this embodiment, end effector device 3500A includes a mounting structure 3502, a camera 3200, a first gripper member 3510 (also referred to as a first grip member), a second gripper member 3520, and a third gripper member 3530. FIG. 4C depicts end effector device 3500B, which is similar to end effector device 3500A, but does not have the third gripper member 3530. The camera 3200 of FIGS. 4A-4C may be mounted or otherwise attached on a first surface 3503 (e.g., a top surface) of the mounting structure 3502 (e.g., a mounting plate), while the gripper members 3510-3530 may be mounted or otherwise attached on an opposing second surface 3504 (e.g., a bottom surface) of the mounting structure 3502. In some cases, the end effector device 3500 / 3500A may be mounted on or otherwise coupled to the robot arm 3400 at a first surface (e.g., a top surface) of the mounting structure 3502. For example, the first surface 3503 may have a mounting bracket disposed thereon, which may act as a connection point between the end effector device and the robot arm 1400 / 3400. In these cases, a second surface (e.g., a bottom surface) of the end effector device may be oriented to face one or more crates or other containers in the environment of the robot 3300.

[0040] In an embodiment, some or all of the first gripper member 3510, second gripper member 3520, and third gripper member 3530 may each include a gripper body formed by or attached to a respective gripper finger assembly. For example, FIG. 4D depicts gripper member 3530A, which may be an embodiment of gripper member 3530, including gripper finger assembly 3531 that is part of or attached to gripper body 3533. Gripper finger assembly 3531 may be used to grasp an object (e.g., a container) by clamping or pinching around a portion of the object, such as a corner of lip 3701A that forms an outer edge of container 3701. In the example of Fig. 4D, the gripper finger assembly 3531 may include two components, also referred to as gripper fingers 3531A, 3531B, which may be movable relative to each other (e.g., both gripper fingers 3531A and 3531B may move toward or away from each other, or one of the gripper fingers 3531A / 3531B may remain stationary while the other gripper finger 3531B / 3531A moves). For example, the two gripper fingers 3531A, 3531B may form a chuck or clamp, where the two gripper fingers are movable toward each other to grasp a portion of an object or tighten a grip around the object, and the two gripper fingers 3531A, 3531B are movable away from each other to loosen the grip or release the object. In some scenarios, one of the two gripper fingers (e.g., 3531A) may be the upper gripper finger and the other of the two gripper fingers (e.g., 3531B) may be the lower gripper finger. In the example of Figure 4D, the gripper member 3530A may further include a back plate 3532, while the gripper body 3533 may be movable relative to the back plate 3532. The relative movement may be inward toward the center of the mounting structure 3502 of Figures 4A and 4B, or outward away from the center of the mounting structure 3502.The gripper member 3530A may further include a sensor 3517 configured to detect inward movement of the gripper finger assembly 3531 and the gripper body 3533. In an embodiment, the first gripper member 3510 and the second gripper member 3520 may each have a gripper finger assembly the same as or similar to that depicted in FIG. 4D. Each such gripper finger assembly may include at least a pair of gripper fingers for securing around a portion of an object. When the end effector device 3500 is used to grasp a portion of an object, such as a portion of a lip of a container, at least one of the pair of gripper fingers of the gripper member (e.g., 3510) may be movable in a direction (e.g., upward) toward the other gripper finger such that the two gripper fingers contact the portion of the object, more specifically, to pinch the portion of the object. When the end effector device 3500 is to release the container, at least one gripper finger may be movable in an opposite direction (e.g., downward) away from the other gripper finger such that the pair of gripper fingers releases a portion of the object.

[0041] In the embodiment of Figures 4A and 4B, the first gripper member 3510 and the second gripper member 3520 may each be a movable gripper member, while the third gripper member 3530 may be a fixed gripper member. More specifically, the first gripper member 3510 may be movable (e.g., slidable) along a first edge 3501A of the mounting structure 3502, while the second gripper member 3520 may be movable along a second edge 3501B of the mounting structure 3502 that is perpendicular to the first edge 3501A. More specifically, the first gripper member 3510 may be movable along a first axis, such as the Y' axis of Figure 4B, which may be the longitudinal axis of the first rail 3540. The second gripper member 3520 may be movable along a second axis, such as the X' axis of Figure 4B, which may be the longitudinal axis of the second rail 3542. The first axis may be parallel to the first edge 3501A while the second axis may be parallel to the second edge 3501B such that the first rail 3540 may be perpendicular to the second rail 3542. In this example, the third gripper member 3530 may be located at a corner of the mounting structure 3502, which may be at or near where the first axis intersects with the second axis in FIG. 4B. Each of the gripper members 3510-3530 may be capable of gripping or otherwise engaging a respective portion of an object, or more specifically, a respective portion of the object structure, as discussed in more detail below. In some scenarios, the first gripper member 3510 operates to engage one side of the object (e.g., the left side), while the second gripper member 3520 operates to engage another side of the object (e.g., the front), and the third gripper member 3530 operates to engage a corner of the object. For example, a first gripper member 3510 and a second gripper member 3520 may respectively engage two perpendicular sides of an object, while a third gripper member 3530 may engage a corner of the object located between the two perpendicular sides.

[0042] As described above, the first gripper member 3510 may be movable relative to a second surface (e.g., a bottom surface) of the mounting structure 3502 via a first rail 3540, while the second gripper member 3520 may be movable relative to a second surface of the mounting structure 3502 via a second rail 3542. The first rail 3540 may extend along a Y′ axis, while the second rail 3542 may extend along an X′ axis perpendicular to the Y′ axis. In some scenarios, the first rail 3540 may extend from a position near a first corner of the mounting structure 3502 (e.g., the corner where the third gripper member 3530 is located) to another position near a second corner of the mounting structure 3502. Further, the second rail 3542 in such scenarios may extend from a position near a first corner of the mounting structure 3502 to a position near a third corner of the mounting structure 3502. The first rail 3540 and the second rail 3542 may allow the end effector device 3500A to accommodate a range of different object sizes. For example, when the first gripper member 3510 and the second gripper member 3520 grip an object, sliding the first gripper member 3510 along the first rail 3540 may be different from sliding the second gripper member 3520 along the second rail 3542 (i.e., the grip point at which the end effector device 3500A grips the object changes).

[0043] More specifically, sliding the first gripper member 3510 may enable the end effector device 3500A to accommodate different values ​​of a first dimension (e.g., width dimension) of various objects, while sliding the second gripper member 3520 along the second rail 3550 may enable the end effector device 3500A to accommodate different values ​​of a second dimension (e.g., length dimension) of various objects. For example, the end effector device 3500A may have a variable grip size (also referred to as a variable span), which may describe the size of the area defined by where the first gripper member 3510 and the second gripper member 3520 are located. The area may represent the reach or coverage of the gripper members 3510, 3520. More specifically, the region may have a first corner at the location of the first gripper member 3510, a second corner at the location of the third gripper member 3520, and a third corner at a location where a first axis (e.g., Y' axis) intersects with a second axis (e.g., X' axis), also referred to as a crossover location. Increasing the size of the region, and therefore the grip size of the end effector device 3500A, may increase the object size that the end effector device 3500A can grasp. As the first gripper member 3510 or the second gripper member 3510 move away from the crossover location, the grip size may increase. More specifically, the grip size of the end effector device 3500A may be defined by at least a first dimension and a second dimension. The first dimension of the grip size may be defined by the distance from the crossover location to the location of the first gripper member, while the second dimension of the grip size may be defined by the distance from the crossover location to the location of the second gripper member. In this example, as the first gripper member 3510 moves away from the intersection position along the first rail 3540, a first dimension of the grip size increases in value, while as the second gripper member moves away from the intersection position along the second rail 3542, a second dimension of the grip size increases in value.

[0044] In an embodiment, the first rail 3540 and the second rail 3542 may have the same size. In another embodiment, the first rail 3540 and the second rail 3542 may have different sizes. For example, as shown in FIG. 4B, the second rail 3542 may be longer than the first rail 3540. As described above, the grip size of the end effector device 3500A may have a first dimension defined by the distance between the first gripper member 3510 and the intersection where the X′ axis intersects the Y′ axis, and may have a second dimension defined by the distance between the second gripper member 3520 and the intersection. In this embodiment, the longer size of the second rail 3542 may allow the maximum distance between the second gripper member 3520 and the intersection to be greater than the maximum distance between the first gripper member 3510 and the intersection. That is, the maximum distance between the second gripper member 3520 and the intersection is based on the size of the second rail 3542, while the maximum distance between the first gripper member 3510 and the intersection is based on the size of the first rail 3540. Thus, a longer sized second rail 3542 may allow the end effector device 3500A to have a larger maximum grip size for the second dimension than the maximum grip size for the first dimension. Such an embodiment may allow the end effector device 3500A to accommodate an object having a first dimension (e.g., width dimension) that differs in value from the second dimension (e.g., length dimension). For example, if the end effector device 3500A is used to grip a rectangular object having a first side and a second side that is longer than the first side, the end effector device 3500A may be oriented such that the second rail 3542 is aligned with the second side of the rectangular object. This is because the second rail 3542 is longer than the first rail 3540, such that the maximum distance that the second gripper member 3520 can slide (relative to the intersection points discussed above) is greater than the maximum distance that the first gripper member 3510 can slide (relative to the intersection points).As a result, the second rail 3542 and second gripper member 3520 may be better able to accommodate the longer second side of the rectangular object, while the first rail 3540 and first gripper member 3510 can be used to accommodate the shorter first side of the rectangular object.

[0045] In embodiments, the computing system 1100 and / or the robot 1300 / 3300 may be configured to control the amount the first gripper member 3510 moves along the first rail 3540 and / or the amount the second gripper member 3520 moves along the second rail 3542. For example, as discussed in more detail below, the computing system 1100 and / or the robot 1300 / 3300 may be configured to control one or more actuators used to cause movement of the first gripper member 3510 and the second gripper member 3520 and / or control a braking mechanism used to stop the movement. The one or more actuators may be controlled, for example, by one or more gripper member positioning commands, which the computing system 1100 may be configured to generate and output to the robot 1300 / 3300 (e.g., via a communication interface). In some scenarios, the computing system 1100 and / or the robot 1300 / 3300 may control the amount of movement of each of the first gripper member 3510 and the second gripper member 3520 based on the object size (e.g., based on the respective values ​​of the length dimension and the width dimension) relative to the object to be grasped by the end effector device 3500A. For example, the amount of movement of the first gripper member 3510 along the first rail 3540 may be controlled such that the first dimension of the grip size of the end effector device 3500A has at least a predefined percentage value relative to the first dimension of the object (e.g., the first dimension of the grip size is at least 50% of or equal to the value of the width dimension of the object). Similarly, the amount that the second gripper member 3520 moves along the second rail 3542 may be controlled in a manner such that the second dimension of the grip size of the end effector device 3500A has a value that is at least a predefined percentage of a value relative to the second dimension of the object (e.g., the second dimension of the grip size is at least 50% of the value of the length dimension of the object or equal to the value of the length dimension). In such an example, a corner of the mounting structure 3502 (e.g., the corner where the third gripper member 3530 is located) may align with a corner of the object.In this example, a corner of the object may be gripped by the third gripper member 3530, while the arrangement of the first gripper member 3510 and the second gripper member 3520 may ensure that the gripping point at which the gripper members 3510, 3520 grip the object is sufficiently far away from that corner of the object (gripped by the third gripper member 3530) so that the overall grip of the object by the gripper members 3510, 3520 and / or 3530 is balanced and stable.

[0046] In some scenarios, the first gripper member 3510 and the second gripper member 3520 may be configured to move along the first rail 3540 and the second rail 3542, respectively, by one or more actuators, such as pneumatic actuators, electromagnetic actuators, electromechanical actuators, any other actuators, or combinations thereof. The one or more actuators may be part of the end effector device 3500A, or more broadly, the robot 1300 / 3300 or system 1000 of FIG. 1A. The one or more actuators may be configured to generate a first force that actuates the first gripper member 3510 in a first direction along the Y′ axis or in an opposite second direction along the Y′ axis to move the first gripper member 3510 in a first direction or a second direction along the Y′ axis. A first direction along the Y′ axis may move the first gripper member 3510 towards the third gripper member 3530, while a second direction along the Y′ axis may move the first gripper member 3510 away from the third gripper member 3530. One or more actuators may be configured to generate a second force that actuates the second gripper member 3520 in a first direction along the X′ axis or in an opposite second direction along the X′ axis to move the second gripper member 3520 in a first direction or move the second gripper member 3520 in a second direction along the X′ axis. A first direction along the X′ axis may move the second gripper member 3520 towards the third gripper member 3530, while a second direction along the X′ axis may move the second gripper member 3520 away from the third gripper member 3530.

[0047] In an embodiment, the one or more actuators may include any type of actuator, such as a pneumatic actuator, an electromagnetic actuator, or an electromechanical actuator, as described above. The one or more actuators may be part of the end effector device 3500 or may be considered separate from the end effector device 3500. For example, the one or more actuators may include a plurality of electromagnetic actuators (e.g., motors or solenoids) mounted on the mounting structure 3502 and part of the end effector device 3500. In another example, the one or more actuators may include a pneumatic actuator (e.g., a pump) configured to generate hydraulic or hydraulic pressure within a hydraulic or hydraulic line, and the end effector device 3500 may include a port that is coupled to or otherwise configured to receive the hydraulic or hydraulic line. The port may direct hydraulic or hydraulic pressure generated by the pneumatic actuator to the first gripper member 3510 and / or the second gripper member 3520. Hydraulic or water pressure may be used to push the gripper body of the first gripper member 3510, causing it to move along the first rail 3540, and / or to push the gripper body of the second gripper member 3530, causing it to move along the second rail 3542.

[0048] In embodiments, one or more actuators may be configured to cause other motions in the end effector device 3500A. For example, one or more actuators may be configured to cause relative motion within each of the gripper finger assemblies described above, or more specifically, between a first gripper finger and a second gripper finger of the gripper finger assembly.

[0049] In some scenarios, one or more actuators may be configured to cause the gripper finger assembly and / or the gripper body of the first gripper member 3510 (e.g., a portion of the gripper body including the gripper fingers of the first gripper member 3510) to extend along an axis perpendicular to the first rail 3540. The motion may be inward or outward relative to the mounting plate 3502 and may be parallel to a top or bottom surface of the mounting plate 3502. Similarly, one or more actuators may be configured to cause the gripper finger assembly and / or the gripper body of the second gripper member 3520 (e.g., a portion of the gripper body including the gripper fingers of the second gripper member 3520) to extend along an axis perpendicular to the second rail 3542. The motion may also be inward or outward relative to the mounting plate 3502 and may be parallel to a top or bottom surface of the mounting plate 3502. For example, if the end effector device 3500A is used to grip a container, and a lip of the container forms or surrounds the edge of the container, the above described actions may occur after the first gripper member 3510 is positioned at a particular location along the first rail 3540, and the gripper finger assembly of the first gripper member 3510 may be moved closer toward a first portion of the container's lip such that a first portion of the container's lip is between a pair of gripper fingers of the gripper finger assembly. Such actions may allow the gripper fingers to clamp around the first portion of the container's lip. The above described actions may further allow the gripper finger assembly of the second gripper member 3520 to be moved closer toward a second portion of the container's lip such that its gripper fingers can clamp around a second portion of the container's lip. Additionally, one or more actuators may be configured to move gripper finger assembly 3531A of the third gripper member 3530 toward a corner of the container rim, as shown in Figure 4D. The movement may be along an axis oblique to the X' and Y' axes of Figure 4B (e.g., 45 degrees to the X' axis).

[0050] In embodiments, the end effector device 3500A may be configured to engage and move objects that are of different respective sizes. To accomplish this, the movement of the first gripper member 3510 along the first rail 3540 and the movement of the second gripper member 3520 along the second rail 3542 may be controlled by the computing system 1100 and / or the robot 3300. For example, the first gripper member 3510 may be moved to an end position E1, as shown in FIG. 4B. y’ and E2 y’ and the second gripper member 3520 may be movable between an end position E1 x’ and E2 x’ The first gripper member 3510 may be further movable between two end positions E1 y’ and E2 y’ Intermediate positions between (e.g., E3 y Similarly, the second gripper member 3510 may be further movable between two end positions E1 x’ and E2 x’ Intermediate positions between (e.g., E3 x Accordingly, the first gripper member 3510 and the second gripper member 3520 may be movable to different positional configurations, thereby enabling the end effector device 3500 to engage objects having different sizes.

[0051] In embodiments, the computing system 1100 and / or robot 1300 / 3300 may be configured to control the movement of the first gripper member 3510 along the first rail 3540 and the movement of the second gripper member 3520 along the second rail 3542 by controlling one or more actuators and / or stopping mechanisms (e.g., brake mechanisms). For example, the computing system 1100 and / or robot 1300 / 3300 may be configured to control whether one or more actuators are activated, which of the one or more actuators are activated, the level (e.g., power level) at which the one or more actuators are activated, and / or the duration for which the one or more actuators are activated. For example, the computing system 1100 and / or robot 1300 / 3300 may be configured to determine a position (e.g., E3) at which the first gripper member 3510 or the second gripper member 3520 should be positioned. x ' or E3 y If the end effector device 3500A determines the position of the first gripper member 3510 or the second gripper member 3520, the computing system 1100 and / or the robot 1300 / 3300 may activate the actuator to move the first gripper member 3510 or the second gripper member 3520 to the determined position and stop the actuator when the first gripper member 3510 or the second gripper member 3520 stops at the determined position. In some scenarios, if the end effector device 3500A includes a stopping mechanism, the computing system 1100 and / or the robot 1300 / 3300 may be configured to activate the stopping mechanism as the first gripper member 3510 or the second gripper member 3520 approaches the determined position to stop the first gripper member 3510 or the second gripper member 3520 at the determined position.

[0052] In an embodiment, the end effector device 3500A may include one or more sensors to measure the movement of the gripper members 3510, 3520 and / or detect the presence (e.g., proximity) of a container or other object to be engaged (e.g., grasped) by the end effector device 3500A. For example, the one or more sensors may include a first gripper body sensor (e.g., optical sensor, mechanical sensor, electromechanical sensor) configured to measure or otherwise determine the position of the first gripper member 3510 along the first rail 3540 and a second gripper body sensor configured to measure or otherwise determine the position of the second gripper member 3520 along the second rail 3542.

[0053] In some scenarios, the one or more sensors may include a first gripper member proximity sensor 3570, a second gripper member proximity sensor 3572, and a third gripper member proximity sensor 3574, as shown in FIG. 4B. The first gripper member proximity sensor 3570 may be disposed on and / or part of the first gripper member 3510, while the second gripper member proximity sensor 3572 may be disposed on and / or part of the second gripper member 3520, and the third gripper member proximity sensor 3574 may be disposed on and / or part of the third gripper member 3530. The gripper member proximity sensors 3570, 3572, 3574 operate to detect the proximity of a container or other object to be grasped or otherwise engaged by the end effector device 3500A. For example, to engage and pick up an object, such as object 3711 in FIG. 3A , the computing system 1100 and / or robot 3300 may move the end effector device 3500 by way of the robot arm 3400 towards the object 3711. The gripper body proximity sensors 3570, 3572, 3574 operate to detect when the gripper members 3510, 3520, and 3530 are within a defined (e.g., predefined) threshold distance from the object 3711 and / or when the gripper members 3510, 3520, 3530 are aligned with the object 3711. In some instances, when the end effector device 3500A approaches the object 3711 by lowering towards the object 3711, the gripper member proximity sensors 3570, 3572, 3574 may detect when the gripper members 3510, 3520, 3530 of the end effector device 3500A are low enough to be flush with a portion of the object 3711 to be grasped (e.g., the lip of a container). The gripper member proximity sensors 3570, 3572, and 3574 may each include a mechanical sensor, an electromechanical sensor, an optical sensor, or any other type of sensor configured to detect proximity between the sensor and the object.

[0054] In some scenarios, the one or more sensors may include a first gripper finger sensor, a second gripper finger sensor, and a third gripper finger sensor. In these scenarios, the first gripper member 3510, the second gripper member 3520, and the third gripper member 3530 may each include a respective gripper finger assembly having at least a pair of gripper fingers. The first gripper finger sensor, the second gripper finger sensor, and the third gripper finger sensor may each be configured to measure or otherwise determine the relative position of the respective pair of gripper fingers for the respective gripper finger assembly and / or detect whether an object or a portion thereof is between the respective pair of gripper fingers. Each gripper finger sensor may be used to control the relative motion between the respective pair of gripper fingers. For example, if the gripper finger sensors indicate that the edge of a container is positioned between each pair of gripper fingers monitored by a particular gripper finger sensor, the computing system 1100 and / or robot 1300 / 3300 may control one or more actuators discussed above to move the pair of gripper fingers toward each other to clamp around a portion of the object.

[0055] As mentioned above, one aspect of the present application relates to performing motion planning that may be used to facilitate robotic interactions, such as interactions in which a robot moves an object from a current location to a destination location. FIG. 5 depicts a flow diagram of an example method 5000 for performing motion planning. Method 5000 may be performed, for example, by computing system 1100 of FIGS. 2A-2D or 3A, or more specifically, by at least one processing circuit 1110 of computing system 1100. In some scenarios, at least one processing circuit 1100 may perform method 5000 by executing instructions stored on a non-transitory computer readable medium, such as non-transitory computer readable medium 1120. For example, the instructions may cause processing circuit 1110 to execute motion planning module 1122, which may perform method 5000. In an embodiment, method 5000 may be performed in an environment where computing system 1100 is in communication with a robot and camera, such as robot 3300 and camera 3200 of Figures 3A and 3B, or with any other robot discussed in this disclosure. In some scenarios, the camera (e.g., 3200) may be mounted on the end effector device (e.g., 3500) of the robot (e.g., 3300). In other scenarios, the camera may be mounted elsewhere and / or may be fixed.

[0056] In an embodiment, the method 5000 may begin or otherwise include step 5002 in which the computing system 1100 receives first image information representing a structure of an object (also referred to as an object structure) that is or was within a field of view (also referred to as a camera field of view) of a camera. For example, FIG. 6A depicts a scenario in which a stack 3720 of objects 3721-3726 is disposed within the camera field of view 3202 of the camera 3200. Each of the objects 3721-3726 may be, for example, a box, crate, or other container. In the example of FIG. 6A, the objects 3721-3726 may be disposed on a pallet 3728. In an embodiment, the pallet 3728 may be used with a wide variety of stacking configurations to stack containers or other objects that may have a wide variety of sizes (e.g., a wide variety of length, width, and height values).

[0057] In an embodiment, the first image information received by the computing system 1100 may be generated by the camera (e.g., 3200) when the camera has a first camera pose, such as the camera pose shown in FIG. 3A or the camera pose shown in FIG. 6A. The camera pose may refer to the position and orientation of the camera (e.g., 3200). In some scenarios, the camera pose may affect the view or viewpoint of the camera (e.g., 3200). For example, the first camera pose depicted in 6A may involve the camera 3200 having a position that is directly above the stack 3720 and an orientation in which the camera 3200 faces the top of the stack 3720, or more specifically, the objects 3721, 3722 that form the top of the stack 3720. In some instances, the steps of the method 5000 may be performed to facilitate robot interaction with individual objects of the stack 3720, such as the object 3722. In such instances, a particular object that is a target of the robot interaction may be referred to as a target object. In some scenarios, the steps of method 5000 may be performed multiple times or in multiple iterations to facilitate robot interaction with multiple target objects.

[0058] In some scenarios, the first image information may represent a particular view of the stack 3720, or more specifically, a particular view of one or more objects forming the stack 3720. In the example of FIG. 6A, the first image information may represent a top view of the stack 3720, or more specifically, of the objects 3721, 3722, since the first image information may be generated when the camera 3200 has a first camera pose as shown in FIG. 6A, where the camera 3200 is above the objects 3721, 3722 and points toward the top ends of the objects 3721, 3722. In the example of FIG. 6A, the objects 3721, 3722 may each be a wooden box or other open-top container having one or more walls surrounding a bottom inner surface of the container. The one or more walls may form a rim at the top end of the container. In such an example, the plan view of the object 3721 / 3722 may include a view of the edge surface (also referred to as the edge surface) of the object 3721 / 3722 and may include a view of the bottom interior surface of the object 3721 / 3722. In another example, the bottom interior surface of the object 3721 / 3722 is not represented by the first image information or is only partially represented by the first image information. Such an example may occur, for example, when the object 3721 / 3722 is a container that is partially or completely filled with items that are positioned or stacked over the bottom interior surface of the object 3721 / 3722, such as the situation shown for a portion of the container in FIG. 3B. In such a situation, the first image information may describe or otherwise represent items that are positioned within the container, but those items may partially or completely block or obscure the view of the bottom interior surface of the container. In another example, a fully enclosed box, such as a container with a lid, or other container may be within the camera field of view (e.g., 3202) of camera 3200. In this example, a top view of the fully enclosed container may include a view of the exterior surface of the container (e.g., the top exterior surface, also referred to as the top surface).

[0059] In an embodiment, the first image information may describe the appearance of the stack 3720, or more specifically, one or more objects (e.g., 3721 and 3722) that form the stack 3720. For example, FIG. 6B provides an example in which the first image information includes or forms a 2D image 6082 (e.g., a grayscale or color image) that includes an image portion 6021 (e.g., a region of pixels) that describes the appearance of the object 3721, an image portion 6022 that describes the appearance of the object 3722, and an image portion 6028 that describes the appearance of the palette 3728. More specifically, the image 6082 may describe the appearance of the objects 3721, 3722, and the palette 3728 from the perspective of the camera 3200 of FIG. 6A, and more specifically may represent a top view of the objects 3721, 3722. As described above, the image 6082 may be generated when the camera 3200 has the first camera pose depicted in FIG. 6A. More specifically, the 2D image 6082 may represent one or more surfaces of the object 3721 and one or more surfaces of the object 3722. For example, the image portion 6021 of the 2D image 6082 may include, more specifically, an image portion 6021A representing a first surface (e.g., an edge surface) of the object 3721 and an image portion 6021B representing a second surface (e.g., a bottom inner surface) of the object 3721. Similarly, the image portion 6022 may include an image portion 6022A representing a first surface (e.g., an edge surface) of the object 3722 and an image portion 6022B representing a second surface (e.g., a bottom inner surface) of the object 6022. In another example, if the first object 3722 is a container filled with items, as discussed above, the image portion 6022, or more specifically, the image portion 6022B, may describe the appearance of the items disposed within the container.

[0060] In an embodiment, the first image information may describe a structure of the stack (also referred to as the stack structure), or at least a portion of the stack structure, which may be defined by the structure of the objects 3721-3726 forming the stack. More specifically, the first image information may describe a structure of the objects forming the stack (also referred to as the object structure), or at least a portion of the object structure. In such an embodiment, the camera (e.g., 3200) generating the first image information may be a 3D camera (also referred to as a spatial structure sensing device). As described above, the first image information received in step 5002 may represent a particular viewpoint of the camera when the first image information is generated, such as a top view of the stack structure. In some scenarios, the first image information may include spatial structure information, which may be referred to as three-dimensional (3D) information, which describes how the objects are arranged in 3D space. For example, the spatial structure information may include depth information that describes the depth of a portion or more of the object, or the depth of the object structure, relative to a reference point, such as a point where the camera (e.g., 3200) is located when the camera generates the first image information.

[0061] In some scenarios, the spatial structure information may describe depth values ​​for each of a plurality of positions (also referred to as points) on one or more surfaces of the object structure. For example, Fig. 6C depicts a first image information having 3D image information 6084 describing respective depth values ​​for various positions on the surfaces of objects (also referred to as object surfaces) in a camera field of view (e.g., 3202) of a camera (e.g., 3200), such as object surfaces of object 3721, object 3722, and pallet 3728 of Fig. 6A. In the example of Fig. 6C, the various positions identified or otherwise described by the spatial structure information are positions 37281 to 3728 on the top surface of pallet 3728. n (depicted as open circles), positions 3721A1 to 3721A on a first surface (e.g., an edge surface) of object 3721. n , positions 3721B1 to 3721B2 on a second surface (e.g., a bottom inner surface) of the first object 3721. n, positions 3722A1 to 3722A on a first surface (e.g., an edge surface) of the object 3722 n , and positions 3722B1 to 3722B2 on a second surface (e.g., bottom inner surface) of the object 3722. n In another example, if the object 3721 / 3722 is a container filled with items, as discussed above, positions 3721B1 to 3721B n and / or positions 3722B1 to 3722B n may be positions on one or more surfaces within the container. In some scenarios, the first image information may describe the respective depth values ​​along with a depth map, which may include, for example, an array of pixels corresponding to a grid of positions on one or more object surfaces in the camera field of view (e.g., 3202). In such a scenario, some or all of the pixels may each include a respective depth value for a respective position corresponding to the pixel, the respective positions being on one or more object surfaces. In some circumstances, the first image information may describe respective depth values ​​through a plurality of 3D coordinates, which may describe various positions on one or more object surfaces. For example, the 3D coordinates may be from positions 37281 to 37282 in FIG. 6C . n , positions 3721A1 to 3721A n , positions 3721B1 to 3721B n , positions 3722A1 to 3722A n , and positions 3722B1 to 3722B n A plurality of 3D coordinates may form a point cloud or a portion of a point cloud describing at least a portion of an object structure, such as, for example, a top of the object structure of object 3721 and / or a top of the object structure of object 3722. The 3D coordinates may be expressed in the camera coordinate system or in some other coordinate system. In some instances, a depth value for a particular location may be represented by or based on components of the 3D coordinate for that location. As an example, if the 3D coordinate for a location is an [XYZ] coordinate, then the depth value for that location may be equal to or based on the Z component of the 3D coordinate.

[0062] Returning to FIG. 5, the method 5000 may include step 5004 in which, in an embodiment, the computing system 1100 may determine a first estimate of an object structure of an object in the camera field of view (e.g., 3202), such as the object structure of object 3721 or the object structure of object 3722 in FIG. 6A, based on the first image information. In some instances, the first estimate of the object structure may include an estimate of one or more object dimensions, such as an estimate for the length of the corresponding object (e.g., 3721 or 3722) and / or an estimate for the width of the corresponding object. In some instances, the first estimate of the object structure may include an estimate of the shape of the corresponding object (also referred to as object shape). For example, the first estimate of the object structure may indicate that the object structure has a rectangular shape. In some scenarios, the first estimate of the object structure may include a point cloud describing a first outer surface (e.g., a top outer surface) forming the object structure, or more specifically, describing positions on the first outer surface. In some implementations, the point cloud may be later updated to incorporate a different set of image information representing different perspectives of the object structure, such as first image information and second image information, discussed in more detail below with respect to steps 5012 and 5014. In such implementations, the point cloud may be referred to as a global point cloud. In some instances, the point cloud may specifically represent the object structure for a target object, such as object 3722. In some instances, if the target object is part of a stack, such as stack 3720, the point cloud may represent the stack structure for the stack. In such instances, a portion of the point cloud may specifically represent the object structure for the target object (e.g., 3722).

[0063] In some scenarios, a first estimate of the object structure (e.g., an estimate of the object size or shape) may be determined directly based on the first image information. For example, the first image information may include images of positions 3722A1 to 3722A2 on an edge surface of the object 3722 in FIG. nIf the first image information includes 3D coordinates of the first object structure, the computing system 1100 may determine a first estimate of the object structure by using the 3D coordinates. More specifically, if the first estimate of the object structure is or includes a point cloud, the computing system 1100 may determine the first estimate of the object structure by including the 3D coordinates in the point cloud. For example, the computing system 1100 may insert or add the 3D coordinates to the point cloud, or more specifically, to a file or other data structure that represents the point cloud. The 3D coordinates from the first image information may form a partial point cloud, for example, representing a portion of the object structure (e.g., an edge that forms a top of the object structure) or a particular viewpoint (e.g., a top view) of the object structure. In this example, the computing system 1100 may incorporate information from the partial point cloud into the global point cloud in step 5014. In step 5004, the global point cloud may include only or mainly information from the partial point cloud discussed above that represents a viewpoint associated with the first camera pose. As discussed below, the wide-range point cloud may ultimately incorporate additional image information representing one or more additional viewpoints (e.g., perspective viewpoints), which may enable the wide-range point cloud to be a more complete object structure representation of the object (e.g., 3722) relative to the partial point cloud associated with the first camera pose. In some implementations, the computing system 1100 may determine whether the 3D coordinates in the partial point cloud use a coordinate system that is different from the coordinate system used by the 3D coordinates of the wide-range point cloud. If so, the computing system 1100 may transform the 3D coordinates in the partial point cloud to be expressed relative to the coordinate system of the wide-range point cloud and add the transformed coordinates to the wide-range point cloud.

[0064] In some instances, when the first image information includes the 3D coordinates discussed above, and when the first estimate for the object structure includes an estimate for the object length and an estimate for the object width of the object structure, the computing system 1100 may be configured to directly determine the estimate based on a difference between some of the 3D coordinates. For example, when the computing system 1100 determines the 3D coordinate [X 3721A1 Y3721A1 Z 3721A1 ] and position 3721A n 3D coordinates [X 3721An Y 3721An Z 3721An ]. More specifically, the computing system 1100 may determine the estimate based on the difference between Y 3721An ~Y 3721A1 An estimate for the object length of object 3721 may be determined to be equal to or based on the absolute value of X 3721An ~X 3721A1 (X-axis may correspond to the width dimension). Similarly, the calculation system 1100 may determine an estimate for the object width to be equal to or based on Y 3722An ~Y 3722A1 An estimate for the object length of object 3722 may be determined to be equal to or based on X 3722An ~X 3722A1 An estimate for the object width of object 3722 may be determined to be equal to or based on:

[0065] In an embodiment, the first image information may be generated by a camera (e.g., 3200) while the camera has a first camera pose in which the camera is pointing directly at a first exterior surface of an object structure, such as the object structure of object 3721 / 3722. The first exterior surface (e.g., the top exterior surface) may therefore be contained within the camera field of view (e.g., 3202 in FIG. 6A) of the camera (e.g., 3200), and the camera may point to the camera with line of sight to the first exterior surface or at least a portion thereof. The first image information may thus describe the first exterior surface (e.g., the top exterior surface) of the object structure. In some scenarios, the camera may lack line of sight to some or all other exterior surfaces of the object structure, such as the entire outer side surface and bottom exterior surface of the object structure of object 3721 / 3722, when in the first camera pose. In such scenarios, the first image information may be devoid of description of such exterior surfaces (e.g., the outer side surfaces and bottom exterior surfaces, also referred to as side and bottom surfaces). For example, the first image information shown in Figures 6B and 6C may describe the top outer surface of the object structure for object 3721 / 3722, but may have little or no description of the outer side surface of the object structure for object 3721 / 3722. As another example, if the first estimate for the object structure includes estimates for one or more object dimensions, such as object length and object width, the first estimate may omit estimates for one or more object dimensions, such as object height. In this example, the object height may be omitted from the first estimate of the object structure because the first image information on which the first estimate is based may represent a plan view of the object structure (e.g., of objects 3721, 3722). In such an example, the plan view may lack information that would allow the computing system 1100 to directly determine the object height. If the first estimate of the object structure in step 5004 includes a point cloud, or more specifically, the wide range point cloud discussed above, the wide range point cloud of step 5004 may include 3D coordinates representing the top of the object structure of the object (e.g., 3721 / 3722) but may lack 3D coordinates representing bottom and / or side portions of the object structure because those portions may not be within the camera's line of sight when the camera (e.g., 3200) generated the first image information.

[0066] In some scenarios, the computing system 1100 may determine a first estimate of the object structure based on a defined maximum value for a characteristic of the object structure, such as the object height or other object dimension. In this example, the computing system 1100 may use the defined maximum value to make an initial estimate for an object dimension or other characteristic that the first image information may not fully describe or represent (if described at all). For example, if the first image information is based on a plan view of the object structure and does not describe the object height of the object structure, the computing system 1100 may determine an initial estimate for the object height to be equal to or based on the defined maximum object height. The computing system 1100 may use the initial estimate for the object height or other characteristic as a first estimate or part of the first estimate of the object structure. The defined maximum object height, or some other defined maximum value, may, for example, be manually provided to the computing system 1100 indicating the maximum object dimension that the computing system 1100 or a robot (e.g., 3300) is likely to encounter, and / or may be determined by an object registration process in which the computing system 1100 determines and stores information describing the object structure of previously encountered objects.

[0067] In some scenarios, determining a first estimate for the object structure of the object may involve determining an object type that corresponds to the object (e.g., 3722) represented by the first image information. The object type may refer to a particular object design, such as a visual design and / or physical design, for the object (e.g., 3722) or class of objects. For example, if the object discussed above is a container, the object type may refer to a container type, or to a particular container design, which may include a particular visual design and / or physical design, for the container or class of containers. The determined object type may be associated with a particular object structure and thus may be used to determine a first estimate for the object structure. More specifically, the computing system 1100 may, in some implementations, store or otherwise have access to templates (e.g., 1126) that describe various respective object types. As discussed above, the templates may include visual description information and / or object structure descriptions that describe the object type, or more specifically, that describe the object design associated with that object type. The visual description information in the template may describe a visual design that defines an appearance associated with an object type, and the object structure description in the template may describe a physical design that defines a structure associated with an object type. In some scenarios, the object structure description may describe a 3D structure for a physical design associated with an object type. For example, the object structure description may describe a combination of values ​​for each of object length, object width, and object height for the physical design, and / or may include a CAD model that describes the contour, shape, and / or any other aspect of the physical design.

[0068] In some instances, the computing system 1100 may determine an object type corresponding to the object by comparing the first image information to the various templates discussed above, and determine which of the various templates the first image information matches. If the first image information includes or forms a 2D image that represents the appearance of the object (e.g., 3722), the computing system 1100 may compare the 2D image or a portion thereof (e.g., image portion 6021 / 6022 in FIG. 6B) with the visual description information of the template. In some instances, if the first image information includes 3D image information that describes a portion of the object structure (e.g., describing the object length and object width), the computing system 1100 may compare the 3D image information or other description with the description of the object structure (also referred to as structure description information) of the template.

[0069] In some circumstances, if step 5004 involves determining an object type for the object represented by the first image information or a portion thereof, the object type determined in this step may be an initial guess for the object type. More specifically, if the first image information lacks a description of a portion of the object structure, such as its outer side surface, matching a template using the first image information may lead to results with only a medium or low level of certainty. In some scenarios, the first image information may match multiple templates, especially if the templates have visual description information or object structure descriptions that share similarities for a portion (e.g., a top) of their respective physical designs. As discussed in more detail below with respect to steps 5012 and 5014, the computing system 1100 may use the second image information to perform another template matching operation, which may be more successful and / or lead to results with a higher level of certainty.

[0070] In an embodiment, the computing system 1100 may be configured to determine a motion plan based on the first estimate of the object structure. In some scenarios, the motion plan may be an initial motion plan determined immediately or shortly after step 5004. In such scenarios, the computing system 1100 may further generate a motion plan of step 5016, which is an updated motion plan, as discussed in more detail below. In some scenarios, the method 5000 may omit determining an initial motion plan based on the first estimate of the object structure. However, such an initial motion plan, if generated, may include planned actions, or more specifically, a set of one or more actions, for the robot (e.g., 3300) or a portion thereof (e.g., the robot arm 3400 and / or the end effector device 3500). The planned actions may be used to cause an interaction between the robot (e.g., 3300) and an object (e.g., 3722) corresponding to the object structure determined in step 5004. In such an example, the motion command may be referred to as an object interaction motion command. The interactions may include, for example, an end effector device (e.g., 3500) of the robot (e.g., 3300) picking up an object and moving the object to a target location. In some instances, the planned motion may describe a desired motion for the end effector device (e.g., 3500). For example, the planned motion may describe a trajectory for the end effector device (e.g., 3500) to follow. In some implementations, the planned motion may more specifically describe the motion of various components of the robot arm (e.g., 3400), such as the motion of various joints connecting the articulations of the robot arm, or various motors or other actuators configured to actuate the articulations.

[0071] In some instances, if the motion plan includes a trajectory for the end effector device (e.g., 3500) or other component to follow, the computing system 1100 may determine an end point for the trajectory. The end point may specify, for example, a position (or more specifically, a pose) where the robot (e.g., 3500) or a component thereof (e.g., end effector device 3500) stops moving and ends interaction with a particular object (e.g., 3722). Ending the interaction may involve, for example, the release of the object from the grip of the end effector device (e.g., 3500). In some implementations, the computing system 1100 may determine the end point of the trajectory based on an object height for the object, which may have been determined from a first estimate of the object structure.

[0072] More specifically, the computing system 1100 may determine a final end effector height based on an estimate of the object height and determine the end point of the trajectory based on the final end effector height (also referred to as a determined final end effector height or a planned final end effector height). The determined final end effector height may refer to the height of the end effector device (e.g., 3500) when it releases or otherwise stops interacting with the object (e.g., 3722). In some scenarios, the determined final end effector height may be expressed relative to a destination location. If the destination location is part of a destination structure that receives the object, the destination location may refer to a location or area of ​​the destination structure where the earliest or initial contact between the object and the destination structure will occur. For example, if the destination structure is a roller conveyor having a set of rollers, the destination location may be the highest position of one or more of the rollers, since at the destination location the end effector device (e.g., 3500) will first contact the object during its trajectory to lower the object toward the roller conveyor. If the destination structure is, for example, a conveyor belt or a floor having a top surface, the destination location may be a location on the top surface or floor. The final end effector height may represent, for example, a height that the end effector device (e.g., 3500) is planned to have or is likely to have when a bottom portion (e.g., bottom outer surface) of the object contacts the destination location. More specifically, the final end effector height may represent a height that the end effector device (e.g., 3500) should have when the operation of the end effector device (e.g., 3500) is terminated. Thus, the computing system 1100 may determine the end point of the trajectory based on the final end effector height. In some scenarios, the computing system 1100 may determine the final end effector height to be equal to or based on an estimate of the object height, which may be from a first estimate for the object structure of a particular object (e.g., 3722). However, as discussed above, the estimate of the object height from a first estimate of the object structure may lack accuracy.As a result, the first estimate of the object structure may affect the reliability of the final end effector height and trajectory determined by the computation system 1100. As discussed in more detail below, the computation system 1100 may determine a second estimate for the object structure in step 5014. The second estimate may provide greater accuracy and may be used to determine a more reliable motion plan in step 5016.

[0073] Returning to FIG. 5, the method 5000 may, in an embodiment, include step 5006, in which the computing system 1100 may identify corners of the object structure, i.e., more specifically, external or convex corners of the object structure. In some scenarios, the corners of the object structure (also referred to as object corners) may be determined based on a first estimate of the object structure or based on first image information. In some scenarios, determining the object corners may involve determining the exact or approximate location of the object corners. For example, the computing system 1100 may determine the 3D coordinates [X 3722A1 Y 3722A1 Z 3722A1 6C with ] may be identified as an object corner of object 3722. In some instances, identifying the corners may involve identifying vertices (also called contour points) from the point cloud and identifying convex corners based on the vertices. Identifying convex corners is discussed in detail in U.S. Patent Application No. 16 / 578,900 (MJ0037-US / 0077-0006US1), the entire contents of which are incorporated herein by reference.

[0074] In an embodiment, if the first estimate of the object structure describes multiple object corners, the computing system 1100 may select from among the multiple object corners in step 5006. For example, a first estimate of the object structure of the object 3722 in FIG. 6A, which may be based on the first image information depicted in FIGS. 6B and 6C, may include, for example, positions 3722A1, 3722A4, 3722A5, and 3722A6. nIn some implementations, the computing system 1100 may make the selection based on at least one of (i) the respective amount of occlusion experienced by the plurality of object corners, or (ii) the respective reachability of the plurality of object corners by an end effector device (e.g., 3500). For example, the computing system 1100 may be configured to select as the corner identified in step 5006 the object corner of the object structure that experiences the least amount of occlusion among the plurality of object corners and / or has the highest level of reachability by the end effector device.

[0075] Referring again to FIG. 5, the method 5000 may, in an embodiment, include step 5008, in which the computing system 1100 may determine a second camera pose, such as the camera pose of FIG. 7A. The second camera pose may include a particular position and orientation combination of the camera (e.g., 3200) that, when adopted by the camera (e.g., 3200), directs the camera (e.g., 3200) to a corner of the object structure determined in step 5006. For example, the second camera pose depicted in FIG. 7A may direct the camera 3200 to a corner of the object 3722 of FIG. 6A, which object corner may correspond to position 3722A1 of FIG. 6C. The second camera pose may also direct the camera (e.g., 3200) to an exterior surface, such as an exterior side surface, that was not previously in the camera's line of sight when the camera (e.g., 3200) was in the first camera pose. In other words, when the camera (e.g., 3200) has a first camera pose, it may have a camera field of view (e.g., 3202) that encompasses a first exterior surface (e.g., a top exterior surface) of the object structure without encompassing a second exterior surface (e.g., a side exterior surface). When the camera moves to the second camera pose, the camera field of view may alternatively or additionally encompass the second exterior surface. In an embodiment, the second camera pose may provide the camera 3200 with a different perspective or viewpoint relative to the perspective or viewpoint of the first camera pose. More specifically, the second camera pose may provide the camera 3200 with a perspective view of the object structure for the object 3722, as discussed in more detail below.

[0076] Returning to FIG. 5, the method 5000 may, in an embodiment, include a step 5010 of moving, by the computing system 1100, the camera (e.g., 3200) to a second camera pose. For example, the computing system 1100 may be configured to generate one or more motion commands, such as one or more motor commands, for a robot arm (e.g., 3400) and / or an end effector device (e.g., 3500) to which the camera (e.g., 3200) is attached, to move the camera to the second camera pose. The computing system 1100 may output one or more motion commands, which may be referred to as camera placement motion commands, to the robot (e.g., 3300) via a communication interface (e.g., 1130 in FIG. 2B) of the computing system 1100. Upon receiving the one or more camera placement motion commands, the robot (e.g., 3300) may be configured to execute them to move the camera (e.g., 3200) to the second camera pose.

[0077] Returning to FIG. 5, the method 5000 may, in an embodiment, include step 5012, in which the computing system 1100 may receive second image information. In this example, at least a portion of the second image information represents an object structure, such as object 3722 in FIG. 7A, corresponding to the object and object corners of steps 5006 and 5008. The second image information may be generated by a camera (e.g., 3200) while the camera (e.g., 3200) has a second camera pose as shown in FIG. 7A. The second image information may include a 2D image describing an appearance of the object (e.g., 3722) and / or 3D image information describing an object structure for the object. In some instances, the second image information may describe a stack structure formed by the object structure. In such an example, the object structure may be represented by a portion of the second image information. For example, FIG. 7B illustrates an embodiment in which the second image information includes a 2D image 7082 describing the appearance of the object 3722, or more broadly, the appearance of the stack 3720 including the object 3722. In addition, FIG. 7C depicts an embodiment in which the second image information includes 3D image information 7084 describing the object structure for the object 3722, and more broadly, the stack structure for the stack 3720.

[0078] More specifically, the 2D image 7082 of Fig. 7B may represent a perspective view of various objects 3721-3726 of the stack 3720 of Figs. 7A and 6A. As depicted in Fig. 7B, the image 7082 may include an image portion 7028 (e.g., a region of pixels) representing the appearance of the pallet 3728 of Fig. 7 on which the stack 3720 is arranged, and may include image portions 7021-7026 representing the appearance of each of the objects 3721-3726, respectively, from a perspective view. The first image information shown in Fig. 6B may include an image portion (e.g., 6022) representing a first outer surface of the object, while the second image information shown in Fig. 7B may include an image portion (e.g., 7022) representing one or more further outer surfaces that are not visible, i.e. not represented more generally, in the first image information. For example, the first image information may represent an edge surface, or more broadly, a top surface (also referred to as a top face), of the structure of the object 3722, while the second image information may represent a first outer side surface and a second outer side surface (also referred to as a first side surface and a second side surface) of the structure of the object 3722 that are not represented by the first image information. In certain circumstances, one or more objects in the stack 3720, such as the object 3722, may have a 2D or 3D pattern on the outer side surface. The 2D pattern may include, for example, a visual pattern or other visual details (e.g., a logo or picture) that appears on the outer side surface, while the 3D pattern may include, for example, a pattern of ridges or protrusions (collectively also referred to as a ridge pattern) that protrudes from the outer side surface, such as the ridge patterns depicted in Figures 3A and 3B. The 2D or 3D pattern may be partially or completely hidden in the first image information or may otherwise be partially or completely omitted from the first image information, but may be more fully represented in the second image information. In the example of Figure 7B, the second image information may also represent the top outer surface. In some circumstances, one or more additional surfaces represented in the second image information may be perpendicular to the first outer surface, angled relative to the first outer surface, or more broadly, non-parallel to the first outer surface.

[0079] In an embodiment, where the second image information includes 3D image information, the 3D information may include a plurality of 3D coordinates describing various positions on one or more object surfaces within a camera field of view (e.g., 3202 in FIG. 7A). For example, FIG. 7C illustrates a plurality of 3D coordinates describing various positions on a surface (e.g., a top surface) of a pallet 3728 from positions 37281 to 37282. n 7D depicts 3D image information 7084 including 3D coordinates of locations 3722A1 to 3722A2 on an edge surface of object 3722, as well as 3D coordinates of various locations on one or more object surfaces of objects 3721 to 3726. The one or more surfaces may include, for example, a top exterior surface (e.g., an edge surface), one or more interior side surfaces, a bottom interior surface, and / or one or more exterior side surfaces. By way of example, FIG. 7D depicts a portion of 3D information 7084, the portion of which includes locations 3722A1 to 3722A2 on an edge surface of object 3722. n , positions 3722B1 to 3722B2 on the bottom inner surface of the object 3722, positions 3722C1 to 3722C2 on the first outer side surface (also referred to as the outer side surface of the object) of the object 3722. n , and positions 3722D1 to 3722D2 on a second outer side surface of the object 3722. n The second image information may include 3D coordinates of positions on various surfaces of object 3722, including, for example, [XYZ] coordinates in a camera's (e.g., 3200) coordinate system when the camera was in a second camera pose, or some other coordinate system. In some scenarios, the second image information may include a depth map describing depth values ​​for each of the positions discussed above, and computing system 1100 may be configured to determine the 3D coordinates of those positions based on the respective depth values.

[0080] Returning to FIG. 5, the method 5000 may in an embodiment include a step 5014 in which the computing system 1100 determines a second estimate of the object structure (e.g., for the object 3722) based on the second image information. The second estimate of the object structure may include, for example, a point cloud describing the object structure, an estimate of object dimensions, and / or an estimate of object shape. If the second estimate of the object structure includes a point cloud, step 5014 may include inserting or adding 3D coordinates to the point cloud, which may be included in the second image information or determined based on the second image information. In some scenarios, the determination of the second estimate of the object structure may involve the generation or updating of a global point cloud. As discussed above with respect to step 5004, the global point cloud may already include 3D coordinates, in some instances, included in the first image information or determined based on the first image information. As shown in FIG. 6C, these 3D coordinates may be, for example, a point cloud for positions 3722A1 to 3722A2 on the edge surface of the object 3722. n In some scenarios, the 3D coordinates may also represent positions 3722B1 to 3722B2 on the bottom inner surface of the object 3722. n In such an example, 3D coordinates representing other surfaces and / or other locations on the object structure may be inserted or added to the global point cloud by the computing system 1100 in step 5014. For example, as shown in FIG. 7D , the 3D coordinates used to update the global point cloud may represent locations on the inner surface of the object 3722 from location 3722C1 to location 3722C2 on the first outer side surface of the object 3722. n , and positions 3722D1 to 3722D2 on a second outer side surface of the object 3722. nIn some scenarios, when step 5004 involves generating an initial wide area point cloud based on the first image information, the 3D coordinates added or inserted from the second image information of steps 5012 and 5014 may represent one or more surfaces (e.g., outer surfaces) not described by the initial wide area point cloud of step 5004. In such an example, step 5012 may update the initial wide area point cloud to generate an updated wide area point cloud. As described above, when the object 3722 has a 3D pattern (e.g., a ridge pattern) on its outer side surface, the first image information may lack information regarding the 3D pattern, and as a result, a description of the 3D pattern may be omitted in the initial wide area point cloud. The second image information may capture or otherwise represent the 3D pattern such that the updated wide area point cloud represents the 3D pattern on the outer side surface of the object 3722. If the 3D coordinates included in the second image information use a coordinate system different from the coordinate system used by the initial wide-area point cloud, the computing system 1100 may be configured to update the initial wide-area point cloud by transforming the 3D coordinates so that they are expressed in the coordinate system of the initial wide-area point cloud and adding or inserting the transformed 3D coordinates into the initial wide-area point cloud.

[0081] In some instances, if the second estimate of the object structure includes estimates for object dimensions, the estimated object dimensions may be dimensions not described by the first estimate of the object structure. For example, the first estimate of the object structure determined in step 5004 may include an estimate for a first object dimension (e.g., object length) and an estimate for a second object dimension (e.g., object width), but may lack an estimate for a third object dimension (e.g., object height). In this example, the second estimate of the object structure may include an estimate for a third object dimension (e.g., object height). In some instances, the first estimate of the object structure determined in step 5004 may already include an estimate for the third object dimension, but this estimate may be inaccurate. As discussed above, this inaccuracy may arise because step 5004 may be based on a plan view of the object structure. If step 5004 involves determining an estimate for the object height based on a plan view of the object structure, such estimate may lack a high degree of precision or certainty. In such an example, step 5014 may be used to generate updated estimates for the object dimensions, as discussed in more detail below. The updated estimates may have greater accuracy or a greater degree of certainty.

[0082] In an embodiment, the computing system 1100 may be configured to determine estimates for object dimensions, such as object height, based on 3D coordinates. These 3D coordinates may be within a global point cloud and may include 3D coordinates contained in or determined based on the second image information. As an example, the computing system 1100 may determine estimates for 3D coordinates [X 3722An Y 3722An Z 3722An ] and [X 3722Dn Y 3722Dn Z 3722Dn ], the system 1100 may determine an estimate of the object height for the structure of the object 3722 based on the difference between two of the 3D coordinates. 3722An~ Z 3722Dn In this example, an estimate for the object height may be determined to be equal to or based on the 3D coordinate [X3722An Y 3722An Z 3722An ] may represent a location on an edge surface or other top outer surface of the object 3722 that may form the apex of the object 3722, while the 3D coordinate [X 3722Dn Y 3722Dn Z 3722Dn ] may describe a location that is part of the bottom portion of the object 3722. More specifically, the 3D coordinates [X 3722Dn Y 3722Dn Z 3722Dn ] may represent a location on an outer side surface of object 3722 and near a bottom outer side surface of object 3722. In some scenarios, if the first estimate for the object structure already includes an estimate for an object dimension (e.g., object length or object width), such as an estimate based on the first image information, step 5014 may involve determining an updated estimate for the object dimension, where the updated estimate is based on the second image information.

[0083] In an embodiment, determining a second estimate for the object structure in step 5014 may involve determining an object type for an object corresponding to the object structure, such as object 3722. As discussed above, the computing system 1100 may store or otherwise access templates describing various respective object types. The templates may include visual description information and / or a description of the object structure, such as a CAD model or respective values ​​of various object dimensions. The description of the object structure in the template may, in some circumstances, include a more complete description of the object's structure than that provided by the first image information and / or the second image information and may be used as a second estimate for the object structure. For example, the second image information may have a sufficient level of detail to be used to compare with the various templates in step 5014 to determine whether the second image information matches any of the templates. If one of the templates matches the second image information, the matching template may have a description of the object structure that has a higher level of detail than the second image information. In some scenarios, the object type may already have been determined in step 5004 based on the first image information, but such determination may be intended as an initial guess for the object type. As discussed above, using the first image information to perform template matching may result in a lack of a high level of accuracy or confidence, especially when the first image information lacks a description of a certain part of the object structure, such as its outer side surface. As discussed above, the first image information may lack a description of a 2D or 3D pattern on the outer side surface of the object structure. Meanwhile, the second image information may capture or otherwise represent a 2D, 3D pattern, or other visual or structural details on the side surface of the object structure. If step 5014 also involves performing template matching, this step may result in a higher level of accuracy or confidence, since step 5014 uses the second image information, which may extend the first image information by describing a certain part of the object structure that is not included in or omitted from the first image information.In some scenarios, the second image information may represent a portion of the object structure, such as multiple outer side surfaces of the object structure, which may be particularly useful for template matching. More specifically, the second image information may describe visual details (e.g., visual patterns) or structural details (e.g., ridge patterns) on one or more side surfaces of the object structure. This visual or structural details described by the second image information may improve the accuracy or effectiveness of template matching, especially when many of the different types of containers or other objects undergoing robotic interaction have similar sizes. In such a situation, the size of the object may match the object structure description of each of many templates, each of which may be associated with a different object type. However, the visual or structural details (e.g., ridge patterns) on the side surfaces of the object represented by the second image information may only match the visual description information or object structure description of one template or a few templates, thereby narrowing down the object types to which the object (e.g., 3722) may belong. Thus, visual or structural details in the second image information, which may provide a better description of the side surface of the object than the first image information, may improve the accuracy or effectiveness of template matching and improve the accuracy and effectiveness of determining which object type is associated with the object represented by the second image information.

[0084] As mentioned above, the pallet 3728 may be used in embodiments to stack containers or other objects, which may have a wide variety of sizes. A wide variety of object sizes may result in a wide variety of stacking configurations. In other words, different pallets may have significantly different stacking configurations for how the containers or other objects are arranged. Thus, when the computing system 1100 is determining a motion plan to remove an object from a pallet, the location of the object (e.g., the location of a corner or edge of the object) may have a wide range of possible values. Thus, the second image information may be particularly useful since it may be utilized by the computing system 1100 to provide fine / precise detection of the location of the object and / or some other characteristic of the object (e.g., size).

[0085] In an embodiment, the second image information may be used by the computing system 1100 to identify grip points, which may be locations or portions on an object (e.g., 3722) that are grasped by the robot 1300 / 3300 of FIG. 3A / 6A, or more specifically, the end effector device 3500. These grip points may be identified as part of a motion plan, determined during step 5016, discussed in more detail below. As mentioned above, the end effector device 3500 may include gripper fingers that, in some scenarios, clamp around or pinch a portion of an object, such as a portion of the edge of a container. In some situations, grip points may need to be determined with high accuracy and a high degree of reliability. For example, having a grip point on or near a broken portion of an object may result in a difficult or unstable grip, so the determination of the grip point may need to take into account which portion of the object (e.g., any portion on the rim of the container of the object 3722) is broken, inaccessible, or otherwise difficult to grasp, and therefore the grip point is placed or otherwise positioned on or near that portion of the object. The second image information may provide sufficient precision for the computing system 1100 to identify the broken portion of the object, allowing the grip point to be determined with a sufficiently high level of reliability and accuracy. The second image information may also be used to eliminate grip points that are inaccessible, such as due to the presence or positioning of adjacent objects that would prevent the end effector device 3500 from accessing the grip point.

[0086] In some implementations, if the second image information includes or forms a 2D image representing an object (e.g., 3722), the computing system 1100 may compare the 2D image or a portion thereof (e.g., image portion 7022 of FIG. 7B) with the visual description information of the templates discussed above. Such a comparison may be used to determine whether the object appearance represented by the 2D image or a portion thereof matches the visual description information of one of the templates. In some cases, if the second image information includes 3D image information (e.g., 7084 of FIG. 7C), the matching of the templates may be based on a comparison of the 3D image information or a portion of the 3D image information representing the structure of the object (e.g., a portion of FIG. 7D) with the object structure description of each of the templates. In one example, the matching of the templates may involve comparing the object structure description of the templates with the global point cloud discussed above, which may be based on the second image information and the first image information.

[0087] Returning to FIG. 5, the method 5000 may, in an embodiment, include step 5016, in which the computing system 1100 generates a motion plan based on at least the second estimate of the object structure. The motion plan generated in step 5016 may be for initiating a robotic interaction between the robot and an object (e.g., 3722) associated with the estimated object structure. The interaction may involve, for example, the end effector device (e.g., 3500) approaching the object (also referred to as a target object), picking up the target object, and moving the target object to a destination location. In some cases, the motion plan generated in step 5016 may be an updated motion plan that updates an initial motion plan based on the first estimate of the object structure, as discussed above. In some scenarios, generating the initial motion plan prior to step 5016 may facilitate more timely execution of the robotic interaction. For example, generating the initial motion plan may involve performing calculations or determining information that can be reused when an updated motion plan is determined. If all such calculations or decisions are made during step 5016, there may be scenarios where there is an excessive time difference between when the camera (e.g., 3200) generates the second image information and when a motion plan is generated based on the second image information. Such a time difference may delay the execution of the robot interaction. In such scenarios, at least some of those calculations or decisions may be made as part of determining an initial motion plan based on the first image information. These calculations may be made before step 5016, thus reducing the time required to generate an updated motion plan in step 5016. However, in some embodiments, method 5000 may omit determining an initial motion plan based on a first estimate of the object structure.

[0088] In an embodiment, the motion plan determined in step 5016 may include a trajectory to be followed by an end effector device (e.g., 3500) of the robot (e.g., 3300). For example, FIG. 8A depicts an example trajectory 8010 that, as the end effector device 3500 tracks, causes the end effector device 3500 to approach an object 3722, engage the object 3722 (e.g., pick up the object 3722), and move the object to a destination location 8004. In some instances, step 5016 may include determining a grip point where the end effector device 3500 grips or otherwise engages the object, as discussed above. The grip point may be based on a second estimate of the object structure. For example, the grip point may be determined based on the object dimensions as indicated by the second estimate of the object structure. As an example, if the object structure includes at least four coplanar corners (e.g., having a rectangular shape), the first grip point may be located along a first edge of the object structure and between a first corner and a second corner of the at least four coplanar corners, while the second grip point may be located along a second edge of the object structure and between a first corner and a third corner of the at least four coplanar corners. The first grip point may be closer to the second corner than the first corner, while the second grip point may be closer to the third corner than the first corner. That is, the first grip point may have a distance from the first corner that is at least a predefined percentage of a value for a first dimension of the object (e.g., the distance from the first grip point to the first corner is at least 50% of the value of the width dimension of the object), while the second grip point may have a distance from the first corner that is at least a predefined percentage of a value for a second dimension of the object (e.g., the distance from the second grip point to the first corner is at least 50% of the value of the length dimension of the object). These example grip points may facilitate balanced or otherwise stable grasping of an object.In an embodiment, the location of the grip point may define or otherwise correspond to a grip size for an end effector device (eg, 3500A), as discussed above with respect to Figures 3A and 3B.

[0089] As an example of the motion plan discussed above, Figures 8B-8F depict an end effector device 3500 on a robot arm 3400 of a robot 3300 tracking the trajectory 8010 of Figure 8A to pick up an object 3722. As described above, if the destination location 8004 is part of a destination structure that receives an object (e.g., 3722), the destination location may refer to a location where the earliest contact between the object and the destination structure will occur. For example, if the destination structure is a conveyor belt or a floor, the destination location 8004 may be a location on the top surface of the conveyor belt or on the floor. If the destination structure is a roller conveyor with a set of rollers, the destination location 8004 may be the highest position on one or more of the rollers, as shown in Figure 8F.

[0090] In some cases, determining a trajectory (e.g., 8010) may involve verifying that the trajectory does not result in a collision between the object (e.g., 3722) undergoing robot interaction and a physical element or item in the object's (e.g., 3722) and / or the robot's (e.g., 3300) environment. Examples of physical elements include walls, support beams, power cables, etc. The verification that no collision will occur may be based on an estimate of the object structure for the object (e.g., 3722), which may be determined, for example, from step 5014. For example, the computing system 1100 may determine whether the trajectory (e.g., 8010) will cause the object structure to occupy a space that is also occupied by any of the physical elements discussed above. In this example, the space occupied by the object structure may be defined by the global point cloud discussed above, an estimated shape of the object structure, and / or estimates for various dimensions (e.g., length, width, height) of the object structure.

[0091] In an embodiment, where the end effector device (e.g., 3500) includes at least a first gripper member, a second gripper member, and a third gripper member, such as those shown in Figures 4A and 4B, the computing system 1100 may determine the motion of the gripper members as part of the motion plan. For example, the computing system 1100 may generate the motion plan by determining a motion for the first gripper member (e.g., 3510) to engage one of the first edge or the second edge of the object structure and the second gripper member (e.g., 3520) to engage another of the first edge or the second edge of the object structure. The first and second edges may be perpendicular to each other, for example. The determined motion may further cause the third gripper member (e.g., 3530) to engage an object corner associated with the second camera pose, such as the corner represented by position 3722A1 in Figure 6C, or may cause the third gripper member to engage another object corner of the object structure.

[0092] In some instances, as depicted in Figures 4A and 4B, when a first gripper member (e.g., 3510) is slidable along a first rail (e.g., 3540) of an end effector device (e.g., 3500A) and a second gripper member (e.g., 3520) is slidable along a second rail (e.g., 3542) that is longer than the first rail, the computing system 1100 may be configured to generate a motion plan by determining an operation that causes the first gripper member to engage the shorter of the first edge or the second edge of the object structure and the second gripper member to engage the longer of the first edge or the second edge of the object structure. The computing system 1100 may further be configured to control one or more actuator or stop mechanisms discussed above to slide a first gripper member (e.g., 3510) along a first rail (e.g., 3540) to a position where it can grasp a first grip point described by the motion plan, and to slide a second gripper member (e.g., 3520) along a second rail (e.g., 3542) to a position where it can grasp a second grip point described by the motion plan.

[0093] In an embodiment, step 5016 may involve determining an end point for the trajectory, such as end point 8012 for trajectory 8010 depicted in Figures 8A and 8F. The end point may, for example, specify a position (or more specifically, a pose) where the robot (e.g., 3500) or a component thereof (e.g., end effector device 3500) stops moving and ends interaction with a particular object (e.g., 3722). Ending the interaction may involve, for example, the release of the object from the grip of the end effector device (e.g., 3500). In some implementations, the computing system 1100 may determine an object height estimate h for the object 3722, such as based on the object height of the object structure, as shown in Figure 8A. 3722The endpoint of the trajectory may be determined based on the second estimate of the object structure determined in step 5014. If the motion plan determined in step 5016 is an updated motion plan, and if the computing system 1100 has already determined an initial motion plan having a first endpoint (e.g., based on the first image information), the endpoint determined in step 5016 may be an updated endpoint. In some cases, the updated endpoint may be more reliable than the first endpoint for performing robot interaction because the updated endpoint may be based on a second estimate for the object structure, which may have greater accuracy than the first estimate of the object structure.

[0094] In the examples of Figures 8A and 8F, the computing system 1100 may determine the end point 8012 of the trajectory 8010 based on a final end effector height determined or planned for the end effector device 3500. As discussed above with respect to determining the initial motion plan, the final end effector height may refer to the height of the end effector device when the end effector device 3500 releases or otherwise stops interacting with the object (e.g., 3722) and / or the height of the end effector device when the operation of the end effector device 3500 ends. In some cases, the final end effector height may be expressed relative to the destination position (e.g., 8004) discussed above. In an embodiment, the computing system 1100 may determine the final end effector height based on an estimate of the object height relative to the object, which may be determined in step 5014 based on the second image information. In some cases, the computing system 1100 of FIG. 8A may generate an estimate of the object height h 3722 The final end effector height may be determined to be a height that is above the destination position 8004 by an amount equal to or based on the estimated value h 3722 is part of or based on a second estimate for the object structure of the object 3722 determined in step 5014. More broadly, the computing system 1100 of FIG. 3722The end point 8012 may be determined to be a position away from the destination location 8004 by a distance equal to or based on the final end effector height. By generating a trajectory based on the final end effector height, the computing system 1100 may control the end effector device 3500 to stop operation of the end effector device 3500 substantially simultaneously when a bottom portion of an object (e.g., 3722) being carried by the end effector device 3500 is placed (e.g., positioned) on or otherwise in contact with the destination location 8004. Thus, such a trajectory may be particularly suitable for the end effector device 3500 to stop operation and release the object.

[0095] In an embodiment, the computing system 1100 may be configured to detect the arrival of an object (e.g., 3722) at a destination location. For example, as shown in FIGS. 8G, 8H, and 8I, the roller conveyor 3800 may include one or more sensors 3810, such as a first line sensor 3811 and a second line sensor 3812. The first line sensor 3811 may be positioned at a first distance (e.g., a first height) relative to the roller conveyor 3800, while the second line sensor 3812 may be positioned at a second distance (e.g., a second height) relative to the roller conveyor 3800. The computing system 1100 may generate and output a control signal that causes the robot to move the end effector device 3500 toward the roller conveyor 3800. As shown in FIG. 8H, the first line sensor 3811 may output a first sensor signal to the computing system 1100 indicative of a detection of the proximity of the object 3722 and / or the end effector device 3500 within a first distance from the roller conveyor 3800. Upon receiving the first sensor signal, the computing system 1100 may output (e.g., via the communication interface) one or more motion commands to decelerate or otherwise slow down the motion of the robot arm 3400 and the end effector device 3500 towards the roller conveyor 3800. As shown in FIG. 8I, the second line sensor 3812 may output a second sensor signal to the computing system 1100 indicative of a detection of the proximity of the object 3722 and / or the end effector device 3500 within a second distance from the roller conveyor 3800. Upon receiving the second sensor signal, the computing system 1100 may output one or more motion commands to stop motion of the end effector device 3500 and / or cause the end effector device 3500 to release or otherwise disengage the object 3722.

[0096] Returning to FIG. 5, the method 5000 may, in an embodiment, include a step of causing the computing system 1100 to cause the robot (e.g., 3300), or more specifically, the robot arm (e.g., 3400) and / or the end effector device (e.g., 3500), to perform a robotic interaction by following the motion plan determined in step 5016. In some instances, the computing system 1100 may generate one or more motion commands based on the motion plan and may output the one or more motion commands to the robot (e.g., 3300) via, for example, the communication interface 1130 of FIG. 2B. The one or more motion commands (also referred to as one or more object interaction motion commands), when received and executed by the robot (e.g., 3300), may cause the robot (e.g., 3300) to follow the motion plan and perform a robotic interaction with the object (e.g., 3722) discussed above.

[0097] In an embodiment, the object that receives or is targeted for robotic interaction as a result of the motion plan from step 5016 may be one of a plurality of objects, such as a stack 3720 of crates or other containers, as depicted in Figures 6A and 7A-7C. In that example, the object 3722 that is moved according to the motion plan of step 5016 may be a first object that is moved from the stack 3720 to a destination location 8004, as shown in Figures 8A-8F. In such an example, the first image information and the second image information received in steps 5002 and 5012 may represent an appearance of the stack 3720 and / or describe the structure of the stack (also referred to as the stack structure). Additionally, the computing system 1100 may use the first image information and / or the second image information to determine an estimate of the stack structure. For example, if the first estimation of step 5004 and the second estimation of step 5014 include a global point cloud, the global point cloud may more specifically describe the stacking structure for the stack 3720, and different portions of the global point cloud may describe the different respective objects 3721-3726 that form the stack 3720. The global point cloud in the above example may represent the stack 3720 prior to removing the first object 3722 discussed above. In an embodiment, the method 5000 may involve interaction with additional objects on the stack, such as one or more of objects 3721 and 3723-3726. The interaction may involve, for example, picking up each of the objects 3721, 3723-3726 and moving them to a destination location (e.g., a conveyor belt) as part of an off-pallet operation.

[0098] In an embodiment, an interaction with a further object (e.g., 3721) may involve determination of an updated stacking structure reflecting the removal or other movement of a first object (e.g., 3722), which is moved according to the motion plan of step 5016. While this updated estimate of the stacking structure may be determined based on use of a camera (e.g., 3200) to generate additional image information after the first object (e.g., 3722) is removed from the stack (e.g., 3720), the computing system 1100 may alternatively or additionally use a second estimate for the object structure of the first object (e.g., 3722) to determine an updated estimate of the stacking structure of the stack 3720.

[0099] For example, FIG. 9A illustrates an embodiment in which the computing system 1100 has determined an estimate of the stacking structure for the stack 3720 before removing the first object (e.g., 3722). The estimate of the stacking structure may be, for example, a global point cloud representing the outline or shape of the stack 3720, and may be the same as or similar to the 3D image information 7084 of FIG. 7C. In this example, the computing system 1100 may have determined an estimate of the object structure for the first object (e.g., 3722). This estimate may be, for example, the second estimate of the object structure determined in step 5016. Furthermore, this estimate of the object structure may be, for example, a portion of the global point cloud for the stacking structure. Because an estimate for the object structure of the first object (e.g., 3722) was already determined, the computing system 1100 may directly determine an updated estimate for the stacking structure by removing the portion of the estimate that corresponds to the first object. As an example, an estimate of the object structure of a first object (e.g., 3722) may identify 3D coordinates that are on various surfaces of the first object. The computing system 1100 may be configured to remove these 3D coordinates from the estimate of the global point cloud representing the stacked structure, such as by masking out those 3D coordinates from the global point cloud, as shown in FIGS. 9B and 9C. More specifically, FIG. 9B depicts with open circles 3D coordinates that have been deleted or otherwise removed from the global point cloud for stack 3720. FIG. 9C depicts the resulting global point cloud representing an updated estimate for stack 3720 after removing first object 3722. As shown in FIG. 9C, the updated estimate may no longer represent first object 3722, but instead represent the empty space that first object 3722 previously occupied before it was removed from stack 3720.

[0100] In an embodiment, where the method 5000 involves interacting with a second object (e.g., 3721) on the stack (e.g., 3720) after removing the first object (e.g., 3722), the method 5000 may involve generating a second motion plan. The second motion plan may be generated based on an updated estimate of the stack structure and may cause a robotic interaction with the second object, such as an interaction in which the end effector device (e.g., 3500) approaches the second object, engages the second object, and moves the second object to a destination location (e.g., 8004). In some cases, generating the second motion plan may involve determining a new corner of the stack structure that is exposed by the removal of the first object (e.g., 3722) based on the updated estimate of the stack structure. For example, the new corner may be located at location 3722A in FIG. 9B . n The corners may be associated with a second object (e.g., 3721), such as the corner represented by: Accordingly, the computing system 1100 may identify new object corners based on the updated estimate of the stack structure.

[0101] In the above example, the computing system 1100 may return the camera (e.g., 3200) to the first camera pose to generate additional image information, e.g., representing a plan view of the stack (e.g., 3720) after the first object (e.g., 3722) has been removed, but may not need to do this since the computing system 1100 has already determined an estimate of the object structure of the first object in step 5016. In other words, after the first object (e.g., 3722) is removed from the stack (e.g., 3720), the computing system 1100 may determine an updated estimate of the stack structure for the stack by determining which portion of the estimated stack structure corresponds to the first object and masking out or otherwise removing that portion. In some cases, the computing system 1100 may use estimates for the object dimensions of the first object and / or use a point cloud representing the first object to determine which portion of the estimated stack structure corresponds to the first object. After generating the updated estimate of the stack structure, the computing system 1100 may use the updated estimate of the stack structure to identify object corners of the remaining objects. In an embodiment, the computing system 1100 may specifically identify convex corners (e.g., exterior corners) of the remaining objects. Such corners may also be, for example, convex corners of the stack. In some cases, the location 3721A in FIG. 9C may be used to identify object corners of the remaining objects. n One corner of the remaining objects, such as a corner of , may become a convex corner after removal of the first object (e.g., 3722). More specifically, the corner may be directly adjacent to the first object (e.g., 3722) or may become exposed due to removal of the first object. The computing system 1100 may select the new object corner discussed above from among the convex corners of the remaining objects.

[0102] In an embodiment, the new object corner may be used to obtain image information representing a perspective view of a second object (e.g., 3721) to be removed from the stack (e.g., 3720). For example, the computing system 1100 may determine an additional camera pose that points the camera (e.g., 3200) toward the new object corner. The computing system 1100 may repeat steps 5006-5016 to move the camera to the additional camera poses and may receive additional image information generated by the camera (e.g., 3200) while the camera has the additional camera pose. In this example, the computing system 1100 may use the additional image information to generate a second motion plan for causing a robot interaction with the second object (e.g., 3721) in the same or similar manner as steps 5014 and 5016.

[0103] As mentioned above, one aspect of the present application relates to interactions in which a robot moves an object from a current position to a destination position. FIG. 10 depicts a flow diagram for an exemplary method 10000 for moving an object (e.g., 3722 of FIGS. 8A and 8B). Method 10000 may be performed, for example, by the computing system 1100 of FIGS. 2A-2D. In an embodiment, method 10000 may begin with or otherwise include step 10002, in which the computing system selects an object to be moved. For example, computing system 1100 may select a container or other object to be moved and determine a motion plan for the robot to engage the object and move it to a destination position. The motion plan may include a trajectory for lowering an end effector device (e.g., 3500) by the robot (e.g., 3300) to approach the object, align with an edge or corner of the object, and grasp the object.

[0104] Method 10000 may, in an embodiment, include step 10004, in which computing system 1100 may output one or more motion commands to cause the robot to place or otherwise position the end effector device (e.g., 3500) directly above the object. In an embodiment, computing system 1100 may use image information generated by camera 3200 of FIG. 8A when engaging the object to determine or verify the location of the object, e.g., object 3722. Once the object location is determined, computing system 1100 may generate and output one or more motion commands to cause robot 3300 to place end effector device 3500 directly above object 3722, as shown in FIG. 8B. In an embodiment, computing system 1100 may cause robot 3300 to orient end effector device 3500 such that a bottom surface of end effector device 3500 faces object 3722.

[0105] In step 10006, the computing system 1100 may cause an end effector device (e.g., 3500) of the robot (e.g., 3300) to grasp or otherwise engage the object. In an embodiment, step 10006 may involve generating one or more motion commands to cause the end effector device 3500 to lower toward the object, or more broadly, in a negative Z direction, as shown in FIG. 8C. In an embodiment, the computing system 1100 may be configured to cause motion of the first gripper member (e.g., 3510) and the second gripper member (e.g., 3520) along the first rail (e.g., 3540) and the second rail (e.g., 3542), respectively, to adjust the grip size of the end effector device (e.g., 3500) based on the size of the object (e.g., 3722), as discussed above. More specifically, the computing system 1100 may cause the area defined by the grip size of the end effector device to have a size that substantially matches or is otherwise based on the size of the object 3701. In a more specific example, the computing system 1100 may determine a grip point on the object 3701 and control the operation of the first gripper member and the second gripper member to grip the object 3701 at the determined grip point. The computing system 1100 may further cause the end effector device 3500 to engage the object 3722, as shown in FIG. 8D.

[0106] In step 10008, the computing system 1100 may cause the robot to move the object to a destination location. For example, the computing system 100 may generate and output one or more motion commands to cause the robot 3300 to move the end effector device 3500 to a destination location, such as a position on a conveyor 3800, as shown in Figures 8E-8G. In an embodiment, the one or more motion commands may be generated based on the motion plan discussed above.

[0107] In step 10010, the computing system 1100 may detect the arrival of an object at the destination location. In an embodiment, the computing system 1100 may detect the arrival of an object at the destination location using one or more sensors at the destination location, such as the line sensors discussed above with respect to Figures 8G-8I. In step 10012, the computing system 1100 may generate one or more motion commands to cause the end effector device 3500 of the robot 3300 to release the object 3722 at the destination location.

[0108] Additional Considerations Regarding Various Embodiments

[0109] A first embodiment relates to a computing system including a communication interface and at least one processing circuit. The communication interface is configured to communicate with (i) a robot having an end effector device, and (ii) a camera mounted on the end effector device and having a camera field of view. The at least one processing circuit includes: receiving first image information representing at least a first exterior surface of an object structure associated with the object when the object is or was in the camera field of view, the first image information being generated by the camera when the camera has a first camera pose in which the camera is oriented toward the first exterior surface such that the camera field of view encompasses the first exterior surface; determining a first estimate of the object structure based on the first image information; identifying a corner of the object structure based on the first estimate of the object structure or based on the first image information; and determining a second camera pose that, when adopted by the camera, orients the camera toward the corner of the object structure such that the camera field of view encompasses the corner of the object structure and at least a portion of the second exterior surface. and, when executed by the robot, outputting to the end effector device one or more camera positioning action commands that move the camera to a second camera pose; receiving second image information representing the object structure, the second image information being generated by the camera while the camera has the second camera pose; determining a second estimate of the object structure based on the second image information; generating a motion plan based on at least the second estimate of the object structure, the motion plan being for causing a robotic interaction between the robot and the object; and outputting one or more object interaction action commands for causing the robotic interaction, the one or more object interaction action commands being generated based on the motion plan.

[0110] Example 2 includes the computing system of example 1, wherein the first estimate for the object structure includes at least an estimate for a first object dimension of the object structure and an estimate for a second object dimension of the object structure, and the second estimate for the object structure includes at least an estimate for a third object dimension of the object structure.

[0111] A third embodiment includes the calculation system of the second embodiment, wherein the first object dimension is an object length, the second object dimension is an object width, and the third object dimension is an object height.

[0112] Example 4 includes the computing system of example 2 or 3, wherein the second estimate for the object structure includes an updated estimate for the first object dimension and an updated estimate for the second object dimension.

[0113] A fifth embodiment includes the computing system of any one of the first to fourth embodiments, wherein the second estimate for the object structure includes an estimated shape for the object structure.

[0114] Embodiment 6 includes a computational system of any one of embodiments 1 to 5, wherein a first estimation of the object structure includes a point cloud that identifies a location on a first outer surface of the object structure without identifying a location on the second outer surface of the object structure, and a second estimation of the object structure includes an updated point cloud that identifies a location on the first outer surface and a location on the second outer surface of the object structure.

[0115] Embodiment 7 includes a computing system of any one of embodiments 1 to 6, wherein at least one processing circuit is configured to determine a second estimation of the object structure by determining an object type corresponding to the object based on the second image information, and determining a defined object structure description associated with the object type, wherein the object structure description describes a structure associated with the object type, and determining a second estimation of the object structure based on the object structure description.

[0116] Example 8 includes the computing system of example 7, wherein at least one processing circuit is configured to determine the object type by comparing the second image information to one or more templates that include a description of one or more respective object structures.

[0117] Example 9 includes a computational system of any one of Examples 1 to 8, and the motion plan includes a trajectory that, when tracked by the end effector device, causes the end effector device to approach an object, engage the object, and move the object to a destination position.

[0118] Example 10 includes the computing system of Example 9, wherein the motion plan is an updated motion plan, and at least one processing circuit is configured to generate an initial motion plan based on a first estimate of the object structure, and to generate an updated motion plan based on the initial motion plan and based on a second estimate of the object structure.

[0119] Example 11 includes the computing system of example 9 or 10, wherein the second estimate of the object structure includes an estimate for the object height, and the at least one processing circuit is configured to determine a final end effector height relative to the destination position based on the estimate for the object height, and determine an endpoint of the trajectory based on the final end effector height.

[0120] Embodiment 12 includes a calculation system of any one of embodiments 1 to 11, and when the end effector device includes at least a first gripper member, a second gripper member, and a third gripper member, at least one processing circuit is configured to generate a motion plan by determining an operation that causes the first gripper member to engage one of the first edge or the second edge of the object structure, the second gripper member to engage another of the first edge or the second edge of the object structure, and the third gripper member to engage a corner associated with the second camera pose or engage another corner of the object structure.

[0121] Example 13 includes a computing system of any one of Examples 1 to 12, wherein when the first estimate of the object structure describes a plurality of corners, at least one processing circuit is configured to select a corner from among the plurality of corners, the selection being based on at least one of (i) a respective amount of occlusion experienced by the plurality of corners, or (ii) a respective reachability of the plurality of corners by the end effector device.

[0122] Embodiment 14 includes a computing system of any one of embodiments 1 to 13, and when the object is a first object in a stack of multiple objects and the motion plan is a first motion plan to remove the first object from the stack, at least one processing circuit is configured to: determine an estimate of the stacking structure based on next first image information or second image information, where the estimate of the stacking structure represents the stack before the first object is removed; determine an updated estimate of the stacking structure based on a second estimate of the object structure, where the updated estimate of the stacking structure represents the stack after the first object is removed; and generate a second motion plan based on the updated estimate of the stacking structure, where the second motion plan is to cause a robot interaction with a second object in the stack.

[0123] Example 15 includes the computing system of example 14, wherein at least one processing circuit is configured to generate a second motion plan by determining a new corner of the stacked structure exposed by removal of the first object based on the updated estimate of the stacked structure, where the new corner is associated with a second object, determining an additional camera pose that directs the camera toward the new corner, and receiving additional image information generated by the camera while the camera has the additional camera pose, where the second motion plan is generated based on the additional image information.

[0124] Example 16 includes the computing system of example 15, wherein the estimate for the stacked structure includes a point cloud describing positions on the stack, and at least one processing circuit is configured to determine an updated estimate of the stacked structure by updating the point cloud to remove positions on the stack that also belong to the object structure, and the positions on the stack that also belong to the object structure are identified by a second estimate of the object structure.

[0125] It will be apparent to those skilled in the relevant art that other suitable modifications and adaptations to the methods and applications described herein can be made without departing from the scope of any of the embodiments. The above-described embodiments are illustrative examples, and the present invention should not be construed as being limited to these particular embodiments. It should be understood that the various embodiments disclosed herein may be combined in different combinations than those specifically presented in the description and accompanying figures. It should also be understood that, by way of example, certain acts or events of any of the processes or methods described herein may be performed in a different order, or may be added, integrated, or omitted entirely (e.g., not all acts or events described may be necessary to perform a method or process). In addition, although certain features of the embodiments herein are described as being performed by a single component, module, or unit for clarity, it should be understood that the features and functions described herein may be performed by any combination of components, modules, or units. Thus, various changes and modifications may be effected by those skilled in the art without departing from the spirit or scope of the invention, as defined in the appended claims.

Claims

1. (i) a robot having an end effector device; and (ii) a communications interface mounted on the end effector device and configured to communicate with a camera having a camera field of view; at least one processing circuit; The at least one processing circuit, when an object is or was within the camera field of view, estimating the positions of corners of the object; generating commands to move the camera to a camera pose corresponding to a camera orientation that points the camera toward the corner of the object and at least a portion of an outer surface of the object structure; receiving image information representative of the object structure, the image information being generated by the camera while the camera is in the camera pose; determining an estimate of the object structure based on the image information; and generating a motion plan based on the estimation of the object structure, the motion plan being for causing a robotic interaction between the robot and the object; A computing system configured to:

2. The at least one processing circuit further comprises: determining a first estimate of an object structure associated with the object; determining a second estimate of the object structure based on the image information; and generating the motion plan based on at least the second estimate of the object structure; the corners of the object are estimated based on the first estimate of the object structure; the first estimate for the object structure includes at least an estimate for a first object dimension of the object structure and an estimate for a second object dimension of the object structure; The computing system of claim 1 , wherein the second estimate for the object structure includes at least an estimate for a third object dimension of the object structure.

3. The computing system of claim 2 , wherein the first object dimension is an object length, the second object dimension is an object width, and the third object dimension is an object height.

4. The computing system of claim 2 , wherein the second estimate for the object structure includes an updated estimate for the first object dimension and an updated estimate for the second object dimension.

5. The computing system of claim 2 , wherein the second estimate for the object structure includes an estimated shape for the object structure.

6. The computing system of claim 2 , wherein the second estimate for the object structure includes an updated point cloud that identifies locations on the exterior surface of the object structure.

7. The at least one processing circuit comprises: determining an object type corresponding to the object based on the image information; determining a defined object structure description associated with the object type, the defined object structure description describing a structure associated with the object type; determining the second estimate of the object structure based on the defined object structure description; The computing system of claim 2 , configured to determine the second estimate of the object structure by:

8. 8. The computing system of claim 7, wherein the at least one processing circuit is configured to determine the object type by comparing the image information to one or more templates that include a description of one or more respective object structures.

9. The motion plan includes: When the end effector device tracks, The computing system of claim 2 , further comprising a trajectory for causing the end effector device to approach, engage, and move the object to a destination location.

10. the motion plan is an updated motion plan; The at least one processing circuit comprises: generating an initial motion plan based on the first estimate of the object structure; and generating the updated motion plan based on the initial motion plan and based on the second estimate of the object structure.

11. the second estimate of the object structure includes an estimate for an object height; The at least one processing circuit comprises: determining a final end effector height relative to the destination position based on the estimate for the object height; and determining an end point of the trajectory based on the final end effector height.

12. When the end effector apparatus includes at least a first gripper member, a second gripper member, and a third gripper member, The at least one processing circuit comprises: engaging the first gripper member with one of a first edge or a second edge of the object structure; engaging the second gripper member with another of the first edge or a second edge of the object structure; causing the third gripper member to engage the corner associated with the camera pose or to engage another corner of the object structure; The computing system of claim 1 , configured to generate the motion plan by determining a motion.

13. 3. The computing system of claim 2, wherein when the first estimate of the object structure describes multiple corners, the at least one processing circuit is configured to select the corner from among the multiple corners, the selection being based on at least one of (i) the amount of occlusion experienced by the multiple corners, or (ii) the reachability of each of the multiple corners by the end effector device.

14. when the object is a first object in a stack of objects, and the motion plan is a first motion plan for removing the first object from the stack of objects; The at least one processing circuit comprises: determining an estimate of a stacking structure based on the image information, the estimate of the stacking structure representing a stack of the plurality of objects prior to removing the first object; determining an updated estimate of the stack structure based on the second estimate of the object structure, such that the updated estimate of the stack structure represents a stack of the plurality of objects after removing the first object; 3. The computing system of claim 2, configured to: generate a second motion plan based on the updated estimate of the stack structure, the second motion plan being for causing a robotic interaction with a second object in the stack of objects.

15. The at least one processing circuit comprises: determining a new corner of the stacked structure that is exposed by removal of the first object based on the updated estimate of the stacked structure, the new corner being associated with the second object; determining an additional camera pose that points the camera toward the new corner; receiving additional image information generated by the camera while the camera has the additional camera pose, the second motion plan being generated based on the additional image information; The computing system of claim 14 , configured to generate the second motion plan by:

16. the estimation of the stacked structure includes a cloud of points describing positions of the objects on the stack; 15. The computing system of claim 14, wherein the at least one processing circuit is configured to determine the updated estimate of the stacked structure by updating the point cloud to remove locations on the stack of the plurality of objects that also belong to the object structure, and locations on the stack of the plurality of objects that also belong to the object structure are identified by the second estimate of the object structure.

17. When executed by at least one processing circuit of a computing system configured to communicate with (i) a robot having an end effector device, and (ii) a camera mounted on the end effector device, the camera having a camera field of view, the at least one processing circuit is caused to: Estimating the location of corners of an object; generating commands to move the camera to a camera pose corresponding to a camera orientation that points the camera toward the corner of the object and at least a portion of an outer surface of the object structure; receiving image information representative of the object structure, the image information being generated by the camera while the camera is in the camera pose; determining an estimate of the object structure based on the image information; and generating a motion plan based on the estimation of the object structure, the motion plan being for invoking a robotic interaction between the robot and the object; A non-transitory computer-readable medium having instructions to cause a

18. The at least one processing circuit further comprises: determining a first estimate of an object structure associated with the object; determining a second estimate of the object structure based on the image information; and generating the motion plan based on at least the second estimate of the object structure; the corners of the object are estimated based on the first estimate of the object structure; the first estimate for the object structure includes at least an estimate for a first object dimension of the object structure and an estimate for a second object dimension of the object structure; 20. The non-transitory computer-readable medium of claim 17, wherein the second estimate for the object structure includes an estimate for at least a third object dimension of the object structure.

19. 1. A method performed by (i) a robot having an end effector device, and (ii) a computing system configured to communicate with a camera mounted on the end effector device and having a camera field of view, the method comprising: Estimating the location of corners of an object; moving the camera to a camera pose corresponding to a camera orientation that directs the camera toward the corner of the object and at least a portion of an outer surface of the object structure; receiving image information representative of the object structure, the image information being generated by the camera while the camera is in the camera pose; determining an estimate of the object structure based on the image information; and generating a motion plan based on the estimation of the object structure, the motion plan being for causing a robotic interaction between the robot and the object.

20. determining a first estimate of an object structure associated with the object; determining a second estimate of the object structure based on the image information; and generating the motion plan based on at least the second estimate of the object structure; the corners of the object are estimated based on the first estimate of the object structure; the first estimate for the object structure includes at least an estimate for a first object dimension of the object structure and an estimate for a second object dimension of the object structure; The method of claim 19 , wherein the second estimate for the object structure includes an estimate for at least a third object dimension of the object structure.

Images