Method and Computing System for Implementing Grip Area Detection - Patent application

Abstract

A method implemented by a computing system for performing grip area detection is presented. The method may include receiving image information representing an object surface associated with a flexible object and identifying, based on the image information, a surface region of the object surface that meets a defined smoothness condition and has a region size equal to or greater than a defined region size threshold as a grip region. The method may further include identifying, as a safety region, a three-dimensional (3D) region that surrounds the grip region in one or more horizontal dimensions and extends from the grip region along a vertical dimension perpendicular to the one or more horizontal dimensions. The method may further include performing robot motion planning based on the grip region and the safety region.

Description

[Technical field]

[0001] The present disclosure relates to computational systems and methods for performing grip area detection of flexible objects and the like. [Background technology]

[0002] As automation becomes more commonplace, robots are being used in more environments, such as warehousing and retail environments. For example, robots may be used to interact with objects in a warehouse. The robot's actions may be fixed or may be based on inputs, such as information generated by sensors in the warehouse. Summary of the Invention

[0003] An aspect of the disclosure relates to a computing system, a method, and a non-transitory computer-readable medium having instructions that may be executed by the computing system to perform the method. The system may include a communication interface and at least one processing circuit. The communication interface is configured to communicate with a robot having an end effector device and a camera having a camera field of view. The at least one processing circuit is configured to perform the method when the flexible object is within the camera field of view. The method may include receiving image information representing an object surface associated with the flexible object, the image information being generated by the camera. The method may further include identifying a surface region of the object surface as a grip region based on the image information, the surface region having a region size that satisfies a defined smoothness condition and is equal to or greater than a defined region size threshold. The method may further include identifying a three-dimensional (3D) region as a safety region that encloses the grip region in one or more horizontal dimensions and extends from the grip region along a vertical dimension perpendicular to the one or more horizontal dimensions. The method may further include implementing a robot motion plan for grasping the flexible object at the gripping region by the end effector device based on the gripping region and the safety region. [Brief description of the drawings]

[0004] [Figure 1A] 1 illustrates a system for performing grip area detection consistent with embodiments herein. [Figure 1B] 1 illustrates a system for performing grip area detection consistent with embodiments herein. [Figure 1C] 1 illustrates a system for performing grip area detection consistent with embodiments herein. [Figure 1D] 1 illustrates a system for performing grip area detection consistent with embodiments herein. [Figure 2A] 1 provides a block diagram illustrating a computing system configured to perform grip area detection consistent with embodiments herein. [Figure 2B] 1 provides a block diagram illustrating a computing system configured to perform grip area detection consistent with embodiments herein. [Figure 2C] 1 provides a block diagram illustrating a computing system configured to perform grip area detection consistent with embodiments herein. [Figure 3A] 1 illustrates an environment in which grip area detection may occur, according to embodiments herein. [Figure 3B] 1 illustrates an environment in which grip area detection may occur, according to embodiments herein. [Figure 3C] 1 illustrates an environment in which grip area detection may occur, according to embodiments herein. [Figure 4] 1 provides a flow diagram illustrating a method for identifying grip and safety areas according to an embodiment herein. [Figure 5A] 1 illustrates example image information in which grip areas and / or safety areas may be identified, according to embodiments herein. [Figure 5B] 1 illustrates example image information in which grip areas and / or safety areas may be identified, according to embodiments herein. [Figure 6] 1 illustrates surface areas that may be identified as grip areas, according to embodiments herein. [Figure 7A]1 illustrates surface areas that may be identified as grip areas, according to embodiments herein. [Figure 7B] 1 illustrates surface areas that may be identified as grip areas, according to embodiments herein. [Figure 8A-8G] 8A-8G illustrate various ways of identifying grip areas according to embodiments herein. [Figure 9A-9B] 9A and 9B show various aspects of identifying grip areas according to embodiments herein. [Figure 10] 1 illustrates a 3D region identified as a safe area according to an embodiment herein. [Figure 11A-11B] 11A and 11B illustrate various aspects of identifying a safe region according to embodiments herein. [Figure 12A-12B] 12A and 12B illustrate various aspects of identifying a safety zone or a safety zone based exercise plan according to embodiments herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] One aspect of the disclosure relates to identifying a grip area and / or a safety area for gripping an object. The grip area, which may also be referred to as a pickable area, may be used to facilitate a robot picking up or otherwise gripping an object. The safety area, which may also be referred to as a safety volume, may be used to determine a trajectory of the object with a reduced risk of collision. In one example, the object may be a package in a warehouse or retail space, and the grip area and / or the safety area may be identified to facilitate the process of controlling the robot to grip the object and move the object to a destination location.

[0006] In an embodiment, identifying a grip area for picking up an object may involve identifying a sufficiently smooth and sufficiently large surface area on the surface of the object as the grip area. The grip area may be a surface area identified from image information generated by a camera, where the image information may represent a scene including the object to be gripped. The surface area in this embodiment may provide a suitable candidate area for an end effector device of the robot to engage and grasp. In some instances, the robot may include an end effector device having one or more suction cups configured to be pressed against or otherwise contact the surface of the object. The one or more suction cups may create adhesion with the surface of the object when there is sufficient contact between the surface of the object and the one or more suction cups. In some instances, the object to be gripped may be a flexible object, such as a package of clothing. More specifically, the package may include a piece of clothing wrapped in a sheet of plastic or other packaging material. Such flexible objects may be sufficiently soft or otherwise deformable that one or more portions of the flexible object may temporarily form one or more non-smooth portions as a result of how the flexible object is physically arranged in its environment. The non-smooth portions may include, for example, bumps, wrinkles, ridges, depressions, or other physical features that introduce non-smoothness to respective local portions of the surface of the flexible object. The non-smooth portions of the surface of the object may reduce the amount of possible contact between the end effector device and that portion of the surface of the object, which may hinder the ability of the end effector device to create adhesion between itself and that portion of the surface of the object. Thus, the computing system of the embodiments herein may attempt to identify sufficiently smooth surface regions as grip regions so that the end effector device can more reliably create adhesion between itself and the grip region. The identified grip regions may be larger than a defined region size threshold to be large enough to fit a component of the end effector device (e.g., a suction cup) used to contact the surface of the object.

[0007] In an embodiment, the computing system may attempt to find a sufficiently smooth surface region by searching for a surface region that satisfies a defined smoothness condition. Such a condition may involve portions of the surface region having a sufficiently similar orientation and a sufficiently similar depth. For example, the computing system may divide one or more object surfaces into a number of two-dimensional (2D) patches, also called 2D planes, as discussed in more detail below, and search for contiguous 2D patches that satisfy a defined orientation similarity criterion and / or a defined depth similarity criterion.

[0008] In an embodiment, the computing system may attempt to identify the grip region as a surface region that is formed by or fits within a set of contiguous 2D patches that satisfy a defined smoothness condition. In some implementations, the computing system may identify a 2D patch that represents the smoothest location or cluster of locations in a scene represented by image information generated by a camera. The computing system may use the 2D patch as a seed or initial 2D patch and attempt to expand the initial 2D patch into a set of contiguous 2D patches. The expansion operation may define a local region that includes the initial 2D patch and may involve searching for additional 2D patches that satisfy the defined smoothness condition. In some instances, the computing system may search for the largest local region that can be formed from such an expansion operation while still satisfying the defined smoothness condition. In some implementations, the grip region may be such a local region that is formed by a set of contiguous 2D patches. In some implementations, the grip region may be a smaller region that fits within such a local region.

[0009] In an embodiment, the computing system may determine as the safe area a three-dimensional (3D) area that surrounds the grip area in one or more horizontal dimensions. In some instances, the safe area may surround the grip area in a vertical dimension. The safe area may define a safety margin around the grip area that may be taken into account by motion planning operations to reduce the risk of collision between an object grasped by the robot and other objects or structures in the robot environment. In some instances, the safe area may be large enough such that when the safe area surrounds the safe area in one or more horizontal dimensions, it is likely to surround the object associated with the grip area. For example, the safe area may have edges that are determined based on a defined lateral dimension size associated with an object or group of objects having the same size, or based on a maximum lateral dimension size associated with a group of objects having different sizes.

[0010] In embodiments, the safe region may be determined by the computing system to have a height based on the lateral dimension size of the object. In such embodiments, the object may be a flexible object that is soft enough to sag under its own weight when the object is lifted by the robot. In such embodiments, the height of the safe region may take into account the sagging of the flexible object, which may be used by the motion planning to reduce the risk of collision between the sagging portion of the flexible object and other structures or objects in the robot's environment.

[0011] FIG. 1A illustrates a system 1000 for performing grip area detection, or more specifically, generating detection results that identify grip areas, based on image information. More specifically, the system 1000 may include a computing system 1100 and a camera 1200. In this example, the camera 1200 may be configured to generate image information that describes or otherwise represents an environment in which the camera 1200 is located, i.e., more specifically, represents an environment in a field of view (also referred to as a camera field of view) of the camera 1200. The environment may be, for example, a warehouse, a retail space, or some other facility. In such an instance, the image information may represent objects, such as packages of clothing or other flexible objects, stored and / or handled in such a facility. The system 1000 may be configured to generate, receive, and / or process the image information, such as by performing robotic motion planning based on the image information, as discussed in more detail below. The robot motion planning may be used to control the robots in the facility to facilitate robotic interactions between the robots and objects, such as, for example, a robot picking up one of the objects from an original position or grabbing one of the objects and moving the object to a destination position. The computing system 1100 and the camera 1200 may be located in the same facility or may be located remotely from each other. For example, the computing system 1100 may be part of a cloud computing platform hosted in a data center remote from a warehouse or retail space and may communicate with the camera 1200 via a network connection.

[0012] In an embodiment, the camera 1200 may be a 2D camera and / or a 3D camera (the terms "and / or" and "or" are used interchangeably in this disclosure). For example, FIG. 1B illustrates a system 1000A (which may be an embodiment of the system 1000) including a computing system 1100, and a camera 1200A and a camera 1200B (both of which may be embodiments of the camera 1200). In this example, the camera 1200A may be a 2D camera configured to generate 2D image information that includes or forms a 2D image describing a description of the visual appearance of an environment in the field of view of the camera. The camera 1200B may be a 3D camera (also referred to as a spatial structure sensing camera or a spatial structure sensing device) configured to generate 3D image information that includes or forms spatial structure information regarding the environment in the field of view of the camera. The spatial structure information may include depth information that describes the respective depth values ​​of various positions relative to the camera 1200B, such as positions on the surface of various objects in the field of view of the camera 1200. The depth information of this example may be used to estimate how objects are spatially arranged in three-dimensional (3D) space. In some instances, the spatial structure information may include or be used to generate a depth map or point cloud that describes positions on the surface of one or more objects (also referred to as one or more object surfaces) in the field of view of the camera 1200B. In some instances, the spatial structure information may describe various positions on the structure of the object (also referred to as the object structure).

[0013] In an embodiment, the system 1000 may be a robotic manipulation system for facilitating interaction between a robot and various objects in the environment of the camera 1200. For example, FIG. 1C shows a robotic manipulation system 1000B, which may be an embodiment of the system 1000 / 1000A of FIGS. 1A and 1B. The robotic manipulation system 1000B may include a computing system 1100, a camera 1200, and a robot 1300. As described above, the robot 1300 may be used to interact with one or more objects in the environment of the camera 1200, such as a package of clothes placed in a bin. For example, the robot 1300 may be configured to pick up or otherwise grab a package with a piece of clothing, lift the package upward from the bin, and move the package to another location, such as a location on a conveyor belt. In some implementations, the camera 1200 may be attached to the robot 1300, such as to a robotic arm of the robot 1300. Such implementations allow the camera 1200 to be moved to different positions and / or orientations via the robotic arm. In some implementations, the camera 1200 may be separate from the robot 1300. For example, the camera 1200 may be mounted to a ceiling or other structure in a warehouse and may remain stationary relative to the structure.

[0014] In an embodiment, the computing system 1100 of FIGS. 1A-1C may form or be part of a robot control system (also referred to as a robot controller) that is part of the robot manipulation system 1000B. The robot control system may be, for example, a system configured to generate motion commands or other commands for the robot 1300. In such an embodiment, the computing system 1100 may be configured to generate such commands based on image information generated by, for example, the cameras 1200 / 1200A / 1200B. For example, the computing system 1100 may be configured to determine a motion plan based on the image information and generate one or more motion commands to execute the motion plan. The motion plan may be, for example, a robot motion plan, which may be a plan for the robot to execute to move one or more objects. For example, the motion plan may include an end effector device of the robot 1300 approaching an object and then picking up or otherwise grasping the object (e.g., via a suction cup or other component of the end effector device) and moving the object to a destination location.

[0015] In an embodiment, the computing system 1100 may form or be part of a vision system. The vision system may be, for example, a system that generates visual information describing the environment in which the robot 1300 is located, or more specifically, the environment in which the camera 1200 is located. The visual information may include 3D image information or 2D image information as discussed above, or some other image information. In some scenarios, when the computing system 1100 forms a vision system, the vision system may be part of the robot control system as discussed above, or may be separate from the robot control system. When the vision system is separate from the robot control system, it may be configured to output information describing the environment in which the robot 1300 is located. The information may be output to the robot control system, which may receive such information from the vision system and control the operation of the robot 1300 based on the information, or more specifically, to implement a robot motion plan.

[0016] In an embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 through a direct connection, such as a connection provided through a dedicated wired communication interface, such as an RS-232 interface, a universal serial bus (USB) interface, and / or a local computer bus, such as a peripheral component interconnect (PCI) bus. In an embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 through a network. The network may be any type and / or form of network, such as a personal area network (PAN), a local area network (LAN), e.g., an intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The network may utilize different technologies and layers or stacks of protocols, including, for example, Ethernet protocols, Internet Protocol Suite (TCP / IP), Asynchronous Transfer Mode (ATM) technologies, Synchronous Optical Networking (SONET) protocols, or Synchronous Digital Hierarchy (SDH) protocols.

[0017] In an embodiment, the computing system 1100 may communicate directly with the camera 1200 and / or the robot 1300 or may communicate via an intermediate storage device or, more broadly, an intermediate non-transitory computer-readable medium. For example, FIG. 1D illustrates a system 1000C, which may be an embodiment of the system 1000 / 1000A / 1000B, including a non-transitory computer-readable medium 1400 that may be external to the computing system 1100 and may act, for example, as an external buffer or repository for storing image information generated by the camera 1200. In one such example, the computing system 1100 may retrieve or otherwise receive image information from the non-transitory computer-readable medium 1400. Examples of the non-transitory computer-readable medium 1400 include electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. The non-transitory computer readable medium may form, for example, a computer diskette, a hard disk drive (HDD), a solid state drive (SDD), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read only memory (CD-ROM), a digital versatile disc (DVD), and / or a memory stick.

[0018] As mentioned above, the camera 1200 may be a 3D camera and / or a 2D camera. The 2D camera may be configured to generate a 2D image, such as a color image or a grayscale image. The 3D camera may be a depth-sensing camera, such as a time-of-flight (TOF) camera, a structured light camera, a stereoscopic camera or camera system (which may include multiple cameras that collectively provide a stereoscopic image), or any other type of 3D camera. In some cases, the 2D camera and / or the 3D camera may include an image sensor, such as a charge-coupled device (CCD) sensor and / or a complementary metal-oxide semiconductor (CMOS) sensor. In an embodiment, the 3D camera may include a laser, a LIDAR device, an infrared device, a light / dark sensor, a motion sensor, a microwave detector, an ultrasonic detector, a radar detector, or any other device configured to capture spatial structure information.

[0019] As described above, the image information may be processed by the computing system 1100. In an embodiment, the computing system 1100 may include or be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop, etc.), a smartphone, a tablet computing device, and / or any other computing system. In an embodiment, all of the functionality of the computing system 1100 may be performed as part of a cloud computing platform. The computing system 1100 may be a single computing device (e.g., a desktop computer) or may include multiple computing devices.

[0020] 2A provides a block diagram illustrating an embodiment of a computing system 1100. The computing system 1100 includes at least one processing circuit 1110 and a non-transitory computer-readable medium (or media) 1120. In an embodiment, the processing circuit 1110 includes one or more processors, one or more processing cores, a programmable logic controller ("PLC"), an application specific integrated circuit ("ASIC"), a programmable gate array ("PGA"), a field programmable gate array ("FPGA"), any combination thereof, or any other processing circuit.

[0021] In an embodiment, the non-transitory computer readable medium 1120 that is part of the computing system 1100 may be a replacement for or in addition to the intermediate non-transitory computer readable medium 1400 discussed above. The non-transitory computer readable medium 1120 may be a storage device such as an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, such as a computer diskette, a hard disk drive (HDD), a solid state drive (SSD), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, any combination thereof, or any other storage device, etc. In some instances, the non-transitory computer readable medium 1120 may include multiple storage devices. In particular implementations, the non-transitory computer-readable medium 1120 is configured to store image information generated by the camera 1200 and received by the computing system 1100. The non-transitory computer-readable medium 1120 may alternatively or additionally store computer-readable program instructions that, when executed by the processing circuit 1110, cause the processing circuit 1110 to perform one or more of the techniques described herein, such as the operations described with respect to FIG.

[0022] FIG. 2B depicts an embodiment of the computing system 1100, the computing system 1100A, including a communication interface 1130. The communication interface 1130 may be configured to receive image information generated by, for example, the camera 1200 of FIGS. 1A-1D. The image information may be received via an intermediate non-transitory computer-readable medium 1400 or a network as discussed above, or via a more direct connection between the camera 1200 and the computing system 1100 / 1100A. In an embodiment, the communication interface 1130 may be configured to communicate with the robot 1300 of FIG. 1C. If the computing system 1100 is external to the robot control system, the communication interface 1130 of the computing system 1100 may be configured to communicate with the robot control system. The communication interface 1130 may also be referred to as a communication component or communication circuitry, and may include, for example, communication circuitry configured to communicate via a wired or wireless protocol. By way of example, the communications circuitry may include an RS-232 port controller, a USB controller, an Ethernet controller, a Bluetooth controller, a PCI bus controller, any other communications circuitry, or a combination thereof.

[0023] In an embodiment, the processing circuit 1110 may be programmed by one or more computer readable program instructions stored on a non-transitory computer readable medium 1120. For example, FIG. 2C illustrates an embodiment of computing system 1100 / 1100A, computing system 1100B, in which the processing circuit 1110 is programmed by one or more modules, including a grip area detection module 1127 and / or a motion planning module 1129.

[0024] In an embodiment, the grip area detection module 1127 may be configured to perform a grip area detection operation, which may involve, for example, performing steps 4002-4006 of method 4000, discussed in more detail below. The grip area detection operation may be used to facilitate gripping of an object, such as a package having a piece of clothing, by a robot. More specifically, the grip area detection operation may identify a grip area, which may be a suitable area on the surface of the object (also referred to as a suitable surface area) where the object may be picked up or otherwise grasped by the robot. In some implementations, the grip area detection operation may identify as a grip area a surface area on the surface of the object that is sufficiently smooth and sufficiently large to allow the robot to reliably grip the object at the surface area. For example, if the robot uses one or more suction cups to adhere to the surface of the object (also referred to as the object surface), the grip area (also referred to as a grippable area or pickable area) identified by the module 1127 may be a surface area that is sufficiently smooth and sufficiently large to engage one or more suction cups and reliably create an adhesion between the surface area and the one or more suction cups. Module 1127 may determine whether the surface region is sufficiently smooth, for example, by determining whether the surface region satisfies a defined smoothness condition, discussed in more detail below. In an embodiment, grip area detection module 1127 may be configured to detect a safety area surrounding the grip area, as discussed in more detail below. In some instances, grip area detection module 1127 may be configured to analyze or otherwise process image information representing a scene within the camera field of view and identify or generate a grip area and / or a safety area based on the image information. In such instances, the grip area and / or the safety area may be an output of grip area detection module 1127.

[0025] In an embodiment, the motion planning module 1129 may be configured to perform motion planning, or more specifically, robot motion planning, based on the output of the grip area detection module 1127, or more generally, based on image information representative of a scene in the camera field of view. More specifically, motion planning may involve determining motions to be executed by a robot (e.g., 1300) or components thereof. For example, motion planning may involve determining a trajectory of the robot 1300, or more specifically, its end effector device, to grip an object in the camera field of view and move the object to a destination location. As an example, the trajectory may cause a suction cup or other component of the end effector device to engage or otherwise contact a grip area on a surface of the object, which may be identified by module 1127 to grip the object. The trajectory may further cause the end effector device to lift the object and move the object to a destination location. In some instances, the motion planning module 1129 may determine a trajectory based on a safe area, which may also be identified by module 1127. The safety area may be used by module 1129 to reduce the risk of collision between the grasped object and another object or structure in the robot's environment, as discussed in more detail below.

[0026] In embodiments, the non-transitory computer readable medium 1120 may store information used by and / or output of modules 1127 / 1129. For example, if the grip area detection module 1127 uses one or more thresholds, such as an area size threshold, the one or more thresholds may be stored on the non-transitory computer readable medium 1120. The one or more thresholds may be dynamically defined by the computing system 1100 or manually defined, and thus may be referred to as one or more defined thresholds or thresholds.

[0027] In various embodiments, the terms "computer readable instructions" and "computer readable program instructions" are used to describe software instructions or computer code configured to perform various tasks and operations. In various embodiments, the term "module" refers broadly to a collection of software instructions or code configured to cause the processing circuitry 1110 to perform one or more functional tasks. The modules and computer readable instructions may be described as performing various operations or tasks when the processing circuitry or other hardware components execute the modules or computer readable instructions.

[0028] 3A-3C illustrate an example environment in which grip area detection and / or motion planning operations may be performed. More specifically, FIG. 3A depicts a system 3000 (which may be an embodiment of systems 1000 / 1000A / 1000B / 1000C of FIGS. 1A-1D ) that includes a computing system 1100, a robot 3300, and a camera 3200. Camera 3200 may be an embodiment of camera 1200 and represents a scene within a camera field of view 3210 of camera 3200, or more specifically, objects 35201, 35202, 35203, 35204, 35205...35206, for example. n The objects 35201 to 3520 may include n 3A-3C, the robot 3300 may be configured to generate image information representing objects or structures within the camera field of view 3210, such as objects 35201-3520. n 35201-3520 by picking up or otherwise grasping one of the objects 35201-3520, lifting the object from its current location, and moving the object to a destination location. n , may be configured to operate or otherwise interact with each of one or more of the

[0029] In some instances, objects 35201-3520 n Some or all of the objects 35201 to 3520 may be flexible objects. nEach of objects 35201-3520 may be a package having a piece of clothing (e.g., a shirt or pair of pants) or other fabric or textile, which may be wrapped in a sheet of packaging material, such as a sheet of plastic. In some scenarios, the sheet of plastic or other packaging material may be generally impermeable to air or other fluids. In the example of FIG. 3A, objects 35201-3520 n 35201~3520 n may be placed in a container 3510, such as a bin or box used to hold the garments in a facility, such as a warehouse, associated with a clothing manufacturer or retailer.

[0030] In some scenarios, a flexible object of an embodiment herein (e.g., 35201) may have a sufficiently high level of flexibility that allows the flexible object to deform when being moved or otherwise manipulated by the robot 3300. A sufficiently high level of flexibility may correspond to a sufficiently low level of stiffness or rigidity that prevents the object from maintaining its shape when being moved or otherwise manipulated by the robot 3300. In some instances, a flexible object may have a sufficiently high level of flexibility that allows the weight of the flexible object to cause a deformation of its own shape when the flexible object is being lifted by the robot 3300. The deformation may involve, for example, the flexible object bending, or more specifically, sagging, under its own weight when being lifted by the robot 3300. The flexibility of a flexible object may result, for example, from the dimensions of the flexible object and / or the material of the flexible object. In one example, the flexible object may have a thin profile that may introduce flexibility (also referred to as flexibility) into the flexible object. More specifically, the flexible object may have a thickness dimension that is significantly smaller than the size of the lateral dimension (e.g., length or width dimension). In one embodiment, the flexible object may be made from a material that is soft enough to introduce flexibility into the flexible object. In some circumstances, the material of the flexible object may be soft enough to sag under its own weight when the object is being lifted by the robot 3300. For example, if the flexible object is a package with a piece of clothing, it may be made from a material such as a cotton or wool fabric that lacks sufficient rigidity to prevent the material from sagging under its own weight when being lifted by the robot 3300.

[0031] In an embodiment, the robot 3300 (which may be an embodiment of the robot 1300) may include a robot arm 3320 having one end attached to a robot base 3310 and having another end attached to or formed by an end effector device 3330. The robot base 3310 may be used to mount one end of the robot arm 3320, while the other end of the robot arm 3320, and more specifically, the end effector device 3330, may be used to interact with one or more objects (e.g., 35201, 35202, etc.) in the environment of the robot 3300. The interaction may include, for example, grasping and lifting one or more objects and / or moving one or more objects from a current location to a destination location.

[0032] In the embodiment, the end effector device 3330 is configured to move the objects 35201-3520. n One or more suction cups used to pick up or otherwise lift an object, such as one of n In some implementations, the suction cups 33321 to 3332 nEach of the suction cups (also referred to as end effector suction cups) may be a mechanical device configured to contact a surface of an object (e.g., 35201) to reduce a fluid pressure (e.g., air pressure) in a space between the suction cup and the surface of the object (also referred to as the object surface). In one example, the object surface may be formed by a material that is generally impermeable to fluids, or more commonly, may be non-porous, such as a sheet of plastic packaging material used to wrap a piece of clothing. A reduction in fluid pressure, such as a partial or complete vacuum, may result in a pressure difference between the fluid pressure outside the space and the fluid pressure within the space. More specifically, the fluid pressure within the space may be lower than the fluid pressure outside the space, which may create a negative fluid pressure that causes the higher fluid pressure to provide a net force that sticks the suction cup to the object surface. The net force may act as an adhesive force that allows the suction cup to adhere to the object surface so as to grip the object surface. In an embodiment, the suction cup (e.g., 33321 or 3332) may be configured to apply pressure to the object surface, such as a pressure difference between the fluid pressure outside the space and the fluid pressure within the space. More specifically, the fluid pressure within the space may be lower than the fluid pressure outside the space, which may create a negative fluid pressure that causes the higher fluid pressure to apply a net force that sticks the suction cup to the object surface. The net force may act as an adhesive force that allows the suction cup to adhere to the object surface so as to grip the object surface. nEach of the arms 10, 12, 14, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 112, 124, 130, 140, 150, 162, 170, 184, 196, 198, 102, 103, 104, 105, 106, 107, 108, 109, 110, 120, 131, 142, 144, 156, 166, 170, 188, 199, 200, 210, 220, 2 Thus, as discussed in more detail below, the computing system 1100 may be configured to identify or search for surface regions that are sufficiently smooth to be used as grip regions where the suction cup can securely adhere to and therefore grasp the object surface.

[0033] In the embodiment, the camera 3200 captures the objects 35201 to 3520. nand container 3510, or any other object(s) within the camera field of view 3210. The camera 3200 may be a 3D camera configured to generate 3D image information, and / or a 2D camera configured to generate 2D image information. In an embodiment, the 3D image information may represent an object surface of the object (e.g., 35201), or more specifically, may describe the physical structure of the object surface. For example, the 3D image information may include a depth map, or more generally, may include depth information that may describe respective depth values ​​of various positions of the camera field of view 3210 relative to the camera 3200, or relative to some other reference point. The positions corresponding to each depth value may be determined by the distance between the objects 35201-3520. n In some instances, the 3D image information may be the positions on various surfaces of the camera field of view 3210, such as the positions on the respective object surfaces of the objects 35201-3520 in the camera field of view 3210. n The object may include a cloud of points, which may include multiple 3D coordinates describing various locations on the object surface, respectively.

[0034] In an embodiment, the object surface of an object (e.g., 35201) may refer to the outer surface (e.g., top surface) of the object. In such an embodiment, the 3D image information may represent the outer surface, or more specifically, may describe the physical structure of the outer surface. For example, if the camera 3200 generates the 3D image information by sensing light (e.g., laser or structured light) or other signals reflecting from the outer surface, the 3D information may represent, for example, the surface contour of the outer surface. If the outer surface is formed by a transparent material, such as a thin sheet of flexible plastic used as a packaging material, the 3D information may still represent the outer surface of the object. More specifically, the camera 3200 in such a situation may sense light or other signals reflected from a non-transparent material, such as a point of clothing fabric, that is under or otherwise covered by the transparent material. The reflected light or signal may pass through the transparent material and may be detected by the camera 3200 to generate the 3D information. In this situation, the transparent material (e.g., a plastic sheet) may be thin enough that the distance between the outer surface and the surface of the non-transparent material is considered negligible. Thus, in an embodiment, the 3D information may be considered to describe depth information of various positions on the outer surface of the object. Furthermore, when a transparent material forms the outer surface, the transparent material may be sufficiently flexible such that all or many portions of the transparent material adopt the surface contour of the underlying non-transparent material. Thus, the 3D image information in this situation may be considered to describe the outer surface of the object, more specifically, the physical structure or surface contour of the outer surface.

[0035] In an embodiment, the 2D image information may include, for example, color or grayscale images that represent the appearance of one or more objects within the camera field of view 3210. For example, if the object surfaces have visual markings (e.g., logos) or other visual details printed on them, the 2D image information may describe or otherwise represent the visual details. As described above, the object surface may be the exterior surface of the object, which in some circumstances may be formed from a transparent material. In such circumstances, the 2D image information may represent light (e.g., visible light) or other signals that have reflected from the surface of an underlying non-transparent material (e.g., a shirt) and passed through the transparent material that forms the exterior surface. Because the 2D image information in such circumstances is based on light or other signals passing through the exterior surface, the 2D image information may still be considered to represent the exterior surface. Furthermore, the transparent material that forms the exterior surface may in some instances be thin enough and transparent enough to have little or negligible effect on the appearance of the object, such that the appearance of the object or the appearance of the exterior surface of the object may be considered to refer to the appearance of the underlying non-transparent material (e.g., clothing material).

[0036] In an embodiment, system 3000 may include multiple cameras. For example, Figure 3B shows system 3000A (which may be an embodiment of system 3000) including camera 3200A having camera field of view 3210A and including camera 3200B having camera field of view 3210B. Camera 3200A (which may be an embodiment of camera 1200A) may be, for example, a 2D camera configured to generate 2D images or other 2D image information, while camera 3200B (which may be an embodiment of camera 1200B) may be, for example, a 3D camera configured to generate 3D image information.

[0037] In an embodiment, the camera 3200 / 3200A / 3200B may be stationary relative to a reference point, such as a floor on which the container 3510 rests or relative to the robot base 3310. For example, the camera 3200 of FIG. 3A may be mounted to a mounting frame that remains stationary relative to a ceiling, such as a warehouse ceiling, or a floor, relative to the robot base 3310, or some other reference point. In an embodiment, the camera 3200 may be mounted on a robot arm 3320. For example, FIG. 3C depicts a system 3000B (which may be an embodiment of system 1000) in which the camera 3200 is mounted or otherwise mounted on an end effector device 3330 that forms the distal end of the robot arm 3200. Such an embodiment may provide the robot 3300 with the ability to move the camera 3200 to different poses via movement of the robot arm 3320.

[0038] FIG. 4 depicts a flow diagram of an example method 4000 for performing grip area detection. Method 4000 may be performed, for example, by the computing system 1100 of FIGS. 2A-2C or 3A-3C, or more specifically, by at least one processing circuit 1110 of the computing system 1100. In some scenarios, at least one processing circuit 1100 may perform method 4000 by executing instructions stored on a non-transitory computer-readable medium (e.g., 1120). For example, the instructions may cause the processing circuit 1110 to execute one or more of the modules illustrated in FIG. 2C, which may perform method 4000. As an example, steps 4002-4006 discussed below may be performed by the grip area detection module 1127, and step 4008 may be performed by the motion planning module 1129. In an embodiment, method 4000 may be performed in an environment in which computing system 1100 is in communication with a robot and camera, such as robot 3300 and camera 3200 / 3200A / 3200B of Figures 3A and 3B, or with any other camera or robot discussed in this disclosure.

[0039] In an embodiment, one or more steps of method 4000 may be performed when an object, or more specifically, a flexible object (e.g., 35201, 35202, etc.), is in a camera field of view (e.g., 3210 / 3210A / 3210B) of a camera (e.g., 3200 / 3200A / 3200B). For example, one or more steps of method 4000 may be performed immediately after the object enters the camera field of view (e.g., 3210 / 3210A / 3210B), or more generally, while the object is in the camera field of view. In some scenarios, one or more steps of method 4000 may be performed when the object is in or was in the camera field of view. For example, when an object is in a camera field of view (e.g., 3210 / 3210A / 3210B), the camera (e.g., 3200 / 3200A / 3200B) may generate image information representing the object, or more specifically, the surface of the object (also referred to as the object surface), and may communicate the image information to a computing system (e.g., 1100). The computing system may perform one or more steps of method 4000 based on the image information while the object is still in the camera field of view, or even when the object is no longer in the camera field of view.

[0040] In an embodiment, method 4000 may begin with or may otherwise include step 4002, in which computing system 1100 receives image information representing an object surface of an object, such as a flexible object, within a camera field of view (e.g., 3210 / 3210A / 3210B) of a camera (e.g., 3200 / 3200A / 3200B). In an embodiment, the image information may be generated by a camera (e.g., 3200) when the object is (or was) in the camera field of view, and may include, for example, 3D image information and / or 2D image information.

[0041] 5A illustrates an embodiment in which the image information of step 4002 includes 3D image information 5700. More specifically, the 3D image information 5700 may be, for example, a grid of locations organized into rows and columns, such as locations 57001, 57002, ... 5700. nThe camera may include a depth map or other depth information that indicates the depth values ​​of various positions within the camera field of view (e.g., 3200 / 3200A), such as positions 57001-5700A. In some implementations, the depth map may include positions 57001-5700B, 5700C, 5700D, 5700E, 5700F, 5700G, 5700H, 5700I, 5700IJ, 5700IK, 5700IJ ... n In an embodiment, the positions 57001 to 5700 may include pixels indicating the depth values ​​of the respective positions. n At least a portion of the objects 35201-3520 n For example, the 3D image information 5700 may include image portions 57201, 57202, 57203, 57204, 57205, ... 57206. n-3 , 5720 n-2 , 5720 n-1 , 5720 n , each of the image portions may include a respective object (e.g., 35201, 35202, 35203, ..., or 3520 n ) may include depth values ​​for each set of positions on the object surface. In some circumstances, the 3D image information may include depth values ​​for each set of positions 57001 to 5700 n The coordinates may be 3D coordinates, such as [XYZ] Cartesian coordinates, or may have values ​​relative to the camera coordinate system, or some other coordinate system. In this example, the [XYZ] coordinates for a particular location (e.g., 57001) may have a Z component that is equal to or based on the depth value of that location. The depth value may be relative to the camera (e.g., 3200 / 3200A) generating the 3D image information, or may be relative to some other reference point.

[0042] In an embodiment, the 3D image information may describe a surface contour of an object surface. For example, the 3D image information 5700 in FIG. 5A may have at least an image portion 57201 that describes a surface contour of the object surface of the object surface 35201. The surface contour of the object surface may describe a physical structure of the object surface. In some instances, the physical structure of the object surface may be completely or substantially smooth. In some instances, the physical structure of the object surface may include physical features such as wrinkles, ridges, ridges, folds, or depressions that may form one or more non-smooth portions of the object surface.

[0043] As mentioned above, the object surface may be the outer surface (e.g., the top surface) of the object, and may be formed by a transparent material, a non-transparent material (e.g., a translucent or opaque material), or a combination thereof. As further mentioned above, when the outer surface is formed by a transparent material covering an underlying non-transparent material, the transparent material may be sufficiently thin and flexible to be considered to have a negligible effect on the physical structure or surface contour of the object. In such an instance, the 3D image information representing the physical structure or surface contour of the underlying non-transparent material may also be considered to represent the physical structure or surface contour of the outer surface of the object. Furthermore, if the transparent material is sufficiently thin, its thickness may be considered to have a negligible effect on the depth measurement by the camera (e.g., 3200). In such a situation, various positions having depth values ​​represented in the 3D image information, such as the position of the image portion 57201, may be considered to be positions on the outer surface of the corresponding object (e.g., 35201).

[0044] FIG. 5B illustrates 2D image information, or more specifically, images of objects 35201-3520 of FIGS. 3A-3C, generated by camera 3200 / 3200B. n 35201-35202, and the container 3510. More specifically, the 2D image 5600 shows the objects 35201-3520 n and container 3510 (objects 35201-3520 n More specifically, the 2D image 5600 may describe the appearance of the objects 35201, 35202, 35203, ..., 3520 nImage parts 56101, 56202, 56203, 56204, 56205, ..., 5620 represent visual details of n-3 , 5620 n-2 , 5620 n-1 , 5620 n (e.g., pixel regions). In an embodiment, the 2D image information may represent an object surface of an object (e.g., 35201). As described above, the object surface may be an outer surface (e.g., a top surface) of the object, and may be formed by a transparent material, a non-transparent material (e.g., a translucent or opaque material), or a combination thereof. As further described above, when the outer surface is formed by a transparent material covering an underlying non-transparent material, the transparent material may be sufficiently thin and transparent to be considered to have a negligible effect on the appearance of the object. In such an instance, the appearance of the underlying non-transparent material may also be considered to be the appearance of the outer surface of the object, such that the 2D image information is considered to represent the appearance of the outer surface of the object.

[0045] In an embodiment, the computing system 1100 may be configured to determine whether the received image information has gaps where pixel values ​​are unavailable or has an amount of error that exceeds a certain error threshold. For example, the computing system 1100 may determine whether the image information has highlight regions, which may be regions of pixels that are oversaturated with light or other signals reflecting from a scene in the camera field of view. For example, the highlight regions may represent glare from a portion of an object surface, where the glare is caused by reflected light with too much intensity. The glare may result in noise that may cause pixels of the resulting image information to have inaccurate values, such that the resulting pixels form gaps in the image information. In such an embodiment, the computing system 1100 may be configured to fill the gaps in the image information with estimated pixel values. The estimated values ​​may be determined, for example, based on interpolating from pixel values ​​of adjacent pixels around the gap.

[0046] Returning to FIG. 4, method 4000 may, in an embodiment, include step 4004, where computing system 1100 may identify a grip area based on the image information received in step 4002. More specifically, computing system 1100 may identify as a grip area a surface area of ​​the object surface that meets a defined smoothness condition and has an area size that is equal to or greater than a defined area size threshold. In an embodiment, a grip area (also referred to as a pickable area) may be a surface area of ​​an object (e.g., 35201) that is suitable to be picked up or otherwise grasped by a robot or to receive some other robotic interaction, as identified by computing system 1100. For example, FIG. 6 illustrates an example of a computing system that identifies a surface area of ​​an object 35201 that is suitable to be picked up or otherwise grasped by an end effector device 3330, or more specifically, one or more suction cups 33321-3332 of the end effector device 3330. n 9B, discussed below, shows another example of a grip region 6812. The computing system 1100 may determine that the surface region 6810 satisfies a defined smoothness condition. In an embodiment, the defined smoothness condition may be predefined, for example, in the non-transitory computer readable medium 1120, or may be dynamically defined by the computing system 1100. For example, if the defined smoothness condition is formed by one or more criteria that may involve satisfying one or more thresholds, rules or other information describing the one or more criteria or the one or more thresholds may be predefined and stored in the non-transitory computer readable medium 1120.

[0047] In some instances, the defined smoothness condition of a surface region (e.g., 6810) may be associated with the surface region being sufficiently smooth for its intended purpose, such as to be picked up or otherwise gripped by an end effector device. Grippling by an end effector device may involve, for example, a suction cup or other component of the end effector device contacting the surface region (e.g., 6810). As mentioned above, the end effector device may, in some scenarios, push the suction cup into contact against the surface region in an attempt to create a sealed space with negative fluid pressure that creates adhesion between the suction cup and the surface region. In one such example, meeting the smoothness condition may involve the surface region being sufficiently smooth to provide a sufficient amount of contact between itself and the suction cup or other component of the end effector device. For example, the smoothness condition may involve the surface region (e.g., 6810) being sufficiently smooth such that when pressed against the surface region, the suction cup reliably forms a seal around the space between the suction cup and the surface region to maintain a negative fluid pressure in that space and create a sufficient level of adhesion between the suction cup and the surface region. In such an instance, the computing system 1100 may determine that the surface region (e.g., 6810) is suitable for use as a grip region.

[0048] In some implementations, the smoothness of a surface region or surface may involve the absence of physical features such as wrinkles, ridges, ridges, creases, or depressions that protrude from or sink into the surface region or surface. More specifically, the presence of such physical features may reduce the maximum possible contact area that can be made between the end effector device and the surface region, or more specifically, may interfere with the ability of the suction cup or other components of the end effector device to form a seal around the space between the suction cup and the surface region. Thus, a surface region (e.g., 6810) that meets a smoothness condition may generally be free or substantially free of such physical features.

[0049] In an embodiment, the defined smoothness condition may be defined by one or more criteria, such as an orientation similarity criterion and / or a depth similarity criterion, which are discussed in more detail below. The orientation similarity criterion may involve small patches forming surface regions with sufficiently similar orientations, while the depth similarity criterion may involve small patches that are sufficiently close to each other along the depth dimension, or more specifically, along an axis parallel to the normal vector of one of the patches. As mentioned above, the rules or other information describing the orientation similarity criterion and / or the depth similarity criterion may be predefined or dynamically defined and may be stored in the non-transitory computer-readable medium 1120.

[0050] In some implementations, the defined smoothness condition may be satisfied by a smooth, flat surface region, as well as by a surface region that is smooth and has some curvature. For example, if the computing system 1100 divides the surface region into smaller patches, the orientation similarity criterion in such implementations may be satisfied by patches that have small differences in orientation. Such small differences may account for surface regions with gentle curvatures. Such implementations may reflect an assessment that a surface region that is smooth but has gentle curvatures may still be able to tolerate a large amount of contact between itself and the suction cup or some other component of the end effector device (e.g., 3330), and thus still be able to tolerate a robust amount of adhesion that is reliably formed between the surface region and the suction cup.

[0051] In some implementations, the surface area that meets the defined smoothness condition may have to be both smooth and flat or substantially flat. For example, the orientation similarity criteria in such implementations may be stricter than the examples discussed above involving smooth surfaces with gentle curvature. In these implementations, the patches may have to be much more similar in orientation to meet the defined orientation similarity criteria. As a result, the surface area formed by the patch may have a much smaller level of curvature or no curvature. Such implementations may reflect an appreciation that the surface area must be both smooth and flat or substantially flat to ensure that it can form a robust amount of adhesion between itself and some other component of the suction cup or end effector device (e.g., 3330). Thus, the level of stringency of the orientation similarity criteria may be used to control how flat the surface area identified as a grip area is.

[0052] In an embodiment, the computing system 1100 can determine whether a surface area (e.g., 6810) is equal to or greater than a defined area size threshold as a precondition for identifying a surface area (e.g., 6810 / 6812 in FIG. 6 and FIG. 9B) as a grip area. Such a precondition may ensure that the surface area (e.g., 6810 / 6812) is not only sufficiently smooth but also large enough to be grasped by a robot (e.g., 3300) or receive some other interaction from the robot (e.g., 3300). In some instances, the defined area size threshold may be or may indicate a surface area threshold, such that the computing system 1100 determines whether the total surface area of ​​the surface area (e.g., 6810 / 6812) to be identified as a grip area is equal to or greater than the surface area threshold. The defined area size threshold may be, for example, predefined or dynamically defined and stored in the non-transitory computer-readable medium 1120. In some instances, the defined region size threshold may indicate one or more values ​​for one or more respective dimensions, such as radius, length, and / or width. More specifically, the defined region size threshold may define a contact area or other interaction area for interaction between the object surface and the robot. The contact area may have one or more values ​​for one or more respective dimensions. In such instances, the computing system 1100 may determine whether such contact area falls within the surface area (e.g., 6810).

[0053] More specifically, the defined area size threshold may in some scenarios be a defined contact area size associated with contact between an end effector device (e.g., 3330) and an object (or more specifically, a flexible object). For example, if surface area 6810 of FIG. 6 is a portion of object 35201 of FIGS. 3A-3C, the defined area size threshold may be a contact area size associated with contact between one or more components of the end effector device (e.g., suction cups 33321-3332). n35201). The defined contact area size may be a size of a contact area for one or more components that are pressed against or otherwise in contact with the object (e.g., 35201). As an example, FIG. 7A illustrates one suction cup (e.g., 3332) of one or more suction cups. n 35 depicts a contact area 7334 where the suction cup 3332 contacts the surface area 6810 of the object 35201. In one such example, the defined contact area size may be determined by the suction cup 3332, such that a threshold for the defined area size may be a defined suction cup size (e.g., a radius of the suction cup). n FIG. 7B illustrates a configuration of multiple suction cups (e.g., 33321 and 3332). n 7B depicts another contact area 7335 where the suction cup 35201 contacts the surface area 6810 of the object 35201. In the example of FIG. 7B, the defined area size threshold may be the size of the contact area 7335. If the multiple suction cups form an array of suction cups, the defined area size threshold may be the defined size of the array (also referred to as the defined array size). The computing system 1100 may compare the size of the surface area (e.g., 6810) to the defined area size threshold to more ensure that if the surface area is identified as a grip area, the grip area is large enough to fit into the contact areas (e.g., 7334, 7335) having the defined contact area size. In other words, a surface area (e.g., 6810) that meets the defined area size threshold may be large enough to receive contact, or more generally, may be large enough to be engaged by one or more components of an end effector device (e.g., 3330) used to grip the object (e.g., 35201) at the surface area. The defined sizes discussed above may be, for example, predefined or dynamically defined and may be stored in the non-transitory computer readable medium 1120.

[0054] In some scenarios, the defined region size threshold may be a defined object size associated with the object (e.g., 35201). In such scenarios, the defined object size may, for example, describe a boundary or edge of the object. The computing system 1100 may then determine whether the surface region that meets the defined smoothness condition is still large enough to fit into the object or region having such a boundary or edge. The use of such a defined region size threshold may enable the computing system 1100 to more rigorously check whether the surface region (e.g., 6810) belongs to or is otherwise associated with the object (e.g., 35201) to be grasped and moved by the robot (e.g., 3330).

[0055] In an embodiment, the computing system 1100 may identify smooth surface regions, or more specifically, surface regions that satisfy a defined smoothness condition, by dividing the object surface into a plurality of two-dimensional (2D) patches and by determining whether the plurality of 2D patches includes a set of contiguous 2D patches that form a sufficiently smooth surface region. For example, FIG. 8A illustrates a surface represented by 3D image information divided into a plurality of 2D patches 87401-8740. n 5A depicts an example of dividing the plurality of 2D patches into 2D planes (not necessarily drawn to scale). Each 2D patch of the plurality of 2D patches may be a virtual 2D patch, also referred to as a virtual 2D plane, formed by a group of locations that are part of the plurality of locations represented by the image information received in step 4002. More specifically, the image information represents the plurality of locations 57001-5700 discussed with respect to FIG. 5A. n The image information may describe depth information for multiple locations within the camera field of view, such as 87401-87402. The image information may include, for example, a depth map or a point cloud having a grid of pixels (also called an array of pixels). In this example, the computing system 1100 computes multiple 2D patches 87401-87402. nThe grip region may be identified by determining the smoothness of the plurality of 2D patches and by identifying a set of contiguous 2D patches from the plurality of 2D patches that satisfy the defined smoothness condition. More specifically, the plurality of 2D patches 87401 to 8740 may include a plurality of 2D patches 87402 to 87403. n Object 35201-3520 n For example, the 3D image information 5700 may include an image portion 57201 that represents an object surface of the object 35201, and a plurality of patches 87401-8740. n may include sections of patches that also represent the object surface of object 35201. FIG. 8B illustrates patches, i.e., patch 8740, that may be formed based on image portion 57201. q ~8740 r Therefore, patch 8740 q ~8740 r The section may represent the object surface of the object 35201.

[0056] In an embodiment, a plurality of 2D patches 87401 to 8740 n Multiple positions 57001 to 5700 n That is, each 2D patch may be fitted or otherwise approximated to a respective subset of positions, each of which may be a plurality of positions 57001-57002. n As an example, FIG. 8B illustrates a computing system 1100 generating a plurality of 2D patches 87401-8740. n Part of, or more specifically patch 8740 q ~8740 r It may be part of a section of 2D patch 8740 p ~8740 p+5 In this example, the 2D patches are located at positions 57001-5700. n 8B , which may be part of k ~5700 k+116etc., may be determined or identified based on the location described by the image information received in step 4002.

[0057] As mentioned above, position 5700 k ~5700 k+116 Multiple locations including 57001 to 5700 n may be organized into a grid of locations having rows and columns. For example, FIG. 8C shows location 5700 [kx,ky] ~5700 [kx+12,ky+8] 5700 positions organized as a grid k ~5700 k+116 In one such embodiment, position 5700 [kx,ky] ~5700 [kx+12,ky+8] The grid of pixels may be represented by a corresponding grid of pixels, such as a 13×9 pixel grid, which may be part of the image information received in step 4002. For example, the image information may include a grid of pixels at location 5700. [kx,ky] ~5700 [kx+12,ky+8] More specifically, the image information may be a depth map or point cloud including pixels describing the depth values ​​of each of the positions 5700, 5701, 5702, 5703, 5704, 5705, 5706, 5707, 5708, 5709, 5800, 5810, 5811, 5812, 5813, 5814, 5815, 5816, 5817, 5818, 5819, 5820, 5821, 5822, 5823, 5824, 5825, 5826, 5827, 5828, 5829, 5830, 5831, 5832, 5833, 5834, 5835, 5836, 5 [kx,ky] ~5700 [kx+12,ky+8] In one such example, the 2D patch 8740 may include a point cloud including respective [XYZ] coordinates for each position of the 2D patch 8740. p ~8740 p+5 The sections are located at positions 5700 and [kx,ky] ~5700 [kx+12,ky+8] Each subset may be fitted or otherwise approximated to a respective subset of, for example, a 5×5 grid of locations corresponding to 5×5 pixels of image information. Each pixel in this example may include the [XYZ] coordinates of the corresponding location represented by the pixel.

[0058] As an example, the computing system 1100 may p , position 5700 [kx,ky] ~5700 [kx+4,ky+4] or, more specifically, through the subset containing corner position 5700 [kx,ky] , 5700[kx+4,ky] , 5700 [kx,ky+4] , and 5700 [kx+4,ky+4] The 5×5 grid locations may correspond to a 5×5 grid of respective pixels of image information, which may describe their respective coordinates (e.g., [XYZ] coordinates) relative to the grid of locations. In one such example, the computing system 1100 may determine or identify the 2D patch 8740 as a virtual 2D plane that fits through the respective coordinates by performing a plane fitting operation to determine a virtual 2D plane that fits through the respective coordinates. p The plane fitting operation may involve identifying a virtual 2D plane that has a sufficiently low amount of deviation from the respective coordinates, such as an amount of deviation below a defined deviation threshold. In some instances, the 2D patch 8740 p The plane fitting operation for may involve identifying the best fitting virtual 2D plane through each of the coordinates discussed above.

[0059] In an embodiment, a plurality of 2D patches (e.g., 57001 to 5700 n) may facilitate the ability of the computing system 1100 to evaluate the smoothness of the object surface in a manner that is robust or tolerant to noise. More specifically, the image information received in step 4002 may, in some instances, include orientation information for each individual pixel of the image information. The orientation at a pixel may describe the orientation associated with the location represented by the pixel. Although the computing system 1100 may attempt to use the orientation information associated with the individual pixels to evaluate the smoothness of the object surface, such orientation information may include errors caused by imaging noise or some other error source. To correct for such error sources, the computing system 1100 may determine 2D patches, each based on a plurality of pixels, or more specifically, a grid of pixels (e.g., a 5×5 or 10×10 grid of pixels), and use the orientation associated with each 2D patch to evaluate the smoothness of the object surface. Because each of the 2D patches is based on a plurality of pixels, errors in any one of the pixels may be averaged or otherwise mitigated by the information of the other pixels. Therefore, orientation information derived from 2D patches may provide a more accurate or reliable way to assess smoothness across an object surface.

[0060] In an embodiment, a plurality of 2D patches (e.g., 57001 to 5700 n) may each have a sufficiently small size that allows the multiple 2D patches to represent one or more object surfaces with a high degree of granularity or detail. This degree of granularity may enable the computing system 1100 to use the multiple 2D patches to detect even small physical features that do not result in smoothness on the object surface. For example, each of the 2D patches may have a size of 5×5 pixels. In other words, each of the 2D patches may cover an area represented by a 5×5 grid of pixels of image information. The computing system 1100 may use the multiple 2D patches of this example to detect wrinkles, ridges, creases, depressions, or other physical features that reduce the smoothness of the surface, even if such physical features are relatively small. For example, the multiple 2D patches may be able to detect wrinkles, or other physical features with a resolution of as little as tens of pixels.

[0061] As described above, the computing system 1100, in an embodiment, generates a plurality of 2D patches (e.g., 87401 to 8740 n ) to identify a surface region that satisfies the defined smoothness condition by identifying a set of contiguous 2D patches that satisfy the defined orientation similarity criterion and the defined depth similarity criterion. In such an embodiment, the defined orientation similarity criterion and / or the defined depth similarity criterion may be part of the defined smoothness condition, such that satisfying the defined smoothness condition includes satisfying the defined depth similarity criterion and / or the defined orientation similarity criterion. More specifically, a set of contiguous 2D patches in this embodiment may satisfy the defined smoothness condition if they satisfy the defined orientation similarity criterion and the depth similarity criterion. As an example, FIG. 8D illustrates a set of contiguous 2D patches 8745 that satisfy the depth similarity criterion and the defined depth similarity criterion. The set of contiguous 2D patches 8745 in FIG. 8D may be, for example, a set of 2D patches 8740 in FIG. 8C . p ~8740 p+5As discussed in more detail below, a set of contiguous 2D patches (e.g., 8745) may be used to form or define a surface area (e.g., 6810) that meets a defined smoothness condition and thus can be used to form the grip area discussed above.

[0062] In an embodiment, the set of 2D patches may define a spatially contiguous local region, where no portion of the local region is completely separated from any other portion of the local region. A set of 2D patches may be contiguous if each 2D patch of the set of 2D patches is directly or indirectly connected to every other 2D patch in the set of 2D patches. In this embodiment, a first 2D patch and a second 2D patch may be directly connected and contiguous to each other if they share a common border, have some overlap, or are immediately adjacent. More specifically, a first 2D patch and a second 2D patch may be directly connected and contiguous to each other if they share a location that was used to perform the plane fitting to determine the 2D patches. For example, 2D patch 8740 in FIG. 8C p and 2D patch 8740 p+1 2D patch 8740 p and 2D patch 8740 p+1 In other words, the 2D patch 8740 shares five common locations that are used to determine both the p 8740 Positions and 2D patches used to determine p+1 There is an overlap with the positions used to determine 2D patch 8740. p and 2D patch 8740 p+1 may be considered to be directly connected and contiguous to one another.

[0063] Furthermore, a first 2D patch and a second 2D patch may be considered to be indirectly connected if there is an intermediate patch or patches in the set of 2D patches that connect the first 2D patch and the second 2D patch. For example, 2D patch 8740 in FIG. 8C p+12D patch 8740 p and 2D patch 8740 p+2 As another example, the 2D patch 8740 may be an intermediate patch that indirectly connects the p+3 and 8740 p+4 2D patch 8740 p and 2D patch 8740 p+5 In one embodiment, the intermediate patches may be contiguous themselves, e.g., sharing one boundary with the first 2D patch and another boundary with the second 2D patch. More specifically, one of the intermediate patches may be directly connected to the first 2D patch and another of the intermediate patches may be directly connected to the second 2D patch.

[0064] In an embodiment, the orientation of a set of 2D patches may be defined by the normal vectors of each of those 2D patches, and the orientation similarity criterion may involve the normal vectors being sufficiently similar to one another. For example, FIG. 8E illustrates a set of 2D patches 8740, 8750, and 8760, respectively. p ~8740 p+5 Normal vector for 8750 p ~8750 p+5 , where each normal vector may be a vector perpendicular to the respective 2D patch. In such an embodiment, the first 2D patch (e.g., 8740 p ) and a second 2D patch (e.g., 8740 p+1 ) may involve their normal vectors being sufficiently similar to each other. For example, the computing system may p ), the first normal vector (e.g., 8750 p ) may be determined, and a second 2D patch (e.g., 8740 p+1 ), a second normal vector that can be a vector perpendicular to the p+1 ) may be determined. The computing system 1100 of this embodiment may determine a first normal vector (e.g., 8750 p ) and a second normal vector (e.g., 8750p+1 ) satisfies a defined orientation similarity criterion, the first 2D patch (e.g., 8740 p ) and the second 2D patch (8740 p+1 ) may satisfy a defined orientation similarity criterion. The first normal vector and the second normal vector may satisfy the defined orientation similarity criterion, for example, when the angle between the first normal vector and the second normal vector is below a defined angle threshold and / or when the dot product between the first normal vector and the second normal vector exceeds a defined dot product value threshold. In the example of FIG. 8D , the computing system 1100 may determine whether the defined orientation similarity criterion is satisfied by the 2D patch 8740. p ~8740 p+5 For example, the computing system 1100 may determine whether the 2D patch 8740 is satisfied for a larger group of 2D patches such as 8741 if, for each of the 2D patches, the normal vector of the 2D patch is within a defined angle threshold relative to the normal vectors of its immediately adjacent neighboring patches. p ~8740 p+5 It may be determined that the orientation similarity criteria defined for is satisfied.

[0065] In an embodiment, the defined depth similarity criterion may involve whether one or more distances between the 2D patches are sufficiently small, as measured along a depth dimension. For example, the computing system 1100 may determine whether the defined depth similarity criterion is a first 2D patch (e.g., 8740 in FIG. 8F and FIG. 8G ) by determining whether a first distance between the first 2D patch and the second 2D patch satisfies the defined depth similarity criterion and / or whether a second distance therebetween satisfies the defined depth similarity criterion. p ) and a second 2D patch (e.g., 8740 p+18F ) is satisfied. More specifically, the computing system 1100 may determine that the depth similarity criterion is satisfied for the first 2D patch and the second 2D patch if the first distance and / or the second distance is below a defined distance threshold. The first distance, such as distance d1 in FIG. 8F , may be determined to be satisfied for the first 2D patch (e.g., 8740 p ) to the first 2D patch (e.g., 8740 p ), which may be a vector perpendicular to the p ) along axis A perpendicular to the second 2D patch (e.g., 8740 p+1 ) The second distance, such as distance d2 in FIG. 8F, may be the distance to the first 2D patch (e.g., 8740 p ) to a second 2D patch (e.g., 8740 p+1 ), which may be a vector perpendicular to the p+1 ) along axis B perpendicular to the second 2D patch (e.g., 8740 p+1 In an embodiment, the computing system 1100 may classify a 2D patch 8740 as a 2D patch if, for each 2D patch, one or more distances between the 2D patch and its immediately adjacent neighboring 2D patches are less than a defined distance threshold. p ~8740 p+5 It may be determined that a group of 2D patches such as meets a defined depth similarity criterion. In this embodiment, the one or more distances may include the first distance and the second distance discussed above, which are measured along the normal vectors of each of the two patches.

[0066] As described above, the computational system may, in some implementations, determine a set of contiguous 2D patches that meet a defined orientation similarity criterion and a defined depth similarity criterion to determine a surface region (e.g., 6810 in FIG. 6 and FIG. 9A or 6812 in FIG. 9B) that meets a defined smoothness condition. More specifically, the surface region may be used as a grip region, and may be formed by or fit within a local area defined by the set of contiguous 2D patches.

[0067] In an embodiment, the computing system 1100 may identify an initial set of one or more 2D patches and identify a set of continuous 2D patches (e.g., 8745) by expanding the initial set of one or more 2D patches into a set of continuous 2D patches. As an example, the computing system 1100 may identify 2D patch 8740 of FIG. 8B, FIG. 8C, and FIG. 8E. p In this example, the initial set of single 2D patches can be formed using the single 2D patch 8740. p may be used as a seed that is expanded into a set of contiguous patches 8745 in FIG. 8D.

[0068] In an embodiment, the computing system 1100 may include a first set of one or more 2D patches that are deemed by the computing system 1100 to be, or represent, the smoothest location or group of locations over a particular region, such as 2D patches (e.g., 8740 p ), in one embodiment, the particular region may be a local region encompassing the 2D patch, such as a particular quadrant of the scene represented by the image information of step 4002. In one embodiment, the particular region may be the entire scene represented by the image information. In this embodiment, the 2D patches included in the initial set of one or more 2D patches are located at multiple locations (e.g., 57001-57002) represented by the image information. n ) may represent the smoothest position or group of positions. In this example, the group of positions is the corner position in FIG. 8C (corner position 5700 [kx,ky] , 5700 [kx+4,ky], 5700 [kx,ky+4] , and 5700 [kx+4,ky+4] ) may be a cluster of locations, such as a 5×5 grid of locations bounded by

[0069] In an embodiment, the computing system 1100 may determine a plurality of smoothness scores indicative of respective degrees of smoothness across one or more surfaces represented by the image information, which may include one or more object surfaces (e.g., respective object surfaces of objects 35201, 35202, etc.). More specifically, the plurality of smoothness scores may be determined across a plurality of locations represented by the image information (e.g., 57001-57002). n ) may indicate the degree of smoothness of each of the smoothness scores across the entire scene represented by the image information. As described above, the computing system 1100 may include in the initial set of one or more 2D patches a 2D patch that represents the smoothest location or group of locations across a local region or across the entire scene represented by the image information. In this embodiment, the 2D patch may represent the location having the highest smoothness score among the multiple smoothness scores. In some instances, the multiple smoothness scores may form a smoothness map that measures the smoothness across the scene represented by the image information. In such instances, the 2D patch included in the initial set may have the highest smoothness score across the entire smoothness map or across a local region of the smoothness map (e.g., a local quadrant of the smoothness map).

[0070] As described above, the computing system 1100 may attempt to expand an initial set of one or more 2D patches into a set of contiguous 2D patches. The initial set may be, for example, a set of single 2D patches (e.g., 8740 p ) or multiple 2D patches (e.g., 8740 p and 8740 p+1), which may act as a seed to be expanded into the set of continuous 2D patches. More specifically, the initial set may be an initial version of the set of continuous 2D patches. The computing system 1100 may add one or more 2D patches to this initial version of the set of continuous 2D patches to generate an updated version. In the example of FIG. 8D , the set of continuous patches 8745 may be, for example, a final version of the set of continuous 2D patches. The final version may be, for example, a maximum possible set of continuous 2D patches that the computing system 1100 can identify that satisfies a defined smoothness condition, i.e., more specifically, a defined orientation similarity criterion and a defined depth similarity criterion, as well as being expanded from the initial set of one or more 2D patches (e.g., expanded from 8740p1). In this example, the computing system 1100 may generate an updated version of the set of continuous 2D patches (e.g., 8740p2). 1~ 8740 n ), if it is unable to identify any additional 2D patches, it may stop attempting to extend the set of contiguous 2D patches that are contiguous with existing 2D patches in the set and that satisfy orientation and depth similarity criteria to the existing 2D patches.

[0071] As an example of an extension, the computing system 1100 may search for a 2D patch that (i) is not currently in the set of contiguous 2D patches, or more specifically, is not in the current version of the set of contiguous 2D patches, (ii) is contiguous with at least one 2D patch in the current version of the set of contiguous 2D patches, and (iii) satisfies at least one 2D-defined orientation similarity criterion and a defined depth similarity criterion. If the computing system 1100 can identify such a 2D patch that satisfies the above criteria, it can add the 2D patch to the current version of the set of contiguous 2D patches to generate an updated version of the set. For example, the initial set of one or more 2D patches may be generated by adding the 2D patch to the current version of the set of contiguous 2D patches discussed above (e.g., 87401-87402). n ) the first 2D patch (e.g., 8740 in FIG. 8Cp ), the computing system 1100 may use the initial set as an initial version of the set of contiguous 2D patches. The computing system 1100 may select a second 2D patch (e.g., 8740 ) from among the multiple 2D patches that is contiguous with the first 2D patch and is not currently in the initial set. p+1 or 8740 p+3) The computing system may extend this initial version by identifying a first 2D patch (e.g., 8740 p ) and a second 2D patch (e.g., 8740 p+1 or 8740 p+3 ) may further determine whether the first set of 2D patches satisfies a defined orientation similarity criterion and a defined depth similarity criterion. In response to determining that the above criteria are met, the computing system 1100 may add a second 2D patch to the initial set to generate an extended set of 2D patches. The extended set may be an updated version of the set of continuous 2D patches. In some instances, the extended set may form a final version of the set of continuous 2D patches, such as the set of continuous 2D patches 8745 of FIG. 8D. In some instances, the extended set may be an intermediate version of the set of continuous 2D patches, and the intermediate version may form part of the final version of the set of continuous 2D patches.

[0072] In an embodiment, a first 2D patch (e.g., 8740 p ) and a second 2D patch (e.g., 8740 p+1 ) satisfies the defined orientation similarity criterion by judging the normal vector of the first 2D patch (e.g., 8750 p ) and the normal vector of the second 2D patch (e.g. 8750 p+1 ) and determining whether the two normal vectors meet a defined orientation similarity criterion, as discussed above with respect to FIG. 8E. In an embodiment, the first 2D patch (e.g., 8740 p ) and a second 2D patch (e.g., 8740 p+1Determining whether the 2D patches (d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15, d16, d17, d18, d19, d20, d21, d22, d23, d24, d25, d26, d27, d28, d29, d30, d31, d32, d33, d34, d35, d36, d37, d38, d39, d40, d41, d42, d43, d44, d45, d46, d47, d48, d49, d50, d51, d52, d53, d54, d55, d56, d57, d58, d59, d60, d61, d62, d63, d64, d65, d66, d67, d68, d69, d70, d71, d72, d73, d74, d75, d76, d77, d78, d79, d80, d71, d72, d74, d75, d79, d75, d79, d79, d71, d72, d73, d74, d75, d75, d79, d74, d75, d79, d79, d79, d79, d79,

[0073] In an embodiment, if the expanded set discussed above is an intermediate version of a set of consecutive 2D patches, the computing system 1100 may attempt to update the expanded set. The expansion may involve updating at least one 2D patch (e.g., 8740 p+1 ) and satisfy the defined orientation similarity criterion and the defined depth similarity criterion with respect to at least one of the 2D patches. p+2 ) The search may be performed by searching multiple 2D patches (e.g., 87401 to 8740 n ). The computing system 1100 may update the expanded set by adding one or more remaining 2D patches to the expanded set.

[0074] In an embodiment, the computing system 1100 may use a set of continuous 2D patches to form or identify a surface area that may be used as a grip area. In other words, a surface area identified as a grip area may be formed by and / or fit within a set of continuous 2D patches. More specifically, a set of continuous 2D patches may satisfy a defined smoothness condition because the 2D patches in the set may satisfy a defined orientation similarity criterion and a defined depth similarity criterion. The computing system 1100 may define a grip area as a surface area that fits within a boundary of the set of continuous 2D patches. In some instances, the surface area may have the same shape as the set of continuous 2D patches. In some instances, the surface area may have a defined area shape (e.g., a rectangular shape, a circular shape, or an elliptical shape) that fits within the set of continuous 2D patches. As an example, FIG. 9A depicts an implementation in which the computing system 1100 identifies a grip area as a surface area 6810 that fits within a set of continuous 2D patches 8745. In this example, the computing system 1100 may define the surface region 6810 to have a boundary that is identical or substantially identical to the boundary 8746 formed by the set of contiguous 2D patches 8745. Figure 9B illustrates an example in which the computing system 1100 identifies the grip region as a surface region 6812 having a defined shape. More specifically, the surface region 6812 may be a rectangular region that fits within the boundary 8746 of the set of contiguous 2D patches 8745.

[0075] As described above, surface area 6810 or surface area 6812 may be identified as a grip area, which may be a detection result output by a grip area detection operation, or more specifically, by grip area detection module 1127 of Figure 2C. The detection result may be provided, for example, as an input to a motion planning operation, which may be performed, for example, by motion planning module 1129.

[0076] Returning to FIG. 4, the method 4000 may include step 4006 in which, in an embodiment, the computing system 1100 may identify a safe area (also referred to as a safety volume), which may be a three-dimensional (3D) area surrounding the grip area in one or more horizontal dimensions. The one or more horizontal dimensions (e.g., the X dimension and / or the Y dimension) may be parallel to the ground and / or perpendicular to the depth dimension, which may be a vertical dimension parallel to the gravity axis, for example. For example, FIGS. 10, 11A, and 11B show a 3D area 6820 that the computing system may identify as a safe area. The 3D area 6820 may extend, for example, in the X dimension and the Y dimension. As shown in FIG. 11A, the 3D area 6820 or other safe area may surround a grip area, such as the surface area 6810, in two horizontal dimensions. In some implementations, the safe area may extend along a vertical dimension perpendicular to the one or more horizontal dimensions. For example, the 3D region 6820 that forms the safety area may be the same as that of FIG. 12A and FIG. 12B (object 3520 n-5 and 3520 n-4 68. The safe area may be a rectangular area extending downward from the surface area 6810 along a vertical dimension (e.g., the Z dimension), as shown in FIG. 68. In some instances, the safe area may be defined to completely encompass the surface area 6810, such that no portion of the surface area 6810 extends beyond the top surface or any other surface of the safe area in the Z dimension. In some instances, the surface area 6810 may have one or more portions that protrude beyond the surface of the safe area.

[0077] As discussed in more detail below, the safety area, which may also be an output of the grip area detection operation, may be used during motion planning, or more specifically, during robot motion planning, to grasp an object (e.g., 35201) associated with the grip area. More specifically, the use of the safety area may reduce the likelihood of a collision between the object and another object or structure in the robot's environment. In some instances, the safety area may provide a safety margin around the object such that the computing system 1100 takes the safety margin into account when planning a movement to grasp and move the object (e.g., 35201). For example, the motion planning may involve determining a movement path that the end effector device (e.g., 3330) will follow after grasping the object with the grip area, and the end effector device causes the grip area and the object to substantially follow the same movement path. The safety area surrounding the grip area may be large enough to further surround the object in one or more horizontal dimensions in this example to provide a safety margin. The computing system 1100 may determine a travel path to avoid collisions between a safety region (surrounding the grip region) and structures such as container walls, electrical wiring, and other objects in the robot environment. Such a form of motion planning may move the end effector apparatus in a more conservative manner, such as, for example, by keeping a greater distance between the object and other objects or structures that may be near the object's travel path, thereby further reducing the likelihood of collisions between those other objects or structures.

[0078] As discussed above, the safety area surrounding the grip area may, in embodiments, be large enough to further surround the object associated with the grip area in one or more horizontal dimensions. More specifically, the safety area may be large enough to extend from the grip area beyond the boundary of the object, such that the boundary of the object is encompassed by the boundary of the safety area in one or more horizontal dimensions. In embodiments, the boundary of the safety area may include one or more edges, and the computing system 1100 may determine at least one edge of the safety area based on a defined lateral dimension size. The defined lateral dimension size may be a defined size of the lateral dimension of the object (also referred to as the object lateral dimension). The lateral dimension of the object may be a dimension perpendicular to the thickness dimension of the object. For example, a rectangular shaped object may have a lateral dimension that includes a length dimension and a width dimension, while a disk shaped object may have a lateral dimension that is a radial dimension. As an example, FIG. 11B illustrates a first defined lateral dimension size, which may be a defined size of the length dimension for an object (e.g., 35201) or group of objects, a second defined lateral dimension size, which may be a defined size of the length dimension for the object (e.g., 35201) or group of objects, a third defined lateral dimension size, which may be a defined size of the length dimension for the object (e.g., 35201) or group of objects, a fourth defined lateral dimension size, which may be a defined size of the length dimension for the object (e.g., 35201) or group of objects, a fifth ... sixth defined lateral dimension size, which may be a defined size of the length dimension for the object (e.g., 35201) or group of objects, a seventh defined lateral dimension 3520 FIG. 11B depicts a second defined lateral dimension size, width, which may be a defined size of a width dimension for an object or group of objects. 3520 The defined size may be, for example, predefined and stored in the non-transitory computer-readable medium 1120.

[0079] As discussed above, the defined lateral dimension size may be associated with an object or group of objects. More specifically, the computing system 1100, in an embodiment, may be configured to allow the robot (e.g., 3300) to detect and determine whether objects (e.g., 35201-35202) having the same or substantially the same size are in a single lateral dimension. n), or is expected to grab only a single object (e.g., 35201). For example, the robot may be expected to pick up a package of clothing from a bin that contains only a group of identical clothing packages. The group of packages may be associated with the same stock keeping unit (SKU) number, for example.

[0080] In the above embodiment, the computing system 1100 may determine the edge of the safety area using a defined lateral dimension size, such as a defined object length or a defined object width, associated with a group of packages, or more generally, a group of objects. More specifically, the group of objects may be a group of objects having a length of FIG. 3520 and width 3520 The computing system 1100 may expect that an object associated with a grip region (e.g., surface region 6810 / 6812), such as object 35201, has edges that do not extend beyond the defined lateral dimension size from the grip region. More specifically, object 35201 associated with surface region 6810 in FIG. 11B has edges that are 100 mm long and 120 mm long. 3520 Extends from surface area 6810 or other grip area along a first dimension (e.g., length dimension) a distance of 3520 The object surface may have a length extending along a second dimension (e.g., width dimension) of: 3520 and at least another edge (e.g., 6822 / 6823) of the safety area 6820 is set to be located beyond the surface area 6810 by a distance equal to 3520 If the safety area 6820 is set to lie beyond the surface area 6810 by a distance equal to 35201, then such a safety area 6820 may likely surround the object (e.g., 35201) associated with the surface area 6810 (which in this example is the grip area). Thus, in this example, the computing system may determine the edges of the safety area based on a defined lateral dimension size associated with the lateral dimension of the object.

[0081] In an embodiment, the edges of the safety area may be determined based on a maximum lateral dimension size for a group of objects. Such an embodiment may involve scenarios in which the robot is expected to grip and move a group of objects having different respective lateral dimension sizes. For example, the robot may be expected to pick up a package of clothes from a bin, which may include a group of packages associated with different SKU numbers and having different object lengths and / or different object widths. In such an embodiment, the computing system 1100 may determine the edges of the safety area using a maximum lateral dimension size, such as a defined maximum object length or a defined maximum object width, between the group of objects. For example, the computing system may determine the safety area to have one edge or a pair of edges that are located beyond the grip area by a distance equal to a first defined maximum lateral dimension size (e.g., a defined maximum object length) and may determine the safety area to have another edge or a pair of edges that are located beyond the grip area by a distance equal to a second defined maximum lateral dimension size (e.g., a defined maximum object width).

[0082] In an embodiment, the computing system 1100 may determine an edge of the safe region that is located at a container wall of a container in which an object is placed. For example, the image information 5700 (e.g., point cloud) of FIG. 11A may describe depth information of locations on one or more surfaces within a container, such as the container 3510 of FIGS. 3A-3C. In such an instance, the computing system 1100 may expect that an object (e.g., 35201) placed within the container will not extend through the container wall, which may form the edge of the container. Thus, the safe region may not need to extend beyond the container wall to still encompass the object (e.g., 35201) within the container (e.g., 3510). Thus, as shown in FIG. 11B, the computing system 1100 may determine that an edge 6824 of the 3D region 6820 (which in this example is the safe region) is at a location associated with the container wall of the container 3510. As shown in FIG. 11B, the location of the edge 6824 is determined based on a defined lateral dimension size (e.g., length 3520 ) from the grip area (e.g., surface area 6810). In this example, the computing system 1100 may detect the container wall, for example, by detecting an abrupt change in the depth of the image information 5700 or by detecting an edge of the image information 5700. Thus, in the embodiment of FIG. 11B, the computing system may detect the container wall by detecting (i) a location associated with the container wall, such as a location defining edge 6824, or (ii) a lateral dimension size (e.g., length or width dimension) defined along a horizontal dimension (e.g., length or width dimension), such as a location defining edge 6821 / 6822 / 6823. 3520 , or width 3520 ), a safety area (e.g., 3D area 6820) may be identified as an area extending in the horizontal dimension from the grip area to at least one of the extending locations.

[0083] In an embodiment, the computing system 1100 calculates the height in FIG. 6820 The height of the safe area (also referred to as the safe area height) can be determined based on a defined lateral dimension size (e.g., length) associated with the object. 3520), or the largest lateral dimension associated with a group of objects. 6820 A, length 3520 In such an embodiment, the object associated with the defined lateral dimension size (e.g., 35201) may be a flexible object that sags when being lifted. For example, FIG. 12B illustrates the force exerted by the end effector device 3330, or more specifically, its suction cups 3332. n 35 shows an object 35201, such as a package of clothes, that sags when lifted by a safety zone. By determining the height of the safe zone in this manner, if a safe zone is used for motion planning, the motion planning can take into account the tendency of the object to sag under its own weight when being lifted. As a result, the motion planning can reduce the likelihood of a collision between the object and another object or structure.

[0084] Returning to FIG. 4, the method 4000 may, in an embodiment, include step 4008, in which the computing system 1100 performs a robot motion plan based on the grip area (e.g., surface area 6810 / 6812) and the safety area (e.g., 3D area 6820). The robot motion plan may be for an end effector device (e.g., 3330) to grasp an object (e.g., 35201) with the grip area. As mentioned above, the object may be a flexible object in some instances. In an embodiment, the robot motion plan may involve generating or otherwise determining a trajectory along which the grip area moves with the end effector device. For example, the computing system 1100 may determine a trajectory in which the end effector device (e.g., 35201) approaches and contacts the grip area to engage the grip area. Engaging the grip area may cause the end effector device to grasp the object (e.g., 35201) with the grip area. The trajectory may further include an end effector device that lifts the object and moves the object to a target location. The robot motion planning may be performed in a manner that avoids contact of the safety region with other objects or structures in the environment of the end effector device so as to reduce the risk of collision between the grasped object and other objects or structures in the environment of the end effector device.

[0085] As an example, when an object (e.g., 35201) is placed in a container (e.g., 3510), the robot motion planning may attempt to avoid collisions between the object (e.g., 35201) and structures such as container walls. In this example, the computing system 1100 may determine whether a safety region, such as the 3D region 6820 surrounding the object 35201, overlaps with a portion of the container 3510 as the safety region follows the grip region (e.g., surface region 6810) along the trajectory as discussed above. As mentioned above, an object may sag when lifted, in some instances, as shown in FIG. 12B. If the end effector device 3330 attempts to move such an object sideways before lifting it to a sufficient height, the sagging portion of the object may collide with the container wall. The safety region may be used to take into account the possibility of such collisions. More specifically, the safety region may be a height (e.g., h ) based on a defined lateral dimension size, as discussed above. 6820 ), in one such embodiment, a trajectory that causes a loose portion of the object to collide with a container wall may also cause the safe region to overlap a portion of the container, such as the container wall. Thus, if the computing system 1100 determines that a trajectory causes a portion of the container to overlap the safe region (e.g., 6820), the computing system 1100 may generate an adjusted trajectory that causes the safe region to avoid the container (e.g., 3510) as it follows the grip region along the adjusted trajectory. As a result, the use of the safe region may reduce the likelihood of a collision between the gripped object (e.g., 35201) and the container (e.g., 3510).

[0086] In an embodiment, one or more of the above steps may be repeated. For example, the computing system 1100 may perform steps 4002-4008 to identify a first grip area and a first safety area to facilitate a robotic movement to lift a first object (e.g., 35201) from the bin and deliver the first object to a destination location. The computing system 1100 of this embodiment may repeat steps 4004-4008 to identify a second grip area and a second safety area to facilitate a robotic movement to lift another object (e.g., 35202) from the bin and deliver the other object to a destination location. In an embodiment, one or more steps may be omitted. For example, the computing system 1100 may perform steps 4002-4006 without performing step 4008, or may perform steps 4002 and 4004 without performing step 4006.

[0087] Additional considerations regarding various embodiments: The first embodiment relates to a computing system and a method performed by the computing system, such as when the computing system executes instructions on a non-transitory computer-readable medium. The computing system may include a communication interface and at least one processing circuit. The communication interface is configured to communicate with a robot having an end effector device and a camera having a camera field of view. The at least one processing circuit is configured to receive image information representing an object surface associated with the flexible object when the flexible object is within the camera field of view, the image information being generated by the camera. The at least one processing circuit is further configured to identify a surface area of ​​the object surface as a grip area based on the image information, the surface area having an area size that satisfies a defined smoothness condition and is equal to or greater than a defined area size threshold, identify a three-dimensional (3D) area that surrounds the grip area in one or more horizontal dimensions and extends from the grip area along a vertical dimension perpendicular to the one or more horizontal dimensions, and perform a robot motion plan for gripping the flexible object with the grip area by the end effector device based on the grip area and the safety area.

[0088] Example 2 includes the computing system of Example 1. In this embodiment, the defined area size threshold is a defined contact area size associated with contact between the end effector device and the flexible object, such that the grip area is large enough to fit a contact area having the defined contact area size.

[0089] Example 3 includes the computing system of Example 2. In this embodiment, the defined region size threshold is a defined suction cup size associated with the end effector suction cup.

[0090] Example 4 includes the computing system of Example 1. In this embodiment, the defined region size threshold is a defined object size associated with the flexible object.

[0091] Example 5 includes the computing system of any one of Examples 1 to 4. In this embodiment, when the flexible object has a thickness dimension and a lateral dimension perpendicular to the thickness dimension, the at least one processing circuit is configured to determine an edge of the safety region based on a defined lateral dimension size associated with the lateral dimension of the flexible object.

[0092] Example 6 includes the computing system of any one of Examples 1 to 5. In this embodiment, when the flexible object is one of a group of flexible objects in the camera field of view, each having a respective lateral dimension size, the at least one processing circuit is configured to determine an edge of the safe region based on a maximum lateral dimension size of the different lateral dimension sizes.

[0093] Example 7 includes the computing system of any one of Examples 1 to 6. In this embodiment, if the flexible object is a sagging object when lifted, the at least one processing circuit is configured to determine a safe zone height based on a defined lateral dimension size associated with the flexible object.

[0094] Example 8 includes the computing system of Example 7. In this embodiment, when the flexible object is placed in a container, the at least one processing circuit is configured to perform the robot motion planning by: generating a trajectory along which a gripping area of ​​the object surface will follow as moved by the end effector device; determining whether the trajectory causes the safety area to overlap a portion of the container when the gripping area follows the trajectory; and, in response to determining that the trajectory causes the safety area to overlap a portion of the container, generating an adjusted trajectory that avoids the safety area overlapping the container when the safety area follows the gripping area along the adjusted trajectory.

[0095] Example 9 includes the computing system of any one of Examples 1 to 8. In this embodiment, when the image information includes a point cloud describing depth information of locations on one or more surfaces within the container, the at least one processing circuitry is configured to identify the safety area as (i) a location associated with a container wall, or (ii) an area extending in one or more horizontal dimensions from the grip area to at least one of the locations extending by a lateral dimension size defined along one of the horizontal dimensions.

[0096] A tenth embodiment includes the computing system of any one of the first to ninth embodiments. In this embodiment, when the image information describes depth information of a plurality of positions within the camera field of view, the at least one processing circuit is configured to identify a grip region by determining a plurality of two-dimensional (2D) patches representing the object surface that approximate a respective subset of the plurality of positions, and identifying, from among the plurality of 2D patches, a set of contiguous 2D patches that satisfy a defined orientation similarity criterion and a defined depth similarity criterion, wherein the surface region identified as the grip region is formed by or fits within the set of contiguous 2D patches, and satisfying the defined smoothness condition for the surface region satisfies the defined orientation similarity criterion and the defined depth similarity criterion for the set of contiguous 2D patches.

[0097] Example 11 includes the computing system of Example 10. In this embodiment, the grip area has a defined area shape and fits within a set of contiguous 2D patches.

[0098] Example 12 includes the computing system of example 10 or 11. In this example, the at least one processing circuit is configured to identify the set of contiguous 2D patches by identifying an initial set of one or more 2D patches and expanding the initial set of one or more 2D patches to a set of contiguous 2D patches.

[0099] Example 13 includes the computing system of Example 12. In this embodiment, the initial set of one or more 2D patches includes a first 2D patch of the plurality of 2D patches, and the at least one processing circuit is configured to: determine a first normal vector that is a vector perpendicular to the first 2D patch; identify a second 2D patch from the plurality of 2D patches that is contiguous with the first 2D patch and is not in the initial set of one or more 2D patches; determine a second normal vector that is a vector perpendicular to the second 2D patch; and determine at least one of: (i) a first distance from the first 2D patch to the second 2D patch along an axis parallel to the first normal vector; or (ii) a second distance from the second 2D patch to the first 2D patch along an axis parallel to the second normal vector; and determine the first normal vector and The method is configured to expand the initial set of one or more 2D patches into a set of contiguous 2D patches by: determining whether the second normal vector satisfies a defined orientation similarity criterion; determining whether at least one of the first distances or the second distances satisfies a defined depth similarity criterion; and in response to a determination that the first normal vector and the second normal vector satisfy the defined orientation similarity criterion and at least one of the first distances or the second distances satisfy the defined depth similarity criterion, adding a second 2D patch to the initial set to generate an expanded set of 2D patches, wherein the expanded set forms a set of contiguous 2D patches or forms a part of the set of contiguous 2D patches.

[0100] Example 14 includes the computing system of Example 13. In this embodiment, the at least one processing circuit is further configured to: search for one or more remaining 2D patches among the 2D patches that are part of the plurality of 2D patches and that are not currently in the expanded set of the 2D patches, the one or more remaining 2D patches being (i) contiguous with a 2D patch currently in the expanded set and (ii) satisfying a defined orientation similarity criterion and a defined depth similarity criterion for the 2D patches in the expanded set; and update the expanded set by adding the one or more remaining 2D patches to the expanded set.

[0101] Example 15 includes the computing system of any one of Examples 12 to 14. In this embodiment, the at least one processing circuit is configured to include in the initial set of one or more 2D patches a 2D patch of the plurality of 2D patches that represents a smoothest location or group of locations among the plurality of locations represented by the image information.

[0102] Example 16 includes the computing system of Example 15. In this embodiment, the at least one processing circuit is configured to determine a plurality of smoothness scores indicative of respective degrees of smoothness across the one or more object surfaces at respective locations represented by the image information, and a 2D patch included in the initial set of one or more 2D patches represents a location having a highest smoothness score among the plurality of smoothness scores.

[0103] It will be apparent to those skilled in the relevant art that other suitable modifications and adaptations to the methods and applications described herein can be made without departing from the scope of any of the embodiments. The above-described embodiments are illustrative examples, and the present invention should not be construed as being limited to these particular embodiments. It should be understood that the various embodiments disclosed herein may be combined in different combinations than those specifically presented in the description and accompanying figures. It should also be understood that, by way of example, certain acts or events of any of the processes or methods described herein may be performed in a different order, or may be added, integrated, or omitted entirely (e.g., not all acts or events described may be necessary to perform a method or process). In addition, although certain features of the embodiments herein are described as being performed by a single component, module, or unit for clarity, it should be understood that the features and functions described herein may be performed by any combination of components, modules, or units. Thus, various changes and modifications may be effected by those skilled in the art without departing from the spirit or scope of the invention, as defined in the appended claims.

Claims

1. a communication interface configured to communicate with a robot having an end effector device; at least one processing circuit; The at least one processing circuit is receiving image information including a representation of an object surface associated with the object; identifying, as grip areas, surface areas of the object surface that satisfy a defined smoothness condition related to surface smoothness and have an area size equal to or greater than a defined area size threshold based on the image information; identifying a three-dimensional (3D) region representing a loose portion of the object outside of the grip area, the edge of the 3D region being determined based on a maximum lateral dimension size of the object; performing a robot motion planning based on the grip area and the 3D area; communicating commands to the robot to cause the robot to perform the planned motion; A computing system configured to:

2. The computing system of claim 1 , wherein the defined area size threshold is a defined contact area size associated with contact between the end effector device and the object, such that the grip area encompasses a contact area having the defined contact area size.

3. The computing system of claim 2 , wherein the defined region size threshold is a defined suction cup size associated with an end effector suction cup.

4. The computing system of claim 1 , wherein the defined region size threshold is a defined object size associated with the object.

5. If the object is one of a group of objects each having a different lateral dimension size, The computing system of claim 1 , wherein the at least one processing circuitry is configured to determine the edges of the 3D region based on the maximum lateral dimension size of the respective different lateral dimension sizes.

6. The computing system of claim 1 , wherein the at least one processing circuitry is configured to determine a 3D region height based on a defined lateral dimension size associated with the slack portion of the object.

7. When the object is placed in a container, The at least one processing circuit is generating a trajectory along which the gripping area of ​​the object surface is moved by the end effector device; determining whether the trajectory causes the 3D region to overlap a portion of the container as the 3D region follows the grip region along the trajectory; generating an adjusted trajectory in response to determining that the trajectory causes the 3D region to overlap a portion of the container, the adjusted trajectory causing the 3D region to avoid overlapping the container when the 3D region is following the grip region along the adjusted trajectory; The computing system of claim 6 , configured to implement the robot motion planning by:

8. if the image information comprises a point cloud describing depth information of positions on one or more surfaces within the container; 2. The computing system of claim 1, wherein the at least one processing circuit is configured to identify the 3D region as a region extending in one or more horizontal dimensions to at least one of: (i) a location associated with a container wall; or (ii) a location extending from the grip region a lateral dimension size defined along one of the one or more horizontal dimensions.

9. Where the image information describes depth information for a plurality of positions, The at least one processing circuit is determining a plurality of two-dimensional (2D) patches approximating a respective subset of the plurality of locations, the plurality of 2D patches representing the object surface; identifying a set of contiguous 2D patches from the plurality of 2D patches that satisfy a defined orientation similarity criterion related to the orientation of the surface and a defined depth similarity criterion related to the depth of the surface; and configured to identify the grip area by the surface area identified as the grip area is formed by or fits within the set of contiguous 2D patches; The computing system of claim 1 , wherein satisfying the defined smoothness condition for the surface region satisfies the defined orientation similarity criterion and the defined depth similarity criterion for the set of contiguous 2D patches.

10. The computing system of claim 9 , wherein the grip area has a defined area shape and fits within the set of contiguous 2D patches.

11. the at least one processing circuit: Identifying an initial set of one or more 2D patches; expanding the initial set of one or more 2D patches into the set of contiguous 2D patches; The computing system of claim 9 , configured to identify the set of contiguous 2D patches by:

12. the initial set of one or more 2D patches includes a first 2D patch of the plurality of 2D patches; the at least one processing circuit: determining a first normal vector, the first normal vector being a vector perpendicular to the first 2D patch; identifying a second 2D patch from the plurality of 2D patches that is contiguous with the first 2D patch and is not in the initial set of one or more 2D patches; determining a second normal vector, the second normal vector being a vector perpendicular to the second 2D patch; determining at least one of: (i) a first distance from the first 2D patch to the second 2D patch along an axis parallel to the first normal vector; or (ii) a second distance from the second 2D patch to the first 2D patch along an axis parallel to the second normal vector; determining whether the first normal vector and the second normal vector satisfy the defined orientation similarity criterion; determining whether at least one of the first distance or the second distance satisfies the defined depth similarity criterion; in response to a determination that the first normal vector and the second normal vector satisfy the defined orientation similarity criterion and a determination that at least one of the first distance or the second distance satisfies the defined depth similarity criterion, adding the second 2D patch to the initial set to generate an extended set of 2D patches, the extended set forming the set of continuous 2D patches or forming a part of the set of continuous 2D patches; 12. The computing system of claim 11, configured to expand the initial set of one or more 2D patches to the set of contiguous 2D patches by:

13. the at least one processing circuit: searching for one or more remaining 2D patches among the 2D patches that are part of the plurality of 2D patches and that are not currently in the expanded set of 2D patches, the one or more remaining 2D patches (i) being contiguous with a 2D patch currently in the expanded set and (ii) satisfying the defined orientation similarity criterion and the defined depth similarity criterion with respect to the 2D patch in the expanded set; updating the expanded set by adding the one or more remaining 2D patches to the expanded set; The computing system of claim 12 , further configured to:

14. 12. The computing system of claim 11, wherein the at least one processing circuit is configured to include in the initial set of one or more 2D patches a 2D patch of the plurality of 2D patches that represents a smoothest location or group of locations from the plurality of locations represented by the image information.

15. the at least one processing circuitry is configured to determine a plurality of smoothness scores indicative of respective degrees of smoothness across one or more object surfaces at respective locations represented by the image information; The computing system of claim 14 , wherein the 2D patch included in the initial set of one or more 2D patches represents a location having a highest smoothness score among the plurality of smoothness scores.

16. A non-transitory computer-readable medium having instructions, The instructions, when executed by a processing circuit of a computing system, cause the processing circuit to: receiving image information by the at least one processing circuit of the computing system, the computing system being configured to communicate with a robot having an end effector device, the image information including a representation of an object surface associated with an object; identifying, as grip areas, surface areas of the object surface that satisfy a defined smoothness condition related to surface smoothness and have an area size equal to or greater than a defined area size threshold based on the image information; identifying a three-dimensional (3D) region representing a loose portion of the object outside of the grip area, the edge of the 3D region being determined based on a maximum lateral dimension size of the object; performing a robot motion planning based on the grip area and the 3D area; and communicating commands to the robot to cause the robot to perform the planned motion.

17. 17. The non-transitory computer readable medium of claim 16, wherein the defined area size threshold is a defined contact area size associated with contact between the end effector device and the object, such that the grip area encompasses a contact area having the defined contact area size.

18. 17. The non-transitory computer-readable medium of claim 16, wherein when executed by the processing circuitry and if the object sags while being lifted, the instructions cause the processing circuitry to determine a 3D region height based on a defined lateral dimension size associated with the object.

19. 1. A method performed by a computing system, comprising: receiving image information by the computing system, the computing system configured to communicate with a robot having an end effector device, the image information including a representation of an object surface associated with an object; identifying, as grip areas, surface areas of the object surface that satisfy a defined smoothness condition related to surface smoothness and have an area size equal to or greater than a defined area size threshold based on the image information; identifying a three-dimensional (3D) region indicative of a loose portion of the object outside of the grip area in one or more horizontal dimensions, an edge of the 3D region being determined based on a maximum lateral dimension size of the object; performing a robot motion planning based on the grip area and the 3D area; and communicating a command to the robot to cause the robot to perform the planned motion.

Images