System and method for robotic system with object handling capabilities

Abstract

Systems and methods for robotic systems with object handling capabilities are disclosed. A computing system configured for object transfer is provided. The computing system includes at least one processing circuit configured to identify a pickable region of an object from image information of the object. The pickable region can be determined from a surface cost map indicative of smoothness of regions of the image information, the surface cost map being determined from height differences and normal differences. The identification of the pickable region can be used in motion planning operations for transferring the object.

Description

[0001] This application is a divisional application of invention patent application 202310238393.9, filed on March 8, 2023, entitled "System and method for a robot system with object handling capability".

[0002] Cross-reference to related applications

[0003] This application claims the benefit of U.S. Provisional Application No. 63 / 317,877, entitled “ROBOTIC SYSTEM WITH OBJECT DETECTION,” filed March 8, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0004] This technology is generally directed to robotic systems, and more specifically to systems, processes, and techniques for detecting and handling objects. More particularly, this technology can be used to identify pickable regions of objects within containers. Background Technology

[0005] With the continuous improvement in robot performance and the reduction in cost, many robots (e.g., machines configured to perform physical actions automatically / autonomously) are now widely used in a variety of different fields. For example, robots can be used to perform various tasks in manufacturing and / or assembly, packaging and / or packing, transportation and / or shipping, etc. (e.g., manipulating or conveying objects through a space). When performing tasks, robots can replicate human actions, thereby replacing or reducing human intervention that would otherwise be needed when performing dangerous or repetitive tasks.

[0006] However, despite technological advancements, robots often lack the complexity required to replicate human interactions needed to perform larger and / or more complex tasks. Therefore, improved technologies and systems for managing operations and / or interactions between robots remain in demand. Summary of the Invention

[0007] In one embodiment, a computing system is provided. The computing system includes: a control system configured to communicate with a robot having a robotic arm and a camera, the robotic arm including or attached to an end effector; and at least one processing circuit configured, when the robot is in an object disposal environment including an object source for transfer to a destination within the object disposal environment, to: acquire image information of an object; identify pickable regions of one or more selected objects among the objects by: generating a surface cost map based on the image information; segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to the one or more selected objects; generating a pickable region detection result that includes at least the one or more pickable regions; and generating a motion plan for the robot system to transfer the one or more selected objects, the motion plan being based on the pickable region detection result.

[0008] In one embodiment, an object transfer method executed by a control system is provided. The control system has at least one processing circuit and is configured to communicate with a robot having a robotic arm and a camera, the robotic arm including or attached to an end effector device. The method includes: obtaining image information of one or more objects contained in an object source; identifying pickable regions of one or more selected objects from the selected objects by: generating a surface cost map based on the image information; segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to one or more selected objects; generating a pickable region detection result that includes at least the one or more pickable regions; and generating a motion plan for a robot system to transfer the one or more selected objects, the motion plan being based on the pickable region detection result.

[0009] In one embodiment, a non-transitory computer-readable medium is provided, configured with executable instructions for object transport, the object transport being performed by a control system having at least one processing circuit and configured to communicate with a robot having a robotic arm and a camera, the robotic arm including or attached to an end effector device. The instructions may be configured to: obtain image information of one or more objects contained in an object source; identify pickable regions of one or more selected objects from the selected objects by: generating a surface cost map based on the image information; segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to one or more selected objects; generating a pickable region detection result that includes at least the one or more pickable regions; and generating a motion plan for the robot system to transport the one or more selected objects, the motion plan being based on the pickable region detection result. Attached Figure Description

[0010] Figure 1A The illustration shows a system for performing or facilitating the detection, identification, and acquisition of objects according to embodiments of this document.

[0011] Figure 1B The illustrations depict embodiments of a system for performing or facilitating the detection, identification, and acquisition of objects, according to embodiments of this document.

[0012] Figure 1C Another embodiment of a system for performing or facilitating the detection, identification, and acquisition of objects, according to embodiments of this document, is illustrated.

[0013] Figure 1D The illustration shows yet another embodiment of a system for performing or facilitating the detection, identification, and acquisition of objects according to embodiments of this document.

[0014] Figure 2A This is a block diagram illustrating a computing system configured to perform or facilitate the detection, identification, and acquisition of objects, consistent with the embodiments described herein.

[0015] Figure 2B This is a block diagram illustrating an embodiment of a computing system configured to perform or facilitate the detection, identification, and acquisition of objects, consistent with the embodiments described herein.

[0016] Figure 2C This is a block diagram illustrating another embodiment of a computing system configured to perform or facilitate the detection, identification, and acquisition of objects, consistent with the embodiments described herein.

[0017] Figure 2D This is a block diagram illustrating yet another embodiment of a computing system configured to perform or facilitate the detection, identification, and acquisition of objects, consistent with the embodiments described herein.

[0018] Figure 2E This is an example of image information processed by the system and is consistent with the embodiments described herein.

[0019] Figure 2F This is another example of image information processed by the system and is consistent with the embodiments described herein.

[0020] Figure 3A An exemplary object handling environment for operating a robotic system according to embodiments herein is illustrated.

[0021] Figure 3B An exemplary object handling environment for operating a robotic system according to embodiments herein is illustrated.

[0022] Figure 3CAn exemplary object handling environment for operating a robotic system according to embodiments herein is illustrated.

[0023] Figure 4 This is a flowchart illustrating an example process for handling detected objects.

[0024] Figure 5A An example of 2D image information for a scene consistent with the embodiments herein is illustrated.

[0025] Figure 5B An example of 3D image information for a scene consistent with the embodiments herein is illustrated.

[0026] Figure 6A An example flowchart is provided for a surface cost map generation method consistent with the embodiments herein.

[0027] Figure 6B-6E Examples of various aspects of a surface cost map generation method consistent with the embodiments herein are provided.

[0028] Figure 6F An example of a height gradient costmap consistent with the embodiments in this paper is provided.

[0029] Figure 6G An example of a normal differences costmap consistent with the embodiments in this paper is provided.

[0030] Figure 6H An example of a surface cost map consistent with the embodiments described herein is provided.

[0031] Figure 6I Examples of objects consistent with the embodiments in this article are provided.

[0032] Figure 7A Examples of segmentation methods consistent with the embodiments herein are provided.

[0033] Figures 7B-7E Examples of various aspects of the segmentation method consistent with the embodiments herein are provided.

[0034] Figure 8A and 8B Examples of various aspects of detection mask information generation consistent with the embodiments herein are provided.

[0035] Figure 9A and 9BExamples of various aspects of safety volume generation consistent with the embodiments herein are provided. Detailed Implementation

[0036] This document describes systems and methods related to object detection, identification, and retrieval. In particular, the disclosed systems and methods facilitate object detection, identification of pickable areas, and object retrieval when the object is located within a container. As discussed herein, objects may include boxes, bags, pouches, etc. Object handling in such situations can be challenging due to the irregular arrangement of objects and the difficulty in identifying suitable areas or portions for pickup (e.g., using a suction gripping device). Therefore, the systems and methods described herein are designed to identify pickable areas of objects from a set of objects, where the objects may be arranged in different locations, at different angles, etc. The systems and methods discussed herein may include robotic systems. Robotic systems configured according to embodiments of this document can autonomously perform integrated tasks by coordinating the operations of multiple robots. As described herein, a robotic system may include any suitable combination of robotic devices, actuators, sensors, cameras, and computing systems configured to control, issue commands, receive information from robotic devices and sensors, access, analyze, and process data generated by robotic devices, sensors, and cameras to generate data or information that can be used to control the robotic system and plan actions for robotic devices, sensors, and cameras. As used herein, the robotic system does not require direct access to or control of robot actuators, sensors, or other devices. As described herein, the robotic system can be a computing system configured to improve the performance of such robot actuators, sensors, and other devices by receiving, analyzing, and processing information.

[0037] The techniques described in this paper provide technological improvements to robotic systems configured for object identification, pickable area identification, and object transport. These improvements enhance the speed, accuracy, and precision of these tasks and further facilitate the detection, pickable area identification, and transport of objects from source containers or storage facilities to destinations. The robotic and computational systems described in this paper address the technical problems of identifying, detecting pickable areas, and acquiring objects from containers (where objects may be irregularly arranged). By solving these technical problems, the techniques for object identification, pickable area detection, and object acquisition are improved.

[0038] This application relates to systems and robotic systems. As discussed herein, a robotic system may include robotic actuator components (e.g., robotic arms, robotic grippers, etc.), various sensors (e.g., cameras, etc.), and various computing or control systems. As discussed herein, a computing or control system may be referred to as “controlling” various robotic components, such as robotic arms, robotic grippers, cameras, etc. Such “control” can refer to the direct control and interaction with various actuators, sensors, and other functional aspects of the robotic components. For example, a computing system can control a robotic arm by issuing or providing all necessary signals to cause various motors, actuators, and sensors to move the robot. Such “control” can also refer to issuing abstract or indirect commands to another robotic control system, which then translates such commands into the necessary signals to cause the robot to move. For example, a computing system can control a robotic arm by issuing commands describing the trajectory or destination position to which the robotic arm should move, and another robotic control system associated with the robotic arm can receive and interpret such commands, then provide the necessary direct signals to the various actuators and sensors of the robotic arm to cause the desired movement.

[0039] In particular, the techniques described herein assist robotic systems in interacting with a target object among multiple objects in a container. The methods and systems described herein can identify pickable regions of selected objects from a set of objects. As described herein, robotic transfer mechanisms (e.g., robotic arms) may include suction cups or suction grippers as part of an end effector device for grasping, picking up, or holding objects. Such suction-based gripping devices can perform better when applied to a smooth surface of the object (e.g., a portion of the object with a sufficiently smooth surface profile so that the suction cup can engage between the object's surface and the suction cup to form a seal for lifting and transferring the object). A surface that is sufficiently smooth to properly engage with a suction gripper and large enough to accommodate one or more suction grippers in a robotic transfer system can be referred to as a "pickable region." When objects are loosely organized in a source storage or container, the systems and methods described herein can be used to identify pickable regions of objects.

[0040] In the following, specific details are set forth to provide an understanding of the currently disclosed technology. In the embodiments, the technology described herein may be practiced without including every specific detail disclosed herein. In other instances, well-known features such as specific functions or routines are not described in detail to avoid unnecessarily obscuring this disclosure. References to “embodiment,” “an embodiment,” etc., in this specification mean that a particular feature, structure, material, or characteristic being described is included in at least one embodiment of this disclosure. Therefore, the appearance of such phrases in this specification does not necessarily refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Furthermore, a particular feature, structure, material, or characteristic described with respect to any embodiment may be combined in any suitable manner with those in any other embodiment, unless such items are mutually exclusive. It should be understood that the various embodiments shown in the figures are merely illustrative representations and are not necessarily drawn to scale.

[0041] For clarity, certain details describing structures or processes well-known and commonly associated with robotic systems and subsystems, but which may unnecessarily obscure some important aspects of the disclosed technology, are not set forth in the following description. Furthermore, although the following disclosure sets forth several embodiments of different aspects of the technology, several other embodiments may have different configurations or different components than those described in this section. Therefore, the disclosed technology may have other embodiments with additional elements or without several of the elements described below.

[0042] Many embodiments or aspects of this disclosure described below can take the form of computer or controller executable instructions, including routines executed by a programmable computer or controller. Those skilled in the art will understand that the disclosed technology can be implemented or practiced on computer or controller systems other than those shown and described below. The technology described herein can be embodied in a dedicated computer or data processor specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described below. Therefore, as commonly used herein, the terms “computer” and “controller” refer to any data processor and can include internet tools and handheld devices (including handheld computers, wearable computers, cellular or mobile phones, multiprocessor systems, processor-based or programmable consumer electronics, network computers, minicomputers, etc.). Information processed by these computers and controllers can be presented on any suitable display medium, including a liquid crystal display (LCD). Instructions for performing computer or controller executable tasks can be stored in or on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. Instructions can be contained in any suitable memory device, including, for example, a flash drive, a USB device, and / or other suitable media.

[0043] The terms “coupling” and “connection”, and their derivatives, are used herein to describe structural relationships between components. It should be understood that these terms are not intended to be synonyms. Rather, in certain embodiments, “connection” can be used to indicate that two or more elements are in direct contact with each other. Unless explicitly stated otherwise in the context, the term “coupling” can be used to indicate that two or more elements are in direct or indirect contact with each other (through other inserting elements between them), or that two or more elements cooperate or interact with each other (e.g., as in a causal relationship, such as for signal transmission / reception or function invocation), or both.

[0044] Any reference to image analysis performed via a computational system herein can be performed based on or using spatial structure information, which may include depth information describing corresponding depth values ​​relative to various locations of selected points. The depth information can be used to identify objects or estimate how objects are arranged in space. In some cases, the spatial structure information may include or be used to generate a point cloud describing the positions of one or more surfaces of an object. Spatial structure information is merely one form of possible image analysis, and other forms known to those skilled in the art can be used according to the methods described herein.

[0045] Figure 1AA system 1000 for performing object detection, or more specifically, object recognition, is illustrated. More specifically, system 1000 may include a computing system 1100 and a camera 1200. In this example, camera 1200 may be configured to generate image information that describes or otherwise represents the environment in which camera 1200 is located, or more specifically, the environment within the field of view (also referred to as the camera field of view) of camera 1200. The environment may be, for example, a warehouse, a manufacturing plant, a retail space, or other location. In the specific embodiments described herein, the environment may be an object disposal environment comprising one or more source repositories and one or more destination repositories. In such a case, the image information may represent images of objects (such as boxes, bags, pouches, cabinets, crates, etc.) located in such a location. Such objects may be located in source repositories and destination repositories. System 1000 can be configured to generate, receive, and / or process image information to perform object identification or registration based on the image information, and / or to perform robot interaction planning based on the image information, as discussed in more detail below (the terms "and / or" and "or" are used interchangeably in this disclosure). Robot interaction planning can be used, for example, to control a robot at a location to facilitate robot interaction between the robot and containers or other objects. Computing system 1100 and camera 1200 can be located in the same location or can be located remotely from each other. For example, computing system 1100 may be part of a cloud computing platform hosted in a data center remote from a warehouse or retail space and may communicate with camera 1200 via a network connection.

[0046] In an embodiment, camera 1200 (which may also be referred to as an image sensing device) may be a 2D camera and / or a 3D camera. For example, Figure 1BA system 1500A (which may be an embodiment of system 1000) is illustrated. System 1500A includes a computing system 1100 and cameras 1200A and 1200B, both of which may be embodiments of camera 1200. In this example, camera 1200A may be a 2D camera configured to generate 2D image information that includes or forms a 2D image describing the visual appearance of the environment in the camera's field of view. Camera 1200B may be a 3D camera (also referred to as a spatial structure sensing camera or spatial structure sensing device) configured to generate 3D image information that includes or forms spatial structure information about the environment in the camera's field of view. This spatial structure information may include depth information (e.g., a depth map) that describes corresponding depth values ​​relative to various positions of camera 1200B, such as positions on the surfaces of various objects in the field of view of camera 1200B. These positions in the camera's field of view or on the surfaces of objects may also be referred to as physical positions. In this example, depth information can be used to estimate how objects are spatially arranged in three-dimensional (3D) space. In some cases, spatial structure information can include or can be used to generate point clouds (also known as 3D point clouds) that describe the positions of objects on one or more surfaces within the field of view of the camera 1200B. More specifically, spatial structure information can describe various positions on the structure (also known as object structure) of one or more objects.

[0047] In an embodiment, system 1000 may be a robot operating system for facilitating robot interaction between the robot and various objects in the environment of camera 1200. For example, Figure 1C The robot operating system 1500B is shown, which can be Figure 1A and Figure 1BAn embodiment of system 1000 / 1500A. Robot operating system 1500B may include computing system 1100, camera 1200, and robot 1300. As described above, robot 1300 can be used to interact with one or more objects (such as boxes, bags, pouches, crates, cabinets, pallets, or other containers) in the environment of camera 1200. For example, robot 1300 can be configured to pick up objects from one location and move them to another location. In some cases, robot 1300 can be used to perform depalletizing operations in which a set of containers or other objects are unloaded and moved to, for example, a conveyor belt. In some implementations, as discussed below, camera 1200 can be attached to robot 1300 or robot 3300. This is also referred to as a handheld camera or handheld camera solution. Camera 1200 can be attached to robotic arm 3320 of robot 1300. Robotic arm 3320 can then move to various picking ranges to generate image information about those ranges. In some implementations, camera 1200 can be separate from robot 1300. For example, camera 1200 can be mounted to the ceiling or other structure of a warehouse and can remain fixed relative to that structure. In some implementations, multiple cameras 1200 may be used, including multiple cameras 1200 separate from robot 1300 and / or cameras 1200 separate from robot 1300 used in conjunction with handheld camera 1200. In some implementations, one or more cameras 1200 may be mounted or fixed to a dedicated robotic system separate from robot 1300 for object manipulation, such as a robotic arm, gantry, or other automated system configured for camera movement. Throughout the specification, “control” of camera 1200 may be discussed. For handheld camera solutions, control of camera 1200 also includes control of robot 1300 to which camera 1200 is mounted or attached.

[0048] In an embodiment, Figure 1A-1C The computing system 1100, also referred to as a robot controller, can be formed or integrated into the robot 1300. The robot control system can be included in system 1500B and can be configured to generate commands for the robot 1300, such as robot interaction movement commands for controlling robot interactions between the robot 1300 and containers or other objects. In such embodiments, the computing system 1100 can be configured to generate such commands based on image information, for example, generated by camera 1200. For example, the computing system 1100 can be configured to determine a motion plan based on the image information, wherein the motion plan may be intended for, for example, grasping or otherwise picking up an object. The computing system 1100 can generate one or more robot interaction movement commands to execute the motion plan.

[0049] In this embodiment, the computing system 1100 may form part of a vision system. The vision system may be a system that generates, for example, visual information describing the environment in which the robot 1300 is located, or alternatively or additionally, describing the environment in which the camera 1200 is located. The visual information may include the 3D and / or 2D image information discussed above, or some other image information. In some cases, if the computing system 1100 forms a vision system, the vision system may be part of the robot control system discussed above, or it may be separate from the robot control system. If the vision system is separate from the robot control system, the vision system may be configured to output information describing the environment in which the robot 1300 is located. This information may be output to the robot control system, which may receive such information from the vision system and perform motion planning and / or generate robot interactive movement commands based on that information. Further information regarding the vision system is described in detail below.

[0050] In one embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 via a direct connection (such as a connection provided via a dedicated wired communication interface (such as an RS-232 interface, a Universal Serial Bus (USB) interface) and / or via a local computer bus (such as a Peripheral Component Interconnect (PCI) bus)). In another embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 via a network. The network may be any type and / or form of network, such as a Personal Area Network (PAN), a Local Area Network (LAN) (e.g., an intranet), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), or the Internet. The network may utilize different technologies and protocol layers or stacks, including, for example, Ethernet, Internet Protocol Suite (TCP / IP), ATM (Asynchronous Transfer Mode) technology, SONET (Synchronous Optical Networking) protocol, or SDH (Synchronous Digital Hierarchy) protocol.

[0051] In embodiments, the computing system 1100 may communicate information directly with the camera 1200 and / or with the robot 1300, or it may communicate via an intermediate storage device or more generally via an intermediate non-transitory computer-readable medium. For example, Figure 1DSystem 1500C, which may be an embodiment of system 1000 / 1500A / 1500B, is illustrated. System 1500C includes a non-transitory computer-readable medium 1400, which may be external to computing system 1100 and may act as an external buffer or a repository for storing, for example, image information generated by camera 1200. In such an example, computing system 1100 may retrieve or otherwise receive image information from non-transitory computer-readable medium 1400. Examples of non-transitory computer-readable medium 1400 include electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. Non-transitory computer-readable media may be formed, for example, computer floppy disks, hard disk drives (HDDs), solid-state drives (SDDs), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disk (DVD), and / or memory sticks.

[0052] As described above, camera 1200 can be a 3D camera and / or a 2D camera. A 2D camera can be configured to generate 2D images, such as color or grayscale images. A 3D camera can be, for example, a depth-sensing camera, such as a time-of-flight (TOF) camera or a structured light camera, or any other type of 3D camera. In some cases, the 2D camera and / or 3D camera may include image sensors, such as charge-coupled device (CCD) sensors and / or complementary metal-oxide-semiconductor (CMOS) sensors. In embodiments, the 3D camera may include a laser, a LiDAR device, an infrared device, a light / dark sensor, a motion sensor, a microwave detector, an ultrasonic detector, a RADAR detector, or any other device configured to capture depth information or other spatial structure information.

[0053] As described above, image information can be processed by computing system 1100. In embodiments, computing system 1100 may include or be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, laptop computer, etc.), a smartphone, a tablet computing device, and / or any other computing system. In embodiments, any or all of the functions of computing system 1100 may be performed as part of a cloud computing platform. Computing system 1100 may be a single computing device (e.g., a desktop computer) or may include multiple computing devices.

[0054] Figure 2AA block diagram illustrating an embodiment of a computing system 1100 is provided. The computing system 1100 in this embodiment includes at least one processing circuitry 1110 and a non-transitory computer-readable medium (or media) 1120. In some cases, the processing circuitry 1110 may include a processor (e.g., a central processing unit (CPU), a dedicated computer, and / or an onboard server) configured to execute instructions (e.g., software instructions) stored on the non-transitory computer-readable medium 1120 (e.g., computer memory). In some embodiments, the processor may be included in a separate / independent controller operatively coupled to other electronic / electrical devices. The processor may implement program instructions to control other devices / interface with other devices, thereby enabling the computing system 1100 to perform actions, tasks, and / or operations. In embodiments, the processing circuitry 1110 includes one or more processors, one or more processing cores, a programmable logic controller (“PLC”), an application-specific integrated circuit (“ASIC”), a programmable gate array (“PGA”), a field-programmable gate array (“FPGA”), any combination thereof, or any other processing circuitry.

[0055] In embodiments, the non-transitory computer-readable medium 1120, which is part of the computing system 1100, may be an alternative to or addition to the intermediate non-transitory computer-readable medium 1400 discussed above. The non-transitory computer-readable medium 1120 may be a storage device, such as an electronic storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device, or any suitable combination thereof, for example, such as a computer floppy disk, hard disk drive (HDD), solid-state drive (SDD), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, any combination thereof, or any other storage device. In some cases, the non-transitory computer-readable medium 1120 may include multiple storage devices. In some implementations, the non-transitory computer-readable medium 1120 is configured to store image information generated by the camera 1200 and received by the computing system 1100. In some cases, the non-transitory computer-readable medium 1120 may store one or more object recognition templates for performing the methods and operations discussed herein. The non-transitory computer-readable medium 1120 may alternatively or additionally store computer-readable program instructions that, when executed by the processing circuitry 1110, cause the processing circuitry 1110 to perform one or more methods described herein.

[0056] Figure 2BA computing system 1100A is depicted, which is an embodiment of the computing system 1100 and includes a communication interface 1130. The communication interface 1130 can be configured, for example, to receive signals from... Figure 1A-1D Image information generated by camera 1200. Image information can be received via the intermediate non-transitory computer-readable medium 1400 or network discussed above, or via a more direct connection between camera 1200 and computing system 1100 / 1100A. In embodiments, the communication interface can be configured to... Figure 1C The computing system 1100 communicates with the robot control system. If the computing system 1100 is external to the robot control system, the communication interface of the computing system 1100 can be configured to communicate with the robot control system. The communication interface may also be referred to as a communication component or communication circuit, and may include, for example, communication circuitry configured to perform communication via wired or wireless protocols. As an example, the communication circuitry may include an RS-232 port controller, a USB controller, an Ethernet controller, etc. Controller, PCI bus controller, any other communication circuit or combination thereof.

[0057] In an embodiment, such as Figure 2C As shown, the non-transitory computer-readable medium 1120 may include storage space 1125 configured to store one or more data objects discussed herein. For example, the storage space may store object recognition templates, detection hypotheses, image information, object image information, robotic arm movement commands, and any additional data objects that the computing system discussed herein may need to access.

[0058] In an embodiment, the processing circuitry 1110 may be programmed by one or more computer-readable program instructions stored on a non-transitory computer-readable medium 1120. For example, Figure 2DThe illustration shows a computing system 1100C, an embodiment of computing systems 1100 / 1100A / 1100B, wherein processing circuitry 1110 is programmed with one or more modules including an object recognition module 1121, a motion planning module 1129, and an object manipulation planning module 1126. Processing circuitry 1110 may also be programmed with an object registration module 1130 and a pickable area detection module 1132. Each of these modules may represent computer-readable program instructions configured to perform certain tasks when instantiated on one or more of the processors, processing circuitry, computing systems, etc., described herein. Each of these modules may cooperate with each other to achieve the functions described herein. Various aspects of the functions described herein may be performed by one or more of the aforementioned software modules, and the software modules and their descriptions should not be construed as limiting the computational architecture of the systems disclosed herein. For example, while specific tasks or functions may be described with respect to specific modules, these tasks or functions may also be performed by different modules as needed. Furthermore, the system functions described herein may be performed by different sets of software modules configured with different functional decomposition or allocation methods.

[0059] In embodiments, the object recognition module 1121 may be configured to acquire and analyze image information, as discussed throughout this disclosure. The methods, systems, and techniques discussed herein regarding image information may utilize the object recognition module 1121. The object recognition module may also be configured for object recognition tasks related to object identification, as discussed herein.

[0060] The motion planning module 1129 can be configured to plan and execute the robot's movements. For example, the motion planning module 1129 can interact with other modules described herein to plan the movements of the robot 3300 for object acquisition operations and for camera placement operations. The methods, systems, and techniques discussed herein regarding robotic arm movement and trajectory can be performed by the motion planning module 1129.

[0061] The object manipulation planning module 1126 can be configured to plan and execute object manipulation activities of the robotic arm, such as grasping and releasing objects, and executing robotic arm commands to help and facilitate such grasping and releasing.

[0062] The object registration module 1130 can be configured to acquire, store, generate, and otherwise process object registration and detection information that may be required for the various tasks discussed herein. The object registration module 1130 can be configured to interact or communicate with any other necessary modules.

[0063] Pickable area detection module 1132 can be configured to identify pickable areas on the surface of one or more objects, for example, as per [reference to...]. Figure 4As described, the pickable area detection module 1132 can be configured to interact or communicate with any other necessary modules.

[0064] refer to Figure 2E , 2F 3A, 3B, and 3C explain the methods related to the object recognition module 1121 and the object registration module 1130, which can be executed for image analysis. Figure 2E and 2F The illustration shows example image information associated with the image analysis method, while Figures 3A-3C An example robotic environment associated with an image analysis method is illustrated. References in this paper relating to image analysis performed by a computational system can be performed based on or using spatial structure information, which may include depth information describing corresponding depth values ​​relative to various locations of selected points. The depth information can be used to identify objects or estimate how objects are arranged in space. In some cases, the spatial structure information may include or be used to generate a point cloud describing the positions of one or more surfaces of an object. Spatial structure information is merely one form of possible image analysis, and other forms known to those skilled in the art can be used according to the methods described herein.

[0065] In an embodiment, computing system 1100 can acquire image information representing objects in the camera field of view (e.g., 3210) of camera 1200. The steps and techniques described below for acquiring image information can be image information capture operations. In some cases, the object can be one of a plurality of objects 3520 in a source container 3510 within the field of view 3210 of camera 1200. Image information 2600, 2700 can be generated by the camera (e.g., 1200) when the object 3520 is (or is already) in the camera field of view 3210 and can describe one or more of the individual objects 3520. The object appearance describes the appearance of the object 3520 from the viewpoint of camera 1200. If there are multiple objects 3520 in the camera field of view, the camera can generate image information representing the multiple objects or a single object as needed (such image information associated with a single object can be referred to as object image information). Image information can be generated by a camera (e.g., 1200) when the group of objects is (or has been) in the camera's field of view, and can include, for example, 2D image information and / or 3D image information.

[0066] As an example, Figure 2E The first set of image information, or more specifically, 2D image information 2600, as described above, is generated by camera 1200 and represents object 3520, such as... Figures 3A-3CThe objects shown are examples of this. More specifically, the 2D image information 2600 can be a grayscale or color image and can describe the appearance of object 3520 from the viewpoint of camera 1200. In embodiments, the 2D image information 2600 can correspond to a single color channel (e.g., red, green, or blue channel) of a color image. If camera 1200 is positioned above object 3520, the 2D image information 2600 can represent the appearance of the corresponding top surface of object 3520. Figure 2E In the example, 2D image information 2600 may include corresponding portions 2000A / 2000B / 2000C / 2000D / 2550 representing the corresponding surfaces of object 3520, also referred to as image portions or object image information. Figure 2E In the 2D image information 2600, each image portion 2000A / 2000B / 2000C / 2000D / 2550 can be an image range, or more specifically, a pixel range (if the image is formed by pixels). Each pixel in the pixel range of the 2D image information 2600 can be characterized as having a position described by a set of coordinates [U,V] and can have values ​​relative to the camera coordinate system or some other coordinate system, such as... Figure 2E and 2F As shown. Each pixel may also have an intensity value, such as a value between 0 and 255 or 0 and 1023. In another embodiment, each pixel may include any additional information associated with pixels of various formats (e.g., hue, saturation, intensity, CMYK, RGB, etc.).

[0067] As described above, in some embodiments, image information can be all or part of an image, such as 2D image information 2600. For example, computing system 3100 can be configured to extract image portion 2000A from 2D image information 2600 to obtain image information associated only with the corresponding object 3520. In the case where an image portion (such as image portion 2000A) points to a single object, it can be referred to as object image information. Object image information does not need to contain only information about the object it points to. For example, the object it points to may be near, below, above, or otherwise located near one or more other objects. In such cases, object image information can include information about the object it points to as well as one or more adjacent objects. Computing system 1100 can extract image information based on 2D image information 2600 and / or Figure 2FThe 3D image information 2700 shown performs image segmentation or other analysis or processing operations to extract image portions 2000A. In some implementations, the segmentation or other processing operations may include detecting the image locations where the physical edges of objects (e.g., object edges) appear in the 2D image information 2600, and using such image locations to identify object image information that is limited to individual objects representing the camera's field of view (e.g., 3210) and substantially excludes other objects. "Substantially excludes" means that image segmentation or other processing techniques can be designed and configured to exclude non-target objects from the object image information, but it is understood that errors may occur, noise may be present, and various other factors may cause portions to contain other objects.

[0068] Figure 2F An example is depicted where the image information is 3D image information 2700. More specifically, 3D image information 2700 may include, for example, depth maps or point clouds indicating corresponding depth values ​​at various locations on one or more surfaces (e.g., top surface or other outer surfaces) of object 3520. In some implementations, image segmentation operations for extracting image information may involve detecting the image locations where physical edges of objects (e.g., edges of boxes) appear in 3D image information 2700, and using such image locations to identify image portions (e.g., 2730) limited to representing individual objects (e.g., 3520) within the camera's field of view.

[0069] The corresponding depth values ​​can be relative to the camera 1200 that generates the 3D image information 2700, or they can be relative to another reference point. In some implementations, the 3D image information 2700 may include a point cloud (3D point cloud) that includes the corresponding coordinates of various structural positions of objects within the camera's field of view (e.g., 3210). Figure 2F In the example, the point cloud may include a corresponding set of coordinates describing the position on the corresponding surface of object 3520. The coordinates may be 3D coordinates, such as [XY Z] coordinates, and may have values ​​relative to the camera coordinate system or some other coordinate system. For example, 3D image information 2700 may include a set of positions 27101-2710 on the surface of object 3520, also referred to as physical positions. n The first image portion 2710 (also referred to as the image portion) contains the corresponding depth value. Furthermore, the 3D image information 2700 may also include second, third, fourth, and fifth portions 2720, 2730, 2740, and 2750. These portions can then indicate the depth values ​​that can be represented by 27201-2720 respectively. n ,、27301-2730 n 27401-2740 n and 27501-2750 nThese represent the corresponding depth values ​​for a set of locations. These images are merely examples and any number of objects with corresponding image portions can be used. Similarly, the acquired 3D image information 2700 may, in some cases, be a portion of a first set of 3D image information 2700 generated by the camera. Figure 2E In the example, if the acquired 3D image information 2700 represents Figure 3A If the 3D image information 2700 is a single object 3520, then the 3D image information 2700 can be reduced to refer only to the image portion 2710. Similar to the discussion of the 2D image information 2600, the identified image portion 2710 can belong to an individual object and can be referred to as object image information. Therefore, as used herein, object image information can include 2D and / or 3D image information.

[0070] In an embodiment, as part of acquiring image information, an image normalization operation may be performed by the computing system 1100. The image normalization operation may involve transforming an image or portion of an image generated by the camera 1200 to generate a transformed image or portion of an image. For example, if the acquired image information may include 2D image information 2600, 3D image information 2700, or a combination of both, it may undergo an image normalization operation to attempt to alter the image information in terms of viewpoint, object pose, and lighting conditions associated with visual descriptive information. Such normalization may be performed to facilitate a more accurate comparison between image information and model (e.g., template) information. The viewpoint may refer to the pose of an object relative to the camera 1200, and / or the angle at which the camera 1200 is observing the object when the camera 1200 generates an image representing the object.

[0071] For example, image information can be generated during an object recognition operation, where the target object is within the camera's field of view 3210. When the target object has a specific pose relative to the camera, the camera 1200 can generate image information representing the target object. For example, the target object may have a pose in which its top surface is perpendicular to the optical axis of the camera 1200. In such an example, the image information generated by the camera 1200 can represent a specific viewpoint, such as a top view of the target object. In some cases, when the camera 3210 generates image information during an object recognition operation, the image information can be generated under specific lighting conditions, such as illumination intensity. In such cases, the image information can represent a specific illumination intensity, illumination color, or other lighting conditions.

[0072] In an embodiment, image normalization may involve adjusting an image or portion of an image of a scene generated by a camera to better match the viewpoint and / or lighting conditions associated with information from an object recognition template. The adjustment may involve transforming the image or portion to generate a transformed image that matches at least one of the object pose or lighting conditions associated with the visual descriptive information of the object recognition template.

[0073] Viewpoint adjustment can involve processing, rolling, and / or shifting an image of a scene such that the image represents the same viewpoint as the visual descriptive information that may be included within an object recognition template. For example, processing may include changing the color, contrast, or lighting of an image; rolling the scene may include changing the size, dimensions, or scale of the image; and shifting the image may include changing its position, orientation, or rotation. In example embodiments, processing, rolling, and / or shifting can be used to alter the orientation and / or size of objects in an image of the scene to match or better correspond to the visual descriptive information of the object recognition template. If the object recognition template describes a head-on view (e.g., a top view) of an object, the image of the scene may be rolled so that it also represents a head-on view of objects in the scene.

[0074] Other aspects of the object recognition method performed herein are described in more detail in U.S. Application No. 16 / 991,510, filed August 12, 2020, and U.S. Application No. 16 / 991,466, filed August 12, 2020, each of which is incorporated herein by reference.

[0075] In various embodiments, the terms "computer-readable instructions" and "computer-readable program instructions" are used to describe software instructions or computer code configured to perform various tasks and operations. In various embodiments, the term "module" broadly refers to a collection of software instructions or code configured to cause the processing circuitry 1110 to perform one or more functional tasks. When the processing circuitry or other hardware components are executing modules or computer-readable instructions, these modules and computer-readable instructions can be described as performing various operations or tasks.

[0076] Figures 3A-3C The illustration shows an example environment where pickable region (or graspable region) detection and / or motion planning operations can be performed. More specifically, Figure 3A The system 3000 (which can be) is described Figure 1A-1DThe system (implementations 1000 / 1000A / 1000B / 1000C) includes a computing system 1100, a robot 3300, and a camera 1200. The camera 1200 may be an embodiment of a camera 1200 and may be configured to generate image information representing the scene within the camera field of view 3210 of the camera 1200, or more specifically, objects within the camera field of view 3210 (such as objects 35201 to 3520). n This can include, for example, objects 35201, 35202, 35203, 35204, 35205, ... 3520. n ) or its structure. In Figures 3A-3C In embodiments, robot 3300 may be configured to manipulate or otherwise interact with objects 35201-3520. n Interact with each of one or more objects in the 35201-3520, such as by picking up or otherwise grabbing them. n One of them is to lift the object from its current position and then move the object to its destination position.

[0077] In some cases, objects 35201 to 3520 n Some or all of them can be flexible objects. For example, objects 35201 to 3520. n Each of these can be a package containing a garment (e.g., a shirt or trousers) or other textile or fabric, wherein the garment or other textile may be wrapped in a sheet of packaging material (such as a sheet of plastic). In some cases, the plastic or other packaging material sheet may generally be unaffected by air or other liquids. Figure 3A In the example, objects 35201 to 3520 n It can be deployed in container 3510, such as for 35201 to 3520. n Boxes or crates kept in facilities (such as warehouses associated with clothing manufacturers or retailers). In some cases, items 35201 to 3520. n Some or all of these can include items such as boxes, bags, pouches, and other items.

[0078] In some cases, the flexible object (e.g., 35201) of the embodiments herein may have sufficiently high flexibility to allow the shape of the flexible object to deform when the robot 3300 moves or otherwise manipulates it, or when it is placed in the container 3510. Sufficiently high flexibility may correspond to sufficiently low stiffness or rigidity to prevent the object from maintaining its shape when moved or otherwise manipulated by the robot 3300. In some cases, the flexible object may have sufficiently high flexibility to allow the weight of the flexible object to cause deformation of the object's own shape when the robot 3300 lifts the flexible object. Deformation may involve, for example, bending of the flexible object, or more specifically, sagging under its own weight when lifted by the robot 3300. The flexibility of the flexible object may come from, for example, the dimensions of the flexible object and / or the material of the flexible object. In one example, the flexible object may have a thin cross-section, which can introduce flexibility (also known as suppleness) into the flexible object. More specifically, the thickness dimension of the flexible object may be very small relative to the dimensions of the lateral dimensions (e.g., the length dimension or the width dimension). In one example, the flexible object may be made of a sufficiently soft material to introduce flexibility into the flexible object. In some cases, the material of a flexible object can be soft enough to sag under its own weight when the robot 3300 lifts the object. For example, if the flexible object is a package containing a piece of clothing, it might be made of a material such as cotton or wool fabric, which lacks sufficient stiffness to prevent the material from sag under its own weight when the robot 3300 lifts it.

[0079] In an embodiment, robot 3300 (which may be an embodiment of robot 1300) may include a robotic arm 3320, one end of which is attached to robot base 3310 and the other end of which is attached to or formed by end effector device 3330. Robot base 3310 may be used to mount one end of robotic arm 3320, while the other end of robotic arm 3320, or more specifically end effector device 3330, may be used to interact with one or more objects (e.g., 35201, 35202, etc.) in the environment of robot 3300. Interaction may include, for example, grasping and lifting one or more objects, and / or moving one or more objects from a current position to a destination position.

[0080] In an embodiment, the end effector device 3330 may include one or more suction cups 33321-3332. n (Also referred to herein as a suction gripper and suction gripping device), used to pick up or otherwise lift objects, such as objects 35201-3520. n One of them. In some implementations, the suction cup is 33321-3332. nEach of these (also referred to as an end effector suction cup) can be a mechanical device configured to reduce the fluid pressure (e.g., air pressure) in the space between the suction cup and the surface of the object (also referred to as the object surface) when pressed against the surface of an object (e.g., 35201). In the example, the object surface may be formed of a generally impermeable (or more generally non-porous) material, such as the plastic packaging material used to wrap a garment. The reduced fluid pressure (such as partial or complete vacuum) results in a pressure difference between the fluid pressure outside the space and the fluid pressure inside the space. More specifically, the fluid pressure inside the space may be lower than the fluid pressure outside the space, which creates a negative fluid pressure, causing the higher fluid pressure to exert a net force that causes the suction cup to adhere to the object surface. The net force can act as an adhesive force, enabling the suction cup to adhere to the object surface, thereby gripping the object surface. In embodiments, each of these suction cups (e.g., 33321 or 3332) n The suction cup can have various shapes (e.g., circular) and sizes, and can be made of various materials such as plastics, silicone, nitrogen, viton, vinyl, urethane, rubber, or some other flexible material. The suction cup is discussed in more detail in U.S. Patent No. 10,576,630, entitled “Robotic system with a robot arm suction control mechanism and method of operation thereof,” the entire contents of which are incorporated herein by reference. In embodiments, the strength of the adhesive force between the suction cup and the object surface can depend on how tightly the suction cup can seal the space between itself and the object surface. For example, a tight seal can maintain a pressure difference, thereby maintaining the adhesive force, while a loose seal will prevent the pressure difference from being maintained, thus interfering with the suction cup's ability to grip the object surface. In embodiments, the suction cup's ability to form a tight seal can depend on the smoothness of the area of ​​the object surface to which the suction cup attempts to grip (also referred to as the surface area). Therefore, as discussed in more detail below, the computing system 1100 can be configured to identify or search for surface areas that are smooth enough to be used as gripping areas to which the suction cup can reliably adhere to thereby gripping the object surface.

[0081] In this embodiment, camera 1200 can be configured to generate representations of objects 35201-3520. nImage information of container 3510 or any other object(s) within camera field of view 3210. Camera 1200 may be a 3D camera configured to generate 3D image information and / or a 2D camera configured to generate 2D image information. In embodiments, the 3D image information may represent the collective object surface of object 3520, or more specifically, describe the physical structure of the object surface. For example, the 3D image information may include a depth map, or more generally, depth information that describes depth values ​​of various locations within camera field of view 3210 relative to camera 1200 or relative to other reference points. The locations corresponding to the respective depth values ​​may be locations on various surfaces within camera field of view 3210, such as objects 35201 to 3520. n The position on the corresponding object surface. In some cases, 3D image information may include a point cloud, which may include multiple 3D coordinates describing the position of objects 35201 to 3520 in the camera's field of view 3210. n The position on the corresponding object surface.

[0082] In an embodiment, the object surface of an object (e.g., 35201) may refer to the outer surface (e.g., the top surface) of the object. In such an embodiment, the 3D image information may include information representing the outer surface, or more specifically, may describe the physical structure of the outer surface. For example, if the camera 1200 generates 3D image information by sensing light (e.g., laser or structured light) or other signals reflected from the outer surface, then the 3D information may represent, for example, the surface profile of the outer surface. If the outer surface is formed of a transparent material (such as a flexible plastic sheet used as packaging material), then the 3D information may still represent the outer surface of the object. More specifically, in this case, the camera 1200 may sense light or other signals reflected from a non-transparent material (such as a piece of clothing fabric that is underneath or otherwise covered by a transparent material). The reflected light or signal can pass through the transparent material and can be detected by the camera 1200 to generate 3D information. In this case, the transparent material (e.g., a plastic sheet) may be thin enough that the distance between the outer surface and the surface of the non-transparent material can be considered negligible. Therefore, in this embodiment, the 3D information can be considered to describe the depth information at various locations on the outer surface of the object. Furthermore, if a transparent material forms the outer surface, the transparent material may be flexible enough that all or many portions of the transparent material follow the surface contour of the underlying non-transparent material. Therefore, in this case, the 3D image information can be considered to describe the outer surface of the object, or more specifically, the physical structure or surface contour of the outer surface.

[0083] In embodiments, 2D image information may include, for example, color or grayscale images representing the appearance of one or more objects within the camera's field of view 3210. For example, if the object surface has visual markings (e.g., logos) or other visual details printed thereon, the 2D image information may describe or otherwise represent those visual details. As mentioned above, the object surface may be the outer surface of the object, which in some cases may be formed of a transparent material. In such cases, the 2D image information may represent light (e.g., visible light) or other signals reflected from the surface of the underlying opaque material (e.g., a shirt) and passing through the transparent material forming the outer surface. Because the 2D image information in this case is based on light or other signals passing through the outer surface, it can still be considered that the 2D image information represents the outer surface. Additionally, in some cases, the transparent material forming the outer surface may be thin and transparent enough to have little or negligible effect on the appearance of the object, making it possible to consider the appearance of the object or the outer surface of the object to refer to the appearance of the underlying opaque material (e.g., clothing material).

[0084] In one embodiment, system 3000 may include multiple cameras. For example, Figure 3B The illustration shows a system 3000A (which may be an embodiment of system 3000), which includes a camera 1200A having a camera field of view 3210A, and a camera 1200B having a camera field of view 3210B. Camera 1200A (which may be an embodiment of camera 1200A) may be, for example, a 2D camera configured to generate 2D images or other 2D image information, while camera 1200B (which may be an embodiment of camera 1200B) may be, for example, a 3D camera configured to generate 3D image information.

[0085] In embodiments, cameras 1200 / 1200A / 1200B may be fixed relative to a reference point (such as the floor on which container 3510 is placed) or relative to robot base 3310. For example, Figure 3A The camera 1200 can be mounted to a ceiling (such as a warehouse ceiling) or to a mounting frame that remains fixed relative to the floor, the robot base 3310, or some other reference point. In an embodiment, the camera 1200 can be mounted on a robotic arm 3320. For example, Figure 3C A system 3000B (which may be an embodiment of system 1000) is depicted, wherein a camera 1200 is attached to or otherwise mounted on an end effector device 3330 forming the distal end of a robotic arm 3320. This embodiment can provide a robot 3300 with the ability to move the camera 1200 to different poses via movement of the robotic arm 3320.

[0086] The computing system 1100 can be configured to generate pickable region detection results for one or more objects 3520 at a source container 3510. For example, the source container 3510 may include a container of objects 3520 having random orientation, pose, and position. In addition to the pickable region, the pickable region detection results may also include additional information such as detection mask information, safety volume, or a combination thereof, each of which is described in detail below.

[0087] The robot 3300 may also include other sensors configured to acquire information for performing tasks, such as for manipulating structural members and / or for transporting robot units. These sensors may include devices configured to detect or measure one or more physical characteristics of the robot 3300 and / or its surrounding environment (e.g., the state, condition, and / or position of one or more of its structural members / joints). Some examples of sensors may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

[0088] Figure 4 A flowchart illustrating the overall flow of methods and operations for identifying pickable regions of one or more selected objects within a container is provided. The pickable region identification method 4000 may include any combination of features of the sub-methods and operations described herein. This method 4000 may be performed or executed by any suitable system and device described herein.

[0089] In operation 4002, method 4000 includes acquiring image information. Image information of a group or multiple objects contained in a source container can be acquired by a computing system. Image information can be acquired, for example, by controlling a camera and / or from a data storage device on which image information has been stored. As discussed herein, image information of objects in a scene may include 3D image information 2700. Figure 5A and Figure 5B Representative examples of the scene are provided, which include multiple objects represented by 2D image information 5600. Figure 5A ) and 3D image information representing the scene 5700 ( Figure 5B ).

[0090] Figure 5A Depicting and representing images generated by camera 1200 / 1200A Figures 3A-3C Objects 35201-3520 n And the 2D image information of container 3510, or more specifically, 2D image information 5600. More specifically, 2D image information 5600 can describe objects 35201-3520. nAnd objects 35201-3520 are deployed within them. n The appearance of container 3510. More specifically, 2D image information 5600 may include representations of objects 35201, 35202, 35203, ..., 35200 respectively. n The visual details of the image portions 56101, 56202, 56203, 56204, 56205, ..., 5620 n-3 5620 n-2 5620 n-1 5620 n (e.g., pixel region). In embodiments, 2D image information may represent the object surface of an object (e.g., 35201). As described above, the object surface may be the outer surface of the object (e.g., the top surface) and may be formed of a transparent material, a non-transparent material (e.g., a translucent or opaque material), or a combination thereof. As further described above, if the outer surface is formed of a transparent material covering an underlying non-transparent material, then the transparent material may be thin and transparent enough to be considered to have a negligible effect on the appearance of the object. In such cases, the appearance of the underlying non-transparent material may also be considered as the appearance of the outer surface of the object, such that the 2D image information is considered to represent the appearance of the outer surface of the object.

[0091] Figure 5B An example of 3D image information 5700 is illustrated. More specifically, 3D image information 5700 may include, for example, a depth map or other depth information indicating various locations (such as locations 57001, 57002, ... 5700) within the camera's field of view (e.g., 3210 / 3210A). n This can be a corresponding depth value (which may be a position grid organized into rows and columns). In some implementations, the depth map may include indicators of positions 57001-5700. n The depth value of the pixels. In the embodiment, positions 57001-5700 are... n At least some locations in the text are one or more object surfaces (such as object 35201-3520). n The location on the surface of the object. For example, 3D image information 5700 may include image portions 57201, 57202, 57203, 57204, 57205, ... 5720 n-3 5720 n-2 5720 n-1 5720 n Each image portion may include a corresponding object (e.g., 35201, 35202, 35203, ... or 3520). nThe depth values ​​of the corresponding set of locations on the surface of the object. In some cases, 3D image information may include point clouds, which may include descriptions of locations 57001-5700. n A set of coordinates. The coordinates can be 3D coordinates, such as [XYZ] Cartesian coordinates, and can have values ​​relative to the camera coordinate system or some other coordinate system. In this example, the [XYZ] coordinates of a specific location (e.g., 57001) can have a Z component that is equal to or based on the depth value at that location. The depth value can be relative to the camera that generated the 3D image information (e.g., 1200 / 1200A), or it can be relative to some other reference point.

[0092] In this embodiment, 3D image information can describe the surface contours of an object. For example, Figure 5A The 3D image information 5700 may have at least an image portion 57201 that describes the surface profile of the object surface 35201. The surface profile of the object surface can describe the physical structure of the object surface. In some cases, the physical structure of the object surface may be completely or substantially smooth. In other cases, the physical structure of the object surface may include physical features such as wrinkles, bumps, ridges, creases, or depressions, which may form one or more non-smooth portions of the object surface.

[0093] As described above, the surface of an object can be its outer surface (e.g., top surface) and can be formed of transparent material, opaque material (e.g., translucent or opaque material), or a combination thereof. Further as described above, if the outer surface is formed of a transparent material covering an underlying opaque material, then the transparent material can be thin and flexible enough to be considered to have a negligible effect on the physical structure or surface profile of the object. In such cases, the 3D image information representing the physical structure or surface profile of the underlying opaque material can also be considered to represent the physical structure or surface profile of the object's outer surface. Additionally, if the transparent material is thin enough, then its thickness can be considered to have a negligible effect on the depth measurement of the camera (e.g., 1200). In this case, various locations with depth values ​​represented in the 3D image information (such as the location of image portion 57201) can be considered as locations on the outer surface of the corresponding object (e.g., 35201).

[0094] In embodiments, obtaining image information (which may include object detection and object registration) can be performed by any suitable means. In embodiments, identifying or detecting multiple objects 3520 may include processes such as object registration, template generation, feature extraction, hypothesis generation, hypothesis refinement, and hypothesis verification, as performed, for example, by the object registration module 1130. Such a process is described in detail in U.S. Patent Application No. 17 / 884,081, filed August 9, 2022, the entire contents of which are incorporated herein by reference.

[0095] Object registration is a process that includes acquiring and using object registration data (e.g., known, previously stored information related to object 3520) to generate object recognition templates for identifying and recognizing similar objects in a physical scene. Template generation is a process that includes generating a set of object recognition templates for use by the computing system when identifying object 3520 for further operations related to object picking. Feature extraction (also known as feature generation) is a process that includes extracting or generating features from object image information for use in object recognition template generation. Hypothesis generation is a process that includes generating one or more object detection hypotheses, for example, based on comparisons between object image information and one or more object recognition templates. Hypothesis refinement is the process of matching object recognition templates with object image information (even if the object recognition templates and object image information do not match perfectly). Hypothesis validation is a process by which a single hypothesis is selected from multiple hypotheses as the best fit or best choice for object 3520.

[0096] In operation 4004, method 4000 includes generating surface cost maps of multiple objects 3520 in the scene. The surface cost map may be an image map indicating the smoothness of the surfaces of the collected multiple objects 3520 or a portion of objects 3520. The surface cost map may be an image map identifying irregularities or discontinuities in the surfaces of the collected multiple objects 3520 or a portion of objects 3520. The surface cost map may include surface cost map values ​​for representing each point or pixel of the surfaces or top layer of the collected multiple objects 3520 or a portion thereof. Therefore, the surface cost map can assign a surface cost map value to each point in a point cloud representing the multiple objects 3520 or a portion thereof. As discussed above, each point / pixel in the point cloud can be represented by three coordinates (x, y, z). The surface cost map value represents the difference between a set of points (referred to herein as a kernel or cell) and adjacent kernels. Therefore, the surface cost map value assigned to any point or kernel can represent the difference between that point or kernel and its adjacent points or kernels.

[0097] The surface cost map generated from image information 5700 can represent the height and angle differences between subdivision regions or cells and adjacent subdivision regions or cells. The surface cost map can include a height gradient map and a normal difference map, or can be calculated based on a combination of height gradient maps and normal difference maps. The surface cost map can be calculated or determined using various means to represent the height and angle differences between adjacent portions of 3D image information 5700 representing multiple objects in a scene. In an embodiment, reference... Figure 6A-6I The surface cost map can be calculated as follows.

[0098] Figure 6A A sample flowchart of a surface cost map generation method 6000 is provided. Method 6000 can be executed by any suitable processor or computing device described herein. Figure 6A The steps are provided only through examples. Figure 6A The steps can be performed in any suitable order or combination, and additional steps can be combined as needed. Alternatively, alternative methods for generating surface cost maps can be used without departing from the scope of this disclosure.

[0099] A surface cost map can be generated from 3D image information 5700 to include or provide a combination of a height gradient map and a normal difference map based on several cost map parameters. These cost map parameters, explained in more detail below, may include subdivision region, stride, distance threshold, normal threshold, and normal weight factor. As further described below, the cost map parameters can be determined manually or automatically.

[0100] In operation 6002 of the surface cost map generation method 6000, 3D image information 5700 can be covered by the grid 6100 of cell 6101. Figure 6B and Figure 6C The diagram illustrates the meshing operation of a surface cost map generation method. Cell 6101 can be rectangular or square and its size can be determined based on the subdivision regions. The subdivision regions can represent the size of each cell 6101 in units of points or pixels of the point cloud represented by the 3D image information 5700 (as shown in dimension 6105), such as 2×2, 4×4, 6×6, 8×8, 10×10, 15×15, 20×20, or any other suitable size. Cells 6101 form a mesh on which surface cost map calculations can be performed. In an embodiment, the 3D image information 5700 can be meshed using a single non-overlapping set of cells 6101, such as... Figure 6B As shown in the diagram, each cell center 6102 is spaced one step apart from the others, thus creating a non-overlapping grid. The length of this step (dimension 6106) is equal to the size of the subdivision region.

[0101] In a further embodiment, the grid 6100 covering the 3D image information 5700 may include a set of overlapping cells 6101. Each cell 6101 may overlap with multiple other cells 6101, with cell centers 6102 spaced apart by a stride smaller than the subdivision region size. Thus, for example, as Figure 6C As shown, cell 6101 can separate cell center 6102 by a stride size equal to half the subdivision region size. Figure 6C In the grid 6100, there are cell centers 6102 (each cell center is separated by a stride size) and cells 6101 whose width and length dimensions are twice that stride size. Figure 6C In the diagram, the size of a single cell 6101 is illustrated by the shaded area. Each cell 6101 overlaps with the other four cells 6101.

[0102] In the following discussion of surface cost map calculation, surface cost map values ​​are assigned to cell center 6102, and during the calculation, each cell 6101 is compared with its non-overlapping adjacent cells. Therefore, for clarity, reference will be made to... Figure 6B Non-overlapping arrangement.

[0103] In operation 6004, the surface cost map generation method 6000 may include the step of fitting a plane to each cell 6101. Figure 6D A set of planes 6220 corresponding to grid 6100 is illustrated. For each cell 6101, plane 6201 can be determined based on the x, y, and z coordinates of the points in 3D image information 5700 encompassed by that cell 6101. Therefore, for a subdivided region size of 20×20, 400 points of 3D image information 5700 can be used to determine plane 6201. Plane 6201 can be determined by any suitable method, including, for example, least squares. In another example, plane 6201 can be determined based on the average of the normal vectors at each point within each cell 6101 in 3D image information 5700. Each plane 6201 includes a centroid 6202 and a normal 6203. The centroid 6202 is located at the geometric center of plane 6201, and the normal 6203 extends orthogonally to plane 6201 from the centroid 6202. The height of each plane 6201 can be defined as the height of its centroid 6202.

[0104] In operation 6006, the surface cost map generation method 6000 may include calculating or determining the height gradient of each plane 6201 relative to its neighboring planes 6201. Figure 6FThe illustration shows the height gradient 6200 overlaid on the representation of the source container 3510 containing object 3520. The height gradient of each plane 6201 can be a mathematical combination of the individual height gradients between plane 6201 and its eight adjacent planes 6201. The height gradient of each plane 6201 can be determined in several different ways. For example... Figure 6F As shown, hollow circles represent portions with low height gradients, solid circles represent portions with higher height gradients, and crosses represent portions that cannot be identified as objects, for example, due to unreliable detection or the detection of source container 3510. For illustrative purposes, these values ​​are shown as high and low, but in reality, these values ​​can span a range of possible values. It can be seen that the height gradient at the boundary of object 3520 is greater than the height gradient across the central portion of object 3520.

[0105] In the embodiments, the following references may be made. Figure 6E Determine the cost map height gradient between plane 6201 and its adjacent planes 6201. First, the height difference between the two planes (6201A and 6201B) can be determined. In an embodiment, the height difference between adjacent planes can be based on an extension of one plane 6201B onto another plane 6201A (e.g., an extended plane 6201BA). For example, the height difference can be determined as the height difference between the centroid 6202A of the first extended plane 6201BA and the second plane 6201A, calculated based on the length of the normal vector of either plane or based on the length of the vector in the z-direction of the 3D point cloud. For example, the height difference can be determined as the average height difference between corresponding points on the first extended plane 6201BA and the second plane 6201A, wherein the corresponding points correspond to grid points in the point cloud of the 3D image information 5700. In an embodiment, the height difference between two planes 6201A and 6201B can be determined as the maximum or average height difference between: the height difference determined by extending plane 6201B above (or below) plane 6201A, and the height difference determined by extending plane 6201A above (or below) plane 6201B. This height difference determination method can result in the same height difference regardless of which of the two planes is selected as the "first" plane and which is selected as the "second" plane.

[0106] The height difference between two planes 6201 can be directly assigned to the position between the cells 6101 corresponding to the two planes 6201. For example, the height difference between the planes 6201 corresponding to cells 6101D and 6101E can be assigned (see...). Figure 6BThe position is assigned to point DE. Therefore, since the height difference between planes 6201 corresponding to cells 6101D and 6101E does not correspond to the centroid of plane 6201 corresponding to cell 6101E, a correction can be applied to determine the height difference to be assigned to cell 6101E corresponding to the height difference between planes 6201 corresponding to cells 6101E and 6101D. In an embodiment, this correction can be applied by averaging the height difference assigned to point DE and the height difference assigned to point EF (e.g., the height difference assigned to point EF is based on the height difference between planes 6201 corresponding to cells 6101E and 6101F). The total height gradient of each cell 6101 can be determined as the average of eight height differences with adjacent cells. The total height gradient of each cell 6101 can be assigned as a value associated with the point at the center of that cell in the height gradient cost map 6200.

[0107] In further embodiments, the height difference can be determined using different methods. The height difference can be based, for example, on the height difference between the centroids 6202A / 6202B of planes 6201A / 6201B, or on the height difference (or average height difference) between planar views along the boundaries of cells 6101 corresponding to plane 6201. Other height difference calculations and definitions can be used without departing from the scope of this disclosure.

[0108] about Figure 6B The above discussion refers to the calculation of the height gradient at the center point of each cell 6101 in grid 6100. Because the stride size can be smaller than the subdivision region size, the number of points for which height gradients are calculated can be greater than (even significantly greater than) the number of subdivision regions that can be fitted to grid 6100. For example, for stride size 1, any specific point in the 3D point cloud can have an associated height gradient, each gradient determined based on a grid of cells 6101 of subdivision region size, where that specific point is the center of one of the cells 6101 of subdivision region size. For stride size 2, every other point will have an associated height gradient.

[0109] Therefore, the height gradient cost map 6200 may include a series of values ​​representing the height gradient of a point (in some embodiments, all points) in the 3D point cloud relative to its neighboring points in the 3D point cloud. As discussed above, the points in the height gradient cost map 6200 may be those points in the 3D point cloud image information 5700 that are separated by a frame. For each point in the height gradient cost map 6200 that is assigned a value, the value is calculated based on plane 6201 (whose 2D projection has subdivision region dimensions) and the relationship between that plane and its neighboring planes 6201.

[0110] In embodiments, the calculation of the height gradient cost map 6200 can be simplified or optimized by reusing the height difference between the two planes 6201. For example, in some embodiments, as discussed above, the calculation of the height difference from the first plane 6201 to the second plane 6201 results in the exact same value as the calculation of the height difference between the second plane 6201 and the first plane 6201. Therefore, the height difference between the two planes 6201 can be calculated only once, allowing the total number of height gradient calculations to be reduced by approximately 50%.

[0111] In an embodiment, a distance threshold parameter can be used when determining the height difference. The distance threshold parameter can be a threshold beyond which any height difference is assigned a maximum value. If the height difference between two planes exceeds the distance threshold, then the height difference can be set to a predetermined value (e.g., the distance threshold in some embodiments). Using the distance threshold can reduce the weight of large height differences between two planes when calculating the total height gradient. In an embodiment, the distance threshold parameter can also be used to set a threshold for the height gradient assigned to cell 6101. After averaging the height differences with adjacent cells, if the determined height gradient exceeds the distance threshold, the distance threshold can be applied to change the determined height gradient to a predetermined value.

[0112] In operation 6008 of the surface cost map generation method 6000, the normal difference can be calculated. Figure 6G The illustration shows the normal difference cost map 6300 overlaid on the representation of the source container 3510 containing object 3520. Now refer to... Figure 6D The difference between the normal 6203 of each plane 6201 and the normal 6203 of its neighboring plane 6201 can be determined. The normal difference can be determined as the dot product of the normal 6203 of a plane 6201 and the normal 6203 of its neighboring plane 6201. Therefore, each plane 6201 can have eight different calculated normal differences. The mean of these normal differences can be taken and assigned to the cell 6101 associated with the plane 6201 (e.g., the point at the center of cell 6101). In this way, a normal difference cost map 6300 can be generated, where each point within the surface cost map is assigned a normal difference indicating the angular difference between the plane 6201 centered at that point and its neighboring planes 6201. Figure 6G As shown, hollow circles indicate regions with low normal differences, solid circles indicate regions with larger normal differences, and crosses indicate regions that cannot be identified as objects, for example, due to unreliable detection or the detection of source container 3510. For illustrative purposes, these values ​​are shown as high and low, but in reality, these values ​​can span a range of possible values. It can be seen that the normal difference at the boundary of object 3520 is greater than the normal difference across the central portion of object 3520.

[0113] In an embodiment, a normal threshold parameter can be used when determining the normal difference. The normal threshold parameter can be a threshold beyond which any height difference is assigned a maximum value. If the normal difference between two planes exceeds the normal threshold, then the normal difference can be set to a predetermined value (e.g., the normal threshold in some embodiments). Using the normal threshold can reduce the weight of large normal differences between two planes when calculating the average normal difference.

[0114] In operation 6010 of the surface cost map generation method 6000, a surface cost map can be generated. Figure 6H The illustration shows a surface cost map 6400 overlaid on a representation of a source container 3510 containing object 3520. The surface cost map 6400 can be generated as a mathematical combination of a height gradient cost map 6200 and a normal difference cost map 6300. In an embodiment, a computer system can combine height difference values ​​with normal difference values ​​based on filtering operations such as averaging filters or SOBEL filters. In an embodiment, values ​​in the height gradient cost map 6200 and normal difference cost map 6300 can be normalized and combined. In an embodiment, a weighting factor can be applied to the height difference or normal difference value to control the strength of the surface cost map's dependence on the corresponding difference. The weighting factor can be a normal weighting factor, for example, a factor multiplied by the normalized normal difference to determine the strength of the final surface cost map 6400 as determined by the normal difference or the strength of the final surface cost map 6400 as determined by the height difference. As discussed below, the selection of the normal weighting factor can be performed based on the expected object type. Figure 6H As shown, hollow circles indicate portions with low surface cost map values, solid circles indicate portions with larger surface cost map values, and crosses indicate portions that cannot be identified as objects, for example, due to unreliable detection or detection of source container 3510. For illustrative purposes, the values ​​are shown as high and low, but in reality, these values ​​can span a range of possible values. It can be seen that the surface cost map value at the boundary of object 3520 is greater than the surface cost map value across the central portion of object 3520.

[0115] As discussed above, surface cost map generation can be performed based on one or more parameters, including subdivision size, stride size, distance threshold, normal threshold, and normal weight.

[0116] The subdivision region size and stride size can be selected or determined based on various factors to achieve various results. In one embodiment, a smaller subdivision region size can be selected to provide results more sensitive to subtle changes in the 3D point cloud; however, a smaller subdivision region size will also be more sensitive to noise. In another embodiment, a larger subdivision region size can be selected to smooth out small variations in the 3D point cloud, whether these variations are due to noise or variations in the actual object being imaged. In another embodiment, a small stride size can be selected to provide a high-resolution, detailed surface cost map, although such a small stride size may require increased computing power and / or increased processing time. In another embodiment, a larger stride size can lead to downsampling of the 3D point cloud, which can provide faster results and / or lower computational resource usage, but at the cost of some detail. In another embodiment, selecting a stride size less than 0.5, less than 0.4, and / or less than 0.3 of the subdivision region size can provide an appropriate amount of detail while still providing faster results. In yet another embodiment, a stride size of half or approximately half the subdivision region size can provide a balance between reduced resolution, speed, and level of detail. It is understandable that the choice of subdivision region and stride size will be influenced by the availability of processing or computing power. Increased computing resources can allow for the generation of more detailed surface cost maps without adversely increasing processing time.

[0117] In this embodiment, the composition of objects in the object source affects the optimal values ​​for the subdivision region size and stride size. For example, a collection of objects with small, sharp discontinuities may benefit from a smaller stride size to capture finer details. In another example, a collection of objects with rough but deformable features may benefit from a larger subdivision region size to provide greater smoothness. In yet another example, if the subdivision region size is large compared to the object size (e.g., the object size is only 2, 3, or 4 times the subdivision region size), the surface cost map may include very few smooth areas because many subdivision regions covering the object will also overlap with the edges of objects with discontinuities. In yet another example, if both the subdivision region size and stride size are too large, objects with smooth, curved surfaces of small radii may result in incorrectly high costs.

[0118] In the embodiments, the composition of objects in the object source also affects the optimal values ​​used for the distance threshold, normal threshold, and normal weight factor. For example, now referring to Figure 6IConsider objects 6500 (like a box) and 6501 (like a bag) (these are examples of object 3520). The central portion of objects 6500 / 6501 has a smoothness property that describes the overall or main smoothness of objects 6500 / 6501, while the edges of objects 6500 / 6501 describe the transitions between objects 6500 / 6501. Therefore, it is advantageous to choose parameters that can take advantage of this.

[0119] For example, a distance threshold can be selected based on the object size. Any detected height difference equal to or greater than the distance threshold is set to the maximum value of the height difference. Therefore, a height difference at the edge of object 6500 / 6501 may have the same effect on surface cost map 6400, regardless of whether the object is on top of a stack of several objects or is a single object. A larger height drop at the edge of object 6500 / 6501 (e.g., because object 6500 / 6501 is stacked on top of other objects 6500 / 6501) does not provide any additional information for identifying object transitions.

[0120] In another example, the normal threshold can be chosen based on the object's shape. For instance, for a box-like object 6500, the expected normals will have low variation. In this case, the normal threshold can be chosen to be greater than the expected variation caused by noise. Therefore, in the normal difference cost map, any normal difference identified as greater than any normal difference caused by noise difference is set to the maximum value. In a box-like object 6500, since all object surfaces are likely to be flat, any variation in the normals that can be identified as real (because it exceeds the noise value) can represent discontinuities between objects 6500. For such objects 6500, a normal weighting factor can also be chosen to give approximately equal weight to normal differences and height differences. In another example, a bag-like object (such as object 6501) may have sections with significantly changed angles, which does not represent object discontinuities. In this case, a normal weighting factor can be chosen to give greater weight to height differences, since normal differences provide less information about object discontinuities. In another example, a deformable bag can be expected to have large variations in its normals, so the normal weighting factor can be chosen to give greater weight to the height difference, since the difference in normals provides very little information about the discontinuity of the object.

[0121] As discussed above, different parameters can provide better or worse results in surface cost map generation depending on the type and size of the objects in the source container. In embodiments, surface cost map generation parameters can be manually selected, for example, based on the expected type and size of the objects in the source container. In further embodiments, parameter selection can be automated and can be performed based on, for example, the acquired 2D image information 2600 and / or the acquired 3D image information 5700. As discussed above, object detection (including, for example, object registration) can be performed on the acquired 2D image information 2600 and / or the acquired 3D image information 5700 to identify the size, shape, and / or type of objects in the source container. Based on object detection (e.g., object registration), surface cost map generation parameters, including subdivision region size, stride size, distance threshold, normal threshold, and normal weight factor, can be automatically selected.

[0122] In embodiments that include a source container with multiple different types of objects, the distance threshold, normal threshold, and normal weight factor can be adjusted within the surface cost map for regions associated with different types of registered objects.

[0123] Now return to Figure 4 In operation 4006, method 4000 includes segmentation of image information (e.g., 2D image information 2600 and / or 3D image information 5700). Segmentation can be performed based on a surface cost map 6400 generated by the methods described above or by any suitable method. The segmented image information can provide multiple image fragments, which use the values ​​of the surface cost map 6400 to identify individual objects in the scene. (Regarding...) Figures 7A-7E An image segmentation process according to an embodiment is described.

[0124] In operation 7002 of image segmentation method 7000, initial segmentation of the surface cost map may be performed by applying a cost threshold. The cost threshold generates threshold boundaries 7102 between the object portions 7101 in the thresholded mask 7100, such as... Figure 7B As shown in the diagram. Threshold boundary 7102 represents a region with a surface cost map value exceeding the threshold, while object portion 7101 represents a region with a surface cost map value not exceeding the threshold. Therefore, threshold boundary 7102 can be represented by a "false" value in the thresholded mask 7100, while object portion 7101 is represented by a "true" value. The assignment of "false" and "true" values ​​is merely a convention, and any suitable distinction can be applied. Object portion 7101 represents a first estimate of the object surface, while threshold boundary 7102 represents a first estimate of the object boundary or discontinuity. Object boundary 7103 represents the actual object boundary and is provided for comparison purposes.

[0125] In operation 7004 of image segmentation method 7000, a thresholded mask 7100 can be further defined in the mask definition operation. The mask definition operation may include one or more of connected component analysis and mask erosion, as described above. Figure 7C The thresholded mask 7100 can be further defined to generate a defined mask 7200.

[0126] Generating a defined mask 7200 may include mask erosion performed on a thresholded mask 7100. Mask erosion is an operation that reduces or erodes the boundaries of a mask based on a structured element. A structured element may represent, for example, an N×N group of pixels or points with an output pixel / point, which may be located at the center of the structured element. When placed on a mask, if every point in the mask that coincides with a point in the structured element is true, then the output point of the structured element in the eroded mask is set to true. Therefore, for points in the eroded mask to be true, every surrounding point in the original mask, up to the size of the structured element, must also be true. Thus, erosion has the effect of eliminating one or more layers of points at the mask edges and smoothing any irregularities in the mask. In the example, mask erosion may be performed on a thresholded mask 7100 using a structured element that is half the size of the minimum pickable area (e.g., the minimum area size that can be gripped by a robotic arm, for example, it could be the size required for a suction gripper to achieve a firm grip). This erosion operation can therefore be used to disconnect any part of a mask smaller than the minimum pickable area size.

[0127] In operation 7006 of image segmentation method 7000, object regions can be identified within a defined cost mask 7200. (See also...) Figure 7C Connectivity component analysis can be performed on the defined cost mask 7200 to identify object regions 7201 located within the defined cost mask 7200. Object regions 7201 can represent a more refined estimate of object location and boundaries than the object portion 7101 previously discussed.

[0128] In operation 7008 of image segmentation method 7000, image fragments 7301 from object region 7201 can be selected and further defined. Now refer to Figure 7C and Figure 7DImage fragment 7301 can be selected as an object region 7201 containing a seed 7204. The seed 7204 can be a point with the lowest cost in the surface cost map (e.g., the smoothest point, which is least likely to represent a boundary or discontinuity). A fragment map 7300 containing image fragment 7301 can be generated by removing all object regions 7201 that do not contain the seed. Figure 7D The image fragment 7301 can then be dilated using a structured element corresponding to half the size of the smallest pickable region. Dilation is the opposite of erosion. During dilation, the output pixels / points of the structured element become the input points. When overlaid on fragment image 7300, if the points in fragment image 7300 corresponding to the input points of the structured element are true, then all points in fragment image 7300 corresponding to the structured element are set to true. Dilation has the effect of extending the boundary of image fragment 7301 by an amount corresponding to the size of the structured element.

[0129] In operation 7010 of image segmentation method 7000, image fragment 7301 can be verified. Verification of image fragment 7301 can be performed to determine whether the identified image fragment 7301 represents a feasible object among multiple objects. A bounding box 7305 (e.g., a square or rectangular box) can be fitted around the identified image fragment 7301. The bounding box 7305 can then be compared with the maximum and minimum candidate object sizes. The maximum and minimum candidate object sizes represent the maximum and minimum possible object sizes determined during the object detection process. If the bounding box is larger than the maximum candidate object size or smaller than the minimum candidate object size, then image fragment 7301 can be determined as invalid, requiring iterations of operations 7002, 7004, 7006, and 7008. If the bounding box is larger than the maximum candidate object size, then iterations can be performed using a reduced cost threshold. If the bounding box is smaller than the minimum candidate object size, then iterations can be performed using an increased cost threshold.

[0130] In an embodiment, the bounding box can also be compared to a desired minimum pickable region size. The minimum pickable region size can correspond to the smallest possible region size that can be picked up, such as the size of a single suction gripper of the robotic arm. In an embodiment, the robotic arm can employ more than one suction gripper, such as two or four. The desired minimum pickable region size can be a parameter corresponding to the size of the region necessary to achieve the selected or desired grip (e.g., the region necessary for two or four suction grippers to achieve the grip). If the bounding box is smaller than the desired minimum pickable region size, then operations 7002, 7004, 7006, and 7008 can be iterated using an increased threshold.

[0131] After image fragment 7301 has been verified, it can be stored as a pickable region for further analysis. Image fragment 7301 can then be removed from the surface cost map 6400, and operations 7002-7010 can be repeated to identify additional image fragments 7301. In an embodiment, a cost threshold can be increased before repeating operations 7002-7010. Method 7000 can be repeated, and the cost threshold can be increased until no additional fragments are detected or identified. Figure 7E The illustration shows a set of image segments 7301 identified from the surface cost map 6400. In an embodiment, the identified image segments 7301 may be designated as pickable regions. In an embodiment, the identified image segments 7301 may be further analyzed to determine pickable regions therein.

[0132] In operation 4008, method 4000 includes generating a detection mask. The detection mask can be generated to refine or further define the potential pickable regions of objects corresponding to the image segment 7301 determined from image segmentation operation 4006.

[0133] For example, such as Figure 8A As shown, because the bounding box of operation 7010 is a two-dimensional construction, it may not accurately correspond to the actual height of points on the object. Figure 8A In the process, bounding box 8021 has been fitted to object 8022. However, due to the deformable nature of object 8022, not all actual points 8023 on the surface of object 8022 fall within bounding box 8021. Therefore, in operation 4008, detection mask information can be generated to identify the more or less suitable parts of the object within the bounding box for object picking.

[0134] Figure 8B The illustration shows detection mask information 8300. Detection mask information 8300 may include information about objects within bounding box 8021 (e.g., a bounding box generated during operation 7010 for image segment 7301). Detection mask information 8300 includes identified regions 8024 and 8027 and an unidentified region 8026. Identified regions 8024 and 8027 may include detected region 8024 (which includes detected and unoccluded areas) and occluded region 8027. For object picking, occluded region 8027 may be unsafe or useless, while detected region 8024 may be safe for picking. Unidentified region 8026 may include regions that are not identified for either occlusion or picking and are generally not used for detection or are not relied upon for detection. Figure 8BThe figure also illustrates the minimum pickable region 8025. As shown, it can be seen that the detected region 8024, marked "B", is not large enough to accommodate the minimum pickable region 8025. Therefore, the detection mask information 8300 can be used in conjunction with the aforementioned image segmentation techniques to identify the pickable region of the object.

[0135] In operation 4010, method 4000 may include determining a safety volume to be used in the motion planning operation. The safety volume may represent the volume that an object selected for pickup might occupy. The safety volume is selected to reduce the likelihood of the selected object colliding with other objects in the object disposal environment once picked up.

[0136] Now for reference Figure 9A A safety volume 9100 is provided around the pickable area 9201, which is designated as the pickable region of object 3520. The safety volume can be determined as a size having twice the difference between the designated pickable area 9201 and the expected object size. This safety volume size thus creates a volume around the pickable area 9201 that can provide an error margin for the object's potential dimensions, for example, if the pickable area 9201 is not located at the center of the object 3520 to be picked. The size of the safety volume 9100 can then be modified as follows.

[0137] First, the safe volume 9100 is compared to the 3D point cloud. If the 3D point cloud does not support the size of the safe volume 9100 (e.g., the safe volume 9100 is too large and extends beyond the boundaries of the 3D point cloud, which correspond to the boundaries of the source container 3510), then the size of the safe volume 9100 can be reduced to a size supported by the 3D point cloud. The safe volume 9100 can then be aligned with the edges of the 3D point cloud. Figure 9B The illustration shows the case where the safety volume 9100 is reduced to the safety volume 9101, because the boundary of the safety volume 9100 extends beyond the 3D point cloud associated with the source container 3511.

[0138] If the safety volume 9100 / 9101 is greater than the maximum permissible size specified for the destination container, then the safety volume 9100 / 9101 can be further reduced. For example, if the destination container is smaller than the source container, then the safety volume 9100 / 9101 may be too large for the destination container. The safety volume 9100 / 9101 can therefore be reduced or adjusted accordingly. In an embodiment, if the safety volume 9100 / 9101 is greater than the destination container and cannot be adjusted to a size smaller than the destination container, then if it is known that the object 3520 can be fitted into the target container, then a motion plan that takes into account this uncertainty can be generated.

[0139] If the detection bounding box of operation 7010 extends beyond the safety volume 9100 / 9101, then the safety volume 9100 / 9101 can be further adjusted. This may occur, for example, due to shrinking or realigning the safety volume as described above, or if the bounding box is arranged in an inconvenient manner relative to the pickable area 9201 that forms the basis of the safety volume 9100 / 9101. In an embodiment, to address this issue, the safety volume 9100 / 9101 can be moved to include the bounding box, or the bounding box can be moved and aligned to the safety volume 9100 / 9101.

[0140] In operation 4012, method 4000 includes outputting a pickable region detection result. The pickable region detection result may include any or all information generated in operations 4002-4010, including, for example, identified image segments 7301, their associated bounding boxes 7305, identified pickable regions 9201, and safe volumes 9100 / 9101. The pickable region detection result may include pickable region detection result information regarding any or all detected objects 3520 within the source container 3510.

[0141] In operation 4014, method 4000 may include generating and / or outputting motion planning based on the detection results of the pickable area. The motion planning may include robot instructions for following a trajectory, grasping or picking up object 3520 through the identified pickable area 9201, and transporting object 3520 to a destination container, while considering potential collisions based on the determined safe volume 9100 / 9101 of object 3520.

[0142] It will be apparent to those skilled in the art that other suitable modifications and adaptations can be made to the methods and applications described herein without departing from the scope of any of the embodiments. The above embodiments are illustrative examples and should not be construed as limiting this disclosure to these specific embodiments. It should be understood that the various embodiments disclosed herein can be combined in combinations different from those specifically presented in the specification and drawings. It should also be understood that, according to examples, certain actions or events of any process or method described herein may be performed in a different order, and may be added, combined, or omitted entirely (e.g., all described actions or events may not be necessary for performing the method or process). Furthermore, while certain features of the embodiments herein are described for clarity, they are performed by a single component, module, or unit; it should be understood that the features and functions described herein can be performed by any combination of components, units, or modules. Therefore, various changes and modifications can be made by those skilled in the art without departing from the spirit or scope of the invention as defined by the appended claims.

[0143] Further embodiments may include:

[0144] Example 1 is a computing system comprising: a control system configured to communicate with a robot having a robotic arm and a camera, the robotic arm including or attached to an end effector; and at least one processing circuit configured to perform the following operations when the robot is located in an object disposal environment including an object source for transfer to a destination within the object disposal environment: acquiring image information of the object; identifying pickable regions of one or more selected objects among the objects by: generating a surface cost map based on the image information; segmenting the surface cost map to obtain one or more image segments, the one or more image segments identifying one or more pickable regions corresponding to the one or more selected objects; generating a pickable region detection result that includes at least the one or more pickable regions; and generating a motion plan for the robot system to transfer the one or more selected objects, the motion plan being based on the pickable region detection result.

[0145] Example 2 is the system of Example 1, wherein the surface cost map represents the smoothness of the one or more selected objects.

[0146] Example 3 is a system of Example 1 or 2, wherein the image information includes three-dimensional information, and the processing circuit is further configured to generate the surface cost map based on the height gradient and normal difference between defined cells in the image information.

[0147] Example 4 is a system of any one of Examples 1 to 3, wherein the at least one processing circuit is further configured to generate the surface cost map based on the surface cost map parameters.

[0148] Example 5 is a system of any one of Examples 1 to 4, wherein the at least one processing circuit is further configured to: register the one or more objects based on the image information to create object registration information; and determine the surface cost map parameters based on the object registration information.

[0149] Example 6 is a system of any one of Examples 1 to 5, wherein the at least one processing circuit is further configured to generate detection mask information indicating the one or more pickable regions of the image segment, the detection mask information including detected regions and occluded regions within the one or more image segments.

[0150] Example 7 is a system of any one of Examples 1 to 6, wherein segmenting the surface cost map includes: applying a cost threshold to the surface cost map to generate a thresholded mask; eroding the thresholded mask to generate an eroded mask; and applying connectivity component analysis to the eroded mask to identify a first image segment.

[0151] Example 8 is a system of any one of Examples 1 to 7, wherein segmenting the surface cost map further includes: removing the first image segment from the surface cost map; applying a second cost threshold to the remainder of the surface cost map to generate a second thresholded mask; eroding the second thresholded mask to generate a second eroded mask; and applying connectivity component analysis to the second eroded mask to identify the second image segment.

[0152] Example 9 is a system of any one of Examples 1 to 8, wherein generating the pickable region detection result further includes: generating a safety volume around the one or more pickable regions, the safety volume indicating the estimated remaining portion of the one or more selected objects.

[0153] Example 10 is an object transfer method executed by a control system having at least one processing circuit and configured to communicate with a robot having a robotic arm and a camera, the robotic arm including or attached to an end effector device. The method includes: obtaining image information of one or more objects contained in an object source; identifying pickable regions of one or more selected objects among the selected objects by: generating a surface cost map based on the image information; segmenting the surface cost map to obtain one or more image segments, the one or more image segments identifying one or more pickable regions corresponding to the one or more selected objects; generating a pickable region detection result that includes at least the one or more pickable regions; and generating a motion plan for a robot system to transfer the one or more selected objects, the motion plan being based on the pickable region detection result.

[0154] Example 11 is the method of Example 10, wherein the surface cost map represents the smoothness of the one or more selected objects.

[0155] Example 12 is a method of Example 10 or 11, wherein the image information includes three-dimensional information, and the method further includes generating the surface cost map based on the height gradient and normal difference between defined cells in the image information.

[0156] Example 13 is a method of any one of Examples 10 to 12, and further includes generating the surface cost map based on surface cost map parameters.

[0157] Example 14 is a method of any one of Examples 10 to 13, further comprising: registering the one or more objects based on the image information to create object registration information; and determining the surface cost map parameters based on the object registration information.

[0158] Example 15 is a method of any one of Examples 10 to 14, further comprising generating detection mask information indicating the one or more pickable regions of the image segment, the detection mask information including detected regions and occluded regions within the one or more image segments.

[0159] Example 16 is a method of any one of Examples 10 to 15, wherein segmenting the surface cost map includes: applying a cost threshold to the surface cost map to generate a thresholded mask; eroding the thresholded mask to generate an eroded mask; and applying connectivity component analysis to the eroded mask to identify a first image segment.

[0160] Example 17 is a method of any one of Examples 10 to 16, wherein segmenting the surface cost map further includes: removing the first image segment from the surface cost map; applying a second cost threshold to the remainder of the surface cost map to generate a second thresholded mask; eroding the second thresholded mask to generate a second eroded mask; and applying connectivity component analysis to the second eroded mask to identify the second image segment.

[0161] Example 18 is a method of any one of Examples 10 to 17, wherein generating the pickable region detection result further includes: generating a safety volume around the one or more pickable regions, the safety volume indicating the estimated remaining portion of the one or more selected objects.

[0162] Example 19 is a non-transitory computer-readable medium configured with executable instructions for object transport performed by a control system having at least one processing circuit and configured to communicate with a robot having a robotic arm and a camera, the robotic arm including or attached to an end effector device. The instructions are configured to: obtain image information of one or more objects contained in an object source; identify pickable regions of one or more selected objects from the selected objects by: generating a surface cost map based on the image information; segmenting the surface cost map to obtain one or more image segments, the one or more image segments identifying one or more pickable regions corresponding to the one or more selected objects; generating a pickable region detection result that includes at least the one or more pickable regions; and generating a motion plan for the robot system to transport the one or more selected objects, the motion plan being based on the pickable region detection result.

[0163] Example 20 is a non-transitory computer-readable medium of Example 19, wherein the image information includes three-dimensional information, and the instructions are further configured to generate the surface cost map based on the height gradient and normal difference between defined cells in the image information.

Claims

1. A computing system, comprising: The control system is configured to communicate with a robot having a robotic arm and a camera, the robotic arm including or attached to an end effector device. At least one processing circuit is configured to perform the following operations when the robot is in the object disposal environment, which includes an object source for delivery to a destination within the object disposal environment: Obtain image information of the object, including three-dimensional information; The following steps are used to identify the pickable region of one or more selected objects within the object: A surface cost map is generated based on the defined height gradient and normal differences between cells in the image information. Segmenting the surface cost map to obtain one or more image fragments, the one or more image fragments identifying one or more pickable regions corresponding to the one or more selected objects, wherein segmenting the surface cost map includes: A cost threshold is applied to the surface cost map to generate a thresholded mask; The thresholded mask is eroded by reducing the boundary of the thresholded mask to generate an eroded mask; Connectivity component analysis is applied to the eroded mask to identify the first image fragment; and The first image segment is dilated using structured elements selected based on the minimum pickable region size to define one or more pickable regions; and Generate a pickable region detection result that includes at least one or more of the pickable regions; and Generate a motion plan for the robot system to transport the one or more selected objects, the motion plan being based on the detection results of the pickable area.

2. The system as claimed in claim 1, wherein, The surface cost map represents the smoothness of the one or more selected objects.

3. The system as described in claim 1, wherein, The at least one processing circuit is further configured to verify the one or more pickable regions by: The bounding box that identifies the one or more pickable regions, and The bounding box is compared with one or more of the minimum object candidate size, the maximum object candidate size, and the minimum pickable region size.

4. The system as described in claim 3, wherein, The at least one processing circuit is further configured to generate the surface cost map according to surface cost map generation parameters that define parameters for generating the surface cost map.

5. The system as described in claim 4, wherein, The at least one processing circuit is further configured to: Register the one or more objects based on the image information to create object registration information; and The surface cost map generation parameters are determined based on the object registration information.

6. The system of claim 1, wherein, The at least one processing circuit is further configured to: Generate detection mask information indicating one or more pickable regions of an image segment, the detection mask information including unoccluded regions and occluded regions detected within the one or more image segments; as well as The one or more pickable regions are compared with the minimum pickable region size to identify the suitability of object picking.

7. The system as claimed in claim 1, wherein, The surface cost map includes values ​​corresponding to points in the image information.

8. The system as claimed in claim 1, wherein, The segmentation of the surface cost map also includes: Remove the first image fragment from the surface cost map; A second cost threshold higher than the stated cost threshold is applied to the remainder of the surface cost map to generate a second thresholded mask; Eroding a second thresholded mask to generate a second eroded mask; and Connectivity component analysis is applied to the second eroded mask to identify the second image fragment.

9. The system of claim 1, wherein, Generating the pickable region detection result also includes: Generate a safety volume around the one or more pickable regions, the safety volume indicating that the one or more selected objects extend beyond the estimated range of the one or more pickable regions.

10. A method for transferring an object performed by a control system, the control system having at least one processing circuit and configured to communicate with a robot having a robotic arm and a camera, the robotic arm including or attached to an end effector device, the method comprising: Obtain image information of the object, including its three-dimensional information; The following steps are used to identify the pickable region of one or more selected objects within the object: A surface cost map is generated based on the defined height gradient and normal differences between cells in the image information. Segmenting the surface cost map to obtain one or more image fragments, the one or more image fragments identifying one or more pickable regions corresponding to the one or more selected objects, wherein segmenting the surface cost map includes: A cost threshold is applied to the surface cost map to generate a thresholded mask; The thresholded mask is eroded by reducing the boundary of the thresholded mask to generate an eroded mask; Connectivity component analysis is applied to the eroded mask to identify the first image fragment; and The first image segment is dilated using structured elements selected based on the minimum pickable region size to define one or more pickable regions; Generate a pickable region detection result that includes at least one or more of the pickable regions; and Generate a motion plan for the robot system to transport the one or more selected objects, the motion plan being based on the detection results of the pickable area.

11. The method of claim 10, wherein, The surface cost map represents the smoothness of the one or more selected objects.

12. The method of claim 10, further comprising verifying the one or more pickable regions by: The bounding box that identifies the one or more pickable regions, and The bounding box is compared with one or more of the minimum object candidate size, the maximum object candidate size, and the minimum pickable region size.

13. The method of claim 10, further comprising generating the surface cost map based on surface cost map generation parameters.

14. The method of claim 13, further comprising: Register the one or more objects based on the image information to create object registration information; as well as The surface cost map generation parameters are determined based on the object registration information.

15. The method of claim 10, further comprising: Generate detection mask information indicating one or more pickable regions of the image segment, the detection mask information including detected unoccluded regions and detected occluded regions within the one or more image segments; as well as The one or more pickable regions are compared with the minimum pickable region size to identify the suitability of object picking.

16. The method of claim 10, wherein, The surface cost map includes values ​​corresponding to points in the image information.

17. The method of claim 10, wherein, The segmentation of the surface cost map also includes: Remove the first image fragment from the surface cost map; A second cost threshold higher than the stated cost threshold is applied to the remainder of the surface cost map to generate a second thresholded mask; Eroding the second thresholded mask to generate a second eroded mask; and Connectivity component analysis is applied to the second eroded mask to identify the second image fragment.

18. The method of claim 10, wherein, Generating the pickable region detection result also includes: Generate a safety volume around the one or more pickable regions, the safety volume indicating that the one or more selected objects extend beyond the estimated range of the one or more pickable regions.

19. A non-transitory computer-readable medium configured with executable instructions for object transfer, said object transfer being performed by a control system having at least one processing circuit and configured to communicate with a robot having a robotic arm and a camera, said robotic arm including or attached to an end effector device, the instructions being configured to: Obtain image information of the object, including its three-dimensional information; The following steps are used to identify the pickable region of one or more selected objects within the object: A surface cost map is generated based on the defined height gradient and normal differences between cells in the image information. Segmenting the surface cost map to obtain one or more image fragments, the one or more image fragments identifying one or more pickable regions corresponding to the one or more selected objects, wherein segmenting the surface cost map includes: A cost threshold is applied to the surface cost map to generate a thresholded mask; The thresholded mask is eroded by reducing the boundary of the thresholded mask to generate an eroded mask; Connectivity component analysis is applied to the eroded mask to identify the first image fragment; and The first image segment is dilated using structured elements selected based on the minimum pickable region size to define one or more pickable regions; Generate a pickable region detection result that includes at least one or more of the pickable regions; and Generate a motion plan for the robot system to transport the one or more selected objects, the motion plan being based on the detection results of the pickable area.

20. The non-transitory computer-readable medium of claim 19, wherein, The instructions are further configured to generate the surface cost map according to surface cost map generation parameters defined for generating the surface cost map.

21. The non-transitory computer-readable medium of claim 20, wherein, The instruction is further configured to: Register the one or more objects based on the image information to create object registration information; and The surface cost map generation parameters are determined based on the object registration information.

22. The non-transitory computer-readable medium of claim 19, wherein, The instruction is further configured to: Generate detection mask information indicating one or more pickable regions of an image segment, the detection mask information including unoccluded regions and occluded regions detected within the one or more image segments; as well as The one or more pickable regions are compared with the minimum pickable region size to identify the suitability of object picking.

23. The non-transitory computer-readable medium of claim 19, wherein, The instructions are further configured to segment the surface cost map by also performing the following operations: Remove the first image fragment from the surface cost map; A second cost threshold higher than the stated cost threshold is applied to the remainder of the surface cost map to generate a second thresholded mask; Erosion of a second thresholded mask to generate a second eroded mask; as well as Connectivity component analysis is applied to the second eroded mask to identify the second image fragment.

24. The non-transitory computer-readable medium of claim 19, wherein, The instructions are further configured to verify the one or more pickable regions by: The bounding box that identifies the one or more pickable regions, and The bounding box is compared with one or more of the minimum object candidate size, the maximum object candidate size, and the minimum pickable region size.

25. The non-transitory computer-readable medium of claim 19, wherein, The surface cost map includes values ​​corresponding to points in the image information.

Images