Robotic system and method with object detection

Abstract

The invention relates to a robotic system and method with object detection. A computing system includes a processor configured to identify a contour of a stack of objects based on obtained 3D image information, identify a top surface of the stack of objects based on the contour of the stack of objects, detect an extension line extending from a perimeter to a center of the top surface based on 2D image information of the top surface of the stack of objects, and determine a contour of the stack of objects based on the extension line. Each extension line corresponds to an edge of an object or a gap between objects, detecting a presence of a layer divider based on a length of each extension line being less than a line length threshold, identifying a contour of the layer divider from a point on the top surface corresponding to a termination point of the extension line, and determining a trajectory to control movement of the robot based on the contour.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 524,668, entitled “ROBOTIC SYSTEM WITH OBJECT DETECTION” and filed on July 2, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present technology is generally directed to robotic systems, and more specifically, to systems, processes and techniques for identifying and detecting objects. More particularly, the present technology enables identification and detection of layer separators between layers of objects arranged on a loading or transport platform. Background Art

[0004] With the ever-increasing performance and decreasing cost of robots, many robots (e.g., machines configured to automatically / autonomously perform physical actions) are now widely used in a variety of different fields. Robots can be used, for example, to perform various tasks (e.g., manipulating or conveying objects through space) in manufacturing and / or assembly, packaging and / or packaging, transportation and / or shipping, etc. In performing tasks, robots can replicate human actions, thereby replacing or reducing human involvement that would otherwise be required in performing dangerous or repetitive tasks.

[0005] Despite advances in technology, robots often lack the sophistication to replicate human interactions to perform larger and / or more complex tasks. Thus, there remains a need for improved techniques and systems for managing operations and / or interactions between robots. Summary of the invention

[0006] In some aspects, a computing system for detecting a layer divider between a stack of objects is provided, comprising a control system configured to communicate with a robot having at least one camera and a robotic arm including an end effector device; and a processing circuit configured to identify at least one edge and an outline of the stack of objects based on 3D image information obtained from at least one image captured by the at least one camera, identify a top surface of the stack of objects based on the outline and at least one edge of the stack of objects, detect extension lines extending from a perimeter of the top surface to a center of the top surface based on 2D image information of the top surface of the stack of objects, each extension line corresponding to at least one of a gap between objects in the stack of objects or an edge of an object, detect the presence of a layer divider based on a length of each extension line being less than a line length threshold, identify an outline of the layer divider based on a point on the top surface corresponding to a termination point of the extension line, and determine a trajectory based on the outline of the layer divider to control movement of the robotic arm.

[0007] In some aspects, a computing system for detecting a layer divider between a stack of objects is provided, including a control system configured to communicate with a robot having at least one camera and a robotic arm including an end effector device, and processing circuitry configured to identify at least one edge and an outline of the stack of objects based on 3D image information obtained from at least one image captured by the at least one camera, identify a top surface of the stack of objects based on the outline and at least one edge of the stack of objects, detect extension lines extending from a perimeter of the top surface to a center of the top surface based on 2D image information of the top surface of the stack of objects, each extension line corresponding to at least one of a gap between objects in the stack of objects or an edge of an object, and detect the presence of a layer divider in response to a length of each extension line being less than a line length threshold.

[0008] In some aspects, a computer processor-implemented method for detecting a layer divider between a stack of objects is provided, comprising: identifying at least one edge and an outline of the stack of objects based on 3D image information obtained from at least one image captured by at least one camera, identifying a top surface of the stack of objects based on the outline and at least one edge of the stack of objects, detecting extension lines extending from a perimeter of the top surface to a center of the top surface based on 2D image information of the top surface of the stack of objects, each extension line corresponding to at least one of a gap between objects in the stack of objects or an edge of the objects, detecting the presence of a layer divider based on a length of each extension line being less than a line length threshold, identifying an outline of the layer divider based on a point on the top surface corresponding to a termination point of the extension line, and identifying a trajectory based on the outline of the layer divider to control movement of a robot arm of a robot.

[0009] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] This Summary is provided to introduce some concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0011] Figure 1A An example of a system for detecting, identifying, and retrieving objects is illustrated in accordance with one or more embodiments.

[0012] Figure 1B An example of a system for detecting, identifying, and retrieving objects is illustrated in accordance with one or more embodiments.

[0013] Figure 1CAn example of a system for detecting, identifying, and retrieving objects is illustrated in accordance with one or more embodiments.

[0014] Figure 1D An example of a system for detecting, identifying, and retrieving objects is illustrated in accordance with one or more embodiments.

[0015] Figure 2A An example of a computing system configured to detect, identify, and retrieve objects is illustrated in accordance with one or more embodiments.

[0016] Figure 2B An example of a computing system configured to detect, identify, and retrieve objects is illustrated in accordance with one or more embodiments.

[0017] Figure 2C An example of a computing system configured to detect, identify, and retrieve objects is illustrated in accordance with one or more embodiments.

[0018] Figure 2D An example of a computing system configured to detect, identify, and retrieve objects is illustrated in accordance with one or more embodiments.

[0019] Figure 2E Illustrated are examples of image information processed by the systems and methods disclosed herein according to one or more embodiments.

[0020] Figure 2F is an example of image information processed by the systems and methods disclosed herein, according to one or more embodiments.

[0021] Figure 3A Illustrated are examples of environments for operating a robotic system in accordance with one or more embodiments.

[0022] Figure 3B Illustrated is an example of an environment for detecting, identifying, and retrieving objects by a robotic system in accordance with one or more embodiments.

[0023] Figure 4A

[0013] An example of a method for detecting, identifying, and retrieving layer dividers by a robotic system in accordance with one or more embodiments is illustrated.

[0024] Figure 4B An example of a method for determining, by a robotic system, a top surface of a stack of objects is illustrated in accordance with one or more embodiments.

[0025] Figure 5A An example of an image of a stack of objects and a layer divider is illustrated in accordance with one or more embodiments.

[0026] Figure 5BAn example of an image of a stack of objects and a layer divider is illustrated in accordance with one or more embodiments.

[0027] Figure 5C An example of an image of a stack of objects and a layer divider is illustrated in accordance with one or more embodiments.

[0028] Fig. 6A An example of a method for determining outlines of layer dividers in a stack of objects in accordance with one or more embodiments is illustrated.

[0029] Figure 6B Illustrated are examples of images used to determine a layer divider in accordance with one or more embodiments.

[0030] Figure 7 An example of a method for verifying the presence of a layer divider in accordance with one or more embodiments is illustrated.

[0031] Figure 8 An example of an environment in which two layers of objects are stacked on a pallet is illustrated in accordance with one or more embodiments.

[0032] Throughout the drawings and detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0033] Systems and methods related to object detection, identification, and retrieval are described herein. In some examples, the disclosed systems and methods can facilitate detection, identification, and retrieval of layer dividers deployed between layers of objects arranged on a loading or transport platform. As discussed herein, the objects can be boxes, boxes, crates, etc., which are arranged on a loading or transport platform (such as, for example, a slide or a pallet). In the detailed description below, the terms loading or transport platform and pallet can be used interchangeably, and these terms can refer to any platform on which objects can be stacked for storage or transportation.

[0034] In some examples, the shapes and sizes of the objects may vary. In some examples, the shapes of the objects may generally be cuboids or cubes. The objects may be arranged on the loading or transport platform in an organized, unorganized, or irregular manner. For example, the objects arranged on the loading or transport platform may be arranged in shapes such as, for example, a cuboid, a rectangular cone, or a cube. In other examples, the objects may be arranged in layers on the loading or transport platform, and each layer may or may not have a similar or equal number of objects.

[0035] Objects in a layer can be separated from objects in another layer by a separator, layer separator, or layer divider. A layer separator or layer divider can separate layers of objects on a loading or transport platform and can distribute weight, protect products from dust and dirt, and help prevent damage to one or more objects by strapping and forklifts. In some examples, the layer divider can be a thin layer deployed between objects, such as, for example, a slip sheet, a separator, cardboard, corrugated fiberboard, paperboard (solid board), corrugated board, card stock, paper, plastic board, or layer pad that can be placed between layers of objects. In some examples, the thickness of the layer divider can be extremely small, such as less than 10%, less than 5%, less than 2%, or less than 1%, compared to the size of any object or the length or width of the layer divider. Some examples of the thickness of the layer divider are: the thickness of corrugated fiberboard can vary from 3mm to 15mm, depending on whether the corrugated fiberboard is single-corrugated, double-corrugated, or triple-corrugated, the thickness of cardboard can vary from 0.2mm to 1mm, and the thickness of card stock can vary from 0.25mm to 0.8mm.

[0036] In some examples, the layer divider can be deployed between layers of some or all objects on a loading or transport platform. In some examples, the size of the layer divider can cover all, almost all, or substantially all (e.g., greater than 50%, greater than 80%, greater than 90%, greater than 95%, or greater than 98%) of the top surface of the layer of the object. In some examples, the length and width of the layer divider can be less than the corresponding length and width of the top surface of the layer of the object on which it is deployed. In some examples, the layer divider can be substantially quadrilateral in shape, such as a rectangular shape or a square shape. In some examples, the shape of the layer divider can be deformed due to folding, bending, or tearing of one or more edges or sides of the layer divider.

[0037] In such cases, due to the irregular arrangement and shape of the layer dividers, it may be challenging to detect and identify the layer dividers. The systems and methods discussed herein improve object detection, identification and retrieval of layer dividers with different shapes and arranged in a regular or semi-regular manner. Thus, the systems and methods described herein are designed to detect the presence of layer dividers and identify the contours of layer dividers. The systems and methods discussed herein may include a robotic system. The robotic system configured according to the embodiments of this article may autonomously perform integrated tasks by coordinating the operation of multiple robots. As described herein, the robotic system may include any suitable combination of robotic devices, actuators, sensors, cameras and computing systems, which are configured to control, issue commands, receive information from robotic devices and sensors, access, analyze and process data generated by robotic devices, sensors and cameras, generate data or information that can be used to control the robotic system, and plan the actions of robotic devices, sensors and cameras. As used herein, it is not required that the robotic system immediately access or control robotic actuators, sensors or other devices. The robotic system as described herein may be a computing system configured to improve the performance of such robotic actuators, sensors and other devices by receiving, analyzing and processing information.

[0038] The technology described herein provides technical improvements to robotic systems configured for object recognition, detection, and retrieval. The technical improvements described herein increase the speed, precision, and accuracy of these tasks and further facilitate the detection, recognition, and retrieval of layer separators between layers of objects deployed on a loading or transport platform. The robotic systems and computing systems described herein solve the technical problem of recognizing, detecting, and retrieving layer separators between layers of objects deployed on a loading or transport platform. By solving this technical problem, the technology of object recognition, detection, and retrieval is improved.

[0039] The present application relates to systems and robotic systems. As discussed herein, a robotic system may include a robotic actuator assembly (e.g., a robotic arm, a robotic gripper, etc.), various sensors (e.g., a camera, etc.), and various computing or control systems. As discussed herein, a computing system or control system may be referred to as "controlling" various robotic components, such as a robotic arm, a robotic gripper, a camera, etc. Such "control" may refer to direct control of various actuators, sensors, and other functional aspects of a robotic assembly and interaction with various actuators, sensors, and other functional aspects of a robotic assembly. For example, a computing system may control a robotic arm by issuing or providing all the required signals for various motors, actuators, and sensors to cause the robot to move. Such "control" may also refer to issuing abstract or indirect commands to another robotic control system, which then converts such commands into necessary signals for causing the robot to move. For example, a computing system may control a robotic arm by issuing commands describing a trajectory or destination position to which the robotic arm should move, and another robotic control system associated with the robotic arm may receive and interpret such commands, and then provide necessary direct signals to various actuators and sensors of the robotic arm to cause the required movement.

[0040] In particular, the present technology described herein assists a robotic system in detecting, identifying, and interacting with a layer divider that is deployed between layers of objects arranged on a loading or transport platform. Detecting, identifying, and retrieving a layer divider from between layers of objects requires several steps, including identifying edges and contours of a stack of objects deployed on a loading or transport platform, identifying a top surface of the stack of objects based on the contours and edges of the stack of objects, detecting extension lines corresponding to gaps between objects in the stack of objects or edges of the objects, the extension lines extending from a perimeter of the top surface toward a center of the top surface of the stack of objects, detecting the presence of a layer divider based on a length of each extension line being less than a threshold, and identifying the contour of the layer divider based on points on the top surface corresponding to termination points of the extension lines. For example, because the arrangement and shape of the layer divider may be irregular, it may be necessary to discern and identify the contour of the layer divider.

[0041] In the following description, specific details are set forth to provide an understanding of the currently disclosed technology. In an embodiment, the technology introduced herein can be practiced without including each specific detail disclosed herein. In other cases, well-known features, such as specific functions or routines, are not described in detail to avoid unnecessarily obscuring the present disclosure. References to "example", "embodiment", "one embodiment" and the like in this description mean that the specific features, structures, materials or characteristics described are included in at least one embodiment of the present disclosure. Therefore, such phrases appearing in this specification do not necessarily all refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. In addition, the specific features, structures, materials or characteristics described with respect to any one embodiment can be combined with the features, structures, materials or characteristics of any other embodiment in any suitable manner, unless these items are mutually exclusive. It should be understood that the various embodiments shown in the figures are merely illustrative representations and are not necessarily drawn to scale.

[0042] For the sake of clarity, several details describing structures or processes that are well known and often associated with robotic systems and subsystems but that may unnecessarily obscure some important aspects of the disclosed technology are not set forth in the following description. Moreover, although the following disclosure sets forth several embodiments of different aspects of the present technology, several other embodiments may have different configurations or different components than those described in this section. Therefore, the disclosed technology may have other embodiments with additional elements or without several of the elements described below.

[0043] Many embodiments or aspects of the present disclosure described below may take the form of computer executable instructions or controller executable instructions, including routines executed by programmable computers or controllers. It will be understood by those skilled in the relevant art that the disclosed technology may be practiced on a computer or controller system other than those shown and described below or practiced with a computer or controller system other than those shown and described below. The technology described herein may be embodied in a special-purpose computer or data processor that is specially programmed, configured or constructed to execute one or more of the computer executable instructions described below. Therefore, the terms processor, processing unit, computer and controller as generally used herein refer to any data processor, and may include Internet devices and handheld devices (including palmtop computers, wearable computers, cellular or mobile phones, multi-processor systems, processor-based or programmable consumer electronics, network computers, minicomputers, etc.). For ease of description, the singular term processor, controller or computer may be used in the description of the examples described in this application, but multiple processors or computers may be used in other examples, or the processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, two or more processors or a processor and a controller. One or more processors, processors or controllers can implement a single hardware component or two or more hardware components. The information processed by these computers and controllers can be presented on any suitable display medium including a liquid crystal display (LCD). The instructions for performing computer executable tasks or controller executable tasks can be stored in or on any suitable computer readable medium, including hardware, firmware or a combination of hardware and firmware. Instructions can be included in any suitable memory device, including, for example, a flash drive, a USB device and / or other suitable media.

[0044] The terms "coupled" and "connected", together with their derivatives, can be used in this article to describe the structural relationship between components. It should be understood that these terms are not intended to be synonymous with each other. Instead, in a particular embodiment, "connected" can be used to indicate that two or more elements are in direct contact with each other. Unless otherwise specified in the context, the term "coupled" can be used to indicate that two or more elements are in direct or indirect contact with each other (there are other intervening elements between them), or two or more elements cooperate or interact with each other (for example, as in a causal relationship, such as for signal transmission / reception or for function call), or both.

[0045] Any reference herein to image analysis performed by a computing system may be performed based on or using spatial structure information, which may include depth information describing respective depth values ​​of various locations relative to a selected point. Depth information may be used to identify objects or estimate how objects are arranged in space. In some instances, spatial structure information may include or may be used to generate a point cloud describing the locations of one or more surfaces of an object. Spatial structure information is only one form of possible image analysis, and other forms known to those skilled in the art may be used in accordance with the methods described herein.

[0046] Figure 1A A system 1000 for performing object detection (or more specifically, object recognition) is illustrated. More specifically, the system 1000 may include a computing system 1100 and a camera 1200. In this example, the camera 1200 may be configured to generate image information that describes or otherwise represents an environment in which the camera 1200 is located, or more specifically represents an environment in a field of view (also referred to as a camera field of view) of the camera 1200. The environment may be, for example, a warehouse, a manufacturing plant, a retail space, or other location. In such an instance, the image information may represent objects located in such a location, such as boxes, bins, boxes, crates, pallets, or other containers, and layer dividers discussed herein. The system 1000 may be configured to generate, receive, and / or process image information, such as by using the image information to distinguish individual objects in the camera field of view, to perform object recognition or object registration based on the image information, and / or to perform robot interaction planning based on the image information, as discussed in more detail below (the terms "and / or" and "or" are used interchangeably in this disclosure). Robot interaction planning can be used, for example, to control a robot at a location to facilitate robot interaction between the robot and a container or other object. The computing system 1100 and the camera 1200 can be located at the same location or can be remote from each other. For example, the computing system 1100 can be part of a cloud computing platform hosted in a data center remote from a warehouse or retail space and can communicate with the camera 1200 via a network connection.

[0047] In embodiments, the camera 1200 (which may also be referred to as an image sensing device) may be a 2D camera and / or a 3D camera. Figure 1BA system 1500A (which may be an embodiment of the system 1000) is shown, which includes a computing system 1100 and cameras 1200A and 1200B, both of which may be embodiments of the camera 1200. In this example, the camera 1200A may be a 2D camera configured to generate 2D image information, which includes or forms a 2D image describing the visual appearance of the environment in the camera's field of view. The camera 1200B may be a 3D camera (also referred to as a spatial structure sensing camera or spatial structure sensing device) configured to generate 3D image information, which includes or forms spatial structure information about the environment in the camera's field of view. The spatial structure information may include depth information (e.g., a depth map) that describes the corresponding depth values ​​of various locations (such as locations on the surface of various objects in the field of view of the camera 1200B) relative to the camera 1200B. These locations in the field of view of the camera or on the surface of the object may also be referred to as physical locations. The depth information in this example may be used to estimate how to spatially arrange objects in a three-dimensional (3D) space. In some examples, the spatial structure information may include or may be used to generate a point cloud describing locations on one or more surfaces of an object in the field of view of the camera 1200B. More specifically, the spatial structure information may describe various locations on the structure of the object (also referred to as object structure).

[0048] In an embodiment, system 1000 may be a robot operating system for facilitating robotic interaction between the robot and various objects in the environment of camera 1200. For example, Figure 1C A robot operating system 1500B is shown, which may be Figure 1A and Figure 1BEmbodiments of system 1000 / 1500A. Robot operating system 1500B may include computing system 1100, camera 1200, and robot 1300. As described above, robot 1300 may be used to interact with one or more objects in the environment of camera 1200, such as interacting with boxes, crates, boxes, boxes, pallets, other containers, and / or layer dividers discussed herein. For example, robot 1300 may be configured to pick up containers from one location and move them to another location. In some cases, robot 1300 may be used to perform unloading operations, in which a group of containers or other objects are unloaded and moved to, for example, a conveyor belt, or may be used to perform palletizing operations, in which containers are stacked onto a pallet in preparation for transportation. In some examples, during unloading operations, robot 1300 may identify and pick up layer dividers between layers of objects arranged on a pallet, and store them or move them to a location away from the conveyor belt. In some examples, camera 1200 may be attached to robot 1300 or robot 3300 discussed below. This is also referred to as a handheld camera or camera-in-hand solution. The camera 1200 can be attached to the robot arm 3320 of the robot 1300. The robot arm 3320 can be moved to various picking areas to generate image information about these areas. In some embodiments, the camera 1200 can be separated from the robot 1300. For example, the camera 1200 can be mounted on the ceiling of a warehouse or other structure and can remain stationary relative to the robot 1300. Multiple cameras 1200 can be used, including multiple cameras 1200 separate from the robot 1300 / 306 and / or cameras 1200 separate from the robot 1300 being used in conjunction with the camera 1200 in hand. In some embodiments, one or more cameras 1200 can be mounted or fixed to a dedicated robot system separate from the robot 1300 for object manipulation, such as a robot arm, a crane frame, or other automated system configured for camera movement. Throughout the specification, "controlling" the camera 1200 can be discussed. For the camera-in-hand solution, control of the camera 1200 also includes control of the robot 1300 to which the camera 1200 is mounted or attached.

[0049] In an embodiment, Figure 1A-Figure 1CThe computing system 1100 may be formed or integrated into the robot 1300, which may also be referred to as a robot controller. The robot control system may be included in the robot operating system 1500B and is configured to, for example, generate commands for the robot 1300, such as robot interaction movement commands for controlling the robot 1300 to interact with a container, layer separator, or other object. In such an embodiment, the computing system 1100 may be configured to generate such commands based on image information generated, for example, by the camera 1200. In some examples, the computing system 1100 may be configured to determine a motion plan based on the image information, wherein the motion plan may be intended for, for example, picking up or otherwise grasping an object. The computing system 1100 may generate one or more robot interaction movement commands to execute the motion plan.

[0050] In an embodiment, the computing system 1100 may form part of a visual system or be part of a visual system. The visual system may be a system that generates, for example, visual information that describes the environment in which the robot 1300 is located, or, in addition, describes the environment in which the camera 1200 is located. The visual information may include 3D image information and / or 2D image information discussed above, or some other image information. In some scenarios, if the computing system 1100 forms a visual system, the visual system may be part of the robot control system discussed above or may be separate from the robot control system. If the visual system is separate from the robot control system, the visual system may be configured to output information describing the environment in which the robot 1300 is located. The information may be output to the robot control system, and the robot control system may receive such information from the visual system and perform motion planning and / or generate robot interactive movement commands based on the information. Further information about the visual system is detailed below.

[0051] In embodiments, the computing system 1100 may communicate with the camera 1200 and / or with the robot 1300 via a direct connection, such as via a dedicated wired communication interface, such as an RS-232 interface, a universal serial bus (USB) interface, and / or via a connection provided by a local computer bus, such as a peripheral component interconnect (PCI) bus. In embodiments, the computing system 1100 may communicate with the camera 1200 and / or with the robot 1300 via a network. The network may be any type and / or form of network, such as a personal area network (PAN), a local area network (LAN) (e.g., an intranet), a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The network may utilize different technologies and protocol layers or protocol stacks, including, for example, Ethernet protocols, Internet protocol suite (TCP / IP), ATM (Asynchronous Transfer Mode) technology, SONET (Synchronous Optical Network) protocols, or SDH (Synchronous Digital Hierarchy) protocols.

[0052] In embodiments, computing system 1100 may communicate information directly with camera 1200 and / or with robot 1300, or may communicate via an intermediate storage device, or more generally, an intermediate non-transitory computer-readable medium. Figure 1D System 1500C is illustrated, which may be an embodiment of system 1000 / 1500A / 1500B, and includes a non-transitory computer-readable medium 1400, which may be external to computing system 1100 and may act as an external buffer or repository for storing image information generated, for example, by camera 1200. In such an example, computing system 1100 may retrieve or otherwise receive image information from non-transitory computer-readable medium 1400. Examples of non-transitory computer-readable medium 1400 include electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. The non-transitory computer-readable medium may be in the form of, for example, a computer floppy disk, a hard disk drive (HDD), a solid-state drive (SDD), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk-read-only memory (CD-ROM), a digital versatile disk (DVD), and / or a memory stick.

[0053] As described above, camera 1200 may be a 3D camera and / or a 2D camera. A 2D camera may be configured to generate a 2D image, such as a color image or a grayscale image. A 3D camera may be, for example, a depth sensing camera, such as a time-of-flight (TOF) camera or a structured light camera, or any other type of 3D camera. In some cases, a 2D camera and / or a 3D camera may include an image sensor, such as a charge coupled device (CCD) sensor and / or a complementary metal oxide semiconductor (CMOS) sensor. In an embodiment, a 3D camera may include a laser, a lidar device, an infrared device, a light / dark sensor, a motion sensor, a microwave detector, an ultrasonic detector, a radar detector, or any other device configured to capture depth information or other spatial structure information.

[0054] As described above, the image information may be processed by the computing system 1100. In an embodiment, the computing system 1100 may include or be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop computer, etc.), a smart phone, a tablet computing device, and / or any other computing system. In an embodiment, any or all functions of the computing system 1100 may be performed as part of a cloud computing platform. The computing system 1100 may be a single computing device (e.g., a desktop computer), or may include multiple computing devices.

[0055] Figure 2A An example of a computing system 1100 is illustrated. The computing system 1100 in this embodiment includes a processing circuit 1110 and one or more non-transitory computer-readable media 1120. In some instances, the processing circuit 1110 may include a processor (e.g., a central processing unit (CPU), a dedicated computer, and / or an onboard server) configured to execute instructions (e.g., software instructions) stored on a non-transitory computer-readable medium 1120 (e.g., a computer memory). In some embodiments, the processor may be included in a separate / independent controller operably coupled to other electronic / electrical devices. The processor may implement program instructions to control other devices / interface with other devices, thereby causing the computing system 1100 to perform actions, tasks, and / or operations. In an embodiment, the processing circuit 1110 includes one or more processors, one or more processing cores, a programmable logic controller ("PLC"), an application specific integrated circuit ("ASIC"), a programmable gate array ("PGA"), a field programmable gate array ("FPGA"), any combination thereof, or any other processing circuit.

[0056] In an embodiment, the non-transitory computer-readable medium 1120 as part of the computing system 1100 may be a replacement or supplement to the intermediate non-transitory computer-readable medium 1400 discussed above. The non-transitory computer-readable medium 1120 may be a storage device, such as an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, for example, such as a computer floppy disk, a hard disk drive (HDD), a solid-state drive (SSD), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, any combination thereof, any other storage device, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In some cases, the non-transitory computer-readable medium 1120 may include multiple storage devices. In an example, the instructions or software and any associated data, data files, and data structures are distributed on a network-coupled computer system so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers. In some embodiments, the non-transitory computer-readable medium 1120 is configured to store image information generated by the camera 1200 and received by the computing system 1100. In some cases, the non-transitory computer-readable medium 1120 may store one or more object recognition templates for performing the methods and operations discussed herein. The non-transitory computer-readable medium 1120 may alternatively or additionally store computer-readable program instructions that, when executed by the processing circuit 1110, cause the processing circuit 1110 to perform one or more methods described herein.

[0057] Figure 2B The computing system 1100A is described, which is an embodiment of the computing system 1100 and includes a communication interface 1131. The communication interface 1131 can be configured to receive, for example, Figure 1A-Figure 1D The image information generated by the camera 1200 may be received via the intermediate non-transitory computer-readable medium 1400 or the network discussed above, or via a more direct connection between the camera 1200 and the computing system 1100 / 1100A. In an embodiment, the communication interface 1131 may be configured to communicate with Figure 1CIf the computing system 1100 is located outside the robot control system, the communication interface 1131 of the computing system 1100 may be configured to communicate with the robot control system. The communication interface 1131 may also be referred to as a communication component or a communication circuit, and may include, for example, a communication circuit configured to perform communication via a wired or wireless protocol. As an example, the communication circuit may include an RS-232 port controller, a USB controller, an Ethernet controller, a Bluetooth controller, PCI bus controller, any other communication circuit, or a combination thereof.

[0058] In an embodiment, Figure 2C As shown in , the non-transitory computer readable medium 1120 may include a storage space 1125 configured to store one or more data objects discussed herein. For example, the storage space may store object recognition templates, image information, object image information, thresholds, robot arm movement commands, and any additional data objects that the computing system discussed herein may need to access.

[0059] In an embodiment, the processing circuit 1110 may be programmed by one or more computer-readable program instructions stored on a non-transitory computer-readable medium 1120. For example, Figure 2D A computing system 1100C is illustrated as an embodiment of a computing system 1100 / 1100A / 1100B, wherein a processing circuit 1110 is programmed with one or more modules including an object recognition module 1121, a motion planning module 1129, and an object manipulation planning module 1126. The processing circuit 1110 may also be programmed with an object registration module 1130, a feature extraction module 1134, and a verification module 1138. Each of the above modules may represent computer-readable program instructions configured to perform certain tasks when instantiated on one or more of the processors, processing circuits, computing systems, etc. described herein. Each of the above modules may operate in conjunction with each other to implement the functions described herein. Various aspects of the functions described herein may be performed by one or more of the software modules described above, and the software modules and their descriptions should not be understood as limiting the computing structure of the system disclosed herein. For example, although a specific task or function may be described with respect to a specific module, the task or function may also be performed by a different module as needed. In addition, the system functions described herein may be performed by a different set of software modules configured with different functional decompositions or functional allocations.

[0060] In an embodiment, the object recognition module 1121 can be configured to acquire and analyze image information as discussed throughout the disclosure. The methods, systems, and techniques discussed herein with respect to image information can use the object recognition module 1121. The object recognition module can also be configured for object recognition tasks related to object recognition, as discussed herein.

[0061] The motion planning module 1129 can be configured to plan and execute the movement of the robot. For example, the motion planning module 1129 can interact with other modules described herein to plan the motion of the robot 3300 for object acquisition operations and for camera placement operations. The methods, systems, and techniques discussed herein with respect to robot arm motion and trajectory can be performed by the motion planning module 1129.

[0062] The object manipulation planning module 1126 may be configured to plan and execute object manipulation activities of the robotic arm, such as grasping and releasing objects and executing robotic arm commands to assist and facilitate such grasping and releasing.

[0063] The object registration module 1130 may be configured to obtain, store, generate, and otherwise process object registration information that may be required for the various tasks discussed herein. The object registration module 1130 may store dimensions and thresholds associated with various objects that the robotic system may detect, identify, and retrieve. The object registration module 1130 may be configured to interact or communicate with any other modules as needed.

[0064] The feature extraction module 1134 can be configured to perform the feature extraction and generation tasks discussed herein, such as with Figures 4A-5C The feature extraction module 1134 can be configured to interact with the object registration module 1130 and any other modules as needed.

[0065] Verification module 1138 can be configured to complete the verification tasks described herein, such as with Figure 4A and 7 The verification module 1138 can be configured to interact with the object registration module 1130, the feature extraction module 1134, and any other modules as needed.

[0066] refer to Figure 2E , Figure 2F , Figure 3A and Figure 3B , explains methods related to the object recognition module 1121 that can be executed for image analysis. Figure 2E and Figure 2F illustrates example image information associated with an image analysis method, and Figure 3A and Figure 3BAn example robotic environment associated with an image analysis method is illustrated. References herein to image analysis performed by a computing system may be performed based on or using spatial structure information, which may include depth information describing corresponding depth values ​​of various locations relative to a selected point. Depth information may be used to identify objects or estimate how objects are arranged in space. In some instances, the spatial structure information may include or may be used to generate a point cloud that describes the location of one or more surfaces of an object. Spatial structure information is only one form of image analysis, and other forms known to those skilled in the art may be used in accordance with the methods described herein.

[0067] In an embodiment, computing system 1100 may obtain image information representing an object in a camera field of view (e.g., 3200) of camera 1200. In some examples, the object may be a layer divider 3600 on a top surface 3120 of an object 3100A / 3100B / 3100C / 3100D / 3600 arranged on a pallet 3400 in the field of view 3200 of camera 1200. Figure 2E and Figure 2F As shown in , the image information 2600, 2700 may be generated by a camera (e.g., 1200) when the layer divider 3600 is located (or has been located) in the camera field of view 3200, and may describe the layer divider 3600, the pallet 3400, one or more individual objects 3100A / 3100B / 3100C / 3100D, and / or the scene 3800. The object appearance describes the appearance of the layer divider 3600 and the surrounding objects 3100A / 3100B / 3100C / 3100D and the pallet 3400 from the perspective of the camera 1200. If there are multiple objects in the camera field of view, the camera may generate image information representing the multiple objects or a single object as needed (such image information related to a single object may be referred to as object image information). The image information may be generated by a camera (e.g., 1200) when the object group is located (or has been located) in the camera field of view, and may include, for example, 2D image information and / or 3D image information.

[0068] As an example, Figure 2E Depicted is a first set of image information, or more specifically, 2D image information 2600, which is generated by camera 1200 as described above and represents Figure 3A100A / 3100B / 3100C / 3100D, layer divider 3600, and pallet 3400 in the 2D image information. In some examples, the 2D image information 2600 may be a grayscale image or a color image, and may describe the appearance of the object 3100A / 3100B / 3100C / 3100D, the layer divider 3600, and the pallet 3400 from the viewpoint of the camera 1200. In an embodiment, the 2D image information 2600 may correspond to a monochrome channel (e.g., a red, green, or blue channel) of a color image. If the camera 1200 is deployed above the object 3100A / 3100B / 3100C / 3100D, the 2D image information 2600 may represent the appearance of the respective top surfaces of the object 3100A / 3100B / 3100C / 3100D, the layer divider 3600, and the pallet 3400. Figure 2E In the example of , the 2D image information 2600 may include corresponding parts 2000A / 2000B / 2000C / 2000D, also referred to as image parts or object image information, which represent the corresponding top surfaces of the objects 3100A / 3100B / 3100C / 3100D. Figure 2E In the example of , the 2D image information 2600 may include an image portion 2100, also referred to as an image portion, which represents a top surface of a layer divider 3600 disposed on the object 3100A / 3100B / 3100C / 3100D. Figure 2E In the example of , the 2D image information 2600 may include 2550, also referred to as image portion or object image information, which represents the corresponding surface of the pallet 3400 on which the object 3100A / 3100B / 3100C / 3100D is deployed.

[0069] exist Figure 2E In the 2D image information 2600, each image portion 2000A / 2000B / 2000C / 2000D / 2100 / 2550 may be an image region, or more specifically a pixel region (if the image is formed by pixels). Each pixel in the pixel region of the 2D image information 2600 may be characterized as having a position described by a set of coordinates [U, V] and may have a value relative to the camera coordinate system or some other coordinate system, such as Figure 2E and Figure 2F Each of the pixels may also have an intensity value, such as a value between 0 and 255 or 0 and 1023. In further embodiments, each of the pixels may include any additional information associated with the pixel in various formats (e.g., hue, saturation, intensity, CMYK, RGB, etc.).

[0070] As described above, the image information may be all or a portion of an image, such as 2D image information 2600, in some embodiments. In an example, the computing system 1100 may be configured to extract the image portion 2100 from the 2D image information 2600 to obtain only the image information associated with the corresponding object (i.e., the layer classifier 3600). In the case where an image portion (such as image portion 2000A) is directed to a single object, it may be referred to as object image information. Object image information need not contain only information about the object to which it is directed. For example, the object to which it is directed may be close to one or more other objects, below one or more other objects, above one or more other objects, or otherwise located in the vicinity of one or more other objects. In this case, the object image information may include information about the object to which it is directed and to one or more neighboring objects. The computing system 1100 may perform a multi-step process based on the 2D image information 2600 and / or Figure 2F The 3D image information 2700 shown performs image segmentation or other analysis or processing operations to extract the image portion 2000A. In some implementations, the image segmentation or other processing operations may include detecting image locations where physical edges of objects (e.g., edges of objects in the image or gaps between objects) appear in the 2D image information 2600 and using such image locations to identify object image information that is limited to representing a single object in the camera field of view (e.g., 3200) and substantially excludes other objects. By "substantially exclude," it is meant that the image segmentation or other processing techniques are designed and configured to exclude non-target objects from the object image information, but it is understood that errors may occur, noise may be present, and various other factors may result in portions that include other objects.

[0071] Figure 2F An example where the image information is 3D image information 2700 is depicted. In some examples, the 3D image information 2700 may include, for example, a depth map or point cloud indicating corresponding depth values ​​for various locations on one or more surfaces (e.g., top surface or other exterior surface) of the objects 3100A / 3100B / 3100C / 3100D, the layer divider 3600, and the pallet 3400. In some embodiments, an image segmentation operation for extracting the image information may involve detecting image locations where physical edges of objects (e.g., edges of a box) appear in the 3D image information 2700 and using such image locations to identify image portions (e.g., 2730) limited to representing individual objects in the camera field of view (e.g., 3100A).

[0072] The corresponding depth values ​​may be relative to the camera 1200 that generated the 3D image information 2700 or may be relative to some other reference point. In some embodiments, the 3D image information 2700 may include a point cloud that includes corresponding coordinates of various locations on the structure of an object in the camera's field of view (e.g., field of view 3200). Figure 2F In the example of , the point cloud may include corresponding sets of coordinates describing the positions of the corresponding surfaces of the objects 3100A / 3100B / 3100C / 3100D, the layer divider 3600, and the pallet 3400. The coordinates may be 3D coordinates, such as [XYZ] coordinates, and may have values ​​relative to the camera coordinate system or some other coordinate system. For example, the 3D image information 2700 may include a first image portion 2710, also referred to as an image portion, which indicates a set of positions 27101-2710, also referred to as physical positions, on the surface of the object 3100D. n In another example, the 3D image information 2700 may also include second, third, fourth and fifth parts 2720, 2730, 2740, 2750 and 2760. These parts may then further indicate that the 3D image information 2700 may be respectively represented by 27201-2720 n 、27301-2730 n 、27401-2740 n 、27501-2750 n and 27601-2760 n denoted by a set of corresponding depth values ​​of a position. These figures are merely examples, and any number of objects having corresponding image portions may be used. Similar to the above, the obtained 3D image information 2700 may in some instances be part of a first set of 3D image information 2700 generated by a camera. Figure 2F In the example of Figure 3A If the 3D image information 2700 is an object 3100A, then the 3D image information 2700 can be reduced to refer to only the first image portion 2710. Similar to the discussion of the 2D image information 2600, each of the first, second, third, fourth, and fifth image portions 2730 can belong to a separate object and can be referred to as object image information. Therefore, the object image information used herein may include 2D and / or 3D image information.

[0073] In an embodiment, an image normalization operation may be performed by the computing system 1100 as part of obtaining the image information. The image normalization operation may involve transforming an image or image portion generated by the camera 1200 to generate a transformed image or a transformed image portion. For example, image information that may include 2D image information 2600, 3D image information 2700, or a combination of both may undergo an image normalization operation in an attempt to cause the image information to be changed in terms of viewpoint, object pose, lighting conditions associated with the visual description information. Such normalization may be performed to facilitate a more accurate comparison between the image information and the model (e.g., template) information. The viewpoint may refer to the pose of the object relative to the camera 1200, and / or the angle at which the camera 1200 is viewing the object when the camera 1200 generates an image representing the object.

[0074] For example, image information may be generated during an object recognition operation in which a target object is in the camera field of view 3200. The camera 1200 may generate image information representing a target object when the target object has a particular posture relative to the camera. For example, the target object may have a posture such that its top surface is perpendicular to the optical axis of the camera 1200. In such an example, the image information generated by the camera 1200 may represent a particular viewpoint, such as a top view of the target object. In some instances, when the camera 1200 is generating image information during an object recognition operation, the image information may be generated with a particular lighting condition, such as lighting intensity. In such an instance, the image information may represent a particular lighting intensity, lighting color, or other lighting condition.

[0075] In an embodiment, the image normalization operation may involve adjusting an image or image portion of a scene generated by a camera so that the image or image portion better matches the viewpoint and / or lighting conditions associated with the information of the object recognition template. The adjustment may involve transforming the image or image portion to generate a transformed image that matches at least one of an object pose or lighting conditions associated with the visual description information of the object recognition template.

[0076] Viewpoint adjustment may involve processing, warping, and / or shifting of an image of a scene so that the image represents the same viewpoint as the visual description information that may be included in the object recognition template. Processing may, for example, include changing the color, contrast, or lighting of the image, warping of the scene may include changing the size, dimension, or scale of the image, and shifting of the image may include changing the position, orientation, or rotation of the image. In an example embodiment, processing, warping, and / or shifting may be used to change objects in the image of the scene to have an orientation and / or size that matches or better corresponds to the visual description information of the object recognition template. If the object recognition template describes a front view (e.g., a top view) of an object, the image of the scene may be warped so as to also represent a front view of the object in the scene.

[0077] Further aspects of the object recognition methods performed herein are described in more detail in U.S. Application No. 16 / 991,510 filed on August 12, 2020 and U.S. Application No. 16 / 991,466 filed on August 12, 2020, each of which is incorporated herein by reference.

[0078] In various embodiments, the terms "computer-readable instructions" and "computer-readable program instructions" are used to describe software instructions or computer code that are configured to perform various tasks and operations. In various embodiments, the term "module" broadly refers to a collection of software instructions or codes that are configured to cause the processing circuit 1110 to perform one or more functional tasks. These modules and computer-readable instructions can be described as performing various operations or tasks when the processing circuit or other hardware components are executing these modules or computer-readable instructions.

[0079] Figure 3A-3B An exemplary environment is illustrated in which computer readable program instructions stored on a non-transitory computer readable medium 1120 are utilized via a computing system 1100 to increase the efficiency of object recognition, detection, and acquisition operations and methods. Figure 3A The image information illustrated in affects the system's decision-making process and command output to the robot 3300 present within the object environment.

[0080] Figure 3A-3B An example environment is shown in which the processes and methods described herein may be performed. Figure 3A Depicted is a system 3000 (which may be Figure 1A-Figure 1DIn an embodiment of the system 1000 / 1500A / 1500B / 1500C in FIG. 1 ), the system 3000 includes at least a computing system 1100, a robot 3300, and a camera 1200. The camera 1200 may be an embodiment of the camera 1200 and may be configured to generate image information representing a scene 3800 in a camera field of view 3200 of the camera 1200, or more specifically, a stack of objects 3700 (such as boxes), such as objects 3100A, 3100B, 3100C, 3100D, in the camera field of view 3200. In one example, each of the objects 3100A-3100D may be, for example, a container such as a box or a crate, which is disposed on an object such as a pallet 3400. The layer divider 3600 may be disposed on a top surface 3120 of the object 3100A / 3100B / 3100C / 3100D. The layer divider can divide two layers of objects in the stack of objects 3700. In some examples, the stack of objects 3700 can be collectively referred to as objects 3100A / 3100B / 3100C / 3100D, layer divider 3600, and pallet 3400. For ease of illustration and explanation, Figure 3A and 3B In the example of FIG. 3 , only one layer of objects 3100A / 3100B / 3100C / 3100D is shown with a layer divider 3600. However, it should be understood that the stack 3700 of objects may include multiple layers of objects, wherein the layer divider 3600 may or may not be disposed between two layers of objects. Figure 8 , an example of two layers of objects 3100A / 3100B / 3100C / 3100D / 3100E / 3100F / 3100G / 3100H stacked on a pallet 3400 is illustrated, wherein a layer divider 3600 is disposed on a top surface of the second layer.

[0081] Figure 3A An embodiment including a plurality of objects 3100A-3100D is illustrated, wherein a camera field of view (eg, 3200) is captured using a camera 1200 attached to a robot 3300. Figure 3BAn embodiment of capturing a camera field of view (e.g., 3200) using a camera 1200 deployed separately from a robot 3300 is illustrated. Each of the objects 3100A / 3100B / 3100C / 3100D can be, for example, a box, crate, box, case, or other container. The object (i.e., object 3100A / 3100B / 3100C / 3100D) can be deployed on another object (such as a pallet 3400), which can also be at least partially deployed within the camera field of view 3200. In some scenarios, the pallet 3400 can be used to stack or otherwise deploy containers or other objects, which can have a variety of sizes (e.g., a variety of lengths, widths, and height values) and have a variety of stacking or placement configurations. The layer divider 3600 may or may not be deployed between the object layers on the pallet 3400.

[0082] In an embodiment, Figure 3A The system 3000 may include one or more light sources. The light source may be, for example, a light emitting diode (LED), a halogen lamp, or any other light source, and may be configured to emit visible light, infrared radiation, or any other form of light toward the surface of the objects 3100A-3100D, the layer divider 3600, and the pallet 3400. In some embodiments, the computing system 1100 may be configured to communicate with the light source to control when the light source is activated. In other implementations, the light source may operate independently of the computing system 1100.

[0083] In embodiments, system 3000 may include a camera 1200 or multiple cameras 1200, including a 2D camera configured to generate 2D image information 2600 and a 3D camera configured to generate 3D image information 2700. Camera 1200 or multiple cameras 1200 may be mounted or fixed to robot 3300, may be fixed within an environment, and / or may be fixed to a dedicated robotic system separate from robot 3300 for object manipulation, such as a robotic arm, gantry, or other automated system configured for camera movement.

[0084] Figure 3A An example with a fixed camera 1200 and a handheld camera 1200 is shown, and Figure 3BAn example with only a fixed camera 1200 is shown. The 2D image information 2600 (e.g., a color image or a grayscale image) may describe the appearance of one or more objects, such as the objects 3100A / 3100B / 3100C / 3100D or the layer divider 3600 in the camera field of view 3200. For example, the 2D image information 2600 may capture or otherwise represent visual details disposed on corresponding outer surfaces (e.g., top surfaces) of the objects 3100A / 3100B / 3100C / 3100D and the layer divider 3600, and / or the contours of these outer surfaces. In an embodiment, the 3D image information 2700 may describe the structure of one or more of the objects 3100A / 3100B / 3100C / 3100D and the layer divider 3600, where the structure of the object may also be referred to as the object structure or the physical structure of the object. In some examples, the 3D image information 2700 may describe the outlines of the objects 3100A / 3100B / 3100C / 3100D and the layer divider 3600 arranged on the pallet 3400. For example, the 3D image information 2700 may include a depth map, or more generally, depth information, which may describe respective depth values ​​of various locations in the camera field of view 3200 relative to the camera 1200 or relative to some other reference point. The locations corresponding to the respective depth values ​​may be locations on respective surfaces in the camera field of view 3200 (also referred to as physical locations), such as locations on respective top surfaces of the objects 3100A / 3100B / 3100C / 3100D and the layer divider 3600. In some cases, the 3D image information 2700 may include a point cloud that may include multiple 3D coordinates that describe various locations on one or more exterior surfaces of the objects 3100A / 3100B / 3100C / 3100D and the layer divider 3600, or various locations of some other objects in the camera field of view 3200. Figure 2F An example of a point cloud is shown.

[0085] exist Figure 3A and Figure 3BIn an example of the robot 3300, the robot 3300 (which may be an embodiment of the robot 1300) may include a robot arm 3320 having one end attached to a robot base 3310 and having another end attached to or formed by an end effector device 3330 (such as a robot gripper). The robot base 3310 may be used to mount the robot arm 3320, and the robot arm 3320, or more specifically the end effector device 3330, may be used to interact with one or more objects in the environment of the robot 3300. The interaction (also referred to as robot interaction) may include, for example, grasping or otherwise picking up at least one of the objects 3100A-3100D and 3600. For example, the robot interaction may be part of an object picking operation to identify, detect, and retrieve the layer divider 3600 from the top surface 3120 of the layer of the object 3100A / 3100B / 3100C / 3100D. The end effector device 3330 may have a suction cup or other component for grasping or gripping the layer divider 3600. The end effector device 3330 may be configured to grasp or grip an object using a suction cup or other gripping component through contact with a single face or surface of the object (e.g., via the top surface).

[0086] The robot 3300 may also include additional sensors configured to obtain information for achieving tasks, such as for manipulating structural members and / or for transporting the robot unit. The sensors may include devices configured to detect or measure one or more physical properties of the robot 3300 (e.g., the state, condition, and / or position of one or more structural members / joints thereof) and / or one or more physical properties of the surrounding environment. Some examples of sensors may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

[0087] Figure 4A Illustrated is an example of a method 400 for detecting, identifying, and retrieving layer dividers by a robotic system in accordance with one or more embodiments. Figure 4A The operations can be performed in the order and manner shown. However, the order of some operations can be changed, or some operations can be omitted without departing from the spirit and scope of the examples shown. In addition, Figure 4A The operations shown in can be performed in parallel or simultaneously. Figure 4A One or more blocks and combinations of blocks may be implemented by a computer based on dedicated hardware or a combination of dedicated hardware and instructions (eg, computer or processor instructions) that performs the specified functions. Figure 4A In addition to the description of Figure 1- Figure 3B The description also applies to Figure 4A , and combined in this article.

[0088] In image capture operation 410, camera 1200 captures an image of a stack of objects deployed on a pallet. In an embodiment, computing system 1100 (e.g., via object recognition module 1121) receives image information (which may also be referred to as a first set of image information) representing at least a first portion of an object structure associated with an object in field of view 3200 (also referred to as camera field of view) of camera 1200. For example, Figure 3A A scene is depicted in which a stack 3700 of objects is located within the camera field of view 3200 of camera 1200 .

[0089] In some examples, the image information received by computing system 1100 may be an image of a camera in a state where the camera is at or has a first camera pose (eg, Figure 3A The camera pose shown in ) is generated by a camera (e.g., 1200). The camera pose may refer to the position and orientation of the camera (e.g., 1200). In some scenes, the camera pose may affect the perspective or viewpoint of the camera (e.g., 3200). For example, Figure 3A 3100A / 3100B / 3100C / 3100D, and having an orientation with the camera 1200 pointing toward a first portion (or more specifically a top portion (e.g., a top surface)) of the object 3100A / 3100B / 3100C / 3100D. In some examples, the orientation of the first camera pose may be associated with the camera 1200 having a top view of the object. For example, the first camera pose may involve the camera 1200 having an orientation with its image sensor directly facing the top portion of the object 3100A / 3100B / 3100C / 3100D, and / or an orientation with the focal axis of one or more lenses of the camera 1200 vertical or substantially vertical.

[0090] In contour operation 420, computing system 1100 identifies contours of stack 3700 of objects based on information obtained from images captured by a camera. Edges of stack 3700 of objects may represent boundaries in an image where there is a sharp change in image brightness or image color. Such changes may be due to discontinuities in image intensity or transitions between different regions within an image. Edges may be generated by one or more of boundaries of objects, boundaries between objects, reflectivity changes within an object, reflectivity changes between objects, surface orientation changes within an object, and lighting changes (e.g., cast shadow boundaries). Edges of an image may be detected by one or more methods, such as, for example, a Sobel operator, a Canny operator, a Prewitt operator, a Roberts operator, and a Gabor energy filter.

[0091] The contour of an object may refer to a curve connecting all consecutive points of the same color or intensity along the boundary. Contour detection methods may follow the edges identified in the image to find curves that depict the boundaries of the object, which may be continuous. Unlike edges that focus primarily on high contrast areas in an image, contours focus on outlining the shape and structure of the object within the image. The contours depicted in the image may be determined by converting the image of the stack 3700 of objects into a binary image by the computing system 1100, preprocessing the binary image to enhance edge features, and extracting contours.

[0092] In an example process for determining contours depicted in an image, the computing system 1100 may apply a filter to smooth the image. Gaussian blur may be used for this purpose to reduce image noise without blurring edges. Next, the computing system 1100 may determine the gradient of the image to highlight areas with high spatial frequencies, which may correspond to edges. Any of the techniques discussed above, such as Sobel, Prewitt, or Canny edge detectors, may be used to calculate the magnitude and direction of the gradient at each pixel. The computing system 1100 may also perform non-maximum suppression to thin out edges. Here, pixels that are not in areas of high spatial frequency may be suppressed, leaving pixels with sparse lines of high gradients. High spatial frequency areas may be areas where image intensity changes abruptly over a small spatial distance, such as at edges between different objects or between an object and a background. Next, the computing system 1100 may apply a threshold to the gradient image to identify edges. The threshold may be a simple binarization threshold, an adaptive threshold, or a hysteresis threshold used in a Canny edge detector, which distinguishes strong edges from weak edges based on two thresholds. The threshold may be stored in the object registration module 1130. Once edges are identified, the computing system 1100 can link edge pixels together to form contours, which can be continuous. A contour tracing algorithm, such as, for example, the cv2.findContours function and the cv2.drawContours function, can be employed to track edges and outline the stack 3700 of objects.

[0093] In some cases, additional processing may be performed by the computing system 1100 to simplify the detected contours and reduce complexity. For example, the computing system 1100 may apply methods such as the Douglas-Peucker algorithm to reduce the number of points in the curve that deviate negligibly from the original contour. After extracting the contours of the stack 3700 of objects, the feature extraction module 1134 may extract features such as shape, size, orientation, or perform object recognition. The computing system 1100 may analyze the contours to extract features such as shape, size, orientation, or perform object recognition by comparing the contours to known shapes stored in the object registration module 1130. One or more steps of contour extraction and analysis may be enhanced or even bypassed by deploying a neural network trained for edge detection and / or extracting contours from an input image. The neural network may be a U-Net convolutional neural network (CNN) architecture.

[0094] In some examples, as described above, the computing system 1100 identifies the edges and contours of the stack 3700 of objects based on the features extracted from the image by the feature extraction module 1134. The image may be a 3D image (and the image information may be 3D image information), and in other examples, the image may be a 2D image (and the image information may be 2D image information). The computing system 1100 (e.g., via the object recognition module 1121) may generate contours of the objects (e.g., 3100A / 3100B / 31000C / 3100D) and the layer divider 3600 deployed on the pallet 3400 in the camera field of view (e.g., 3200) based on the image information. The computing system 1100 may analyze the contours of the stack 3700 of objects to determine structural information of the contours of the stack 3700 of objects, such as shape, size, and orientation. The contours may be generated based on multiple sets of image information, all of which represent top views of one or more objects in the camera field of view (e.g., 3200). Multiple sets of image information may be generated by a camera (e.g., 1200) at different corresponding positions (as the camera moves laterally), but with the same or similar orientation of the camera, such as, for example, the orientation of the camera's image sensor directly facing a corresponding area of ​​the top portion of one or more objects (e.g., 3100A / 3100B / 31000C / 3100D). In this example, the image information may be one of the multiple sets of image information. The image information may include a point cloud or a depth map, which may be used by the object recognition module 1121 to identify the outline of the stack 3700 of objects together with other modules of the computing system 1100.

[0095] The computing system 1100 can analyze the outline of the stack of objects 3700 to identify edges associated with the outline of the stack of objects 3700. For example, the edges can be Figure 3A3700 of objects in a stack. In some embodiments, computing system 1100 can identify object corners based on sensed structural information. For example, computing system 1100 can identify multiple outer edges described by sensed structural information, or multiple edge regions described by sensed structural information. In such examples, computing system 1100 can identify an edge as a location located at or near the intersection of multiple outer edges, and / or a location located in an area where multiple edge regions intersect. Identification of edges and contours of a stack of objects is discussed in more detail in U.S. Patent No. 11,883,966 B2, the entire contents of which are incorporated herein by reference.

[0096] In surface recognition operation 430, computing system 1100 may recognize a top surface of a stack of objects based on the outline of the objects. Object recognition module 1121 may recognize a top surface of a layer of objects based on features extracted from an image by feature extraction module 1134. Object recognition module 1121 may recognize a top surface of a layer of objects 3700 based on analyzing the outline of stack 3700 of objects to determine an uppermost boundary of the outline representing the top surface. The uppermost boundary of the outline may refer to a line or surface marking the limit of a horizontal plane representing the uppermost surface of stack 3700 of objects.

[0097] Figure 4B Surface identification operations 430 for determining, by a robotic system, a top surface of a stack of objects are illustrated in greater detail in accordance with one or more embodiments. Figure 4B The operations can be performed in the order and manner shown. However, the order of some operations can be changed, or some operations can be omitted without departing from the spirit and scope of the examples shown. In addition, Figure 4B The operations shown in can be performed in parallel or simultaneously. Figure 4B One or more blocks and combinations of blocks may be implemented by a computer based on dedicated hardware or a combination of dedicated hardware and instructions (eg, computer or processor instructions) that performs the specified functions. Figure 4B In addition to the description of Figure 1- Figure 4A The description also applies to Figure 4B , and combined in this article.

[0098] In positioning operation 431, computing system 1100 may generate instructions to position camera 1200 toward stack of objects 3700. Computing system 1100 causes camera 1200 (e.g., via one or more camera placement movement commands) to be positioned to have a second camera pose in which camera (e.g., 1200) is configured to capture an image of a top view of stack of objects 3700 identified in contour operation 420. Object identification module 1121 may identify a top surface of layer 3700 of objects based on analyzing the contour of stack of objects 3700 to determine an uppermost boundary of the contour representing the top surface.

[0099] In some examples, the second camera pose may be the same as the camera pose used to capture the image of stack 3700 of objects in operation 410. In some examples, the second camera pose of operation 431 may be different from the camera pose associated with operation 410. As described above, the camera pose may be a combination of the position and orientation of the camera (e.g., 1200).

[0100] In capture operation 432, a camera (e.g., 1200) may capture a 2D image of a top view of stack of objects 3700. When camera 1200 has a second camera pose, the top surface of stack of objects 3700 may enter within camera field of view 3200 of camera 1200 and may be within the line of sight of the image sensor of camera 1200. Figure 5A An example of a 2D image 5000A of the top surface of a stack of objects is illustrated, which will be described further below. In other embodiments, in capture operation 432, the camera (eg, 1200) may also capture a 3D image of a top view of the stack 3700 of objects.

[0101] In top surface determination operation 433, computing system 1100 may determine a top surface of stack of objects 3700 based on the 2D image captured in operation 432. The top surface of stack of objects 3700 may represent a horizontal plane representing the highest surface of stack of objects 3700. The top surface of stack of objects 3700 may be determined or identified by image segmentation or edge detection as described above of the image captured in capture operation 432. The perimeter of the top surface of stack of objects 3700 may be determined by Figure 5A 5100A-5100I shown in FIG. 5200A is defined by the top view of the objects 5100A-5100I shown in FIG. The computing system 1100 can determine the perimeter of the top surface of the stack of objects 3700 that delineates the outer boundary of the stack of objects based on the edge detection method described above. In some embodiments, the vertical height of the top surface of the stack of objects 3700 from the pallet 3400 can be determined by a sensor or image segmentation or edge detection, as described above.

[0102] Now return to Figure 4A In the extension line detection operation 440, the computing system 1100 may detect extension lines extending from the perimeter of the top surface to the center of the top surface based on the image information of the top surface of the stack of objects. Each extension line may correspond to at least one of a gap between objects in the stack of objects in the image or an edge of an object. Figure 5AAs shown in , extension line 5500A may correspond to an edge detected in the image information of the image of the top surface of stack 3700 of objects, and the edge represents a gap between objects 5100A and 5100B in the image information. As described below, the progression of the extension line from the perimeter to the center of the top surface of stack 3700 of objects is limited by layer dividers 5600A deployed on objects 5100A-5100I (or hidden from view of the image sensor of camera 1200). Reference will be made to Figure 5A-Figure 5C Further details are provided regarding detecting extension lines between objects. In another embodiment, when there are no layer dividers between the layers of objects, the extension lines may trace the contours of the surfaces of all objects disposed on the top surface of the stack of objects. In other words, when there are no layer dividers between the layers of objects, the extension lines may not terminate, but may define the shape of the top surface of the objects 5100A-5100I.

[0103] Figure 5A An example of a 2D image 5000A of a top surface of a stack of objects, such as objects 5100A-5100I arranged on a pallet 5400, is illustrated. A layer divider 5600A may be disposed on the top surface of the objects 5100A-5100I. The layer divider 5600A may cover all or part of the top surface of the objects 5100A-5100I. As described above, image segmentation, edge detection, or other processing operations may be performed to identify objects 5100A-5100I, boundaries 5200A of top views of objects 5100A-5100I, and extension lines 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H (representing gaps between objects in image 5000A) extending inward from the corresponding boundaries (and substantially perpendicular to the corresponding boundaries, e.g., within 10%, within 5%, within 2%, or within 1% of perpendicularity to the corresponding boundaries) toward the center portion of the 2D image 5000A. Boundary 5200A defines the boundaries of the outer edges of objects 5100A-5100I deployed on the pallet 5400. Figure 5A As shown in , extension lines 5500A and 5500B extend inward from the right side boundary 5200A of the 2D image 5000A, extension lines 5500C, 5500D and 5500E extend inward from the bottom side boundary 5200A of the 2D image 5000A, and extension lines 5500F, 5500G and 5500H extend inward from the top side boundary 5200A of the 2D image 5000A.

[0104] The computing system 1100 may determine extension lines 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H starting from the boundary 5200A and proceeding toward the center portion of the 2D image 5000A based on image segmentation, edge detection, or other processing operations as described above. The boundary 5200A may enclose a detection area 5700 of the image, which includes the objects 5100A-5100I, the extension lines 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H, and the layer divider 5600A.

[0105] Figure 5B An example of a 3D image 5000B of a top surface of a stack of objects is illustrated. Figure 5B An example of a top view of an object, such as objects 5100A-5100I arranged on a pallet, is illustrated. A layer divider 5600A may be deployed on the top surface of the object 5100A-5100I. As described above with reference to the above, image segmentation, edge deduction, or other processing operations may be performed on the depth map or point cloud data to identify the objects 5100A-5100I, the boundaries 5200A of the top views of the objects 5100A-5100I, extension lines 5500C / 5500D / 5500E extending from the respective boundaries (and substantially perpendicular to the respective boundaries, e.g., within 10%, within 5%, within 2%, or within 1% of the perpendicularity to the respective boundaries) toward the center portion of the 3D image 5000B (representing gaps between the objects 5100A-5100I in the image 5000B), and a detection area 5700 of the image. For example, computing system 1100 may determine extension lines 5500C / 5500D / 5500E starting from boundary 5200A and advancing toward a central portion of 3D image 5000B. Boundary 5200A defines the boundary of the outer edges of objects 5100A-5100I placed on pallet 5400. Extension lines 5500C / 5500D / 5500E and boundary 5200A may be determined based on changes in depth information of adjacent 3D points in the 3D image or changes in depth gradients. As described above, spatial depth information in the 3D image may be analyzed to detect edges based on discontinuities or significant changes in depth. These detected edges may be analyzed to determine extension lines 5500C / 5500D / 5500E (representing gaps between objects in image 5000A).

[0106] Returning now to FIG. 4 , in a layer divider detection operation 450 , the computing system 1100 may measure the length of the extension lines detected in the extension line detection operation 440 . The computing system 1100 may detect the presence of the layer divider 3600 disposed on the top surface 3120 of the layer of objects based on (e.g., in response to) the length of each extension line being less than a first threshold (i.e., a line length threshold). In some examples, the size of the objects in the stack of objects and the threshold may be stored in the object registration module 1130 . In some examples, the line length threshold may correspond to the length of the smallest side of any object in the objects 5100A-5100I. The line length threshold may be retrieved by the computing system 1100 and compared to the measured length. When the length of each extension line is less than the line length threshold, the computing system 1100 may detect the presence of the layer divider. When the length of each extended line representing the gaps between objects 5100A-5100I in image 5000A is less than the length of the smallest side of objects 5100A-5100I, it may indicate that some portion of the image of the objects is occluded by the presence of another object, which may be layer divider 5600A. Figure 5C The line length measurement process can be further understood.

[0107] Figure 5C The diagram shows the Figure 5A The 2D images and methods shown in or Figure 5B Examples of 3D images and extension lines and edges of detection areas identified by the method are shown in FIG. Figure 5C The image 5000C shows the use of detection area 5700. Figure 5A A 2D image or Figure 5BThe computing system 1100 may determine the end point of each of the extension lines 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H as 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 / 5500T8, respectively, based on image segmentation, edge detection, or other processing operations discussed above. Termination points 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 / 5500T8 are at the far end of the perimeter of the top surface defined by boundary 5200A. The end points 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 / 5500T8 of the extension lines 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H indicate that an object (i.e., the layer divider 5600A) blocks the extension lines 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H from extending from their corresponding boundaries 5200A. Thus, the termination points 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 / 5500T8 may indicate that a layer divider is deployed on top of the objects 5100A-5100I, and tracking the termination points may provide an outline of the layer divider 5600A.

[0108] The computing system 1100 may determine the length of each extended line 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H based on image segmentation, edge detection, or other processing operations discussed above. In some examples, the length of each respective line may be determined based on the length of the shortest line from the endpoint of the line to the boundary 5200A of the detection area 5700 from which the respective line extends.

[0109] In some examples, computing system 1100 may also be configured to detect the presence of layer divider 3600 based on no extension lines extending from the perimeter of top surface 3120 to the center of top surface 3120 of stack 3700 of objects. When no extension lines extend from the perimeter of top surface 3120 to the center of top surface 3120 of stack 3700 of objects, this may indicate that some portion of the image of objects 5100A-5100I is obscured by the presence of another object, which may be layer divider 5600A. In some examples, the line length threshold may be less than half the length of the smallest side of a quadrilateral defining top surface 3120 of stack 3700 of objects. In some examples, the line length threshold may be less than one-quarter the length of the smallest side of a quadrilateral defining top surface 3120 of stack 3700 of objects. In some examples, the line length threshold may be half the length of the side of a quadrilateral defining top surface 3120 of stack 3700 of objects. In some examples, the wire length threshold can be a predefined number that is stored in the object registration module 1130. The predefined number can be based on the environment in which the robotic system is operated and the objects that the robotic system is configured to handle. The predefined wire length threshold can be selectively adjusted based on changes in the size of objects that the robotic system can handle or process.

[0110] In verification operation 460, computing system 1100 may verify the presence of the layer divider based on one or more of the surface area of ​​the layer divider, the thickness of the layer divider, the weight of the layer divider, the smoothness of the layer divider, or texture artifacts. In some examples, operation 460 may be performed selectively. In some examples, the outline of the layer divider may be identified after the presence of the layer divider has been verified. In some examples, verification may be performed after the outline of the layer divider has been identified.

[0111] Figure 7 An example of a method for verifying the presence of a layer divider deployed between layers of an object or above layers of an object is illustrated. Figure 7 One or more blocks and combinations of blocks in the embodiment may be implemented by a computer based on special-purpose hardware or a combination of special-purpose hardware and instructions (e.g., computer or processor instructions) that performs the specified functions. Figure 7 One or more operations of may be performed by the verification module 1138 of the computing system 1100C. Figure 4A The order of some operations may be changed, or some operations may be omitted without departing from the spirit and scope of the illustrated examples. Figure 7 In addition to the description of Figure 1- Figure 6B The description also applies to Figure 7 and are incorporated into this article.

[0112] In surface area minimum threshold operation 461, when (e.g., in response to) the presence of a layer divider having been detected, the verification module 1138 of the computing system 1100C may determine the surface area of ​​the outline of the layer divider. The computing system 1100C may determine the surface area of ​​the outline of the layer divider 5600A by calculating the area enclosed by the outline that defines the boundary of the layer divider 5600A. If the outline of the layer divider 5600A defines a standard geometric shape (such as a rectangle, circle, or triangle), then the surface area may be determined by applying standard geometric formulas, such as, for a rectangle / square, multiplying the length by the width, applying the formula πr 2 , where r is the radius of the circle, determines the area of ​​a triangle based on dividing 1 / 2 × base × height or by applying Heron's formula when all sides of the triangle are known.

[0113] For a layer divider 5600A of a non-standard shape, the surface area can be determined by decomposing the shape into smaller, more manageable parts (such as triangles) or by using numerical methods. For example, the computing system 1100C can divide the area enclosed by the outline of the layer divider 5600A into triangles by selecting a point inside (or outside) the outline of the layer divider 5600A as a reference and connecting it to the vertices of the outline, thereby effectively triangulating the interior of the outline of the layer divider 5600A. The computing system 1100C can calculate the area of ​​each triangle by the above formula, and these areas can be added to obtain the area of ​​the outline of the layer divider 5600. In another example, the computing system 1100C can use the shoelace formula (also known as the Gaussian area formula) to determine the surface area of ​​the outline of the layer divider 5600A that can be represented as a polygon.

[0114] In other examples, the computing system 1100C may extract surface area from a digital image using a library (such as OpenCV) that provides functions such as a contourArea function that may directly calculate the area enclosed by a contour.

[0115] The computing system 1100C may compare the surface area of ​​the outline of the layer divider to a surface area minimum threshold stored in the object registration module 1130. The computing system 1100C may verify the presence of the layer divider based on whether the surface area of ​​the outline of the layer divider is greater than the surface area minimum threshold (e.g., in response to the surface area of ​​the outline of the layer divider being greater than the surface area minimum threshold). If the surface area of ​​the outline of the potentially detected layer divider is large enough, it may be considered an accurate detection, e.g., verified.

[0116] In some examples, the surface area minimum threshold may be equal to or greater than the surface area of ​​the smallest side of the smallest object handled in the operating environment. The surface area minimum threshold may be equal to or greater than the surface area of ​​the smallest side of the smallest object in a stack of objects. The surface area minimum threshold may be equal to the surface area of ​​the largest object in a stack of objects. In some examples, the surface area minimum threshold may be equal to the surface area of ​​the largest side of the largest object in a stack of objects. In some examples, the surface area minimum threshold is a predefined number that is stored in the object registration module 1130. The predefined number may be based on the environment used to operate the robotic system and the objects that the robotic system is configured to handle. In some examples, the predefined number may vary based on the type and size of the object or the type or size of items contained in the object.

[0117] In a surface area maximum threshold operation 462, the verification module 1138 of the computing system 1100C may determine a surface area of ​​the outline of the separator after detecting the presence of the layer divider (e.g., in response to detecting the layer divider). The surface area of ​​the outline of the layer divider may be determined as described above with reference to the surface area determination of the surface area minimum threshold operation 461. The surface area may be compared to a surface area maximum threshold stored in the object registration module 1130. The computing system 1100C may verify the presence of the layer divider based on whether the surface area of ​​the outline of the layer divider is less than the surface area maximum threshold (e.g., in response to the surface area of ​​the outline of the layer divider being less than the surface area maximum threshold). If the surface area of ​​the outline of the potentially detected layer divider is small enough, it may be considered an accurate detection, e.g., verified.

[0118] In some examples, the surface area maximum threshold can be equal to the surface area of ​​the stacked pallet on which the object is deployed. In some examples, the surface area maximum threshold is a predefined number that is stored in the object registration module 1130. In some examples, the predefined number can vary based on the environment in which the object is stored or positioned. Based on the environment for operating the robotic system and the object that the robotic system is configured to handle, the predefined surface area maximum threshold can be obtained by multiplying the size of the side of the pallet. In some examples, the size of the pallet can be any of the following: 45.9"×45.9", 48"×40", 47.2"×31.5", 47.2"×39.4", 47.2"×31.5", 44.9"×44.9", 43.3"×43.3", 39.4"×47.2", 31.5"×23.6", 24"×20", 20"×24", or any other size suitable for use in an environment for operating a robotic system.

[0119] In the layer divider thickness operation 463, when the presence of the layer divider is detected, the verification module 1138 of the computing system 1100C may determine the thickness of the layer divider based on the 3D image information. In some examples, the outline of the layer divider may be identified after the presence of the layer divider has been verified based on the layer divider thickness operation 463. In other examples, the layer divider thickness operation 463 may verify the presence of the layer divider after the outline of the layer divider has been identified. In some examples, the computing system 1100C may determine the thickness of the layer divider based on a change in depth information of adjacent 3D points in the 3D image or a change in a depth gradient between the layer divider and the top surface of the stack of objects indicated by the outline of the layer divider. The computing system 1100C may verify the presence of the layer divider based on whether the thickness of the layer divider indicated by the outline of the layer divider is less than a thickness threshold (e.g., in response to the thickness of the layer divider indicated by the outline of the layer divider being less than the thickness threshold). In some examples, the thickness threshold is a predefined number that is stored in the object registration module 1130. As described above, the layer divider can be made of many different materials, such as, for example, corrugated fiberboard, corrugated board, and cardstock. The predefined thickness threshold can vary based on the material of the layer divider. For example, when the layer divider is made of corrugated fiberboard, the predefined thickness threshold can vary from 3 mm to 15 mm, depending on whether the corrugated fiberboard is single-walled, double-walled, or triple-walled. When the layer divider is made of cardboard, the predefined thickness threshold can vary from 0.2 mm to 1 mm, and when the layer divider is made of cardstock, the predefined thickness threshold can vary from 0.25 mm to 0.8 mm.

[0120] In the layer divider weight operation 464, when the presence of the layer divider is detected, the verification module 1138 of the computing system 1100C can determine the weight of the layer divider based on the sensor information. In some examples, the outline of the layer divider can be identified after the presence of the layer divider has been verified based on the layer divider weight operation 464. In other examples, the layer divider weight operation 464 can verify the presence of the layer divider after the outline of the layer divider has been identified. The computing system 1100C can verify the presence of the layer divider based on whether the weight of the layer divider is less than a weight threshold (e.g., in response to the weight of the layer divider being less than the weight threshold). In some examples, the weight threshold is a predefined number that is stored in the object registration module 1130. The predefined weight threshold can vary based on the material of the layer divider and the expected size of the layer divider. For example, the predefined weight threshold of a triple-wall corrugated fiberboard can be greater than the predefined weight threshold of a double-wall corrugated fiberboard, and the predefined weight threshold of a single-wall corrugated fiberboard can be greater than the predefined weight threshold of paperboard or card stock. The expected size of the layer divider can depend on the size of the object that the robotic system is configured to handle. In some examples, the weight threshold can be approximately two and a half (2.5) kilograms. In other examples, the weight threshold can vary between 100 grams and 2500 grams based on the environment for operating the robotic system and the object that the robotic system is configured to handle.

[0121] In the layer divider smoothness operation 465, the verification module 1138 of the computing system 1100C can measure the smoothness of the layer divider and compare the measured smoothness with the smoothness threshold to verify the identified layer divider. If the measured smoothness is less than the smoothness threshold, that is, the layer divider is coarser than the smoothness threshold, then it can be determined that the identified object is not a layer divider, for example, it can be invalidated.

[0122] In an example, the smoothness measured by analyzing the z-values ​​in the points in the point cloud representing the layer divider 5600A can be compared with the z-value smoothness threshold. The verification for the z-value smoothness threshold can be performed in a variety of ways. The verification module 1138 can extract the z-coordinate (height) of the point. The z-value can represent the height or depth of the point in the point cloud from an imaging device (such as camera 1200). Then, the verification module 1138 can use a data structure (such as KDTree) to identify the nearest neighbors of each point. The verification module 1138 can determine the local z variation by calculating the standard deviation of the difference between the z-value of each point and the z-value of its neighbors. The verification module 1138 can analyze the mean and standard deviation of these local z variations to quantify the surface smoothness, and a lower standard deviation indicates that the surface of the layer divider 5600A is smoother. The surface smoothness can be compared with the z-value smoothness threshold to verify the identified layer divider or invalidate it.

[0123] In an example, the validation module 1138 can analyze the range and distribution of depth values ​​(ie, z-values) to identify changes. The changes can be compared to a z-value smoothness threshold to validate or invalidate the identified layer divider.

[0124] In another example, when (ie, in response to) a depth variation between a point and its neighboring points is greater than a z-value smoothness threshold, the computing system 1100C may determine that the object is not a layer divider.

[0125] In another example, the computing system 1100C may segment the point cloud into clusters of similar depth to examine depth variations within and between the clusters. When the depth variation is greater than a z-value smoothness threshold (i.e., in response thereto), the computing system 1100C may determine that the object is not a layer divider.

[0126] In an example, exceeding a z-value smoothness threshold of approximately 5 mm, 6 mm, or more may be considered to invalidate an object as a potential layer divider. In another example, when (i.e., in response to) the depth variation is greater than the z-value smoothness threshold for between 5% and 12%, between 7% and 9%, or approximately 8% of the surface area of ​​an object representing a point cloud (e.g., identified as a potential layer divider), the computing system 1100C may determine that the object is not a layer divider.

[0127] In other examples, the computing system 1100C can measure the smoothness of the layer divider based on a sensor, such as, for example, an optical profiler, which uses light to measure surface topography. These optical profilers can project a light beam onto the surface of the layer divider and analyze the reflected light to determine surface features and determine the smoothness of the layer divider. The smoothness of the layer divider can be represented by the arithmetic mean roughness (Ra), which is a measure of the average surface roughness of the material. Ra is the average deviation of the surface peaks and valleys measured over a specified length from the average line. The computing system 1100C can verify the presence of the layer divider based on whether the smoothness is greater than a smoothness threshold (e.g., in response to the smoothness being greater than the smoothness threshold). In some examples, the smoothness threshold is a predefined number that is stored in the object registration module 1130. The predefined smoothness threshold can vary based on the material of the layer divider, such as corrugated fiberboard, paperboard, and card stock.

[0128] In some examples, the computing system 1100C may determine that the object is a layer divider based on a comparison with either or both of a measured smoothness threshold and a z-value smoothness threshold (which may be different thresholds). When (i.e., in response to) both corresponding thresholds being met, the computing system 1100C may determine that the object is a layer divider.

[0129] In some examples, the outline of the layer divider may be identified after the presence of the layer divider has been verified based on the layer divider smoothness operation 465. In some examples, the outline of the layer divider may be identified after the presence of the layer has been verified by the layer divider smoothness operation 465. In other examples, the layer divider smoothness operation 465 may verify the presence of the layer divider after the outline of the layer divider has been identified.

[0130] In the layer divider texture operation 466, the feature extraction module 1134 of the computing system 1100C may generate image features based on the 2D image information and the 3D image information corresponding to the upper surface of the layer divider. In some examples, the outline of the layer divider may be identified after the existence of the layer divider has been verified based on the layer divider texture operation 466. In other examples, the layer divider texture operation 466 may verify the existence of the layer divider after the outline of the layer divider has been identified. In some examples, the verification module 1138 may interact with the feature extraction module 1134 to identify texture artifacts based on the image features. The computing system 1100C may verify the existence of the layer divider based on whether the texture artifact is identified in the image information (e.g., in response to identifying the texture artifact in the image information). When the texture artifact is identified in the image information (e.g., in response thereto), the computing system 1100C may determine that the existence of the layer divider is a false positive. In some examples, texture artifacts include any one or any combination of logos, graphics, labels, markings, logos, scan codes, universal product codes (“UPC”), packaging tape, visual patterns, and high spatial frequency regions in the image information. High spatial frequency regions can be areas where image intensity changes abruptly over a small spatial distance, such as between an object and a background, or between a solid color background of a packaging box and text or picture artifacts. When texture artifacts are not identified in the image information (e.g., in response thereto), the computing system 1100C can verify the presence of a layer divider.

[0131] Now return to Figure 4A In the outline recognition operation 470, the computing system 1100 may recognize an outline of the layer divider according to points on the top surface corresponding to the termination points of the extension lines.

[0132] Fig. 6A Illustrated is an example of a method 670 for determining a delineation of a layer divider in accordance with one or more embodiments. Fig. 6A The operations can be performed in the order and manner shown. However, the order of some operations can be changed, or some operations can be omitted without departing from the spirit and scope of the examples shown. In addition, Fig. 6A The operations shown in can be performed in parallel or simultaneously. Fig. 6AOne or more blocks and combinations of blocks may be implemented by a computer based on dedicated hardware or a combination of dedicated hardware and instructions (eg, computer or processor instructions) that performs the specified functions. Fig. 6A In addition to the description of Figure 1- Figure 5C The description also applies to Fig. 6A and are incorporated into this article.

[0133] In the termination point determination operation 671, the computing system 1100 may determine the termination point of each extension line 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H as 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7, respectively. The termination points 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 are respectively Figure 5C and referenced above Figure 5C The termination points 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 / 5500T8 of the extension lines indicate that a certain object (i.e., the layer divider 5600A) blocks the extension lines 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H extending from their corresponding boundaries 5200A. Therefore, the termination points 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 / 5500T8 may indicate that the layer divider is arranged on top of the objects 5100A-5100I, and tracking the termination points may provide an outline of the layer divider 5600A.

[0134] In the top surface corresponding operation 672, the computing system 1100 may determine the outline of the layer divider based on the points on the top surface corresponding to the termination points 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 of the extension lines. In an example, the outline of the layer divider may be based on connecting the termination points 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 of the extension lines.

[0135] In another example, the contour of the layer divider can be based on the end points 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 of the extension line and the image segmentation or other processing operation of the 2D image of the top view of the object (such as the objects 5100A-5100I arranged on the pallet). During the image segmentation, the computing system 1100 can identify objects (such as objects 5100A-5100I, layer divider 5600A, boundary 5200A) within the 2D image of the top view based on criteria such as the color, intensity or texture of the image. The computing system 1100 can track the contour of the layer divider based on the end points and objects from the image segmentation operation. If the computing system 1100 encounters the boundary 5200A element while tracking the contour, the computing system 1100 can confirm that the boundary of the detection area 5700 has been reached, thereby prompting it to stop contour tracking towards the boundary 5200 of the detection area 5700. In another example, the outline of the layer divider can be based on the end points 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 of the extension line and based on the change in depth information of adjacent 3D points in the 3D image or the change in the depth gradient of the top view of the object (such as the objects 5100A-5100I arranged on the pallet). Similar to the example described for image segmentation, the change in depth information can identify objects in the image, which in turn can help the computing system 1100 track the outline of the layer divider.

[0136] In the detection area portion removal operation 673, the computing system 1100 may determine the outline of the layer divider by filtering out the portion corresponding to the length of the corresponding extension line 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H in the detection area 5700, which may be determined as described above. In one example, the computing system 1100 may identify the outline of the layer divider based on eliminating the portion from the perimeter to the termination point 5500T1 / 5500T2 / 5500T3 / 5500T4 / 5500T5 / 5500T6 / 5500T7 of each extension line in the 2D image information of the top surface of the stack of objects. Figure 6B , an operation of removing a portion of the inspection area corresponding to the length of the corresponding extension line 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H is illustrated.

[0137] In the detection area partial removal operation 674, the computing system 1100 may determine the longest extension line (Ext Line 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H) among the extension lines 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H. long The computing system 1100 can filter out the longest extension line (Ext Line ) in the detection area 5700. long ) to determine the contour of the layer divider. The computing system 1100 may be based on eliminating (or filtering out) the longest extension line (ExtLine) from the perimeter of the detection area 5700 to each side of the quadrilateral forming the detection area 5700 in the 2D image information of the top surface of the stack of objects. long ) is used to identify the contour of the layer divider.

[0138] In the detection area portion removal operation 675, the computing system 1100 may determine the longest extension line (Ext Line) of each side of the quadrilateral forming the detection area 5700. long ).exist Figure 5C In the example of FIG. 5 , the computing system 1100 may determine the first longest extension line (1 st ExtLine long The computing system 1100 may determine the second longest extension line (2 nd Ext Line long The computing system 1100 may determine the third longest extension line (3 ) from the extension lines 5500F / 5500G / 5500H of the top side of the quadrilateral forming the detection area 5700. rd Ext Line long ). Since there is no extension line on the left side of the quadrilateral forming the detection area 5700, it may not be possible to perform this operation on the fourth side of the quadrilateral. In some examples, there will be no extension line on the fourth side of the quadrilateral because the layer divider 5600A may extend all the way to the perimeter of the fourth side.

[0139] The computing system 1100 may determine the outline of the layer divider by eliminating (or filtering out) a portion of the detection area 5700 corresponding to the length of the longest extension line of each side of the quadrilateral forming the detection area 5700. For example, the computing system 1100 may determine the outline of the layer divider by eliminating (or filtering out) a portion of the detection area 5700 corresponding to the length of the longest extension line of the corresponding side. st Ext Line long) of equal length, and filtering out the second longest extension line (2 nd Ext Line long ) and filtering out the third longest extension line (3 rd Ext Line long ) to determine the outline of the layer divider. Since no extension line is detected on the left side of the quadrilateral forming the detection area 5700, the computing system 1100 can determine that the detection area 5700 extends to the perimeter of the quadrilateral forming the detection area 5700 and does not need to be eliminated (or filtered out).

[0140] In another example, if no extension line is detected on the left side of the quadrilateral forming the detection area 5700, the computing system 1100 may select 1 st Ext Line long , 2 nd Ext Line long and 3 rd Ext Line long , to ensure that the risk of image processing failure is avoided. If the image processing fails to detect the extension line on the left side of the quadrilateral forming the detection area 5700, the outline of the layer divider may be determined to be larger than the actual layer divider, resulting in a potential accident when the robot attempts to pick up or remove the layer divider from the stack of objects. Therefore, out of caution and to avoid accidents, when no extension line is detected on the left side of the quadrilateral forming the detection area 5700, the computing system 1100 can eliminate (or filter out) the portion of the detection area 5700 corresponding to the length of the longest extension line from the left side of the quadrilateral forming the detection area 5700. This is done because if the outline of the layer divider is determined to be larger than the actual layer divider, an accident may occur when the robot attempts to pick up or remove the layer divider from the stack of objects. The computing system 1100 can eliminate (or filter out) the portion of the detection area 5700 corresponding to the length of the longest extension line from the left side of the quadrilateral forming the detection area 5700.

[0141] Figure 6B The top portion of 2D image 6000A illustrates detection region 5700 delimited by boundary 5200A and extension lines 5500A / 5500B / 5500C / 5500D / 5500E arranged in detection region 5700, starting from boundary 5200A and progressing toward the center of 2D image 6000A. Figure 6BThe bottom portion of FIG. 5 illustrates the longest extension line (ExtLine 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G) in the detection area 5700 by removing the longest extension line (ExtLine 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G) from each side of the quadrilateral forming the detection area 5700. long ) as described above for operation 674.

[0142] After determining the outline of the layer divider, in the outline sharpening operation 676, the computing system 1100 can perform a morphological operation to sharpen the outline of the identified layer divider. In some examples, the morphological operation is based on a square structure element, the side length of which is equal to the length of the largest extension line in the extension lines 5500A / 5500B / 5500C / 5500D / 5500E / 5500F / 5500G / 5500H. The morphological operation can include an erosion operation based on sliding the structure element over the layer divider and retaining the minimum pixel value covered by the structure element. Erosion is a morphological operation that removes pixels on the boundary of an object in an image, effectively shrinking the object. It works by sliding the structure element over the image and retaining the pixel at its origin only when all pixels under the structure element match the element, resulting in a thinner object and eliminating small protrusions. Erosion erodes the boundaries of foreground (usually white) objects in the image.

[0143] Morphological operations can include a dilation operation based on sliding a structuring element over a layer divider and retaining the maximum pixel value covered by the structuring element. Dilation adds pixels to the boundaries of objects in the image. The structuring element is slid over the image and if any pixel under the structuring element is 1, then the pixel at the origin is set to 1. Dilation is used to emphasize features and close small holes or gaps in objects.

[0144] In some examples, the morphological operation may include an erosion operation followed by a dilation operation. In some examples, the morphological operation may include performing a dilation operation followed by an erosion operation.

[0145] In the trajectory determination operation 480, when the computing system 1100 determines the contour of the layer divider (e.g., in response thereto), the object manipulation planning module 1126 of the computing system 1100 may determine a trajectory based on the contour of the layer divider to control the movement of the robot arm of the robot 1300 or the robot 3300. For example, the object manipulation planning module 1126 may configure the robot 3300 to grasp and remove the layer divider 3600 from the top surface 3120 of the object 3100A / 3100B / 3100C / 3100D based on the trajectory. In some examples, the trajectory configures the robot arm 330 to grasp and remove the layer divider 3600 from above the stack 3700 of objects using the end effector device 3330. In some examples, the trajectory configures the gripper of the end effector device 3330 to grasp the layer divider 3600.

[0146] Figure 8 An example of an environment is illustrated in which two layers 3710 and 3720 of objects 3100A / 3100B / 3100C / 3100D / 3100E / 3100F / 3100G / 3100H are stacked on a pallet 3400, with a layer divider 3600 disposed on a top surface of the second layer. The first layer 3710 includes objects 3100A / 3100B / 3100C / 3100D, and the second layer 3720 includes objects 3100E / 3100F / 3100G / 3100H. The layer divider 3600 is disposed on a top surface of the second layer 3720. Figure 8 The existence of the layer divider 3600 can be determined based on the above-mentioned methods and systems.

[0147] The specific implementation of the examples of the disclosed technology above is not intended to be exhaustive or to limit the disclosed technology to the precise form disclosed above. Although the specific examples of the disclosed technology are described above for illustrative purposes, various equivalent modifications are possible within the scope of the disclosed technology, as will be recognized by those skilled in the relevant art. For example, although the process or block is presented in a given order, alternative implementations can perform routines with steps in different orders, or adopt a system with blocks, and some processes or blocks can be deleted, moved, added, subdivided, combined and / or modified to provide alternatives or sub-combinations. Each of these processes or blocks can be implemented in various different ways. In addition, although the process or block is sometimes shown to be performed in series, these processes or blocks can be performed or implemented in parallel instead, or can be performed at different times. In addition, any specific numbers mentioned herein are only examples, and alternative implementations can adopt different values ​​or ranges.

[0148] These and other changes can be made to the disclosed technology in light of the above specific embodiments. Although the specific embodiments describe certain examples of the disclosed technology and the best mode contemplated, the disclosed technology can be practiced in many ways, no matter how detailed the above description appears in the text. The details of the system may vary considerably in its specific implementation while still being covered by the technology disclosed herein. As described above, specific terms used in describing certain features or aspects of the disclosed technology should not be taken as implying that the term is redefined herein to be limited to any specific characteristic, feature, or aspect of the disclosed technology associated with the term. Therefore, the present invention is not limited unless limited by the appended claims. In general, the terms used in the following claims should not be interpreted as limiting the disclosed technology to the specific examples disclosed in the specification, unless such terms are explicitly defined in the above specific embodiment section.

[0149] It will be apparent to those of ordinary skill in the relevant art that other suitable modifications and adaptations may be made to the methods and applications described herein without departing from the scope of any embodiment. The embodiments described above are illustrative examples and should not be construed as limiting the disclosure to these specific embodiments. It should be understood that the various embodiments disclosed herein may be combined in combinations different from those specifically presented in the description and drawings. It should also be understood that, depending on the examples, certain behaviors or events of any process or method described herein may be performed in different orders, may be added, merged or completely excluded (e.g., all described behaviors or events may not be necessary for performing the method or process). In addition, although for the purpose of clarity, certain features of the embodiments herein are described as being performed by a single component, module or unit, it should be understood that the features and functions described herein may be performed by any combination of components, units or modules. Therefore, various changes and modifications may be affected by those skilled in the art without departing from the spirit or scope of the present invention as defined in the appended claims.

[0150] Additional embodiments include:

[0151] Embodiment 1. A computing system for detecting a layer divider between stacks of objects, comprising: at least one camera; a control system configured to communicate with a robot having a robotic arm including an end effector device; and a processing circuit configured to: identify an outline of the stack of objects based on 3D image information obtained from at least one image captured by the at least one camera, identify a top surface of the stack of objects based on the outline of the stack of objects, detect extension lines extending from a perimeter of the top surface to a center of the top surface based on 2D image information of the top surface of the stack of objects, each extension line corresponding to at least one of a gap between objects in the stack of objects or an edge of an object, detect the presence of a layer divider based on a length of each extension line being less than a line length threshold, identify the outline of the layer divider based on a point on the top surface corresponding to a termination point of the extension line, and determine a trajectory based on the outline of the layer divider to control the movement of the robotic arm.

[0152] Embodiment 2. The system of embodiment 1, wherein the termination point is distal to the perimeter of the top surface.

[0153] Embodiment 3. The system of embodiment 1, wherein the processing circuit is further configured to detect the extension line on the top surface based on the depth gradient between the objects detected in the 3D image information.

[0154] Embodiment 4. The system of Embodiment 1, wherein the processing circuit is further configured to detect the presence of the layer divider based on the extension line not extending from the perimeter of the top surface to the center of the top surface.

[0155] Embodiment 5. The system of Embodiment 1, wherein the processing circuit is further configured to identify an outline of the layer divider based on a portion of the image information of the top surface of the stack of removed objects from the perimeter to the termination point of each extension line.

[0156] Embodiment 6. A system as described in Embodiment 1, wherein the processing circuit is further configured to: determine a surface area of ​​the layer divider in response to detecting the presence of the layer divider; and verify the presence of the layer divider in response to the surface area being greater than a minimum surface area threshold.

[0157] Embodiment 7. A system as described in Embodiment 1, wherein the processing circuit is further configured to: determine a surface area of ​​the layer divider in response to detecting the presence of the layer divider; and verify the presence of the layer divider in response to the surface area being less than a maximum surface area threshold.

[0158] Embodiment 8. A system as described in Embodiment 1, wherein the processing circuit is further configured to: in response to detecting the presence of a layer divider, determine the smoothness of the layer divider based on changes in depth information of neighboring pixels in 3D image information corresponding to an upper surface of the layer divider; and in response to the smoothness being greater than a smoothness threshold, verify the presence of the layer divider.

[0159] Embodiment 9. A system as described in Embodiment 1, wherein the processing circuit is further configured to: generate image features based on 2D image information and 3D image information corresponding to the upper surface of the layer divider; identify texture artifacts based on the image features; and in response to identifying texture artifacts, determine that the presence of the layer divider is a false positive.

[0160] Embodiment 10. The system of Embodiment 1, wherein the trajectory configures the robot to grasp and remove a layer divider.

[0161] Embodiment 11. The system of Embodiment 1, wherein the line length threshold is less than half the length of the smallest side of a quadrilateral defining the top surface.

[0162] Embodiment 12. The system of Embodiment 1, wherein the line length threshold is less than one quarter of the length of the smallest side of a quadrilateral defining the top surface.

[0163] Embodiment 13. A system as described in Embodiment 6, wherein the minimum surface area threshold is equal to the surface area of ​​the largest object in the stack of objects.

[0164] Embodiment 14. A system as described in Embodiment 6, wherein the minimum surface area threshold is equal to the surface area of ​​the largest side of the largest object in the stack of objects.

[0165] Embodiment 15. The system of embodiment 7, wherein the surface area maximum threshold is equal to the surface area of ​​the stacked pallet on which the objects are arranged.

[0166] Embodiment 16. A system as described in Embodiment 1, wherein the processing circuit is further configured to: in response to detecting the presence of the layer divider, determine the thickness of the layer divider based on the 3D image information; and in response to the thickness being less than a thickness threshold, verify the presence of the layer divider.

[0167] Embodiment 17. A system as described in Embodiment 1, wherein the processing circuit is further configured to: determine the weight of the layer divider in response to detecting the presence of the layer divider; and verify the presence of the layer divider in response to the weight being less than a weight threshold.

[0168] Embodiment 18. A system as described in Embodiment 9, wherein the texture artifacts include any one or any combination of logos, graphics, labels, markings, logos, scan codes, universal product codes ("UPC"), packaging tape, visual patterns, and intensity variations.

[0169] Embodiment 19. A system as described in Embodiment 1, wherein the processing circuit is further configured to perform a morphological operation to sharpen the outline of the layer divider, wherein the morphological operation is based on a square structuring element having a side length equal to the length of the largest extended line among the extended lines.

[0170] Embodiment 20. The system of embodiment 1, wherein the morphological operation comprises a dilation operation based on sliding a structuring element over a layer divider and retaining a maximum pixel value covered by the structuring element.

[0171] Embodiment 21. A system as described in Embodiment 1, wherein the morphological operation includes an erosion operation based on sliding a structuring element over a layer divider and retaining a minimum pixel value covered by the structuring element.

[0172] Embodiment 22. A system as described in Embodiment 1, wherein the morphological operation includes performing a dilation operation and then performing an erosion operation.

[0173] Embodiment 23. The system of Embodiment 1, wherein the trajectory configures the robotic arm to grasp and remove a layer divider from above a stack of objects using an end effector device.

[0174] Embodiment 24. The system of Embodiment 1, wherein the trajectory configures a gripper of the end effector assembly to grip the layer divider according to at least one of a plurality of gripping models.

[0175] Embodiment 25. A system as described in Embodiment 1, wherein identifying the top surface of the stack of objects includes: positioning the at least one camera toward the stack of objects; capturing a 2D image of a top view of the stack of objects based on a contour of the stack of objects; and determining the top surface of the stack of objects based on the 2D image.

[0176] Embodiment 26. A non-transitory computer-readable medium having instructions thereon, which when executed by a control circuit configure the control circuit to: identify an outline of a stack of objects based on 3D image information obtained from at least one image captured by at least one camera, identify a top surface of the stack of objects based on the outline of the stack of objects, detect extension lines extending from a perimeter of the top surface to a center of the top surface based on 2D image information of the top surface of the stack of objects, each extension line corresponding to at least one of a gap between objects in the stack of objects or an edge of an object, detect the presence of a layer divider based on a length of each extension line being less than a line length threshold, identify an outline of the layer divider based on a point on the top surface corresponding to an end point of the extension line, and determine a trajectory based on the outline of the layer divider to control movement of a robot arm of a robot.

[0177] Embodiment 27. The non-transitory computer readable medium of Embodiment 26, wherein the termination point is distal to the perimeter of the top surface.

[0178] Embodiment 28. The non-transitory computer readable medium of Embodiment 26, wherein the control circuit is further configured to detect the extension line on the top surface based on a depth gradient between objects detected in the 3D image information.

[0179] Embodiment 29. The non-transitory computer-readable medium of Embodiment 26, wherein the control circuit is further configured to detect the presence of the layer divider based on the absence of an extension line extending from the perimeter of the top surface to the center of the top surface.

[0180] Embodiment 30. A non-transitory computer-readable medium as described in Embodiment 26, wherein the control circuit is further configured to identify the outline of the layer divider based on a portion of the image information of the top surface of the stack of removed objects from the perimeter to the termination point of each extension line.

[0181] Embodiment 31. A processor-implemented method for detecting a layer divider between stacks of objects, comprising: identifying an outline of the stack of objects based on 3D image information obtained from at least one image, identifying a top surface of the stack of objects based on the outline of the stack of objects, detecting extension lines extending from a perimeter of the top surface to a center of the top surface based on 2D image information of the top surface of the stack of objects, each extension line corresponding to at least one of a gap between objects in the stack of objects or an edge of an object, detecting the presence of a layer divider based on a length of each extension line being less than a line length threshold, identifying the outline of the layer divider based on a point on the top surface corresponding to an end point of the extension line, and determining a trajectory based on the outline of the layer divider to control movement of a robot arm of a robot.

[0182] Embodiment 32. The method of embodiment 31, wherein the termination point is distal to the perimeter of the top surface.

[0183] Embodiment 33. The method of Embodiment 31 further includes detecting an extension line on the top surface based on a depth gradient between objects detected in the 3D image information.

[0184] Embodiment 34. The method of Embodiment 31 further comprises detecting the presence of a layer divider based on a lack of an extension line extending from a perimeter of the top surface to a center of the top surface.

[0185] Embodiment 35. A method as described in Embodiment 31, wherein identifying the outline of the layer divider includes identifying the outline of the layer divider based on a portion of the image information of the top surface of the stack of removed objects, from the perimeter to the termination point of each extension line.

Claims

1. A computing system for detecting a layer divider between a stack of objects, comprising: At least one camera; a control system configured to communicate with a robot having a robotic arm including an end effector device; as well as The processing circuit is configured to: identifying an outline of a stack of objects based on 3D image information obtained from at least one image captured by the at least one camera, identifying a top surface of the stack of objects based on a contour of the stack of objects, detecting extension lines extending from a perimeter of the top surface toward a center of the top surface based on the 2D image information of the top surface of the stack of objects, each extension line corresponding to at least one of a gap between objects of the stack of objects or an edge of an object, detecting the presence of a layer divider based on the length of each extended line being less than a line length threshold, identifying the outline of the layer divider based on points on the top surface corresponding to the termination points of the extended lines, and The trajectory is determined based on the contour of the layer divider to control the movement of the robot arm.

2. The system of claim 1, wherein the termination point is located distal to the perimeter of the top surface. 3 . The system of claim 1 , wherein the processing circuit is further configured to detect an extension line on the top surface based on a depth gradient between objects detected in the 3D image information. 4 . The system of claim 1 , wherein the processing circuit is further configured to detect the presence of the layer divider based on the extension line not extending from the perimeter of the top surface to the center of the top surface.

5. The system of claim 1, wherein the processing circuit is further configured to identify an outline of the layer divider based on a portion of the image information of the top surface of the stack of removed objects from the perimeter to a termination point of each extension line.

6. The system of claim 1 , wherein the processing circuit is further configured to: In response to detecting the presence of the layer divider, determining a surface area of ​​the layer divider; and In response to the surface area being greater than a surface area minimum threshold, the presence of a layer divider is verified.

7. The system of claim 1 , wherein the processing circuit is further configured to: In response to detecting the presence of the layer divider, determining a surface area of ​​the layer divider; and In response to the surface area being less than a surface area maximum threshold, the presence of a layer divider is verified.

8. The system of claim 1 , wherein the processing circuit is further configured to: In response to detecting the presence of the layer divider, determining a smoothness of the layer divider based on a change in depth information of neighboring pixels in the 3D image information corresponding to an upper surface of the layer divider; and In response to the smoothness being greater than a smoothness threshold, existence of a layer divider is verified.

9. The system of claim 1 , wherein the processing circuit is further configured to: generating an image feature based on 2D image information and 3D image information corresponding to an upper surface of the layer divider; Identifying texture artifacts based on image features; and In response to identifying a texture artifact, determining that the presence of a layer divider is a false positive.

10. The system of claim 1, wherein the trajectory configures the robot to grasp and remove a layer divider.

11. A non-transitory computer readable medium having instructions thereon, which when executed by a control circuit configure the control circuit to: identifying an outline of a stack of objects based on 3D image information obtained from at least one image captured by at least one camera, identifying a top surface of the stack of objects based on a contour of the stack of objects, detecting extension lines extending from a perimeter of the top surface toward a center of the top surface based on the 2D image information of the top surface of the stack of objects, each extension line corresponding to at least one of a gap between objects of the stack of objects or an edge of an object, detecting the presence of a layer divider based on the length of each extended line being less than a line length threshold, identifying the outline of the layer divider based on points on the top surface corresponding to the termination points of the extended lines, and A trajectory is determined based on the profile of the layer divider to control the movement of a robot arm of the robot. 12 . The non-transitory computer readable medium of claim 11 , wherein the termination point is located at a distal end of a perimeter of the top surface. 13 . The non-transitory computer readable medium of claim 11 , wherein the control circuit is further configured to detect the extension line on the top surface based on a depth gradient between objects detected in the 3D image information. 14 . The non-transitory computer readable medium of claim 11 , wherein the control circuit is further configured to detect the presence of the layer divider based on the extension line not extending from the perimeter of the top surface to the center of the top surface.

15. The non-transitory computer readable medium of claim 11, wherein the control circuit is further configured to identify an outline of the layer divider based on a portion of the image information of the top surface of the stack of removed objects from the perimeter to a termination point of each extension line.

16. A processor-implemented method for detecting a layer divider between a stack of objects, comprising: identifying an outline of a stack of objects based on 3D image information obtained from at least one image, identifying a top surface of the stack of objects based on a contour of the stack of objects, detecting extension lines extending from a perimeter of the top surface toward a center of the top surface based on the 2D image information of the top surface of the stack of objects, each extension line corresponding to at least one of a gap between objects of the stack of objects or an edge of an object, detecting the presence of a layer divider based on the length of each extended line being less than a line length threshold, identifying the outline of the layer divider based on points on the top surface corresponding to the termination points of the extended lines, and A trajectory is determined based on the profile of the layer divider to control the movement of a robot arm of the robot.

17. The method of claim 16, wherein the termination point is located distal to the perimeter of the top surface. 18 . The method of claim 16 , further comprising detecting an extension line on the top surface based on a depth gradient between objects detected in the 3D image information.

19. The method of claim 16, further comprising detecting the presence of a layer divider based on an extension line not extending from a perimeter of the top surface to a center of the top surface.

20. The method of claim 16, wherein the identifying of the outline of the layer divider comprises identifying the outline of the layer divider based on a portion of the image information of the top surface of the stack of removed objects from the perimeter to the termination point of each extension line.

Images