Method and computer system for performing container detection and object detection
The computer system addresses robotic interaction challenges by using container edge measurements to compensate for noise, ensuring precise object pose determination and robotic interaction accuracy.
Patent Information
- Application Number
- DE102020124221
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-07-15
- Filing Date
- 2020-09-17
- Publication Date
- 2024-11-07
- Estimated Expiration
- 2040-09-17
AI Technical Summary
Existing robotic systems face challenges in accurately determining the pose of objects within containers due to measurement errors and image noise, which can affect the precision of robotic interactions, particularly when millimeter-level accuracy is required.
A computer system and method that utilizes spatial structure information from a camera to determine the pose of objects within containers by compensating for measurement errors by using measurements from the container edges, which are less affected by noise, to plan precise robotic interactions.
Enhances the accuracy of robotic interactions by robustly determining object poses despite image noise, ensuring precise movement and grasping of objects within containers.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] The present disclosure relates to computer systems and methods for container detection and object detection.
[0002] With increasing automation, robots are being used in more environments, such as warehousing and retail environments. For example, robots can be used to interact with goods or other objects in a warehouse. The robot's movement can be fixed or based on an input, such as information generated by a sensor in the warehouse. DE 10 2013 001 603 B4 discloses an image processing device aimed at preventing misdetection or non-detection of workpieces and reducing the processing time for workpiece detection by adjusting the search window to be used when detecting the workpieces by the image processing device.The image processing device photographs an image using a camera, sets a search window in the image as an area for detecting an object accommodated in a container device, and detects the object. DE 10 2013 012 224 B4 shows a device for removing objects that detects the positions of objects stored loosely or as loose goods in a storage container and removes the detected objects using a robot. US 2019 / 0 351 547 A1 discloses a holding device that includes, among other things, a first detection unit configured to detect a displayed held object and a second detection unit configured to detect a label in the vicinity of the held object upon detection of the held object by the first detection unit. US 10 369 698 B1 describes a robot control system and a method for automatic camera calibration.US 10 150 564 B2 describes an automatic storage system for use in a warehouse with a storage rack comprising drawers divided into several compartments for containing various packages. The drawers are located at specific locations in individual cells in the storage racks and can be opened and closed. JP S61-206 709 A discloses a workstation with a robot arm for storing and retrieving goods. The object of the present invention relates to facilitating robot interactions with the contents of a container, such as goods or any other objects arranged in the container.
[0003] This object is achieved by a computer system having the features of claim 1, a method having the features of claim 19, and / or a non-transitory computer-readable medium with instructions for performing object detection having the features of claim 16. The computer system comprises a communication interface configured to communicate with a robot having a robot arm having a spatial structure detection camera arranged on the robot arm, wherein the spatial structure detection camera has a camera field of view. The at least one processing circuit is configured to execute the method when an object is located or has been located within a container in the camera field of view while the container is in an open position.The method comprises receiving spatial structure information including depth information for an environment in the camera field of view, wherein the spatial structure information is generated by the spatial structure capture camera, and determining a container pose based on the spatial structure information, wherein the container pose is for describing at least one of an orientation for the container or a depth value for at least a portion of the container. The method further includes determining an object pose based on the container pose, wherein the object pose is for describing at least one of an orientation for the object or a depth value for at least a portion of the object, and issuing a motion command for effecting a robot interaction with the object, wherein the motion command is generated based on the object pose. The Fig. 1A through 1E illustrate a computer system configured to receive and process spatial structure information and / or detected object identification information consistent with embodiments hereof. The Fig. 2A through 2C provide block diagrams illustrating a computer system configured to receive and process spatial structure information and / or detected object identification information consistent with embodiments hereof. The Fig. 3A to 3D illustrate an environment with multiple bins (e.g., drawers) and a robot for interacting with the bins based on spatial structure information generated by a spatial structure sensing camera, according to one embodiment thereof. Fig. 4 provides a flowchart illustrating a method for determining information about an object disposed in a container, according to one embodiment thereof. The Fig. 5A to 5C illustrate a container and objects within the container according to one embodiment thereof. The Fig. 6A to 6C illustrate spatial structure information describing a container or objects arranged in the container, according to one embodiment thereof. Fig. 6D illustrates a container and objects disposed therein according to one embodiment thereof. Fig. 6E illustrates spatial structure information describing a container or objects arranged in the container, according to one embodiment thereof. Fig. 6F illustrates a relationship between a container surface and a container edge according to one embodiment hereof. Fig. 7A illustrates an environment including both a spatial structure detection camera and an object identifier detection device, or more particularly a barcode detection device, according to one embodiment thereof. The Fig. 7B and Fig. 7C illustrate determining a barcode location used to determine one or more object locations, according to one embodiment hereof. Fig. 8 illustrates barcodes located adjacent to an object used to determine a location for the object, according to one embodiment thereof. Fig. 9 illustrates spatial structure information and / or acquired object identification information or, in particular, acquired barcode information covering only a portion of a container, according to one embodiment thereof. Fig. 10 illustrates a segmentation that connects different regions of the container to different segments, according to one embodiment hereof. The Fig. 11A to 11C illustrate the movement of a container from a closed position to an open position according to one embodiment thereof.
[0004] One aspect of the present disclosure relates to facilitating robotic interaction with the contents of a drawer or other container, such as merchandise or any other objects disposed within the container (the term "or" may be used herein to refer to "and / or"). The robotic interaction may include, for example, a robotic hand grasping or otherwise picking up an object disposed within the container. The robotic interaction may occur, for example, in a warehouse, a retail store, or other environment. In some cases, facilitating the robotic interaction includes determining a pose of the object within the container, where the pose may relate to at least one of an orientation or depth of the object relative to a camera or other reference point, such that the robotic hand may be appropriately moved to retrieve or otherwise pick up the object.
[0005] Various embodiments relate to determining the pose of an object (also referred to as object pose) by performing open container detection, in which information about an open container is determined, wherein the object may be located within the container. These embodiments may provide a way of determining the object pose that is, for example, more robust and tolerant to image noise or other sources of measurement error. The image noise may, for example, affect a point cloud or other spatial structure information used to measure the object. The measurement error introduced into the point cloud may cause an erroneous determination of, for example, the orientation and / or depth of the object.In some cases, an error of even a few millimeters or degrees can affect robot interaction, which under certain circumstances may rely on millimeter-level accuracy or better in determining a relative location between the robot hand and the object. Since a measurement error for the object can prevent or impede such accuracy, one aspect of the present disclosure relates to the use of measurements regarding the container in which the object is disposed and the use of such measurements to infer or otherwise determine a pose or other information regarding the object.
[0006] In some cases, image noise may also affect the direct measurement of a portion of the container, such as a surface on which the object is arranged (the surface may also be referred to as the container surface). One aspect of the present disclosure relates to compensating for measurement errors affecting the container surface by performing measurements on a different portion of the container, such as an edge of the container (also referred to as the container edge). In these cases, the container edge may occupy a space that is less affected by image noise and may therefore provide more reliable or trustworthy measurements. The measurements relating to the container edge may be used to infer or otherwise determine a pose or other information relating to the container surface. Such a determination may, for example,based on a known distance separating the container surface and the container edge.
[0007] In some cases, measurements taken relative to the container can be used for motion planning, such as motion planning for removing an object from the container. For example, measurements relative to the container's rim can provide information about the location of a side wall of the container. When a robot removes the object from the container, an object's motion path can be planned to avoid a collision between the side wall of the container and the robot or the object. In some cases, measurements relative to the container can be used to virtually divide the container into different segments, as discussed in more detail below.
[0008] In some cases, facilitating robot interaction with an object may rely on information about an object identifier (if any) located on the object. The object identifier may comprise a visual marker, such as a barcode, a logo or symbol (e.g., an alphanumeric symbol), or other visual pattern that identifies the object. In some cases, the object identifier may be printed on a surface of the object. In some cases, the object identifier may be printed on a sticker or other layer of material adhered or otherwise attached to the object. If the object is a box containing one or more items, the object identifier may identify the one or more items, or more generally, a content of the box.The information regarding the object identifier used to facilitate robot interaction may include, for example, a location of the object identifier (also referred to as an object identifier location) or information encoded into the object identifier, such as information encoded in a barcode. In some cases, the object identifier location may be used to narrow down which part of a point cloud or other spatial structure information should be searched to detect a particular object. For example, if the object identifier is a barcode location, such a search may be limited to a part of the point cloud corresponding to a region around the barcode location. Such an embodiment may facilitate a more targeted and efficient search for the object.In some cases, if an object size is encoded into the barcode or other object identifier, this information can be used to locate the object using the point cloud or other spatial structure information, or to plan how the robot can grasp or otherwise interact with it.
[0009] Fig. Figure 1A illustrates a system 100 for processing spatial structure information for object detection, as discussed in more detail below. In the embodiment of Fig. 1A, the system 100 may include a computer system 101 and a spatial structure detection camera 151 (also referred to as a spatial structure detection device 151). In this example, the spatial structure detection device 151 may be configured to generate spatial structure information (also referred to as spatial information or spatial structure data) that includes depth information about an environment in which the spatial structure detection camera 151 is located, or in particular about an environment in the field of view of the camera 151 (also referred to as the camera field of view). The computer system 101 in Fig. 1A may be configured to receive and process the spatial structure information. For example, computer system 101 may be configured to use the depth information in the spatial structure information to distinguish between different structures in the camera field of view, or more generally to identify one or more structures in the camera field of view. The depth information, in this example, may be used to determine an estimate of how the one or more structures are spatially arranged in three-dimensional (3D) space.
[0010] In one example, the spatial structure detection camera 151 may be located in a warehouse, a sales area (e.g., a store), or other premises. In such an example, the warehouse or sales area may include various goods or other objects. The spatial structure detection camera 151 may be used to capture information about the objects and / or structures containing the objects, such as drawers or other types of containers. As stated above, the spatial structure detection camera 151 may be configured to generate spatial structure information that may, for example, describe how a structure for a good and / or how a structure of a container is / are arranged in 3D space. The computer system 101 in such an example may be configured to receive and process the spatial structure information from the spatial structure detection camera 151.The computer system 101 may be located on the same premises or remotely. For example, the computer system 101 may be part of a cloud computing platform hosted in a data center remote from the warehouse or sales floor and may communicate with the spatial structure detection camera 151 via a network connection.
[0011] In one embodiment, the system 100 may be a robot operating system for interacting with various objects in an environment of the spatial structure detection camera 151. Fig. For example, Figure 1B illustrates a robot operating system 100A that includes an embodiment of the system 100 of Fig. 1A. Robot operating system 100A may include computer system 101, spatial structure sensing camera 151, and a robot 161. In one embodiment, robot 161 may be used to interact with one or more objects in the vicinity of spatial structure sensing camera 151, such as goods or other objects in a warehouse. For example, robot 161 may be configured to pick up goods from a drawer or other container and move the goods from the container to another location (e.g., a conveyor belt outside the drawer).
[0012] In one embodiment, the computer system 101 of the Fig. 1A and Fig. 1B may form or be part of a robot control system (also referred to as a robot controller) that is part of the robot operating system 100A. The robot control system may be a system configured to generate, for example, motion commands or other commands for the robot 161. In such an embodiment, the computer system 101 may be configured to generate such commands, for example, based on spatial structure information generated by the spatial structure detection camera 151. In such an embodiment, the computer system 101 may form or be part of an image processing system. The image processing system may be a system that generates, for example, image processing information describing an environment in which the robot 161 is located, or in particular describing an environment in which the spatial structure detection camera 151 is located.The image processing information may include spatial structure information, which may also be referred to as 3D information or 3D image information because it may indicate how a structure is laid out or otherwise arranged in 3D space. In some cases, the robot 161 may include a robot arm having a robot hand or other end effector forming one end of the robot arm, and the spatial structure information may be used by the computer system 101 to control the placement of the robot hand. In some cases, when the computer system 101 forms a vision system, the vision system may be part of the robot control system discussed above or separate from the robot control system. If the vision system is separate from the robot control system, it may be configured to output information about the environment in which the robot 161 is located.In such an example, the robot control system may receive such information and control a movement of the robot 161 based on this information.
[0013] In one embodiment, the system 100 may include an object identifier detection device 152, such as a barcode detection device (also referred to as a barcode reader). In particular, Fig. 1C illustrates a system 100B (which is an embodiment of system 100 / 100A) that includes computer system 101, spatial structure detection camera 151, robot 161, and further object identifier detection device 152. In some cases, object identifier detection device 152 may be configured to detect an object identifier disposed on or adjacent to an object. As noted above, the object identifier may be a visual marker that identifies the object. If the object is a box or other object for containing merchandise or any other article, in one embodiment, the object identifier may identify the article or other contents of the box. As also noted above, in some examples, the object identifier may be a barcode. In some cases, the barcode may include a spatial pattern, such as a series of dark stripes or an array of dark squares (e.g.,a QR code) or any other barcode within a field of view of the object identifier capture device 152 (e.g., barcode capture device). The barcode may be affixed, for example, to a commodity or other object in a warehouse. The object identifier capture device 152 may be configured to capture information about the object identifier. This information (which may also be referred to as captured object identifier information) may include information encoded into the object identifier, a location of the object identifier (also referred to as object identifier location), or any other information related to the object identifier. If the object identifier is a barcode, the information encoded into the barcode may include, for example, a stock unit (SKU) code or a universal product code (UPC).
[0014] In one embodiment, the object identifier detection device 152 and / or the spatial structure detection camera 151 may be attached to a stationary attachment point, such as an attachment point that is stationary within a warehouse or a sales floor. In one embodiment, the spatial structure detection camera 151 and / or the object identifier detection device 152 may be attached to a robot arm of the robot 161. In a more specific example, the object identifier detection device 152 and / or the spatial structure detection camera 151 may be attached to, or disposed on (or near) a robot hand or other end effector forming one end of the robot arm. In such an example, the object identifier detection device 152 and the spatial structure detection camera 151 may be respectively referred to as a handheld object identifier detection device (e.g.,an available barcode reader) and a handheld spatial structure sensing camera. In some cases, the computer system 101 may be configured to control the robot 161 to move the handheld spatial structure sensing camera and / or the handheld object recognition detection device to an optimal location for sensing an environment of the robot 161, as discussed in more detail below.
[0015] In one embodiment, when computer system 101 is part of a robot control system, computer system 101 may be configured to generate one or more motion commands for controlling the movement of robot 161, as discussed in more detail below. These motion commands may include, for example, object motion commands, sensor motion commands, and container motion commands. A sensor motion command may be used to move spatial structure detection camera 151 and / or object identifier detection device 152. A container motion command may be used to move a container containing a commodity or other object, such as a motion command to open or close the container. An object motion command may be used to move commodity or another object in a warehouse or other premises, or more specifically, an object disposed within the container.In one embodiment, the components of system 100 may be configured to communicate via a network and / or a storage device. In particular, FIG. Fig. 1D illustrates a system 100C that is an embodiment of the system 100 / 100A / 100B of the Fig. 1A to 1C. System 100C includes computer system 101, spatial structure capture camera 151, robot 161, object identifier capture device 152, and further includes a network 199 and a data storage device 198 (or any other type of non-transitory computer-readable medium) separate from computer system 101. In some cases, storage device 198 may be configured to store information generated by object identifier capture device 152, spatial structure capture camera 151, and / or robot 161 and to make the stored information available to computer system 101. In such an example, computer system 101 may be configured to access the stored information by retrieving (or more generally, receiving) the information from data storage device 198.
[0016] In Fig. 1D, the storage device 198 may comprise any type of non-transitory computer-readable medium (or media), which may also be referred to as a non-transitory computer-readable storage device. Such a non-transitory computer-readable medium or storage device may be configured to store and provide access to stored information (also referred to as stored data).Examples of the non-transitory computer-readable medium or storage device may include, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, such as, but not limited to, a computer diskette, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a solid-state drive, static random access memory (SRAM), portable read-only memory (CD-ROM), a digital versatile disk (DVD), and / or a memory stick.
[0017] In one embodiment, network 199 may facilitate communication between computer system 101, spatial structure capture camera 151, object identifier capture device 152, and / or robot 161. For example, computer system 101 and / or storage device 198 may receive information generated by spatial structure capture camera 151 and / or object identifier capture device 152 (e.g., spatial structure information or captured object identifier information) via network 199. In such an example, computer system 101 may be configured to provide one or more commands (e.g., motion commands) to robot 161 via network 199. Network 199 may provide a single network connection or a series of network connections for computer system 101 to receive information and / or issue a command in accordance with embodiments thereof.
[0018] In Fig. 1D, the network 199 may be connected via wired or wireless connections. Wired connections may include Digital Subscriber Line (DSL), coaxial cable, or fiber optic lines. Wireless connections may include Bluetooth®, Bluetooth Low Energy (BLE), ANT / ANT+, ZigBee, Z-Wave, Thread, Wi-Fi®, Worldwide Interoperability for Microwave Access (WiMAX®), Mobile WiMAX®, WiMAX®-Advanced, NFC, SigFox, LoRa, Random Phase Multiple Access (RPMA), Weightless-N / P / W, an infrared channel, or a satellite band. The wireless connections may also include any cellular network standards for communication between mobile devices, including standards that qualify as 2G, 3G, 4G, or 5G. Wireless standards may use various channel access methods, such as FDMA, TDMA, CDMA, OFDM, or SDMA. In some embodiments, different types of information may be transmitted over different connections and standards.In other embodiments, the same types of information may be transmitted over different connections and standards. Network communication may occur over any suitable protocol, including, for example, HTTP, TCP / IP, UDP, Ethernet, ATM, etc.
[0019] The network 199 may be any type and / or form of network. The geographic scope of the network can vary greatly, and the network 199 may be a corporate area network (BAN), a personal area network (PAN), a local area network (LAN), e.g., an intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network 199 may be in any form and may include, for example, any of the following: point-to-point, bus, star, ring, mesh, or tree. The network 199 may have any network topology known to one of ordinary skill in the art and capable of supporting the modes of operation described herein. The network 199 may use various techniques and layers or protocol stacks, including, for example,the Ethernet protocol, the Internet Protocol (TCP / IP) suite of protocols, the ATM (Asynchronous Transfer Mode) technology, the SONET (Synchronous Optical Networking) protocol, or the SDH (Synchronous Digital Hierarchy) protocol. The TCP / IP Internet Protocol suite can include the application layer, transport layer, Internet layer (including, for example, IPv4 and IPv4), or link layer. Network 199 can be some type of broadcast network, a telecommunications network, a data communications network, or a computer network.
[0020] In one embodiment, the computer system 101, the spatial structure capture camera 151, the object recognition detection device 152, and / or the robot 161 may be capable of communicating with each other via a direct connection rather than via a network connection. For example, in such an embodiment, the computer system 101 may be configured to receive information from the spatial structure capture camera 151 and / or the object recognition detection device 152 via a dedicated wired communication interface, such as an RS-232 interface, a Universal Serial Bus (USB) interface, and / or via a local computer bus, such as a Peripheral Component Interconnect (PCI) bus.
[0021] In one embodiment, the spatial structure information generated by the spatial structure capture camera 151 may refer to any type of information that describes how a structure is laid out or otherwise arranged in space, such as a three-dimensional (3D) space. In particular, the spatial structure information may describe a 3D layout of the structure or a 3D pose or arrangement of the structure in a 3D space. For example, the structure may belong to a container in an environment or field of view of the spatial structure capture camera 151 or to an object arranged within the container. In some cases, the spatial structure information may indicate how the structure is oriented in a 3D space.In some cases, the spatial structure information may include depth information indicating one or more corresponding depth values for one or more locations on the structure, or more specifically, on one or more surfaces of the structure. The depth value for a particular location may be relative to the spatial structure detection camera 151 or relative to another reference frame (e.g., a ceiling or a wall of the warehouse or sales floor). In some cases, the depth values may be measured along an axis orthogonal to an imaginary plane on which the spatial structure detection camera 151 is located. For example, if the spatial structure detection camera 151 has an image sensor, the imaginary plane may be an image plane defined by the image sensor. In one embodiment, the spatial structure information may be used to determine a contour, or more generally, a boundary of a structure. The contour may, for example,that of a container, a portion of the container, or an object within the container. For example, the spatial structure information can be used to detect one or more locations where there is a strong discontinuity in depth values, where such locations may indicate a boundary (e.g., edge) of a structure. In some cases, the boundary of the structure can be used to determine its shape or size.
[0022] In some cases, the spatial structure information may comprise or form a depth map. The depth map may be a bitmap having a plurality of pixels representing or otherwise corresponding to various locations in the camera field of view, such as locations on one or more structures in the camera field of view. In such cases, some or all of the pixels may each have a corresponding depth value indicating a depth of a corresponding location represented by or otherwise corresponding to that pixel. In some cases, the depth map may comprise 2D image information describing a 2D appearance of the one or more structures in the camera field of view. The depth map may, for example, comprise a 2D image.In such an example, each of the pixels of the depth map may further include a color intensity value or grayscale intensity value indicating an amount of visible light reflected from or otherwise corresponding to the location represented by the pixel.
[0023] In one embodiment, the spatial structure information may be or include a point cloud. The point cloud may identify a plurality of locations that describe one or more structures, such as a structure of a container and / or a structure of an object within the container. In some cases, the plurality of points may be corresponding locations on one or more surfaces of the one or more structures. In some cases, the point cloud may include a plurality of coordinates (e.g., 3D coordinates) that identify or otherwise describe the plurality of points. For example, the point cloud may include a series of Cartesian or polar coordinates (or other data values) that specify corresponding locations or other features of the one or more structures. The corresponding coordinates may be relative to a frame of reference (e.g.,Coordinate system) of the spatial structure capture camera 151 or with respect to another frame of reference. In some cases, the corresponding coordinates are discrete and spaced apart from each other, but can be understood as representative of one or more contiguous areas of the one or more structures. In one embodiment, the point cloud can be generated from a depth map or other information (e.g., by the computer system 101).
[0024] In some embodiments, the spatial structure information may further be stored in any suitable format, such as polygon or triangle mesh models, non-uniform rational basis spline models, CAD models, parameterization of primitives (e.g., a rectangle may be defined according to a center and extensions in the x, y, and z directions, a cylinder may be defined by a center, a height, a top and bottom radius, etc.), etc.
[0025] As stated above, the spatial structure information is captured or otherwise generated via the spatial structure capture camera 151. In one embodiment, the spatial structure capture camera 151 may be or include a 3D camera or any other 3D image capture device. The 3D camera may be a depth-sensing camera, such as a time-of-flight (TOF) camera or a structured-light camera, or any other type of 3D camera. In some cases, the 3D camera may include an image sensor, such as a charge-coupled device (CCD) sensor and / or a complementary metal-oxide-semiconductor (CMOS) sensor.In one embodiment, the spatial structure detection camera 151 may include a laser, a LIDAR device, an infrared device, a light / dark sensor, a motion sensor, a microwave detector, an ultrasonic detector, a RADAR detector, or any other device configured to detect spatial structure information.
[0026] As stated above, the object identifier detection device 152 may be configured to detect an object identifier and generate detected object identifier information, such as detected barcode information describing a barcode. The detected object identifier information may describe, for example, a location of the object identifier (e.g., a barcode location), information encoded into the object identifier, or some other object identifier information. When the object identifier detection device 152 is a barcode detection device, in some cases, the barcode detection device may include a laser or a photodiode.configured to emit light or another signal toward a region of the barcode, such as a region occupied by a dark stripe or a dark square of the barcode, and may include a sensor configured to measure an amount of light or other signal reflected from the region. In some cases, the object identifier detection device 152 may be configured as shown in FIG. Fig. 1E may include a 2D camera 153. The 2D camera 153 may, for example, include a grayscale camera or a color camera. The 2D camera 153 may be configured to capture or otherwise generate 2D image information describing or otherwise representing a visual appearance of an environment within a field of view of the 2D camera 153, including the appearance of a barcode or other object identifier (if present) on an object in the field of view. Such a 2D camera 153 may, for example, also include an image sensor, such as a charge-coupled device (CCD) sensor and / or a complementary metal-oxide-semiconductor (CMOS) sensor. In some cases, the 2D image information may include multiple pixels forming a 2D image. Each pixel of the 2D image information may, for example, represent an intensity or other property of light reflected from a location corresponding to the pixel.In some cases, the 2D camera 153 may include processing circuitry configured to detect the barcode or other object identifier within the 2D image and generate object identification information based on the object identifier. In some cases, if the spatial structure information includes a depth map that includes 2D image information, the 2D image information may be generated by the 2D camera 153.
[0027] In one embodiment, the spatial structure detection camera 151 and the object identification detection device 152 may be integrated into a single device. For example, they may be enclosed in a single housing and have a fixed relative location and orientation. In some cases, they may share a single communications interface and / or a single power supply. In one embodiment, the spatial structure detection camera 151 and the object identification detection device 152 may be two separate devices attached to the robot 161 or otherwise secured, such as to a robot arm of the robot 161, as discussed in more detail below.
[0028] As indicated above, the spatial structure information and / or the acquired object identification information may be processed by the computer system 101. The computer system 101, in one embodiment, may include or be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop computer, etc.), a smartphone, a tablet computing device, and / or another computing system. In one embodiment, some or all of the functionality of the computer system 101 may be executed as part of a cloud computing platform. The computer system 101 may be a single computing device (e.g., a desktop computer) or may include multiple computing devices.
[0029] Fig. Figure 2A provides a block diagram illustrating one embodiment of computer system 101. Computer system 101 includes at least one processing circuit 110 and a non-transitory computer-readable medium (or media) 120. In one embodiment, processing circuit 110 includes one or more processors, one or more processing cores, a programmable logic controller ("PLC"), an application-specific integrated circuit ("ASIC"), a programmable gate array ("PGA"), a field-programmable gate array ("FPGA"), any combination thereof, or any other processing circuitry.In one embodiment, the non-transitory computer-readable medium 120 may be a storage device, such as an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, e.g., a computer diskette, a hard disk, a solid-state drive (SSD), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, any combination thereof, or any other storage device. In some cases, the non-transitory computer-readable medium may include multiple storage devices.In certain cases, the non-transitory computer-readable medium 120 is configured to store spatial structure information generated by the spatial structure detection camera 151 and / or detected object identification information generated by the object identification detection device 152. In certain cases, the non-transitory computer-readable medium 120 further stores computer-readable program instructions that, when executed by the processing circuit 110, cause the processing circuit 110 to perform one or more methodologies described herein, such as those described in . Fig. 4 described operation.
[0030] Fig. 2B shows a computer system 101A, which is an embodiment of computer system 101 and includes a communication interface 130. The communication interface 130 may be configured to receive, for example, spatial structure information generated by the spatial structure detection camera 151 and / or captured object identification information (e.g., captured barcode information) generated by the object identification detection device 152, such as via the storage device 198 and / or the network 199 of Fig. 1D or via a more direct connection from the spatial structure detection camera 151 or from the object recognition detection device 152. In one embodiment, the communication interface 130 may be configured to communicate with the robot 161 of Fig. 1B. If the computer system 101 is not part of a robot control system, the communication interface 130 of the computer system 101 may be configured to communicate with the robot control system. The communication interface 130 may, for example, include communication circuitry configured to perform communication via a wired or wireless protocol. The communication circuitry may, for example, include an RS-232 port controller, a USB controller, an Ethernet controller, a Bluetooth® controller, a PCI bus controller, any other communication circuitry, or a combination thereof.
[0031] In one embodiment, processing circuitry 110 may be programmed by one or more computer-readable program instructions stored on non-transitory computer-readable medium 120. Fig. For example, Figure 2C illustrates a computer system 101B, which is an embodiment of computer system 101 / 101A, in which processing circuitry 110 is programmed by one or more modules including a container detection module, an object detection module 204, and / or a motion planning module 206, which are discussed in more detail below.
[0032] In one embodiment, the container detection module 202 may be configured to detect a container, such as a drawer, or more particularly, to determine information about how the drawer is arranged in 3D space, such as its orientation and / or depth. As discussed in more detail below, the container detection module 202 may be configured to make such a determination using at least the spatial structure information. In some cases, the container detection module 202 may determine how a particular portion of the drawer, such as its lower interior surface, is arranged in 3D space. In some implementations, the container detection module 202 may be configured to determine how the lower interior surface is arranged in 3D space based on how another portion of the drawer, such as its edge, is arranged in 3D space.
[0033] In one embodiment, the object detection module 204 may be configured to detect an object located within the container, such as a commodity located on the lower interior surface of the drawer, or more specifically, to determine how the object is arranged in 3D space, such as an orientation and / or depth of the object. In some cases, the object detection module 204 may make such a determination based on information generated by the container detection module 202 regarding how the drawer or other container is arranged in 3D space. In some cases, the object detection module 204 may be configured to use acquired object identifier information (e.g., acquired barcode information) to identify the object and / or a size of the object, as discussed in more detail below.
[0034] In one embodiment, the motion planning module 206 may be configured to determine a robot motion for interacting with a container, interacting with an object within the container, and / or moving the spatial structure detection camera 151 and / or the object identifier detection device 152. For example, the robot motion may be part of a robot operation to grasp or otherwise retrieve the object from the container and move the object to another location. The robot motion may be determined based on information generated, for example, by the object detection module 204 regarding how the object is arranged in 3D space and / or by the container detection module 202 regarding how the container is arranged in 3D space. It is understood that the functionality of the modules as discussed herein is representative and not limiting.
[0035] In various embodiments, the terms "computer-readable instructions" and "computer-readable program instructions" are used to describe software instructions or computer code configured to perform various tasks and operations. In various embodiments, the term "module" generally refers to a collection of software instructions or code configured to cause processing circuitry 110 to perform one or more functional tasks. The modules and computer-readable instructions may be described as performing various operations or tasks when processing circuitry or other hardware component executes the modules or computer-readable instructions. In some cases, the modules and computer-readable instructions may implement a method for performing container detection and scheduling a robot interaction based on the container detection.
[0036] The Fig. 3A to 3C illustrate an environment in which a method for container detection and / or robot interaction may take place. In particular, Fig. 3A illustrates a system 300 (which is an embodiment of the system 100 / 100A / 100B / 100C of Fig. 1A to 1E), which includes the computer system 101, a robot 361, and a spatial structure detection camera 351 (which may be an embodiment of the spatial structure detection camera 151). In one embodiment, the robot 361 may include a base 362 and a robot arm 363. The base 362 may be used to mount the robot 361, while the robot arm 363 may be used to interact with the environment of the robot 361. In one embodiment, the robot arm 363 may include multiple arm sections that are movable relative to one another. For example, Fig. 3A, the arm portions 363A, 363B, 363C, and 363D are rotatable and / or extendable relative to one another. For example, the robot arm 363 may include one or more motors or other actuators configured to rotate the arm portion 363A relative to the base 362, the arm portion 363B relative to the arm portion 363A, and the arm portion 363C relative to the arm portion 363B. In this example, the arm portions 363A, 363B, and 363C may be members of the robot arm 363, while the arm portion 363D may be an end effector, such as a robot hand. In some cases, the robot hand may include a gripper configured to grasp or otherwise pick up an object, allowing the robot arm 363 to interact with objects.
[0037] In the embodiment of Fig. 3A, the spatial structure detection camera 351 may be mounted or otherwise secured to the robot 361, or in particular to the robot arm 363 of the robot 361, at a location that is part of or close to the end effector 363D. The spatial structure detection camera 351 may be part of an imaging system. In some scenarios, the imaging system may further include an object identification detection device, which is described below with respect to Fig. 7A. In some cases, the object identification detection device may also be mounted on the robot arm 363, such as at a location on or near the end effector 363D.
[0038] As in Fig. 3A, the system 300 may further include one or more containers 384, such as containers 384A through 384L. In some scenarios, containers 384A through 384L may be located in a warehouse or in a retail space and may be used to hold items such as merchandise or other objects. In the example of Fig. 3A, the bins 384A through 384L may be housed in a cabinet 380, which may provide a housing 381 that arranges the bins 384A through 384L in a stack. In this example, the bins 384A through 384L may each be a drawer movable between a closed and an open position. Each of the bins 384A through 384L may be attached to the cabinet 380 via one or more fasteners. Fig. For example, Figure 3A illustrates a pair of rails 382A, 383A attached to an interior surface of the housing 381 that allow the container 382A to slide between an open and a closed position. In some situations, at least one of the containers 384A through 384L may contain one or more objects. For example, the container 384A may include the objects 371, 373, which may be goods in the warehouse or sales floor. As an example, the objects 371, 373 may each be a box that is or contains items, such as goods to be shipped or sold.
[0039] As stated above, the spatial structure detection camera 351 (and / or the object identification detection device) may be a handheld camera device attached to or otherwise disposed on the robot arm 363. This placement may allow flexibility regarding the location and / or orientation of the spatial structure detection camera 351 (and / or the object identification detection device). Instead of attaching the spatial structure detection camera 351 (and / or the object identification detection device) to a stationary attachment point, the Fig. 3A to 3C, in particular, illustrate an embodiment in which the robot arm 363 can move the spatial structure detection camera 351 (and / or the object identification detection device) to different locations and / or orientations. Such an embodiment allows, for example, the robot arm 363 to adjust the distance between an object being detected and the spatial structure detection camera 351 (and / or the object identification detection device) to adjust a focus and / or resolution level of the resulting spatial structure information (and / or the detected object identification information).
[0040] Fig. Figure 3B further illustrates a situation in which the containers 384A to 384L are Fig. 3A, such as the container 384A, may each be in a closed position. When the container 384A is in the closed position, its contents (e.g., the objects 371, 373) may be inaccessible to the robot 361 or, in particular, to the robot arm 363. For example, an interior surface (e.g., the lower interior surface) of the container 384A may be substantially unexposed to an environment outside the housing 381 of the cabinet 380 when the container 384A is in the closed position. Furthermore, the contents of the container 384A (e.g., the objects 371, 373) may be obscured. In particular, they may be obscured from a camera field of view 353 of the spatial structure detection camera 351 (and / or a field of view of the object identification detection device). In such an example, an external portion of one or more containers, such as a handle of container 384A, may be within camera field of view 353.As discussed in more detail below, in some embodiments, the robotic arm 363 may be configured to grasp the handle of, for example, the container 384A and pull it to move the container 384A (e.g., over the rails 382A, 383A of FIG. Fig. 3A) into an open position.
[0041] Fig. 3C illustrates the container 384A in an open position (also referred to as the open position). When the container 384A is in the open position, its contents are accessible to the robot 361, or more specifically, the robot arm 363. For example, the container 384A may be slid along the rails 382A, 383A to a location where part or all of a lower interior surface of the container 382A is exposed to an environment outside the housing 381 of the cabinet 380. In such a situation, at least part of the lower interior surface may be within a camera field of view 353 of the spatial structure detection camera 351 (and / or within a field of view of the object recognition detection device). Furthermore, the contents of the container 384A, such as the objects 371, 373 arranged on the lower inner surface of the container 384A, may also be within the camera field of view 353 when the container 384A is in the open position.As stated above, the spatial structure detection camera 351 may be configured to generate spatial structure information describing the container 384A and / or the objects 371, 373 contained therein.
[0042] The spatial structure information can be used to detect a pose of the objects 371, 373 and to facilitate the interaction between the robot arm 363 and the objects 371, 373, such as an interaction in which the end effector 363D of the robot arm 363 picks up the objects 371, 373 and moves them away from the container 384A.
[0043] Fig. 3D shows a view of the container 384A of the Fig. 3A to 3C. As in Fig. 3D, in one embodiment, the container 384A may include a surface 384A-1 (also referred to as a container surface 384A-1) on which one or more objects, such as objects 371, 373, are disposed within the container 384A. The container surface 384A-1 may, for example, be a lower interior surface of the container 384A.
[0044] In one embodiment, the container 384A may have a rim 384A-2 (also referred to as a container edge 384A-2) that is offset from the container surface 384A-1. The container edge 384A-2 may be formed by one or more side walls 384A-3, 384A-4, and 384A-5, each having a common height h (as shown in Fig. 3D) or may have different corresponding heights. In this example, the container edge 384A-2 may include a top surface of the one or more side walls 384A-3, 384A-4, and 384A-5. The container edge 384A-2 and the container surface 384A-1 may be separated by a distance equal to or based on the height h, which may be a known value that the computer system 101 is configured to receive or determine. In some cases, the computer system 101 of the Fig. 3A to 3C determine information describing the container surface 384A-1 and / or the container edge 384A-2 and use this information to determine additional information describing one or more objects (e.g., 371, 373) disposed within the container 384A, as discussed in more detail below. As in Fig. 3D, the container 384A may in some cases include a handle 384A-6. The computer system 101 may, in some embodiments, be configured to move the robot arm 363 (the Fig. 3A to 3C) to move the container 384A into an open position by pulling on the handle 384A-6 or by otherwise interacting therewith.
[0045] Fig. 4 shows a flowchart for a method 400 for facilitating robot interaction with an object contained in a container. The method 400 may include determining information describing how the object is arranged in space so that a robot can move appropriately to, for example, grasp the object. In one example, the information may describe a pose of the object (also referred to as an object pose), which may include at least one of an orientation or depth value for the object (e.g., relative to the spatial structure capture camera 351 of the Fig. 3A to 3C). As discussed in more detail below, the method 400 may determine how the object is arranged in space based on information describing how the container (e.g., 384A) is arranged in space. In one embodiment, the method 400 may be performed by the computer system 101 or, more specifically, by the processing circuitry 110. In some cases, the method 400 may be performed when the processing circuitry 110 executes instructions stored on the non-transitory computer-readable medium 120 of the Fig. 2A to 2C are stored.
[0046] In one embodiment, method 400 may begin in a scenario where a container (e.g., 384A) is in a closed position, and may include computer system 101 controlling a robot (e.g., 361) to move the container to an open position, as discussed in more detail below. Such movement may include, for example, sliding container 384A on rails 382A, 383A included in the Fig. 3A and Fig. 3C. As described with reference to Fig. 3A and Fig. 3C, in one embodiment, the rails 382A, 383A may be attached to an inner side surface of the housing 381 in which the container 384A is housed, and may allow the container 384A to slide into and out of the housing 381.
[0047] In one embodiment, the method 400 may begin in a scenario where the container (e.g., 384A) is in an open position or was already in an open position, as described in the Fig. 3C and Fig. 5A. Similar to the example in Fig. 3C, the container 384A contains Fig. 5A shows objects 371 and 373 disposed within container 384A, and more particularly, disposed on a container surface 384A-1 of container 384A. Since container 384A is in the open position, objects 371, 373 within container 384A may be within camera field of view 353 of spatial structure detection camera 351.
[0048] In one embodiment, the method 400 may include a step 402 in which the computer system 101 receives a spatial structure detection camera (e.g., 351 of Fig. 5A). The spatial structure information may include depth information for an environment in the camera field of view 353. In particular, the spatial structure information may describe how various structures are spatially arranged in the camera field of view 353 (i.e., how they are arranged in space). The various structures may, for example, include the container 384A and the objects 371, 373 arranged within the container 384A.
[0049] In one embodiment, the spatial structure information can be used to detect the inclination of one or more structures, or in particular, a degree of inclination and / or an orientation of inclination. In particular, the container 384A can have one or more connecting elements, such as the rails 382A, 383A, which attach the container 384A to a housing 381. As the container 384A is moved to the open position, it can be tilted downward relative to the one or more members and relative to the housing 381. As the container 384A is moved along the rails 382A, 383A in Fig. 5A slides from a closed position to an open position, the weight of the container 384A may cause it to tilt downward relative to the rails 382A, 383A. An example of tilting is shown in Fig. 5B. In particular, Fig. 5B represents an axis 582 which allows alignment of the rails 383A, 383A of Fig. 5A and, in particular, runs parallel to the rails 382A, 383A. The figure further illustrates another axis 582P that runs perpendicular to the axis 582. In some cases, the axis 582P may be parallel to a vertical wall of a housing 381 for the cabinet 380. Fig. 5B further illustrates an axis 584 and an axis 584P, both of which may represent an orientation of the container 384A. In particular, the axis 584 may be parallel to the container 384A or, more specifically, parallel to the container surface 384A-1. The axis 584P may be a vertical axis for the container surface 384A-1 and may be perpendicular to the axis 584. When the container 384A is in the closed position, the axis 584 connected to the container 384 may be parallel to the axis 582 connected to the rails 382A, 383A. Further, the axis 584P may be parallel to the axis 582P. As discussed above, when the container 384A slides from the closed to the open position, the container 384A may tilt downward, causing the axis 584 connected to the container 384A to deviate from the axis 582 and the vertical axis 584P to deviate from the axis 582P, as shown in Fig. 5B. In other words, axis 584 may become oblique to axis 582, and axis 584P may become oblique to axis 582P.
[0050] In one embodiment, tilting the container 384A may cause the container 384A and any object (e.g., object 371 or 373) within the container 384A to be adjusted in depth and / or orientation relative to, for example, the robot arm 363 and / or the spatial structure capture camera 351 of the Fig. 3A to 3C. For example, if the container 384 is not tilted downward in the open position, the objects 371, 373 may have a first depth value and a first orientation relative to the spatial structure detection camera 351 and / or the robot arm 363. Tilting the container 384A may cause the objects 371, 373 to have a second depth value and a second orientation relative to the spatial structure detection camera 351 and / or the robot arm 363.In some cases, the second depth value may be only a few millimeters greater than the first depth value, and the second orientation may differ from the first orientation by only a few degrees or a fraction of a degree, but such a difference may be sufficient to affect the ability of the robot arm 363 to grasp or otherwise interact with the objects 371, 373 in a precise manner, particularly if the computer system 101 assumes that the objects 371, 373 are arranged in space according to the first depth value or the first orientation. Furthermore, the computer system 101 may need to determine with millimeter-level accuracy or better how the objects 371, 373 are arranged in space to ensure correct interaction between the robot 361 and the . Fig. 3A to 3C and objects 371, 373.
[0051] In some cases, it may be difficult to predict an amount or effect of tilt with millimeter-level accuracy because there may be multiple degrees of freedom for the container (e.g., 384A) as it moves from the closed position to the open position. Therefore, one aspect of the present application relates to the use of spatial structure information, such as that received in step 402 of method 400, to determine how an object (e.g., 371 / 373) is arranged in space to support an ability to control the robot arm 363 to correctly interact with the object (e.g., 371 / 373).
[0052] As indicated above, the spatial structure information of step 402 may include depth information for the environment in the camera field of view (e.g., 353). The depth information may include one or more depth values, each of which may indicate a depth of a particular location in the camera field of view 353 relative to the spatial structure capture camera (e.g., 351) or relative to another reference point or frame of reference. In some cases, the location associated with the depth value may be a location on a surface of a structure in the camera field of view 353. Fig. For example, Figure 5C illustrates depth information that represents the depth values of the ObjektA , Ort1 ; d Rand , Ort1 ; d Fläche, Ort1 ; and d Boden, Ort1 In this example, the depth value d ObjektA, Ort1indicate a depth value for a location on the object 371 relative to the spatial structure detection camera 351 or, in particular, relative to an image plane 354 formed by an image sensor or another sensor of the spatial structure detection camera 351. In particular, the depth value d ObjektA, Ort1 indicate a distance between the location on the object 371 (e.g., on a surface of the object 371) and the image plane 354. This distance may be measured along an axis perpendicular to the image plane 354. In one embodiment, the depth information may include one or more corresponding depth values for one or more parts of the container, such as the container surface 384A-1 and the container edge 384A-2. The depth value d Fläche, Ort1 For example, may specify a depth value for the container surface 384A-1 or, in particular, for a location on the container surface 384A-1. The depth value d Rand, Ort1may indicate a depth value for the container edge 384A-2 relative to the image plane 354 or, in particular, for a location on the container edge 384A-2. In addition, the depth value d Boden, Ort1 specify a depth value for a floor or other surface on which the housing 381 of the cabinet 380 is arranged, or in particular for a location on the floor.
[0053] Fig. 6A illustrates a representation of the spatial structure information obtained in step 402. In this example, the spatial structure information may include or identify multiple depth values for multiple locations in a camera field of view (e.g., 353). In particular, the figure illustrates various sets 610-660 of locations (also referred to as points) for which the spatial structure information identifies corresponding depth values. The set 610 of locations (denoted as striped hexagons) may correspond to a floor or other surface on which the housing 381 of Fig. 5C. The set 610 may, for example, include the position 6101 corresponding to the depth value d Boden, Ort1 from Fig. 5C. The set 620 of locations (denoted as white circles) may be associated with the surface 384A-1 of the container 384A (e.g., lower inner surface). For example, the set 620 may include the location 6201 corresponding to the depth value d Fläche, Ort1from Fig. 5C. The set 630 of locations (denoted as dark circles) may belong to the container edge 384A-2. For example, the set 630 may include the location 6301 corresponding to the depth value d Rand, Ort1 from Fig. 5C. The set 640 of locations (marked as dark ellipses) may correspond to the handle 384A-6 of the container 384A. Furthermore, the set 650 of locations (marked as shaded rectangles) may correspond to the object 371 of Fig. 5C. For example, the set 650 may include the location 6501 corresponding to the depth value d ObjektA, Ort1 from Fig. 5C. In addition, the set 660 of locations (marked as white rectangles) can be assigned to the object 373 of Fig. 5C belong.
[0054] In one embodiment, the spatial structure information may include a depth map and / or a point cloud. The point cloud may, for example, include corresponding coordinates of locations on one or more structures in the camera field of view (e.g., 353) of the spatial structure capture camera (e.g., 351). The point cloud may, for example, include 3D coordinates, such as [xyz] coordinates in a frame of reference (e.g., coordinate system) of the spatial structure capture camera or another frame of reference. In such an example, a coordinate for a location may indicate a depth value for that location. The depth value for the location may, for example, be equal to or based on a z-component of the coordinate.
[0055] In one embodiment, the spatial structure information may be influenced by or include measurement errors or other errors. For example, location 6501 may have a depth value equal to d ObjektA, Ort1is, but the point cloud or other spatial structure information may indicate that location 6501 at object 371 has an [xyz] coordinate, where z = d ObjektA, Ort1 + ε ObjektA, Ort1 is, where ε ObjektA, Ort1 refers to an error associated with location 6501. In this situation, the spatial structure information may incorrectly indicate that location 6501 has the depth value d ObjektA, Ort1 + ε ObjektA, Ort1The error may be due, for example, to image noise or another error source. The error may be based on a variety of factors. In some cases, the object 371 or another structure may have a shape that interferes with the operation of the spatial structure detection camera (e.g., 351). In some cases, the object 371 may be made of a material (e.g., transparent or translucent material) that interferes with the operation of the spatial structure detection camera. In some cases, light or other signals from another object 373 (from Fig. 5C) or an interior surface of the container 384A, and such a signal reflected from the other object 373 may act as image noise that interferes with an ability of the spatial structure detection camera (e.g., 351) to accurately measure a depth value for the object 371.
[0056] Fig. Figure 6B provides examples of locations (represented by shaded triangles) that correspond to a portion of the spatial structure information that is substantially affected by noise or other error sources. In the example of Fig. 6B, locations 6202 through 6205 of the set 620 of locations (corresponding to the container surface 384A-1) may be substantially affected by noise, and the spatial structure information corresponding to these locations may include a significant amount of error. Furthermore, locations 6501 through 6503 of the set 650 of locations (corresponding to the object 371) and locations 6601 through 6603 of the set 660 of locations (corresponding to the object 373) may be substantially affected by noise, and the spatial structure information corresponding to these locations may also include a significant amount of error.In such an example, determining how the object 371 or the object 373 are arranged in space by directly using the spatial structure information for the set 650 of locations or the set 660 of locations may result in an inaccurate or otherwise unreliable result, as a significant percentage of these locations may be influenced by noise or other sources of error. Therefore, one aspect of the present disclosure relates to determining how the container (e.g., 384A) is arranged in space and determining how an object (e.g., 371 / 373) within the container (e.g., 384A) is arranged in space based on how the container is arranged in space.
[0057] With further reference to Fig. 4, in one embodiment, the method 400 may include a step 404 in which the computer system 101 determines a container pose based on the spatial structure information. In some implementations, step 404 may be performed by the container detection module 202 of Fig. 2C. In one embodiment, the container pose may refer to a pose of a container, such as container 384A, and may be used to describe at least one of an orientation of the container (e.g., 384A) or a depth value for at least a portion of the container (e.g., 384A). In some cases, the portion of the container (e.g., 384A) may refer to a component of the container, such as the container edge (e.g., 384A-2) or the container surface (e.g., 384A-1). In some cases, the portion of the container (e.g., 384A) may refer to a region on the container or, more generally, to a location (e.g., 6201 or 6301 of Fig. 6A) on a container surface, such as the container surface (e.g., 384A-1) on which the container contents are deposited, or a surface of the container rim (e.g., 384A-2).
[0058] In some cases, the container pose may describe a position or arrangement of the container (e.g., 384A) or, more generally, how the container (e.g., 384A) is arranged in a 3D space. For example, the container pose may describe an orientation of the container (e.g., 384A), which may describe an amount (if any) by which the container (e.g., 384) or a portion thereof tilts downward. As indicated above, the container pose may describe a depth value, which may, for example, indicate how far the container (e.g., 384A) or a portion thereof is from the spatial structure capture camera (e.g., 351 of the Fig. 3A to 3C) or from a robot arm (e.g. 363) or another part of a robot (e.g. 361).
[0059] In some cases, the container pose may describe both an orientation of the container and a depth value for a location (e.g., 6201 or 6301) on the container (e.g., 384A). For example, the depth value may be equal to or indicate a component of a 3D coordinate for that location. For example, the 3D coordinate may be an [xyz] coordinate that includes a 2D component or a 2D coordinate (e.g., an [xy] coordinate) and a depth component (e.g., a z-component or a z-coordinate). The z-component or z-coordinate may be equal to or based on the depth value for that location relative to the spatial structure capture camera (e.g., 351) or another frame of reference. In such an example, the container pose may describe both an orientation of the container and a 3D coordinate for a location on the container.
[0060] In one embodiment, the container pose determined in step 404 may be a container surface pose, which may be a pose of a container surface (e.g., 384A-1). The container surface may be, for example, a bottom interior surface or another surface on which an object or other contents of the container (e.g., 384A) are disposed within the container. The container surface pose may, for example, describe at least one of: an orientation of the container surface (e.g., 384A-1) or a depth value for at least one location (e.g., 6201) of or at the container surface (e.g., 384A-1).
[0061] In one embodiment, determining the container surface pose may include directly using a portion of the spatial structure information corresponding to locations on the container surface (e.g., 384A-1). For example, in one embodiment, computer system 101 may determine the container surface pose directly based on spatial structure information corresponding to the set 620 of locations of Fig. 6B, which correspond to locations 6201 to 620 n The corresponding spatial structure information can, for example, include corresponding depth values for the locations 6201 to 620 n In some cases, the computer system 101 may be configured to include locations 6201 to 620 nas belonging to the container surface (e.g., 384A-1) or, more generally, as belonging to a common layer, to distinguish these locations from the locations representing other layers in a camera field of view (e.g., 353). For example, computer system 101 may be configured to identify locations 6201 to 620 n as locations with corresponding depth values that are substantially continuous and do not exhibit strong discontinuities between them. In some cases, in this embodiment, the computer system 101 may determine the container surface pose by determining a plane that best passes through all or some of the locations 6201 to 620 nfits. The computer system 101 may determine an orientation for the container surface (e.g., 384A-1) as equal to or based on a property of the plane, such as its slope or normal vector. In some cases, the computer system 101 may estimate or otherwise determine a depth value for a location on the container surface (e.g., 384A-1) directly based on the spatial structure information. For example, if the spatial structure information provides a 3D coordinate for that location, such as an [xyz] coordinate, the depth value may be equal to or based on a z-component of the 3D coordinate. In some cases, the computer system 101 may use the plane to estimate a depth value for a location on the container surface (e.g., 384A-1), since that location may fall on the plane or be substantially close to the plane. For example, if the computer system 101an [xy] component) for a location on the container surface (e.g., 384A-1), it may be configured to determine a 3D coordinate corresponding to the plane that also has the 2D component. In such an example, the 3D coordinate on the plane may specify or approximate the location on the container surface (e.g., 384A-1). Therefore, a z component of the 3D coordinate on the plane may correspond to or approximate a depth value for the location on the container surface (e.g., 384A-1).
[0062] In one embodiment, determining the container face pose may include indirectly using spatial structure information corresponding to another part of the container (e.g., 384A), such as spatial structure information corresponding to the container edge (e.g., 384A-2). In particular, an embodiment of step 404 may include determining a container edge pose and determining the container face pose based on the container edge pose. The container edge pose may describe at least one of an orientation of the container edge (e.g., 384A-2) or a depth value for at least one location (e.g., 6301) on the container edge (e.g., 384A-2).
[0063] In some cases, determining the container surface pose based on the container edge pose may provide a determination that is more robust to noise or other sources of error. In particular, noise may affect not only locations (e.g., 6501 to 6503) on an object (e.g., 371) located in a container (e.g., 384), but also locations on the container surface (e.g., 384A-1) on which the object is located. Therefore, depth information or other spatial structure information corresponding to these locations may be unreliable. For example, Fig. 6C a scenario in which the locations 6202, 6203, 6204, 6205, 6206, 6207, 6208, 6209, 620 10 , ... 620 k on the container surface 384A-1 (which is a subset of all locations 6201 to 620 identified by the spatial structure information nmay be) be affected by image noise, which may introduce errors into a corresponding part of the spatial structure information for these locations and, in particular, errors into the depth information for these locations. While Fig. 6B also illustrates an example with noise (at locations 6202 to 6206), Fig. 6C shows an example of a much noisier environment at the container surface 384A-1. In the example of Fig. 6C, the noise-affected locations (6202 to 620 k ) a large percentage of all places (6201 to 620 n) for which spatial structure information is available. The noise may arise, for example, from the presence of signals reflected from the container surface (e.g., 384A-1) or from an object (e.g., 371 / 373) on the container surface (e.g., 384A-1), where the reflected signals may interfere with each other and interfere with the direct measurement of depth values for locations on the container surface (e.g., 384A-1).
[0064] In some cases, a crowded bin (i.e., a bin containing a number of objects that may obscure the bin surface) may also interfere with the direct measurement of depth values for locations on the bin surface (e.g., 384A-1). Fig. For example, Figure 6D illustrates a scenario in which many objects, such as objects 371 to 375, are arranged on the container surface 384A-1. The objects 371 to 375 may cover a large portion of the container surface 384A-1. In particular, Fig. 6E Spatial structure information for the Fig. 6D. As shown in Fig. 6E, objects 371 through 375 may cover or otherwise occupy regions 652 through 692 on container surface 384A-1. Although some portions of container surface 384A-1 are not covered by objects 371 through 375, these portions may still be affected by noise, limiting the ability to use spatial structure capture camera 371 to directly obtain accurate depth measurements for container surface 384A-1.
[0065] Therefore, one aspect of the present disclosure relates to the use of spatial structure information corresponding to another part of the container, such as the container edge (e.g., 384A-2), to indirectly determine information about the container surface (e.g., 384A-1). In one embodiment, the computer system 101 may determine a container edge pose as noted above and use the container edge pose to determine the container surface pose. The container edge (e.g., 384A-2) may, in some cases, be less affected by noise or measurement error sources. That is, these measurement error sources may affect the direct measurement of the container surface (e.g., 384A-1) or objects disposed on the container surface (e.g., 384A-1). However, the container rim (e.g., 384A-2) may extend from the container surface (e.g., 384A-1) around one or more side walls of the container (e.g., side walls 384A-3 to 384A-5 in Fig. 3D), which may have a height h that causes the container edge (e.g., 384A-2) to be completely above the objects (e.g., 371, 373). Therefore, the container edge (e.g., 384A-2) may be significantly less affected by measurement error sources, and the direct depth measurement of the container edge (e.g., 384A-2) may be considerably more accurate than the direct depth measurement of the container surface (e.g., 384A-1).
[0066] In one embodiment, the direct measurement of the container edge (e.g., 384A-2) may include spatial structure information corresponding to locations on the container edge (e.g., 384A-2), and the container edge pose may be determined based on this spatial structure information. As in Fig. 6C, the spatial structure information may, for example, include depth information containing corresponding depth values for the locations 6301, 6302, 6303, 6304, ... 630 non the container edge (e.g., 384A-2). In some cases, the computer system 101 may be configured to specify these locations 6301 to 630 n from locations that represent another component of the container (e.g. from 6201 to 620 n ) by using the places 6301 to 630 n are identified as not having a strong discontinuity in depth between each other and therefore belong to a common layer that is separate from other layers in the camera field of view (e.g., 353). In one embodiment, the computer system 101 may be configured to identify the locations 6301 to 6301 as belonging to the container edge (e.g., 384A-2) by determining an estimated region in which the container edge (e.g., 384A-2) should be located and searching for the locations (e.g., 6301 to 630 n) in the estimated region. For example, the computer system 101 may have access to defined or otherwise known information about a structure of the container (e.g., 384A) and a structure of a cabinet (e.g., 380) or enclosure (e.g., 381) in which the container (e.g., 384A) is located. The information may, for example, identify a size (e.g., dimensions), physical configuration, shape, and / or geometry of the container (e.g., 384A) or cabinet (e.g., 380). The computer system 101 may be configured to determine the estimated region based on this information. For example, the computer system may estimate that the container rim (e.g., 384A-2) should have a depth value of approximately 600 mm with an error margin of approximately 10 mm. The computer system 101 can then search for the container edge (e.g.384A-2) in an estimated region that occupies a space having a depth value in the range of 590 mm to 610 mm.
[0067] As indicated above, the container edge pose may indicate at least one of an orientation of the container edge (e.g., 384A-2) or a depth value for at least one location on the container edge (e.g., 482A-2). In one embodiment, computer system 101 may determine an orientation for the container edge pose by determining a difference (if any) between corresponding depth values for some or all locations (e.g., 6301 to 630 n ) at the container edge (e.g. 384A-2). If the corresponding depth values for locations 6301 to 630 nFor example, if the container edge 384A-2 has an orientation that is substantially flat relative to the spatial structure capture camera 351 or another reference frame, the computer system 101 may determine a slope representing this change. The orientation for the container edge pose may be equal to or based on the slope.
[0068] In one embodiment, the computer system 101 may determine the container edge pose by determining a plane that substantially passes through some or all of the locations (e.g., 6301 to 630 n ) at the container edge (e.g., 384A-2), for which spatial structure information is provided. For example, the computer system 101 may use a 3D coordinate (e.g., [x n y n z n ]) for each of the locations (e.g. 6301 to 630 n) or a subset of the locations, where the 3D coordinate contains a depth component derived from the spatial structure information (e.g. z n ). The computer system 101 can determine a plane that best fits the corresponding 3D coordinates. The plane can be represented, for example, by the equation a(x-x0) + b(y-y0) + c(z±z0) = 0, where [x0, y0, z0] can be a 3D coordinate for one of the locations (e.g., 6301) at the container edge (e.g., 384A-2), and [xyz] can be a 3D coordinate for some or all of the remaining locations (e.g., 6302 to 630 n) at the container edge (e.g., 384A-2). The computer system 101 may generate a set of simultaneous equations based on the above coordinates and solve them for the coefficients a, b, c that best satisfy the simultaneous equations. In such an example, the computer system 101 may determine an orientation for the container edge pose as equal to or based on a property of the plane, such as its slope or normal vector (e.g., a vector parallel to ). As stated above, the computer system 101 may determine the container surface pose based on the container edge pose. In some cases, such a determination may be based on a defined distance between the container edge (e.g., 384A-2) and the container surface (e.g., 384A-1). The defined distance may, for example, be a height h of one or more side walls (e.g., 384A-3 to 384A-3 of Fig. 3D) that form the container edge (e.g., 384A-2). In some cases, the defined distance may be a known value stored on a non-transitory computer-readable medium (e.g., 120) accessible by computer system 101.
[0069] As further explained above, the container face pose may describe at least one of an orientation of the container face (e.g., 384A-1) or a depth value for a location on the container face (e.g., 384A-1). In some cases, the computer system 101 may determine the orientation for the container face pose based on an orientation for the container edge pose. In particular, the computer system 101 may determine an orientation for the container face (e.g., 384A-1) to be equal to, or based on, an orientation for the container edge (e.g., 384A-2). Such a determination may be based on an assumption that the container edge (e.g., 384A-2) is parallel to the container face (e.g., 384A-1).As an example, the computer system 101 may determine a first plane defining an orientation for the container edge pose, as discussed above, and may use the first plane to determine a second plane defining an orientation for the container face pose. Fig. For example, Figure 6F illustrates a first level 684A-2 that is based on the spatial structure information for locations (e.g., 6301 to 630 n ) at the container edge 384A-2 can be determined and which can define an orientation for the container edge 384A-2. The computer system 101 can be configured to determine a second plane 684A-1 based on the first plane 684A-2, wherein the second plane 684A-1 can define an orientation for the container surface 384A-1. In some cases, the first plane 684A-2 and the second plane 684A-1 can be parallel to each other and offset by the defined distance h. For example, if the first level 684A-2 is defined by the equation a(x-x0) + b(y-y0) + c(z-z0) = 0, as discussed above, the computer system 101 may determine the second level 684A-1 defined by the equation a(x-x0) + b(y-y0) + c(z-z0-h) = 0.
[0070] In one embodiment, computer system 101 may use second plane 684A-1 to determine or represent corresponding depth values for locations on surface 384A-1 of the container, since locations on container surface 384A-1 may fall on or be substantially close to plane 684A-1. For example, when computer system 101 receives a 2D component or a 2D coordinate (e.g., [xy] coordinate) for a location on container surface 384A-1, it may be configured to determine a 3D coordinate on plane 684A-1 corresponding to that 2D component and determine the depth value for that location based on a depth component (e.g., z component) of the 3D coordinate. In particular, the computer system 101 may determine a 3D coordinate [xyz] that satisfies the equation a(x-x0) + b(y-y0) + c(z-z0-h) = 0, where x and y may belong to the received 2D component, and where z may be the depth component of the 3D coordinate.Therefore, the information determined from the container edge 384A-2 can be used to make a reliable determination regarding orientation and / or depth for the container surface 384A-1. Referring again to FIG. Fig. 4, in one embodiment, the method 400 may include a step 406 in which the computer system 101 determines an object pose based on the container pose. This step may be performed, for example, by the object identifier detection module 204 of Fig. 2C. In this embodiment, the object pose may be at least one of an orientation for an object (e.g., 371 / 373 of the Fig. 3A to 3C) located within the container (e.g., 384A), or a depth value for at least a portion of the object. In some cases, the object may be a target object for which robot interaction is desired. The portion of the object may, for example, refer to a location on a surface (e.g., the top surface) of the object and / or a physical feature of the object, such as a corner or edge of the top surface.
[0071] In one embodiment, the container pose used to determine the object pose may be a container face pose. For example, computer system 101 may determine an orientation for the object pose as equal to or based on an orientation for the container face pose. In particular, computer system 101 may determine that an orientation for the object (e.g., 371) is equal to an orientation for the container face (e.g., 384A-1).
[0072] In some cases, computer system 101 may determine the orientation for the object (e.g., 371 / 373) based on the orientation of the container surface (e.g., 384A-1). For example, computer system 101 may determine that the orientation for the object (e.g., 371 / 373) is equal to the orientation for the container surface (e.g., 384A-1) on which the item is disposed. Such a determination may be based on an assumption that the object, such as a box of merchandise, sits flush with the container surface (e.g., 384A-1).
[0073] In some cases, computer system 101 may determine a depth value for a location on the object (also referred to as an object location) based on a depth value for a corresponding location on the container surface (e.g., 384A-1). The corresponding location may be a location on the container surface (e.g., 384A-1) on which the object is resting, or more generally, one that has the same 2D component or 2D coordinate as the object location. In some cases, the determination may be based on a defined quantity for the object (e.g., 371), such as a defined height h Objekt , which may be stored in a non-transitory computer-readable medium (e.g., 120) accessible by the computer system 101. If the object location is located on an upper surface of the object (e.g., 371), the computer system 101 may determine the location of the object as being separated from the location on the container surface (e.g., 384A-1) by the defined height h Objekt separately. For example, if the location on the container surface (e.g. 384A-1) has the 3D coordinate [xyz Fläche ], the computer system 101 may determine that the object location has a 3D coordinate [xyz Fläche ± h Objekt ] In such an example, the computer system 101 may initially determine the 2D component for the object location and use the 2D component to determine the 3D coordinate [xyz Fläche ], which may be determined, for example, based on solving the equation for plane 684A-1 or more generally based on the container surface pose as discussed above. Computer system 101 may then determine a depth value for the object location as equal to or based on z Fläche ± h Objekt Such a technique provides a robust way to accurately determine an orientation and / or depth value for the object (e.g., 371), even in an environment with significant image noise. Such image noise may prevent the spatial structure detection camera (e.g., 351) from directly and accurately measuring the depth for the object. However, the computer system 101 may make an indirect measurement by using spatial structure information for the container to determine a container surface pose and a depth value for a corresponding location on the container surface. The depth value for the object may then be extrapolated based on the depth value for the corresponding location on the container surface.
[0074] In some cases, step 406 may be performed in an environment that includes an object identifier capture device, such as the barcode capture device 352 of Fig. 7A. The barcode capture device 352 (which may be an embodiment of the object identifier capture device 152) may be attached to a robotic arm (e.g., 363) or to a stationary attachment point. The barcode capture device 352 may have a field of view 355 (also referred to as a reader field of view) and may be configured to capture a barcode or other object identifier (if any) located on an object (e.g., 371 / 373) disposed in a container (e.g., 384A). Fig. For example, Figure 7B provides an example in which a barcode 711 is disposed on object 371 and a barcode 713 is disposed on object 373. As noted above, object identification information may be configured to generate captured object identification information. In the example of Fig. 7B, the barcode capture device 352 may be configured to generate captured barcode information that may describe a location of the barcode 711 / 713, information encoded into the barcode 711 / 713, or any other information about the barcode 711 / 713. The information encoded into the barcode 711 / 713 may describe an object 371 / 373 on which the barcode is located, such as an identity of the object 711 / 713 or a size of the object 711 / 713.
[0075] In some cases, computer system 101 may be configured to determine whether information encoded in a barcode (e.g., 711 / 713) or other object identifier matches the object identification information received by computer system 101. For example, computer system 101 may receive object identification information such as a stock number (SKU) and / or a universal product code (UPC) that identifies a particular item, such as a commodity, for retrieval by a robot (e.g., 361). In such a case, computer system 101 may be configured to determine whether any object in the container (e.g., 384A) has a barcode (e.g., 711 / 713) or other object identifier affixed thereto that matches the object identification information, or more specifically, whether the information encoded in the barcode (e.g., 711) or other object identifier matches the object identification information.If there is a barcode (e.g., 711) or other object identifier whose encoded information matches the object identifier information, the computer system 101 may use the barcode to determine, for example, a 2D component for one or more locations associated with the object (e.g., 371) where the barcode (e.g., 711) or other object identifier is affixed.
[0076] In one embodiment, computer system 101 may be configured to determine an object identifier location, such as a barcode location. The barcode location may describe a 2D location of a barcode (e.g., 711 / 713), such as a barcode that matches the object identifier information. In some cases, the 2D location of the object identifier may be represented by a 2D coordinate (also referred to as a 2D object identifier coordinate). If the object identifier is a barcode, the 2D object identifier coordinate may be a 2D barcode coordinate. For example, computer system 101 may be configured to determine a 2D barcode coordinate [x BarcodeA , y BarcodeA ] that determines a position of the barcode 711 in Fig. 7B, and a 2D barcode coordinate [x BarcodeB , y BarcodeB ] that represents a location for the barcode 713. In some cases, the 2D barcode coordinate may be determined by the barcode detection device 352 from Fig. 7A or by another object identifier detection device. As indicated above, the barcode location may be determined for a barcode that matches the object identifier information discussed above. For example, if computer system 101 determines that the information encoded in barcode 711 on object 371 matches a received SKU number, computer system 101 may determine a 2D barcode coordinate [x ObjektA,Bareode , y ObjektA,Barcode ] for a location of the barcode 711. In one embodiment, the computer system 101 may use information encoded in a barcode (e.g., 711) or other object identifier to determine a size or other information about an object (e.g., 371) on which the barcode is located. For example, the barcode 711 or other object identifier may determine the height of the object 371 discussed above. Objekt , a length or width of the object 371 or any other information regarding a size of the object 371 (also referred to as object size). The height h Objekt can be used to determine a depth value for a location on the object 371, as also discussed above.
[0077] In one embodiment, computer system 101 may be configured to use the acquired object identifier information associated with an object identifier (e.g., acquired barcode information associated with barcode 711) to determine one or more 2D locations for an object (e.g., 371) where the object identifier is located, or more specifically, to determine one or more 2D coordinates for one or more corresponding locations on the object (e.g., 371). The 2D object coordinates may be combined with the depth values discussed above to plan a robot interaction with the object (also referred to as 2D object coordinates). In some cases, the 2D object coordinates may approximate a contour of the object (e.g., 371), such as a 2D boundary of a top surface of the object.
[0078] In one embodiment, the 2D object coordinates for the object locations may be determined based on the spatial structure information. The spatial structure information may, for example, be a point cloud representing multiple locations on one or more surfaces captured by an environment in the camera field of view (e.g., 353). The locations represented by the point cloud may, for example, be the locations defined in the Fig. 6A to 6C and 7C, such as the various sets 610 to 660 of locations on the container surface (e.g., 384A-1), a surface of objects (e.g., 371, 373), and a surface of a container rim (e.g., 384A-2). In such an embodiment, the computer system 101 may be configured to use acquired object identification information, such as acquired barcode information, to search the point cloud and determine at least one set of one or more 2D object coordinates. The 2D object coordinates may, for example, be [xy] coordinates representing corresponding object locations. For example, the 2D object coordinates may be corresponding 2D coordinates representing some or all of the object locations 6501 to 6504 on object 371 in Fig. 7C or some or all of the object locations 6601 to 6605 on object 373. The 2D object coordinates can be combined with corresponding depth values or the orientation of an object to generate a move command to interact with the object (e.g., 371 / 373), as discussed in more detail below.
[0079] In one embodiment, the 2D object coordinates for the object locations can be determined based on the spatial structure information and based on the location of an object identifier or, in particular, its 2D object identifier coordinate. If the object identifier is a barcode, the 2D object coordinates can be determined based on the spatial structure information and the location of the barcode. In particular, the location of the barcode (e.g., the 2D barcode coordinate) can be used to narrow down which part of the spatial structure information to search for the 2D object coordinates. In particular, the search can be restricted to a part of the spatial structure information that corresponds to a region surrounding the location of the object identifier or, in particular, the location of the barcode. For example, to search for 2D locations on the object 371, the computer system 101 can define a region 721 of Fig. 7C, which provides a barcode location for the barcode 711 of Fig. 7B. This means that region 721 contains the 2D barcode coordinate [x ObjektA,Barcode , y ObjektA,Barcode ] In some cases, region 721 may be a 2D region or a 3D region.
[0080] In one embodiment, the computer system 101 may be configured to search a portion of the spatial structure information corresponding to the region 721 to identify object locations corresponding to the object 711, or more particularly, to search for locations that fall on the object 711. Instead of searching all locations identified by the Fig. 7C, the computer system 101 may also search a subset of the locations represented by the spatial structure information. In particular, the subset of locations may be those in region 721. The computer system 101 may be configured to search for locations on the object 371, for example, by identifying locations that have a sufficiently large depth difference from surrounding locations located on the surface 384A-1 of the container. In some cases, the computer system 101 may determine 2D object coordinates for locations on the object. For example, if the spatial structure information provides an [xyz] coordinate for location 6501 incident on the object 371, the computer system 101 may determine [xy] as the 2D object coordinate for that object location.Although the spatial structure information in this example may also provide a z-component for this location, as discussed above, the z-component may be unreliable due to noise. In particular, the z-component may have sufficient accuracy for computer system 101 to distinguish between locations on object 371 and locations on a surrounding container surface 384A-1, but may lack sufficient accuracy to plan a robot interaction with object 371. Therefore, as further discussed above, computer system 101 may use the container pose to determine a corresponding depth value for object location 6501, or more generally, for an object location having the 2D object coordinate or 2D component [xy]. In some cases, the 2D object coordinate may be for, e.g.,the object location 6501 and the corresponding depth value for that location are combined to form a more trustworthy 3D coordinate for that object location.
[0081] Fig. Figure 7C further illustrates a region 723 containing a 2D object identifier coordinate or, in particular, the 2D barcode coordinate [x ObjektB,Barcode , y ObjektB,Barcode ] for the barcode 713. The computer system 101 may be configured to search the region 723 to determine object locations for the object 373. In one embodiment, the region 721 / 723 may have a defined fixed size. In one embodiment, the region 721 / 723 may have a size based on an object size of the object 371 / 373.
[0082] In one embodiment, the object locations for an object (e.g., 371) may be determined based on its object size, which may, for example, be encoded in a barcode (e.g., 711) or other object identifier located on the object. For example, the object size may indicate the length and width of the object (e.g., 371). Computer system 101 may be configured to estimate 2D coordinates, representing, for example, an edge or other boundary of the object (e.g., 371), based on the object size. For example, based on the length or width of the object, computer system 101 may estimate that a particular edge of the object is a particular distance from the barcode location or other object identifier location. Computer system 101 may use this distance to determine a 2D coordinate indicating where that particular edge is located.
[0083] In one embodiment, a 2D object identifier coordinate, such as the 2D barcode coordinate for the barcode location, may be determined based on information received from the object identifier detection device (e.g., the barcode detection device 352 of Fig. 7A). The barcode capture device (e.g., 352) may, for example, generate an [xy] coordinate as a 2D barcode coordinate and communicate the [xy] coordinate to the computer system 101. The computer system 101 may, if necessary, be configured to convert the [xy] coordinate from the representation in one coordinate system of the object identifier capture device (e.g., the barcode capture device 352) to another coordinate system, such as that of the spatial structure capture camera (e.g., 351). As stated above, the object identifier capture device (e.g., the barcode capture device 352) may, in some cases, use a 2D camera (e.g., 153 of Fig. 1E). In such cases, the object identifier detection device (e.g., the barcode detection device 352) may be configured to capture a 2D image. For example, Fig. 7B illustrates a 2D image representing a field of view (e.g., 355) of the barcode capture device 352. The object identifier capture device (e.g., the barcode capture device 352) and / or the computer system 101 may be configured to detect a barcode (e.g., 711 / 713) or other object identifier from the 2D image and determine the 2D object identifier coordinate based on where the object identifier (e.g., barcode 711 / 713) appears in the 2D image.
[0084] In one embodiment, when a 2D image is generated, it may be used to determine 2D object coordinates. For example, when computer system 101 receives the 2D image, it may be configured to detect an edge or other boundary of an object (e.g., 371) appearing in the 2D image and determine a 2D object coordinate representing the object based on where the edge appears in the 2D image. In some cases, computer system 101 may be configured to limit its search for the edge or other boundary to a portion of the 2D image. In such cases, the portion of the 2D image in which the search is performed may be based on an object identifier location, such as a barcode location for a barcode (e.g., 711) located on the object, or where the barcode appears in the 2D image.
[0085] In one embodiment, computer system 101 may be configured to estimate a 2D position for an object based on object identifier locations, such as barcode locations of neighboring barcodes. The neighboring barcodes are not located on the object and may be located on neighboring objects. Fig. For example, Figure 8 illustrates a scenario in which an object 377 is disposed within the container 384A that does not have a barcode thereon. In this example, the computer system 101 may be configured to use 2D barcode locations of the barcodes 711, 713, 716 disposed on neighboring objects 371, 373, and 376, respectively, to triangulate or otherwise determine a 2D location for the object 377. For example, the computer system 101 may be configured to determine a region whose boundary is defined by 2D barcode coordinates [x ObjektA,Bareode , y ObjektA,Barcode ], [x ObjektB,Barcode , y ObjektB,Barcode ], [x ObjektC,Barcode , y ObjektC,Barcode ] defined accordingly for the barcodes 711, 713, and 716, and to search this region for the object 377. In particular, the computer system 101 can search a portion of the spatial structure information corresponding to this region for locations that fall on the object 377.
[0086] In one embodiment, when the barcode detection device (or other object identifier detection device) and / or the spatial structure detection camera are mounted on a robot arm (e.g., 353 of Fig. 3A), the computer system 101 may be configured to control the placement of the device / camera (e.g., 352 / 351) by moving the robot arm. For example, the computer system 101 may be configured to generate and issue a sensor movement command that causes the robot arm 363 to move the object identifier detection device (e.g., the barcode detection device 352) and / or the spatial structure detection camera (e.g., 351) to a desired location and / or orientation. The sensor movement command may, for example, cause the devices (e.g., 352 / 351) to be moved to a location that is within a defined degree of proximity. In some cases, the defined degree of proximity may be based on a focal length of the object identifier detection device. In particular, the sensor movement command may cause the object identification detection device to move sufficiently close to objects in the container (e.g.354A) is moved so that any barcode (e.g., 711) on the objects is within the focal length of the object identifier detection device. In one embodiment, the spatial structure information received in step 402 and the detected barcode information or other object identifier information may be generated after the devices (e.g., 352 / 351) have been moved as a result of the sensor move command. In one embodiment, the sensor move command may cause the spatial structure information and / or the barcode detection device (or any other object identifier detection device) to be moved within a proximity range such that the spatial structure information and / or detected barcode information represents or covers only a portion of the container surface (e.g., 384A-1). Fig. For example, Figure 9 illustrates a situation where only a portion of the container surface (e.g., 384A-1) or, more generally, one side of the container (e.g., 384A) is captured by the spatial structure capture camera 351 and / or the barcode capture device 352. That is, at such a degree of proximity, only a portion of the container surface (e.g., 384A-1) may be within the camera field of view (e.g., 353) or reader field of view (e.g., 355). It may not be necessary to capture information for an entire container surface (e.g., 384A-1) or an entire container (e.g., 384A). Rather, capturing only a portion of the container (e.g., 384A) may allow the computer system 101 to focus on a specific portion of the container (e.g., 384), such as its right half, and particularly on detecting objects in that portion. In some cases, the computer system 101 may limit how often the spatial structure detection camera (e.g.,351) and / or the object identifier capture device (e.g., the barcode capture device 352) is moved, or to how many locations the camera / device (e.g., 351 / 352) is moved to capture information about a particular container (e.g., 384A). For example, the camera / device (e.g., 351 / 352) may only be moved once to a single location to capture a snapshot of a particular container.
[0087] In one embodiment, computer system 101 may be configured to perform segmentation of a particular container (e.g., 384A) by associating different regions on the container surface (e.g., 384A-1) to different segments. Fig. For example, Figure 10 illustrates a situation where the bin surface 384A-1 may be virtually divided into segments 1001 through 1006. In this scenario, the computer system 101 may be configured to receive a bin segment identifier associated with an object. In one example, the computer system 101 may receive a bin segment identifier that identifies the segment 1006 or, more specifically, indicates that robot interaction is desired for an object (e.g., 371) located within segment 1006. In one embodiment, the computer system 101 may be configured to determine locations on the bin surface (e.g., 384A-1) associated with the bin segment identifier. In some cases, determining these locations may include determining their depth values, which may include using at least one of the bin edge pose or the bin face pose, as discussed above.
[0088] In one embodiment, method 400 may include a step 408 in which computer system 101 issues a motion command to cause a robot interaction with the object (e.g., 371 / 373), such as grasping or otherwise picking up the object by the robot arm (e.g., 363). Such a motion command may also be referred to as an object motion command. In some cases, step 408 may be performed by the motion planning module 206 of Fig. 2C, which may be configured to generate, for example, object movement commands, the sensor movement commands discussed above, and container movement commands (discussed below). The object movement command may be generated by the computer system 101, for example, based on the object pose determined in step 406, such as the orientation or depth value for the object (e.g., 371). The object movement command may be determined, for example, to move a robot hand or other end effector on the robot arm (e.g., 363) so that it is within reach for manipulating or otherwise interacting with the object (e.g., 371), and to an orientation that matches an orientation for the object (e.g., 371). In one embodiment, the movement command may, for example, cause a rotation or other actuation that places the end effector in such a location and / or orientation.In some cases, the movement command may be generated based on the 2D object coordinates discussed above and their corresponding depth values, which may be provided by the object pose. For example, the object movement command may be generated such that the end effector approaches the 2D object coordinates to a defined degree of proximity, allowing the end effector to manipulate or otherwise interact with the object (e.g., 371).
[0089] In one embodiment, the object movement command may be determined to avoid a collision event. The collision event may be a collision between an object being moved (e.g., 371) and, for example, a container sidewall (e.g., 384A-5 of Fig. 3D) or another container boundary forming the container edge (e.g., 384A-2). In some cases, computer system 101 may be configured to determine an object motion path that avoids such a collision event. The object motion path may be a motion path for the object (e.g., 371) being moved. The object motion command may be generated based on the object motion path. In some cases, accurately determining the object pose in step 406 may facilitate such collision avoidance.
[0090] As discussed above, in some embodiments, method 400 may begin in a scenario where the container (e.g., 384A) is already in an open position, such as that shown in Fig. 3C. In one embodiment, the method 400 may begin in a scenario where the container (e.g., 384A) is in a closed position, such as that illustrated in Fig. 3A and Fig. 11A. In such an embodiment, method 400 may include a step in which computer system 101 controls the robot arm (e.g., 363) to move the container (e.g., 384A) to an open position. Such a container opening step may occur before receiving spatial structure information in step 402.
[0091] For example, the Fig. 11A and Fig. 11B illustrates a situation in which the container 384A is in a closed position. In such a situation, the spatial structure detection camera 351 may be configured to generate spatial structure information describing an outer surface 384A-7 of the container 384A, or more specifically, describing locations on the outer surface 384A-7. The spatial structure information in this example may differ from the spatial structure information of step 402, which involves an open container situation. The computer system 101 may be configured to determine one or more locations representing the handle 384A-6 of the container 384A based on the spatial structure information describing the outer surface 384A-7.Further, computer system 101 may be configured to generate and issue a bin movement command to cause robot arm 363 to move bin 384A from the closed position to the open position. The bin movement command may be generated based on the one or more locations representing handle 384A-6 (also referred to as bin handle positions). In particular, as shown in FIGS. Fig. 11B and Fig. As illustrated in Figure 11C, the bin move command causes the robot hand 363D or another end effector of the robot arm 363 to pull on the handle 384A-6 to push the bin 384A into the open position. After the bin 384A is in the open position, in some scenarios, the spatial structure capture camera 351 may be moved to a different location (e.g., via a sensor move command) to capture spatial structure information of a bin surface and objects disposed thereon, after which an object pose may be determined and an object may be moved based on the object pose (e.g., via an object move command). Additional discussion of various embodiments:
[0092] Embodiment 1 relates to a computer system comprising a communication interface and at least one processing circuit. The communication interface is configured to communicate with a robot having a robot arm having a spatial structure detection camera arranged on the robot arm, wherein the spatial structure detection camera has a camera field of view. The at least one processing circuit is configured to perform the following method when an object is or has been within a container in the camera field of view while the container is in an open position: receiving spatial structure information comprising depth information for an environment in the camera field of view, wherein the spatial structure information is generated by the spatial structure detection camera;Determining a container pose based on the spatial structure information, wherein the container pose serves to describe at least one of an orientation for the container or a depth value for at least a portion of the container; Determining an object pose based on the container pose, wherein the object pose serves to describe at least one of an orientation for the object or a depth value for at least a portion of the object; Issuing a motion command to effect robot interaction with the object, wherein the motion command is generated based on the object pose.
[0093] Embodiment 2 includes the computer system of Embodiment 1. In this embodiment, the at least one processing circuit is configured to determine the container pose as a container surface pose to describe at least one of: an orientation for a container surface on which the object is disposed, or a depth value for at least one location on the container surface.
[0094] Embodiment 3 comprises the computer system of Embodiment 2, wherein the at least one processing circuit is configured, when the container is a drawer with a container edge offset from the container surface, to determine a container edge pose to describe at least one of: an orientation for the container edge or a depth value for at least one location on the container edge, wherein the container edge pose is determined based on the spatial structure information. Furthermore, the container surface pose is determined based on the container edge pose and based on a defined distance between the container edge and the container surface. Embodiment 4 comprises the computer system of Embodiment 2 or 3.In this embodiment, the at least one processing circuit is configured to: receive a bin segment identifier associated with the object, wherein the bin segment identifier is for identifying a segment of the bin surface, and determine locations associated with the bin segment identifier, wherein the determining is based on the bin surface pose.
[0095] Embodiment 5 includes the computer system of Embodiment 3 or 4. In this embodiment, the at least one processing circuit is configured to determine an object movement path that avoids a collision event, wherein the collision event represents a collision between the object and a container boundary forming the container edge, and wherein the movement command is generated based on the object movement path.
[0096] Embodiment 6 includes the computer system of any one of Embodiments 1 to 5. In this embodiment, when an object identifier is located on the object, the at least one processing circuit is configured to: determine an object identifier location for describing a 2D location of the object identifier; and determine a set of one or more object locations, which are one or more locations representing the object, based on the object identifier location and the spatial structure information, wherein the movement command is generated based on the set of one or more object locations.
[0097] Embodiment 7 includes the computer system of Embodiment 6. In this embodiment, the at least one processing circuit is configured to determine a region surrounding the object identifier location and to determine the set of one or more object locations by searching a portion of the spatial structure information corresponding to the determined region surrounding the object identifier location.
[0098] Embodiment 8 includes the computer system of Embodiment 7. In this embodiment, the spatial structure information includes a point cloud representing a plurality of locations on one or more surfaces captured by the environment in the camera field of view, wherein the portion of the spatial structure information from which the set of one or more object locations is searched includes a subset of the plurality of locations located within the particular region surrounding the object identification location.
[0099] Embodiment 9 includes the computer system of any one of Embodiments 6 to 8. In this embodiment, the at least one processing circuit is configured to: determine at least one 2D object identifier coordinate that is a 2D coordinate representing the location of the object identifier, and determine at least one set of one or more 2D object coordinates based on the 2D object identifier coordinate, wherein the one or more 2D object coordinates are one or more corresponding 2D coordinates representing the one or more object locations, wherein the move command is generated based on the set of one or more 2D object coordinates and based on an orientation and depth value for the object.
[0100] Embodiment 10 includes the computer system of any of Embodiments 6 to 9. In this embodiment, the at least one processing circuit is configured to determine an object size based on information encoded into the object identifier, wherein the one or more object locations represent a boundary for the object and are determined based on the object size.
[0101] Embodiment 11 includes the computer system of any of Embodiments 6 to 10. In this embodiment, the at least one processing circuit is configured to receive object identifier information associated with the object and determine whether the information encoded into the object identifier matches the object identifier information.
[0102] Embodiment 12 includes the computer system of any one of Embodiments 6 to 11. In this embodiment, the at least one processing circuit is configured, when an object identifier detection device is disposed on the robot arm, to determine the object identifier location based on the information detected by the object identifier detection device.
[0103] Embodiment 13 includes the computer system of Embodiment 12. In this embodiment, the move command is an object move command to cause the robot to move the object, wherein the at least one processing circuit is configured to issue a sensor move command to cause the robot arm to move the object identifier detection device to be within a defined degree of proximity to the container, and wherein the object identifier location is determined after the sensor move command is issued.
[0104] Embodiment 14 includes the computer system of Embodiment 13. In this embodiment, the sensor movement command also serves to cause the robot arm to move the spatial structure detection camera such that it is within the defined degree of proximity to the container, wherein the spatial structure information is generated when the spatial structure detection camera is within the defined degree of proximity to the container and represents a part of a container surface on which the object is arranged.
[0105] Embodiment 15 includes the computer system of Embodiment 13 or 14. In this embodiment, when the container is in a closed position and includes a handle, the at least one processing circuit is configured to: receive additional spatial structure information for describing locations on an exterior surface for the container, determine, based on the additional spatial structure information, one or more handle locations for representing the handle, and issue a container movement command to cause the robot arm to move the container from the closed position to the open position, wherein the container movement command is generated based on the one or more handle locations, and wherein the sensor movement command and the object movement command are issued after the container movement command.
Claims
[1] Computer system (101) comprising: a communication interface (130) configured to communicate with a robot (161) having a robot arm (363) having a spatial structure detection camera (151, 351) arranged on the robot arm (363), the spatial structure detection camera (151, 351) having a camera field of view (353); at least one processing circuit (110) configured to perform the following when an object (371, 373) is or has been within a container (384A) in the camera field of view (353) while the container (384A) is in an open position: Receiving spatial structure information comprising depth information for an environment in the camera field of view (353), wherein the spatial structure information is generated by the spatial structure detection camera (151, 351), Determining a container pose based on the spatial structure information, Determining an object pose based on the container pose, and issuing a movement command to effect a robot interaction with the object (371, 373), wherein the movement command is generated based on the object pose, where the container pose serves to describe an orientation for the container (384A) and the object pose serves to describe an orientation for the object (371, 373), and / or the container pose serves to describe a depth value for at least a part of the container (384A) and the object pose serves to describe a depth value for at least a part of the object (317, 373). [2] The computer system (101) of claim 1, wherein the at least one processing circuit (110) is configured to determine the container pose as a container surface pose to describe at least one of: an orientation for a container surface (384A-1) on which the object (371, 373) is disposed, or a depth value for at least one location on the container surface (384A-1). [3] The computer system (101) of claim 2, wherein the at least one processing circuit (110) is configured to, when the container (384A) is a drawer having a container edge (384A-2) offset from the container surface (384A-1), determine a container edge pose to describe at least one of: an orientation for the container edge (384A-2) or a depth value for at least one location on the container edge (384A-2), wherein the container edge pose is determined based on the spatial structure information, and wherein the container surface pose is determined based on the container edge pose and based on a defined distance between the container edge (384A-2) and the container surface (384A-1). [4] The computer system (101) of claim 3, wherein the at least one processing circuit (110) is configured to: Receiving a container segment identifier associated with the object (371, 373), the container segment identifier for identifying a segment of the container surface (384A-1), and Determining locations associated with the container segment identifier, wherein the determining is based on the container face pose. [5] The computer system (101) of claim 3, wherein the at least one processing circuit (110) is configured to determine an object movement path that avoids a collision event, wherein the collision event represents a collision between the object (371, 373) and a container boundary forming the container edge (384A-2), and wherein the movement command is generated based on the object movement path. [6] The computer system (101) of claim 1, wherein the at least one processing circuit (110), when an object identifier is located on the object (371, 373), is configured to: Determining an object identifier location for describing a 2D location of the object identifier; and Determining a set of one or more object locations that are one or more locations representing the object (371, 373) based on the object identification location and the spatial structure information, wherein the movement command is generated based on the set of one or more object locations. [7] The computer system (101) of claim 6, wherein the at least one processing circuit (110) is configured to determine a region surrounding the object identifier location and to determine the set of one or more object locations by searching a portion of the spatial structure information corresponding to the determined region surrounding the object identifier location. [8] The computer system (101) of claim 7, wherein the spatial structure information comprises a point cloud representing a plurality of locations on one or more surfaces captured by the environment in the camera field of view (353), wherein the portion of the spatial structure information from which the set of one or more object locations is searched comprises a subset of the plurality of locations located within the particular region surrounding the object identification location. [9] The computer system (101) of claim 6, wherein the at least one processing circuit (110) is configured to: Determining at least one 2D object identifier coordinate, which is a 2D coordinate for representing the location of the object identifier, and Determining at least one set of one or more 2D object coordinates based on the 2D object identification coordinate, wherein the one or more 2D object coordinates are one or more corresponding 2D coordinates for representing the one or more object locations, wherein the movement command is generated based on the set of one or more 2D object coordinates and based on an orientation and depth value for the object (371, 373). [10] The computer system (101) of claim 6, wherein the at least one processing circuit (110) is configured to determine an object size based on information encoded in the object identifier, wherein the one or more object locations represent a boundary for the object (371, 373) and are determined based on the object size. [11] The computer system (101) of claim 6, wherein the at least one processing circuit (110) is configured to receive object identifier information associated with the object (371, 373) and to determine whether the information encoded into the object identifier matches the object identifier information. [12] The computer system (101) of claim 6, wherein, when an object identifier detection device (152) is disposed on the robot arm (363), the at least one processing circuit (110) is configured to determine the object identifier location based on the information detected by the object identifier detection device (152). [13] The computer system (101) of claim 12, wherein the movement command is an object movement command to cause the robot (161) to move the object (371, 373), wherein the at least one processing circuit (110) is configured to output a sensor movement command to cause the robot arm (363) to move the object identification detection device (152) such that it is within a defined degree of proximity to the container (384A), where the object identification location is determined after issuing the sensor movement command. [14] The computer system (101) of claim 13, wherein the sensor movement command also serves to cause the robot arm (363) to move the spatial structure detection camera (151, 351) such that it is within the defined degree of proximity to the container (384A), wherein the spatial structure information is generated when the spatial structure detection camera (151, 351) is within the defined degree of approach of the container (384A) and represents a part of a container surface (384A-1) on which the object (371, 373) is arranged. [15] The computer system (101) of claim 13, wherein the at least one processing circuit (110), when the container (384A) is in a closed position and includes a handle, is configured to: Receiving additional spatial structure information for describing locations on an outer surface (384A-7) for the container (384A), Determining, based on the additional spatial structure information, one or more grip locations to represent the grip and Issuing a bin move command to cause the robot arm (363) to move the bin (384A) from the closed position to the open position, wherein the bin move command is generated based on the one or more grasp locations, and wherein the sensor move command and the object move command are issued after the bin move command. [16] A non-transitory computer-readable medium comprising instructions that, when executed by at least one processing circuit (110) of a computer system (101), cause the at least one processing circuit (110) to: Receiving spatial structure information, wherein the computer system (101) is configured to communicate with a robot (161) having a robot arm (363) having a spatial structure detection camera (151, 351) with a camera field of view (353) arranged on the robot arm, wherein the spatial structure information is generated by the spatial structure detection camera (151, 351), wherein the spatial structure information comprises depth information for an environment in the camera field of view (353) and is generated when an object (371, 373) is or was located within a container (384A) in the camera field of view (353) while the container (384A) is in an open position; Determining a container pose based on the spatial structure information; Determining an object pose based on the container pose, and issuing a movement command to effect a robot interaction with the object (371, 373), wherein the movement command is generated based on the object pose, where the container pose serves to describe an orientation for the container (384A) and the object pose serves to describe an orientation for the object (371, 373), and / or the container pose serves to describe a depth value for at least a part of the container (384A) and the object pose serves to describe a depth value for at least a part of the object (371, 373). [17] The non-transitory computer-readable medium of claim 16, wherein the instructions, when executed by the at least one processing circuit (110), cause the at least one processing circuit (110) to determine the container pose as a container surface pose to describe at least one of: an orientation for a container surface (384A-1) on which the object (371, 373) is disposed, or a depth value for at least one location on the container surface (384A-1). [18] The non-transitory computer-readable medium of claim 17, wherein the instructions, when executed by the at least one processing circuit (110), and when the container (384A) is a drawer having a container edge (384A-2) offset from the container surface (384A-1), cause the at least one processing circuit (110) to determine a container edge pose to describe at least one of: an orientation for the container edge or a depth value for at least one location on the container edge (384A-2), wherein the container edge pose is determined based on the spatial structure information, and wherein the container surface pose is determined based on the container edge pose and based on a defined distance between the container edge (384A-2) and the container surface (384A-1). [19] A method for object detection, comprising: Receiving, by a computer system (101), spatial structure information, wherein the computer system (101) is configured to communicate with a robot (161) having a robot arm (363) having a spatial structure detection camera (151, 351) with a camera field of view (353) arranged on the robot arm (363), wherein the spatial structure information is generated by the spatial structure detection camera (151, 351), wherein the spatial structure information comprises depth information for an environment in the camera field of view (353) and is generated when an object (371, 373) is or has been within a container (384A) in the camera field of view (353) while the container (384A) is in an open position; Determining a container pose based on the spatial structure information; Determining an object pose based on the container pose, and issuing a movement command to effect a robot interaction with the object (371, 373), wherein the movement command is generated based on the object pose, where the container pose serves to describe an orientation for the container (384A) and the object pose serves to describe an orientation for the object (371, 373), and / or the container pose serves to describe a depth value for at least a part of the container (384A) and the object pose serves to describe a depth value for at least a part of the object (371, 373). [20] The method of claim 19, wherein determining the container pose comprises determining a container surface pose for describing at least one of: an orientation for a container surface (384A-1) on which the object (371, 373) is arranged, or a depth value for at least one location on the container surface (384A-1).
Citation Information
Patent Citations
Image processing device with a function for automatically adjusting the search window
DE102013001603B4
Device for removing loosely stored items by a robot
DE102013012224B4
Tray delivery apparatus
JP1986206709A
System and method for overhead warehousing
US10150564B2
Method and system for performing automatic camera calibration for robot control
US10369698B1
