System and method for object detection based on image data

Abstract

To solve the problem in which systems in the prior art relying on a single imaging device may have an increased probability of failing to find or detect boundaries or edges around objects.SOLUTION: A method of detecting objects comprises: generating first edge information from first image data representing an object based on a first mode of image capture; generating second edge information from second image data representing the object based on a second mode of image capture different from the first mode of image capture; fusing the first edge information with the second edge information to generate fused edge information; generating an object detection hypothesis based on the fused edge information; and validating the object detection hypothesis based on the fused edge information, the first edge information, and / or the second edge information.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001]

[01] This disclosure relates to object detection, and more particularly to systems and methods for detecting edges of objects. [Background technology]

[0002]

[02] In some machine vision applications, detecting edges of palletized or containerized objects or boxes moving along a conveyor system may be important to detect gaps or determine the size or type of the object. However, prior art systems may have difficulty detecting contrast or boundaries between two objects or boxes, between an object and the surrounding environment, or between multiple objects mounted on a single pallet due to the color or optical resolution of the camera or other imaging device used. Prior art systems that rely on a single imaging device may have a high probability of failing to find or detect boundaries or edges around an object. Exemplary implementations of the present invention may address deficiencies in prior art systems. Summary of the Invention

[0003]

[03] Aspects of the invention may include a method for detecting an object. The method may include generating first edge information from first image data representing the object based on a first image capture mode, generating second edge information from second image data representing the object based on a second image capture mode different from the first image capture mode, fusing the first edge information and the second edge information to generate fused edge information, generating an object detection hypothesis based on the fused edge information, and verifying the object detection hypothesis based on the fused edge information, the first edge information, and / or the second edge information.

[0004]

[04] Additional aspects of the invention may include a non-transitory computer-readable medium having instructions embodied therein that cause a computing device to perform a method of detecting an object. The method may include generating first edge information from first image data representing the object based on a first image capture mode, generating second edge information from second image data representing the object based on a second image capture mode different from the first image capture mode, fusing the first edge information and the second edge information to generate fused edge information, generating an object detection hypothesis based on the fused edge information, and verifying the object detection hypothesis based on the fused edge information, the first edge information, and / or the second edge information.

[0005]

[05] A further aspect of the invention may include a computer system for detecting an object using machine vision. The computer system may include a system interface and a processor communicatively coupled to the system interface. The system interface may be configured to receive first image data representing an object based on a first image capture mode and to receive second image data representing the object based on a second image capture mode different from the first image capture mode. The processor may be configured to perform an object detection method. The method may include generating first edge information from the first image data representing the object based on the first image capture mode, generating second edge information from the second image data representing the object based on the second image capture mode different from the first image capture mode, fusing the first edge information and the second edge information to generate fused edge information, generating an object detection hypothesis based on the fused edge information, and validating the object detection hypothesis based on the fused edge information, the first edge information, and / or the second edge information.

[0006]

[06] An additional aspect of the invention may include another object detection system for detecting objects using machine vision. The system may include a device configured to capture images in a first image capture mode and capture images in a second image capture mode, a device configured to generate first object edge information from image data collected in the first image capture mode and generate second object edge information from image data collected in a second image capture mode different from the first image capture mode, and a processor. The processor may be configured to fuse the first object edge information and the second object edge information to generate fused edge information, generate an object detection hypothesis based on the fused edge information, and verify the object detection hypothesis based on the fused edge information, the first object edge information, and the second object edge information. [Brief description of the drawings]

[0007]

[07] Exemplary embodiments of the present invention will now be described in detail with reference to the following figures.

[0008] [Figure 1A]

[08] FIG. 1 illustrates an exemplary environment in which a robotic system incorporating an exemplary embodiment of the present invention may be used. [Figure 1B]

[09] FIG. 1 is a block diagram illustrating a robotic system incorporating an exemplary embodiment of the present invention. [Diagram 2]

[10] shows a device for detecting objects moving along a conveyor according to a first exemplary implementation of the present invention. [Diagram 3] 1 shows a device for detecting objects moving along a conveyor according to a second exemplary implementation of the present invention. [Figure 4]

[12] shows a device for detecting objects moving along a conveying apparatus according to a third exemplary implementation of the present invention. [Diagram 5]

[13] FIG. 1 shows a schematic diagram of image data that may represent edge detection results of a single image capture device or camera according to a comparative example. [Figure 6]

[14] FIG. 1 shows a schematic diagram of a fused image data representation that may represent a fused edge detection result using an implementation of the present invention as an example. [Figure 7A] FIG. 15 shows a schematic diagram of inaccurate detection hypotheses that can be avoided by utilizing fused detection edge data, according to an exemplary implementation of the present invention. [Figure 7B] FIG. 15 shows a schematic diagram of inaccurate detection hypotheses that can be avoided by utilizing fused detection edge data, according to an exemplary implementation of the present invention. [Figure 8]

[16] FIG. 1 shows a flowchart of a process for object detection and verification according to an exemplary implementation of the present invention. [Figure 9]

[17] An exemplary computing environment including an exemplary computing device suitable for use with some exemplary implementations of the present invention is illustrated. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009]

[18] The following detailed description reveals further details of the drawings and exemplary implementations of the present invention. Reference numbers and descriptions of elements that overlap between the drawings are omitted for clarity. Terms used throughout the description are provided by way of example and are not intended to be limiting. For example, the use of the term "automatic" can include fully automated or semi-automated implementations, including user or operator control over certain aspects of the implementation, depending on the implementation desired by those skilled in the art practicing the implementation of the present invention. Furthermore, sequential terms such as "first," "second," and "third" may be used in the present specification and claims for descriptive purposes only and are not intended to be limited to references to described acts or items occurring in the order described. Acts or items may be arranged in different orders or performed in parallel or dynamically without departing from the scope of the present invention.

[0010]

[19] The term "computer-readable medium" in the present invention may include a local storage device, a cloud-based storage device, a remotely located server, or any other storage device that may be apparent to one of ordinary skill in the art.

[0011]

[20] Exemplary implementations of the machine vision or object detection system of the present invention can increase the probability of detecting object edges, for example by detecting gaps or spaces between objects or boxes, by using two or more image data modes to detect gaps or contrast differences between objects. By using two or more image data modes, differences in materials, backgrounds, cases or objects, ambient lighting, or reflections can be detected, and based on these detections, the computer system can generate fused edge information that identifies edges between gaps, or point clouds, to determine the size of the object. Furthermore, by fusing edge information detected using different image data modes, as described below, exemplary implementations can read the contour around the object or box, which is the edge of a box on a map, thereby defining this area and increasing the likelihood of providing an accurate detection hypothesis for subsequent processing. By increasing the probability of detecting object location and contours, the object can be better tracked and located within a production processing, packaging, or shipping facility.

[0012]

[21] Exemplary implementations described herein may relate to tracking pallets or boxes passing through a sorting or manufacturing facility. However, the exemplary implementations are not limited to these applications. Exemplary implementations of the processes and systems described herein may also be used for 3D mapping, autonomous vision, or any other application that may benefit from improved edge detection. Exemplary implementations provide improved edge detection results by using more modules of data compared to traditional edge detection systems, overcoming the shortcomings and limitations of different sensing devices and combining their strengths and capabilities in a given environment.

[0013] The right environment

[22] Figure 1A illustrates an exemplary environment for a robotic system 100 having a machine vision or object detection system, according to an exemplary embodiment of the present application. The robotic system 100 includes one or more structures (e.g., robots) configured to perform one or more tasks. Aspects of the piece loss management mechanism may be performed or implemented by a variety of structures.

[0014]

[23] In the example shown in FIG. 1A, the robotic system 100 may include an unloading unit 102, a transfer unit 104, a transport unit 106, a loading unit 108, or a combination thereof, at a warehouse or distribution / shipping site. Each unit of the robotic system 100 may be configured to perform one or more tasks. Tasks may be combined in sequence to perform operations to achieve an objective, such as, for example, unloading an object from a truck or van for storage in a warehouse, or unloading an object from a storage location and loading it onto a truck or van for shipping. In another example, a task may include moving an object from one container to another. Each unit may be configured to perform a series of actions (e.g., manipulating one or more components within the unit) to perform the task.

[0015]

[24] In some embodiments, a task may include manipulating (e.g., moving and / or reorienting) a target object 112 (e.g., a box, case, cage, pallet, etc.) from a start location 114 to a task location 116. For example, the unloading unit 102 (e.g., a devanning robot) may be configured to transfer the target object 112 from a position in a carrier (e.g., a truck) to a position on a conveyor belt, and the transport unit 104 (e.g., a palletizing robot) may be configured to transfer the target object 112 from a position on the conveyor belt to a position on the transport unit 106, for example, to load the target object 112 onto a pallet on the transport unit 106.

[0016]

[25] According to another example embodiment, the transfer unit 104 (e.g., a piece-picking robot) can be configured to transfer the target object 112 from one container to another. When the transport unit 106 completes a task, it can transfer the target object 112 from an area associated with the transfer unit 104 to an area associated with the loading unit 108. The loading unit 108 can transfer the target object 112 from the transfer unit 104 (e.g., by moving a pallet carrying the target object 112) to a storage location (e.g., a location on a shelf). More details about the tasks and associated operations are provided below.

[0017]

[26] For purposes of illustration, robotic system 100 is described in the context of a fulfillment center, although it is understood that robotic system 100 can be configured to perform tasks for other purposes in other environments, such as for manufacturing, assembly, packaging, medical, and / or other types of automation. It is also understood that robotic system 100 can include other units not shown in FIG. 1, such as manipulators, service robots, modular robots, etc.

[0018]

[27] For example, in some embodiments, the robotic system 100 may include a depalletizing unit for transferring objects from a cage, cart, or pallet to a conveyor or another pallet, a container exchange unit for transferring objects from one container to another, a packaging unit for packaging objects, a sorting unit for classifying objects according to one or more characteristics thereof, a piece-picking unit for performing different operations on the objects (e.g., sorting, grouping, and / or transporting) according to one or more characteristics thereof, or combinations thereof.

[0019] The right system 1B is a block diagram illustrating a robotic system 100 having a machine vision or object detection system, according to an exemplary embodiment of the present technology. For example, in some embodiments, the robotic system 100 (e.g., in one or more of the units and / or robots described above) may include, for example, one or more processors 118, one or more storage devices 120, one or more communication devices 122, one or more input / output devices 124, one or more drive devices 128, one or more transport motors 130, one or more sensors 132, or combinations thereof.

[0020]

[29] Various devices may be communicatively coupled to one another via wired and / or wireless connections. For example, the robotic system 100 may include a bus, such as a system bus, a Peripheral Component Interconnect (PCI) bus or a PCI Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also known as "FIREWIRE"). For example, the robotic system 100 may also include bridges, adapters, controllers, or other signal-related devices to provide wired connections between devices. The wireless connection may be based on, for example, a cellular communication protocol (e.g., 3G, 4G, LTE, 5G, etc.), a wireless local area network (LAN) protocol (e.g., Wireless Fidelity (WIFI)), a peer-to-peer or device-to-device communication protocol (e.g., Bluetooth, Near Field Communication (NFC), etc.), an Internet of Things (IoT) protocol (e.g., NB-IoT, LTE-M, etc.), and / or other wireless communication protocols.

[0021]

[30] Processor 118 may include a data processor (e.g., a central processing unit (CPU), a special purpose computer, and / or an on-board server) configured to execute instructions (e.g., software instructions) stored in storage device 120 (e.g., computer memory). Processor 118 executes program instructions to control or interface with other devices, thereby causing robotic system 100 to perform actions, tasks, and / or operations.

[0022]

[31] Storage device 120 may comprise a non-transitory computer-readable medium on which program instructions (e.g., software) are stored. Some examples of storage device 120 may include volatile memory (e.g., cache memory and / or random access memory (RAM)) and / or non-volatile memory (e.g., flash memory and / or magnetic disk storage). Other examples of storage device 120 may include portable memory devices and / or cloud storage devices.

[0023]

[32] In some embodiments, storage device 120 may be used to further store and access processing results and / or data / thresholds (e.g., predefined data and thresholds). For example, storage device 120 may store master data including descriptions of objects (e.g., boxes, cases, and / or products) that robotic system 100 may manipulate. In one or more embodiments, the master data may include dimensions, shapes (e.g., templates of potential poses and / or computer-generated models for recognizing objects in various poses), color schemes, images, identification information (e.g., barcodes, Quick Response (QR) codes, logos, etc., and / or expected locations thereof), expected weights, or combinations thereof, of objects expected to be manipulated by robotic system 100. In some embodiments, the master data may include manipulation-related information about the objects, such as center of gravity locations for each object, expected sensor measurements (e.g., of force, torque, pressure, and / or contact measurements) corresponding to one or more motions / manipulations, or combinations thereof.

[0024]

[33] Also, for example, storage device 120 may store object tracking data. In some embodiments, the object tracking data may include a log of scanned or manipulated objects. In some embodiments, the object tracking data may include image data (e.g., photographs, point clouds, live video images, etc.) of objects at one or more locations (e.g., designated loading or unloading locations and / or conveyor belts). In some embodiments, the object tracking data may include the position and / or orientation of objects at one or more locations.

[0025]

[34] The communication device 122 may comprise circuitry configured to communicate with external or remote devices over a network. The communication device 122 may comprise, for example, a receiver, a transmitter, a modulator / demodulator (modem), a signal detector, a signal encoder / decoder, a connector port, a network card, etc. The communication device 122 may be configured to transmit, receive, and / or process electrical signals according to one or more communication protocols (e.g., Internet Protocol (IP), wireless communication protocols, etc.). In some embodiments, the robotic system 100 may use the communication device 122 to exchange information between units of the robotic system 100 and / or with systems or devices external to the robotic system 100 (e.g., for reporting, data collection, analysis, and / or troubleshooting purposes).

[0026]

[35] The input / output devices 124 can include user interface devices configured to communicate information with an operator (e.g., a human operator). For example, the input / output devices 124 can include a display 126 and / or other output devices (e.g., speakers, haptic circuitry, or haptic feedback devices, etc.) for communicating information to a human operator. The input / output devices 124 can also include control or receiving devices, such as a keyboard, mouse, touch screen, microphone, user interface (UI) sensors (e.g., a camera for receiving motion commands), wearable input devices, etc. In some embodiments, the robotic system 100 can use the input / output devices 124 to interact with a human operator when performing a motion, task, operation, or combination thereof.

[0027]

[36] The robotic system 100 may comprise physical or structural members (e.g., robotic manipulator arms) articulated for movement (e.g., rotation and / or translational displacement). The structural members and joints may form a kinematic chain configured to manipulate an end effector (e.g., gripper) configured to perform one or more tasks (e.g., grasping, rotating, welding, etc.) depending on the use / operation of the robotic system 100. The robotic system 100 may comprise actuation devices 128 (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to actuate or manipulate (e.g., move and / or reorient) the structural members at or about the corresponding joints. In some embodiments, the robotic system 100 may comprise a transport motor 130 configured to transport the corresponding unit / chassis around.

[0028]

[37] The robotic system 100 may include sensors 132 configured to obtain information used to perform tasks, such as to manipulate structural members and / or transport the robotic unit. The sensors 132 may include devices configured to detect or measure one or more physical characteristics of the robotic system 100 (e.g., the state, status, and / or position of one or more structural members / joints thereof) and / or the surrounding environment. Some examples of the sensors 132 may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, and position encoders.

[0029]

[38] In some embodiments, for example, the sensor 132 can include one or more imaging devices 134 (e.g., 2D and / or 3D cameras, including visual and / or infrared cameras, LADAR, RADAR, and / or other distance measuring or imaging devices) configured to detect the surrounding environment. The imaging devices 134 can detect and generate a representation of the detected environment, such as a digital image and / or a point cloud, that is used to perform machine / computer vision (e.g., for automated inspection, robotic guidance, or other robotic applications). As described in more detail below, the robotic system 100 can process (e.g., via the processor 118) the digital image and / or point cloud to identify the target object 112 of FIG. 1A, the start position 114 of FIG. 1A, the task position 116 of FIG. 1A, the pose of the target object 112 of FIG. 1A, or a combination thereof.

[0030]

[39] To manipulate a target object 112, the robotic system 100 can capture (e.g., via various units) and analyze images of a designated area (e.g., inside a truck, inside a container, or a pick-up location for an object on a conveyor belt) to identify the target object 112 and its start location 114. Similarly, the robotic system 100 can capture and analyze images of another designated area (e.g., a drop location for placing the object on a conveyor belt, a location for placing the object inside a container, or a loading location on a pallet) to identify a task location 116.

[0031]

[40] Also for example, the sensors 132 may include position sensors 136 (e.g., position encoders, potentiometers, etc.) configured to detect the positions of structural members (e.g., a robot arm and / or an end effector) and / or corresponding joints of the robotic system 100. The robotic system 100 may use the position sensors 136 to track the positions and / or orientations of the structural members and / or joints during the performance of a task.

[0032]

[41] In some embodiments, the sensor 132 may include a contact sensor 138 (e.g., a pressure sensor, a force sensor, a strain gauge, a piezoresistive / piezoelectric sensor, a capacitive sensor, an elasto-resistive sensor, and / or other tactile sensor) configured to measure a characteristic associated with direct contact between two or more physical structures or surfaces. The contact sensor 138 may measure a characteristic corresponding to the grip of the end effector (e.g., a gripper) on the target object 112. As such, the contact sensor 138 may output a contact measure that represents a quantified measure (e.g., a measured force, torque, position, etc.) corresponding to the degree of contact or intimacy between the gripper and the target object 112. For example, the contact measure may include one or more force or torque measurements associated with a force applied by the end effector to the target object 112. Contact measures are described in more detail below.

[0033]

[42] As described in more detail below, the robotic system 100 (e.g., via the processor 118) can perform various actions to accomplish a task based on the contact measure. For example, the robotic system 100 can re-grasp the target object 112 if the initial contact measure is below a threshold. The robotic system 100 can also intentionally put down the target object 112, adjust the task position 116, adjust the speed or acceleration of the operation, or a combination thereof, if the contact measure falls below a threshold during the performance of a task.

[0034]

[43] Exemplary embodiments of the end effectors (e.g., grippers) of the present application may provide greater flexibility in tool selection and customization by providing the ability to change tool heads based on required capabilities. For example, a variable gripper may be swapped out to match the type of object to be gripped. Matching may be based on the size of the object, the delicacy of the object, or any other factor that would be apparent to one of ordinary skill in the art. This changeability is provided by a connector structure communicatively coupled to a control module configured to detect tool changes and adjust signal I / O operations and the amount of compressed air provided to match the requirements of the new tool.

[0035] Exemplary Implementations

[44] FIG. 2 illustrates an apparatus 205 for detecting objects 275a-275c moving along a conveyor 220 according to a first exemplary implementation of the present invention. The apparatus 205 may include two image data capture devices 210 / 212 spaced apart from the conveyor 220 along the Z axis. As illustrated, the image capture devices 210 / 212 are aligned with the direction of movement of the objects 275a-275c along the conveyor 220 and are located on the same side of the conveyor 220 as the objects 275a-275c. However, the exemplary implementation is not limited to this configuration and may be configured in other configurations as would be apparent to one of ordinary skill in the art. Other exemplary configurations are illustrated in FIGS. 3 and 4, discussed below. Additionally, for illustrative purposes, the apparatus 205 is illustrated with the objects 275a-275c on the conveyor 220, but it will be understood that the objects 275a-275c may be arranged in different configurations. For example, objects 275a-275c may be loaded or placed on a platform, such as a pallet, or within an object container, such as a cart or bin.

[0036]

[45] In some example implementations, one or both of image data capture devices 210 / 212 may be a two-dimensional imaging device or camera, such as a color still camera (e.g., an RGB camera), a color video camera, a monochrome or grayscale still camera, a monochrome or grayscale video camera, or any other two-dimensional imaging device as would be apparent to one of ordinary skill in the art. Also, in some example implementations, each of image data capture devices 210 / 212 may be a different type of two-dimensional imaging device (e.g., one image capture device may be an RGB camera and the other image capture device may be a monochrome camera, or any other combination of two-dimensional imaging devices as would be apparent to one of ordinary skill in the art).

[0037]

[46] Additionally, in some example implementations, one or both of the image data capture devices 210 / 212 may be three-dimensional imaging devices capable of generating 3D point cloud information, such as, for example, a stereo vision camera, a range imaging camera (i.e., a time-of-flight (TOF) camera), or any other device capable of generating 3D point cloud information. Also, in some example implementations, each of the image data capture devices 210 / 212 may be a different type of three-dimensional imaging device (e.g., one image capture device may be a stereo vision camera and the other image capture device may be a time-of-flight camera, or any other combination of three-dimensional imaging devices as would be apparent to one of ordinary skill in the art).

[0038]

[47] Furthermore, in some example implementations, one of the image capture devices (e.g., device 210) may be a two-dimensional image capture device and the other of the image capture devices (e.g., device 212) may be a three-dimensional image capture device.

[0039] 2, the image capture devices 210 / 212 can be configured to capture or generate image data of the objects 275a-275c as the objects 275a-275c pass beneath the image capture devices 210 / 212 during movement on the conveyor 220. In some example implementations, the objects 275a-275c may be boxes, packages, or pallets moving along the conveyor 220 or a similar device in an industrial, manufacturing, or sorting facility.

[0040]

[49] The apparatus 205 may also include a computing device 225 coupled to the image capture device 210 / 212 and configured to process image data of the objects 275a-275c. In some example implementations, the computing device 225 may include a system interface 280 (e.g., an input / output (I / O) interface) that communicatively couples the image capture device 210 / 212 to one or more processors within the computing device 225. For example, the system interface 280 of the computing device 225 may receive image data captured or generated by the image capture device 210 / 212 and provide the image data to the one or more processors of the computing device 225.

[0041]

[50] Additionally, the one or more processors of the computing device 225 may be configured to detect edges of the objects 275a-275c based on image data captured or generated by the image capture devices 210 / 212. As another example, the one or more processors of the computing device 225 may be configured to blend the detected edges into the image data of the image capture devices 210 / 212 to detect gaps between the objects 275a-275c. The edge detection and blending processes are discussed in more detail below. In some example implementations, the computing device 225 may be, for example, the computing device 905 of the computing environment 900 of FIG. 9 discussed below. Also, the system interface 280 may be, for example, the I / O interface 925 of the computing device 905 of FIG. 9. Additionally, the one or more processors may be, for example, the processor 910 of the computing device 905 of FIG. 9.

[0042]

[51] The computing device 225 can also generate a combined image signal from the output of the image data capture devices 210 / 212 to better detect edges and gaps between the objects 275a-275c. In some example implementations, the computing device 225 can also transmit the combined image signal to a number of devices 235-265 communicatively connected to each other, for example, via a network 230 (e.g., by wired and / or wireless connections), that can use the detected gaps and edges for sorting, tracking, or any other purpose that would be apparent to one of ordinary skill in the art.

[0043]

[52] The plurality of devices 235-265 may include, but are not limited to, a computer 235 (e.g., a laptop computing device), a mobile device 240 (e.g., a smartphone or tablet), a wearable device 255 (e.g., a smart watch), a server computer 245 connected to a storage device 250, and other computing devices 260-265 as would be apparent to one skilled in the art. The devices 235-265 may also have a computing environment 900 of FIG. 9 shown below.

[0044]

[53] In some example implementations, computing device 225 may generate a combined image signal prior to transmitting to multiple devices 235-265, although example implementations of the invention are not limited to this configuration. For example, computing device 225 may combine image data from image data capture device 210 (sometimes referred to as first image data capture device 210) with image data from image data capture device 212 (sometimes referred to as second image data capture device 212) to detect edges and gaps associated with objects 275a-275c, and generate a combined image signal using processes or algorithms discussed below.

[0045]

[54] Conversely, in other example implementations, the computing device 225 may transmit the image data from the image data capture device 210 and the image data from the image data capture device 212 as a separation signal to one or more of the plurality of devices 235-265. Once the separation signals are received, each of the plurality of devices 235-265 may combine the separation signals using a process or algorithm discussed below to generate a combined image signal to detect edges and gaps associated with the objects 275a-275c.

[0046]

[55] The computing device 225 may determine whether to transmit a combined image signal or separate signals corresponding to the image data from the image data capture device 210 and the image data from the image data capture device 212 to generate a combined image signal during an initial setup or reconfigurable setup of the apparatus 205. The determination of whether to transmit a combined image signal or separate signals may be based on querying each of the plurality of devices 235-265 to determine the capabilities of each of the plurality of devices 235-265 and the bandwidth available for data transfer to each of the plurality of devices 235-265.

[0047]

[56] Figure 3 shows an apparatus 305 for detecting objects 375a-375c moving along a conveyor 320 according to a second exemplary implementation of the present invention. The apparatus 305 is similar to the apparatus 205 shown in Figure 2 and described above. Accordingly, similar reference numbers are used to indicate similar structure and components.

[0048]

[57] The apparatus 305 includes two image data capture devices 310 / 312 spaced apart from the transporter 320 along the Z axis. Unlike FIG. 2, however, the image data capture devices 310 / 312 are shown angled relative to each other while being located on the same side of the transporter 320 as the objects 375a-375c. Similarly, the exemplary implementation is not limited to this configuration and may be configured in other configurations as would be apparent to one of ordinary skill in the art. Another exemplary configuration is shown in FIG. 4, discussed below.

[0049]

[58] In some example implementations, one or both of the image data capture devices 310 / 312 may be a two-dimensional imaging device or camera, such as a color still camera (e.g., an RGB camera), a color video camera, a monochrome or grayscale still camera, a monochrome or grayscale video camera, or any other two-dimensional imaging device as would be apparent to one of ordinary skill in the art. Also, in some example implementations, each of the image data capture devices 310 / 312 may be a different type of two-dimensional imaging device (e.g., one image capture device may be an RGB camera and the other image capture device may be a monochrome camera, or any other combination of two-dimensional imaging devices as would be apparent to one of ordinary skill in the art).

[0050]

[59] Additionally, in some example implementations, one or both of the image data capture devices 310 / 312 may be a three-dimensional imaging device capable of capturing a 3D point cloud, such as a stereo vision camera, a time-of-flight (TOF) camera, or any other device capable of capturing a 3D point cloud. Also, in some example implementations, each of the image data capture devices 310 / 312 may be a different type of three-dimensional imaging device (e.g., one image capture device may be a stereo vision camera and the other image capture device may be a time-of-flight camera, or any other combination of three-dimensional imaging devices as would be apparent to one of ordinary skill in the art).

[0051]

[60] Furthermore, in some example implementations, one of the image capture devices (e.g., device 310) may be a two-dimensional capture device and the other of the image capture devices (e.g., device 312) may be a three-dimensional capture device.

[0052]

[61] The image capture devices 310 / 312 can be configured to capture image data of objects 375a-375c passing beneath the image capture devices 310 / 312 during movement on the transporter 320. In some example implementations, the objects 375a-375c can be boxes, packages, or pallets moving along a conveyor belt or similar device in an industrial, manufacturing, or sorting facility.

[0053]

[62] The apparatus 305 may also include a computing device 325 coupled to the image capture devices 310 / 312 and configured to process image data of the objects 375a-375c. In some example implementations, the computing device 325 may include a system interface 380 (e.g., an I / O interface) that communicatively couples the image capture devices 310 / 312 to one or more processors within the computing device 325. For example, the system interface 380 of the computing device 325 may receive image data captured or generated by the image capture devices 310 / 312 and provide the image data to the one or more processors of the computing device 325.

[0054]

[63] The one or more processors of the computing device 325 may be configured to detect edges of the objects 375a-375c and blend the detected edges to detect gaps between the objects 375a-375c. The processes of detecting edges and blending are discussed in more detail below. In some example implementations, the computing device 325 may be, for example, the computing device 905 of the computing environment 900 of FIG. 9 discussed below. Also, the system interface 380 may be, for example, the I / O interface 925 of the computing device 905 of FIG. 9. Furthermore, the one or more processors may be, for example, the processor 910 of the computing device 905 of FIG. 9.

[0055]

[64] The computing device 325 can also generate a combined image signal from the output of the image data capture devices 310 / 312 to better detect edges and gaps between the objects 375a-375c. In some example implementations, the computing device 325 can also transmit the combined image signal to a number of devices 335-365 communicatively connected to each other, for example, via a network 330 (e.g., by wired and / or wireless connections), that can use the detected gaps and edges for sorting, tracking, or any other purpose that would be apparent to one of skill in the art.

[0056]

[65] The plurality of devices 335-365 may include, but are not limited to, a computer 335 (e.g., a laptop computing device), a mobile device 340 (e.g., a smartphone or tablet), a wearable device 355 (e.g., a smart watch), a server computer 345 connected to a storage device 350, and other computing devices 360-365 as would be apparent to one of ordinary skill in the art. The devices 335-365 may also have a computing environment 900 of FIG. 9 shown below.

[0057]

[66] In some example implementations, the computing device 325 may generate a combined image signal prior to transmitting to the plurality of devices 335-365, although example implementations of the invention are not limited to this configuration. For example, the computing device 325 may combine image data from the image data capture device 310 (which may be referred to as the first image data capture device 310) with image data from the image data capture device 312 (which may be referred to as the second image data capture device 312) to detect edges and gaps associated with the objects 375a-375c, and generate a combined image signal using processes or algorithms discussed below.

[0058]

[67] Conversely, in other example implementations, the computing device 325 may transmit the image data from the image data capture device 310 and the image data from the image data capture device 312 as a separation signal to one or more of the plurality of devices 335-365. Once the separation signals are received, each of the plurality of devices 335-365 may combine the separation signals using a process or algorithm discussed below to generate a combined image signal to detect edges and gaps associated with the objects 375a-375c.

[0059]

[68] The computing device 325 may determine whether to transmit a combined image signal or separate signals corresponding to the image data from the image data capture device 310 and the image data from the image data capture device 312 to generate a combined image signal during an initial setup or reconfigurable setup of the apparatus 305. The determination of whether to transmit a combined image signal or separate signals may be based on querying each of the plurality of devices 335-365 to determine the capabilities of each of the plurality of devices 335-365 and the bandwidth available for data transfer to each of the plurality of devices 335-365.

[0060]

[69] Figure 4 illustrates an apparatus 405 for detecting objects 475a-475c moving along a conveyor 420 according to a third exemplary implementation of the present invention. The apparatus 405 is similar to the apparatus 205 illustrated in Figure 2 and described above. Accordingly, similar reference numbers are used to indicate similar structure and components.

[0061]

[70] The apparatus 405 includes two image data capture devices 410 / 412 spaced apart from the transporter along the Z axis. Unlike FIG. 2, however, the image capture devices 410 / 412 are shown aligned with one or more of the X, Y, or Z axes while being located on different sides of the objects 475a-475c, more specifically, on opposite sides of the transporter 420. Similarly, the exemplary implementation is not limited to this configuration and may be configured in other configurations as would be apparent to one of ordinary skill in the art.

[0062]

[71] In some example implementations, one or both of the image data capture devices 410 / 412 may be a two-dimensional imaging device or camera, such as a color still camera (e.g., an RGB camera), a color video camera, a monochrome or grayscale still camera, a monochrome or grayscale video camera, or any other two-dimensional imaging device as would be apparent to one of ordinary skill in the art. Also, in some example implementations, each of the image data capture devices 410 / 412 may be a different type of two-dimensional imaging device (e.g., one image capture device may be an RGB camera and the other image capture device may be a monochrome camera, or any other combination of two-dimensional imaging devices as would be apparent to one of ordinary skill in the art).

[0063]

[72] Additionally, in some example implementations, one or both of the image data capture devices 410 / 412 may be a three-dimensional imaging device capable of capturing a 3D point cloud, such as a stereo vision camera, a time-of-flight (TOF) camera, or any other device capable of capturing a 3D point cloud. Also, in some example implementations, each of the image data capture devices 410 / 412 may be a different type of three-dimensional imaging device (e.g., one image capture device may be a stereo vision camera and the other image capture device may be a time-of-flight camera, or any other combination of three-dimensional imaging devices as would be apparent to one of ordinary skill in the art).

[0064]

[73] Furthermore, in some example implementations, one of the image capture devices (e.g., device 410) may be a two-dimensional capture device and the other of the image capture devices (e.g., device 412) may be a three-dimensional capture device.

[0065]

[74] The image capture devices 410 / 412 can be configured to capture image data of objects 475a-475c passing beneath the image capture devices 410 / 412 during movement on the transporter 420. In some example implementations, the objects 475a-475c can be boxes, packages, or pallets moving along a conveyor belt or similar device in an industrial, manufacturing, or sorting facility.

[0066]

[75] The apparatus 405 may also include a computing device 425 coupled to the image capture devices 410 / 412 and configured to process image data of the objects 475a-475c. In some example implementations, the computing device 425 may include a system interface 480 (e.g., an I / O interface) that communicatively couples the image capture devices 410 / 412 to one or more processors within the computing device 425. For example, the system interface 480 of the computing device 425 may receive image data captured or generated by the image capture devices 410 / 412 and provide the image data to the one or more processors of the computing device 425.

[0067]

[76] The one or more processors of the computing device 425 can be configured to detect edges of objects and blend edges to detect gaps between the objects 475a-475c. The processes of detecting edges and blending edges are discussed in more detail below. In some example implementations, the computing device 425 can be, for example, the computing device 905 of the computing environment 900 of FIG. 9, discussed below. Also, the system interface 480 can be, for example, the I / O interface 925 of the computing device 905 of FIG. 9. Furthermore, the one or more processors can be, for example, the processor 910 of the computing device 905 of FIG. 9.

[0068]

[77] The computing device 425 can also generate a combined image signal from the output of the image data capture devices 410 / 412 to better detect edges and gaps between the objects 475a-475c. In some example implementations, the computing device 425 can also transmit the combined image signal to a number of devices 435-465 communicatively connected to each other, for example, via a network 430 (e.g., by wired and / or wireless connections), that can use the detected gaps and edges for sorting, tracking, or any other purpose that would be apparent to one of skill in the art.

[0069]

[78] The plurality of devices 435-465 may include, but are not limited to, a computer 435 (e.g., a laptop computing device), a mobile device 440 (e.g., a smartphone or tablet), a wearable device 455 (e.g., a smart watch), a server computer 445 connected to a storage device 450, and other computing devices 460-465 as would be apparent to one skilled in the art. The devices 435-465 may also have a computing environment 900 of FIG. 9 shown below.

[0070]

[79] In some example implementations, computing device 425 may generate a combined image signal prior to transmitting to multiple devices 435-465, although example implementations of the invention are not limited to this configuration. For example, computing device 425 may combine image data from image data capture device 410 (which may also be referred to as first image capture device 410) with image data from image data capture device 412 (which may also be referred to as second image capture device 412) to detect edges and gaps associated with objects 475a-475c, and generate a combined image signal using processes or algorithms discussed below.

[0071]

[80] Conversely, in other example implementations, the computing device 425 may transmit the image data from the image data capture device 410 and the image data from the image data capture device 412 as a separation signal to one or more of the plurality of devices 435-465. Once the separation signals are received, each of the plurality of devices 435-465 may combine the separation signals using a process or algorithm discussed below to generate a combined image signal to detect edges and gaps associated with the objects 475a-475c.

[0072]

[81] The computing device 425 may determine whether to transmit a combined image signal or separate signals in response to the image data from the image data capture device 410 and the image data from the image data capture device 412 to generate a combined image signal during an initial setup or reconfigurable setup of the apparatus 405. The determination of whether to transmit a combined image signal or separate signals may be based on querying each of the plurality of devices 435-465 to determine the capabilities of each of the plurality of devices 435-465 and the bandwidth available for data transfer to each of the plurality of devices 435-465.

[0073]

[82] Figure 5 shows a schematic diagram 500 of image data that may represent edge detection results of a single image capture device or camera according to a comparative example. As shown, the edge detection results of the image data may include multiple rectangles 502, 510, and 530 of different sizes that illustrate boxes or objects of approximately the same size on a palette 501. In other words, the image data in Figure 5 shows that the rectangles 502, 510, and 530 have different sizes, indicating that the system is detecting objects or boxes represented by each rectangle having different sizes from each other. However, in reality, the actual boxes or objects being imaged by the image capture device that the system is trying to represent with the rectangles 502, 510, 530 are of approximately the same size and shape in this example. In other words, the system is erroneously detecting the boxes as having different shapes and sizes when in fact they have approximately the same size and shape.

[0074]

[83] Each box and object has a detected shape defined by edges detected in the image data (e.g., lines and curves that represent edges that define the shape of the object). In Figure 5, the edges of a box or object detected by a single image capture device are shown as straight lines represented by reference numerals 505, 512, 515 and 535 that make up the sides of rectangles 502, 510 and 530. However, the detected edges may not be straight, but may be curved or irregularly shaped.

[0075]

[84] As shown, line 505 (representing a detected edge) is detected as having a first length, and line 512 (representing another detected edge) is detected as having a second length longer than line 505. Additionally, lines 515 and 535 (representing detected edges, respectively) are detected as having shorter lengths compared to the detected edges represented by lines 505 and 512. In this example, no additional internal edges (e.g., edges between objects or boxes represented in the image data) have been detected in regions 520 and 525, although internal edges may exist in actual boxes or objects present in regions 520 and 525. An exemplary implementation may blend edges or contours detected from different image data modes using a process 800 described below with respect to FIG. 8 to address errors in contour detection (e.g., undetected internal edges, detected edges having lengths longer or shorter than the edges of actual objects, or any other detection errors that would be apparent to one of ordinary skill in the art).

[0076]

[85] Figure 6 shows a schematic diagram 600 of a fused image data representation that may represent a fused edge detection result in an exemplary implementation of the present invention. As shown, two different image data representations (e.g., a first image data representation 610 and a second image data representation 615) may be combined into a fused image data representation 605 that represents fused image data corresponding to fused edges or contours of detected objects.

[0077]

[86] For example, the first image data representation 610 may be a visual representation of image data captured using 2D image data, such as monochrome camera image data. In some examples, the 2D image capture device (e.g., a 2D image camera) may not have sufficient resolution to determine contrast between different objects, such as boxes. If the resolution of the 2D image capture device is not sufficient to determine contrast between different boxes, some of the edges (e.g., interior edges between adjacent boxes) may not be detected while other edges are accurately detected as represented by line 660, resulting in inaccurately detected edges, such as line 655. In FIG. 6, areas representing detected objects based on inaccurately detected edges are shown as shaded regions 620. The first image data representation 610 alone may be considered a detection failure due to failure to accurately detect interior edges between boxes in the area of ​​inaccurately detected edges (e.g., shaded regions 620). However, as described below, when the first image data representation 610 is combined with the second image data representation 615, a fused image data representation 605 may be generated.

[0078]

[87] As an example, the second image data representation 615 may be a visual representation of 3D point cloud data (also referred to as 3D image data) generated by a 3D camera (e.g., a depth perception camera). The shaded area of ​​region 625 may represent the 3D point cloud data. Each portion of the 3D point cloud may represent a box or height of an object within region 625, but edges around the object may be distorted or incompletely represented due to noise or environmental factors, as shown by determined contour line 640, compared to the actual edges shown by dashed lines 630. In some examples, edge detection based on processing and / or analysis of the 3D image data may accurately detect edges, as represented by line 665, while in other examples, edge detection based on processing and / or analysis of the 3D image data may result in incomplete or inaccurate edge detection, as represented by line 670. Some of the detected edges based on the second image data representation 615 may also be considered detection failures alone because some of the internal edges (i.e., lines 670) and edges around the object may be distorted, incomplete, and / or inaccurate (as shown by the difference between the determined contour line 640 and the actual edges shown by dashed lines 630).

[0079]

[88] The fused image data, shown in image data representation 605, may be generated by fusing image data (representing 2D image data) shown in first image data representation 610 with second image data representation 615 (representing 3D depth camera data) shown in first image data representation 610. As shown, the fused image data representation 605 may enable detection of nine distinct objects 635 based on improved detection accuracy of internal contours or edges 645 and surrounding contours or edges 650, which may be determined by fusing partially detected, imprecise, or incomplete detected edges with one another, such as detected edges represented by lines 670 and 655, and / or by a combination of accurately detected edges (i.e. edges with high detection confidence, such as exterior edges), such as detected edges represented by lines 66- and 665, with one or more of the partially detected, imprecise, or incomplete detected edges.

[0080]

[89] Thus, edge fusion using a process according to an exemplary implementation of the present invention can enable the generation of a fused image data representation 605 from a pair of a first image data representation 610 and a second image data representation 615 that each otherwise result in an object detection hypothesis that may be considered inaccurate or low confidence.

[0081]

[90] Figures 7A and 7B show schematic diagrams of object detection according to an exemplary implementation of the present invention. The robotic system 100 of Figure 1 can generate object detection hypotheses as described below with reference to step 820 in process 800 of Figure 8. Once the object detection hypotheses have been determined as described with reference to step 820, they can be verified as described below with reference to step 825 in process 800 of Figure 8. By using the fused image data, improved generation of object detection hypotheses and their verification can be achieved.

[0082]

[91] Figure 7A illustrates a first example of an inaccurate detection hypothesis that may be avoided by utilizing a fused edge data representation according to an exemplary implementation of the present invention. As shown in Figure 7A, in some examples, an internal contour or edge (e.g., a contour or edge between two boxes) represented by dashed line 715 may not be detected based on an analysis of edge detection data represented by rectangle 710 from a single image capture mode, and the robotic system 100 may not be able to accurately generate a detection hypothesis for the actual object represented by rectangle 705. As a result, the robotic system 100 may erroneously generate a detection hypothesis as rectangle 710. The fusion of edge detection data associated with different image data modes may improve the certainty of edge information and / or identify edges that would otherwise not be considered valid edge detection results for the object contour, thereby allowing the robotic system 100 to improve the generation and / or validation of detection hypotheses.

[0083]

[92] FIG. 7B illustrates a second example of an inaccurate detection hypothesis that may be avoided by utilizing fused detection edge data according to an exemplary implementation of the present invention. As shown, a series of boxes 725 on a palette 720 may be detected. A detection hypothesis (represented by a rectangle 745) may be generated based on image information representing an input image 740. However, in some examples, the robotic system 100 may not be able to detect a split contour (e.g., an internal contour or edge between two boxes) represented by a dashed line 735 by edge detection based on a single image capture mode, and may instead erroneously generate a detection hypothesis represented by a rectangle 745. In other words, the inability to recognize an edge or contour (e.g., an internal edge between two boxes) represented by the dashed line 735 may result in an incorrect or inaccurate object detection hypothesis. Fusion of edge detection data associated with different image data modes may enable better detection of object contours, thereby achieving improved generation and / or validation of object detection hypotheses.

[0084]

[93] Figure 8 shows a flow chart of a process 800 for object detection and verification according to an exemplary implementation of the invention. The process 800 may be performed by a computing device of the robotic system 100 (e.g., computing device 905 of the computing environment 900 shown in Figure 9 below).

[0085]

[94] As shown in FIG. 8, process 800 begins at step 805 with detection of edge information associated with image data captured by a first imaging mode. Edge information is information that defines and / or describes the contours of one or more objects captured in the image data. As discussed above, the edges of an object may be lines and / or curves that represent edges that define the shape of the object. Edge information may include edge dimensions (e.g., edge length), edge shape (e.g., straight, curved, etc.), edge location, and corner location (e.g., location where two or more edges meet). For example, edge information may be coordinate information (i.e., (x,y) coordinates) that corresponds to the location of one or more pixels that form a detected edge in the image data. Edge information may also include height or depth information (e.g., information representing the distance between an image capture device and a point on a surface) and the location of regions having similar heights. As an example, depth information may include a value that represents a distance from an image capture device (i.e., a 3D camera). In some embodiments, depth information may be a depth value assigned to a pixel of the image data.

[0086]

[95] In the case where multiple objects are captured in the image data, the image data may include an external or peripheral contour or edge located around a collection of the multiple objects captured in the image data. For example, the peripheral edge or contour may be represented by reference number 650 in the fused image data representation 605 of FIG. 6. Additionally, the image data may include an internal contour or edge located between adjacent objects. As an example, the internal contour or edge of a box may be perpendicular to the peripheral edge or contour. Additionally, for example, the internal contour or edge may be represented by reference number 645 in the fused image data representation 605 of FIG. 6.

[0087]

[96] The first imaging mode (e.g., first image capture mode) may be a process for capturing or generating image data. In some example implementations, the first imaging mode may be a process for capturing 2D image data to generate an image data representation, such as illustrated in first image data representation 610 discussed above with respect to FIG. 6. In other words, in some example implementations, edge information detection in the first imaging mode may be based on two-dimensional image data, such as data from a monochrome camera (e.g., a grayscale camera), a color camera (e.g., a red-green-blue (RGB) camera), or any other two-dimensional camera as would be apparent to one of ordinary skill in the art.

[0088]

[97] For two-dimensional image data, edge detection can be performed using any type of 2D edge detection algorithm that would be apparent to one skilled in the art. For example, the 2D edge detection method can be CANNY edge detection, region similarity edge detection (RSD), line segment detection (LSS), or any other 2D edge detection method that can be used to detect object contours and extract edge information.

[0089]

[98] In other example implementations, the first imaging mode may be a process of capturing 3D point cloud data that generates an image data representation such as that illustrated in the second image data representation 615 discussed above with respect to Figure 6. In other words, in some example implementations, the edge information detection of the first imaging mode may be based on three-dimensional data, such as data from a stereo vision camera, a range imaging camera (e.g., a time-of-flight (TOF) camera), or any other three-dimensional camera as would be apparent to one of ordinary skill in the art.

[0090]

[99] For three-dimensional image data (e.g., 3D point cloud data), the 3D point cloud may be a projection of points in image space that represent an object. Object contours or edges can then be identified from the 3D point cloud. For example, the robotic system 200 can identify object contours or edges from the projected 3D point cloud based on the change or difference in depth information between one location in the 3D point cloud and an adjacent location in the 3D point cloud.

[0091]

[0100] Edge information associated with the image data collected in the second imaging mode is detected in step 810. As discussed in more detail below, the second imaging mode is different from the first imaging mode associated with the edge information detection of step 805.

[0092]

[0101] As explained above, edge information is information that defines a contour of one or more objects captured in the image data. As discussed above, the contour of an object is a line and / or curve that represents an edge that defines the shape of the object. To reiterate, edge information can include edge dimensions (e.g., edge length), edge shape (e.g., straight, curved, etc.), edge location, and corner location (e.g., location where two or more edges meet). Edge information can also include height or depth information (e.g., information that represents a distance between an image capture device and a point on a surface) and location of areas having similar heights. The contour can include an external contour or edge located around a collection of multiple objects captured in the image data or a peripheral contour or edge. For example, the peripheral edge or contour can be represented by reference number 650 of the fused image data representation 605 of FIG. 6. Furthermore, the image data can include an internal contour or edge located between adjacent objects. As an example, the internal contour or edge of a box can be perpendicular to the peripheral edge or contour. Furthermore, for example, the internal contour or edge can be represented by reference number 645 of the fused image data representation 605 of FIG. 6.

[0093]

[0102] The second imaging mode (e.g., second image capture mode) may also be a process for capturing or generating image data. In some example implementations, the second imaging mode may be a process for capturing 2D image data to generate an image data representation such as illustrated in the first image data representation 610 discussed above with respect to FIG. 6. In other words, in some example implementations, edge information detection of the second imaging mode may be based on two-dimensional image data, such as data from a monochrome camera (e.g., a grayscale camera), a color camera (e.g., an RGB camera), or any other two-dimensional camera as would be apparent to one of ordinary skill in the art.

[0094]

[0103] For two-dimensional image data, edge detection can be performed using any type of 2D edge detection algorithm that would be apparent to one skilled in the art, for example, the 2D edge detection method can be CANNY edge detection, region similarity edge detection (RSD), line segment detection (LSS), or any other 2D edge detection method that can be used to detect object contours and extract edge information.

[0095]

[0104] In other example implementations, the second imaging mode may be a process of capturing 3D point cloud data that generates an image data representation such as that illustrated in second image data representation 615 discussed above with respect to Figure 6. In other words, in some example implementations, edge information detection of the second imaging mode may be based on three-dimensional data, such as data from a stereo vision camera, a range imaging camera (e.g., a time-of-flight (TOF) camera), or any other three-dimensional camera as would be apparent to one of ordinary skill in the art.

[0096]

[0105] For three-dimensional image data (e.g., 3D point cloud data), the 3D point cloud may be a projection of points in an image space that represent an object. Object contours or edges can then be identified from the 3D point cloud. For example, the robotic system 200 can identify object contours or edges from the projected 3D point cloud based on the change or difference in depth information between one location in the 3D point cloud and an adjacent location in the 3D point cloud.

[0097]

[0106] The second imaging mode (e.g., the second image capture mode) is a different type of image capture mode compared to the first imaging mode. For example, if the edge detection associated with the first imaging mode was performed based on two-dimensional data, edge information may be detected in step 810 based on three-dimensional data or a different type of two-dimensional data (e.g., a monochrome camera if a color camera was used in step 805, or a color camera if monochrome camera data was used in step 805).

[0098]

[0107] Conversely, if edge detection associated with the first imaging mode is performed based on three-dimensional data, edge information can be detected in step 810 based on two-dimensional data or a different type of three-dimensional data (e.g., a stereo vision camera if a TOF camera was used in step 805, or a TOF camera if a stereo vision camera was used in step 805).

[0099]

[0108] In step 815, the edge information detected from the data associated with the first imaging mode and the second imaging mode is combined to generate fused edge information using a fusion process. Specifically, the edges detected based on the first imaging mode and the second imaging mode are composited into a "reference" camera space. In other words, the edges detected based on each of the first imaging mode and the second imaging mode are transformed from a native camera space (e.g., coordinate system) associated with each imaging mode to a shared or common camera space (e.g., reference camera space) discussed in more detail below. An exemplary implementation of the reference camera space can include a 3D camera space, such as the camera space of a stereo vision camera or a ToF camera.

[0100]

[0109] In some example implementations, edges detected from two-dimensional image data (e.g., from a 2D imaging mode) can be projected into three-dimensional camera space using depth information that can define edge heights relative to a two-dimensional plane or surface captured adjacent to the detected edge in the 2D image. Additionally, in some example implementations, the three-dimensional spatial data can be converted into a different three-dimensional spatial data mode (e.g., image data from a ToF camera can be converted into image data from a stereo vision camera).

[0101]

[0110] In other words, the fusion process can be performed by generating edge points (e.g., coordinates or positions of points along the detected edges) associated with one of the image data modes projected into a shared image space (e.g., a "reference" camera space). More specifically, the detected edges from one of the image data modes can be projected (or mapped) into a shared image space (also called a selected reference camera space) of a selected camera or imaging device, which can be, for example, a stereo vision camera space. During the fusion process, the robotic system 100 can consider and correct possible errors in the coordinates of the edge points due to factors such as point cloud noise or calibration (e.g., physical offset / angle / distance between the cameras) after transformation and projection.

[0102]

[0111] The projection of the detected edge in the image space of the selected reference camera (e.g., stereo vision camera) can be performed by using the original position information (i.e., coordinates) of the detected edge of the reference camera. In other words, the robotic system 100 needs to perform the projection of the detected edge from the selected reference camera to the shared image space. In some exemplary implementations, the selected reference camera can be associated with a first imaging mode. In other exemplary implementations, the selected camera can be associated with a second imaging mode.

[0103]

[0112] For detecting edges in the image space of other cameras (e.g., cameras that have a native image space different from the selected "reference" camera space), several different steps may be performed depending on the camera type.

[0104]

[0113] If the other imaging mode is a 2D camera, the edge points (also called 2D edge points) detected from the 2D image data of the 2D camera, which are the points constituting the detected edge, can be assigned depth values ​​by projecting the 2D edge points into the selected camera image space, which in this case is the 3D image space of the selected reference camera, using depth information from the selected reference camera corresponding to the projected 2D point. In other words, the 2D edge points are projected into the 3D image space of the selected reference camera using the depth value of the point (i.e., pixel) from the selected reference camera image space (e.g., stereo vision camera) that corresponds to the 2D position (e.g., the (x,y) pixel coordinate where the edge point of the detected edge is located in the 2D image space of the 2D camera). This allows the projected 3D edge points to be generated from the 2D edge points.

[0105]

[0114] Additionally, edge points captured by other cameras (e.g., cameras having a native image space different from the selected reference image space) can be transformed (also referred to as "transformed edge points") into the selected reference image space. The transformed edge points can be generated from the projected 2D edge points described in the previous paragraph, or can be generated from another 3D camera different from the selected reference camera or imaging device. For example, the transformed edge points generated from the projected 2D edge points can be transformed into the 3D space of the selected reference camera such that the transformed edge points can also be considered to be part of the original image data of the selected reference camera. Transforming the edge points of one camera to the selected reference camera can include translating or shifting the position of the projected 3D edge points and / or scaling the projected 3D edge points to align them with the reference camera image space of the selected camera.

[0106]

[0115] Once the projected 3D edge points are translated into the selected reference camera image space, the computer system 100 can generate a 2D representation (also referred to as a 2D bitmap) of the 3D point cloud including the projected 3D edge points (also referred to as a fused 3D point cloud). For example, the 2D bitmap can correspond to a top-down representation of the fused 3D point cloud, which may include the top surfaces of a stack of boxes and a pallet as shown in FIG. 6. In this regard, the 2D bitmap associated with the selected reference camera is independent of the depth values ​​in the 3D image space, but corresponds to the 2D coordinates of the image itself (e.g., the xy coordinates of the pixels of the image or the coordinates along the x and y axes shown in FIGS. 2-4 above). In some embodiments, the pixels of the 2D bitmap may match real-world dimensions (i.e., the length of a pixel may correspond to a unit length or distance in the real world). In such embodiments, the robotic system 100 can use the 2D bitmap of the fused 3D point cloud for downstream processes such as motion planning, grip positioning, and other operations acting on objects.

[0107]

[0116] Once the fused edge information is generated, the robotic system 100 may generate an object detection hypothesis based on the fused edge information in step 820. In some example implementations, the robotic system 100 may generate an object detection hypothesis based on segmenting the 3D point cloud information based on the fused edges of the fused edge information. For example, the robotic system 100 may segment the 3D point cloud information along the fused edges until an edge intersection is reached. The edge intersection may be, for example, a location where two or more fused edges meet, or in some examples, a location where an extrapolation of the detected edges would intersect. In the example of a box-shaped object, the robotic system 100 may generate an object detection hypothesis as a portion of the 3D point cloud information defined by two parallel pairs of fused edges, where one parallel pair of fused edges is perpendicular to the other parallel pair of fused edges, with the edge intersection coinciding with the intersection of the two parallel pairs of fused edges. In some embodiments, the robotic system 100 may generate multiple object detection hypotheses based on different combinations of fused edges.

[0108]

[0117] By using fused edge information to generate object detection hypotheses, as discussed above with respect to Figures 7A and 7B, several important advantages can be obtained. One important advantage that can be obtained is that the fused edge information can be used to more accurately segment the point cloud data compared to edge information generated from a single image capture mode. In applications where objects or boxes of approximately similar heights are detected, such as when the top surfaces of each of the boxes have the same or similar heights to form an approximately horizontal plane or surface, the portion of the point cloud corresponding to the top surfaces of the boxes (also referred to as a "point cloud layer") may be captured to encompass multiple objects or boxes. In this example, the boxes or objects may have the same or similar height / depth values ​​in the point cloud layer, making it difficult to detect individual boxes or objects (i.e., the interior contours 645 corresponding to the boundaries between the boxes). More specifically, segmentation of 3D point cloud data using detected edges based on 2D image data may result in incomplete edges (i.e., detected edges that correspond to a portion of the entire edge of the actual top surface of the box) or false edges (i.e., detected edges that do not actually correspond to the actual edge of the top surface of the box, such as the edge of a box flap, a fold / wrinkle, or a portion of a design / graphic / image on the top surface of the box). Furthermore, the boxes may be densely packed, so that depth differences that may correspond to separations between the boxes cannot be captured in the 3D image data. Thus, fused edge information combining edge information from multiple imaging modes may improve identification of actual object edges by reducing false edge detections and / or identifying edges that would otherwise go undetected, and increase the certainty of correlation of edge information with actual object edges. This may result in better segmentation of the point cloud layer and improved detection of individual objects or boxes.

[0109]

[0118] Another important advantage that can be obtained is that the fused edge information can be used as a feature for detection hypothesis generation. For example, as described above, the fused edge information can be used to detect or generate edge intersections (e.g., where two or more detected edges meet, or where extrapolations of detected edges would intersect) that may correspond to the location of corners of the object's contour. More specifically, analysis of image data from only a single imaging mode (i.e., edge detection based on two-dimensional image data or three-dimensional image data) may have a high probability of error or fail to identify edges, and using the fused edge information to generate detection hypotheses increases the probability that the object detection hypothesis is correct.

[0110]

[0119] After the object detection hypothesis is generated, the robotic system 100 may perform verification of the object detection hypothesis in step 825. In some embodiments, the robotic system 100 may verify the object detection hypothesis based on a comparison of the object detection hypothesis with the object description of the object registered in the master data of FIG. 2. For example, the robotic system 100 may verify the object detection hypothesis by matching the contour characteristics (i.e., the length of the fused edges, the position / location of the edge intersections) of the object detection hypothesis with those corresponding to the registered object in the master data. In some embodiments, the robotic system 100 may consider multiple factors, such as the completeness of the fused edge information (i.e., whether the fused edges include a break or the length of the fused edges spans the entire distance between the two intersections), which may affect the accuracy of the detected contour when comparing with the registered object during the verification of the object detection hypothesis. For example, the robotic system 100 may provide a certainty assessment of the object detection hypothesis or a degree of match with the registered object. As a specific example, the robotic system 100 may provide a higher certainty assessment to the detection hypothesis including a higher degree of completeness of the fused edge information, as described above. The fused edge information can provide the maximum possible information from the combination of available imaging devices and thus can help to increase the accuracy and reliability associated with determining the exact location of the edge or contour of an object based on a detection hypothesis. Furthermore, possible errors in the coordinates of edge points due to factors such as point cloud noise or calibration (e.g., physical offset / angle / distance between the cameras) after transformation and projection can be taken into account when determining the certainty rating or degree of match.

[0111]

[0120] After verifying the object detection hypothesis, the robotic system 100 may use the object contour, pose / orientation information, and / or position information of the target object (such as the target object 112 in FIG. 1 ) corresponding to the object detection hypothesis in step 830 for further processing of the target object. For example, the robotic system may use the object contour, pose / orientation information, and / or position information for operations including manipulation of the target object such as grip placement of the robot's end effector and / or motion plan calculations for moving the target object. Improved accuracy of contour and position detection may improve the shipping or manufacturing process. After the object contour and position are communicated, the process 800 may be repeated as necessary to generate further object detection hypotheses, which may include additional edge blending operations according to the steps above, or the process may end.

[0112]

[0121] Exemplary Computer Environment

[0122] 9 illustrates an example computing environment 900 with an example computing device 905 suitable for use in some example implementations. The computing device 905 of the computing environment 900 may include one or more processing units, cores, or processors 910, memory 915 (e.g., RAM, ROM, etc.), internal storage 920 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interfaces 925, any of which may be coupled over a communication mechanism or bus 930 for communicating information or embedded in the computing device 905.

[0113]

[0123] The computing device 905 can be communicatively coupled to an input / interface 935 and an output device / interface 940. Either or both of the input / interface 935 and the output device / interface 940 can be wired or wireless interfaces and can be detachable. The input / interface 935 can include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touch screen interfaces, keyboards, pointing / cursor control, microphones, cameras, Braille, motion sensors, optical readers, etc.).

[0114]

[0124] The output devices / interfaces 940 can include displays, televisions, monitors, printers, speakers, Braille, etc. In some example implementations, the input / interfaces 935 (e.g., user interfaces) and output devices / interfaces 940 can be embedded in or physically coupled to the computing device 905. In other example implementations, other computing devices can function as or provide the functionality of the input / interfaces 935 and output devices / interfaces 940 of the computing device 905. These elements can include, but are not limited to, well-known AR hardware inputs to allow a user to interact with the AR environment.

[0115]

[0125] Examples of computing devices 905 may include, but are not limited to, highly mobile devices (e.g., smart phones, devices in vehicles and other machines, devices carried by humans and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, server devices, other computers, information kiosks, televisions, radios with one or more processors embedded and / or coupled thereto, etc.).

[0116]

[0126] Computing device 905 may be communicatively coupled (e.g., via I / O interface 925) to external storage 945 and network 950 for communicating with any number of network components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 905, or any connected computing device, may function, provide services, and be referred to as a server, a client, a thin server, a general purpose machine, a special purpose machine, or another label.

[0117]

[0127] I / O interface 925 may include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11xs, Universal System Bus, WiMAX, modem, cellular network protocols, etc.) to communicate information to and from at least all connected components, devices, and networks in computing environment 900. Network 950 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0118]

[0128] The computing device 905 can use and / or communicate using computer usable or computer readable media, including transitory and non-transitory media. Transitory media includes transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, etc. Non-transitory media includes magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid media (e.g., RAM, ROM, flash memory, solid state storage), and other non-volatile storage or memory.

[0119]

[0129] The computing device 905 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. The computer-executable instructions can be retrieved from a transitory medium, stored on a non-transitory medium, or retrieved from a non-transitory medium. The executable instructions can be made from one or more of any programming language, scripting language, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0120]

[0130] The processor 910 may execute under any operating system (OS) (not shown), in a native or virtual environment. One or more applications may be deployed, including a logic unit 955, an application programming interface (API) unit 960, an input unit 965, an output unit 970, an edge detection unit 975, an edge blending unit 980, a hypothesis generation unit 985, a hypothesis verification unit 990, and an inter-unit communication mechanism 995 for different units to communicate with each other, the OS, and other applications (not shown).

[0121]

[0131] For example, the edge detection unit 975, the edge blending unit 980, the hypothesis generation unit 985, and the hypothesis verification unit 990 may implement one or more processes illustrated in Figure 8. The described units and elements may be modified in design, function, configuration, or implementation and are not limited to the description provided.

[0122]

[0132] In some example implementations, the information or execution instructions, once received by the API unit 960, can be communicated to one or more other units (e.g., edge detection unit 975, edge blending unit 980, hypothesis generation unit 985, and hypothesis verification unit 990). For example, the edge detection unit 975 can automatically detect object edges from the captured image data from at least two image capture modes and provide the detected edges to the edge blending unit 980. The edge blending unit 980 can fuse the detected edges to generate edge fusion data to provide to the hypothesis generation unit 985. The hypothesis generation unit 985 can generate object detection hypotheses based on the edge fusion data and provide to the hypothesis verification unit 990. The hypothesis verification unit 990 can verify the generated hypotheses based on the collected image data and the edge fusion data.

[0123]

[0133] In some examples, the logic unit 955 may be configured to control the flow of information between units and direct the services provided by the API unit 960, the input unit 965, the edge detection unit 975, the edge blending unit 980, the hypothesis generation unit 985, and the hypothesis testing unit 990 in some example implementations described above. For example, the flow of one or more processes or implementations may be controlled by the logic unit 955 alone or in cooperation with the API unit 960.

[0124]

[0134] Although several exemplary implementations have been shown and described, these exemplary implementations are provided to convey the subject matter described herein to those skilled in the art. It should be understood that the subject matter described herein can be embodied in various forms, without being limited to the exemplary implementations described. The subject matter described herein can be implemented without the matters specifically defined or described, or using other or different elements or matters not described. Those skilled in the art will understand that changes can be made in these exemplary implementations without departing from the subject matter described herein, as defined in the appended claims and their equivalents.

Claims

1. 1. A method for detecting an object, comprising: generating first edge information corresponding to an edge of an object from first image data representing an object in a plurality of objects, the plurality of objects including a second object adjacent to the object, the object and the second object each having a top surface substantially aligned along a horizontal plane, the first image data being captured or generated by a first image data capture device configured to generate three-dimensional image data; generating second edge information corresponding to the edges of the object from second image data representative of the object, the second image data being captured or generated by a second image data capture device distinct from the first image data capture device and configured to generate two-dimensional image data; fusing the first edge information and the second edge information to generate fused edge information corresponding to the edges of the object; and generating object detection hypotheses based on the fused edge information; A method according to claim 1, wherein the blended edge information represents an edge length that is longer than the edge length of the first edge information and / or the edge length of the second edge information and represents a possible gap between the object and the second object.

2. The method of claim 1 , wherein fusing the first edge information and the second edge information comprises compositing the first edge information and the second edge information into a shared image space.

3. Combining the first edge information and the second edge information into the shared image space, mapping the first edge information based on coordinates of detected edges associated with a first image capture mode by the first image data capture device that generates the three-dimensional image data; and The method of claim 2 , further comprising transforming the second edge information to correspond to an image space associated with the first image capture mode.

4. the image space associated with the first image capture mode is a three-dimensional image space; Transforming the second edge information to correspond to the image space associated with the first image capture mode; projecting the two-dimensional edge points from the second edge information into the three-dimensional image space based on depth information associated with the first edge information that corresponds to two-dimensional positions of the two-dimensional edge points of the second edge information to generate projected three-dimensional points associated with the second edge information; and The method of claim 3 , further comprising transforming the projected 3D points to correspond to the 3D image space.

5. Transforming the second edge information to correspond to the image space associated with the first image capture mode; The method of claim 4 , further comprising projecting the transformed three-dimensional points corresponding to the three-dimensional image space into two-dimensional coordinates of an image associated with the first edge information.

6. the second image data is image data captured by the second image data capture device in a two-dimensional image capture mode; Detecting the second edge information CANNY edge detection, Region Similarity Edge Detection, and The method of claim 1 , further comprising one or more of:

7. the first image data is image data captured by the first image data capture device in a three-dimensional image capture mode; Detecting the first edge information includes: projecting the first image data into an image space associated with one or more objects in the plurality of objects to be detected; and The method of claim 1 , comprising extracting object edges from the projected 3D point cloud.

8. A non-transitory computer readable medium having instructions embodied thereon that cause a computing device to perform the method for detecting objects according to any one of claims 1 to 7.

9. 1. An object detection system for detecting an object, comprising: a system interface configured to receive the first image data and the second image data; An object detection system comprising: a processor communicatively coupled to the system interface and configured to perform the method for detecting an object according to any one of claims 1 to 6.

Images