3D-2D Vision System for Robotic Carton Unloading

Abstract

Robotic carton loaders or unloaders combine 3D and 2D sensors to detect 3D point clouds and 2D images of carton stacks within transport vehicles such as truck trailers or shipping containers, respectively. Using the 3D point cloud, segments too small to be part of a product such as a carton are discarded to perform edge detection. Segments too large to correspond to a carton are processed by 2D image processing to detect additional edges. The results from 3D and 2D edge detection are transformed in the calibrated 3D space of the material carton loader or unloader to perform one of the loading or unloading of the transport vehicle. Image processing can also detect congestion in the product sequence from individual controllable sections of the conveyor of the robotic carton loader or unloader for individualized unloading.

Description

[0001] Priority and cross-reference of related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 410,435, filed October 20, 2016, entitled “3D – 2D Vision System for Robotic Carton Unloading”; U.S. Provisional Application No. 62 / 413,122, filed October 26, 2016, entitled “3D – 2D Vision System for Robotic Carton Unloading”; and U.S. Provisional Application No. 62 / 417,368, filed November 4, 2016, entitled “Conveyor Screening During Robotic Article Unloading”, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure generally relates to machine vision systems and more particularly to autonomous vehicles using machine vision to inspect objects in material handling systems. Background Technology

[0004] Trucks and trailers loaded with goods and products move across the country to deliver products to commercial loading and unloading docks at stores, warehouses, and distribution centers. Trucks may have trailers mounted on them or may have tractor-trailer configurations. To reduce overhead costs at retail stores, the number of products in stores has been reduced, and products in transit are now counted as part of available store inventory. Rapidly unloading trucks at warehouse and regional distribution center unloading docks has gained renewed popularity as a way to replenish depleted inventory.

[0005] If the load is palletized and involves manual labor, and if the products are stacked inside a truck, forklifts are typically used for loading and unloading the truck. Manually unloading large truckloads by human labor can be physically difficult and potentially expensive due to the time and labor involved. Furthermore, the hot or cold conditions within the confined space of a truck trailer or container truck can be considered unpleasant work. Therefore, there is a need for improved unloading systems that can unload large, bulk stacks of boxes and goods from truck trailers more quickly and at a lower cost than human labor.

[0006] For automation to be economical, loading or unloading needs to be relatively fast. Commonly known methods for rapidly unloading cartons have seen extremely limited acceptance. Bulk loading techniques have been proposed to unload trailers without knowing the exact location of the cartons within the trailer. For example, the entire trailer could be tilted toward the rear door to move the product. In another example, cartons are placed on a fabric layer pulled toward the rear door to dump their contents. In both instances, the integrity of the packaging and the contents of the cartons are compromised by applying bulk loading techniques. At the other extreme, articulated robotic arms with machine vision sensors on end effectors are known to scan the scene focusing on the area for the next pick-up or drop-off position. Extensive image processing and 3D point cloud processing have been employed in attempts to detect cartons present in a narrow section of a carton stack. The latency between operations makes it difficult to achieve a reasonable return on investment (ROI) for automated trailer loaders or unloaders. Even with significant detection efforts, individual cartons may fail to be detected due to the difficulty in sensing edges that are too tightly aligned for 3D detection or, conversely, too optically disguised for 2D detection. Summary of the Invention

[0007] In one aspect, this disclosure provides a method for determining the location of individual cartons in a material handling system. In one or more embodiments, the method includes receiving a two-dimensional (2D) image and a three-dimensional (3D) point cloud of at least a portion of a stack of cartons placed on the floor of a transport vehicle. The method includes detecting segments within the 3D point cloud. The method includes removing any segments smaller than a first threshold. The method includes determining whether any segment is smaller than a second threshold. The method includes, in response to determining that a selected segment is smaller than the second threshold, qualifying the selected segment as a 3D-detected carton. The method includes, in response to determining that a selected segment is not smaller than the second threshold, (i) determining a 2D mask corresponding to the selected segment; (ii) determining a portion of a 2D image corresponding to the 2D mask; (iii) detecting segments within that portion of the 2D image; and (iv) qualifying the detected segments as 2D-detected cartons. The method includes combining the 2D and 3D-detected cartons in the detection result. The method involves using calibration information to convert the detection result into a 3D position relative to the robotic carton loading and unloading system for a selected loading and unloading operation.

[0008] In another aspect, this disclosure provides a robotic carton loading and unloading system for unloading cartons from a carton stack. The robotic carton loading and unloading system is movable across a floor. In one or more embodiments, the robotic carton loading and unloading system includes a mobile body. The robotic carton loading and unloading system includes a movable robotic manipulator attached to the mobile body. The movable robotic manipulator includes an end effector at its end. The end effector unloads or loads one or more cartons from the carton stack. A conveyor mounted on the mobile body receives the one or more cartons from the end effector and moves the one or more cartons toward the rear of the robotic carton loading and unloading system. A carton detection system includes one or more sensors correspondingly coupled to one of the mobile body and the movable robotic manipulator. The one or more sensors provide 2D images and 3D point clouds of at least a portion of the carton stack disposed on the floor of a transport vehicle. A processing subsystem communicates with the one or more sensors. The processing subsystem detects segments within the 3D point cloud. The processing subsystem removes any segments smaller than a first threshold. The processing subsystem determines whether any segment is smaller than a second threshold. In response to determining that the selected segment is less than a second threshold, the processing subsystem defines the selected segment as a 3D-detected carton. In response to determining that the selected segment is not less than the second threshold, the processing subsystem (i) determines a 2D mask corresponding to the selected segment; (ii) determines a portion of a 2D image corresponding to the 2D mask; (iii) detects segments within that portion of the 2D image; and (iv) defines the detected segments as 2D-detected cartons. The processing subsystem combines the 2D and 3D-detected cartons in the detection result. For a selected loading or unloading operation, the processing subsystem uses calibration information to convert the detection result into a 3D position relative to the robotic carton loading / unloading system.

[0009] In an additional aspect, this disclosure provides a material handling system including a robotic carton handling system for unloading cartons from a carton stack. The robotic carton handling system is movable across a floor. In one or more embodiments, the robotic carton handling system includes a mobile body and a movable robotic manipulator attached to the mobile body. The movable robotic manipulator includes an end effector at its end. The end effector unloads one or more cartons from the carton stack. A conveyor mounted on the mobile body receives the one or more cartons from the end effector and moves the one or more cartons toward the rear of the robotic carton handling system. A carton detection system includes one or more sensors correspondingly coupled to one of the mobile body and the movable robotic manipulator. The one or more sensors provide 2D images and 3D point clouds of at least a portion of the carton stack disposed on the floor of the transport vehicle. A processing subsystem communicates with the one or more sensors. The processing subsystem detects segments within the 3D point cloud. The processing subsystem removes any segments smaller than a first threshold. The processing subsystem determines whether any segment is smaller than a second threshold. The processing subsystem, in response to determining that the selected segment is less than a second threshold, defines the selected segment as a 3D-detected carton. In response to determining that the selected segment is not less than the second threshold, the processing subsystem (i) determines a 2D mask corresponding to the selected segment; (ii) determines a portion of a 2D image corresponding to the 2D mask; (iii) detects segments within that portion of the 2D image; and (iv) defines the detected segments as 2D-detected cartons. The processing subsystem combines the 2D and 3D-detected cartons in the detection result. The processing subsystem uses calibration information to convert the detection result into a 3D position relative to the robotic carton loading / unloading system for a selected loading or unloading operation. An automation controller communicates with the processing subsystem. The automation controller causes the robotic carton manipulator to use the 3D position to perform a selected loading or unloading operation via the robotic carton loading / unloading system. The extendable conveyor system has a proximal end coupled to a fixed conveyor system. The extendable conveyor has a movable remote end positioned close to the robotic carton handling system to transfer cartons between the stationary conveyor and the robotic carton handling system.

[0010] The above summary contains simplifications, generalizations, and omissions of details and is not intended as a comprehensive description of the claimed subject matter, but rather as a brief overview of some of its associated functions. Other systems, methods, functions, features, and advantages of the claimed subject matter will be, or will become apparent to those skilled in the art upon examination of the figures and the detailed written description below. Attached Figure Description

[0011] The description of the illustrative embodiments can be read in conjunction with the accompanying drawings. It will be appreciated that, for simplicity and clarity, the elements illustrated in the figures are not necessarily drawn to scale. For example, some elements are enlarged relative to others. The figures presented herein illustrate and describe embodiments that incorporate the teachings of this disclosure, in which:

[0012] Figure 1 The illustration shows a functional block diagram of a robotic carton loading and unloading system according to one or more embodiments, and a side view of an extendable conveyor that unloads cartons from a carton stack container using a multi-quadrant and combined two-dimensional (2D) and three-dimensional (3D) vision system.

[0013] Figure 2 The illustration is based on one or more embodiments. Figure 1 Top isometric view of the robotic carton loading and unloading system;

[0014] Figure 3 The illustration is based on one or more embodiments. Figure 1 Bottom isometric view of the robotic carton loading and unloading system;

[0015] Figure 4 The illustration is based on one or more embodiments. Figure 1 Front view of the front part of the robotic carton loading and unloading system;

[0016] Figure 5 The illustration is based on one or more embodiments. Figure 1 Right side view of the front part of the robotic carton loading and unloading system;

[0017] Figure 6 The illustration is for use according to one or more embodiments. Figure 1 An exemplary computing environment for the onboard unloading controller of a robotic carton loading and unloading system;

[0018] Figure 7 The illustration is based on one or more embodiments. Figure 1 The vision system of the robotic carton loading and unloading system;

[0019] Figure 8 The illustration is a flowchart of a method for detecting cardboard boxes on a floor of a transport vehicle, according to one or more embodiments.

[0020] Figure 9 The illustration is a flowchart of an example method for 2D, 3D, and 3D-guided 2D box detection according to one or more embodiments;

[0021] Figure 10 The illustration shows a flowchart of another example method for 2D, 3D, and 3D-guided 2D box detection according to one or more embodiments;

[0022] Figure 11 The illustration is based on one or more embodiments. Figure 10 A flowchart of an example method for 3D processing executed as part of the method;

[0023] Figure 12 The illustration is based on one or more embodiments. Figure 10 A flowchart of an example method for 2D processing executed as part of the method; and

[0024] Figure 13 The figure shows a simplified flowchart of an example method for 2D, 3D, and 3D-guided 2D box detection according to one or more embodiments. Detailed Implementation

[0025] Robotic carton loaders or unloaders combine 3D and 2D sensors to detect 3D point clouds and 2D images of carton stacks within transport vehicles such as truck trailers or shipping containers, respectively. Cartons can be identified within a portion of the 3D point cloud by discarding segments too small to be part of a product such as a carton. Segments too large to correspond to a carton are processed by 2D image processing to detect additional edges. The results from 3D and 2D edge detection are transformed in the calibrated 3D space of the material carton loader or unloader to perform one of the loading or unloading of the transport vehicle.

[0026] In one aspect of this disclosure, a custom red-green-blue and depth (RGB-D) vision solution is provided for an autonomous truck unloader. An RGB-D sensor system is designed using a combination of industrial depth and RGB sensors; specifically tailored to the needs of the truck unloader. Four combined units of this type provide RGB, depth, and RGB-D data across the entire width and height of the trailer. Each of the RGB cameras has unique projection parameters. Using those, along with the relative positions of the depth and RGB sensors, 3D from the depth sensors is mapped onto 2D image data from the RGB sensors, and vice versa. The data from the 3D and 2D RGB sensors can be stitched together at a higher level to obtain the entire scene.

[0027] After product commissioning, the expected lifespan is long. If properly controlled, average ambient operating temperature, source voltage consistency, shock and vibration isolation, and isolation from high-power transmitters will extend the lifespan of the sensor and system. At the end of each component's lifespan, more false alarms and missed alarms are expected. Dead points can be monitored on the sensor until a minimum number of dead points are reached, at which point the sensor can be marked for replacement. Assuming original parts or compatible alternatives are available, the components are serviceable.

[0028] In the following detailed description of exemplary embodiments of this disclosure, specific exemplary embodiments in which the disclosure may be practiced are described in sufficient detail to enable those skilled in the art to implement the disclosed embodiments. For example, specific details such as specific method sequences, structures, elements, and connections have been set forth herein. However, it is to be understood that it is not necessary to implement the embodiments of this disclosure using the specific details provided. It is also to be understood that other embodiments may be utilized and logical, architectural, procedural, mechanical, electrical, and other changes may be made without departing from the general scope of this disclosure. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of this disclosure is defined by the appended claims and their equivalents.

[0029] References to "an embodiment," "an embodiment," "various embodiments," or "one or more embodiments" in the specification are intended to indicate that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of this disclosure. The appearance of such phrases in various places throughout the specification does not necessarily refer to the same embodiment, nor is it necessarily a separate or alternative embodiment mutually exclusive with other embodiments. Furthermore, various features that may be demonstrated by some embodiments but not by others are described. Similarly, various requirements that may be required for some embodiments but not for others are described.

[0030] It should be understood that the use of specific component, device, and / or parameter names and / or their corresponding acronyms (such as those performing the functions, logic, and / or firmware described herein) is for illustrative purposes only and is not intended to imply any limitation on the embodiments described. Therefore, different nomenclature and / or terms used to describe components, devices, parameters, methods, and / or functions herein, rather than as limitations, may be used to describe embodiments. References to any specific protocol or proper noun in the description of one or more elements, features, or concepts of an embodiment are provided merely as examples of an implementation, and such references in no way limit the extension of the claimed embodiments to embodiments in which different element, feature, protocol, or concept names are utilized. Therefore, each term used herein is to be given its broadest interpretation within the context in which it is used.

[0031] Figure 1The illustration shows a robotic carton handling system 100 with manipulators, such as a robotic arm assembly 102 that unloads cartons 104 from a carton stack 106 inside a carton stack container 108 (such as a trailer, shipping container, storage unit, etc.). The robotic arm assembly 102 places the cartons 104 onto a conveyor system 110 of the robotic carton handling system 100, which transports the cartons 104 back to an extendable conveyor 112 that follows the moving body 114 of the robotic carton handling system 100 into the carton stack container 108. The extendable conveyor 112 then transports the cartons 104 to a material handling system 116, such as in a warehouse, store, distribution center, etc.

[0032] In one or more embodiments, the robotic carton handling system 100 autonomously unloads a stack of cartons 106 disposed on the floor 118 of a carton stack container 108. A moving body 114 is self-propelled and can move across the floor 118 from the outside to the innermost part of the carton stack container 108. The right and left lower arms 120 of the robotic arm assembly 102 are pivotally attached at their lower ends 122 to the moving body 114 on opposite lateral sides of the conveyor system 110 passing between them. The right and left lower arms 120 rotate about a lower arm axis 124 perpendicular to the longitudinal axis 126 of the conveyor system 110. The upper arm assembly 128 of the robotic arm assembly 102 has a rear end 130 pivotally attached at the respective upper ends 132 of the right and left lower arms 120 to pivot about an upper arm axis 134 perpendicular to the longitudinal axis 126 of the conveyor system 110 and parallel to the lower arm axis 124. Manipulator head 136 is attached to the front end 138 of upper arm assembly 128 and engages at least one carton 104 for movement to conveyor system 110 as soon as the carton stack 106 is placed on floor 118. The pivoting and simultaneous mirroring of the right and left lower arms 120 maintains the upper arm axis 134 at a relative height above conveyor system 110, allowing the at least one carton 104 to be conveyed by conveyor system 110 without obstruction by robotic arm assembly 102 once manipulator head 136 is unloaded. In one or more embodiments, the robotic carton loading and unloading system 100 includes a lift 140 attached between mobile body 114 and front portion 142 of conveyor system 110. The lift 140 moves the front portion 142 of conveyor system 110 relative to floor 118 to reduce the gap under the at least one carton 104 during movement from carton stack 106 to conveyor system 110.

[0033] A higher-level system can assign an autonomous robotic vehicle controller 144 to a specific carton stack container 108 of the robotic carton loading and unloading system 100 and can receive information about the loading / unloading process and provide channels for remote control. Human operators can selectively intervene when errors occur during loading or unloading. This higher-level system may include a host system 146 that processes external order transactions to be implemented by the material handling system 116. Alternatively or additionally, a warehouse execution system (WES) 148 can provide vertical integration of order fulfillment, worker management of a warehouse management system (WMS) 150, and inventory tracking for facilities 152 such as distribution centers. WES 148 may include a vertically integrated warehouse control system (WCS) 154 that controls the automation of order fulfillment and inventory movement required by the WMS 150.

[0034] In one or more embodiments, once specified by WES 148 or manually enabled, the robotic carton handling system 100 can operate autonomously under the control of a robotic vehicle controller 154 in the following processes: (i) moving into a carton stack container 108, (ii) performing loading or unloading of one of the carton stack containers 108, and (iii) moving out of the carton stack container 108. To navigate within the carton stack container 108 and rapidly load and unload cartons 104 therein, the carton detection system 166 of the robotic vehicle controller 154 includes sensors 157 respectively attached to one of the mobile body 114 and the movable robotic manipulator (robotic arm assembly 102) to provide two-dimensional (2D) images and three-dimensional (3D) point clouds of at least a portion of the carton stack 106 disposed on the floor 159 of the carton stack container 108. The carton stack container 108 can be stationary or mobile, such as a transport vehicle for loading on roads, rail, or navigable waterways.

[0035] Controller 144 provides an exemplary environment in which one or more of the features described in the various embodiments of this disclosure may be implemented. Controller 144 may be implemented as a single device or a distributed processing system. The controller 144 includes functional components that communicate across a system interconnect (depicted for clarity as system bus 156) constructed of one or more conductors or optical fibers. System bus 156 may include a data bus, an address bus, and a control bus for communicating data, address, and control information between any of these coupled units. The functional components of controller 144 may include a processor subsystem 158 consisting of one or more central processing units (CPUs), digital signal processors / multiple digital signal processors (DSPs), and processor memory. Processor subsystem 158 may include any tool or set of tools operable for calculating, classifying, processing, transmitting, receiving, retrieving, creating, switching, storing, displaying, indicating, detecting, recording, reproducing, processing, or utilizing any form of information, intelligence, or data for commercial, scientific, control, or other purposes, including the control of automated equipment for material handling systems.

[0036] According to various aspects of this disclosure, an element or any part of an element or any combination of elements may be implemented using a processor subsystem 158 comprising one or more physical devices, including a processor. Non-limiting examples of processors include microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), programmable logic controllers (PLCs), state machines, gate logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described herein. One or more processors in a processing system can execute instructions. A processing system that executes instructions to produce a result is a processing system configured to perform a task that results in that result, such as by providing instructions to one or more components of a processing system that will cause these components to perform actions independently or in combination with other actions performed by other components of the processing system that will result in that result.

[0037] Controller 144 may include network interface (I / F) devices 160 that enable controller 144 to communicate or engage with other devices, services, and components located outside controller 144 (such as WES 148). These network devices, services, and components may engage with controller 144 via an external network (such as example network 162) using one or more communication protocols. Network 162 may be a local area network, wide area network, personal area network, and the like, and the connections to and / or between the network and controller 144 may be wired, wireless, or a combination thereof. For the purposes of discussion, network 162 is referred to as a single set of components for simplicity. However, it should be appreciated that network 162 may include one or more direct connections to other devices and more complex sets of interconnections, such as those existing within a wide area network (such as the Internet) or on a private intranet. For example, programming workstation 1924 may remotely modify the programming or parameter settings of controller 144 via network 162. The various links in network 162 may be wired or wireless. The controller 144 can communicate with various onboard devices (such as lights, indicators, manual controls, etc.) via the device interface 168. The device interface 168 may include wireless and wired links. For example, the controller 144 can guide the extendable conveyor 112 of the robot carton handling system 100 into or out of the carton stack container 108.

[0038] The controller 144 may include several distributed subsystems that manage specific functions of the robotic carton loading system 100. The automation controller 170 may receive position and spatial calibration information from the 3D / 2D carton detection system 166 and use the data to coordinate the movement of the mobile body 114 via the vehicle interface 172 and via the movement of payload components such as the robot arm assembly 102 and the lift 140 of the front portion 142 of the mobile conveyor system 110.

[0039] The 3D / 2D carton inspection system 166 may include depth sensing using binocular, lidar, radar, or sonar principles. To avoid dependence on consistent ambient lighting conditions, illuminator 169 may provide a consistent or adjustable amount of illumination across one or more spectral bandwidths, such as visible or infrared. This illumination may be narrowly defined within the visible spectrum, which filters out most ambient light. Alternatively, the illumination may be outside the visible range so that it does not distract the human operator. The 3D / 2D carton inspection system 166 may receive 2D and 3D sensor data from RGB-D sensors 176 that view the interior of the carton stack container 108 and the front of the carton stack 106. For these and other purposes, the 3D / 2D carton inspection system 166 may include various applications or components that perform the processes described later in this application. For example, the 3D / 2D carton inspection system 166 may include a 2D process module 180, a 3D process module 182, and a 3D-guided 2D process module 184.

[0040] System memory 164 can be used by processor subsystem 158 to house functional components (such as data) and software (such as 3D / 2D carton inspection system 166). Software can be broadly interpreted as instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, application programs, software applications, software packages, routines, subroutines, objects, executable files, execution threads, programs, functions, etc., regardless of whether it is referred to as software, firmware, middleware, microcode, hardware description language, function block diagram (FBD), ladder diagram (LD), structured text (ST), instruction list (IL), and sequential function chart (SFC) or others. The software may reside on a computer-readable medium.

[0041] For clarity, system memory 164 may include random access memory, which may or may not be a volatile or non-volatile data storage device. System memory 164 includes one or more types of computer-readable media, which may be non-transitory or transient. By way of example, computer-readable media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical disks (e.g., compact discs (CDs), digital versatile discs (DVDs)), smart cards, flash memory devices (e.g., cards, sticks, key drives), random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, removable disks, and any other suitable media for storing software and / or instructions that can be accessed and read by a computer. Computer-readable media may be present in the processing system, outside the processing system, or distributed across multiple entities including the processing system. Computer-readable media may be embodied in a computer program product. By way of example, a computer program product may include computer-readable media in packaging material. Those skilled in the art will recognize how best to implement the functions described herein, depending on the specific application and the overall design constraints imposed on the system as a whole.

[0042] Figure 2 The illustrated robotic carton loading and unloading system 100 includes an upper arm assembly 128 comprising a rotatable platform 201 pivotally attached at the rear ends 130 of left and right lower arms 120 at an upper arm axis 134. The rotatable platform 201 has a lateral guide 203 at its extended end 205. The upper arm assembly 128 includes an end arm 207 proximally attached to the lateral guide 203 for lateral movement and distally attached to a manipulator head 136. This end arm 207 translates laterally to achieve an increased lateral range. This allows for a lighter and more maneuverable manipulator head 136. Figure 2-5 The illustrated equipment cabinet 209 is arched above the rear portion of the conveyor system 110. (Special Reference) Figure 5The clearance below equipment cabinet 209 defines a jam height 210 for any cartons housed on the front portion 142 of conveyor system 110, which can be determined based on sensor data from a rear 3D / 2D sensor 178 mounted on equipment cabinet 209. For example, the rear 3D / 2D sensor 178 may include a 2D infrared sensor 211, a 3D depth sensor 213, and a 2D optical sensor 215. The front 3D / 2D sensor 176 may include spatially separated sensors operating in different spectra and sizes to detect objects such as products, cartons, boxes, crates, handbags, etc. (carton 104) under various stacking arrangements, lighting conditions, etc. Mounting the sensors on end effectors (manipulator heads 136) also allows for changes in vantage points, such as looking down over the carton stack 106 to better distinguish the topmost carton 104.

[0043] Special Reference Figure 2 and Figure 4 In one exemplary embodiment, the front 3D / 2D sensor 176 includes a top left 2D sensor 217, a top left 3D sensor 219, a top right 2D sensor 221, and a top right 3D sensor 223 on the manipulator head 136. The front 3D / 2D sensor 176 also includes a bottom left 2D sensor 227, a bottom left 3D sensor 229, a bottom right 2D sensor 231, and a bottom right 3D sensor 233 on the front end of the moving body 114.

[0044] Figure 6The illustrations include exemplary components of a material handling system 600 suitable for use in various embodiments of a robotic carton handling system 601. The robotic carton handling system 601 may include an external monitor 602, a network interface module 604, an HMI module 606, an input / output module (I / O module 608), a robotic arm and conveyor system 615 including a driver / safety module 612 and a motion module 614, a programmable logic controller (or PLC 618), a basic motion module 620 including a vehicle controller module 622 and a manual control module 624, and a vision system 626 (or visualization system), which may include one or more personal computing devices 628 (or “PCs”) and sensor devices 630. In some embodiments, the vision system 626 of the robotic carton handling system 601 may include a PC 628 connected to each sensor device 630. In embodiments where more than one sensor device 630 exists on the robotic carton handling system 601, the PCs 628 for each sensor device 630 can be networked together, and one of the PCs 628 is operable as a master PC 628 that receives data from the other connected PCs 628, can perform data processing (e.g., coordinate transformation, duplicate elimination, error checking, etc.) on the received data as well as its own data, and can output the combined and processed data from all PCs 628 to a PLC 618. In some embodiments, the network interface module 604 may not have its own PLC inline with the PCs 628, and the PLC 618 may be used as a vehicle controller and / or drive / safety system. The sensor devices 630 may include 2D image capture devices (ICDs) 631 and 3D image capture devices (ICDs) 633 separated into sectors for different viewing portions or advantageous positions. Subsets may include rear-mounted sensors 635, end effector-mounted sensors 637, and vehicle-mounted sensors 639.

[0045] The robotic carton handling system 601 can be connected via a network 603 (such as a local Wi-Fi™ network) to a remote location or a system with a network interface module 604 (such as a Wi-Fi™ radio, etc.). Specifically, the network interface module 604 enables the robotic carton handling system 601 to connect to an external monitor 602. This external monitor 602 can be any of a remote warehouse or distribution center control room, a handheld controller, or a computer, and can provide passive remote viewing via the vision system 626 of the robotic carton handling system 601. Alternatively, the external monitor 602 can overwrite the programming inherent in the vision system 626 and assume active command and control of the robotic carton handling system 601. The programming of the robotic carton handling system 601 can also be communicated, operated, and debugged via an external system (such as the external monitor 602). Examples of the external monitor 602 assuming command and control include a remotely located human operator or a remote system, such as a warehouse or distribution server system (i.e., a remote device as described above). Exemplary embodiments of using an external monitor 602 to command and control the robotic carton loading and unloading system 601 may include human or computer intervention in the process of moving the robotic carton loading and unloading system 601 (e.g., from one unloading bay to another) or having the external monitor 602 take over the control of the robotic arm to remove items (e.g., boxes, cartons, etc.) that are difficult to unload using autonomous routines. The external monitor 602 may include any of the following: a visual monitor, a keyboard, a joystick, an I / O port, a CD reader, a computer, a server, a handheld programming device, or any other device that can be used to perform any part of the above embodiments.

[0046] The robotic carton loading and unloading system 601 may include a human-machine interface module 606 (or HMI module 606), which can be used to control and / or receive output information for the robotic arm and conveyor system 615 and / or the basic motion module 620. The HMI module 606 may be used to control (or may itself include) a joystick, display, and keypad, which can be used to reprogram and rewrite the machine's autonomous control and drive the robotic carton loading and unloading system 601 point-by-point. Actuators 609 may be actuated individually or in any combination via an I / O module 608 by a vision system 626, and a distance sensor 610 may be used to assist in guiding the robotic carton loading and unloading system 601 into an unloading area (e.g., a trailer). The I / O module 608 can connect actuators 609 and distance sensor 610 to a PLC 618. The robotic arm and conveyor system 615 may include all components required for the mobile arm and / or conveyor, such as drives / engines and motion protocols or control devices. The basic motion module 620 can be a component of the overall mobile robot carton loading and unloading system 601. In other words, the basic motion module 620 can be a component required to drive the vehicle into and out of the unloading area.

[0047] PLC 618 can control the overall electromechanical movement of the robotic carton loading and unloading system 601 or control exemplary functions, such as controlling the robotic arm or conveyor system 615. For example, PLC 618 can move the manipulator head of the robotic arm to a position for obtaining an item from the wall of an item (e.g., a box, carton, etc.). As another example, PLC 618 can control the activation, speed, and direction of the rotation of the backflush roller, and / or be configured to move the front-end conveyor (such as the front portion 142 of conveyor system 110). Figure 1 Various adjustments to the support mechanism of the PLC 618 and vision system 626. Other electronic components of the PLC 618 and vision system 626 may be installed in an electronic box (not shown) located below the conveyor, adjacent to the conveyor, or elsewhere on the robotic carton loading and unloading system 601. The PLC 618 can autonomously operate all of the robotic carton loading and unloading system 601 or a portion thereof and can receive position information from a distance sensor (not shown). The I / O module 608 can connect actuators and distance sensors 610 to the PLC 618.

[0048] The robotic carton loading and unloading system 601 may include a vision system 626, which includes sensor devices 630 (e.g., cameras, 3D sensors, etc.) and one or more computing devices 628 (referred to as personal computers or "PCs") 628. The robotic carton loading and unloading system 601 can use the sensor devices 630 and the one or more PCs 628 of the vision system 626 to scan the front of the robotic carton loading and unloading system 601 in real-time or near real-time. A forward scan may be triggered by a PLC 618 in response to determining that the robotic carton loading and unloading system 601 is in a position to begin detecting cartons in the unloading area. This forward scanning capability can be used for collision avoidance, shape recognition (safety) sent to humans, determining the dimensions of the unloading area (e.g., trucks or trailers), and scanning the floor of the unloading area to release items (e.g., cartons, boxes, etc.). The 3D capabilities of the vision system 626 may also provide depth perception, edge recognition, and the ability to create 3D images of the walls of the items (or stacks of cartons). The vision system 626 can operate alone or in conjunction with the PLC 618 to identify the edges, shapes, and near / far distances of objects in front of the robotic carton handling system 601. For example, the edges and distances of each individual carton within the walls of the item can be measured and calculated relative to the robotic carton handling system 601, and the vision system 626 can operate alone or in conjunction with the PLC 618 to select the specific carton for removal.

[0049] In some embodiments, the vision system 626 may provide the PLC with information such as the specific XYZ coordinates of a carton identified as a target for removal from the unloading area, and one or more movement paths for the robotic arm or moving body of the robotic carton loading and unloading system 601. The PLC 618 and vision system 626 may work independently or in conjunction, such as in an iterative movement and visual inspection process for carton visualization, initial positioning, and motion accuracy checks. The same process may be used during vehicle movement or during carton removal as an accuracy check. Alternatively, the PLC 618 may use the movement and visualization process as a check to see if one or more cartons have fallen from the carton stack or been repositioned since the last visual inspection. Although already described separately... Figure 6 Various computing devices and / or processors (such as PLC618, vehicle controller module 622, and PC 628) are included, but regarding... Figure 6 In the various embodiments discussed or all other embodiments described herein, the computing devices and / or processors described may be combined, and the operations performed by individual technical devices and / or processors as described herein may be performed by fewer computing devices and / or processors (such as a single computing device or processor having different modules that perform the operations described herein). As an example, different processors combined on a single circuit board may perform the operations described herein that are attributed to different computing devices and / or processors, a single processor running multiple threads / modules may perform the operations described herein that are attributed to different computing devices and / or processors, and so on.

[0050] The extendable conveyor system 632 can transport objects from the robotic carton handling system 601 to other parts of the material handling system 600. As the robotic carton handling system 601 moves forward or backward, a vision device 634 on one of the extendable conveyor system 632 and the robotic carton handling system 601 can image a target 636 on the other. The vision system 626 can perform image processing to detect changes in the size, orientation, and position of the target 636 within the field of view of the vision device 636. Corresponding device interfaces 638, 640 of the extendable conveyor system 632 and the robotic carton handling system 601 can transmit visual information or movement commands. For example, a PLC 618 can direct the extension motion actuator 642 on the extendable conveyor system 632 to correspond to the movement of the robotic carton handling system 601 to keep the extendable conveyor system 632 and the robotic carton handling system 601 aligned and properly spaced. In one embodiment, device interface 638 / 640 utilizes a short-range wireless communication protocol, such as a Personal Access Network (PAN) protocol. Examples of PAN protocols that can be used in various embodiments include Bluetooth®, IEEE 802.15.4, and Zigbee® wireless communication protocols and standards.

[0051] Figure 7 The illustrated robot carton loading and unloading system 100 ( Figure 1 The data flow within the example vision system 700 is shown. The end effector 702 includes a first RGD-D unit 704 and a second RGD-D unit 706. The vehicle 708 includes a rear IR-RGB-D unit 710 positioned for conveyor sorting and including an RGB unit 712, a 3D unit 714, and an IR unit 716. The vehicle 708 includes a third RGB-D unit 718 and a fourth RGB-D unit 720, each having an RGB sensor 722 and a depth sensor 724. The first PC 726 is connected to the first and second RGD-D units 704 and 706, and to the robotic carton loading and unloading system 100. Figure 1 The first, second, third, and fourth PCs 726, 730, 732, and 734 communicate with the PLC 728 for automated control. The second PC 730 communicates with the third RGB-D unit 718 and the PLC 728. The third PC 732 communicates with the fourth RGB-D unit 720 and the PLC 728. Each of these PCs includes a 2D process module 736 and a 3D process module 738. The PLC 728 sends a trigger signal 740 to each of the first, second, third, and fourth PCs 726, 730, 732, and 734. Each of the first, second, third, and fourth PCs 726, 730, 732, and 734 then sends a trigger signal 742 for data to the corresponding designated first, second, third, and fourth RGB-D units 704, 706, 718, and 720, as well as the rear IR-RGB-D unit 710. The first, second, third, and fourth RGB-D units 704, 706, 718, and 720 respond with RGB-D data 744. The rear IR-RGB-D unit 710 responds with IR-RGB-D data 746. The first, second, and third PCs 726, 730, and 732 analyze the RGB-D data 744 and provide a Cartesian 3D array 748 for the box position of the designated sector or quadrant. The fourth PC 734 analyzes the IR-RGB-D data 746 and generates band presence and height data 752. The PLC 728 can merge this data, or one of the first, second, third, and fourth PCs 726, 730, 732, and 734 can perform this role for the PLC 728.

[0052] Figure 8The illustration depicts a method 800 for determining the position of individual cartons in a material handling system. In one or more embodiments, method 800 includes receiving 2D images and 3D point clouds from one or more sensors positioned on a robotic carton handling system to detect a portion of a stack of cartons placed on the floor of a transport vehicle (block 802). Method 800 includes receiving 2D images and 3D point clouds from additional one or more sensors positioned on the robotic carton handling system to detect adjacent portions of the carton stack (block 804). Method 800 includes detecting segments within the 3D point cloud (block 806). Method 800 includes removing any segments smaller than a first threshold (block 808). Method 800 includes determining whether any segment is smaller than a second threshold greater than the first threshold (determination block 810). In response to determining in determination block 810 that a selected segment is smaller than the second threshold, the selected segment is defined as a 3D-detected carton (block 812). Additional rules different from the expected size of the carton may be imposed, such as requiring edges to be approximately vertical or horizontal.

[0053] In response to determining in decision block 810 that the selected segment is not less than a second threshold, method 800 includes:

[0054] (i) Determine the 2D mask (block 814) corresponding to the selected segment;

[0055] (ii) Determine a portion of the 2D image corresponding to the 2D mask (block 816);

[0056] (iii) Detecting segments (block 818) within this portion of the 2D image; and

[0057] (iv) The selected segment is defined as a 2D-detected carton (block 820) by at least partially discarding the edges of the smaller rectangle that are fully contained within the larger rectangle.

[0058] Following the 3D detection in block 812 and any 3D-guided 2D detection in block 820, method 800 includes combining the 2D and 3D detected cartons for each part of the scene in the detection result (block 822). Method 800 includes using calibration information to convert the detection result into a 3D position relative to the robotic carton loading / unloading system for a selected loading and unloading operation (block 824). Method 800 includes combining the 2D and 3D detected cartons from both a part and adjacent parts of the scene to form the detection result (block 826). Method 800 includes using the 3D position to perform a selected loading and unloading operation by the robotic carton loading / unloading system (block 828).

[0059] Figure 9The illustration depicts an example method 900 for 2D, 3D, and 3D-guided 2D box detection. According to one or more embodiments, method 900 includes acquiring optical red-green-blue and depth (RGB-D) sensor data (block 902). Method 900 includes a 2D box detection subprocess 904, a 3D box detection subprocess 906, and a 3D-guided 2D box detection subprocess 908 that analyzes the RGB-D sensor data. Each type of detection, either individually or in combination, can locate edges of cartons, boxes, objects, etc., and determine the location of a robotic carton loading and unloading system within a carton stacking carrier such as a truck trailer or a shipping container.

[0060] Beginning with 2D box detection subprocess 904, method 900 includes preparing 2D RGB input based on RGB-D sensor data (block 910). Method 900 includes performing 2D edge detection to create an edge map (block 912). Method 900 includes detecting the contours of the 2D edge map (block 914). Method 900 includes filtering out contours smaller than a threshold (block 916). For example, a carton may have printing or markings that are generally considered rectangular. Method 900 includes compileing the total box detection from the scene derived from 2D processing into a 2D result (block 918). The output from 2D box detection subprocess 904 is 2D box detection data structure 920.

[0061] Continuing with the 3D box detection subprocess 906, method 900 includes preparing a 3D point cloud (block 922). Method 900 includes preprocessing the 3D point cloud for noise removal (block 924). For example, the resolution level of the 3D point cloud can be reduced to perform spatial filtering and smoothing. Method 900 includes extracting 3D segments for 3D box detection using a cluster-based segmentation method (block 926). Method 900 includes determining whether each cluster is defined as a box (decision block 928). For example, the segments can be filtered to form rectangles within a certain size range and a certain aspect ratio range. In response to determining in decision block 928 that some clusters are indeed defined as boxes, method 900 includes editing the total box detection from the 3D point cloud derived from 3D processing into a 3D result (block 930). The 3D box detection subprocess 909 outputs at least partially the 3D box detection data structure 932. In response to determining in decision block 928 that a cluster is not limited to a single bin, method 900 includes creating a 2D mask region from the undefined cluster / multiple clusters (block 934). In this case, the 3D bin detection subprocess 906 at least partially outputs the 2D mask region data structure 936.

[0062] The 3D-guided 2D box detection subprocess 908 receives a 2D box detection data structure 920 from the 2D box detection subprocess 904 and mask region data 932 from the 2D box detection subprocess 906. Method 900 includes filtering the 2D box detection data structure 920 using the 2D mask region data structure 932 (block 938). Method 900 includes converting the filtered 2D box coordinates into 3D coordinates (block 940). The 3D-guided 2D box detection subprocess 908 outputs a data structure 942 containing the 3D boxes found using 2D box detection.

[0063] The full-scene box detection subprocess 944 receives a 3D box detection data structure 932 and a data structure 942 containing 3D boxes found using 2D box detection. Method 900 includes transforming the 3D box coordinates using calibration data (block 946). In one or more embodiments, distributed processing is performed to accelerate box detection, increase the resolution and size of the imaged scene, and improve detection accuracy. The 3D-guided 2D box detection subprocess 908 may combine results from each sector or quadrant of the scene. To this end, method 900 includes combining the transformed 3D box coordinates from each quadrant (block 948).

[0064] Figure 10 The illustration shows another example method 1000 for 2D, 3D, and 3D-guided 2D box detection. In one or more embodiments, method 1000 includes receiving RGB-D sensor data by a box detection system (block 1002). Method 1000 includes performing a 2DRGB process to generate a final segment list (block 1004), such as in... Figure 12 As described in more detail below. See also: Figure 10Method 1000 includes generating 3D point cloud data (block 1006). Method 1000 includes downsampling the 3D point cloud data for noise removal (block 1008). Method 1000 includes segmenting the downsampled 3D point cloud data (block 1010). Method 1000 includes determining whether another segment to be filtered exists (decision block 1012). In response to determining in decision block 1012 that another segment to be filtered exists, method 1000 includes determining whether the selected segment has a size larger than a segmentation threshold (decision block 1014). In response to determining in decision block 1014 that the selected segment does not have a size larger than a segmentation threshold, method 1000 includes discarding smaller segments (block 1016). Method 1000 then returns to decision block 1012. In response to determining in decision block 1014 that the selected segment does not have a size larger than a segmentation threshold, method 1000 returns to decision block 1012. In response to determining in decision block 1012 that another segment to be filtered does not exist, method 1000 includes filtering out large segments for 2D processing, leaving a filtered 3D result and the large segment result (block 1018). Method 1000 includes resolving any large segment result into a mask (block 1020). Method 1000 includes performing a 2D RGB process on the mask (block 1022). Method 1000 includes editing the final 2D segment list from the RGB image and the mask for each sector or quadrant of the detection system into a final 2D result (block 1024). Method 1000 includes editing the filtered 3D result from each sector or quadrant of the detection system into a final 3D result (block 1026). Method 1000 includes combining the final 3D and 2D results into a final result (block 1028). Method 1000 includes passing the final result to an automation controller for carton loading or unloading operations (block 1030).

[0065] Figure 11 The diagram illustrates method 1000. Figure 10This is an example method 1100 for 3D processing, performed as part of a larger implementation. In one or more embodiments, method 1100 includes receiving 3D data (block 1102). Method 1100 includes removing invalid points (block 1104). Method 1100 includes receiving tool rotation (θ) (block 1106). Method 1100 includes rotating the cloud back (-θ) (block 1108). Method 1100 includes trimming data according to quadrants from a 3D sensor (block 1110). Method 1100 includes performing statistical outlier removal to remove particles from the scene (block 1112). Method 1100 includes performing polynomial smoothing on the 3D scene (block 1114). Method 1100 includes removing invalid 3D points (block 1116). Method 1100 includes performing region growing segmentation (block 1118). Method 1100 includes analyzing clusters of 3D data from the region growing segmentation (block 1120). Method 1100 includes determining whether both the cluster length and width are greater than a cluster threshold (decision block 1122). In response to determining in decision block 1122 that both the cluster length and width are not greater than the cluster threshold, method 1100 includes fitting a plane to the cluster (block 1124). Method 1100 includes determining the position of the plane (block 1126).

[0066] In response to determining in decision block 1122 that both the cluster length and width are greater than a cluster threshold, method 1100 includes rotating the cloud back (+θ) (block 1128). Method 1100 includes applying external calibration to find the cluster's rotation and translation.<R,T> Method 1100 includes applying intrinsic calibration to the cluster (block 1132). Method 1100 includes drawing a 2D mask (block 1134). Method 1100 includes obtaining the bin position from the RGB module for use in the 2D mask (block 1136). Method 1100 includes applying extrinsic and intrinsic calibration to obtain 3D coordinates from the RGB pixels of the bin position (block 1138). Method 1100 includes slicing the corresponding region using the pick point and size from the RGB module (block 1140). Method 1100 includes obtaining the lower right corner from the 3D cluster (block 1142). After determining the position of the plane in block 1126 or after obtaining the lower right corner from the 3D cluster in block 1142, method 1100 includes adding the lower right corner coordinates to the pick point stack (block 1144). Method 1100 includes (block 1146).

[0067] Figure 12 The diagram illustrates method 1000. Figure 10This is an example method 1200 for 2D processing, performed as part of an edge detection module 1202. Starting with edge detection module 1202, method 1200 includes receiving a 2D RGB image (block 1204). Method 1200 includes performing adaptive thresholding of the red layer of the RGB image (block 1206). Method 1200 includes performing adaptive thresholding of the green layer of the RGB image (block 1208). Method 1200 includes performing adaptive thresholding of the blue layer of the RGB image (block 1210). Method 1200 includes generating a first common mask using an OR operation (block 1212). Method 1200 includes performing Gaussian difference (DOG) analysis of the blue layer of the RGB image (block 1214). Method 1200 includes generating a second common mask from the first common mask and the DOG mask using an OR operation (block 1216). Method 1200 includes performing edge detection using a graphical model (block 1218). Method 1200 includes generating a third common mask using operations from a second common mask and an edge detection mask (block 1220). Then, in the blob detection and small segment filtering module 1222, method 1200 includes completing the edge map (block 1224). Method 1200 includes performing blob detection (block 1226). Method 1200 includes determining whether the blob size is greater than a blob threshold (decision block 1228). In response to determining in decision block 1228 that the blob size is not greater than the blob threshold, method 1200 includes discarding small blob fragments (block 1230). In response to determining in decision block 1228 that the blob size is greater than the blob threshold, method 1200 includes filtering small bins within large bins (block 1232). Method 1200 includes generating a final bin list (block 1234).

[0068] Figure 13This diagram illustrates a method 1300 that combines the results of 3D and 2D box detection analysis. A box detector acquires 3D data (block 1302). The box detector reads an angle from the sensor angle specified by its position, and the box detector counter rotates the 3D data to a normalized position (block 1304). Method 1300 includes the detector performing 3D processing to identify boxes within the 3D data (block 1306). The result of detection in block 1306 is any box identified from the 3D processing (block 1308). The completion of block 1306 can be used as a trigger for the next iteration of the 3D data (block 1310). After block 1308, the box detector extracts a clump larger than the box from the 3D data requiring further 2D processing to detect the box (block 1312). The clump of the 3D data is rotated and transformed from the 3D frame to an RGB frame (block 1314). Method 1300 then includes acquiring RGB data (block 1316). The RGD data source can be directly from a 2D sensor or it can be rotated and transformed 3D data. The detector performs 2D processing to identify the bin (block 1318). The completion of block 1318 can be used as a trigger for the next iteration of the 2D data (block 1320). Method 1300 then includes rotating and transforming the identified 2D bin to a 3D frame (block 1322). The result of the detection in block 1322 can be any bin identified from the 3D-enhanced 2D processing (block 1324). Alternatively or additionally, the result of the detection in block 1322 can be any bin identified from the 2D processing (block 1326). The identified bins 1308, 1324, and 1326 are combined (block 1328). Method 1300 includes conveying the combined identified bins to the vehicle / tool ​​controller 1330 to perform one of the loading or unloading processes using the known bin positions (block 1330). Method 1300 then ends.

[0069] As used herein, a processor can be any programmable microprocessor, microcomputer, or multiprocessor chip (or multiple chips) that can be configured by software instructions (applications) to perform a wide variety of functions (including those described in the various embodiments above). In various devices, multiple processors may be provided, such as one dedicated to wireless communication functions and another dedicated to running other applications. Typically, software applications may be stored in internal memory before they are accessed and loaded into the processor. The processor may include sufficient internal memory to store the application software instructions. In many devices, this internal memory may be volatile or non-volatile memory (such as flash memory) or a combination of both. For the purposes of this description, the general reference to memory refers to memory accessible by the processor, including internal or removable memory inserted in various devices and memory within the processor.

[0070] The foregoing method descriptions and process flowcharts are provided merely as illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the given order. As will be appreciated by those skilled in the art, the steps in the foregoing embodiments can be performed in any order. Words such as “afterward,” “then,” “next,” etc., are not intended to limit the order of steps; these words are merely used to guide the reader through the description of the method. Furthermore, references to singular claim elements, such as those using the articles “a,” “an,” or “the,” should not be construed as limiting that element to the singular.

[0071] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the system as a whole. A skilled technician may implement the described functionality in different ways for each specific application; however, such implementation decisions should not be construed as causing a deviation from the scope of the invention.

[0072] The hardware used to implement the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor), multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Alternatively, certain steps or methods may be performed by circuitry specific to a given function.

[0073] In one or more exemplary embodiments, the functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored on or transmitted as one or more instructions or code on a non-transitory processor-readable, computer-readable, or server-readable medium or a non-transitory processor-readable storage medium. The steps of the methods or algorithms disclosed herein may be embodied as processor-executable software modules or processor-executable software instructions that may reside on a non-transitory computer-readable storage medium, a non-transitory server-readable storage medium, and / or a non-transitory processor-readable storage medium. In various embodiments, such instructions may be stored processor-executable instructions or stored processor-executable software instructions. A tangible non-transitory computer-readable storage medium may be any available medium accessible to a computer. By way of example, and without limitation, such a non-transitory computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and is accessible to a computer. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital universal discs (DVDs), floppy disks, and Blu-ray™ discs. Here, disks typically copy data magnetically, while optical discs utilize lasers to copy data optically. The above combinations should also be included within the scope of non-transitory computer-readable media. Furthermore, the operation of a method or algorithm may reside as one or any combination or set of code and / or instructions on tangible non-transitory processor-readable storage media and / or computer-readable media that can be incorporated into a computer program product.

[0074] Although this disclosure has been described with reference to exemplary embodiments, those skilled in the art will recognize that various changes can be made and equivalents can be substituted for elements thereunder without departing from the scope of this disclosure. Furthermore, many modifications can be made to adapt a particular system, device, or component thereof to the teachings of this disclosure without departing from the essential scope of the teachings of this disclosure. Therefore, it is intended that this disclosure be limited to the specific embodiments disclosed for implementing this disclosure, but this disclosure will include all embodiments falling within the scope of the appended claims. Moreover, the use of the terms first, second, etc., does not indicate any order or importance, but rather that the terms first, second, etc., are used to distinguish one element from another.

[0075] For clarity, the robotic carton loading and unloading system 100 ( Figure 1This is described herein as unloading cartons, which can be corrugated boxes, wooden crates, polymer or resin suitcases, storage containers, etc. The manipulator head can further engage as an object of a product or a single product that is shrink-wrapped together. In one or more embodiments, aspects of this invention can be extended to other types of manipulator heads particularly suitable for certain types of containers or products. The manipulator head can employ mechanical gripping devices, electrostatic adhesive surfaces, electromagnetic attraction, etc. Aspects of this invention can also be employed on a single conventional articulated arm.

[0076] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the scope of this disclosure. As described herein, the singular forms “a,” “an,” or “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0077] The description of this disclosure has been given for illustrative and descriptive purposes, but is not intended to be exhaustive or to limit the disclosure in its entirety. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of this disclosure. The described embodiments were chosen and described in order to best explain the principles and practical application of this disclosure, and to enable others skilled in the art to understand this disclosure for use in various embodiments with various modifications as suited to the particular intended use.

Claims

1. A method of determining the location of individual cartons in a handling system, the method comprising: receiving, by a carton detection system positioned on a robotic carton handling system, a two-dimensional (2D) image and a three-dimensional (3D) point cloud of at least a portion of a carton stack; detecting, by a processing subsystem in conjunction with the carton detection system, a set of segments within the 3D point cloud; detecting, by the processing subsystem in conjunction with the carton detection system, another set of segments within the 3D point cloud; defining, by the processing subsystem, the set of segments as 3D detected cartons and the other set of segments as 2D detected cartons; merging, by the processing subsystem, the 2D and 3D detected cartons into a detection result; and converting, by the processing subsystem, the detection result into a 3D location of a target carton for removal using calibration information; wherein the set of segments is defined as 3D detected cartons if the set of segments is less than a segment threshold; and wherein the other set of segments is defined as 2D detected cartons if the other set of segments is greater than the segment threshold, wherein when the other set of segments is greater than the segment threshold: determining a 2D mask corresponding to the other set of segments; determining a portion of the 2D image corresponding to the 2D mask; detecting the other set of segments within the portion of the 2D image; and defining the detected other set of segments as 2D detected cartons.

2. The method of claim 1, further comprising: receiving the 2D image and the 3D point cloud from one or more sensors to detect at least one portion of a carton stack; receiving the 2D image and the 3D point cloud from another one or more sensors to detect an adjoining portion of the carton stack; and combining the 2D and 3D detected cartons from both the one portion and the adjoining portion to form the detection result.

3. The method of claim 1, wherein defining the other set of segments as 2D detected cartons further comprises discarding, by the processing subsystem, edges that form smaller rectangles that are completely contained within a larger rectangle.

4. The method of claim 1, further comprising stitching together the 2D image and the 3D point cloud to obtain an image of the carton stack.

5. The method of claim 1, further comprising: using the 3D location by the robotic carton handling system to perform one of a load operation and an unload operation.

6. A carton detection system to facilitate unloading cartons in a carton stack by a robotic carton handling system: a sensor configured to provide a two-dimensional (2D) image and a three- dimensional (3D) point cloud of at least a portion of a carton stack disposed on a floor of a transport carrier; a processing subsystem in communication with the sensor, the processing subsystem: detecting a set of segments within the 3D point cloud; detecting another set of segments within the 3D point cloud; defining the set of segments as 3D detected cartons and the other set of segments as 2D detected cartons; merging the 2D and 3D detected cartons into a detection result; and converting the detection result into a 3D location of a target carton for removal using calibration information to enable the robotic carton handling system to remove the carton from the carton stack; wherein the set of segments is defined as 3D detected cartons if the processing subsystem determines that the set of segments is less than a segment threshold; and wherein the other set of segments is defined as 2D detected cartons if the processing subsystem determines that the other set of segments is greater than the segment threshold. ​ wherein if the processing subsystem determines that the other set of segments is greater than a segment threshold, the other set of segments is defined as a 2D detected carton, wherein in response to determining that the other set of segments is greater than a segment threshold, the processing subsystem: determines that a 2D mask corresponds to the other set of segments; determines a portion of a 2D image corresponding to the 2D mask; detects the other set of segments within the portion of the 2D image; and defines the detected other set of segments as a 2D detected carton.

7. The carton detection system of claim 6, wherein: the sensor is positioned on a robotic carton handling system to detect at least one portion of a carton stack; the robotic carton handling system further comprises a further sensor positioned on the robotic carton handling system to detect an adjoining portion of the carton stack; and the processing subsystem combines 2D and 3D detected cartons from both the at least one portion and the adjoining portion to form a detection result.

8. The carton detection system of claim 6, wherein the processing subsystem defines the other set of segments as a 2D detected carton at least in part by discarding edges that form smaller rectangles that are completely contained within a larger rectangle.

9. The carton detection system of claim 6, wherein the processing subsystem is further configured to stitch together the 2D image and the 3D point cloud to obtain an image of the carton stack.

10. The carton detection system of claim 6, further comprising an automation controller in communication with the processing subsystem, the automation controller causing a robotic carton manipulator to perform one of a load operation and an unload operation using the 3D positions by the robotic carton handling system.

11. A material handling system, comprising: a robotic carton handling system for unloading cartons in a carton stack, the robotic carton handling system being movable across a floor, the robotic carton handling system comprising: a mobile body; a movable robotic manipulator attached to the mobile body and comprising an end effector at an end thereof, the end effector being configured to unload one or more cartons from the carton stack; a conveyor mounted on the mobile body, the conveyor being configured to receive the one or more cartons from the end effector and move the one or more cartons toward a rear of the robotic carton handling system; a carton detection system comprising: one or more sensors respectively coupled to one of the mobile body and the movable robotic manipulator to provide a two-dimensional (2D) optical image and a three-dimensional (3D) point cloud of at least a portion of the carton stack disposed on a floor of a transportation carrier; a processing subsystem in communication with the one or more sensors, the processing subsystem: detecting a set of segments within the 3D point cloud; detecting another set of segments within the 3D point cloud; defining the set of segments as a 3D detected carton and the other set of segments as a 2D detected carton; merging the 2D and 3D detected cartons into a detection result; and converting the detection result into a 3D position of a target carton for removal using calibration information, an automated controller in communication with the processing subsystem, the automated controller causing the robotic carton manipulator to use the 3D position to perform a selected one of a load operation and an unload operation by the robotic carton handling system; and an extendable conveyor system having a proximal end coupled to the fixed conveyor system and a movable distal end positioned proximate the robotic carton handling system to transfer cartons between the fixed conveyor system and the robotic carton handling system; wherein if the processing subsystem determines that the set of segments is less than a segment threshold, the set of segments is defined as 3D-detected cartons; and wherein if the processing subsystem determines that the other set of segments is greater than a segment threshold, the other set of segments is defined as 2D-detected cartons, wherein the processing subsystem is further configured to: determine a 2D mask corresponding to the other set of segments; determine a portion of a 2D image corresponding to the 2D mask; detect the other set of segments within the portion of the 2D image; and define the detected other set of segments as 2D-detected cartons.

12. The material handling system of claim 11, wherein the processing subsystem is further configured to: receive 2D images and 3D point clouds from one or more sensors positioned on the robotic carton handling system to detect at least one portion of a carton stack; receive 2D images and 3D point clouds from another one or more sensors positioned on the robotic carton handling system to detect an adjoining portion of the carton stack; and combine 2D and 3D detected cartons from both the at least one portion and the adjoining portion to form a detection result.

Images