Closed loop solution for loading / unloading cartons at truck unloaders

By integrating robot arms and sensors on truck unloaders, dynamic adjustment of unloading strategies is achieved using vision systems and torque detection, the problems of low unloading efficiency and vulnerability of cargo in the prior art are solved, and the unloading efficiency and cost-effectiveness are improved.

CN114249137BActive Publication Date: 2024-11-12INTELLIGRATED HEADQUARTERS LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111130986.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-09-21
Filing Date
2021-09-26
Publication Date
2024-11-12
Estimated Expiration
2041-09-26

AI Technical Summary

Technical Problem

Existing truck unloaders have packaging integrity problems during unloading operations, cargo and products are easily damaged, and the waiting time and downtime of sweep operations lead to low return on investment and inconvenient work under high or low temperature conditions.

Method used

A truck unloader equipped with a robot arm and multiple sensors is used to detect the distance and torque input of the carton stack through the visual system, dynamically adjust the unloading strategy, and perform continuous sweep operations until the torque input exceeds the threshold, ensuring safe and effective unloading.

Benefits of technology

Improve unloading efficiency and cost-effectiveness, reduce the risk of cargo damage, adapt to work environments with different temperature conditions, and achieve faster and cost-saving cargo unloading.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114249137B_ABST
    Figure CN114249137B_ABST
Patent Text Reader

Abstract

The present invention is entitled "Closed-loop solution for loading / unloading cartons on a truck unloader". The present invention provides a truck unloader for unloading cartons from a carton stack, the truck unloader comprising a controller having a detection system and a processor subsystem, the detection system being configured to periodically receive one or more images from a vision system, the processor subsystem being connected to the detection system. The processor subsystem is configured to calculate a distance from at least a portion of the truck unloader to the carton stack based on the one or more images, and to propel the truck unloader based on the distance to perform a sweeping operation if the distance satisfies a predefined distance threshold. The processor subsystem is further configured to monitor a torque input received as feedback from the one or more sensors while performing the sweeping operation, wherein the sweeping operation is configured to continue until the torque input exceeds a predefined torque threshold.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Exemplary embodiments of the present invention relate generally to truck unloaders and, more particularly, to a closed loop solution for unloading operations of truck unloaders. Background Art

[0002] Trucks and trailers loaded with goods and products travel across the country to deliver products to commercial loading docks at stores, warehouses, and distribution centers. Trucks may have trailers mounted on the truck, or may have semi-tractor trailer configurations. In an effort to reduce overhead costs at retail stores, the number of products in stores has been reduced, and products being shipped are now counted as part of available store inventory. Quickly unloading trucks at unloading docks at warehouses and regional distribution centers has gained new prominence as a method of replenishing depleted inventory.

[0003] Trucks are typically loaded and unloaded with forklifts if the load is palletized, and manually if the products are stacked within the truck. Manually unloading large truck shipments with human labor can be physically difficult and costly due to the time and labor involved. Additionally, the hot or cold conditions within the confined space of a truck trailer or shipping container can be considered unpleasant work. Therefore, there is a need for an improved unloading system that can unload large quantities of cargo and products from a truck trailer faster and more cost-effectively than human labor.

[0004] For economic reasons, the automation of loading or unloading needs to be relatively fast. A generally known method of unloading goods and products from a truck trailer is to perform a sweeping operation by a truck unloader. However, if performed in a random manner, this sweeping operation has several disadvantages. For example, some disadvantages may be the integrity of the packaging, and the contents of the goods and products may be damaged, the waiting time or downtime between sweeping operations may make it difficult to achieve an economic return on investment (ROI) using a truck unloader to automatically unload, the increased sensitivity of goods and products to being damaged during the sweeping operation, etc. Through hard work, ingenuity and innovation, many of these identified problems have been solved by developing solutions included in the embodiments of the present disclosure, many examples of which are described in detail herein. Summary of the invention

[0005] The following is a brief summary of the invention to provide a basic understanding of some aspects of the disclosed material handling system. This summary is not an exhaustive review and is neither intended to identify key or important elements nor to describe the scope of such elements. Its purpose is to serve as a prelude to the detailed embodiments provided later, giving some concepts of the described features in a simplified form.

[0006] Various exemplary embodiments described herein relate to a truck unloader for unloading cartons from a carton stack, the truck unloader comprising a robotic arm and a front portion comprising one or more sensors. A controller comprises a detection system configured to periodically receive one or more images from a vision system. A processor subsystem connected to the detection system is configured to calculate a distance from at least a portion of the truck unloader to the carton stack based on the one or more images, and to propel the truck unloader based on the distance to perform a sweeping operation if the distance satisfies a predefined distance threshold. The processor subsystem is further configured to monitor a torque input received as feedback from the one or more sensors while performing the sweeping operation, wherein the sweeping operation is configured to continue until the torque input exceeds a predefined torque threshold.

[0007] Various exemplary embodiments described herein relate to a truck unloader wherein the front portion includes front rollers and a conveyor.

[0008] Various exemplary embodiments described herein relate to truck unloaders in which the front portion performs a sweep operation by pulling one or more cartons onto a conveyor.

[0009] Various exemplary embodiments described herein relate to a truck unloader wherein one or more sensors are torque / force sensors configured to measure torque when the front portion contacts a stack of cartons.

[0010] Various exemplary embodiments described herein relate to truck unloaders wherein the value of the torque input varies during each sweep operation based on one of carton size or a packaging arrangement of a carton stack or a wall of a carton stack container.

[0011] Various exemplary embodiments described herein relate to a truck unloader, wherein a vision system includes one or more sensors respectively attached to one of a mobile body or a robotic arm to provide a two-dimensional (2D) image and a three-dimensional (3D) point cloud.

[0012] Various exemplary embodiments described herein relate to a truck unloader, wherein the processing subsystem is further configured to perform a transformation from a camera coordinate system to a robot coordinate system on the 3D point cloud data to calculate the distance.

[0013] Various exemplary embodiments described herein relate to a truck unloader wherein the processing subsystem is further configured to perform Euclidean clustering on the transformed 3D point cloud data to calculate distances.

[0014] Various exemplary embodiments described herein relate to a truck unloader wherein the front portion is further configured to cease sweeping operation when the torque input exceeds a predefined torque threshold.

[0015] Various exemplary embodiments described herein relate to a controller for controlling a truck unloader to unload cartons from a carton stack. The controller includes a detection system and a processor subsystem connected to the detection system, the detection system being configured to periodically receive one or more images from one or more sensors, the processor subsystem being configured to: calculate a distance from at least a portion of the truck unloader to the carton stack based on the one or more images, and to propel the truck unloader based on the distance to perform a sweeping operation if the distance satisfies a predefined distance threshold. The processor subsystem is further configured to monitor a torque input received as feedback from one or more torque sensors while performing the sweeping operation, wherein the sweeping operation is configured to continue until the torque input exceeds a predefined torque threshold.

[0016] Various exemplary embodiments described herein relate to a controller wherein the processor subsystem further comprises a segmentation module and a distance generation module configured to calculate a distance between a front of the truck unloader and a portion of the stack of cartons.

[0017] Various exemplary embodiments described herein relate to a controller wherein the processor subsystem further comprises a cloud accumulation and filtering module for accumulating 3D point cloud data points of one or more sensors.

[0018] Various exemplary embodiments described herein relate to a controller wherein the processor subsystem is further configured to estimate the pose of the robotic arm and front portion of the truck unloader using forward kinematics.

[0019] Various exemplary embodiments described herein relate to a method for controlling a truck unloader to unload a group of cartons. The method includes receiving a plurality of images from one or more sensors on the truck unloader, and calculating a distance to propel the truck unloader to perform a sweeping operation based on the plurality of images. The method also includes receiving a torque input as feedback in response to the truck unloader performing a sweeping operation by propulsing toward the group of cartons based on the distance. In response to determining that the torque input is within a predefined torque threshold, a first instruction for performing a sweeping operation is transmitted when the distance is greater than the predefined distance threshold, and another distance is calculated to propel the truck unloader, wherein the distance is recursively calculated before each sweeping cycle. In response to determining that the torque input exceeds the predefined torque threshold, determining whether to perform a pick-up operation or a sweeping operation, or transmitting a second instruction for performing a pick-up operation when the distance is less than the predefined distance threshold.

[0020] Various exemplary embodiments described herein relate to a method for controlling a truck unloader to unload a group of cartons, wherein determining that the torque input is within a predefined torque threshold further comprises determining whether the distance exceeds a predefined distance threshold. In response to determining that the torque input is within the predefined torque threshold and the distance exceeds the predefined distance threshold, a second sweeping operation is performed. In response to determining that the torque input exceeds the predefined torque threshold or the distance is below the predefined distance threshold, a determination is made whether to perform a third sweeping operation or a pick-up operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings illustrate embodiments of the invention and, together with the description (including the general description above and the detailed description below), serve to explain features of the invention.

[0022] Figure 1 A side view and functional block diagram of a truck unloader system using a vision system to unload cartons from a stack of cartons within a truck trailer is shown, according to an embodiment of the present subject matter.

[0023] Figure 2 A functional block diagram depicting modules involved in processing an image from a sensor is shown according to an embodiment of the present subject matter.

[0024] Figure 3 An exemplary 3D point cloud view of an unloading environment with a truck unloader system according to an embodiment of the present subject matter is shown.

[0025] Figure 4 A method for using a Figure 1 An exemplary closed-loop solution for a sweep operation performed by a truck unloader system.

[0026] Figure 5 The embodiment according to the present subject matter is shown by Figure 1 An exemplary flow chart of a sweep operation performed by a truck unloader system.

[0027] Figure 6 The invention is shown for use according to an embodiment of the present subject matter Figure 1 An exemplary computing environment for a controller of a truck unloader system.

[0028] Figure 7 A method for using a Figure 1 An exemplary method of performing a sweep operation for a truck unloader system. DETAILED DESCRIPTION

[0029] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which some but not all embodiments of the present invention are shown. In fact, these inventions can be embodied in many different forms and should not be interpreted as being limited to the embodiments set forth herein; on the contrary, these embodiments are provided so that the present disclosure meets applicable legal requirements. Throughout the text, similar reference numerals refer to similar elements. As used herein, terms such as "front", "rear", "top", "exterior", "interior", etc. are used for illustrative purposes in the examples provided below to describe the relative positions of certain devices or parts of devices. The terms used in this patent are not meant to be restrictive, and the devices described herein or parts thereof may be attached or utilized in other orientations.

[0030] The term "comprising" means including but not limited to, and should be interpreted in the manner commonly used in the patent context. It should be understood that the use of broad terms such as "comprising," "including," and "having" provides support for narrower terms such as "consisting of," "consisting essentially of," and "composed essentially of."

[0031] The phrases "in one embodiment," "according to one embodiment," and the like generally mean that the particular feature, structure, or characteristic following the phrase may be included in at least one embodiment of the invention, and may be included in more than one embodiment of the invention (importantly, such phrases are not necessarily referring to the same embodiment).

[0032] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any implementation described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other implementations.

[0033] The terms "about" or "approximately" and the like, when used with a number, may mean the specific number, or alternatively, a range approximate to the specific number as understood by those skilled in the art.

[0034] If the specification states that a component or feature "may", "can", "could", "should", "will", "preferably", "likely", "usually", "optionally", "for example", "often", or "might" (or other such words) be included or have a property, the particular component or feature is not required to be included or have that property. Such components or features may optionally be included in some embodiments, or may be excluded.

[0035] As used herein, the term "transmitter" refers to any component that can generate radio waves for communication purposes, while "receiver" is used to broadly refer to any component that can receive radio waves and convert that information into a usable form. "Transceiver" broadly refers to a component that can both generate and receive radio waves and is therefore contemplated when discussing either a transmitter or a receiver.

[0036] The term "processor" is used herein to refer to any programmable microprocessor, microcomputer, or one or more multiprocessor chips that can be configured by software instructions (applications) to perform various functions including the functions of the various embodiments described above. In some devices, multiple processors may be provided, such as a processor dedicated to wireless communication functions and a processor dedicated to running other applications. Software applications may be stored in internal memory before being accessed and loaded into the processor. The processor may include internal memory sufficient to store application software instructions. In many devices, the internal memory may be a volatile or non-volatile memory such as flash memory or a mixture of the two. The memory may also be located inside another computing resource (e.g., enabling computer-readable instructions to be downloaded via the Internet or another wired or wireless connection).

[0037] For the purposes of this description, general references to memory refer to memory accessible to the processor, including internal memory inserted into the device or removable memory and memory within the processor itself. For example, the memory can be any non-transitory computer-readable medium on which computer-readable instructions (e.g., computer program instructions) that can be executed by the processor are stored.

[0038] The term "electronic device" as used hereinafter refers to any or all of a handheld device, a mobile phone, a wearable device, a personal data assistant (PDA), a tablet computer, a smart book, a handheld computer, a bar code reader, a scanner, a tag reader, an imager, a radio frequency identification (RFID reader or interrogator), a vehicle mounted computer, a wearable bar code scanner, a wearable tag reader, a point of sale (POS) terminal, a headset device, a programmable logic controller (PLC), a programmable automation controller (PAC), an industrial computer, a laptop computer, a desktop computer, a personal computer, and similar electronic devices equipped with at least one processor configured to perform the various operations described herein.

[0039] For the sake of brevity, the terms "computing platform" or "master device" or "server" or "supervisor device" are used interchangeably herein to describe various embodiments. The term "server" may be used herein to refer to any computing device or distributed network of computing devices that can act as a server (such as a master exchange server, a network server, a mail server, a document server, or any other type of server). A server can be a dedicated computing device or a computing device that includes a server module (e.g., an application that runs an application that enables the computing device to operate as a server). A server module (e.g., a server application) can be a full-featured server module or a lightweight or auxiliary server module (e.g., a lightweight or auxiliary server application) that is configured to provide synchronization services in a dynamic database on a computing device. A lightweight server or auxiliary server can be a streamlined version of a server-type function that can be implemented on a computing device such as a smart phone, so that it can act as an Internet server (e.g., a corporate email server) only when it is required to provide the functions described herein.

[0040] In some exemplary embodiments, the server may correspond to any one of an industrial computer, a cloud-based platform, an external computer, an independent computing device, etc. In some exemplary embodiments, the host device or computing platform may also refer to any electronic device, as described herein. In some exemplary embodiments, the server may include an access point or gateway device that may be able to communicate directly with one or more electronic devices, and may also be able to communicate (directly or indirectly via a communication network such as the Internet) with a network establishment service (e.g., an Internet service provider). In some exemplary embodiments, the server may manage the deployment of one or more electronic devices throughout a physical environment. In some exemplary embodiments, the server may refer to a network establishment service including a distributed system, wherein multiple operations are performed by utilizing multiple computing resources deployed on a network and / or a cloud-based platform or cloud-based service (such as any one of a software-based service (SaaS), an infrastructure-based service (IaaS), or a platform-based service (PaaS), etc.).

[0041] In some exemplary embodiments, the term "server" may be used herein to refer to a programmable logic controller (PLC), a programmable automation controller (PCC), an industrial computer, a desktop computer, a personal data assistant (PDA), a laptop computer, a tablet computer, a smartbook, a handheld computer, a personal computer, a smart phone, a headset, a smart watch, and similar electronic devices equipped with at least a processor configured to perform the various operations described herein. Devices such as smart phones, tablet computers, headsets, and smart watches are generally referred to as mobile devices.

[0042] The components shown in the figures represent components that may or may not be present in the various embodiments of the invention described herein, such that an embodiment may include fewer or more components than shown in the figures without departing from the scope of the invention.

[0043] References in this specification to "one embodiment," "embodiment," "multiple embodiments," or "one or more embodiments" are intended to indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present disclosure. The appearance of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. In addition, various features are described that may be present in some embodiments but not in other embodiments. Similarly, various requirements are described that may be requirements of some embodiments but not in other embodiments.

[0044] It should be understood that the use of specific parts, equipment and / or parameter names and / or their corresponding acronyms such as those of the utility, logic and / or firmware of the execution described herein is only exemplary and does not mean to imply any limitation to the embodiment. Therefore, the embodiment can be described with different naming and / or terms, which are used to describe the parts, equipment, parameters, methods and / or functions herein, but are not limited thereto. The reference to any specific protocol or proprietary name when describing one or more elements, features or concepts of the embodiment is provided only as an example of a specific implementation, and this reference does not limit the claimed embodiment to be extended to the embodiment using different elements, features, protocols or concept names. Therefore, each term utilized herein will be interpreted in its broadest sense in the context of utilizing the term.

[0045] Throughout this specification, the terms "robotic arm," "robotic arm," "robotic arm assembly," and "end effector" are used interchangeably to refer to a robotic arm having 4 to 6 axes of freedom (DOF).

[0046] Throughout this specification, the terms "torque input", "torque feedback", "torque output", and "torque value" may be used interchangeably to refer to the torque value from the torque / force sensor.

[0047] Throughout this specification, the terms "carton stack container", "container", "truck trailer" are used interchangeably to refer to a container having a cartons stack inside.

[0048] Sweeping operation and / or picking operation are important operations for a truck unloader to unload cartons from a carton pile in a truck trailer. Generally speaking, the sweeping operation is performed on cartons at floor height, and the picking operation is performed on cartons at a certain height above the floor height.

[0049] Conventionally, such sweeping operations or picking operations may be performed by a truck unloader using a method known as "sensing plane action". For example, a vision system may capture an image of a carton pile before unloading the carton, and then process the image to determine a motion plan. Thus, the truck unloader may unload the carton from the carton pile according to the motion plan. In this regard, when the "sensing plane action" method is used, in some examples, the truck unloader determines the motion plan based on a single image before starting the unloading operation, which may result in the truck unloader not being aware of changes in the unloading environment around the truck unloader when performing the unloading operation. In addition, when the "sensing plane action" method is used, there is a delay or time delay in processing the image and determining the motion plan. The truck unloader may remain idle during this time delay, which may affect the unloading throughput. Since the truck unloader is unaware of changes in the unloading environment, the truck unloader may lack the decision-making ability to choose between a sweeping operation and a picking operation to unload the carton. This decision may affect the unloading throughput because the sweeping operation is able to accumulate the largest volume of cartons from the carton pile compared to the picking operation. Furthermore, when using the "sense-plane-motion" approach, the truck unloader may capture additional images between unloading operations, however such images may be further processed to determine an alternative motion plan, which in turn may require the truck unloader to remain idle during processing. In this regard, the "sense-plane-motion" approach fails to continuously monitor the unloading environment and lacks the decision-making ability to select between sweep and pick operations without affecting unloading throughput while minimizing idle time of the truck unloader.

[0050] In some examples, and as disclosed herein, a truck unloader uses a closed-loop based solution executed by a robotic controller of the truck unloader. In some examples, the solution enables a vision system on the truck unloader to continuously monitor the unloading environment by accumulating images captured before and during the unloading operation. The solution determines the distance the truck unloader can travel to perform the continuous sweeping operation based on processing the accumulated images. In addition, the solution decides to continue to perform the continuous sweeping operation based on torque feedback from one or more sensors on the front end of the truck unloader.

[0051] In some examples, torque feedback may provide decision-making capabilities that allow the truck unloader to switch from a sweeping operation to a pickup operation, from a pickup operation to a sweeping operation, and / or stop a sweeping operation. In this regard, the solution dynamically updates the distance that the truck unloader must travel to perform a continuous sweeping operation based on processing images accumulated after each sweeping operation. Such a continuous sweeping operation may be performed based on the updated distance until the torque feedback exceeds a certain threshold. Thus, the truck unloader may be aware of the unloading environment by continuously updating the distance based on periodic capture of images. The truck unloader may decide to continuously perform a sweeping operation based on the torque feedback and the updated distance without affecting throughput while minimizing idle time.

[0052] In the following detailed description of the exemplary embodiments of the present disclosure, the specific exemplary embodiments of the present disclosure are described in sufficient detail so that those skilled in the art can practice the disclosed embodiments. For example, specific details such as specific method sequences, structures, elements and connectors have been presented herein. However, it should be understood that it is not necessary to implement the embodiments of the present disclosure using the specific details proposed. It should also be understood that other embodiments can be used, and logical, architectural, programmatic, mechanical, electrical and other changes can be made without departing from the general scope of the present disclosure. Therefore, the following detailed description is not restrictive, and the scope of the present disclosure is limited by the attached claims and their equivalents.

[0053] Figure 1 A truck unloader 100 is shown with a manipulator such as a robotic arm or end effector 102 that unloads cartons 104 from a carton stack 106 within a carton stack container 108 such as a truck trailer, a container, a storage unit, etc. The robotic arm 102 places the carton 104 on a conveyor system 110 of the truck unloader 100, which conveys the carton 104 back to an extendable conveyor 112 that follows a mobile body 114 of the truck unloader 100 into the carton stack container 108. The extendable conveyor 112 in turn conveys the carton 104 to a material handling system 116 such as a warehouse, a store, a distribution center, etc.

[0054] In one or more embodiments, the truck unloader 100 autonomously unloads the carton stack 106 resting on the floor 118 of the carton stack container 108. The mobile body 114 is self-propelled and is capable of moving from the outside to the innermost part of the carton stack container 108 on the floor 118. The right lower arm and the left lower arm 120 of the robot arm assembly 102 are pivotally attached at the lower end 122 to the mobile body 114 on the opposite lateral sides of the conveyor system 110 passing therebetween. The right lower arm and the left lower arm 120 rotate about a lower arm axis 124 perpendicular to the longitudinal axis 126 of the conveyor system 110. The upper arm assembly 128 of the robot arm assembly 102 has a rear end 130 pivotally attached to the upper end 132 of the right lower arm and the left lower arm 120, respectively, to pivotally rotate about an upper arm axis 134 perpendicular to the longitudinal axis 126 of the conveyor system 110 and parallel to the lower arm axis 124. The manipulator head 136 is attached to the front end 138 of the upper arm assembly 128 and engages at least one carton 104 at a time from the carton stack 106 for movement to the conveyor system 110. The pivoting and simultaneous mirroring motion of the right and left lower arms 120 maintains the upper arm axis 134 at a relative height above the conveyor system 110 so that the at least one carton 104 can be conveyed by the conveyor system 110 without being obstructed by the robot arm assembly 102 when the manipulator head 136 is clear. In one or more embodiments, the truck unloader 100 includes a lifter 140 attached between the mobile body 114 and the front portion 142. The lifter 140 moves the front portion 142 relative to the floor 118 to reduce the clearance below the at least one carton 104 during movement from the carton stack 106 to the conveyor system 110. The front portion 142 may include, for example, the conveyor 111. The front portion may be hingedly attached to a base or mobile body 114 of the truck unloader 100 via one or more joints.

[0055] The manipulator head 136 may include an array of vacuum suction cups (not shown). Each vacuum suction cup may be configured to be actuated so as to move forward and / or backward from its initial position, thereby grabbing items based on vacuum suction. In this regard, the pickup operation of the truck unloader 100 may be performed by the suction cups. In addition, the front portion 142 may be configured to be movably positioned so that a portion of the front portion 142 may sweep under a portion of one or more cartons, thereby pulling the one or more cartons onto the conveyor 111 of the front portion 142 of the truck unloader 100. In this regard, the front portion 142 may include one or more mechanical components that are configured to move in any one of the X, Y, and / or Z directions as needed to pull one or more cartons. In this regard, the front portion 142 may be used to perform a sweeping operation or action of the truck unloader 100. In some examples, to perform the sweeping action, the truck unloader 100 lowers the front portion and pushes it upward to the stack of cartons while cartons that have been displaced on the front portion 142 are conveyed by the conveyor 111 .

[0056] In some examples, the front portion 142 may include one or more front rollers 143 that may be disposed on the end of the conveyor 111 to contact the carton and rotate to pull the carton on the conveyor 111. In some examples, the roller 144 may extend from the conveyor 111 or retract toward the conveyor to move the carton away from the carton stack. Alternatively, the front roller 143 may be placed against the carton stack to support the carton stack when the robot arm 102 unloads the carton. According to one embodiment, the front portion 142 includes one or more torque / force sensors for measuring torque. In some examples, the torque / force sensor may be positioned on the front roller 143 to measure the torque applied to the front roller 143 when contacting the carton during the unloading operation. In some examples, the torque / force sensor may be any sensor generally known in the art that is designed to monitor, detect, record and adjust the linear force and rotational force applied thereto. The torque / force sensor may enable the truck unloader 100 to detect the force and torque applied to the front portion 142. For example, when the front portion 142 of the truck unloader 100 sweeps loosely stacked cartons and tightly stacked cartons from the floor and / or carton stack, the torque value may be different. In other words, the torque value may be different based on the packaging arrangement of the cartons in the carton stack. In some examples, the torque value may be different based on the carton size during each sweeping operation. In some examples, when the front portion 142 contacts an obstacle such as the wall of the truck trailer 108 behind the carton stack, a significant difference in the torque value may be noticed.

[0057] A higher level system may assign an autonomous robotic vehicle controller 144 (hereinafter referred to as a robot controller 144) of the truck unloader 100 to a specific carton stack container 108, and may receive information about the progress of loading / unloading and provide access for remote control. When an error is encountered during loading or unloading, a human operator may selectively intervene. A higher level system may include a host system 146 that processes external order transactions to be executed by the material handling system 116. Alternatively or in addition, a warehouse execution system (WES) 148 may provide vertical integration of a warehouse management system (WMS) 150 that executes order fulfillment, labor management, and inventory tracking of facilities 152 such as distribution centers. The WES 148 may include a vertically integrated warehouse control system (WCS) 154 that controls the automation of order fulfillment and inventory movement requested by the WMS 150. According to one embodiment, a higher level system may include a decision system 191 that can be communicatively connected to the robot controller 144. According to another embodiment, the decision system 191 may be installed or provided as a subsystem within the robot controller 144. The decision system 191 may decide whether to perform a pick-up operation or a sweep operation, and transmit the decision to the robot controller 144. The decision system 191 may be a computer system or a cloud computing system, which includes one or more processors and memory / database to execute one or more algorithms, such as but not limited to image processing algorithms, machine learning algorithms, etc.

[0058] The truck unloader 100 may include a vision system that includes one or more sensors 176 (e.g., cameras, microphones, RGB-D sensors, LiDAR (light detection and ranging) sensors, etc.). The vision system may be used to scan in front of the truck unloader 100 in real time or near real time. A forward scan may be triggered in response to the truck unloader 100 being in a position to begin detecting cartons in the unloading environment. The forward scanning capability may be used for collision avoidance, sent to human shape recognition (safety), determining the size of the unloading area (e.g., truck or trailer), and for scanning the floor of the unloading area for loose items (e.g., cartons, boxes, etc.). The 3D capabilities of the sensors 176 may also provide depth perception, edge recognition, and may create a 3D image of a wall of cartons (or a stack of cartons). In some examples, one or more sensors 176 may be mounted on the robotic arm 102. The processor subsystem 158 may operate in conjunction with the detection system 166 to identify the edges, shapes, and near / far distances of items in front of the truck unloader 100. For example, the edges and distances of each individual carton in the carton wall may be measured and calculated relative to the truck unloader 100. Such distance calculations will be performed by the processor subsystem 158. The processor subsystem 158 may include various applications or software modules or components that perform the processes described later in this application. According to one embodiment, the processor subsystem 158 includes a point cloud accumulation module 190, a point cloud filtering module 192, a segmentation module 194, and a distance generation module 196.

[0059] In one or more embodiments, once assigned by the WES 148 or manually enabled, the truck unloader 100 can perform the following autonomous operations under the control of the robotic controller 144: (i) move into a carton container 108, (ii) perform one of loading or unloading a carton stack container 108, and (iii) move out of a carton stack container 108. In order to navigate within the carton stack container 108 and quickly move cartons 104 therein, sensors 176 can be attached to one of the mobile body 114 and the movable robotic manipulator (i.e., the robotic arm assembly 102), respectively, to provide a two-dimensional (2D) optical image and a three-dimensional (3D) point cloud of at least a portion of the carton stack 106 resting on the floor 159 of the carton stack container 108. In addition, the sensor 176 provides a two-dimensional (2D) image and a three-dimensional (3D) point cloud of at least a portion of the truck unloader 100. Additionally, the sensor 176 provides a two-dimensional (2D) image and a three-dimensional (3D) point cloud of at least a portion of the floor 159 and walls of the carton stack container 108. The carton stack container 108 may be stationary or mobile, such as a transport vehicle transported by road, rail, or on navigable waters.

[0060] In some examples, the detection system 166 may be a 3D / 2D detection system 166, which may include depth sensing using binocular principles, lidar principles, radar principles, or sonar principles. In order to avoid dependence on consistent ambient lighting conditions, the illuminator 169 may provide a consistent or adjustable amount of illumination in one or more spectral bandwidths such as visible light or infrared light. The illumination may be narrowly defined in the visible spectrum, enabling filtering of most ambient light. Alternatively, the illumination may be outside the visible distance so that the illumination does not distract the human operator. The 3D / 2D detection system 166 may receive 2D and 3D sensor data from an RGB-D sensor 176, which observes the interior of the carton stack container 108 and the carton stack 106 and at least a portion of the truck unloader 100. For these and other purposes, the 3D / 2D detection system 166 may include various applications or components that perform the processes described later in this application. For example, the 3D / 2D inspection system 166 may include a 2D process module 180 , a 3D process module 182 , and a 3D-guided 2D process module 184 .

[0061] The robot controller 144 provides an exemplary environment in which one or more of the features of the various embodiments of the present disclosure may be implemented. The controller 144 may be implemented as an integrated device or a distributed processing system. The controller 144 includes functional components that communicate across a system interconnect of one or more conductors or fiber optic structures, which are depicted as a system bus 156 for clarity. The system bus 156 may include a data bus, an address bus, and a control bus for transmitting data, address, and control information between any of these coupled units. The functional components of the controller 144 may include a processor subsystem 158, which is composed of one or more central processing units (CPUs), digital signal processors (DSPs), and processor memory. The processor subsystem 158 may include any tool or collection of tools for computing, classifying, processing, transmitting, receiving, retrieving, initiating, switching, storing, displaying, indicating, detecting, recording, copying, applying, or utilizing any form of information, intelligence, or data for business, scientific, control, or other purposes, including controlling automated equipment for material handling systems.

[0062] According to various aspects of the present disclosure, an element, or any part of an element, or any combination of elements can be implemented with a processor subsystem 158, which includes one or more physical devices having a processor. Non-limiting examples of processors include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), programmable logic controllers (PLCs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform various functions described throughout the present disclosure. One or more processors in a processing system can execute instructions. A processing system that executes instructions to achieve a result is a processing system that is configured to perform tasks that lead to a result, such as by providing instructions to one or more components of the processing system, which instructions will cause these components to perform actions that themselves or in combination with other actions performed by other components of the processing system will lead to a result.

[0063] The robot controller 144 may include a network interface (I / F) device 160 that enables the controller 144 to communicate or interface with other devices, services, and components (such as WES 148) located outside the controller 144. These networked devices, services, and components may interface with the controller 144 via an external network (such as the exemplary network 162) using one or more communication protocols. The network 162 may be a local area network, a wide area network, a personal area network, etc., and the connection to the network and the controller 144 and / or between the two may be a wired or wireless connection or a combination thereof. For the purpose of discussion, for simplicity, the network 162 is indicated as a single common component. However, it should be understood that the network 162 may include one or more direct connections to other devices and a more complex set of interconnections that may exist in a wide area network, such as the Internet or a dedicated intranet. For example, a programming workstation may remotely modify the programming or parameter settings of the controller 144 via the network 162. The various links in the network 162 may be wired or wireless. The controller 144 can communicate with a number of onboard devices, such as lights, indicators, manual controls, etc., via the device interface 168. The device interface 168 can include wireless links and wired links. For example, the controller 144 can direct the extendable conveyor 112 to follow the truck unloader 100 into the carton stack container 108 or to lead the truck unloader 100 out of the carton stack container 108.

[0064] The robotic controller 144 may include several distributed subsystems that manage specific functions of the truck unloader 100. The automation controller 170 may receive position and spatial calibration information from the detection system 166 and use this data to coordinate the movement of the mobile body 114 and the movement of payload components such as the robotic arm assembly 102 and the lift 140 that moves the front portion 142 of the conveyor system 110 via the vehicle interface 172. According to one embodiment, the truck unloader 100 may be of the type disclosed in U.S. Patent No. 10,336,562, which is incorporated herein by reference in its entirety.

[0065] The processor subsystem 158 may use the system memory 164 to store functional components, such as data and software, such as the 3D / 2D detection system 166. Software may be broadly construed as instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, execution threads, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description languages, function block diagrams (FBD), ladder diagrams (LD), structured text (ST), instruction lists (IL), and sequential function charts (SFC), or otherwise. The software may reside on a computer readable medium.

[0066] For clarity, the system memory 164 may include random access memory, which may or may not be volatile, non-volatile data storage devices. The system memory 164 contains one or more types of computer-readable media, which may be non-transitory or transient. By way of example, computer-readable media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs)), smart cards, flash memory devices (e.g., cards, sticks, key drives), random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, removable disks, and any other suitable medium for storing software and / or instructions that can be accessed and read by a computer. The computer-readable medium may reside in the processing system, outside the processing system, or distributed across multiple entities including the processing system. The computer-readable medium may be embodied in a computer program product. By way of example, a computer program product may include a computer-readable medium in packaging material. Those skilled in the art will recognize how to best implement the described functionality presented throughout this disclosure depending on the particular application and the overall design constraints imposed on the overall system.

[0067] Figure 2A functional block diagram according to an embodiment of the present subject matter is shown, which depicts modules involved in processing images from sensors. As previously described, the sensor 176 can be used to capture images of the unloading environment. The unloading environment can be, for example, a stack of cartons, a wall of a truck trailer 108, and a truck unloader 100. For example, the sensor 176 can capture an image of a location in front of the robot arm assembly 102 (e.g., a portion of the stack of cartons resting on the floor, a portion of the floor, a portion of the wall, and at least a portion of the truck unloader 100). In some examples, the portion of the truck unloader 100 can be the front 142 of the truck unloader 100 that performs the sweep operation.

[0068] In some examples, the sensor 176 may take the form of multiple sensors, such as sensors 176a, 176b. In some examples, the sensors 176a, 176b are capable of providing both 2D images and 3D images. According to one embodiment, the sensors 176a, 176b are capable of continuously capturing images before starting the unloading process and while performing the unloading process. The images captured by the sensors 176a, 176b may be used as input for the truck unloader 100 to perform a sweeping operation and / or a picking operation. Figure 2 As shown, the captured images from sensors 176a, 176b are further processed by processor subsystem 158. As previously described, the processor subsystem includes point cloud accumulation module 190, point cloud filtering module 192, segmentation module 194, and distance generation module 196.

[0069] According to one embodiment, the point cloud information of the images captured by the sensors 176a, 176b is accumulated by the point cloud accumulation module 190. In some examples, the point cloud information is a set of data points representing the unloading environment. These points can represent 3D shapes. Each point has its own set of X, Y and Z coordinates, and in some cases has additional attributes. In some examples, the sensors 176a, 176b can be LiDAR (light detection and ranging) sensors that can be used to collect point cloud information of the unloading environment. In some examples, the unloading environment may include a pile of cartons, the front of the truck unloader 100, the wall of the truck trailer 108, etc.

[0070] In some examples, the LiDAR sensors 176a, 176b may be mounted to the robot arm 102 or the robot base (i.e., the base of the truck unloader 100) and configured to emit laser pulses to the stack of cartons, and once the laser returns, the LiDAR sensors 176a, 176b will record data points. The data points from the LiDAR sensors 176a, 176b may be combined to create point cloud information.

[0071] The point cloud accumulation module 190 can accumulate the point cloud information from the sensors 176a, 176b to create a larger point cloud. For example, the data points from the first sensor 176a can be 300 data points, and the data points from the second sensor 176b can be 300 data points. These data points are combined to form a larger point cloud of, for example, 600 data points.

[0072] The data points may include X, Y, and Z coordinates, with Z representing the distance of the carton stack (or target carton) relative to the position of the sensors 176a, 176b. Since the reference point for capturing the image is from the sensors 176a, 176b, the resulting point cloud data represents the camera coordinate system (i.e., in a different viewpoint for each sensor 176a, 176b). Therefore, based on the viewpoints of these sensors 176a, 176b from their respective positions, 300 data points from the first sensor 176a may include a Z value that is different from the 300 data points from the second sensor. Such differences in Z values ​​are due to capturing the data points according to the camera coordinate system. Therefore, the data points can be transformed into a robot coordinate system to provide a common reference point for the Z value. This transformation to the robot coordinate system will provide a Z value that can represent the distance from the robot base to the carton stack, rather than representing the distance from the sensors 176a, 176b. The point cloud data can be transformed in the viewpoint of the robot base, which can be a reference point or the starting point of the point cloud data.

[0073] In some examples, the point cloud information from sensors 176a, 176b may be transformed so that each view appears to originate from a common viewing perspective. For example, the common viewing perspective may be a viewpoint from the base of the robot.

[0074] According to one embodiment, the transformation of the image (i.e., 3D point cloud in the camera coordinate system) from the sensors 176a, 176b to the robot base (i.e., 3D point cloud in the robot coordinate system) can be performed by forward kinematics. In some examples, forward kinematics refers to the robot using a set of kinematic formulas to calculate the position of the robot arm or end effector from predefined values ​​of joint parameters. The joint parameters can be known position information values ​​of the robot joints, such as robot arm joint angles or elbow joint angles. Such information can be obtained from robot joint encoders positioned in the robot arm. In this regard, the joint parameter information is provided as input to calculate the position or posture of the robot arm using forward kinematics. Based on the prior knowledge of the position of the sensors 176a, 176b on the robot arm, forward kinematics outputs a transformation from the base to the sensors 176a, 176b on the robot arm. In this regard, data points from the camera coordinate system (i.e., provided by the sensors 176a, 176b) are transformed into the robot coordinate system (i.e., provided by the transformation of the image relative to the robot base) so that each view appears to originate from a common viewing angle of the robot base. Those skilled in the art will appreciate that any type of kinematics may be employed to rotate and transform the 3D point cloud relative to the robot base. According to one embodiment, a larger point cloud is formed in the point cloud accumulation module 190 by applying a transformation to the data points accumulated from the sensors 176a, 176b. As discussed in the previous example, a transformation may be applied to the 600 data points accumulated at the point cloud accumulation module 190, such that the processing time for transforming 300 data points independently from each sensor 176a, 176b is reduced.

[0075] According to one embodiment, the position of the front portion 142 of the truck unloader 100 may be calculated using forward kinematics. In some examples, the front portion 142 of the truck unloader 100 may be used to perform a sweeping operation. The front portion 142 may be hingedly connected to a robot base. The position of the front portion 142 of the truck unloader 100 may be calculated based on predefined values ​​of joint parameters of the front portion 142 according to a kinematic formula. The joint parameters may be known position information values ​​of robot joints, such as joint angles of robot joints of the front portion 142. Such information is obtained from robot joint encoders positioned in the truck unloader 100. In this regard, the joint parameter information is provided as an input to calculate the position or pose of the front portion 142 using forward kinematics. Using a priori knowledge of the transformation from the base to the sensors 176a, 176b on the robot arm 102, the forward kinematics outputs a transformation from the robot base to the front portion 142. As discussed in the previous example, a transformation may be applied to the accumulated 600 data points so that the Z value may represent the distance from the robot base to the carton stack in the robot coordinate system, rather than the distance from the viewpoint of the sensor 176a, 176b in the camera coordinate system. In this regard, the pose of the robot arm 102 and the pose of the front portion 142 are obtained using forward kinematics. The pose information may indicate the X, Y, Z coordinate information of the robot arm 102 and the front portion 142. In this regard, with the X, Y, Z coordinate information of the robot arm 102 and the front portion 142 determined, the data points in the large point cloud may be transformed into the robot coordinate system.

[0076] According to one embodiment, the point cloud filtering module 192 filters out portions of the larger point cloud data by utilizing bilateral filters and blur functions known in the art to improve the efficiency of detecting the actual area of ​​interest. In some examples, the front 142 or the robot arm 102 can be filtered out from the rest of the unloading environment to generate a point cloud with an actual area of ​​interest. For example, since the X, Y, and Z coordinate information of the robot arm 102 and the front 142 is known, the filtering module can filter out the data points indicating the robot arm and the front, and classify the remaining data points as the unloading environment. In some examples, the unloading environment indicates the wall of the floor, the carton pile, and the truck trailer. For example, let us assume that the Z value after the transformation of the data point to the robot coordinate system represents the distance from the robot base (i.e., the reference point) to the carton pile. Let us assume that the distance from the robot base to the carton pile is 6 meters. Now, the Z value of the front 142 from the robot base (i.e., the reference point) is 4 meters. Then, the point cloud filtering module can filter out data points below 4 meters from the robot base, and classify data points above 4 meters from the robot base as actual areas of interest. Data points below 4 meters can be the front 142 and other parts of the robot, while data points above 4 meters can be the unloading environment. The filtered point cloud data with actual areas of interest are transmitted to the segmentation module 194.

[0077] According to one embodiment, the segmentation module 194 divides the filtered point cloud data into clusters. In some examples, the segmentation module may use a Euclidean clustering algorithm generally known in the art to divide the filtered point cloud data into multiple clusters. For example, a cluster represents the walls of each carton, floor, or truck trailer. In some examples, a threshold value may be selected for the maximum number of data points in a cluster, and the clusters may be divided according to the threshold value. For example, 300 data points may be clustered, where 15 data points are the maximum threshold that meets the cluster. According to one embodiment, the distance generation module 196 may identify a seed point within a cluster, which is called the centroid of the cluster. In some examples, the distance generation module 196 may use a centroid-based averaging algorithm known in the art to identify the centroid of each cluster. For example, the center point of each cluster may be called the centroid. The algorithm may calculate the average of all data points in the cluster and move the centroid to the average position. This calculation may be performed on all clusters in the filtered point cloud, thereby generating a centroid for each cluster. After locating the centroid of each cluster, the algorithm further calculates the distance between the clusters. For example, the distance between the clusters is the distance between the centroids of each cluster. Any technique known in the art can be used to calculate these distances, such as cosine distance or Euclidean distance. In some examples, the first cluster is represented as N, and then the adjacent clusters are represented as N-1, N-2, etc. The centroid of each cluster can be used to calculate the distance between these clusters N and N-1 or N-1 and N-2, N can indicate the first carton in the carton pile, N-1 indicates the second carton, and so on. As mentioned above, these clusters N, N-1, N-2 are observed from a common viewing angle (i.e., from the robot base as a reference point) realized by using the transformation of forward kinematics. In addition, the X, Y, Z coordinate information of the front 142 is known using forward kinematics. In this regard, the X, Y, Z coordinate information of the front 142 can be considered to calculate the distance of any cluster N, N-1, N-2 from the robot base. For example, let us assume that the Z value representing the distance from the robot base to the carton pile is 6 meters. Let us assume that according to the coordinate information of the front 142, the Z value representing the distance from the robot base to the endpoint on the front 142 is 4 meters. The algorithm can calculate the distance from the endpoint on the front 142 to cluster N as (6 meters-4 meters)2 meters. Let us assume that the distance between the centroids of cluster N and cluster N-1 is 1 meter, then the algorithm can calculate the distance from the endpoint on the front 142 to cluster N-1 as (2 meters+1 meter)3 meters. In this regard, the distance between the front 142 and the cluster (i.e., carton) is determined based on the distance for advancing the truck unloader 100. In some examples, the N clusters may represent more than one carton arranged in the first row of the carton pile, and the N-1 cluster may represent more than one carton arranged in the second row of the carton pile, and so on. The truck unloader 100 may advance toward the stack of cartons based on the calculated distance to unload each clustered carton.In some examples, the algorithm calculates the cluster farthest from the endpoint on the front 142. The farthest cluster may be the cluster with the greatest distance from the endpoint on the front 142 or the robot base (i.e., a carton or truck wall). In some examples, the maximum distance may indicate the distance that the truck unloader 100 may advance to perform a sweep operation on the clusters. The truck unloader may advance up to the maximum distance while sequentially sweeping clusters N, N-1, N-2. Various techniques exist in the art for efficiently calculating the centroid of each cluster and calculating the distance between each centroid, and those skilled in the art will recognize that any such technique may be utilized.

[0078] In some examples, the processes of accumulation, filtering, clustering, and calculating distances may be performed by the processor subsystem 158 during each sweep cycle. For example, a sweep cycle may indicate one or more sweeping operations or actions performed by the truck unloader 100 in a specific time period. The sensors 176a, 176b may continuously capture images with time stamps in each sweep cycle. Therefore, the point cloud accumulation module 190, the point cloud filtering module 192, the segmentation module 194, and the distance generation module 196 may process images based on time stamps. For example, the first sweep cycle includes at least 3 sweep actions to be performed by the truck unloader 100, and the modules 190, 192, 194, 196 may process images with first time stamps captured before initiating the first sweep cycle. Therefore, at the end of the first sweep cycle, the modules 190, 192, 194, 196 may begin processing images with second time stamps captured before initiating the second sweep cycle. In this regard, continuous sweep actions may be performed using iteratively captured images with different time stamps.

[0079] Figure 3 An exemplary 3D point cloud view of an unloading environment with a truck unloader according to an embodiment of the present subject matter is shown. Figure 3As shown in the example of , the actual area of ​​interest is a row of cartons 302 in the carton stack and a wall 304 of the truck trailer 108. The distance between the endpoint 301 of the front portion 142 and the base of the truck unloader 100 is known. As described above, the distance is known based on the transformation and coordinate information calculated by forward kinematics. For ease of explanation, we assume that the first cluster is the first carton 302a and the second cluster is the wall 304 of the truck trailer 108, which may be the cluster farthest from the front portion 142. As described above, the distance from the base of the truck unloader 100 to the carton stack may be 6 meters. The distance from the base of the truck unloader to the endpoint 301 of the front portion 142 may be 4 meters. The distance generation module 196 may calculate that the distance from the endpoint 301 of the front portion 142 to the carton stack (i.e., the first carton 302a) is 2 meters. The truck unloader 100 may advance a distance of 2 meters to perform a sweeping motion. As described above, the distance generation module 906 can use the centroid information to perform a comparison of distances between clusters to identify the farthest cluster. The distance between the base of the truck unloader and the farthest cluster (i.e., the wall) is represented as 306. The truck unloader 100 can continue to advance based on the distance 306 to perform a sweeping operation until it reaches the farthest cluster.

[0080] In this regard and in some examples, the sweep operation is performed based on the distance provided by the processor subsystem 158 in response to processing the image. In some examples, for each sweep cycle, the sweep operation can be performed without waiting for motion planning instructions from the decision system 191. In this regard, control can be relayed from the processor subsystem 158 to the decision system 191 only when the truck unloader 100 encounters an error during the sweep operation. Figure 4 and Figure 5 This control is further described.

[0081] An exemplary flow chart of a closed loop solution for performing a continuous sweep operation is shown in Figure 4 According to one embodiment, in addition to the combination Figure 2In addition to the visual feedback from sensors 176, 176b, the closed-loop solution also includes torque feedback. According to one embodiment, torque feedback is provided by a torque / force sensor positioned at the front position 142 of the truck unloader 100. The torque value is monitored for each sweeping motion performed by the truck unloader 100. At box 402, the detection system 166 and the processing subsystem 158 receive images with varying timestamps from the image sensor. The processing system 158 accumulates point cloud data from multiple images, transforms the images to the robot coordinate system and filters the point cloud data to identify areas in the unloading environment other than the truck unloader 100. The processing subsystem 158 also divides the point cloud data into multiple clusters representing cartons in each sweeping cycle and calculates the distance between the cartons and the truck unloader 100 to advance the truck unloader so as to perform the sweeping motion.

[0082] At block 404, the distance is communicated to the sweep controller (ie, the automation controller 170). Based on the calculated distance, the truck unloader 100 advances. When advancing to the distance, a sweep operation is performed by the front portion 142 of the truck unloader 100.

[0083] The truck unloader 100 continues to perform the sweeping operation based on the distance, while at box 404, the sweep controller monitors the torque feedback during each sweeping operation. In this regard, the closed-loop solution allows the truck unloader 100 to continuously perform the sweeping operation based on the captured image (e.g., visual feedback) and the torque feedback. The torque feedback is to ensure that the cartons are not damaged in any way during the sweeping operation. According to one embodiment, the torque feedback is monitored by comparing to a predefined torque threshold to determine whether the sweeping operation is feasible. According to one embodiment, the distance transmitted to the sweep controller at box 404 is monitored for a predefined distance threshold to determine whether the sweeping operation is feasible. In combination Figure 5 The flowchart of FIG. 19 illustrates the feasibility check performed by the closed loop solution during the sweep operation. In addition, when the sweep operation is not feasible, the decision system 191 provides a decision to stop the sweep operation or perform a pickup operation or shut down the truck unloader 100 at box 406. Such a decision is made by the decision system 191 in response to the feasibility check, which is combined with the feasibility check. Figure 5 The flowchart is described below.

[0084] Figure 5 The embodiment according to the present subject matter is shown by Figure 1 At block 502, the processor subsystem 158 connected to the detection system 166 receives images from the sensors 176a, 176b and based on the Figure 2The exemplary embodiment disclosed in processes an image to dynamically calculate a distance from at least a portion of the truck unloader 100 to the carton stack. In some examples, the at least a portion of the truck unloader 100 is a front portion 142 of the truck unloader 100. In some examples, the image indicates an unloading environment. In some examples, the distance is iteratively calculated before each sweep cycle of the truck unloader 100.

[0085] At box 504, the processor subsystem 158 may compare the distance with a predefined distance threshold before performing the sweep operation to determine the feasibility of performing the sweep operation. For example, the predefined distance threshold may be zero. If the calculated distance is greater than zero, the sweep operation may be performed. In some examples, when the processor subsystem 158 cannot detect clusters within the point cloud data from the sensors 176a, 176b, the processor subsystem 158 may not be able to calculate the distance from the truck unloader 100 to the cluster because there is no cluster (i.e., no cartons). In this case, the distance output from the processor subsystem 158 may be zero or less than zero. In this regard, the sweep controller (i.e., the automation controller 170) may not perform the sweep operation, and control may be relayed to the decision system 191. In some examples, when the truck unloader 100 is closer to the wall of the truck trailer 108, the processor subsystem 158 is able to detect only one cluster indicating the wall. In this case, the distance output from the processor subsystem 158 may be zero or less than zero or greater than zero. If it is determined that the distance output is zero or less than zero, control may be relayed to the decision system 158 and the sweep operation may not be performed.

[0086] At box 506, if the predefined distance threshold is met at box 504, the processor subsystem 158 may transmit instructions to the sweep controller (i.e., the automation controller 170) to perform a sweep operation by advancing the truck unloader 100 based on the calculated distance. When the truck unloader 100 advances the calculated distance to perform the sweep operation, the torque output when unloading cartons during the sweep operation is also monitored at box 508. If it is determined that the torque output of the truck unloader 100 exceeds the predefined torque threshold, control is relayed back to the decision system 191. In other words, the front portion 142 of the truck unloader 100 is configured to continuously perform the sweep operation based on the calculated distance until the torque input exceeds the predefined torque threshold when the calculated distance is greater than the predefined distance threshold. The predefined torque threshold may be a maximum torque value, and if the maximum torque value is exceeded, the truck unloader 100 may cause damage to the front portion 142 or the unloaded cartons.

[0087] As discussed in the previous example, when the truck unloader 100 is closer to the wall of the truck trailer 108, the processor subsystem 158 detects only one cluster indicating the wall. In this case, the distance output from the processor subsystem 158 may be zero, less than zero, or greater than zero. If it is determined that the distance output is greater than zero, the sweep controller may perform a sweep operation, however, when the truck unloader 100 attempts to advance to the wall and perform the sweep operation, the torque output may exceed the predefined torque threshold. In other words, the front portion 142 of the truck unloader 100 is configured to continuously perform a sweep operation based on the calculated distance until it encounters the wall of the truck trailer 108 when the calculated distance exceeds the predefined distance threshold.

[0088] As previously described, at box 510, when the distance is less than a predefined threshold or the torque output is greater than a predefined threshold, control is relayed to the decision system 191. At box 512, the decision system 191 may decide whether to perform a pick-up operation or a sweep operation.

[0089] In some examples, the decision system 191 may execute a machine learning algorithm or a segmentation algorithm or a combination of the two to decide whether to perform a pick-up operation or a sweep operation. In some examples, the machine learning algorithm may include a machine learning model trained using multiple images indicating the posture of the robot arm 102 and the condition of the carton stack. The decision system 191 may provide the current image from the sensors 176a, 176b to the machine learning model, which may compare the image with the trained data set and output an instruction for performing a pick-up operation or a sweep operation of the truck unloader 100 or stopping its operation.

[0090] In some examples, the segmentation algorithm can use a bottom-top or top-bottom clustering method to divide the point cloud data into clusters to identify the height of each carton (i.e., the Y coordinate of the data point of the point cloud), and then sum it to find the height of the carton stack from the bottom plate. The decision system 191 may include a memory that stores the operations to be performed based on the height of the carton stack in the form of a lookup table. For example, if the height of the carton stack is less than 6 feet, a sweeping operation is performed, or if the height of the carton stack is greater than 6 feet, a picking operation is performed. Therefore, the height of the cluster (i.e., the carton in the carton stack) can be compared with the predefined height stored in the memory to decide whether to perform a sweeping operation or a picking operation. In this regard, the decision system 191 can decide whether to perform a picking operation or a sweeping operation, or decide to stop the operation of the truck unloader in response to an infeasible condition for performing a sweeping operation. In this regard, the feasibility check of the continuous sweeping operation for the truck unloader 100 is performed in real time or near real time using the torque value and the distance value so that the cartons are effectively unloaded without suffering any damage.

[0091] Figure 6 Exemplary components of a material handling system 600 including a truck unloader 601 suitable for use with various embodiments are shown. The truck unloader 601 may include an external monitor 602, a network interface module 604, an HMI module 606, an input / output module (I / O module 608), a robotic arm and conveyor system 615 including a drive / safety module 612 and a motion module 614, a programmable logic controller (or PLC 618), a base motion module 620 including a vehicle controller module 622 and a manual control module 624, and a vision system 626 (or visualization system) that may include one or more personal computing devices 628 (or "PC") and sensor devices 630. In some embodiments, the vision system 626 of the truck unloader 601 may include a PC 628 connected to each sensor device 630. In embodiments where there is more than one sensor device 630 on the truck unloader 601, the PCs 628 of each sensor device 630 may be networked together, and one of the PCs 628 may operate as a master PC 628, which receives data from other connected PCs 628, may perform data processing (e.g., coordinate transformations, deduplication, error checking, etc.) on the received data and its own data, and may output the combined and processed data from all PCs 628 to the PLC 618. In some embodiments, the network interface module 604 may not have a PLC inline between itself and the PCs 628, and the PLC 618 may act as a vehicle controller and / or drive / safety system. The sensor devices 630 may include 2D image capture devices (ICDs) 631 and 3D image capture devices (ICDs) 633, which are separated into sections for different viewing sections or vantage points. Subsets may include rear mounted sensors 635, end effector mounted sensors 637, and vehicle mounted sensors 639. The truck unloader 601 may also include a processor subsystem 158, which may be part of the vehicle controller 622. In some examples, the processor subsystem 158 may communicate with the vision system 626 to receive one or more images and generate a corresponding image based on the image. Figure 2 The described exemplary embodiments process these images.

[0092] The processor subsystem 158 may be a functional component of the controller 622 comprised of one or more central processing units (CPUs), digital signal processors (DSPs), and processor memory. The processor subsystem 158 may include any tool or collection of tools for computing, classifying, processing, transmitting, receiving, retrieving, initiating, switching, storing, displaying, indicating, detecting, recording, reproducing, applying, or utilizing any form of information, intelligence, or data for business, scientific, control, or other purposes, including controlling automated equipment for material handling systems.

[0093] According to various aspects of the present disclosure, an element, or any portion of an element, or any combination of elements may be implemented with a processor subsystem 158, which includes one or more physical devices having a processor. Non-limiting examples of processors include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), programmable logic controllers (PLCs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform various functions described throughout the present disclosure. One or more processors in the processing subsystem 158 may execute instructions. A processing subsystem 158 that executes instructions to produce a result is a processing system that is configured to perform tasks that lead to the result, such as by providing instructions to one or more components of the processing subsystem 158 that will cause these components to perform actions that, by themselves or in combination with other actions performed by other components of the processing subsystem 158, will lead to the result.

[0094] The truck unloader 601 may also include a decision system 191 that can be communicatively connected to the processor subsystem 158. In some examples, the decision system 191 may be installed or configured as a subsystem within the vehicle controller 622. The decision system 191 may decide whether to perform a pick-up operation or a sweep operation, and transmit the decision to the vehicle controller 622. The decision system 191 may be a computer system or a cloud computing system that includes one or more processors and a memory / database to execute one or more algorithms, such as, but not limited to, image processing algorithms, machine learning algorithms, etc. Figure 5 The functions performed by the decision making system 191 are described. The truck unloader 601 can communicate with the truck via a network 603 such as a local Wi-Fi network. TM The network is connected to a network interface module 604 (e.g., Wi-Fi TMIn particular, the network interface module 604 can enable the truck unloader 601 to be connected to an external monitor 602. The external monitor 602 can be any one of a remote warehouse or distribution center control room, a handheld controller, or a computer, and can provide passive remote viewing through the visual system 626 of the truck unloader 601. Alternatively, the external monitor 602 can replace the programming inherent in the visual system 626 and assume the active command and control of the truck unloader 601. The programming of the truck unloader 601 can also be communicated, operated, and debugged through an external system such as the external monitor 602. Examples of external monitors 602 that assume command and control may include remotely located operators or remote systems, such as warehouse or distribution server systems (i.e., remote devices as described above). Exemplary embodiments of using the external monitor 602 to assume command and control of the truck unloader 601 may include manual or computer intervention in the movement of the truck unloader 601, such as moving from one unloading bay to another, or having the external monitor 602 assume control of a robotic arm to remove items that are difficult to unload with an automated routine (e.g., boxes, cartons, etc.). External monitor 602 may include any of the following: a visual monitor, a keyboard, a joystick, an I / O port, a CD reader, a computer, a server, a handheld programming device, or any other device that may be used to perform any portion of the above-described embodiments.

[0095] The truck unloader 601 may include a human-machine interface module 606 (or HMI module 606) that can be used to control and / or receive output information for the robot arm and conveyor system 615 and / or the basic motion module 620. The HMI module 606 can be used to control (or can include itself) a joystick, display, and keypad that can be used to reprogram, override the automatic control of the machine, and drive the truck unloader 601 from point to point. Actuators 609, which can be actuated by the vision system 626 via the I / O module 608 alone or in any combination, and distance sensors 610 can be used to help guide the truck unloader 601 into the unloading area (e.g., trailer). The I / O module 608 can connect the actuator 609 and the distance sensor 610 to the PLC 618. The robot arm and conveyor system 615 may include all the components required to move the arm and / or conveyor, such as the drive / engine and motion protocol or control. The basic motion module 620 can be a component for moving the entire truck unloader 601. In other words, the base motion module 620 may be the component required to guide the vehicle in and out of the unloading area.

[0096] The PLC 618 can control the overall electromechanical motion of the truck unloader 601 or control exemplary functions, such as controlling a robotic arm or conveyor system 615. For example, the PLC 618 can move the manipulator head of the robotic arm to a position for picking up an item (e.g., a box, a carton, etc.) from an item wall. As another example, the PLC 618 can control the activation, speed, and direction of the kick roller rotation, and / or control a front rack conveyor (such as the front 142 ( Figure 1 ))'s support mechanism. The PLC 618 and other electronic components of the vision system 626 can be installed in an electrical box (not shown) located below the conveyor, near the conveyor, or elsewhere on the truck unloader 601. The PLC 618 can operate all or part of the truck unloader 601 autonomously and can receive position information from distance sensors (not shown). The I / O module 608 can connect the actuator and distance sensor 610 to the PLC 618.

[0097] The truck unloader 601 may include a vision system 626 that includes a sensor device 630 (e.g., a camera, a 3D sensor, etc.) and one or more computing devices 628 (referred to as personal computers or "PCs" 628). The truck unloader 601 may scan in front of the truck unloader 601 in real time or near real time using the sensor device 630 and the one or more PCs 628 of the vision system 626. A forward scan may be triggered by the PLC 618 in response to determining that the truck unloader 601 is in a position to begin detecting cartons in the unloading area, such as a trigger sent in response to the truck unloader 601 being in a position to begin detecting cartons in the unloading area. The forward scanning capability may be used for collision avoidance, sent to human shape recognition (safety), determining the size of the unloading area (e.g., truck or trailer), and for scanning the floor of the unloading area for loose items (e.g., cartons, boxes, etc.). The 3D capability of the vision system 626 may also provide depth perception, edge recognition, and may create a 3D image of a wall of items (or a pile of cartons). Vision system 626 may operate alone or in conjunction with PLC 618 to identify the edges, shapes, and near / far distances of items in front of truck unloader 601. For example, the edges and distances of each individual carton in a wall of items may be measured and calculated relative to truck unloader 601, and vision system 626 may operate alone or in conjunction with PLC 618 to select specific cartons for removal.

[0098] In some embodiments, the vision system 626 may provide the PLC with information such as the specific XYZ coordinate location of the cartons intended to be removed from the unloading area and one or more motion paths traveled by the robotic arm or mobile body of the truck unloader 601. The PLC 618 and vision system 626 can work independently or together, such as an iterative movement and visual inspection process for carton visualization, initial homing, and motion accuracy checks. The same process can be used as an accuracy check during vehicle movement or during carton removal. Alternatively, the PLC 618 can use the movement and visualization process as a check to see if one or more cartons have fallen off or repositioned from the carton stack since the last visual inspection. Although Figure 6 Various computing devices and / or processors in the vehicle such as PLC 618, vehicle controller module 622 and PC 628 have been described separately, but in the context of Figure 6 In the various embodiments discussed and all other embodiments described herein, the described computing devices and / or processors may be combined, and the operations described herein that are performed by separate computing devices and / or processors may be performed by fewer computing devices and / or processors, such as a single computing device or processor having different modules that perform the operations described herein. For example, different processors combined on a single circuit board may perform the operations described herein that are attributed to different computing devices and / or processors, a single processor running multiple threads / modules may perform the operations described herein that are attributed to different computing devices and / or processors, etc.

[0099] The extendable conveyor system 632 can transfer items from the truck unloader 601 to other parts of the material handling system 600. When the truck unloader 601 moves forward or backward, the visual device 634 on one of the extendable conveyor system 632 and the truck unloader 601 can image the target 636 on the other. The visual system 626 can perform image processing to detect changes in the size, direction, and position of the target 636 within the field of view of the visual device 636. The device interfaces 638, 640 of the extendable conveyor system 632 and the truck unloader 601 can transmit visual information or motion commands. For example, the PLC 618 can command the extension motion actuator 642 on the extendable conveyor system 632 to correspond to the movement of the truck unloader 601 to keep the extendable conveyor system 632 and the truck unloader 601 aligned and maintain the appropriate spacing. In one embodiment, the device interfaces 638, 640 utilize a short-range wireless communication protocol such as a personal access network (PAN) protocol. Examples of PAN protocols that can be used in various embodiments include IEEE 802.15.4 and Wireless communication protocols and standards.

[0100] In some example embodiments, some of the operations herein may be modified or further amplified as described below. In addition, in some embodiments, additional optional operations may also be included. It should be understood that each of the operations in the modifications, optional additions, or amplifications described herein may be included in the operations herein, either individually or in combination with any other of the features described herein.

[0101] Figure 7 A method for using a Figure 1 An exemplary method for performing a sweeping operation on a truck unloader of FIG. As previously described, the truck unloader can perform a continuous sweeping operation based on visual feedback and torque feedback. The visual feedback and torque feedback are processed by the detection system and the processor subsystem. The detection system 166 and the processor subsystem 158 process the visual feedback, such as in combination with Figure 2 The detection system 166 and the processor subsystem 158 may be communicatively coupled to the controllers 144, 170 and one or more sensors of the truck unloader to perform a continuous sweep operation.

[0102] Thus, at step 702, the detection system 144 receives a plurality of images from one or more sensors on the truck unloader. The one or more sensors may be 2D or 3D cameras as part of a vision system. The one or more sensors capture images from a plurality of fields of view based on the posture of the robot arm during the unloading operation. In some examples, the images may indicate an unloading environment, which may include a carton stack, a portion of a truck unloader, a carton stack container, etc.

[0103] In addition, at step 704, the processor subsystem 158 may calculate the distance the truck unloader must advance to perform the sweep operation based on the image. The distance indicates the distance the truck unloader must advance to perform the sweep operation. Figure 2 The disclosed accumulation, filtering and clustering techniques process the image to calculate the distance. The processor subsystem 158 transmits the distance to the controller to advance the truck unloader to the calculated distance from its current position. The sweep operation is performed by advancing the truck unloader to the calculated distance. The above-described combination of the above-described distance threshold and the above-described torque threshold may be performed before the truck unloader is advanced to the calculated distance. Figure 5 The feasibility check described above.

[0104] When the sweeping operation is initiated, at step 706, a torque input is received as feedback from one or more force / torque sensors on the truck unloader. At step 708, the torque input is compared to a predefined torque threshold. In some examples, the comparison may be performed before calculating the next distance for the truck unloader to advance, and the sweeping operation is performed based on the previously calculated distance. When the torque input does not exceed the predefined torque threshold, then at step 710, the processor subsystem 158 transmits a first instruction for performing the sweeping operation. In this case, at step 708, the feasibility check on the distance may also be verified by comparing the calculated distance to a predefined distance threshold. At step 708, at least one but preferably two conditions are verified, such as a distance threshold and a torque threshold.

[0105] Alternatively or additionally and in some examples, the processor subsystem 158 may refrain from transmitting any instructions to the controller for performing a sweeping operation until the torque limit exceeds a predefined torque threshold. In some examples, the processor subsystem 158 may refrain from transmitting any instructions to the controller for performing a sweeping operation until the calculated distance value drops below a predefined distance threshold.

[0106] Thereafter, a sweep operation is performed by checking at least one but preferably two conditions at 710, and the processor subsystem begins calculating another distance for advancing the truck unloader for the next sweep operation at step 712. In this regard, the distance is recursively calculated prior to each sweep operation (i.e., prior to advancing the truck unloader forward toward the carton stack for the next sweep cycle).

[0107] Referring back to step 708, when it is determined that the torque input exceeds the predefined torque threshold, then at step 714, the processor subsystem determines whether to perform a pick-up operation or a sweeping operation. In some examples, the processor subsystem 158 may instruct the controller to stop performing a sweeping operation until a decision is made to perform a pick-up operation or a sweeping operation. The decision may be based on torque values, distance values, or image data currently or previously processed by the processor subsystem 158.

[0108] Referring back to step 708, when it is determined that the torque input exceeds the predefined torque threshold and further when it is determined that the distance is less than the predefined distance threshold, then at step 716, the processor subsystem transmits a second instruction for performing a pick-up operation. In other words, if both conditions based on the torque input and the calculated distance fail, the processor subsystem may decide to perform a pick-up operation instead of a sweep operation. In some examples, the processor subsystem may transmit the values ​​of the distance, torque, and image data to the decision system 191 to provide a decision whether to sweep or pick up. In this regard, the truck unloader may perform both a sweep operation and a pick-up operation by continuously monitoring and comparing the distance value and the torque value.

[0109] The various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above according to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the whole system. The technician can implement the described functions in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing deviations from the scope of the present invention.

[0110] Hardware for implementing the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the aspects disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination designed to perform the functions described herein. A general purpose processor may be a microprocessor, but, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuits specific to a given function.

[0111] In one or more exemplary aspects, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored as one or more instructions or codes on a non-transient computer-readable medium or a non-transient processor-readable storage medium. The steps of the method or algorithm disclosed herein may be embodied in a processor-executable software module (or processor-executable instructions), which may reside on a non-transient computer-readable or processor-readable storage medium. A non-transient computer-readable or processor-readable storage medium may be any storage medium accessible by a computer or processor. As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage devices, disk storage devices or other magnetic storage devices, or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. As used herein, disks and optical disks include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks, wherein disks typically copy data magnetically, while optical disks copy data optically using lasers. The above combinations are also included within the scope of non-transient computer-readable and processor-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0112] Although various embodiments according to the principles disclosed herein have been shown and described above, modifications may be made thereto by those skilled in the art without departing from the essence and teachings of the present disclosure. The embodiments described herein are representative only and are not intended to be limiting. Many variations, combinations and modifications are possible and are within the scope of the present disclosure. Alternative embodiments obtained by merging, integrating and / or omitting the features of one or more embodiments are also within the scope of the present disclosure. Therefore, the scope of protection is not limited by the description given above, but by the following claims, which include all equivalents of the subject matter of the claims. Each claim is incorporated into the specification as further disclosure, and the claims are one or more embodiments of one or more of the present invention. In addition, any of the above advantages and features may be associated with a specific embodiment, but the application of such published claims should not be limited to processes and structures that achieve any or all of the above advantages or have any or all of the above features.

[0113] In addition, without departing from the scope of the present disclosure, the techniques, systems, subsystems and methods described and shown as discrete or independent in various embodiments may be combined or integrated with other systems, modules, techniques or methods. Other items shown or discussed as being directly coupled or communicating with each other may be indirectly coupled or communicated through an interface, device or intermediate component, whether such coupling or communication is performed electrically, mechanically or otherwise. Other examples that may be determined by those skilled in the art and that may be varied, replaced and altered without departing from the spirit and scope disclosed herein.

[0114] Those skilled in the art to which the present invention belongs will think of many modifications and other embodiments of the present invention set forth herein, which have the benefit of the teachings presented in the foregoing description and the associated drawings. Although the figures show only certain components of the apparatus and system described herein, it should be understood that various other components may be used in conjunction with the supply management system. Therefore, it should be understood that the present invention is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. For example, various elements or components may be combined or integrated into another system, or certain features may be omitted or not implemented. In addition, the steps in the above method may not necessarily occur in the order depicted in the drawings, and in some cases, one or more of the depicted steps may occur substantially simultaneously, or may involve additional steps. Although specific terms are employed herein, they are used only in a general and descriptive sense, and not for purposes of limitation.

[0115] As will be understood, any such computer program instructions and / or other types of code may be loaded onto the circuitry of a computer, processor, or other programmable device to produce a machine, such that the computer, processor, or other programmable circuitry that executes the code on the machine forms a device for performing various functions, including those described herein.

[0116] It should also be noted that all or some of the information presented by the exemplary displays discussed herein may be based on data received, generated, and / or maintained by one or more components of a local or networked system and / or circuit. In some embodiments, one or more external systems (such as remote cloud computing and / or data storage systems) may also be utilized to provide at least some of the functionality discussed herein.

[0117] As described above and based on the present disclosure, embodiments of the present invention may be configured as methods, personal computers, servers, mobile devices, back-end network devices, etc. Therefore, embodiments may include various devices that completely include hardware or include any combination of software and hardware. In addition, embodiments may take the form of a computer program product on at least one non-transient computer-readable storage medium, which has computer-readable program instructions (e.g., computer software) embodied in a storage medium. Any suitable computer-readable storage medium may be utilized, including a non-transient hard disk, a CD-ROM, a flash memory, an optical storage device, or a magnetic storage device.

[0118] The foregoing description of the embodiments has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. According to the above teachings, obvious modifications or changes can be made. The embodiments are selected and described in order to best illustrate the principles of the present invention and its practical application, so that those of ordinary skill in the art can best utilize the present invention in various embodiments and with various modifications suitable for the intended specific use. Although only a limited number of embodiments of the present invention are explained in detail, it should be understood that the scope of the present invention is not limited to the details of the construction and arrangement of the components set forth in the preceding description or shown in the accompanying drawings. The present invention can have other embodiments and can be practiced or implemented in various ways. In addition, when describing the embodiments, for the sake of clarity, specific terms are used. It should be understood that each specific term includes all technical equivalents that operate in a similar manner to achieve similar purposes.

Claims

1. A truck unloader for unloading cartons from a stack of cartons, wherein the truck unloader comprises: Robotic arm; a front part, said front part comprising a front roller, a conveyor and one or more sensors, wherein the front portion is movably positioned and configured to perform a sweeping operation, wherein to perform the sweeping operation, the front portion is configured to sweep under one or more cartons of the carton stack and pull the one or more cartons onto the conveyor, and wherein the one or more sensors are force sensors positioned on the front roller and configured to measure a torque applied to the front roller when the front portion contacts the carton stack; A controller, the controller comprising: a detection system configured to periodically receive one or more images from the vision system; A processor subsystem connected to the detection system, the processor subsystem being configured to: calculating a distance from at least a portion of the truck unloader to the stack of cartons based on the one or more images; If the distance satisfies a predefined distance threshold, advancing the truck unloader based on the distance to perform a sweeping operation; and A torque input is monitored while performing the sweeping operation, the torque input being feedback received from the one or more sensors, wherein the sweeping operation is configured to continue until the torque input exceeds a predefined torque threshold.

2. The truck unloader of claim 1, wherein the one or more sensors are torque sensors.

3. The truck unloader of claim 1, wherein during each sweep operation, the value of the torque input varies based on one of a carton size or a packaging arrangement of the carton stack or a wall of a carton stack container. 4 . The truck unloader of claim 1 , wherein the vision system comprises one or more sensors respectively attached to one of the mobile body or the robotic arm to provide a two-dimensional image and a three-dimensional point cloud. 5 . The truck unloader of claim 1 , wherein the processor subsystem is further configured to perform a transformation from a camera coordinate system to a robot coordinate system on the three-dimensional point cloud data to calculate the distance. 6 . The truck unloader of claim 5 , wherein the processor subsystem is further configured to perform Euclidean clustering on the transformed three-dimensional point cloud data to calculate the distance. 7 . The truck unloader of claim 1 , wherein the front portion is further configured to cease the sweeping operation when the torque input exceeds the predefined torque threshold.

8. The truck unloader of claim 5, wherein the processor subsystem further comprises: A point cloud accumulation module is provided for accumulating the three-dimensional point cloud data of the one or more sensors.

9. The truck unloader of claim 1, wherein the processor subsystem further comprises: A segmentation module and a distance generation module are configured to calculate a distance between the front of the truck unloader and a portion of the stack of cartons.

10. The truck unloader of claim 1, wherein the processor subsystem is further configured to estimate a pose of the robotic arm and the front portion of the truck unloader using forward kinematics.

11. The truck unloader according to claim 1, wherein: The processor subsystem is further configured to perform a pick-up operation based on at least one of the distance and the torque input.

12. A method for controlling a truck unloader to unload a group of cartons, the method comprising receiving, by a detection system, a plurality of images from one or more sensors on the truck unloader, calculating, by a processor subsystem coupled to the detection system, a distance to advance the truck unloader to perform a sweeping operation based on the plurality of images, in, The sweeping operation is performed by the front of the truck unloader, and wherein, in order to perform the sweeping operation, the front portion sweeps under the group of cartons and pulls the group of cartons onto the conveyor; If the calculated distance is greater than a predefined distance threshold, the processor subsystem transmits a first instruction to advance the truck unloader based on the distance to perform a sweep operation; receiving a torque input as feedback by the processor subsystem while the truck unloader advances toward the group of cartons based on the distance to perform a sweeping operation, wherein the torque is received from a force sensor positioned on a front roller of the front portion and configured to measure the torque applied to the front roller when the front portion contacts the group of cartons; wherein, in response to the processor subsystem determining that the torque input is within a predefined torque threshold, a sweep operation is continuously performed based on a calculated distance, wherein the distance is iteratively calculated prior to each sweep cycle; and wherein, in response to the processor subsystem determining that the torque input exceeds the predefined torque threshold, the processor subsystem determines whether to perform a pick-up operation or the sweep operation; or When the distance is less than a predefined distance threshold, the processor subsystem transmits a second instruction for performing a pick operation.

13. The method according to claim 12, wherein: Determining that the torque input is within the predefined torque threshold further comprises: determining whether the distance exceeds the predefined distance threshold; In response to determining that the torque input is within the predefined torque threshold and the distance exceeds the predefined distance threshold, performing a second sweeping operation; and In response to determining that the torque input exceeds the predefined torque threshold or the distance is below the predefined distance threshold, it is determined whether to perform a third sweep operation or the pick-up operation.

14. The method according to claim 12, wherein: Torque is received from a torque sensor.