Machine learning based decision making for robotic material handling

Abstract

A method for controlling a robotic article handler is described. The method includes obtaining first point cloud data related to a first three-dimensional (3D) image and second point cloud data related to a second 3D image, the first and second 3D images captured by first and second sensor devices, respectively. Further, the method can include transforming the first and second point cloud data into combined point cloud data used as input to a convolutional neural network to construct a machine learning model. The machine learning model can output a decision classification indicating a first probability associated with a first operating mode and a second probability associated with a second operating mode. Further, the method can include operating the robotic article handler according to the first or second operating mode based on a comparison of the first and second probabilities.

Description

TECHNICAL FIELD

[0001] The example embodiments described herein relate generally to systems, methods, and apparatus for performing machine learning based decision making for machine physical goods handling, and more particularly to performing machine learning based decision making for machine physical goods loading and / or unloading. BACKGROUND

[0002] Generally, in a material handling site such as, but not limited to, a distribution center, warehouse, storage, or shipping center, various machines such as conveyors, pallet jack, robotic arms, truck loaders / unloaders, and / or conveyor top mount units are used to perform various operations. For example, vehicles (e.g., trucks, trailers, etc.) loaded with goods, e.g., shipments, products, transports, etc., are used to deliver products to these material handling sites. Today, to achieve lower overhead costs at retail stores, the in-store product quantity at distribution centers has been reduced and the products being transported on vehicles (e.g., trucks, trailers, etc.) are also counted as part of the available store inventory of the inventory. Typically, in a material handling environment, if the containers are individual pieces, then manual labor can be used to load / unload the vehicles, or if the containers are palletized, then a forklift can be used to load / unload the vehicles, or typically by using other truck loader / unloader units. Using manual labor personnel to load or unload and load large vehicle transports can be physically difficult and can be expensive due to the time and labor involved. Therefore, automating the loading / unloading of such vehicles quickly at the transport loading / unloading docks of the material handling sites as a way to refill dwindling inventory has gained new prominence. Thus, there is a great need for an improved loading / unloading system that can unload / load large stacks of boxes and goods from these vehicles faster and more efficiently than manual labor personnel and more cost effectively. SUMMARY

[0003] Various example embodiments described herein relate to a method for controlling a robotic article handler. The method can include obtaining first point cloud data related to a first three-dimensional image captured by a first sensor device of the robotic article handler. Further, the method can include obtaining second point cloud data related to a second three-dimensional image captured by a second sensor device of the robotic article handler. Further, the method can include transforming the first point cloud data and the second point cloud data to generate combined point cloud data. The method can also include constructing a machine learning model based on the combined point cloud data. In this regard, the combined point cloud data can be provided as an input to a convolutional neural network. Further, the method can include outputting, by the machine learning model, a decision classification indicative of a first probability associated with a first operating mode and a second probability associated with a second operating mode. Further, the method can include operating the robotic article handler according to the first operating mode in response to the first probability being higher than the second probability, and operating the robotic article handler according to the second operating mode in response to the second probability being higher than the first probability.

[0004] In some example embodiments, the first sensor device can be coupled to a robotic arm of the robotic article handler, and the second sensor device can be coupled to a platform of the robotic article handler.

[0005] According to some example embodiments, the robotic article handler can be a robotic carton unloader, which can be configured to perform at least one of the following operations: loading and unloading of articles.

[0006] In some example embodiments, the first operating mode can be associated with picking up an article by grasping the article using an end effector of a robotic arm of the robotic article handler. Further, the second operating mode can be associated with sweeping a pile of articles with a platform of the robotic article handler from an article docking station.

[0007] According to some example embodiments, the method can further include evaluating a selection of an operating mode of the robotic article handler based on the decision classification output by the machine learning model and a predefined heuristic associated with past operations of the robotic article handler. Further, the method can include controlling the robotic article handler based on the operating mode selected according to the evaluation.

[0008] According to some example embodiments, the method can include adjusting a first weight associated with the decision classification and a second weight associated with the predefined heuristic based on a performance associated with outputs of the machine learning model over a period of time, the adjustment for the evaluation of the selection.

[0009] Some example embodiments described herein relate to a robotic article unloader. The robotic article unloader can include a vision system, a platform, a robotic arm, and a processing unit. The processing unit can be communicatively coupled to the vision system, the platform, and the robotic arm, respectively. The vision system can include a first sensor device and a second sensor device. The first sensor device can be positioned at a first location on the robotic article unloader, and the second sensor device can be positioned at a second location on the robotic article unloader. Further, the platform of the robotic article unloader can include a conveyor. The platform can be configured to operate in a first operational mode. Further, the robotic arm of the robotic article unloader can include an end effector that can be configured to operate in a second operational mode. According to the example embodiments described, the processing unit can be configured to obtain first point cloud data related to a first three-dimensional image captured by the first sensor device. Further, the processing unit can be configured to obtain second point cloud data related to a second three-dimensional image captured by the second sensor device. Further, the processing unit can be configured to transform the first point cloud data and the second point cloud data to generate combined point cloud data. Further, the combined point cloud data can be used as an input to a convolutional neural network to build a machine learning model. Further, the processing unit can be configured to output a decision classification from the machine learning model. The decision classification can indicate a probability distribution associated with a first probability and a second probability. In this regard, the first probability can be associated with a selection of the first operational mode, and the second probability can be associated with a selection of the second operational mode. Further, the processing unit can be configured to control the robotic article unloader based on the first operational mode or the second operational mode.

[0010] According to the example embodiments described, the processing unit can be configured to control the robotic article unloader by operating the platform of the robotic article unloader according to the first operational mode in response to the first probability being higher than the second probability. Further, the processing unit can be configured to control the robotic article unloader by operating the robotic arm of the robotic article unloader according to the second operational mode in response to the second probability being higher than the first probability.

[0011] According to some example embodiments described herein, in the first operational mode, a portion of the platform of the robotic article unloader can be configured to sweep across a pile of articles, thereby directing one or more articles in the pile of articles onto the conveyor. Further, in some example embodiments, in the second operational mode, a portion of the robotic arm of the robotic article unloader can be configured to pick up an article by using the end effector of the robotic arm to grasp a portion of the article.

[0012] According to some example embodiments, a processing unit of the robotic article unloader can be configured to evaluate the selection of the first operational mode and the second operational mode based on the first probability and the second probability output by the machine learning model and a predefined heuristic associated with past operations of the robotic article unloader. Further, the processing unit can be configured to control the robotic article unloader based on the evaluation of the selection.

[0013] According to some example embodiments, the first sensor device and the second sensor device can include at least one depth camera and a color camera. Further, the first sensor device can be coupled to a robotic arm of the robotic article unloader and the second sensor device can be coupled to a platform of the robotic article unloader.

[0014] According to some example embodiments, to perform at least one of the following operations: loading and unloading the plurality of articles from the article docking station, a processing unit of the robotic article unloader can be configured to generate first commands for operating the robotic arm according to the first operational mode and generate second commands for operating the platform according to the second operational mode.

[0015] According to some example embodiments, a processing unit of the robotic article unloader can be configured to further adjust a first weight associated with the decision classification and a second weight associated with the predefined heuristic based on performance associated with outputs of the machine learning model over a period of time, the adjustment for the evaluation of the selection.

[0016] Some example embodiments described herein relate to a non-transitory computer- readable medium having stored thereon computer-executable instructions that, in response to execution by a processor, perform operations. The operations can include obtaining first point cloud data related to a first three-dimensional image captured by a first sensor device of a robotic article handler. Further, the operations can include obtaining second point cloud data related to a second three-dimensional image captured by a second sensor device of the robotic article handler. Further, the operations can include transforming the first point cloud data and the second point cloud data to generate combined point cloud data. Further, the operations can include constructing a machine learning model by using the combined point cloud data as input to a convolutional neural network. The operations can also include outputting, by the machine learning model, a decision classification indicating a probability associated with an operational mode of the robotic article handler. Further, the operations can include generating first commands for operating the robotic article handler based on the decision classification.

[0017] According to some example embodiments, the non-transitory computer- readable medium can further store thereon computer-executable instructions that, in response to execution by the processor, perform operations that can include evaluating a selection of an operating mode of the robotic material handling machine based on the decision classification and a predefined heuristic associated with past operations of the robotic material handling machine. Further, the operations can include generating a second command for operating the robotic material handling machine based on the evaluation of the selection. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings illustrate embodiments of the application and, together with the description (including the above general description and the detailed description below), serve to explain its features.

[0019] Figure 1 A schematic diagram of a system including a robotic material handling machine according to example embodiments is shown;

[0020] Figure 2 An example scenario of decision making based on a machine learning model for selecting an operating mode of a robotic material handling machine according to example embodiments is shown.

[0021] Figure 3 A component block diagram of elements of a robotic material handling machine according to example embodiments is shown;

[0022] Figure 4 An example architecture of a machine learning model for making decisions for operating modes to control a robotic material handling machine according to example embodiments is shown.

[0023] Figure 5 A schematic diagram of a robotic arm of a robotic material handling machine in operation according to a first operating mode according to example embodiments is shown;

[0024] Figure 6 A flowchart representing a method for controlling a robotic material handling machine according to example embodiments is shown;

[0025] Figure 7 A flowchart representing a method for evaluating decisions made by a machine learning model for selecting an operating mode of a robotic material handling machine for operation according to example embodiments is shown;

[0026] Figure 8 An example scenario of generating a combined point cloud based on data from various sensor devices associated with a robotic material handling machine according to example embodiments is shown;

[0027] Figure 9 A perspective view of a robotic material unloader according to example embodiments described herein is shown.

[0028] Figure 10 a perspective view of a robotic article unloader having multiple sensor devices is shown in accordance with an example embodiment; and

[0029] Figure 11 a schematic view of an example computing device for use with a robotic article handler is shown in accordance with an example embodiment. DETAILED DESCRIPTION

[0030] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the inventions are shown. Indeed, these inventions can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like numbers refer to like elements throughout. As used herein, the terms such as "front," "rear," "top," "exterior," "interior," and the like, are used in the examples provided below for illustrative purposes to describe the relative positions of certain devices or portions of devices. The terms used in this patent are not meant to be limiting, and the devices described herein, or portions thereof, can be attached or utilized in other orientations.

[0031] The term "comprising" means including, but not limited to, and should be interpreted in the manner set out in the passage of the Patent Statutes which follows the term. It is understood that the use of broadening language such as "comprising", "having", "including", and "containing" presents support for narrower language such as "consisting of", "consisting essentially of", and "substantially comprised of".

[0032] The phrases "in one embodiment", "according to one embodiment", etc., generally mean that a particular feature, structure, or characteristic described in connection with the phrase is included in at least one embodiment of the invention, and can be included in more than one embodiment of the invention (importantly, such phrases are not necessarily referring to the same embodiment).

[0033] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0034] The term "about" or "approximately," or the like, when used in connection with a number, can mean that the number is precise or, alternatively, that the number is within a range that is near the precise number, as understood by those skilled in the art.

[0035] If the specification states that a component or feature is "may," "can," "should," "will," "preferably," "possibly," "usually," "optionally," "for example," "often," or "may" (or other such words) included or has that characteristic, then that particular component or feature is not necessarily included or has that characteristic. Such components or features may be optionally included in some embodiments or may be excluded.

[0036] As used herein, the term "transmitter" refers to any component that generates radio waves for communication purposes, while "receiver" is used generally to refer to any component that receives radio waves and converts that information into a usable form. "Transceiver" is used generally to refer to a component that can both generate and receive radio waves, and is therefore conceived when discussing transmitters or receivers.

[0037] This document uses the term "processor" to refer to any programmable microprocessor, microcomputer, or one or more multiprocessor chips that can be configured by software instructions (applications) to perform various functions including those described in the various embodiments above. In some devices, multiple processors may be provided, such as one dedicated to wireless communication functions and another dedicated to running other applications. Software applications may be stored in internal memory before being accessed and loaded into the processor. The processor may include sufficient internal memory to store the application software instructions. In many devices, the internal memory may be volatile or non-volatile memory such as flash memory or a combination of both. The memory may also be located within another computing resource (e.g., enabling computer-readable instructions to be downloaded via the Internet or another wired or wireless connection).

[0038] For the purposes of this specification, the general reference to "memory" means processor-accessible memory, including internal or removable memory inserted into a device and memory within the processor itself. For example, memory can be any non-transitory computer-readable medium on which computer-readable instructions (e.g., computer program instructions) executable by a processor are stored.

[0039] As used herein, the term "electronic device" refers to any or all of the following: handheld devices, mobile phones, wearable devices, personal data assistants (PDAs), tablet computers, smartbooks, PDAs, barcode readers, scanners, tag readers, imagers, radio frequency identification (RFID readers or interrogators), in-vehicle computers, wearable barcode scanners, wearable tag readers, point-of-sale (POS) terminals, headsets, programmable logic controllers (PLCs), programmable automation controllers (PACs), industrial computers, laptop computers, desktop computers, personal computers, and similar electronic devices equipped with at least one processor configured to perform the various operations described herein.

[0040] For the sake of brevity, this document uses the interchangeable terms "computing platform," "master device," "server," or "supervisory device" to describe various implementations. The term "server" may be used herein to refer to any computing device or a distributed network of computing devices capable of acting as a server (such as a primary switch server, web server, mail server, document server, or any other type of server). A server may be a dedicated computing device or a computing device that includes a server module (e.g., an application running an application that enables the computing device to operate as a server). A server module (e.g., a server application) may be a full-featured server module or a lightweight or auxiliary server module (e.g., a lightweight or auxiliary server application) configured to provide synchronization services to a dynamic database on the computing device. A lightweight or auxiliary server may be a simplified version of server-type functionality, implemented on a computing device such as a smartphone, enabling it to act as an internet server (e.g., an enterprise email server) only when necessary to provide the functionality described herein.

[0041] In some exemplary embodiments, a server may correspond to any of an industrial computer, a cloud-based platform, an external computer, a standalone computing device, etc. In some exemplary embodiments, a host device or computing platform may also refer to any electronic device as described herein. In some exemplary embodiments, a server may include an access point or gateway device that is capable of communicating directly with one or more electronic devices and also capable of communicating (directly or indirectly via a communication network such as the Internet) with a network establishment service (e.g., an Internet service provider). In some exemplary embodiments, a server may manage the deployment of one or more electronic devices throughout a physical environment. In some exemplary embodiments, a server may refer to a network establishment service comprising a distributed system, wherein multiple operations are performed by utilizing multiple computing resources deployed on a network and / or a cloud-based platform or cloud-based service (such as any of Software-Based Services (SaaS), Infrastructure-as-a-Service (IaaS), or Platform-as-a-Service (PaaS)).

[0042] In some exemplary embodiments, the term "server" may be used herein to refer to a programmable logic controller (PLC), a programmable automation controller (PCC), an industrial computer, a desktop computer, a personal data assistant (PDA), a laptop computer, a tablet computer, a smartbook, a handheld computer, a personal computer, a smartphone, a headset, a smartwatch, and similar electronic devices equipped with at least a processor configured to perform the various operations described herein. Devices such as smartphones, tablet computers, headsets, and smartwatches are generally collectively referred to as mobile devices.

[0043] The components shown in the figures represent components that may or may not be present in the various embodiments of the invention described herein, such that the embodiments may include fewer or more components than those shown in the figures without departing from the scope of the invention.

[0044] In material handling environments, items such as cartons, goods, shipments, parcels, and containers are typically moved from one location to another. For example, in some cases, these items are unloaded from a transport vehicle onto a conveyor in the material handling environment. In another example, these items are loaded from a conveyor onto the cargo compartment of a transport vehicle. Manually loading / unloading such items can be expensive, less productive, and inefficient. In this regard, robotic item handling equipment (e.g., robotic manipulators, robotic unloaders, etc.) is used in some environments to reduce human effort. Some examples of robotic item handling machines include end effectors (e.g., vacuum suction cups) that can pick up one or more items by gripping them. Furthermore, some examples of robotic item handling machines also include platforms that can be moved from one location to another. These platforms are configured to move such that a portion of the platform can sweep under portions of the items, thereby pulling these items from the cargo compartment of a transport vehicle onto the platform. Therefore, various other configurations of robotic item handling machines are used for loading / unloading items in material handling environments.

[0045] In some exemplary cases, the operating modes of these robotic goods handlers (e.g., but not limited to, picking up based on gripping or picking up based on sweeping under items) also depend on how the items (e.g., in a goods docking station) are stacked. For example, in some cases, items may be organized / stacked into piles. In other cases, items may be spread out above an area (e.g., on the floor of a goods docking station or a section of a carriage). That is, for cases where items are stacked above each other, the robotic goods handler may be configured to operate in a mode to grip each item using the end effector of the robotic goods handler. Alternatively, for cases where items are spread out, the robotic goods handler may be configured to operate in a mode to pull one or more items by sweeping a portion of the platform of the robotic goods handler under some portions of these items. Thus, the robotic goods handler will be configured to operate in different operating modes to efficiently handle items according to various (predictable or unpredictable) conditions that may arise from the various arrangements in which the items are organized or stacked (pile, wall, spread out, etc.).

[0046] The various exemplary embodiments described herein relate to machine learning-based decision-making for robotic goods handling. According to some exemplary embodiments, machine learning (ML) models can be used to determine appropriate operating modes for a robotic goods handling machine (e.g., a robotic carton unloader). In one example, the robotic goods handling machine may include a first sensor device that can be positioned at a first location and a second sensor device that can be positioned at a second location on the robotic goods handling machine. In this regard, the first sensor device may capture a first three-dimensional (3D) image of a portion of the environment within its field of view (e.g., a first portion). Similarly, the second sensor device may capture a second 3D image of another portion of the environment (e.g., a second portion) based on its field of view. According to the various exemplary embodiments described herein, first point cloud data corresponding to the first 3D image and second point cloud data corresponding to the second 3D image may be obtained and transformed to generate combined point cloud data. In one example, the combined point cloud data may indicate a 3D representation of the entire environment in front of the robotic goods handling machine. For example, the combined point cloud data may represent a 3D view of a goods docking station that may include items (e.g., cartons, packages, etc.) in various orientations and arrangements to be handled by the robotic goods handling machine. According to the exemplary embodiment described, the combined point cloud data can be used as input to a convolutional neural network to construct an ML model. Furthermore, the ML model can output a decision classification that indicates a probability distribution function associated with the selection of an operating mode for using the robotic goods handler. For example, the ML model can output a first probability of using a first operating mode (e.g., picking up an item by gripping it) or a second operating mode (e.g., picking up an item by sweeping it across). In this regard, in one example, an operating mode with a higher probability can be selected, and the robotic goods handler can be operated according to that operating mode.

[0047] Furthermore, in some exemplary implementations, the performance of the ML model's output can be re-evaluated over a period of time. In this regard, the decision to select the operating mode of the robotic goods handling machine can be based on several other factors, including but not limited to predefined heuristics associated with past selections of the operating mode in various situations, rather than using the ML model's output as the sole factor. In other words, according to some examples, the decision to select the operating mode of the robotic goods handling machine can be based on evaluating the performance of the ML model and calculating a weighted average of multiple factors, including the ML model's output. More details on machine learning-based selection of the operating mode of the robotic goods handling machine can be found in [reference needed]. Figures 1 to 11 The description is as follows.

[0048] Figure 1An example of a goods handling system 100 including an exemplary robotic goods handler 101 is shown. In some examples, the robotic goods handler 101 may be a robotic carton unloader, which can be used to load and / or unload one or more cartons in a material handling environment. As shown, the robotic goods handler 101 may be configured to have a plurality of sensor devices 102 to 108. According to some exemplary embodiments, the robotic goods handler 101 may be a mobile vehicle having a platform 118 (or frame) with wheels. Mobile vehicle type robotic goods handlers may be able to move through areas 120 (e.g., carton unloading areas, cargo areas, transport docking areas, etc.). In one example, the robotic goods handler 101 may be designed to be used within the trailer of a cargo truck 121. In some examples, the robotic goods handler 101 may be able to move within a material handling site and further perform loading and / or unloading of goods from the cargo truck 121 by movably positioning a portion of the platform 118 of the robotic goods handler 101 within the cargo truck 121.

[0049] According to some exemplary embodiments described herein, the robotic goods handler 101 may be equipped with various components that can be mounted on or otherwise included within the robotic goods handler 101. For example, as shown, the robotic goods handler 101 may include a robotic arm 115. In some examples, the robotic arm 115 may include an end effector capable of removing one or more items (e.g., boxes) from a wall-like object 125 and placing the removed items 126 onto a conveyor 180. For example, in some cases, the end effector of the robotic arm 115 may include a vacuum suction cup array that can be selectively actuated to grip a portion of an item, thereby removing the item from a wall-like object 125. (Described later in the specification) Figure 5 This example of an end effector is shown. According to some exemplary embodiments, as shown, the platform 118 of a robotic item handling machine can be coupled to a conveyor 180, which can be used to transport removed items 126 away from area 120.

[0050] Examplely, the robotic goods handling machine 101 may include a plurality of sensor devices 102 to 108. In some exemplary embodiments, the plurality of sensor devices 102 to 108 may include, for example, but not limited to, a three-dimensional imaging system (e.g., a depth-sensing camera, a color camera, a pattern projection unit, etc.), which is capable of performing various imaging operations, such as capturing 3D images, performing forward scanning based on the pattern projection area, and recording a video stream of the environment. In this regard, the plurality of sensor devices 102 to 108 may be able to capture one or more 3D images of the same or different portions of the environment that may fall within their respective fields of view.

[0051] According to some exemplary embodiments, the robotic goods handler 101 may utilize one or more distance sensors (not shown) to guide the robotic goods handler 101 into area 120 (e.g., a semi-trailer). For example, in some exemplary embodiments, the robotic goods handler 101 may utilize a "curb feeler" distance sensor that uses contact to measure the distance from the wall of area 120. Alternatively, such a distance sensor may use light, sound, or other methods to sense distance.

[0052] In some exemplary embodiments, the robotic goods handler 101 may also include a computing device (e.g., a personal computer, etc.) for commanding and controlling various operations of the robotic goods handler 101. In some embodiments, the robotic goods handler 101 may include an operator interface (not shown) or a human-machine interface (HMI). In some examples, the HMI may be attached to the conveyor 180 and may also include input / output devices such as joysticks, displays (e.g., monitors), and / or keypads (not shown). The robotic goods handler 101 may also include an electrical box (not shown) that may contain electronic components for the control and vision systems of the robotic goods handler 101. According to some exemplary embodiments, the robotic goods handler 101 may include components such as an internal structural frame, wheel drive motors, a vehicle controller, and an energy source (not shown) such as a battery or liquefied petroleum gas (LPG) for powering the robotic goods handler 101. Further details of such internal components of the robotic goods handler 101 can be found in [reference needed]. Figure 3 Describe it.

[0053] According to some exemplary embodiments, the robotic arm 115 may include an end effector configured to remove items from a wall-like object 125 using a gripping mechanism (e.g., clamps, grippers, etc.), a suction cup mechanism, and / or a lifting mechanism (e.g., ramps, shovels, etc.). For example, as previously described, the robotic arm 115 may include an end effector configured to grip and / or lift boxes from the top of a wall-like object 125 to place them on a conveyor 180. In other embodiments, the robotic item handler 101 may be configured to remove items from area 120 rather than placing items 126 on the conveyor 180. In some embodiments, the robotic arm 115 may utilize a manipulator head that uses vacuum to remove items from a wall-like object 125. Examples of such handling of these items by gripping them with an end effector will be referenced later. Figure 5The description is as follows. As previously mentioned, the articles described herein may correspond to any of the following: cardboard boxes, shipping containers, parcels, etc. In some exemplary embodiments, the articles in a wall-like article 125 may be in the shape of a rectangular prism, with at least one face oriented toward the rear of region 120 (e.g., the rear of a semi-trailer). Furthermore, the surface of the articles in a wall-like article 125 may be textured or otherwise include features that can be sufficiently observed by sensor devices (102 to 108) (e.g., recording images with sufficient detail, etc.), the details of which are described below.

[0054] According to some exemplary embodiments, in order to move one or more items, the platform 118 of the robotic goods handler 101 can be movablely positioned relative to a portion of an area 120 (e.g., a cargo compartment or item docking station) inside a cargo truck 121. In this regard, according to some exemplary embodiments, a portion of the platform 118 can be moved to sweep under a portion of some items, thereby pulling the items onto the platform 118 and further onto a conveyor 180. In other words, the platform 118 of the robotic goods handler 101 can be configured to be movablely positioned such that the front portion of the platform 118 can sweep under a portion of some items, thereby pulling some items upward onto the platform 118 and further onto a conveyor associated with the platform 118. In this regard, the following will describe... Figure 9 and Figure 10 More details are provided regarding the Robotic Item Handling Machine 101.

[0055] According to some exemplary embodiments, computing device 110 may be included in robotic goods handler 101, such as as part of a vision system. Computing device 110 may be any electronic device including one or more processing units (or processors) capable of executing software, receiving input (e.g., signals from sensor devices 102 to 108), and / or outputting data. For example, computing device 110 may be a custom mobile computing device attached to robotic goods handler 101. As a specific example, computing device 110 may be a collection of connected computing devices, including computing devices (such as a PLC) dedicated to controlling robotic goods handler 101 and computing devices (such as a personal computer) dedicated to processing image data from each sensor device (e.g., three computing devices for sensor devices 102 to 108). In various exemplary embodiments, computing device 110 may be configured to control the operation of robotic goods handler 101, such as by controlling the operation of robotic arm 115 and / or receiving input from a controller (e.g., a joystick, a remote system, etc.) to move robotic goods handler 101 within area 120.

[0056] According to some exemplary embodiments, multiple sensor devices 102 to 108 may be coupled to computing device 110 via wired or wireless connections 103, 105, 107, etc. For example, each of sensor devices 102 to 108 may be directly plugged into computing device 110 via a physical wire (e.g., Universal Serial Bus (USB) connection, FireWire connection, etc.) or wirelessly connected to the computing device via short-range wireless signals (e.g., Bluetooth, Zigbee, Wi-Fi, Wi-Fi Direct, RF, etc.). In some exemplary embodiments, computing device 110 may include a user interface device (not shown) capable of displaying data such as readouts, application interfaces, and captured sensor data, such as a display, user terminal, etc.

[0057] As previously described, sensor devices 102 to 108 may be configured to capture various representations of region 120 according to various exemplary embodiments. In some exemplary embodiments, sensor devices 102 to 108 may be able to generate ranging information based on acquired visual data (e.g., light) and / or audible data, as well as thermal data or other forms of energy (e.g., infrared) data. In some exemplary embodiments, sensor devices 102 to 108 may be able to track real-time movement and three-dimensional scanning (i.e., 3D scanning) objects. According to some exemplary embodiments, sensor devices 102 to 108 may include one or more camera units and / or microphones. In some embodiments, sensor devices 102 to 108 may be stereo cameras, which include more than one lens and image sensor, and may also include different cameras for capturing different types of data, such as infrared sensors and color cameras. The camera units within sensor devices 102 to 108 may include specialized cameras, lenses, controllers, and / or software that enable these cameras to acquire various types of images. In some exemplary embodiments, sensor devices 102 to 108 may include a camera for capturing color images (or "red-green-blue" (RGB)). Sensor devices 102 to 108 may be able to sense (or calculate) depth using the captured and processed visual portion from the camera and / or audio from a microphone. In some embodiments, sensor devices 102 to 108 may include an infrared laser projector and a monochromatic complementary metal-oxide-semiconductor (CMOS) sensor capable of recording 3D data.

[0058] In some embodiments, sensor devices 102 to 108 may include a battery and / or be connectable to a power source, such as by coupling to a battery cell included within the robotic goods handling unit 101. Additionally, sensor devices 102 to 108 may include various input and output ports or interfaces, such as USB ports, and other elements configured to enable data to be received and / or stored within sensor devices 102 to 108. In some exemplary embodiments, sensor devices 102 to 108 may be capable of recording LiDAR data. In some exemplary embodiments, sensor devices 102 to 108 may be commercially available sensor devices, such as Microsoft Kinect. TM Sensors (e.g., Kinect Version 1, Kinect Version 2, etc.). In some embodiments, the robotic goods handler 101 may include a stereo camera, a monocular camera, and / or a LiDAR sensor unit (e.g., a two-dimensional (2D) LiDAR sensor unit). In some embodiments, sensor devices 102 to 108 may be stand-alone monocular cameras mounted as stereo-to-stereo baselines. Furthermore, in some exemplary embodiments, the movement speed of the robotic goods handler 101 may be controlled or limited to improve the synchronized operation of sensor devices 102 to 108 with computing device 110.

[0059] In some embodiments, computing device 110 may include a transceiver capable of exchanging signals with router device 132 (e.g., a Wi-Fi router) via wireless connection 112. Router device 132 may be associated with a local area network (LAN) 130, which may be connected to a wide area network (WAN), such as the Internet 150, via connection 131. In some embodiments, computing device 110 may be connected to router device 132 via wired connection 113. In some embodiments, computing device 110 may include a transceiver capable of exchanging signals with base station 142 via wireless connection 109. Base station 142 may be associated with a cellular network 140 connected to the Internet 150 via connection 141.

[0060] In various implementations, computing device 110 may be configured to communicate with remote device 160 via various networks 130, 140. Remote device 160 may be connected to local area network 130 via wired or wireless connection 162 and / or may otherwise be connected to the Internet 150 via wired or wireless connection 161. Through these communication links, computing device 110 and remote device 160 may be able to exchange various data, such as cargo reports based on unloading operations performed in area 120. For example, remote device 160 and computing device 110 may be configured to communicate via a public local area network (LAN), such as by exchanging peer-to-peer communication within a distribution center via LAN 130. In some implementations, computing device 110 may be directly connected to remote device 160 via wired or wireless connection 114.

[0061] In various implementations, remote device 160 may be one or more computing devices (e.g., servers, desktop computers, laptop computers, etc.) configured to store, process, retrieve, and otherwise handle data related to unloading goods. For example, remote device 160 may be a warehouse management server connected to a database and configured to execute software for organizing the delivery of goods from area 120 to various departments or zones within the distribution warehouse. In some implementations, computing device 110 and remote device 160 may exchange data between them and with other devices (e.g., scanners, remote sensors, actuators, diagnostic systems, material handling equipment controls (such as conveyor controls), data storage devices, gauges, printers, etc.) via various network connections (including TCP / IP connections, Ethernet connections, EtherCAT connections, PROFIBUS connections, RS232 connections, USB connections, Wi-Fi connections, cellular connections, etc.).

[0062] According to some exemplary embodiments, the robotic goods handling machine 101 may include various mounting options for mounting various sensor devices 102 to 108. In exemplary embodiments, the sensor devices 102 to 108 may be positioned on one or more of the following: the frame of the robotic goods handling machine 101, the robotic arm 115, the platform 118, etc. According to some exemplary embodiments, the sensor devices 102 to 108 may be positioned such that this placement facilitates simpler mutual calibration between the sensor devices 102 to 108. In this respect, the location of the sensor devices 102 to 108 may be a location for capturing the most complete and / or most relevant view of an area 120 including a wall-like object 125.

[0063] According to some exemplary embodiments described herein, the robotic goods handler 101 may be able to operate according to one or more operating modes configured for it. In an exemplary embodiment, the robotic goods handler 101 may be configured to operate in two operating modes (i.e., a first operating mode and a second operating mode). In the first operating mode, the end effector of the robotic arm 115 of the robotic goods handler 101 may be actuated to pick up an item by gripping a portion of the item. Furthermore, in the second operating mode, a portion of the platform 118 of the robotic goods handler 101 (e.g., the front end / lead edge end) may be actuated and moved to sweep across under a portion of a pile of items (e.g., a wall-like pile of items 125), thereby pulling one or more items from the pile onto the platform 118 and further guiding the one or more items onto the conveyor 180. In other words, a first operating mode of the robotic goods handler 101 can be associated with picking up an item by gripping it with the robotic arm 115, and a second operating mode of the robotic goods handler 101 can be associated with sweeping a pile of items from a docking station (e.g., area 120) using the platform 118 of the robotic goods handler 101. According to the various exemplary embodiments described herein, the selection of the operating mode of the robotic goods handler 101 can be based on utilizing a machine learning model, details of which will be referred to later. Figures 2 to 11 Describe it.

[0064] Figure 2 An exemplary scenario 200 is illustrated, illustrating a decision-making process based on a machine learning model for selecting an operating mode for a robotic goods handler, according to an exemplary embodiment. According to some exemplary embodiments described herein, a computing device (e.g., computing device 110) communicatively coupled to the robotic goods handler 101 may include a processing unit 202 capable of performing various operations related to the decision-making process regarding the selection of an operating mode for the robotic goods handler 101. (See previous references...) Figure 1 In some examples, the operating modes associated with the robotic item handling machine 101 may correspond to using the robotic arm 115 to pick up one or more items or using the platform 118 to sweep across the one or more items.

[0065] According to an exemplary embodiment, a system capable of performing such machine learning model-based decision-making may include several hardware and / or software / firmware components, including but not limited to a processing unit 202, a normalization unit 204, a decision evaluation unit 206, and a control unit 208. As shown, the processing unit 202 may be communicatively coupled to the normalization unit 204, the decision evaluation unit 206, and the control unit 208 via a communication network.

[0066] According to an exemplary embodiment, normalization unit 204 may receive a probability distribution function that can be associated with one or more operating modes of the robotic goods handler 101. This probability distribution function may be the output of a machine learning model. In this regard, the machine learning model may be constructed from a machine learning engine that can be coupled to processing unit 202. Examples of machine learning engines and machine learning models will be provided later. Figure 4 The following description is provided. Furthermore, the normalization unit 204 can be configured to convert this information (i.e., information in the form of a probability distribution function) into a decision. This decision can indicate the selection of an operating mode for the robotic goods handler 101. For example, the decision can indicate the selection to actuate the robotic arm 115 of the robotic goods handler 101 to perform the handling (e.g., picking up) of items. In another example, the decision can indicate the selection to actuate the platform 118 of the robotic goods handler 101 to sweep items onto a portion of the robotic goods handler 101.

[0067] According to the exemplary embodiment, the decision evaluation unit 206 can perform an evaluation of the decision (i.e., the output of the normalization unit 204). In this regard, according to some exemplary embodiments, the decision to select the operating mode of the robotic goods handler 101 can be evaluated based on several factors. In other words, the decision can be evaluated based on the output of a machine learning engine and / or several other factors. As shown, in addition to the output of the machine learning, the decision evaluation unit 206 can also evaluate the decision to select the operating mode of the robotic goods handler 101 based on some predefined heuristics. According to some examples, the predefined heuristics can be associated with past operations of the robotic goods handler 101. In other words, the predefined heuristics can correspond to a set of rules or methods followed to select the operating mode of the robotic goods handler 101 over a period of time based on different external conditions (e.g., related to the stacking or placement of items). The following paragraphs describe some examples of predefined heuristics that can be used to evaluate the decision to select the operating mode of the robotic goods handler 101.

[0068] In some exemplary embodiments, a predefined heuristic may be associated with the coordinates (e.g., coordinates corresponding to boxes or cartons) of one or more items identified in three-dimensional space for pickup by the robotic goods handler 101. Furthermore, in some exemplary embodiments, decisions regarding operating modes based on the predefined heuristic may be decisions to ensure that the maximum number of boxes can be picked up in a single action (i.e., robotic operation). In this regard, in some examples, items perceived at the maximum distance from the sensor device (i.e., items associated with the maximum z-coordinate distance) may be selected for pickup, such that a region of interest created around the item results in the pickup of the maximum number of items. Furthermore, in some exemplary embodiments, the predefined heuristic may involve forward scanning or picking up items layer by layer in a pile or wall-like stack. Additionally, in some exemplary embodiments, the predefined heuristic may also be based on several factors that may affect the safety of the robotic goods handler 101 when one or more mechanical components of the robotic goods handler 101 (e.g., robotic arm 115 and / or platform 118) move or operate. For example, rules based on a predefined heuristic may include determining the operating mode of the robotic goods handler 101 such that any mechanical component does not collide with sidewalls or create obstacles during the movement of the robotic goods handler 101. Furthermore, in some exemplary embodiments, the predefined heuristic may also include rules that ensure the robotic arm 115 is not used to pick up objects at lower heights (i.e., avoiding low-pickup). In this regard, the predefined heuristic may include rules indicating that items identified at a docking station at a height below a predefined height can be handled by the platform 118 by performing a sweep-to-handle operation. Further details related to the evaluation of decisions based on machine learning models will be discussed later. Figure 8 Describe it.

[0069] Therefore, the decision evaluation unit 206 can perform an evaluation to provide the control unit 208 with a final decision on the selection of the operating mode of the robotic goods handler 101. Furthermore, based on the final decision, the control unit 208 can generate one or more commands to actuate one or more mechanical components of the robotic goods handler 101 (e.g., the end effector of the robotic arm and / or the platform 118) to perform one or more desired operations.

[0070] Figure 3 A component block diagram of the elements of a robotic item unloader 301 according to various exemplary embodiments described herein is shown. In exemplary embodiments, the robotic item unloader 301 may correspond to, for example... Figure 1The robotic goods handling machine 101 is described herein. In illustrative terms, the robotic goods unloading machine 301 may include various components such as an external monitor 302, a network interface module 304, a human-machine interface (HMI) module 306, an input / output module (I / O module 308), an actuator / sensor module 310, and a conveyor system 315, which may include a robotic arm 311, a drive / safety module 312, and a motion module 314. Furthermore, the robotic goods unloading machine 301 may include processing equipment such as a programmable logic controller (PLC) 318. Additionally, the robotic goods unloading machine 301 may include a basic motion module 320, which may include a vehicle controller module 322 and a manual control module 324. Furthermore, the robotic goods unloading machine 301 may include a vision system 326 (or visualization system), which may include one or more computing devices 328 (or “PC”) and sensor devices 330. According to some exemplary embodiments, the sensor device 330 mentioned herein may correspond to, as previously described... Figure 1 The one or more sensor devices (102 to 108) described herein. In some exemplary embodiments, the vision system 326 of the robotic item unloader 301 may include a PC 328 connected to each sensor device 330. The sensor device 330 may include one or more depth cameras, color cameras, etc., capable of capturing one or more 3D images. For example, the sensor device 330 may be configured to capture 3D images of the environment in which the robotic item unloader 301 will be used. Additionally, the vision system 326 may construct point cloud data from the images captured by the sensor devices 330.

[0071] In some exemplary embodiments, more than one sensor device 330 may be present on the robotic item unloader 301. In this regard, PCs 328 for each sensor device 330 may be networked together, and one of the PCs 328 may operate as a master PC 328. The master PC 328 may receive data (e.g., point cloud data) from another connected PC 328 and may also perform data processing on the received data to generate combined point cloud data. Alternatively, in some exemplary embodiments, the sensor devices 330 may be communicatively coupled to an external processing device that can access image data from the various sensor devices 330 and generate combined point cloud data based on that image data.

[0072] According to some exemplary embodiments, the robotic goods unloader 301 may be able to connect to a remote location or system via a network 303, such as a local Wi-Fi network, using a network interface module 304 (e.g., a Wi-Fi radio component). In some examples, the network interface module 304 may enable the robotic goods unloader 301 to connect to an external monitor 302. The external monitor 302 may be any of a remote warehouse or distribution center control room, a handheld controller, or a computer, and may provide passive remote observation via the vision system 326 of the robotic goods unloader 301. Alternatively, in some exemplary embodiments, the external monitor 302 may override the programming inherent in the vision system 326 and assume active command and control of the robotic goods unloader 301.

[0073] According to some exemplary embodiments described herein, the programming of the robotic unloader 301 can also be communicated, operated, and debugged via an external system such as an external monitor 302. Some examples of the external monitor 302 that performs command and control may include a remotely located human operator or a remote system, such as a warehouse or distribution server system (i.e., remote device 160 as described above). According to some examples, the external monitor 302 may include any of the following: a visual monitor, a keyboard, a joystick, an I / O port, a CD reader, a computer, a server, a handheld programming device, or any other device that can be used to perform various operations.

[0074] According to some exemplary embodiments, the robotic unloader 301 may include an HMI module 306, which can be used to control the robotic arm 311 and the conveyor system 315 and / or the basic motion module 320 and / or receive their output information. In some examples, the HMI module 306 can be used to control (or may include itself) joysticks, displays, and keypads, which can be used for reprogramming, overriding the automatic control of the machine, and driving the robotic unloader 301 from point to point. According to some exemplary embodiments, actuators 310 and distance sensors, which can be actuated individually or in any combination by the vision system 326, can be used to help guide one or more components of the robotic unloader 301 into the unloading area (e.g., the cargo compartment of a trailer).

[0075] According to some exemplary embodiments, I / O module 308 can connect actuators and distance sensors to PLC 318. Robotic arm 115 and conveyor system 315 may include all components required for the mobile arm and / or conveyor, such as drives / engines and motion protocols or controls. Basic motion module 320 may be a component for moving the entire robotic unloader 301. In other words, basic motion module 320 may be a component required to guide the vehicle (i.e., robotic unloader 301) in and out of the unloading area.

[0076] According to some exemplary embodiments, PLC 318 can control the overall electromechanical movement of the robotic unloader 301 or control exemplary functions, such as controlling the robotic arm 311 or the conveyor system 315. For example, PLC 318 can move the manipulator head (i.e., end effector) of the robotic arm to a position for retrieving an item (e.g., a box, carton, etc.) from a wall-like object. An example reference is provided for the end effector of the robotic arm 311. Figure 5 Further description is provided. In other exemplary embodiments, instead of PLC 318, external processing devices such as computing devices 110 may perform similar operations or exemplary functions to control the overall electromechanical movement of the robotic unloader 301, such as controlling the robotic arm 311 and / or the conveyor system 315.

[0077] According to some exemplary embodiments, other electronic components of the PLC 318 and vision system 326 may be housed in an electrical box (not shown) located below, adjacent to, or elsewhere on the robotic unloader 301. The PLC 318 may automatically operate all or part of the robotic unloader 301 and may receive position information from distance sensors. I / O module 308 may connect actuators and distance sensors to the PLC 318.

[0078] According to some exemplary embodiments, the robotic goods unloader 301 may include a vision system 326, which includes sensor devices 330 (e.g., cameras, microphones, 3D sensors, etc.) and one or more computing devices 328 (referred to as personal computers or "PCs") 328. The robotic goods unloader 301 may use the sensor devices 330 and the vision system 326, or the one or more PCs 328, to scan in front of the robotic goods unloader 301 in real-time or near real-time. According to some exemplary embodiments, a forward scan of the sensor devices 330 may be triggered in response to determining an event associated with the robotic goods unloader 301. For example, a forward scan may be initiated by a trigger sent in response to the robotic goods unloader 301 being in a position for initiating detection of cartons in the unloading area. According to some exemplary embodiments, the sensor devices 330 may be configured to perform a forward scan that can be used for various purposes, such as, but not limited to, collision avoidance, human shape recognition (safety), setting the dimensions of the unloading area (e.g., a truck or trailer), and scanning the floor of the unloading area to locate bulk items (e.g., cartons, boxes, etc.).

[0079] According to some exemplary embodiments, the 3D sensing capability of vision system 326 can also provide depth perception, edge recognition, and the ability to create 3D images of a wall-like object (or a stack of cartons). According to some exemplary embodiments, vision system 326 can operate independently or in conjunction with PLC 318 and / or computing device 110 to identify the edges, shapes, and near / far distances of objects in front of robotic unloader 301. For example, the edges and distances of each individual carton in a wall-like object can be measured and calculated relative to robotic unloader 301. In some examples, based on such measurements, vision system 326 can be operated to select specific items or all items from a stack for removal.

[0080] According to some exemplary embodiments, vision system 326 may be configured to provide information such as the specific XYZ coordinate positions of one or more items (e.g., cartons) that are targets for removal from the unloading area. In some exemplary embodiments, these coordinate positions may also correspond to one or more movement paths traveled by the moving body of robotic arm 115 or robotic item unloader 301.

[0081] Although Figure 3 The various computing devices and / or processors, such as PLC 318, vehicle controller 322, and PC 328, have been described separately, but regarding Figure 3 In the various embodiments discussed and all other embodiments described herein, the described computing devices and / or processors may be combined, and the operations described herein that are performed by individual computing devices and / or processors may be performed by fewer computing devices and / or processors, such as a single computing device or processor having different modules for performing the operations described herein. For example, different processors combined on a single circuit board may perform the operations described herein that are attributed to different computing devices and / or processors, and a single processor running multiple threads / modules may perform the operations described herein that are attributed to different computing devices and / or processors, etc.

[0082] Other embodiments of the robotic carton unloader and descriptions of components suitable for use according to various embodiments are described in the following applications: U.S. Provisional Patent Application No. 61 / 894,889, filed October 23, 2013, entitled “Robotic Carton Unloader with Visualization System”; U.S. Provisional Patent Application No. 61 / 973,188, filed March 31, 2014, entitled “Robotic Truck Loader with Alternate Vacuum Head”; U.S. Non-Provisional Patent Application Serial No. 14 / 279,694, filed Cancel, 2014, entitled “Robotic Carton Unloader”; and U.S. Non-Provisional Patent Application Serial No. 14 / 445,929, filed July 29, 2014, entitled “Robotic Carton Unloader”. The entire contents of all four applications are incorporated herein by reference.

[0083] Figure 4 An exemplary architecture 400 of a machine learning model is shown for making decisions on selecting an operating mode of a controlled robotic goods unloader (e.g., robotic goods handler 101). According to an exemplary embodiment, the machine learning model may output a decision classification. This decision classification may include a probability distribution function that indicates and is associated with one or more probabilities of a decision to select among various operating modes associated with the robotic goods handler 101. For example, the decision classification may be a classification of a first probability of the robotic goods handler 101 selecting a first operating mode (e.g., pickup by a robotic arm) or a second probability of selecting a second operating mode (e.g., sweeping by a platform). According to an exemplary embodiment, the machine learning model may be generated by a machine learning engine 402. In an exemplary embodiment, the machine learning engine 402 may be deployed by a computing device 110, as referenced... Figure 1 The machine learning engine 402 may include its own processing unit 404. In some exemplary embodiments, the machine learning engine 402 may be communicatively coupled to an external processing unit, such as the processing unit 202 of computing device 110 or PLC 318.

[0084] According to the exemplary embodiments described, the machine learning engine 402 may employ one or more machine learning processes and / or one or more artificial intelligence techniques to generate a machine learning model. The machine learning model can help select an operating mode from several operating modes to operate the robotic goods handling machine 101. According to some exemplary embodiments, the machine learning engine 402 is based on input data that can be processed by one or more layers of a neural network architecture to output a decision classification, which helps the processing device select an operating mode and command operations according to the selected operating mode, thereby operating the robotic goods handling machine 101.

[0085] According to an exemplary implementation, machine learning engine 402 can build machine learning models that are based on convolutional neural networks (CNNs).

[0086] In this regard, the machine learning engine 402 can use combined point cloud data as input to the CNN. As previously mentioned, the combined point cloud data may correspond to a point cloud obtained by merging multiple point cloud data corresponding to several 3D images, which may be captured by different sensor devices associated with the robotic goods handling machine 101. For example, in an exemplary embodiment, the combined point cloud may be obtained by transforming first point cloud data associated with a first 3D image captured by a first sensor device (e.g., sensor device 106 associated with the robotic arm 115) and second point cloud data associated with a second 3D image captured by a second sensor device (e.g., sensor devices 104 and / or 106 associated with the platform of the robotic goods handling machine 101). Further details on generating the combined point cloud data can be found in [reference needed]. Figure 8 Describe it.

[0087] As previously described, the machine learning model generated by machine learning engine 402 can output a decision classification that indicates the probability associated with an operating mode for operating the robotic goods handler. In one example, machine learning engine 402 can output a decision classification that indicates a first probability associated with a first operating mode of the robotic goods handler (e.g., operating the robotic arm to pick up items) and a second probability associated with a second operating mode (e.g., operating the platform of the robotic goods handler to sweep items). Therefore, the output of the machine learning model can be used to select an operating mode for operating the robotic goods handler 101. For example, in one example, if the first probability output by the machine learning model is higher than the second probability, the robotic goods handler 101 may operate according to the first operating mode. Alternatively, if the second probability output by the machine learning model is higher than the first probability, the robotic goods handler 101 may operate according to the second operating mode.

[0088] According to various exemplary embodiments, the machine learning engine 402 may employ various techniques to generate and train machine learning models. For example, in some examples, the machine learning engine 402 may use a support vector machine (SVM) classifier to determine one or more classifications, one or more correlations, one or more expressions, one or more inferences, one or more patterns, one or more features, and / or other learning information based on input data (i.e., combined point cloud). In another exemplary embodiment, the machine learning engine 402 may employ one or more machine learning classification techniques, such as, but not limited to, techniques associated with Bayesian machine learning networks, binary classification models, multi-class classification models, linear classifier models, quadratic classifier models, neural network models, probabilistic classification models, decision trees, and / or one or more other classification models. According to some exemplary embodiments, the machine learning model (e.g., classification model, machine learning classifier, etc.) employed by the machine learning engine 402 may be explicitly trained (e.g., via training data) and / or implicitly trained (e.g., via extrinsic data received by the machine learning model). For example, the machine learning model (e.g., classification model, machine learning classifier, etc.) employed by the machine learning engine 402 may be trained using training data that may include combined point cloud data. In some examples, the training data may also include one or more samples of predefined heuristics associated with past operating patterns of the robotic goods handler 101. In this regard, the predefined heuristics may include prior selections of operating patterns of the robotic goods handler 101 in various past scenarios (e.g., to handle items stacked in various formations such as piles, walls, spread out, etc.).

[0089] In an exemplary embodiment, the machine learning engine 402 may implement a convolutional neural network (CNN), which may include multiple partially connected or fully connected neural network layers. The network layers of the CNN may be based on one or more convolutional filters and / or pooling functions. Figure 4An exemplary illustration of such a neural network is depicted, which can be implemented by a machine learning engine 402 to build and train machine learning models. According to an exemplary implementation, as shown, the input to the CNN can be merged point cloud data 406 (e.g., combined point cloud data as previously described). In this respect, merged point cloud data 406 can correspond to the combined point cloud as previously described (i.e., a point cloud derived after combining point cloud data from various sensor devices (102 to 108) associated with the robotic goods handling machine 101). In some exemplary implementations, as shown, a first portion from the merged point cloud data 406 may include RGB information from three channels (referred to herein as RGB image data 408), and a second portion from the merged point cloud data 406 may include depth information from additional channels (referred herein as depth image data 410). As shown, RGB image data 408 can be processed by a first set of layers of the CNN, and depth image data 410 can be processed by a second set of network layers of the CNN. In other words, information about the RGB and depth channels from the merged point cloud data can be processed by the machine learning engine 402 through separate layers of the neural network.

[0090] According to some exemplary embodiments described herein, the input data to the CNN (i.e., the merged point cloud data 406) can be initially normalized for training the machine learning model. Furthermore, the input data can be passed through a set of convolutional layers, max-pooling layers, and finally through a fully connected layer combined with a SoftMax layer to obtain a class probability distribution as the output of the machine learning model.

[0091] By way of example, according to an exemplary embodiment, the RGB image data 408 of the merged point cloud data 406 can be processed through a first set of convolutional 7*7 64 filters 412, further through a second set of convolutional 5*5 64 filters 414, further through a max pooling layer 416, further through a first set of convolutional 3*3 64 filters 418 and a second set of 3*3 64 filters 420, and finally through a first fully connected 1024 output layer 422 and a second fully connected 1024 output layer 424. In the exemplary embodiment, the rectified linear unit (ReLU) function can be used as an activation function throughout the network layers of the CNN. Alternatively, other known activation functions can also be used. According to an exemplary embodiment, a second part of the merged point cloud data 406 (i.e., depth image data 410) can be processed through a separate channel from the layer of the CNN used to process the first part. As shown, the depth image data 410 can be an input to a third fully connected 1024 output layer 426 and also to a second fully connected 1024 output layer 424. Furthermore, the output of the second fully connected 1024-output layer 424 can be provided as input to the SoftMax fully connected 2-output network layer 428. Therefore, during inference based on the input data processed by the CNN (i.e., the merged point cloud data 406), the machine learning engine 402 can output a decision classification. As previously described, the decision classification indicates a first probability associated with a first operating mode of the robotic goods handler 101 and a second probability associated with a second operating mode of the robotic goods handler 101.

[0092] According to an exemplary embodiment, a CNN can process input data, identify a scene from that input data, and provide probabilistic classification outputs as pile 430 and wall 432. The classification output by the CNN can be represented by a probability distribution and indicates the type of item stacks identified from the image data, which can be derived from each cumulative iteration of the image. For example, the CNN output pile 430 can indicate a pile-type item stack, and the CNN output wall 432 can indicate a wall-type item stack. Therefore, the output of the CNN can be a decision classification indicating the probability distributions of two or more categories (e.g., pile 430 and wall 432). According to the various exemplary embodiments described herein, the probability distribution output (i.e., the decision classification) from a CNN-based machine learning model can be (e.g., by computing device 110) converted into a next action that can be performed by the robotic item handling machine 101. For example, computing device 110 can use the output of the CNN to operate robotic goods handler 101 in either a first operating mode (i.e., performing a pick-up operation using robotic arm 115) or a second operating mode (i.e., performing a sweep operation using the platform of robotic goods handler 101). For example, if a decision classification indicator shows a first probability associated with output pile 430 that is higher than a second probability associated with output wall 432, then robotic goods handler 101 can operate based on the first operating mode. Alternatively, if a decision classification indicator shows a first probability associated with output pile 430 that is lower than a second probability associated with output wall 432, then robotic goods handler 101 can operate based on the second operating mode.

[0093] According to some exemplary embodiments, various other factors and the output of the machine learning model can be evaluated to select the operating mode of the robotic goods handling machine 101. In other words, the output of the machine learning model and several other factors can be evaluated to determine the operating mode of the robotic goods handling machine 101, details of which can be found in [reference needed]. Figure 2 and Figure 8 It is described in the instruction manual.

[0094] Figure 5 A view of an exemplary end effector 500 of the robotic arm 115 of a robotic goods handler 101 according to an exemplary embodiment described herein is shown. As previously described, the robotic goods handler 101 can operate in a first operating mode in which the robotic arm 115 of the robotic goods handler 101 can be actuated to pick up one or more items. In this respect, in the first operating mode, the end effector 500 of the robotic arm 115 can be actuated to pick up one or more items based on gripping the items. Figure 5A first operating mode of the robotic goods handling machine 101 is shown, in which the end effector 500 approaches a pile or wall of items 502. As shown, multiple items (504-1, 504-2…504-n) such as, but not limited to, cartons, parcels, containers, etc., can be stacked on top of each other or in any arrangement at the goods docking station 503 (e.g., the cargo compartment of a truck).

[0095] As previously described, the end effector 500 can pick up one or more items by grasping a portion of an item. Illustratively, the end effector 500 may include a plurality of vacuum suction cups 506-1…506-n that can be configured to grasp a portion of an item by vacuum suction. In this respect, each of the plurality of vacuum suction cups 506-1…506-n may be mounted at the end of a corresponding guide rod 508. Furthermore, each guide rod 508 may be slidably mounted in a guide frame 510. Additionally, a plurality of springs 512 may be connected between each guide rod 508 and the guide frame 510 to bias the guide rod 508 forward (e.g., toward a stack of items 502).

[0096] According to an exemplary embodiment, an end effector 500 coupled to the end of the robotic arm 115 can move toward a stack of items 502, such that one or more vacuum suction cups among a plurality of vacuum suction cups 506-1…506-n can contact one or more items among items 504-1…504-n. Thus, during such contact, the vacuum suction cups 506-1…506-n can grasp the items based on vacuum suction. Furthermore, a guide frame 510 can be raised to lift one or more items (e.g., items 504-1 and 504-2) from the stack of items 502 and position them on a portion of the conveyor 514 of the robotic item handling machine, thereby performing item loading and / or unloading operations. Figure 5 The end effector 500 shown is one such example of a goods handling tool that can be used with a robotic arm in a first operating mode of a robotic goods handler. Alternatively, in other exemplary embodiments, end effectors of types other than end effector 500 may also be used.

[0097] Figures 6 to 7An exemplary flowchart illustrating operations performed by a device (such as a robotic goods handler 101 or a robotic goods unloader 301) according to an exemplary embodiment of the present invention is shown. It should be understood that each block in the flowchart and combinations of blocks in the flowchart can be implemented by various devices (such as hardware, firmware, one or more processors, circuitry, and / or other devices associated with the execution of software including one or more computer program instructions). For example, one or more processes described above can be embodied by computer program instructions. In this regard, the computer program instructions embodying the processes described above can be stored in the memory of a device employing an embodiment of the present invention and executed by a processor in that device. As will be understood, any such computer program instructions can be loaded onto a computer or other programmable device (e.g., hardware) to produce a machine such that the resulting computer or other programmable device provides an implementation of the functions specified in one or more flowchart blocks. These computer program instructions can also be stored in a non-transitory computer-readable storage memory that can instruct a computer or other programmable device to operate in a particular manner, such that the instructions stored in the computer-readable storage memory produce an article of writing whose execution can implement the functions specified in one or more flowchart blocks. Computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operations to be performed on the computer or other programmable device, thereby producing a computer-implemented process, such that the instructions, which execute on the computer or other programmable device, provide operations for implementing the functions specified in one or more flowchart boxes. Therefore, Figures 6 to 7 When executed, this operation transforms the computer or processing circuitry into a specific machine configured to perform exemplary embodiments of the present invention. Therefore, Figures 6 to 7 The operation is limited to configuring a computer or processor to execute the algorithm of the exemplary implementation. In some cases, an instance of a processor may be provided for a general-purpose computer that executes... Figures 6 to 7 The algorithm transforms a general-purpose computer into a specific machine configured to execute an exemplary implementation.

[0098] Therefore, the boxes in a flowchart support combinations of devices for performing a specified function and combinations of operations for performing the specified function. It will also be understood that one or more boxes in a flowchart, as well as combinations of boxes in a flowchart, can be implemented by a hardware-based dedicated computer system or a combination of dedicated hardware and computer instructions to perform the specified function.

[0099] Figure 6A flowchart illustrating a method 600 for controlling a robotic goods handler 101 according to an exemplary embodiment is shown. Method 600 begins at step 602. At step 604, the robotic goods handler 101 may include a processing unit 202, such as a computing device 110, to obtain first point cloud data. The first point cloud data may be correlated with a first 3D image that may be captured by a first sensor device of the robotic goods handler (e.g., sensor device 108). In one example, the first sensor device may be located on the robotic arm 115 of the robotic goods handler 101. At step 606, the processing unit 202 may obtain second point cloud data correlated with a second 3D image. The second 3D image may be captured by a second sensor device (e.g., sensor device 106). The second sensor device may be located on a platform 118 of the robotic goods handler 101.

[0100] At step 608, processing unit 202 may transform the first point cloud data and the second point cloud data to generate combined point cloud data (e.g., as shown in the image). Figure 4 The merged point cloud data 406. In one example, the merged point cloud data may represent a complete or overall view of a docking station in which the robotic goods handler 101 will perform goods handling. For example, the merged point cloud data may correspond to a 3D view of the entire cargo compartment of a truck from which the robotic goods handler 101 may pick up one or more items for unloading. Details of the transformation from point cloud data to merged point cloud data can be found in [reference needed]. Figure 8 Describe it.

[0101] At step 610, processing unit 202 may construct a machine learning model based on the combined point cloud data. In this regard, the combined point cloud data may be provided as input to a convolutional neural network. Details regarding constructing a machine learning model based on the combined point cloud data can be found in previous references. Figure 4 It has been described.

[0102] Moving to step 612, processing unit 202 may output a decision classification via a machine learning model. This decision classification may indicate the probability associated with an operating mode of the robotic goods handler 101. For example, in one example, the decision classification may indicate a first probability associated with a first operating mode of the robotic goods handler 101 and a second probability associated with a second operating mode of the robotic goods handler 101. Moving to step 614, processing unit 202 may generate a command for operating the robotic goods handler 101. In this regard, the command for operating the robotic goods handler 101 may include one or more configuration settings associated with an operating mode upon which the robotic goods handler 101 operates. According to an exemplary embodiment, a command for operating the robotic goods handler 101 according to a first operating mode may be generated in response to a first probability being higher than a second probability. Alternatively, a command for operating the robotic goods handler 101 according to a second operating mode may be generated in response to a second probability being higher than a first probability. Method 600 stops at step 616.

[0103] Therefore, as a specific implementation of the exemplary embodiment, the robotic goods handling machine 101 can operate according to an operating mode selectable based on the output of a machine learning model. In some exemplary embodiments, the selection of the operating mode may be based on an evaluation of the output of the machine learning model and some other additional factors, details of which can be found in [reference needed]. Figure 2 and 7 Describe it.

[0104] Figure 7 A flowchart illustrating a method 700 for controlling a robotic goods handler 101 according to an exemplary embodiment is shown. The method begins at step 702. In some exemplary embodiments, method 700 may begin in response to a decision classification output by a machine learning model, such as... Figure 6 As described in step 612. Alternatively, method 700 may begin in any other circumstances.

[0105] At step 704, the robotic goods handler 101 may include a processing unit 202, such as a computing device 110, to evaluate the selection of an operating mode for the robotic goods handler 101. In this regard, the processing unit 202 may evaluate operating modes that can be selected for operating the robotic goods handler 101. For example, the processing unit 202 may select whether the robotic goods handler 101 can be operated based on a first operating mode or a second operating mode. In this regard, as previously described, the robotic goods handler 101 may be operated based on a first operating mode, which may involve moving items using a robotic arm 115 (i.e., by gripping a portion of an item). Alternatively, the robotic goods handler 101 may be operated based on a second operating mode, which may involve moving items using a platform 117 (i.e., by sweeping under a portion of an item). According to some exemplary embodiments, the selection of the operating mode may be based on an evaluation that may be performed, for example, by the processing unit of the computing device 110. Some details related to the evaluation described herein are previously referenced. Figure 2 The description was provided.

[0106] According to an exemplary implementation, the evaluation of the selection of the operating mode can be based on several factors. One such factor could be a decision classification output by a machine learning model, as previously... Figures 1 to 6 In other words, according to some exemplary embodiments, decision classification may not be the only factor that can be used to select the operating mode of the robotic goods handling machine 101.

[0107] According to one exemplary embodiment, processing unit 202 may evaluate the selection of an operating mode for the robotic goods handler 101 based on: a decision classification output by a machine learning model, a predefined heuristic associated with past operations of the robotic goods handler 101, and several other factors (e.g., but not limited to, the layout of the material handling environment, safety measures associated with the operation of the robotic goods handler 101, and structural characteristics associated with items such as fragile items, consumable items, etc.). According to some exemplary embodiments, the predefined heuristics mentioned herein may correspond to one or more rules or methods that one or more operators follow during a period of time working at a material handling site to operate the robotic goods handler 101 under various conditions (e.g., related to the stacking or placement of items). For example, the predefined heuristics may include one or more rules defined by an engineer to operate the robotic goods handler 101 according to engineering standards or guidelines. Some examples of predefined heuristics used to evaluate the decision to select an operating mode have been previously referenced. Figure 2 It has been described.

[0108] According to the exemplary embodiment described, the processing unit 202 may assign scores or weights to each factor that contributes to the selection of an operating mode for the robotic goods handler 101. For example, scores / weights may be assigned to each of predefined heuristics, decision classifications, etc., to evaluate the selection of an operating mode for the robotic goods handler 101.

[0109] In an exemplary implementation, the decision to select the operating mode of the robotic goods handler 101 may be based on a weighted average evaluation of multiple factors. For example, in one example, a first weight may be associated with a decision classification, and a second weight may be associated with a predefined heuristic. In some examples, the first and second weights may be defined based on user input. Alternatively, the first and second weights may be dynamically defined by the processing unit 202. For example, the processing unit may dynamically define the first and second weights based on the type of goods handling operation, or the result or performance of the robotic goods handler 101 performing the task according to an operating mode that may have been selected using only the decision classification output by the ML model or a predefined heuristic.

[0110] Moving to step 706, processing unit 202 can adjust the first weight associated with the decision classification and the second weight associated with a predefined heuristic. In some examples, the adjustment of the first and second weights can be performed dynamically by the processing unit. In one example, the weight adjustment can be based on the performance associated with the output of the ML model over a period of time. For example, in the initial stage of training the ML model, the processing unit can assign a smaller weight to the ML model output compared to the weights of the predefined heuristic. Gradually, as the ML model is trained and utilized for a longer period, the processing unit can adjust the first weight associated with the ML model output to be higher than the second weight associated with the predefined heuristic. Therefore, the weights associated with various factors can be adjusted.

[0111] At step 708, processing unit 202 may generate commands for controlling the robotic goods handler 101 based on an evaluation of the selected operating mode, as described at steps 704 and 706. Thereafter, one or more components of the robotic goods handler 101 (e.g., robotic arm 115 or platform 117) may be actuated based on the commands to perform goods handling (e.g., loading and / or unloading). The method stops at step 710.

[0112] Figure 8 An exemplary scene 800 is shown, illustrating the generation of a merged point cloud based on input from various sensor devices that can be associated with a robotic item unloader. These sensor devices can be configured to perform forward scanning by capturing one or more images (e.g., 3D images and / or color images) of a portion of the environment in front of the robotic item unloader. For example,Figure 1 Multiple sensor devices (102 to 108) are described at various locations on a robotic goods handling machine 101. According to some exemplary embodiments, image data captured by the various sensor devices (e.g., sensor devices 102 to 108) can be acquired and transformed to generate a merged point cloud. Furthermore, this merged point cloud data can be used as input to a CNN-based machine learning model, which outputs a decision classification for selecting an operating mode of the robotic goods unloader. Details of these aspects have been previously referenced. Figures 1 to 7 The description was provided.

[0113] By way of example, according to an exemplary embodiment, four sensor devices (802, 804, 806, and 808) may be associated with a robotic goods unloader (e.g., robotic goods transporter 101). In an exemplary embodiment, the first sensor device 802 and the second sensor device 804 may be associated with the top portion of the robotic goods unloader (e.g., the robotic arm 115 of the robotic goods transporter 101). Furthermore, the third sensor device 806 and the fourth sensor device 808 may be associated with the bottom portion of the robotic goods unloader (e.g., the platform 118 of the robotic goods transporter 101).

[0114] Examplely, each of the sensor devices (802 to 808) may include an IFM camera and an RGB camera. The IFM camera may be configured to capture an image including data in three dimensions (X, Y, and Z axes). In this respect, the IFM camera may correspond to any 3D or depth imaging camera capable of capturing a 3D image of the environment within its respective field of view. In some examples, to capture 3D image data, the IFM camera may be configured to measure the distance between the IFM sensor of the IFM camera and the nearest surface point in its field of view point by point using the time-of-flight (TOF) principle. In this respect, the IFM sensor may be configured to illuminate the scene in its field of view using an internal infrared light source, and this distance can be calculated using the time of travel of light reflected from the surface. Thus, the IFM camera associated with each of the sensor devices (802 to 808) can capture a 3D image. Alternatively, in some exemplary embodiments, the sensor devices (802 to 808) may use other techniques for capturing 3D image information. Furthermore, the RGB camera of each of the sensor devices (802 to 808) can be configured to capture color images of the scene within its field of view. In some exemplary embodiments, the color image information obtained from the RGB camera can be used to clearly identify different objects in the scene (e.g., characteristics of boxes, packages). Additionally, in some exemplary embodiments, the RGB information obtained from the RGB camera of the sensor device can be mapped back to the depth frame of the corresponding sensor device.

[0115] According to an exemplary implementation, with a robotic item unloader (e.g., such as...) Figure 3 The vision system 326) associated with the perception system can be derived from an array of IFM / 3D cameras and RGB cameras (e.g., such as...). Figure 8 The sensor device receives input (e.g., point cloud data). As shown, each RGB camera associated with the IFM camera can represent a single sensor unit of the sensor device. In this respect, the sensor device (802 to 808) may have one or more such sensor units.

[0116] According to some exemplary embodiments described herein, point clouds can be constructed based on image information from IFM cameras and RGB cameras. For example, a first sensor device 802 can provide first point cloud data 810, a second sensor device can provide second point cloud data 812, a third sensor device 806 can provide third point cloud data 814, and a fourth sensor device 808 can provide fourth point cloud data 816, respectively. In an exemplary embodiment, the first point cloud data 810 may include image information corresponding to the upper left portion of the environment in front of the robot unloader. Furthermore, the second point cloud data 812 may include image information corresponding to the upper right portion of the environment. Additionally, the third point cloud data 814 may include image information corresponding to the lower left portion of the environment, and the fourth point cloud data 816 may include image information corresponding to the lower right portion of the environment in front of the robot unloader.

[0117] By way of example, information from the first point cloud data 810, the second point cloud data 812, the third point cloud data 814, and the fourth point cloud data 816 may be transformed to generate a merged point cloud 820. According to an exemplary embodiment, the merged point cloud 820 may be a single representation of the entire field of view in front of the robot item unloader. In some examples, the merged point cloud 820 obtained after combining the upper left, upper right, lower left, and lower right portions may represent the entire view of an item docking station that can be located in front of the robot item unloader.

[0118] According to an exemplary embodiment, transforming individual point cloud information from each sensor device into a merged point cloud 820 may include image stitching (i.e., stitching together corresponding image data corresponding to individual point cloud data). Furthermore, image data (i.e., point cloud data) from each of the sensor devices (802 to 808) may be processed such that each image data appears to originate from a common viewpoint point (e.g., base frame 818) of the individual sensor devices of the robotic item unloader. Figure 8The exemplary embodiment shown may include a transformation to merged point cloud 820 that may include stitching together image information from first point cloud data 810, second point cloud data 812, third point cloud data 814, and fourth point cloud data 816. In some exemplary embodiments, the transformation may further include de-skewing the images captured by the sensor devices (802-808) so that objects initially appearing at an angle in the images are transformed to appear directly in front of (or parallel to) the sensor devices (802-808) and / or the robot item unloader. In this way, the outputs (i.e., 3D image information) from each of the sensor devices (802-808) may be adjusted so that they have a common reference point. Once the image data have the same reference point (e.g., centered via a correction operation), image stitching can be performed to generate merged point cloud 820. As shown, merged point cloud 820 may be used as input to a CNN by machine learning engine 402, the details of which have been previously referenced. Figure 4 It has been described.

[0119] Figure 9 A perspective view 900 of a robotic item unloader 902 according to an exemplary embodiment described herein is shown. The robotic item unloader 902 may correspond to either the robotic item handler 101 and / or the robotic item unloader 301, as previously referred to respectively. Figure 1 and Figure 3 As illustrated, the robotic article unloader 902 may include a robotic arm 904 having an end effector 906. The end effector 906 may include an array of vacuum suction cups 908. Similar to the previous references. Figure 5 Each vacuum suction cup in the vacuum suction cups 908 is configured to be actuated to move forward and / or backward from its initial position to grasp an item based on vacuum suction. Furthermore, the robotic item unloader 902 also includes a platform 910. The platform 910 is connectable to a conveyor 911. The platform 910 is configured to be movably positioned such that a portion 912 of the platform 910 can sweep across under a portion of one or more items, thereby pulling the one or more items onto the conveyor 911 of the robotic item unloader 902. In this respect, the platform 910 may include one or more mechanical components and may be configured to move in any of the X, Y, and / or Z directions as needed to pull the one or more items.

[0120] As shown in the figure, the robotic unloader 902 may include multiple sensor devices (914, 916, 918, 920, 922, etc.) that can be located at various positions on the robotic unloader 902. For example, sensor devices 916 and 914 may be located on the end effector 906 of the robotic arm 904. Furthermore, sensor devices 920 and 922 may be located on the platform 910 of the robotic unloader 902. Each of these sensor devices (914 to 922) can be configured to capture a 3D image of the environment within its respective field of view. Figure 9 An example of this operation is illustrated, wherein each of the sensor devices (914 to 922) performs a forward 3D scan for its corresponding field of view. Furthermore, similar to what has been described previously, the sensor devices (914 to 922) of the robotic unloader 902 can be communicatively coupled to a processing device (e.g., processing unit 202 of computing device 110), which can acquire 3D images captured by the sensor devices (914 to 918) and further perform processing on these 3D images to construct a merged point cloud from them. As previously described, the merged point cloud can represent the entire field of view in front of the robotic unloader 902 and can be used as input to a CNN to build a machine learning model. Furthermore, the machine learning model can be used to select the operating mode of the robotic unloader 902, as previously referenced. Figures 1 to 9 As stated above.

[0121] Figure 10 A perspective view 1000 of a robotic item unloader 902 having multiple sensor devices (914 to 922) according to an exemplary embodiment is shown. As shown, each of the sensor devices (914 to 922) can be configured to perform a forward 3D scan of a portion of the environment within its respective field of view (1002, 1004, 1006, 1008, etc.). In this respect, each of the sensor devices (914 to 922) can capture a 3D image that may include a portion of the environment in front of the robotic item unloader 902 within its respective field of view (1002 to 1008). For example, the sensor devices (914, 916) on the robotic arm 904 can capture 3D images, for example, of the upper left and upper right portions of the environment in front of the robotic item unloader 902. Similarly, the sensor devices (920, 922) on the platform 910 can capture images of the lower left and lower right portions of the environment in front of the robotic item unloader 902. Furthermore, as previously referenced... Figures 1 to 9 The point cloud data from each of these 3D images captured by sensor devices (914 to 922) can be transformed to generate a merged point cloud representing the entire environment.

[0122] Figure 11A schematic diagram 1100 illustrates an example of an electronic device 1101 (e.g., a computing device 110) that can be used with a robotic goods handler 101 according to another exemplary embodiment described herein. In some exemplary embodiments, the electronic device 1101 may correspond to, for example... Figure 1 The computing device 110 is described above. In some exemplary embodiments, one or more components of the electronic device 1101 may be used to perform reference... Figures 6 to 8 The one or more operations described above.

[0123] See now Figure 11 It illustrates a block diagram for the functions and operations performed in the exemplary embodiments described. In some exemplary embodiments, electronic device 1101 may provide networking and communication capabilities between a wired or wireless communication network and a server and / or communication devices. To provide additional context for its various aspects, Figure 11 The following discussion is intended to provide a brief, general description of a suitable computing environment in which various aspects of the implementation scheme can be implemented to facilitate the establishment of transactions between an entity and a third party. While the above description is in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that various implementation schemes can also be implemented in combination with other program modules and / or as a combination of hardware and software.

[0124] According to the exemplary embodiments described, the program module includes routines, programs, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will understand that the method of the present invention can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, and personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, each operatively coupled to one or more associated devices.

[0125] The aspects illustrated in the various implementations can also be practiced in a distributed computing environment, where certain tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules may reside in local memory storage devices and / or remote memory storage devices. According to some exemplary implementations, computing devices typically include various media, which may include computer-readable storage media or communication media; these two terms are used differently herein, as described below.

[0126] According to some exemplary embodiments, a computer-readable storage medium can be any available storage medium accessible by a computer, and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, a computer-readable storage medium can be implemented in conjunction with any method or technology used for storing information, such as computer-readable instructions, program modules, structured data, or unstructured data. Computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD ROM, digital versatile optical disc (DVD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or other tangible and / or non-transitory media that can be used to store desired information. A computer-readable storage medium can be accessed by one or more local or remote computing devices, for example, via access requests, queries, or other data retrieval protocols, to perform various operations on the information stored on the medium.

[0127] In some examples, communication media may embody computer-readable instructions, data structures, program modules, or other structured or unstructured data in data signals (such as modulated data signals, e.g., carrier waves or other transmission mechanisms), and include any information delivery or transport medium. The term "modulated data signal" or signal refers to one or more signals whose characteristics are set or altered in a manner that encodes information in one or more signals. By way of example and not limitation, communication media include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media).

[0128] refer to Figure 11 Implementing the various aspects described herein with respect to electronic device 1101 may include a processing unit 1104, a system memory 1106, and a system bus 1108. The system bus 1108 may be configured to couple system components, including but not limited to the system memory 1106, to the processing unit 1104. In some exemplary embodiments, the processing unit 1104 may be any of a variety of commercially available processors. Therefore, in some examples, dual-microprocessor and other multiprocessor architectures may also be used as the processing unit 1104.

[0129] According to some exemplary embodiments, system bus 1108 may be any of several types of bus architectures that can be further interconnected to memory buses (with or without memory controllers), peripheral buses, and local buses, using any of a variety of commercially available bus architectures. In some examples, system memory 1106 may include read-only memory (ROM) 1127 and random access memory (RAM) 1112. According to some exemplary embodiments, a basic input / output system (BIOS) is stored in non-volatile memory 1127 (such as ROM, EPROM, EEPROM), which contains basic routines that facilitate the transfer of information between components within electronic device 1101 (such as during startup). RAM 1112 may also include high-speed RAM, such as static RAM for caching data.

[0130] According to some exemplary embodiments, computing device 1101 may also include an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA), which may also be configured for external use within a suitable chassis (not shown), a floppy disk drive (FDD) 1118 (e.g., reading from or writing to a removable disk 1116), and an optical disk drive 1120 (e.g., reading from a CD-ROM, or reading from or writing to other mass-produced optical media such as a DVD). In some examples, hard disk drive 1114, disk drive 1122, and optical disk drive 1120 may be connected to system bus 1108 via hard disk drive interface 1124, disk drive interface 1126, and optical disk drive interface 1128, respectively. According to some exemplary embodiments, interface 1124 for an external drive implementation may include at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies. Other external drive connection technologies are also contemplated in embodiments of this subject matter.

[0131] According to some exemplary embodiments described herein, the drive and its associated computer-readable medium provide non-volatile storage of data, data structures, computer-executable instructions, etc. For electronic device 1101, the drive and medium are adapted to store any data in a suitable digital format. Although the above description of computer-readable media refers to HDDs, removable disks, and removable optical media (such as CDs or DVDs), those skilled in the art will understand that other types of media readable by electronic device 1101 (such as zip drives, magnetic tape cassettes, flash memory cards, magnetic tapes, etc.) may also be used in the exemplary operating environment. Furthermore, any such media may contain computer-executable instructions for performing the methods of the embodiments disclosed herein.

[0132] In some exemplary embodiments, multiple program modules may be stored in the drive and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. Therefore, in some examples, all or part of the operating system, application programs, modules, and / or data may also be cached in RAM 1112. It should be understood that various embodiments can be implemented using a variety of commercially available operating systems or combinations of operating systems.

[0133] According to some exemplary embodiments, a user can input commands and information into computing device 1101 via one or more wired / wireless input devices (e.g., keyboard 1138) and pointing devices (such as mouse 1140). Other input devices (not shown) may include microphones, IR remote controls, joysticks, game controllers, styluses, touchscreens, etc. In some examples, these and other input devices are typically connected to processing unit 1104 via input device interface 1142 coupled to system bus 1108, but may be connected via other interfaces such as parallel ports, IEEE 13114 serial ports, game ports, USB ports, IR interfaces, etc.

[0134] According to some exemplary embodiments, monitor 1144 or other types of display devices may also be connected to system bus 1108 via an interface (such as video adapter 1146). In addition to monitor 1144, computing device 1101 may also include other peripheral output devices (not shown), such as speakers, printers, etc.

[0135] According to some exemplary embodiments, computing device 1101 can operate in a networked environment using a logical connection to one or more remote computers (such as remote computer 1148) via wired and / or wireless communication. In some examples, remote computer 1148 may be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer-to-peer device, or other public network node, and typically includes many or all of the elements described with respect to a computer, although for simplicity only memory / storage device 1150 is shown. According to some exemplary embodiments, the depicted logical connection includes a wired / wireless connection to a local area network (LAN) 1152 and / or a larger network (e.g., a wide area network (WAN) 1154). Such LAN and WAN network environments are common in offices and companies and facilitate enterprise-wide computer networks such as corporate intranets, all of which can be connected to global communications networks, such as the Internet.

[0136] In some examples, when used in a LAN networking environment, computing device 1101 can be connected to LAN 1152 via a wired and / or wireless communication network interface or adapter 1156. Adapter 1156 facilitates wired or wireless communication with LAN 1152, which may also include a wireless access point configured thereon for communicating with wireless adapter 1156.

[0137] In an alternative example, when used in a WAN networking environment, computing device 1101 may include modem 1158, or a communication server connectable to WAN 1154, or other means for establishing communication over WAN 1154 (such as via the Internet). Modem 1158 (which may be an internal or external wired or wireless device) is connected to system bus 1108 via input device interface 1142. In a networking environment, program modules depicted with respect to the computer or parts thereof may be stored in remote memory / storage device 1150. It should be understood that the network connections shown are exemplary and other means for establishing communication links between computers may be used.

[0138] According to some exemplary embodiments, computing device 1101 is operable to communicate with any wireless device or entity operatively configured in wireless communication, such as a printer, scanner, desktop and / or laptop computer, portable data assistant, communication satellite, any equipment or location associated with a wirelessly detectable tag (e.g., a kiosk, newspaper stand, restroom), and telephone. This may also include at least Wi-Fi and Bluetooth. TM Wireless technology. Therefore, the communication can be a predefined structure like a regular network, or simply self-organizing communication between at least two devices.

[0139] According to some exemplary embodiments, Wi-Fi, or Wireless Fidelity, allows connection to the Internet from a sofa at home, a bed in a hotel room, or a conference room in the workplace without a network cable. For this purpose, Wi-Fi, as referred to herein, is a wireless technology similar to that used in mobile phones, enabling such devices, such as computers, to send and receive data anywhere indoors and outdoors, and within the range of a base station. Wi-Fi networks use radio technology known as IEEE 1102.11 (a, b, g, n, etc.) to provide secure, reliable, and fast wireless connectivity. Furthermore, according to some exemplary embodiments described herein, Wi-Fi networks can be used to connect computers or multiple electronic devices 102-10N to each other, to the Internet, and to wired networks (which use IEEE 1102.3 or Ethernet). Wi-Fi networks operate at data rates of, for example, 11 Mbps (1102.11b) or 54 Mbps (1102.11a) in unlicensed 2.4 and 5 GHz radio bands, or on products that include two bands (dual-band), thus providing real-world performance similar to the basic “9BaseT” wired Ethernet used in many offices.

[0140] In some example implementations, some of the operations described herein may be modified or further amplified as described below. Furthermore, in some implementations, additional optional operations may be included. It should be understood that each of the modifications, optional additions, or amplifications described herein may be included in the operations herein, either alone or in combination with any other feature described herein.

[0141] The foregoing method descriptions and process flowcharts are provided as illustrative examples only and are not intended to require or imply that the steps of the various embodiments must be performed in the presented order. As those skilled in the art will understand, the order of steps in the above embodiments can be performed in any order. Words such as “after,” “then,” “next,” etc., are not intended to limit the order of steps; these words are merely used to guide the reader through the description of the method. Furthermore, any reference to singular claim elements, for example, using the articles “a,” “an,” or “the,” should not be construed as limiting the element to the singular.

[0142] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally according to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application; however, such specific implementation decisions should not be construed as departing from the scope of the invention.

[0143] The hardware used to implement the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the aspects disclosed herein may be implemented or performed by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination designed to perform the functions described herein. The general-purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors incorporating a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuitry specific to a given function.

[0144] In one or more exemplary aspects, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored as one or more instructions or code on a non-transitory computer-readable medium or a non-transitory processor-readable storage medium. The steps of the methods or algorithms disclosed herein may be embodied in a processor-executable software module (or processor-executable instructions) that may reside on a non-transitory computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable storage medium may be any storage medium accessible to a computer or processor. By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible to a computer. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically copy data magnetically, while optical discs use lasers to copy data optically. Combinations of the above are also included within the scope of non-transitory computer-readable and processor-readable media. Additionally, the operation of a method or algorithm may reside as one or any combination or set of code and / or instructions on a non-transitory processor-readable medium and / or a computer-readable medium, which may be incorporated into a computer program product.

[0145] While various embodiments based on the principles disclosed herein have been shown and described above, modifications can be made by those skilled in the art without departing from the spirit and teachings of this disclosure. The embodiments described herein are representative only and not intended to be limiting. Many variations, combinations, and modifications are possible and are within the scope of this disclosure. Alternative embodiments resulting from the merging, integration, and / or omission of features of one or more embodiments are also within the scope of this disclosure. Therefore, the scope of protection is not limited by the description given above, but is defined by the following claims, which include all equivalents of the subject matter of the claims. Each claim is incorporated into the specification as further disclosure, and the claims are for one or more embodiments of the invention. Furthermore, any of the foregoing advantages and features may be associated with a particular embodiment, but the application of such published claims should not be limited to processes and structures that achieve any or all of the foregoing advantages or have any or all of the foregoing features.

[0146] Furthermore, the section headings used herein are intended to align with the recommendations of 37C.FR1.77 or to provide organizational clues. These headings should not limit or characterize one or more inventions that can be set forth in any of the claims published in this disclosure. For example, the description of the technology in the “Background Art” section should not be construed as an admission that a particular technology is prior art to any invention in this disclosure. Nor should “Summary of the Invention” be considered a limiting characterization of one or more inventions set forth in the published claims. Furthermore, any reference to the singular form “invention” in this disclosure should not be used to prove that there is only one novel point in this disclosure. Multiple inventions may be set forth according to the limitations of multiple claims published in this disclosure, and such claims accordingly define the inventions protected by them and their equivalents. In all cases, the scope of these claims should be considered in accordance with the advantages of the claims themselves, and should not be limited by the headings set forth herein.

[0147] Furthermore, without departing from the scope of this disclosure, technologies, systems, subsystems, and methods described and illustrated as separate or independent in the various embodiments may be combined or integrated with other systems, modules, technologies, or methods. Other items shown or discussed as directly coupled or communicating with each other may be indirectly coupled or communicating through an interface, device, or intermediate component, whether such coupling or communication is electrical, mechanical, or otherwise. Other examples of variations, substitutions, and modifications that can be identified by those skilled in the art without departing from the spirit and scope of the disclosure herein will also be provided.

[0148] Those skilled in the art will recognize many modifications and other embodiments of the invention set forth herein, which have the benefits of the teachings presented in the foregoing description and the associated drawings. Although the figures show only some components of the apparatus and systems described herein, it should be understood that various other components may be used in conjunction with the supply management system. Therefore, it should be understood that the invention is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. For example, various elements or components may be combined or integrated into another system, or certain features may be omitted or not implemented. Furthermore, the steps in the above-described methods may not necessarily occur in the order depicted in the drawings, and in some cases, one or more of the depicted steps may occur substantially simultaneously, or additional steps may be involved. Although specific terminology is used herein, it is used only in a general and descriptive sense and not for limiting purposes.

[0149] As will be understood, any such computer program instructions and / or other type of code may be loaded onto the circuitry of a computer, processor or other programmable device to produce a machine, such that the computer, processor, or other programmable circuitry executing the code on that machine forms means for performing various functions, including those described herein.

[0150] It should also be noted that all or some of the information presented by the exemplary displays discussed herein may be based on data received, generated, and / or maintained by one or more components of a local or networked system and / or circuitry. In some embodiments, one or more external systems, such as remote cloud computing and / or data storage systems, may also be utilized to provide at least some of the functions discussed herein.

[0151] As will be understood from the above and based on this disclosure, embodiments of the present invention can be configured as methods, personal computers, servers, mobile devices, backend network devices, etc. Therefore, embodiments can include various means that comprise entirely hardware or any combination of software and hardware. Furthermore, embodiments can take the form of a computer program product on at least one non-transitory computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied in the storage medium. Any suitable computer-readable storage medium can be utilized, including non-transitory hard disks, CD-ROMs, flash memory, optical storage devices, or magnetic storage devices.

Claims

1. A robotic unloading machine, the robotic unloading machine comprising: A vision system, the vision system comprising: A first sensor device, the first sensor device being positioned at a first location on the robot unloader; and A second sensor device is located at a second position on the robot unloader. The platform includes a conveyor configured to be movably positioned such that, during operation in a first operating mode, the front portion of the platform can sweep across under a portion of some items, thereby guiding the items onto the conveyor. A robotic arm, comprising an end effector configured to pick up an article by grasping a portion of the article when operating in a second operating mode; and A processing unit, communicatively coupled to at least one of the vision system, the platform, and the robotic arm, is configured to: Obtain first point cloud data associated with a first three-dimensional image captured by the first sensor device; Obtain second point cloud data associated with the second three-dimensional image captured by the second sensor device; Transform the first point cloud data and the second point cloud data to generate combined point cloud data; The combined point cloud data is used as input to the convolutional neural network to implement the convolutional neural network as a machine learning model; The machine learning model outputs a decision classification indicating a first probability associated with the first operating mode and a second probability associated with the second operating mode; Based on the decision classification output by the machine learning model, one of the first and second operating modes of the robotic goods handling machine is selected, wherein: In response to the first probability being higher than the second probability, the first operating mode is selected; In response to the second probability being higher than the first probability, the second operating mode is selected; and the robot unloading machine is controlled in the following manner: Operate the platform of the robot unloading machine according to the first operating mode; or The robot arm of the robot unloading machine is operated according to the second operating mode; The selection of the first and second operating modes is evaluated based on the first and second probabilities output by the machine learning model and a predefined heuristic associated with the past operations of the robotic unloader. Based on the performance associated with the output of the machine learning model over a period of time, adjust a first weight associated with the decision classification and a second weight associated with the predefined heuristic used to evaluate the selection, wherein during a first time period, the first weight is less than the second weight, and during a second time period, the second weight is less than the first weight, wherein the first time period is earlier than the second time period; and The robot unloader is controlled based on the evaluation of the selection.

2. The robotic unloading machine according to claim 1, wherein the first sensor device and the second sensor device each include at least one depth camera and a color camera, and wherein the first sensor device is coupled to the robotic arm of the robotic unloading machine, and the second sensor device is coupled to the platform of the robotic unloading machine.

3. The robotic item unloading machine according to claim 1, wherein, in order to perform at least one of loading and unloading of a plurality of items from the item docking station, the processing unit is configured to: Generate a first command for operating the robotic arm according to the first operating mode; or Generate a second command for operating the platform according to the second operating mode.

4. The robotic unloading machine according to claim 1, wherein the first sensor device is coupled to the robotic arm of the robotic unloading machine.

5. The robot unloading machine according to claim 1, wherein the second sensor device is coupled to the platform of the robot unloading machine.

6. A method for controlling a robotic goods handling machine, the method comprising: Obtain first point cloud data associated with a first three-dimensional image captured by a first sensor device of a robotic goods handling machine; Obtain second point cloud data associated with a second three-dimensional image captured by a second sensor device of a robotic goods handling machine; Transform the first point cloud data and the second point cloud data to generate combined point cloud data; The combined point cloud data is used as input to the convolutional neural network to implement the convolutional neural network as a machine learning model; The machine learning model outputs a decision classification indicating a first probability associated with a first operating mode and a second probability associated with a second operating mode; and The first operating mode is associated with picking up an item by gripping it with the end effector of the robotic arm of the robotic goods handling machine; and the second operating mode is associated with sweeping a pile of items from a docking station using the platform of the robotic goods handling machine. Based on the decision classification output by the machine learning model, one of the first and second operating modes of the robotic goods handling machine is selected. in: In response to the first probability being higher than the second probability, the first operating mode is selected, and In response to the second probability being higher than the first probability, the second operating mode is selected. Operate the robot item handling machine according to the first operating mode or the second operating mode; The selection of one of the first and second operating modes of the robotic goods handler is evaluated based on the decision classification output by the machine learning model and a predefined heuristic associated with the robot's past operations. Based on the performance associated with the output of the machine learning model over a period of time, adjust a first weight associated with the decision classification and a second weight associated with the predefined heuristic used to evaluate the selection, wherein during a first time period, the first weight is less than the second weight, and during a second time period, the second weight is less than the first weight, wherein the first time period is earlier than the second time period; and The robotic item handling machine is controlled based on the evaluation of the selection.

7. The method of claim 6, wherein the first sensor device is coupled to the robotic arm of the robotic goods handling machine, and the second sensor device is coupled to the platform of the robotic goods handling machine.

8. The method of claim 6, wherein the robotic article handler is a robotic carton unloader, the robotic carton handler being configured to perform at least one of loading and unloading articles.

9. A non-transitory computer-readable medium having computer-executable instructions stored thereon, the instructions performing operations including the following in response to execution by a processor: Obtain first point cloud data associated with a first three-dimensional image captured by a first sensor device of a robotic goods handling machine; Obtain second point cloud data associated with a second three-dimensional image captured by a second sensor device of a robotic goods handling machine; Transform the first point cloud data and the second point cloud data to generate combined point cloud data; The convolutional neural network is implemented as a machine learning model by using the combined point cloud data as input to the convolutional neural network; The machine learning model outputs a decision classification indicating a first probability associated with a first operating mode and a second probability associated with a second operating mode, wherein the first operating mode is associated with picking up an item by gripping it with the end effector of the robotic arm of the robotic item handling machine, and the second operating mode is associated with sweeping a pile of items from an item docking station using the platform of the robotic item handling machine. Based on the decision classification output by the machine learning model, one of the first and second operating modes of the robotic goods handling machine is selected, wherein: In response to the first probability being higher than the second probability, the first operating mode is selected, and In response to the second probability being higher than the first probability, the second operating mode is selected; and Commands for operating the robotic goods handling machine are generated based on the selection of one of the first and second operating modes. The selection of one of the first and second operating modes of the robotic goods handler is evaluated based on the decision classification output by the machine learning model and a predefined heuristic associated with the robot's past operations. Based on the performance associated with the output of the machine learning model over a period of time, the first weight associated with the decision classification and the second weight associated with the predefined heuristic used to evaluate the selection are adjusted, wherein during a first time period, the first weight is less than the second weight, and during a second time period, the second weight is less than the first weight, wherein the first time period is earlier than the second time period; and The commands for operating the robotic goods handling machine are generated based on the decision classification and the evaluation of the selection.

Images