Managing robots in workflows with human intervention

Abstract

A robotic system includes a plurality of humanoid robots configured to move within a commercial environment to autonomously perform one or more tasks; and a control system communicably coupled to the one or more humanoid robots and including a human operator interface. The control system is configured to perform operations including: identifying a communication from a particular humanoid robot of the plurality of humanoid robots during performance of a particular task of the one or more tasks; determining that the communication includes a request for supervision; notifying a human operator through the human operator interface of the request for supervision; and facilitating control of the particular humanoid robot by the human operator to resume or complete the particular task in response to the notifying.

Description

MANAGING ROBOTS IN WORKFLOWS WITH HUMAN INTERVENTIONTECHNICAL FIELD

[0001] The present disclosure describes systems and methods for managing robots, such as humanoid robots that operate autonomously or semi-autonomously, in workflows with human intervention.BACKGROUND

[0002] Humanoid robots, like other types of robots, are designed to work as autonomously as possible to complete the tasks assigned to a robot or group of robots. As with other types of autonomous robots, there can be times where the robots need human intervention with a portion of a task, navigation, localization, or a myriad of other temporary challenges that can be overcome quickly. This often requires an employee, potentially a process or automation engineer, in a warehouse, distribution center, factory, or other commercial environment to traverse to the robot and provide the assistance the robot needs to resume autonomous operation. This can require traversing a large facility or even traveling to a facility just to perform a simple operation that enables the robot to resume its task. This situation is exacerbated by a large fleet of robots deployed across a number of facilities and / or locations.SUMMARY

[0003] In an example implementation, a robotic system includes a plurality of humanoid robots configured to move within a commercial environment to autonomously perform one or more tasks; and a control system communicably coupled to the one or more humanoid robots and including a human operator interface. The control system is configured to perform operations including: identifying a communication from a particular humanoid robot of the plurality of humanoid robots during performance of a particular task of the one or more tasks; determining that the communication includes a request for supervision; notifying a human operator through the human operator interface of the request for supervision; and facilitating control of the particular humanoid robot by the human operator to resume or complete the particular task in response to the notifying.

[0004] In an aspect combinable with the example implementation, the human operator is remote from the commercial environment.

[0005] In another aspect combinable with one, some, or all of the previous aspects, the human operator is a single human operator.

[0006] In another aspect combinable with one, some, or all of the previous aspects, the commercial environment includes a plurality of commercial facilities separately located within a geographic area.

[0007] In another aspect combinable with one, some, or all of the previous aspects, the geographic area includes a city, a state, or a country.

[0008] In another aspect combinable with one, some, or all of the previous aspects, the geographic area includes a first geographic area and a second geographic area different than the first geographic area.

[0009] In another aspect combinable with one, some, or all of the previous aspects, at least one of the plurality of commercial facilities is located within the first geographic area and at least another of the plurality of commercial facilities is located within the second geographic area.

[0010] In another aspect combinable with one, some, or all of the previous aspects, the commercial environment is located within the first geographic area and the human operator is located within the second geographic area.

[0011] In another aspect combinable with one, some, or all of the previous aspects, the control system includes a local controller located within or near the commercial environment; and a remote controller including the human operator interface and located external to the commercial environment.

[0012] In another aspect combinable with one, some, or all of the previous aspects, the operation of notifying the human operator through the human operator interface of the request for supervision includes visually notifying the human operator through the human operator interface of the request for supervision.

[0013] In another aspect combinable with one, some, or all of the previous aspects, the operation of visually notifying the human operator through the human operator interface of the request for supervision includes providing one or more visual images captured by a vision system of the particular humanoid robot to the human operator interface.

[0014] In another aspect combinable with one, some, or all of the previous aspects, the local controller is part of the particular humanoid robot or another humanoid robot of the plurality of humanoid robots.

[0015] In another aspect combinable with one, some, or all of the previous aspects, the operations include subsequent to resuming or completing the particular task, releasing control of the particular humanoid robot by the human operator to facilitate autonomous operation of the particular humanoid robot to perform another task of the one or more tasks.

[0016] In another aspect combinable with one, some, or all of the previous aspects, the operations include facilitating control of a sub-set of humanoid robots of the plurality of humanoid robots by the human operator to resume or complete the particular task in response to the notifying.

[0017] In another aspect combinable with one, some, or all of the previous aspects, the operations include facilitating control of all of the plurality of humanoid robots by the human operator to resume or complete the particular task in response to the notifying.

[0018] In another aspect combinable with one, some, or all of the previous aspects, the operation of facilitating control of the particular humanoid robot by the human operator to resume or complete the particular task in response to the notifying includes adjusting an operation mode of the particular humanoid robot from an autonomous mode to a semi-autonomous mode to facilitate control of the particular humanoid robot by the human operator to resume or complete the particular task in response to the notifying.

[0019] In another example implementation, a computer-implemented method includes registering, with a control system, a plurality of humanoid robots within a commercial environment that includes a plurality of commercial products; identifying, with the control system, a communication from a particular humanoid robot of the plurality of humanoid robots during performance of a particular task of the one or more tasks; determining, with the control system, that the communication includes a request for supervision; notifying a human operator through a human operator interface of the control system of the request for supervision; and facilitating control of the particular humanoid robot by the human operator, with the control system, to resume or complete the particular task in response to the notifying.

[0020] In an aspect combinable with the example implementation, the human operator is remote from the commercial environment.

[0021] In another aspect combinable with one, some, or all of the previous aspects, the human operator is a single human operator.

[0022] In another aspect combinable with one, some, or all of the previous aspects, the commercial environment includes a plurality of commercial facilities separately located within a geographic area.

[0023] In another aspect combinable with one, some, or all of the previous aspects, the geographic area includes a city, a state, or a country.

[0024] In another aspect combinable with one, some, or all of the previous aspects, the geographic area includes a first geographic area and a second geographic area different than the first geographic area.

[0025] In another aspect combinable with one, some, or all of the previous aspects, at least one of the plurality of commercial facilities is located within the first geographic area and at least another of the plurality of commercial facilities is located within the second geographic area.

[0026] In another aspect combinable with one, some, or all of the previous aspects, the commercial environment is located within the first geographic area and the human operator is located within the second geographic area.

[0027] In another aspect combinable with one, some, or all of the previous aspects, notifying the human operator through the human operator interface of the control system of the request for supervision includes providing the request for supervision from a local controller located within or near the commercial environment to a remote controller that includes the human operator interface and is located external to the commercial environment.

[0028] In another aspect combinable with one, some, or all of the previous aspects, notifying the human operator through the human operator interface of the request for supervision includes visually notifying the human operator through the human operator interface of the request for supervision.

[0029] In another aspect combinable with one, some, or all of the previous aspects, visually notifying the human operator through the human operator interface of the request for supervision includes providing one or more visual images captured by a vision system of the particular humanoid robot to the human operator interface.

[0030] In another aspect combinable with one, some, or all of the previous aspects, the local controller is part of the particular humanoid robot or another humanoid robot of the plurality of humanoid robots.

[0031] Another aspect combinable with one, some, or all of the previous aspects includes subsequent to resuming or completing the particular task, releasing control of the particular humanoid robot by the human operator to facilitate autonomous operation of the particular humanoid robot to perform another task of the one or more tasks.

[0032] Another aspect combinable with one, some, or all of the previous aspects includes facilitating control of a sub-set of humanoid robots of the plurality of humanoid robots by the human operator, with the control system, to resume or complete the particular task in response to the notifying.

[0033] Another aspect combinable with one, some, or all of the previous aspects includes facilitating control of all of the plurality of humanoid robots by the human operator, with the control system, to resume or complete the particular task in response to the notifying.

[0034] In another aspect combinable with one, some, or all of the previous aspects, facilitating control of the particular humanoid robot by the human operator, with the control system, to resume or complete the particular task in response to the notifying includes adjusting an operation mode of the particular humanoid robot from an autonomous mode to a semi- autonomous mode to facilitate control of the particular humanoid robot by the human operator to resume or complete the particular task in response to the notifying.

[0035] Implementations of systems and methods according to the present disclosure can include one, some, or all of the following features. For example, implementations according to the present disclosure can utilize one-to-n (human operator-to-humanoid robot) supervised autonomy of a humanoid robot workforce to eliminate a need for a human operator to traverse across a facility or even travel to a facility just to perform a simple operation that enables a humanoid robot to begin, resume, and / or complete a task, whether that task is leveraging gross manipulation or dexterous manipulation that requires more fine motor skills. As another example, implementations according to the present disclosure can allow a human operator to leverage supervised autonomy for a large fleet of robots deployed across a number of facilities and / or locations. Further, implementations according to the present disclosure can allow a human operator to be employed with or otherwise contracted to an organization that supplies automation services or technical support to a different organization that has deployed the humanoid robots in a commercial environment.

[0036] The details of one or more implementations of the subject matter described in this disclosure are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG. l is a schematic illustration of an example implementation of a workflow in a commercial environment that includes multiple, autonomous humanoid robots under supervision by a human operator according to the present disclosure.

[0038] FIGS. 2A and 2B are example implementations of a humanoid robot according to the present disclosure.

[0039] FIG. 3 is a schematic illustration of an example remote controller (and view of a commercial environment with one or more humanoid robots) used by a human operator in a commercial environment during supervision of multiple, autonomous humanoid robots according to the present disclosure.

[0040] FIG. 4 shows an example implementation of a robotic workflow control suite according to the present disclosure.

[0041] FIG. 5 shows a schematic drawing of a control system that can be used in the workflow of FIG. 1 according to the present disclosure.DETAILED DESCRIPTION

[0042] FIG. l is a schematic illustration of an example implementation of a workflow in a commercial environment 100 that includes multiple, autonomous humanoid robots under supervision by a human operator (e.g., a single human operator) according to the present disclosure. Generally, the commercial environment 100 illustrates an example implementation of a one-to-n (or l:n) workflow in which a human operator 900 (e.g., one or more human operators 900) supervises the actions (and, in some cases, intervenes therein) of multiple humanoid robots 112a- 112n. The commercial environment 100 enables humanoid robots to operate autonomously (e.g., exclusive of direct control by the human operator 900 during the completion of one or more assigned or determined tasks) while also enabling one or more humans (such as human operator 900) to intervene (e.g., remotely) when the humanoid robot needs assistance to complete a task or with navigation, localization, or any other temporary challenges that the humanoid robot mightencounter. Some examples of assistance that the humanoid robots 112a- 112n can encounter are, e.g., dealing with plastic wrap on a box, a set of cases that are tightly stacked, a box that is mislabeled, or a piece that falls out of a box or a tote and needs to be placed back in the box or the tote.

[0043] In example aspects of the commercial environment 100, one-to-n human supervision of the multiple autonomous humanoid robots 112a-112n can be accomplished by remote teleoperation or remote action selection for a particular humanoid robot (e.g., one of 112a- 112n) or a fleet of humanoid robots (e.g., all of 112a-112n). The human operator 900 can be notified by a particular humanoid robot 112a- 112n that the robot has encountered a challenge where it needs human supervision or intervention. The human operator 900 can, in some aspects, leverage a vision system on the humanoid robot 112a- 112n to see the challenge that the robot is encountering. The human operator 900 can then leverage one or more controllers and / or a graphical user interfaces to, e.g., (a) directly control appendages or components (e.g., the arms and / or the hands) of the humanoid robot 112a- 112n, (b) drive the robot’s body through the world, and / or (c) select an action from a context-dependent list of available autonomous or semi- autonomous behaviors, all while seeing through the robot vision system, to complete the task or solve the challenge in which the robot 112a- 112n encounters.

[0044] Other forms of notification can also be implemented by the humanoid robot 112a- 112n in order for the human operator 900 to be made aware when the robot has encountered a challenge where it needs human supervision or intervention. For example, the humanoid robot 112a- 112n can transmit an electronic communication to the human operator 900. The electronic communication can be, for example, an electronic mail communication, a text communication, an instant messaging communication, or otherwise. In some aspects, the electronic communication can be part of communications 995 between the humanoid robot 112a-112n and the human operator 900 (e.g., through remote control station 990).

[0045] As a specific example, a humanoid robot could be faced with a situation where it is trying to pick a case from a tightly stacked storage pallet of boxes to be placed on an order pallet or conveyor but is unable to grasp a case because of how tightly stacked the cases are on the storage pallet. With a one-to-n supervised autonomy of a humanoid robot workforce, a human is notified that a particular humanoid robot cannot complete (or begin, or resume) a task. The human operator can then connect (e.g., remotely through a controller that is part of a control system of theenvironment) into the that particular humanoid robot and teleoperate the humanoid robot to create space between the tightly packed cases so that the humanoid robot can complete its task. The human operator can also remotely connect to a different humanoid robot to resolve an issue that the different humanoid robot might be facing, such as encountering an empty tote of parts that should have been full of parts.

[0046] After a humanoid robot has completed (or started or resumed) a task in which the human operator supervised the robot through remote control, the human operator can cede autonomous control of the robot’s actions back to the robot. In such aspects, therefore, each humanoid robot in the workforce of multiple humanoid robots can be semi-autonomous, in that for tasks or jobs that the humanoid robot(s) does not need supervision, the tasks or jobs are completed autonomously (e.g., without human operator intervention or interruption) by the robot(s) but for tasks or jobs in which supervision or intervention is required as determined by the humanoid robot(s), the human operator can communicate and control the robot(s) to provide such intervention or supervision. Of course, the concepts described herein regarding one-to-n supervised autonomy of a humanoid robot workforce can apply to all of the humanoid robots (i . e. , multiple) in a workforce, as well as a sub-set of all of the humanoid robots, as well as a single humanoid robot within the workforce.

[0047] Commercial environment 100 represents an example workflow in which a workforce of humanoid robots 112a- 112n. Optionally, this example workflow can include mobile robots (including autonomous guided vehicles). For example, commercial environment 100 can be a warehouse (in any industry) or other facility (construction warehouse, shipping center, packaging center, warehouse, distribution or fulfillment center, manufacturing facility or otherwise) in which one or more commercial products (i.e., products sold or otherwise transferred in commerce) are gathered and stored (at least transiently) and arranged in combinations for sale, shipping, further distribution channels, or packaging.

[0048] The workflow illustrated and described in FIG. 1 in which one-to-n supervised autonomy of a humanoid robot workforce can apply can represent a case pick-to-pallet process that integrates humanoid robots 112a-n and (optionally) autonomous mobile robots (AMRs) 108a- n to solve existing issues with existing case pick-to-pallet processes regardless of if those processes leverage humans along with pallet jacks, walkie riders, or other pallet transportation equipment, or leverage humans along with AMRs. The example workflow that integrates humanoid robots112a-n and AMRs 108a-n can eliminate or reduce a need for humans to walk long distances, lift and move heavy cases, and pulling pallet transportation equipment while also ensuring that cases are layered in the proper order on order pallets. And with the described one-to-n supervised autonomy of a humanoid robot workforce according to the present disclosure, human operator 900 need not even be co-located in the commercial environment 100 in order to provide controlling supervision or intervention with one or more of the humanoid robots 112a- 112n.

[0049] As shown in FIG. 1, commercial environment 100 (such as a warehouse) includes or encloses an occupiable volume 104 in a building structure 102. The volume 104 can store donor pallets 106a-n (with “n” representing a variable number according to the present disclosure), for example, on a ground surface, pallet racking, or shelving. Each donor pallet 106a-n can enclose, contain, or otherwise support one or more commercial products. In some aspects, each donor pallet 106a-n supports a particular, unique commercial product (e.g., in a large quantity). For example, as shown, donor pallet 106a supports commercial products 105, while donor pallet 106n supports commercial products 107. There can be as many donor pallets 106a-n and commercial products as desired or necessary.

[0050] In this example, one or more AMRs 108a-n are also positioned in the volume 104 and operable to move throughout the volume 104 to receive commercial products (105, 107, or others) onto order pallets 114a-n. In this example, each AMR 108a-n can include an order pallet or otherwise have an order pallet 114a-n placed onto the AMR (either by its own actions or by a human operator). Each order pallet 114a-n can have a specified or predetermined quantity and type of commercial products to be placed on the order pallet according to a commercial transaction that has been (or will be) completed. As shown in this example, each AMR 108a-n includes a safety field 110, which represents a volume surrounding the particular AMR into which other objects (such as humanoid robots 112a-n) should not enter (e.g., for safety or other reasons).

[0051] In this example, supervision control of the humanoid robots 112a-112n by the human operator 900 can occur when one or more of the humanoid robots 112a- 112n, while performing tasks autonomously, determines that supervision or intervention is necessary to complete or otherwise work on the task(s). For example, a donor pallet 106a-n may be overturned or spilling commercial product 105, or inaccessible in a particular area of the volume 104 (in a comer, stuck against a wall, etc.). As another example, order pallets 114a-n may not be accessiblefor commercial product 105 or 107 to be placed therein. There could be functional problems with one or more of the AMRs 108a-n as recognized by the humanoid robot(s) 112a-n.

[0052] As shown in this example, the one or more humanoid robots 112a-n are positioned in the volume 104, such as proximate to (e.g., within 1 ft., 2 ft., 3 ft., 4 ft., 5-10 ft., or otherwise) the donor pallets 106a-n. In some aspects, there can be a one-to-one ratio of humanoid robots 112a-n to donor pallets 106a-n. Alternatively, there can be more humanoid robots 112a-n than donor pallets 106a-n (e.g., more than one humanoid robot 112a-n can be positioned adjacent or assigned to a particular donor pallet 106a-n). Alternatively, there can be fewer humanoid robots 112a-n than donor pallets 106a-n (e.g., a particular humanoid robot 112a-n can be positioned adjacent or assigned to more than one donor pallet 106a-n). Generally, each humanoid robot 112a- n is operable to move to remove one or more commercial products from a particular donor pallet 106a-n (or more than one donor pallet) and, as described herein, place the removed commercial product(s) onto an order pallet.

[0053] Turning briefly, to FIGS. 2A and 2B, these figures illustrate example implementations of a humanoid robot 112a according to the present disclosure. As shown, the humanoid robot 112a can include a torso and lower body, with appendages (e.g., legs, feet, arms, hands, head) that mimic or otherwise resemble and functional similarly to the corresponding human body appendages. The humanoid robot 112a can include multifunction movement, such as: walking, squatting, bending at waist, kneeling, torso rotation, head rotation, lifting (from ground, to torso height, above head), carrying (e.g., commercial product 105 as shown in FIG. 2B) as well as other movements that mimic human natural movement). In addition, in some aspects, the humanoid robot 112a can include visual image sensing and recognition (e.g., built into the head or otherwise), radar, or lidar to detect objects within its path. As such, the humanoid robot 112a can be distinguished from the AMRs 108a-n, which can have movement capability (e.g., through wheels, tracks, rollers, or other non-humanlike apparatus) but do not include humanlike appendages. Example AMRs 108a-n can be, for example robots made by Fetch Robotics, Mobile Industrial Robots, OTTO Robotics or otherwise. Among other components, the humanoid robot 112a can include a visual assembly 155, such as visual receptors, that records (in real time) images (still or moving or both) of the robot’s surrounding environment. Such images can be provided, e g., to a control system for the commercial environment 100.

[0054] In this example workflow, the commercial environment 100 can include a Warehouse Management System (WMS) 999 that can, in some aspects, include, interface with, or incorporate a Warehouse Execution System (WES), a Warehouse Control System (WCS), and / or WES (or pick path optimization) functionality. Other functionality can be included with or interface with the WMS 999, such as cubing (or so-called “Tetris”) functionality, which allows the WMS 999 to instruct the humanoid robots 112a-n how and in what order to place product on one or more pallets (e.g., based on size and / or weight of the product, an order of removal of the product from the pallet such as last on-first off, an arrangement of the unloaded product within a store or other commercial enterprise, or a combination thereof).

[0055] The WMS 999, WES or WCS can be, for example, a microprocessor based control system that controls the operations of the humanoid robots 112a-n and AMRs 108a-n according to software instructions executable by the WMS 999, WES or WCS. In some aspects, the WMS 999, WES or WCS controls operations of the humanoid robots 112a-n and AMRs 108a-n to move commercial products (105, 107, and others) from donor pallets 106a-n to order pallets 114a-n (e.g., in specified quantities and in a specified order of loading) in order to fulfill a commercial transaction or otherwise. In some aspects, WMS 999, WES or WCS can be a physically separate control system that communicates (e.g., wired or wirelessly) with the humanoid robots 112a-n and AMRs 108a-n. Alternatively, some or all of the functionality (e.g., processing capability, memory storage, communications, software instructions) can be located in one or more of the humanoid robots 112a-n (such as, for example, within a head or torso of a humanoid robot). Thus, in some aspects, one or more of the humanoid robots 112a-n can act as the WMS 999 to control the humanoid robots 112a-n and AMRs 108a-n.

[0056] In some aspects of an example workflow of commercial environment 100, the WMS 999, WES or WCS can identify or register all of the humanoid robots 112a-n and AMRs 109a-n within the volume 104 (e.g., in order to determine which of the robots are activated or operable). The WMS 999, WES or WCS can then communicate with AMR 108c and direct AMR 108c toward the donor pallets 106a (e.g., in some cases, subsequent to picking up an empty order pallet 114a-n) in order to load an empty order pallet 114c with commercial products (105, 107, or others) to fulfill a specified transaction. In some aspects, the WMS 999, WES or WCS directs the AMR 108c toward a particular donor pallet 106a-n (and subsequently to other donor pallets 106a-n in a specific order) based on a size or weight (or both) of the commercial product supported on the particular donor pallets 106a-n.

[0057] As shown in this example, the human operator 900 can communicate 995 with the WMS 999, and in some aspects, to the humanoid robots 112a-n directly or through the WMS 999 with a remote control station 990 (or remote controller 990). Remote controller 990 can take many forms of control devices (as shown in FIGS. 3 and 4), such as computing workstations, smart phones, tablets, laptops, or other computing devices. Such communications 995 can be wireless, such as over an intranet, the global network known as the World Wide Web, a VPN network, or other communication network with any appropriate protocol. Communications 995 can be bidirectional between the remote controller 990 and the WMS 999 (which, in some aspects, combine to form an overall control system of the commercial environment 100).

[0058] In some aspects, human operator 900 can be remotely located from the commercial environment 100, which itself can represent multiple commercial facilities (e.g., multiple volumes 104) that are co-owned, co-operated, or independently operated. The commercial environment 100 can be the responsibility of an organization that also bears responsibility (ownership, control, or otherwise) of the humanoid robots 112a-n. Alternatively, the commercial environment 100 can be the responsibility of an organization that is different than an organization that bears responsibility (ownership, control, or otherwise) of the humanoid robots 112a-n.

[0059] In some aspects, one or more of the multiple commercial facilities represented by the commercial environment 100 can be located in a different geographic area (e.g., city, state, country or other geographical or geopolitically-defined area), relative to other of the multiple commercial facilities. In some aspects, the human operator 900 can be co-located in the same geographic area of some of the multiple commercial facilities but not others of the multiple commercial facilities. In some aspects, the human operator 900 can be co-located in none of the geographic areas in which the multiple commercial facilities are located. In some aspects, the human operator 900 can be co-located in the same geographic areas as all of the multiple commercial facilities.

[0060] In example operations of a workflow involving the commercial environment 100, in loading the empty order pallet 114c, it may be beneficial or advantageous to load commercial product heaviest (or largest) to lightest (or smallest) according to cubing functionality built into or interfaced with the WMS 999, WES or WCS. In such aspects, lighter commercial product maynot be crushed or damaged by later-loaded and heavier commercial product. In some instances, the WMS can provide one or more AMRs 108a-n and / or one or more humanoid robots 112a-n one or more tasks. In some aspects, such as with the inclusion of cubing functionality, the WMS 999, WES or WCS can instruct, e.g., the humanoid robots 112a-n told to pick multiple cases and place them on a pallet, with a location specificity of placement on the pallet varying depending on the aforementioned criteria.

[0061] After the AMR 108c is directed to a particular donor pallet, such as donor pallet 106a to pick up commercial product 105, the WMS 999 controls the humanoid robot 112a at the donor pallet 106a to pick a specific number of commercial product 105 from the donor pallet 106a and place the picked commercial product 105 onto the order pallet 114c. In some aspects, the WMS 999 directs the humanoid robot 112a to remain close to the donor pallet 106a until the AMR 108c has stopped at a specific location to ensure that the humanoid robot 112a does not trigger the safety field 110 of the AMR 108c. In some aspects, the WMS 999 or controller of the humanoid robot 112a (e.g., in the head or torso of the humanoid robot 112a) directs a proper placement of each commercial product 105 on the order pallet 114c to create the correct layers of product on the order pallet 114c. After picking and placing the appropriate number of commercial product 105 in the correct location(s) on the order pallet 114c, the humanoid robot 112a repositions itself close to the donor pallet 106a (e.g., at pallet racking or shelving) so that the AMR 108c can leave the area around the donor pallet 106a without its safety field 110 being triggered.

[0062] The above-described operations can be repeated at one or more additional donor pallets 106a-n. For example, the AMR 108c can next be directed to donor pallet 106n to pick up commercial product 107. After picking and placing the appropriate number of commercial product 107 in the correct location(s) on the order pallet 114c, the humanoid robot 112n repositions itself close to the donor pallet 106n (e.g., at pallet racking or shelving) so that the AMR 108c can leave the area around the donor pallet 106n without its safety field 110 being triggered. If the order pallet 114c is complete, the AMR 108c can move the order pallet 114c toward a location in the volume 104 in which the order pallet 114c can be shipped or otherwise packaged (e.g., as shown with AMR 108b moving a completed order pallet 114b).

[0063] In the above described example operations, the humanoid robot 112a (or other robots in the workforce of robots 112a-n) can encounter problems or other issues that prevent or hinder a task from being completed autonomously. When this occurs, the humanoid robot 112acan notify, e.g., through the WMS 999, the remote controller 990 and thus the human operator 900 of the problem or issue. In response, the human operator 900 can take control of the humanoid robot 112a (i.e., adjust the robot’s operating mode from autonomous to semi-autonomous) and direct the humanoid robot 112a to take specific action (move, pick up, put down, etc.) through the remote controller 990 (either directly to the robot 112a or through the WMS 999) to resolve the problem or issue. Once the problem or issues is resolved, the human operator 900 can cede control back to the humanoid robot 112a to carry on in autonomous operation mode for future tasks.

[0064] FIG. 3 is a schematic illustration of an example remote controller (and view of a commercial environment with one or more humanoid robots) used by a human operator in a commercial environment during supervision of multiple, autonomous humanoid robots according to the present disclosure. For example, as described, each humanoid robot 112a-n can include visual receptors (e.g., cameras, video links) that can record (e.g., in real time) a surrounding environment. Here, view 305 shows a surrounding environment of a humanoid robot, as well an object (“box”) and command tasks (“pick up” and “move”).

[0065] In example implementations, if a particular humanoid robot 112a-n requires supervision or intervention from the human operator 900 during any of the tasks described above in the commercial environment 100, the humanoid robot 112a-n (or WMS 999) can notify the human operator 900 through the remote controller 990 (e.g., through an operator interface, such as a graphical user interface). In doing so, the visual receptors (or visual assembly 155) can record one or more images that show or illustrate the environment and provide the image(s) to the human operator 900 through view 305. The human operator 900, therefore, can “see” the surrounding environment as the humanoid robot 112a “sees” it and can make more informed decisions regarding taking control of the robot 112a for supervision or intervention. Additionally, or alternatively, in the example of a workforce of multiple human robots 112a-n, the particular humanoid robot 112a-n that determines a need for supervision or intervention can be different than the humanoid robot 112a-n that captures the image(s) of the task that needs supervision or intervention and provides such image(s) to the human operator 900 through view 305.

[0066] In some aspects, sub-tasks performed by the humanoid robots 112a-n (during the performance of overall tasks within the commercial environment 100) can include both gross and fine movements. For example, gross movements can include the movement, carrying, picking up of totes and cases within the commercial environment 100. Fine movements can include dexterousmanipulation such as picking individual objects, which can include picking and pack-out in logistics as well as kitting and part placement in robotic work cells in manufacturing.

[0067] FIG. 4 shows an example implementation of a robotic workflow control suite 400 according to the present disclosure. For example, the described workflow functionality of a human operator that can supervise multiple humanoid robots in a commercial environment in a one-to-n supervision role can be accessed or otherwise completed from several different control components, including controller 300, back-end devices such as servers, or front end user devices such as tablets, phones, laptops, and desktop workstations as shown in the suite 400.

[0068] FIG. 5 shows a schematic drawing of a control system 500 that can be used in the example workflow of FIG. 1 according to the present disclosure. For example, all or parts of the control system (or controller) 500 can be used for the operations described previously, for example as or as part of the Warehouse Management System (WMS) 999. Some or all of the example control system 500 (or WMS 999 generally) can be implemented as cloud-based system and / or service, alone or in combination with other portions of the example control system 500 that can be implemented at the commercial environment 100. The controller 500 is intended to include various forms of digital computers, such as printed circuit boards (PCB), processors, digital circuitry, or otherwise. Additionally, the system can include portable storage media, such as, Universal Serial Bus (USB) flash drives. For example, the USB flash drives may store operating systems and other applications. The USB flash drives can include input / output components, such as a wireless transmitter or USB connector that may be inserted into a USB port of another computing device.

[0069] The controller 500 includes a processor 510, a memory 520, a storage device 530, and an input / output device 540. Each of the components 510, 520, 530, and 540 are interconnected using a system bus 550. The processor 510 is capable of processing instructions for execution within the controller 500. The processor may be designed using any of a number of architectures. For example, the processor 510 may be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor.

[0070] In one implementation, the processor 510 is a single-threaded processor. In another implementation, the processor 510 is a multi -threaded processor. The processor 510 is capable ofprocessing instructions stored in the memory 520 or on the storage device 530 to display graphical information for a user interface on the input / output device 540.

[0071] The memory 520 stores information within the control system 500. In one implementation, the memory 520 is a computer-readable medium. In one implementation, the memory 520 is a volatile memory unit. In another implementation, the memory 520 is a nonvolatile memory unit.

[0072] The storage device 530 is capable of providing mass storage for the controller 500. In one implementation, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, a tape device, flash memory, a solid state device (SSD), or a combination thereof.

[0073] The input / output device 540 provides input / output operations for the controller 500. In one implementation, the input / output device 540 includes a keyboard and / or pointing device. In another implementation, the input / output device 540 includes a display unit for displaying graphical user interfaces.

[0074] The features described can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The apparatus can be implemented in a computer program product tangibly embodied in an information carrier, for example, in a machine-readable storage device for execution by a programmable processor; and method steps can be performed by a programmable processor executing a program of instructions to perform functions of the described implementations by operating on input data and generating output. The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0075] Suitable processors for the execution of a program of instructions include, by way of example, both general and special purpose microprocessors, and the sole processor or one ofmultiple processors of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, solid state drives (SSDs), and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD- ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

[0076] To provide for interaction with a user, the features can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) or LED (light-emitting diode) monitor for displaying information to the user and a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer. Additionally, such activities can be implemented via touchscreen flat-panel displays and other appropriate mechanisms.

[0077] The features can be implemented in a control system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or an Internet server, or that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. The components of the system can be connected by any form or medium of digital data communication such as a communication network. Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), peer-to-peer networks (having ad-hoc or static members), grid computing infrastructures, and the Internet.

[0078] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular implementations of particular inventions. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various featuresthat are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0079] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0080] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. For example, example operations, methods, or processes described herein may include more steps or fewer steps than those described. Further, the steps in such example operations, methods, or processes may be performed in different successions than that described or illustrated in the figures. Accordingly, other implementations are within the scope of the following claims.

Claims

WHAT IS CLAIMED IS:

1. A robotic system, comprising: a plurality of humanoid robots configured to move within a commercial environment to autonomously perform one or more tasks; and a control system communicably coupled to the one or more humanoid robots and comprising a human operator interface, the control system configured to perform operations comprising: identifying a communication from a particular humanoid robot of the plurality of humanoid robots during performance of a particular task of the one or more tasks; determining that the communication comprises a request for supervision; notifying a human operator through the human operator interface of the request for supervision; and facilitating control of the particular humanoid robot by the human operator to resume or complete the particular task in response to the notifying.

2. The robotic system of claim 1, wherein the human operator is remote from the commercial environment.

3. The robotic system of either one of claims 1 or 2, wherein the human operator is a single human operator.

4. The robotic system of any one of the previous claims 1-3, wherein the commercial environment comprises a plurality of commercial facilities separately located within a geographic area.

5. The robotic system of claim 4, wherein the geographic area comprises a city, a state, or a country.

6. The robotic system of claim 4, wherein the geographic area comprises a first geographic area and a second geographic area different than the first geographic area.

7. The robotic system of claim 6, wherein at least one of the plurality of commercial facilities is located within the first geographic area and at least another of the plurality of commercial facilities is located within the second geographic area.

8. The robotic system of claim 6, wherein the commercial environment is located within the first geographic area and the human operator is located within the second geographic area.

9. The robotic system of any one of the previous claims 1-8, wherein the control system comprises: a local controller located within or near the commercial environment; and a remote controller comprising the human operator interface and located external to the commercial environment.

10. The robotic system of claim 9, wherein the operation of notifying the human operator through the human operator interface of the request for supervision comprises: visually notifying the human operator through the human operator interface of the request for supervision.

11. The robotic system of claim 10, wherein the operation of visually notifying the human operator through the human operator interface of the request for supervision comprises: providing one or more visual images captured by a vision system of the particular humanoid robot to the human operator interface.

12. The robotic system of claim 9, wherein the local controller is part of the particular humanoid robot or another humanoid robot of the plurality of humanoid robots.

13. The robotic system of any one of the previous claims 1-12, wherein the operations comprise: subsequent to resuming or completing the particular task, releasing control of the particular humanoid robot by the human operator to facilitate autonomous operation of the particular humanoid robot to perform another task of the one or more tasks.

14. The robotic system of any one of the previous claims 1-13, wherein the operations comprise: facilitating control of a sub-set of humanoid robots of the plurality of humanoid robots by the human operator to resume or complete the particular task in response to the notifying.

15. The robotic system of any one of the previous claims, wherein the operations comprise: facilitating control of all of the plurality of humanoid robots by the human operator to resume or complete the particular task in response to the notifying.

16. The robotic system of any one of the previous claims 1-14, wherein the operation of facilitating control of the particular humanoid robot by the human operator to resume or complete the particular task in response to the notifying comprises: adjusting an operation mode of the particular humanoid robot from an autonomous mode to a semi-autonomous mode to facilitate control of the particular humanoid robot by the human operator to resume or complete the particular task in response to the notifying.

17. A computer-implemented method, comprising: registering, with a control system, a plurality of humanoid robots within a commercial environment that includes a plurality of commercial products; identifying, with the control system, a communication from a particular humanoid robot of the plurality of humanoid robots during performance of a particular task of the one or more tasks; determining, with the control system, that the communication comprises a request for supervision; notifying a human operator through a human operator interface of the control system of the request for supervision; and facilitating control of the particular humanoid robot by the human operator, with the control system, to resume or complete the particular task in response to the notifying.

18. The computer-implemented method of claim 17, wherein the human operator is remote from the commercial environment.

19. The computer-implemented method of either one of claims 17 or 18, wherein the human operator is a single human operator.

20. The computer-implemented method of any one of the previous claims 17-19, wherein the commercial environment comprises a plurality of commercial facilities separately located within a geographic area.

21. The computer-implemented method of claim 20, wherein the geographic area comprises a city, a state, or a country.

22. The computer-implemented method of claim 20, wherein the geographic area comprises a first geographic area and a second geographic area different than the first geographic area.

23. The computer-implemented method of claim 22, wherein at least one of the plurality of commercial facilities is located within the first geographic area and at least another of the plurality of commercial facilities is located within the second geographic area.

24. The computer-implemented method of claim 22, wherein the commercial environment is located within the first geographic area and the human operator is located within the second geographic area.

25. The computer-implemented method of any one of the previous claims 17-24, wherein notifying the human operator through the human operator interface of the control system of the request for supervision comprises: providing the request for supervision from a local controller located within or near the commercial environment to a remote controller that comprises the human operator interface and is located external to the commercial environment.

26. The computer-implemented method of claim 25, wherein notifying the human operator through the human operator interface of the request for supervision comprises: visually notifying the human operator through the human operator interface of the request for supervision.

27. The computer-implemented method of claim 26, wherein visually notifying the human operator through the human operator interface of the request for supervision comprises: providing one or more visual images captured by a vision system of the particular humanoid robot to the human operator interface.

28. The computer-implemented method of claim 25, wherein the local controller is part of the particular humanoid robot or another humanoid robot of the plurality of humanoid robots.

29. The computer-implemented method of any one of the previous claims 17-28, comprising: subsequent to resuming or completing the particular task, releasing control of the particular humanoid robot by the human operator to facilitate autonomous operation of the particular humanoid robot to perform another task of the one or more tasks.

30. The computer-implemented method of any one of the previous claims 17-29, comprising: facilitating control of a sub-set of humanoid robots of the plurality of humanoid robots by the human operator, with the control system, to resume or complete the particular task in response to the notifying.

31. The computer-implemented method of any one of the previous claims 17-30, comprising: facilitating control of all of the plurality of humanoid robots by the human operator, with the control system, to resume or complete the particular task in response to the notifying.

32. The computer-implemented method of any one of the previous claims 17-31, wherein facilitating control of the particular humanoid robot by the human operator, with the control system, to resume or complete the particular task in response to the notifying comprises: adjusting an operation mode of the particular humanoid robot from an autonomous mode to a semi-autonomous mode to facilitate control of the particular humanoid robot by the human operator to resume or complete the particular task in response to the notifying.

Images