Systems, devices and methods for developing robotic autonomy via teleoperation

Abstract

In a method of operating a robot, the robot identifies a set of candidate actions that can be performed by the robot, and for each candidate action in the set of candidate actions, collects a corresponding auxiliary data set. The robot transmits a request for instructions to a teleoperation system communicatively coupled to the robot. The request for instructions includes each candidate action and each corresponding auxiliary data set. The robot receives and executes the instructions from the teleoperation system. The robot updates a control model based at least in part on each candidate action, each corresponding auxiliary data set, and the instructions to increase the level of autonomy of the robot. The robot may transmit a request for instructions to the teleoperation system in response to determining that the robot cannot select a candidate action to be performed to facilitate a goal.

Description

Technical Field

[0001] The present systems, apparatus, and methods relate generally to robots, and more particularly to training robots to operate at least semi-autonomously.

[0002] background

[0003] Robots are machines that can assist humans or replace humans. Robots can be used in a variety of applications, including construction, manufacturing, monitoring, exploration, learning, and entertainment. For example, robots can be used in dangerous or uninhabitable environments.

[0004] Some robots require user input and can be operated by humans. Other robots have some degree of autonomy and, in at least some cases, can operate without human intervention. Some autonomous robots are designed to mimic human behavior. Autonomous robots can be particularly useful in applications that require the robot to operate for extended periods of time without operator intervention, navigate its operating environment, and / or adapt to changing conditions.

[0005] Brief Overview

[0006] A robot operation method performed by a robot can be summarized as including the following items: identifying a set of candidate actions that can be performed by the robot, for each candidate action in the set of candidate actions, collecting a corresponding auxiliary data set, transmitting a request for instructions to a tele-operation system, the tele-operation system being communicatively coupled to the robot, the request for instructions including each candidate action in the set of candidate actions and each corresponding auxiliary data set, receiving instructions from the tele-operation system, executing the instructions, and updating a control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and the instructions to increase the level of autonomy of the robot.

[0007] In some embodiments, identifying a set of candidate actions that can be performed by the robot includes identifying a set of candidate actions in response to detecting at least one of an event or a condition. Identifying a set of candidate actions in response to detecting at least one of an event or a condition may include identifying a set of candidate actions in response to detecting at least one of an external event or an external condition. In some embodiments, the robot includes one or more sensors, wherein identifying a set of candidate actions in response to detecting at least one of an external event or an external condition includes sensing at least one of the external event or the external condition by the one or more sensors.

[0008] In some embodiments, the method further includes receiving a goal to be completed by the robot, wherein identifying a set of candidate actions that can be performed by the robot includes: identifying a set of candidate actions that can be performed by the robot in furtherance of the goal. Receiving the goal to be completed by the robot may include receiving the goal from a teleoperation system. Transmitting a request for instructions to the teleoperation system may include transmitting a request for instructions to the teleoperation system in response to determining that the robot cannot select a candidate action from the set of candidate actions to perform.

[0009] In some embodiments, transmitting a request for instructions to the teleoperation system in response to determining that the robot cannot select a candidate action from the set of candidate actions to perform includes: transmitting a request for instructions to the teleoperation system in response to determining that the robot cannot select a candidate action from the set of candidate actions to perform to facilitate a goal.

[0010] In some embodiments, the method further includes scoring each candidate action in the set of candidate actions, wherein scoring each candidate action in the set of candidate actions includes assigning a corresponding probability to each candidate action, the corresponding probability being based at least in part on a corresponding expectation of progress toward a goal that can be achieved by each candidate action. In some embodiments, the method further includes sorting each candidate action in a preference order, the preference order being based at least in part on the corresponding probability of each candidate action. In some embodiments, the method further includes using at least one of the corresponding probability and the preference order of each candidate action to determine whether the robot can select a candidate action in the set of candidate actions to perform.

[0011] In some embodiments, transmitting a request for instructions to the teleoperation system includes transmitting a request for instructions including each corresponding probability and a preference order. In some embodiments, using at least one of the corresponding probability and the preference order of each candidate action to determine whether the robot can select a candidate action from the set of candidate actions to perform includes: determining whether the corresponding probability meets a predetermined probability criterion. In some embodiments, determining whether the corresponding probability meets the predetermined probability criterion includes determining whether the corresponding probability exceeds a predetermined probability threshold. In some embodiments, using at least one of the corresponding probability and the preference order of each candidate action to determine whether the robot can select a candidate action from the set of candidate actions to perform includes: determining whether a difference between a first corresponding probability and a second corresponding probability meets a predetermined difference criterion.

[0012] In some embodiments, the robot is a semi-autonomous robot characterized by a level of autonomy, the level of autonomy being determined at least in part by a control model, and the method further comprises comparing the level of autonomy of the robot to a threshold level of autonomy, above which the robot is operable to autonomously select a candidate action from the set of candidate actions to be performed, wherein transmitting a request for instructions to the teleoperation system comprises transmitting a request for instructions to the teleoperation system in response to determining that the level of autonomy of the robot is below the threshold level of autonomy.

[0013] In some embodiments, the method further comprises, in response to determining that the level of autonomy of the robot meets or exceeds a threshold level of autonomy, performing one of the candidate actions in the set of candidate actions. In some embodiments, the method further comprises determining a threshold level of autonomy.

[0014] In some implementations, the method further includes receiving a goal to be accomplished by the robot, wherein identifying a set of candidate actions that can be performed by the robot includes identifying a set of candidate actions that can be performed by the robot to facilitate the goal. Receiving the goal to be accomplished by the robot may include receiving the goal from a teleoperation system.

[0015] In some embodiments, the method further includes determining a threshold level of autonomy, wherein determining the threshold level of autonomy includes assigning a respective probability to each candidate action, the respective probability being based at least in part on a respective expectation of progress toward a goal that can be achieved by each candidate action. In some embodiments, the method further includes ranking each candidate action in an order of preference, the order of preference being based at least in part on the respective probability. Determining that the level of autonomy of the robot is below the threshold level of autonomy may include determining that the respective probability of each candidate action in the set of candidate actions fails to satisfy at least one predetermined criterion.

[0016] In some embodiments, the robot includes one or more sensors and a data repository, and for each candidate action in the set of candidate actions, collecting the corresponding auxiliary data set includes at least one of: collecting a first subset of the corresponding auxiliary data set by the one or more sensors or extracting a second subset of the corresponding auxiliary data set from the data repository. Collecting the first subset of the corresponding auxiliary data set by the one or more sensors may include collecting at least one of audio, imagery, video, or environmental data.

[0017] In some embodiments, the robot includes a communication subsystem, and transmitting the request for instructions to the teleoperation system and receiving instructions from the teleoperation system includes transmitting the request for instructions by the communication subsystem and receiving instructions by the communication subsystem.

[0018] In some embodiments, the robot includes a controller, wherein executing the instructions and updating the control model includes executing the instructions by the controller and updating the control model by the controller.

[0019] Transmitting the request for the instruction to the teleoperation system may include transmitting the request for the instruction to a piloted teleoperation system. Transmitting the request for the instruction to the teleoperation system may include transmitting the request for the instruction to a teleoperation system located at a remote location relative to the robot. Transmitting the request for the instruction to the teleoperation system may include transmitting the request for the instruction to the teleoperation system over a data network. The data network may be a wireless data network.

[0020] Transmitting the request for instructions to the teleoperation system may include transmitting a request to the teleoperation system to determine which candidate action the robot should perform. Transmitting the request for instructions to the teleoperation system may include entering a standby mode.

[0021] Executing the instruction may include invoking at least one candidate action from the set of candidate actions. In some embodiments, the robot includes a motion actuator, and invoking at least one candidate action includes causing the robot to perform a movement via the motion actuator. Executing the instruction may include identifying one or more auxiliary actions that can be performed by the robot.

[0022] In some embodiments, updating the control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and instructions includes increasing the autonomy level of the robot from a first autonomy level to a second autonomy level, the second autonomy level being higher than the first autonomy level. Updating the control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and instructions may include updating at least one of control instructions, control data, or control parameters. Updating the control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and instructions may include updating at least one of a configuration setting, a control method, a strategy, a process, a classifier, or a rule.

[0023] A robot can be summarized as comprising: a body; a controller, the controller including at least one processor; at least one physically actuatable component mechanically coupled to the body and communicatively coupled to the controller; at least one sensor communicatively coupled to the controller; at least one sensor operable to collect data; and at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to identify a set of candidate actions that can be performed by the robot, for each candidate action in the set of candidate actions, collect a corresponding auxiliary data set, transmit a request for instructions to a teleoperation system communicatively coupled to the robot, the request for instructions including each candidate action in the set of candidate actions and each corresponding auxiliary data set, receive instructions from the teleoperation system, execute the instructions and update a control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set and the instructions to increase a level of autonomy of the robot.

[0024] The robot may be a humanoid robot and the body may have the appearance of a human body. The at least one physically actuatable component may include a motion actuator. The at least one sensor may be operable to collect at least one of audio, image, video or environmental data.

[0025] In some embodiments, at least one non-transitory processor-readable storage medium also stores processor-executable instructions and / or data, which, when executed by at least one processor, cause the robot to detect at least one of an event or condition through at least one sensor; and processor-executable instructions and / or data, which, when executed by at least one processor, cause the robot to identify a set of candidate actions that can be performed by the robot, and also cause the robot to identify a set of candidate actions that can be performed by the robot in response to detecting at least one of the events or conditions.

[0026] In some embodiments, at least one non-transitory processor-readable storage medium also stores processor-executable instructions and / or data, which, when executed by at least one processor, cause the robot to receive a goal to be completed by the robot received from a teleoperation system; and processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to identify a set of candidate actions that can be performed by the robot, and also cause the robot to identify a set of candidate actions that can be performed by the robot to facilitate the goal.

[0027] In some embodiments, the processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to transmit a request for instructions to the teleoperation system further cause the robot to transmit a request for instructions to the teleoperation system in response to determining that the robot cannot select a candidate action in the set of candidate actions to be performed. In some embodiments, at least one non-transitory processor-readable storage medium further stores processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to score each candidate action in the set of candidate actions by assigning a respective probability to each candidate action, the respective probability being based at least in part on a respective expectation of progress toward a goal that can be achieved by each candidate action. In some embodiments, at least one non-transitory processor-readable storage medium further stores processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to rank each candidate action in an order of preference, the order of preference being based at least in part on the respective probability of each candidate action.

[0028] In some embodiments, at least one non-transitory processor-readable storage medium further stores processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to use at least one of the corresponding probability and the preference order of each candidate action to determine whether the robot can select a candidate action from the set of candidate actions to be performed. In some embodiments, the processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to use at least one of the corresponding probability and the preference order of each candidate action to determine whether the robot can select a candidate action from the set of candidate actions to be performed also cause the robot to determine whether the corresponding probability meets a predetermined probability criterion. The processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to use at least one of the corresponding probability and the preference order of each candidate action to determine whether the robot can select a candidate action from the set of candidate actions to be performed may also cause the robot to determine whether a difference between the first corresponding probability and the second corresponding probability meets a predetermined difference criterion.

[0029] In some embodiments, the robot is a semi-autonomous robot characterized by a level of autonomy, the level of autonomy being determined at least in part by a control model, wherein at least one non-transitory processor-readable storage medium also stores processor-executable instructions and / or data which, when executed by at least one processor, cause the robot to compare the level of autonomy of the robot to a threshold level of autonomy, above which the robot is operable to autonomously select a candidate action from the set of candidate actions to be performed; and processor-executable instructions and / or data which, when executed by at least one processor, cause the robot to transmit a request for instructions to a teleoperation system, and further cause the robot to transmit a request for instructions to the teleoperation system in response to determining that the level of autonomy of the robot is below the threshold level of autonomy.

[0030] In some embodiments, at least one non-transitory processor-readable storage medium also stores processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to perform one of the candidate actions in the set of candidate actions in response to determining that the robot's level of autonomy meets or exceeds a threshold level of autonomy.

[0031] In some embodiments, at least one non-transitory processor-readable storage medium further stores processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to determine a threshold level of autonomy. The processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to determine a threshold level of autonomy may also cause the robot to assign a corresponding probability to each candidate action, the corresponding probability being based at least in part on a corresponding expectation of progress toward a goal that can be achieved by each candidate action. The at least one non-transitory processor-readable storage medium further stores processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to rank each candidate action in an order of preference, the order of preference being based at least in part on the corresponding probability. The processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to determine that the level of autonomy of the robot is below the threshold level of autonomy may also cause the robot to determine that the corresponding probability of each candidate action in the set of candidate actions fails to satisfy at least one criterion.

[0032] In some embodiments, the robot also includes a data repository, wherein the processor executable instructions and / or data that, when executed by at least one processor, cause the robot to collect a corresponding auxiliary data set for each candidate action in the set of candidate actions also cause the robot to collect a first subset of the corresponding auxiliary data set through one or more sensors and / or extract a second subset of the corresponding auxiliary data set from the data repository.

[0033] In some embodiments, the robot further comprises a communication subsystem, wherein the processor executable instructions and / or data that, when executed by at least one processor, cause the robot to transmit a request for instructions to the teleoperation system also cause the robot to transmit a request for instructions through the communication subsystem and receive instructions through the communication subsystem.

[0034] The teleoperation system may be a guided teleoperation system. The teleoperation system may be located at a remote location relative to the robot.

[0035] In some embodiments, the processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to transmit a request for instructions to the teleoperation system further cause the robot to transmit a request for instructions to the teleoperation system over a data network.

[0036] In some embodiments, the processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to transmit a request for instructions to the teleoperation system also cause the robot to enter a standby mode.

[0037] In some embodiments, the processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to execute the instructions further cause the robot to invoke at least one candidate action in the set of candidate actions. The at least one physically actuatable component may include a motion actuator, wherein the processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to invoke at least one candidate action further cause the robot to perform a movement via the motion actuator.

[0038] In some embodiments, the processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to perform the instructions also cause the robot to identify one or more auxiliary actions that can be performed by the robot.

[0039] In some embodiments, the processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to update the control model also cause the robot to increase the robot's autonomy level from a first autonomy level to a second autonomy level, the second autonomy level being higher than the first autonomy level.

[0040] In some embodiments, the processor executable instructions and / or data that, when executed by at least one processor, cause the robot to update a control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and instructions, also cause the robot to update at least one of control instructions, control data, or control parameters.

[0041] In some embodiments, the processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to update a control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and the instructions also cause the robot to update at least one of a configuration setting, a control method, a strategy, a process, a classifier, or a rule.

[0042] A method for teleoperating a robot performed by a teleoperation system communicatively coupled to the robot can be summarized as including: initializing the robot by transmitting a target to the robot, receiving a request for instructions from the robot, the request for instructions including a set of candidate actions that can be performed by the robot and a corresponding auxiliary data set for each candidate action in the set of candidate actions, analyzing each candidate action in the set of candidate actions and each corresponding auxiliary data set to determine a corresponding relative likelihood that each candidate action will result in facilitating the target when performed by the robot, and transmitting instructions to the robot based at least in part on each corresponding relative likelihood to increase the level of autonomy of the robot.

[0043] In some embodiments, receiving a request for instructions from the robot includes receiving a set of candidate actions from the robot that can be performed by the robot to facilitate the goal. In some embodiments, receiving a request for instructions from the robot includes receiving a request for instructions from the robot in response to the robot determining that the robot cannot select a candidate action in the set of candidate actions to be performed. Receiving a request for instructions from the robot may include receiving a corresponding score for each candidate action in the set of candidate actions, the corresponding score including a corresponding probability for each candidate action, the corresponding probability being based at least in part on an expectation of progress toward the goal that can be achieved by each candidate action. Receiving a request for instructions from the robot may also include receiving a preference order based at least in part on the corresponding probability of each candidate action. Analyzing each candidate action in the set of candidate actions and each corresponding auxiliary data set may include analyzing the corresponding score for each candidate action to determine the corresponding relative likelihood that each candidate action will result in facilitating the goal if performed by the robot.

[0044] In some embodiments, the method may further include updating at least one of a machine learning model or a machine learning process of the teleoperation system based at least in part on the respective score of each candidate action.

[0045] In some embodiments, the robot is a semi-autonomous robot characterized by a level of autonomy, the level of autonomy being determined at least in part by the control model. Receiving a request for an instruction from the robot may include receiving the request for an instruction in response to the robot determining that the level of autonomy of the robot is below a threshold level of autonomy. The threshold level of autonomy is a level of autonomy above which the robot is operable to autonomously select a candidate action from the set of candidate actions to perform.

[0046] In some embodiments, analyzing each candidate action in the set of candidate actions and each corresponding auxiliary data set to determine a corresponding relative likelihood that each candidate action, if performed by the robot, will result in facilitating the goal includes analyzing, by a pilot, at least a subset of the set of candidate actions and each of each corresponding auxiliary data set to determine a corresponding relative likelihood that each candidate action, if performed by the robot, will result in facilitating the goal.

[0047] In some embodiments, transmitting instructions to the robot to increase the level of autonomy of the robot based at least in part on each respective relative likelihood includes transmitting instructions to the robot to cause the robot to update a control model. Transmitting instructions to the robot to cause the robot to update the control model may include transmitting instructions to the robot to cause the robot to update at least one of control instructions, control data, or control parameters.

[0048] BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0049] The various elements and actions depicted in the accompanying drawings are provided for illustrative purposes to support the detailed description. Unless the specific context requires otherwise, the size, shape, and relative position of the elements and actions shown are not necessarily shown to scale and are not necessarily intended to convey any information or limitations. In general, the same reference numerals are used to identify similar elements or actions.

[0050] Figure 1 is a block diagram of a system for increasing the level of autonomy of a teleoperated semi-autonomous robot in accordance with the present systems, apparatus, and methods.

[0051] Figure 2 is a block diagram of a teleoperated semi-autonomous robot in accordance with the present systems, apparatus, and methods.

[0052] Figure 3A A semi-autonomous robot (e.g., Figure 2 Flowchart of an exemplary method of operating a remotely operated semi-autonomous robot).

[0053] Figure 3B A semi-autonomous robot (e.g., Figure 2 A flowchart of an exemplary method for determining whether a teleoperated semi-autonomous robot) is able to select a preferred course of action.

[0054] Figure 4 is a method for developing a teleoperated semi-autonomous robot (e.g., Figure 2 Data flow diagram of an exemplary embodiment of an autonomous system of a teleoperated semi-autonomous robot).

[0055] Figure 5 According to the present systems, devices and methods, a robot (e.g., Figure 2 Flowchart of an exemplary method for teleoperation of a remotely operated semi-autonomous robot).

[0056] Detailed Description

[0057] The following description sets forth specific details in order to illustrate and provide an understanding of various embodiments and examples of the present systems, devices, and methods. Those skilled in the art will appreciate that some of the specific details described herein may be omitted or modified in alternative embodiments and examples, and that the various embodiments and examples described herein may be combined with each other and / or with other methods, components, materials, etc., to produce additional embodiments and examples.

[0058] In some instances, well-known structures and / or processes associated with computer systems and data processing have not been shown or provided in detail in order to avoid unnecessarily complicating or obscuring the description of the implementations and examples.

[0059] Unless the specific context requires otherwise, throughout this specification and the appended claims, the term "comprise" and variations thereof (such as "comprises" and "comprising") are used in an open, inclusive sense to mean "including, but not limited to."

[0060] Throughout this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the specific context requires otherwise. For example, references to "an embodiment" and "the embodiment" include "embodiments" and "the embodiments," respectively, and references to "an implementation" and "the implementation" include "implementations" and "the implementations," respectively. Similarly, the term "or" is generally employed in its broadest sense to mean "and / or," unless the specific context clearly dictates otherwise.

[0061] The title and abstract of the present disclosure are provided for convenience only and are not intended to, and should not be construed to, describe the scope or meaning of the present systems, devices, and methods.

[0062] Various embodiments described herein provide systems, devices, and methods for developing robot autonomy. Developing robot autonomy can include introducing autonomy into the operation of the robot. Developing robot autonomy can include increasing the level of autonomy of a semi-autonomous robot.

[0063] In this application, an automated system refers to a system that is operable to perform a specific set of tasks with well-understood parameters that are known in advance. An automated system is typically built to perform one or more specific functions repeatedly and in an efficient manner. Typically, automated functions can be performed with little or no intervention from a human operator or auxiliary system. For example, decisions made by an automated system can be based on predefined heuristics.

[0064] In this application, an autonomous system refers to a system that is capable of analyzing its environment and determining a preferred course of action (sometimes in a changing, potentially uncertain environment). An autonomous system may be at least partially self-managing, intelligent, sentient, and / or self-aware. In this application, a semi-autonomous system refers to a system that may or may not be capable of determining a preferred course of action based on certain events or conditions, under at least some circumstances, and / or in response to at least some events or conditions.

[0065] Robots can be autonomous, semi-autonomous, or fully autonomous. In practice, autonomous or semi-autonomous systems usually have some degree of automation, that is, the ability to perform certain functions without intervention.

[0066] A humanoid robot is a robot that has human attributes. These attributes may be physical attributes such as body shape and / or behavioral attributes such as empathy. For example, the automated functions of a humanoid robot may include the use and control of limbs.

[0067] In this application, the autonomy of a robot refers to its ability to make independent decisions and its ability to learn and adapt to a dynamic environment. The events and conditions encountered by the robot may exceed the range of events and conditions expected when the robot was originally deployed. Events and conditions may constitute decision points for the robot.

[0068] In this application, the level of autonomy of a robot refers to the degree of independence of the robot in the way it moves and behaves in response to events and conditions. The level of autonomy can be expressed on an ordinal scale (e.g., a scale of 1 to 6, or a scale of A to F). In this application, a semi-autonomous robot refers to a robot that is able to act and behave in response to at least some events and conditions without being guided in real time by another system or a human operator. The level of autonomy can at least partially determine the robot's ability to make decisions when it reaches a decision point (when there is uncertainty about how it should proceed). In the present disclosure, a fully autonomous robot refers to a robot that acts and behaves without relying on a user or operator to make decisions and take actions.

[0069] The semi-autonomous robot may be a teleoperated semi-autonomous robot. In the present application, a teleoperated semi-autonomous robot is a semi-autonomous robot that is communicatively coupled to a teleoperation system. The teleoperation system may be operated by one or more guides. For example, the guide may be a human operator. The teleoperation system may be remote from the robot. The guide may be remote from the robot and / or the teleoperation system.

[0070] When a semi-autonomous robot is exposed to various events and conditions, it may be desirable for the semi-autonomous robot to become more autonomous. Increasing the level of autonomy of a robot may include a) enabling the robot to make more decisions on its own and take actions with less guidance from an external entity (e.g., a teleoperation system or operator), b) enabling the robot to make better decisions on its own and take better actions without guidance (e.g., decisions and actions that provide greater progress toward a goal), and / or c) in a teleoperated robotic system, reducing the amount of time and resources that the teleoperation system devotes to the robot. In a teleoperated robotic system, increasing the level of autonomy of a robot may increase the ability of a remote leader to control more robots simultaneously. Increasing the level of autonomy of a robot may ultimately enable the robot to become fully autonomous. Increasing the level of autonomy of a robot may increase its suitability for different tasks and environments, and may extend its service life.

[0071] There are many examples of semi-autonomous robots. One example of a semi-autonomous robot is a robot responsible for loading items from a warehouse onto a truck. In this example, in response to receiving a bill of lading, the robot locates items in the warehouse and loads them onto a truck. In furtherance of its goals, the robot may encounter situations where it is uncertain how to proceed. For example, the robot may be unsure how or where to grasp an item in order to pick it up, in what order the items should be collected, or where it should be placed in the truck. In response to a particular situation, the robot may be able to identify various actions it can take, but not have enough autonomy to proceed without the intervention of a human operator or an auxiliary system such as a teleoperation system.

[0072] The systems, devices, and methods of the present application can increase the level of autonomy of a robot so that its operation requires less intervention from, for example, a teleoperation system. When the robot reaches a decision point and is unsure how to proceed, it can send a request for assistance to the teleoperation system. The request can include a set of candidate actions (also referred to as options in this application) and associated data (e.g., photos of items to be loaded into a truck).

[0073] The teleoperation system and / or the leader can review the candidate actions and associated data, decide on a course of action, and send instructions to the robot on how to proceed. For example, for an item that is too heavy for the robot to load into the truck, the teleoperation system can instruct the robot to remove the item from the bill of lading and move on to the next item. In addition to executing the course of action provided by the teleoperation system, the robot can also learn from experience and be able or more likely to autonomously decide on an appropriate course of action in similar situations in the future.

[0074] Figure 1 1 is a block diagram of an exemplary embodiment of a system 100 for increasing the level of autonomy of a teleoperated semi-autonomous robot in accordance with the present systems, apparatuses, and methods. The system 100 includes three semi-autonomous robots 102a, 102b, and 102c (collectively referred to as semi-autonomous robots 102), although those skilled in the art will appreciate that in alternative embodiments, the system 100 may include less than three or more than three semi-autonomous robots. The semi-autonomous robots 102 may be members of a robot fleet.

[0075] exist Figure 1In an exemplary embodiment, the semi-autonomous robot 102a is communicatively coupled to the semi-autonomous robot 102b. The semi-autonomous robots 102a and 102b may have peer-to-peer communication via at least one of a wired and / or wireless connection and / or via a network. The semi-autonomous robots 102 may operate independently of each other or in collaboration with each other. The peer-to-peer communication between the semi-autonomous robots 102 may facilitate collaboration between the semi-autonomous robots 102. Respective ones of the semi-autonomous robots 102 may have different respective levels of autonomy.

[0076] The system 100 includes a teleoperation system 104 communicatively coupled to a semi-autonomous robot 102. The teleoperation system 104 may be communicatively coupled to various ones of the semi-autonomous robots 102 via at least one of a wired and / or wireless connection. The semi-autonomous robot 102 may be tethered, i.e., attached to the teleoperation system 104 via a tether. The teleoperation system 104 may be communicatively coupled to various ones of the semi-autonomous robots 102 via a network.

[0077] Various semi-autonomous robots in the semi-autonomous robots 102 may be located at a location remote from the teleoperation system 104. For example, the teleoperation system 104 may be located at an operation center, and various semi-autonomous robots in the semi-autonomous robots 102 may be deployed at a location remote from the operation center. Various semi-autonomous robots in the semi-autonomous robots 102 may be deployed in dangerous environments (e.g., environments that are not suitable for teleoperation systems or human operators).

[0078] The teleoperation system 104 may be operated by two guides 106a and 106b (collectively referred to as guides 106). Those skilled in the art will appreciate that in alternative embodiments, the teleoperation system 104 may be operated by fewer than two or more than two guides. Figure 1 In the exemplary embodiment of , the guide 106a is communicatively coupled to the guide 106b. The guide 106 may be a human operator.

[0079] In some embodiments, the teleoperation system 104 is located at a single location. In other embodiments, the teleoperation system 104 is a distributed system. For example, the guide 106a and / or the guide 106b can be positioned away from each other and / or positioned away from other components of the teleoperation system 104. The distributed components of the teleoperation system 104 can be connected by wire and / or wirelessly and in some embodiments, they are communicatively coupled to each other via a network. The guide 106 is communicatively coupled to the guide interface 108 of the teleoperation system 104. The guide interface 108 may include a display interface and / or an audio speaker. The guide interface 108 may include an interface for data input, such as a keyboard, a mouse, a handheld controller, a microphone and / or various sensors (e.g., accelerometers, inertial measurement units, etc.) worn or otherwise actuatable by the guide 106. The guide interface 108 may include an application programming interface (API).

[0080] The teleoperation system 104 includes a control subsystem 110. The control subsystem 110 may include one or more processors (e.g., one or more microprocessors, central processing units (CPUs), graphics processing units (GPUs), application specific integrated circuits (ASICs), programmable gate arrays (PGAs), or programmable logic controllers (PLCs)) that execute logic (e.g., in the form of instructions stored as software or firmware instructions in one or more non-transitory computer or processor readable media such as memory (e.g., non-volatile memory, volatile memory, read-only memory, random access memory, flash memory, rotating magnetic or optical media)).

[0081] The teleoperation system 104 includes a communication (comms) subsystem 112. The communication subsystem 112 may include a receiver for receiving a request for instructions, such as from the semi-autonomous robot 102, and a transmitter for sending instructions to the semi-autonomous robot 102, such as.

[0082] The teleoperation system 104 also includes a data repository 114. For example, the data repository 114 may include a database and / or one or more files. The data repository 114 may include a memory and / or a storage medium. In some embodiments, the data repository 114 is co-located with the leader interface 108, the control subsystem 110, and the communication subsystem 112. In other embodiments, at least a portion of the data repository 114 is located remotely from the leader interface 108, the control subsystem 110, and / or the communication subsystem 112.

[0083] The data repository 114 includes tele-operation (tele-op) data. Tele-op data may include control instructions, control data, control parameters, configuration settings, control methods, policies, procedures, classifiers, and / or rules associated with the operation of the semi-autonomous robot 102. Tele-op data may include current data, such as the current position, current goal, ongoing actions, and / or current autonomy level of the semi-autonomous robot 102. Tele-op data may include historical data that describes, for example, previous requests received by the teleoperation system 104 from the semi-autonomous robot 102, instructions provided to the semi-autonomous robot 102 by the teleoperation system 104, and previous actions taken by the semi-autonomous robot 102 to facilitate their respective goals.

[0084] In some embodiments, the teleoperation system 104 is fully autonomous and operates without intervention from the leader 106. In embodiments where the teleoperation system 104 is capable of operating without intervention from the leader 106, the leader 106 may still be able to monitor the semi-autonomous robot 102 via the teleoperation system 104 and control the semi-autonomous robot 102 if needed or desired.

[0085] The semi-autonomous robot 102 may be remotely controlled by the teleoperation system 104 (e.g., by a leader 106 of the teleoperation system 104). As previously described, the semi-autonomous robot 102 may have different levels of autonomy. For example, the semi-autonomous robot 102a may have a higher level of autonomy than the semi-autonomous robot 102b. The operation of the semi-autonomous robot 102a may require less intervention by the teleoperation system 104 than the operation of the semi-autonomous robot 102b.

[0086] Operation of each of the semi-autonomous robots 102 may include contributing to a respective goal. In some embodiments, for example, the respective goal is set by or through the teleoperation system 104. In other embodiments, the goal of at least one of the semi-autonomous robots is set by the semi-autonomous robot itself, i.e., the robot is sufficiently autonomous to set its own goal. In still other embodiments, the robot's goal is set by another semi-autonomous robot (either directly via peer-to-peer communication (see, e.g., semi-autonomous robots 102a and 102b) or indirectly via the teleoperation system 104).

[0087] Figure 22 is a block diagram of an exemplary embodiment of a teleoperated semi-autonomous robot 200 according to the present systems, devices, and methods. The semi-autonomous robot 200 includes a controller 202, a processor 204, a power subsystem 206, a sensor 208, a motion actuator 210, and a communication (comms) subsystem 212 housed in a body 214. The body 214 can have a shape, form factor, and / or number of parts suitable for the construction and / or operation of the semi-autonomous robot 200. In some embodiments, the semi-autonomous robot 200 is a humanoid robot, and, for example, the body 214 can be constructed to resemble a human body. The body 214 can have the appearance of a human body.

[0088] The controller 202 may control the actions of the semi-autonomous robot 200. The controller 202 may execute instructions provided by the teleoperation system to cause the robot 200 to take actions.

[0089] The controller 202 may identify candidate actions of or for the semi-autonomous robot 200. Candidate actions are actions that are deemed suitable actions by the controller 202, and may include actions that may, for example, at least partially achieve a goal. The controller 202 may identify candidate actions when executing instructions and / or when taking actions to at least partially achieve a goal. The controller 202 may initiate identifying candidate actions. The controller 202 may identify candidate actions in response to events and / or conditions. The events and / or conditions may be internal events and / or conditions (e.g., low battery charge in the power subsystem 206). The events and / or conditions may be external events and / or conditions, i.e., events and / or conditions external to the semi-autonomous robot 200 (e.g., the proximity of another robot, or an ambient temperature exceeding a predetermined temperature threshold, or the presence or absence of a particular object in the robot's external environment).

[0090] After identifying the candidate actions, the controller 202 can determine whether the semi-autonomous robot 200 can decide on a preferred course of action. A course of action can include one or more candidate actions. If the semi-autonomous robot 200 can decide on a preferred course of action, it can proceed to perform one or more candidate actions. If the semi-autonomous robot 200 cannot decide on a preferred course of action, it can transmit a request for instructions from a teleoperation system (e.g., teleoperation system 104).

[0091] In some embodiments, the semi-autonomous robot 200 evaluates whether it has sufficient confidence, based at least in part on the analysis of the candidate actions, to make a decision about how to proceed. If it does not have sufficient confidence in the decision, it may request assistance from the teleoperation system. The assistance may be in the form of instructions from the teleoperation system to the semi-autonomous robot 200.

[0092] In some embodiments, the evaluation of whether the semi-autonomous robot 200 has sufficient confidence is based at least in part on the corresponding probability of each candidate action and / or the ranking of the candidate actions (also referred to as a preference order in this application). Such probabilities and / or rankings (e.g., such preference orders) can be determined by the semi-autonomous robot 200.

[0093] In some embodiments, the semi-autonomous robot 200 is self-aware of its autonomy level. For example, the semi-autonomous robot 200 may store a control model that defines or includes the autonomy level. In some embodiments, the semi-autonomous robot 200 estimates the current autonomy level based on the control model. In some embodiments, the control model includes at least one or more of the following items: control instructions, control data, control parameters, configuration settings, control methods, strategies, procedures, classifiers, and / or rules.

[0094] Processor 204 may include one or more processors (e.g., one or more microprocessors, central processing units (CPUs), graphics processing units (GPUs), application specific integrated circuits (ASICs), programmable gate arrays (PGAs), or programmable logic controllers (PLCs)) that execute logic (e.g., in the form of instructions stored as software or firmware instructions in one or more non-transitory computer or processor-readable media such as memory (e.g., non-volatile memory, volatile memory, read-only memory, random access memory, flash memory, rotating magnetic or optical media)).

[0095] The power subsystem 206 may include one or more power sources, such as a mains power source, a battery, a fuel cell, a photovoltaic cell panel, a generator, and / or an internal combustion engine. The power subsystem 206 may include a hydraulic system.

[0096] Sensors 208 may include one or more of the following: light sensors (e.g., CCD [charge-coupled device] cameras), sound sensors (e.g., microphones), proximity sensors (e.g., ultrasonic sensors and / or LIDAR systems), tactile sensors (e.g., touch sensors and / or tactile sensors), environmental sensors (e.g., temperature sensors, wind speed sensors, magnetometers, and / or aerosol sensors), navigation and positioning sensors (e.g., GPS [global positioning system]), accelerometers, and diagnostic sensors (e.g., battery charge sensors).

[0097] In operation, the semi-autonomous robot 200 may collect data related to one or more previously identified candidate actions. The data may include sensor data collected by the sensor 208. The data may include, but is not limited to, images, video, audio, wind speed, local magnetic field, ambient temperature, battery charge, and / or aerosol concentration.

[0098] The semi-autonomous robot 200 may include at least one physically actuatable component (e.g., a mobile actuator 210) mechanically coupled to a body 214. The mobile actuator 210 may include, for example, linear and / or rotary actuators, and the actuator may be, for example, hydraulic, pneumatic, and / or electric. DC (direct current) electric actuators may include, for example, DC motors, DC gear motors, stepper motors, and / or linear electric actuators. Depending on the specific design and functionality of the robot 200, the mobile actuator 201 may drive a range of different types of physically actuatable components, including, but not limited to: wheels, legs, arms, end effectors, grippers, hands, and / or heads.

[0099] The communication subsystem 212 may include a transmitter and a receiver, may use technologies such as Wi-Fi local area networks and / or cellular communication technologies (e.g., 4G, LTE, 5G, etc.), and may support one or more protocols (e.g., Ethernet or Bluetooth).

[0100] The semi-autonomous robot 200 also includes a data repository 216. The data repository 216 may store control models.

[0101] Figure 3A is a diagram showing a teleoperated semi-autonomous robot (e.g., Figure 1 Flow chart of an exemplary method 300 of operating a semi-autonomous robot 102a). Figure 3A The method 300 includes fourteen actions 302, 304, 306, 308, 310, 312, 314, 316, 318, 320, 322, 324, 326, 328, and 330. Those skilled in the art will appreciate that in alternative embodiments, the steps 302, 304, 306, 308, 310, 312, 314, 316, 318, 320, 322, 324, 326, 328, and 330 may be omitted. Figure 3A Certain actions of the embodiment and / or additional actions may be added. Those skilled in the art will also understand that the order of actions shown is shown for exemplary purposes only and may be changed in alternative embodiments.

[0102] At 302, in response to a start condition, method 300 starts. The start condition may include starting a remote operation system (e.g., Figure 1 The start condition may include the teleoperation system detecting that the semi-autonomous robot has exited the standby mode, or an instruction from the teleoperation system to cause the robot to exit the standby mode. The start condition may be observed, influenced, or triggered autonomously by the robot. At 304, the semi-autonomous robot receives a target, such as from the teleoperation system.

[0103] When an event or condition is met or triggered at 306, the semi-autonomous robot identifies a set of candidate actions that facilitate the goal based on the event or condition at 308. Observing the event or condition at 306 is optional. In some cases, the robot actively identifies a set of candidate actions. The set of candidate actions identified at 308 includes one or more candidate actions. One of the candidate actions may be to do nothing and wait for another event or condition to occur. If at 310, the semi-autonomous robot determines that it can decide on a preferred course of action, the method 300 proceeds to 312. The preferred course of action may include executing a selected candidate action from the set of candidate actions. Figure 3B An example of a method is described by which a semi-autonomous robot may determine at 310 whether it is able to decide on a preferred course of action.

[0104] At 312, the semi-autonomous robot performs the selected action. If the robot determines at 314 that the goal has been met, the method 300 proceeds to 316, which ends at 316. If the robot determines at 314 that the goal has not been met, the method 300 returns to 306, where the robot waits for, or looks for, or pursues another event or condition.

[0105] If at 310 the robot determines that it does not have sufficient autonomy to decide on a preferred course of action, the method 300 proceeds to 318 where it collects corresponding data associated with each candidate action identified in 308 (e.g., using the sensor 208). At 320, the robot transmits a request for instructions to the teleoperation system and enters a standby mode at 322. For example, in standby mode, the robot may suspend operation and may be powered off. The request for instructions transmitted by the robot at 320 may include the candidate actions identified by the robot at 308 and the corresponding data collected by the robot at 318 regarding the candidate actions.

[0106] At 324, the robot waits to receive instructions from the teleoperation system. At 326, the robot exits the standby mode when instructions have been received from the teleoperation system. At 328, the robot executes the received instructions. At 330, the robot updates the control model in the robot's data repository (e.g., 216), and the method 300 proceeds to 314. Updating the control model in the data repository may include updating control instructions, control data, control parameters, configuration settings, control methods, policies, procedures, classifiers, and / or rules that at least partially govern the robot's ability to operate autonomously. Updating the control model in the data repository may include updating the robot's autonomy level to a higher autonomy level.

[0107] If the robot determines at 314 that the goal has been met, the method 300 proceeds to 316 where it ends. If the robot determines at 314 that the goal has not been met, the method 300 returns to 306 where the robot waits for, or looks for, or pursues another event or condition.

[0108] Figure 3B Is from Figure 3A An expanded view of an exemplary implementation of act 310 of method 300 in FIG. Figure 3A Action 310 in Figure 3B , and itself includes five actions 332, 334, 336, 338 and 340. Those skilled in the art will appreciate that in alternative embodiments, the Figure 3B Certain actions of the embodiment and / or additional actions may be added. Those skilled in the art will also understand that the order of actions shown is shown for exemplary purposes only and may be changed in alternative embodiments.

[0109] exist Figure 3A After identifying the candidate action at 308, method 300 proceeds to Figure 3B 332. Until the semi-autonomous robot exhausts the list of candidate actions at 332, the semi-autonomous robot assigns a corresponding probability to each candidate action at 334.

[0110] At 336, the robot ranks each candidate action based at least in part on the respective probabilities assigned at 334. At 338, the robot determines a criterion based at least in part on the probabilities assigned at 334 and / or the rankings generated at 336. In some implementations, the criterion includes a threshold level of autonomy. The threshold level of autonomy is a level of autonomy above which the robot is operable to autonomously select a candidate action from the set of candidate actions to perform.

[0111] The respective probabilities assigned to each action at 334 may be based at least in part on the respective probabilities and / or the expected degree of progress that each action will result in progress toward the robot's goal. Determining a threshold level of autonomy may include, for example, identifying that one of the respective probabilities exceeds a predetermined probability threshold, or that a difference between the respective probabilities of the most likely action and the second most likely action exceeds a predetermined difference threshold. Other suitable criteria and combinations of criteria may be used to determine the threshold level of autonomy. These criteria may depend at least in part on the respective probabilities and ranking of the candidate actions.

[0112] At 340, if the criteria for decision making by the robot are not met (e.g., if the level of autonomy of the robot is below a threshold level of autonomy), the method 300 proceeds to Figure 3AIf at least one of the criteria is met (e.g., if the level of autonomy of the robot meets or exceeds a threshold level of autonomy), the method 300 proceeds to Figure 3A of 312.

[0113] In some embodiments, the semi-autonomous robot includes one or more processors (e.g., Figure 2 of the semi-autonomous robot 200), and Figure 3A and Figure 3B The control of the various actions 302 to 340 is performed by a processor. In some embodiments, the semi-autonomous robot includes a communication subsystem (e.g., Figure 2 's communication subsystem 212), and actions 320 (transmitting a request for an instruction) and 324 (receiving an instruction) are performed by the communication subsystem.

[0114] Figure 4 is a method for developing a teleoperated semi-autonomous robot (e.g., Figure 1 A data flow diagram (DFD) 400 of an exemplary embodiment of an autonomous system of one of the semi-autonomous robots 102).

[0115] DFD 400 includes a semi-autonomous robot process 402, a teleoperation system 404, and a control model 406. Semi-autonomous robot process 402 is a process that can transform input data into output data for a semi-autonomous robot (e.g., Figure 2 The teleoperation system 404 may be controlled and / or operated by at least one processor (e.g., 204) by accessing and / or executing a control model 406 stored in a non-transitory processor-readable storage medium or data repository (e.g., 216) communicatively coupled to the at least one processor. From the perspective of the semi-autonomous robot process 402, the teleoperation system 404 is an external entity and is an external data source / data sink. The control model 406 is a data repository, such as a database or a file.

[0116] DFD 400 includes four data flows: current control model 408 , request for instructions 410 , instructions 412 , and updated control model 414 .

[0117] The current control model 408 may include, for example, the current level of autonomy, a threshold level of autonomy, actions and corresponding data for each action, as well as other models, classifiers, rules, algorithms, strategies, etc. that govern the (semi-)autonomous control of the robot.

[0118] The request 410 for instructions may include a list of candidate actions and corresponding data for each candidate action. In some implementations, the request 410 for instructions includes a complete list of candidate actions identified by the robot. In other implementations, the request 410 for instructions includes a partial list of candidate actions. The robot may determine the partial list by filtering the complete list of candidate actions and including in the partial list candidate actions that meet one or more criteria (e.g., only the top X candidate actions according to a ranking scheme employed by the robot, where X may be any number greater than 1, such as 2, 3, 4, 5, etc.).

[0119] Instructions 412 may include, for example, a proposed course of action. The proposed course of action may be or include a selection of one or more candidate actions identified by the robot. Additionally, or alternatively, instructions 412 may include a request for the robot to identify additional candidate actions (also referred to herein as auxiliary actions) and / or a request for the robot to collect additional data about one or more candidate actions. Instructions 412 may include actions that were not previously included in the candidate actions. For example, instructions 412 may include instructions to return to a base position or enter a sleep mode.

[0120] The updated control model 414 may include, for example, updating any or all of the following: a classifier (e.g., by adjusting at least one weight in the classifier), a model, a policy, an algorithm, a rule, and / or a parameter (e.g., a current level of autonomy). The updated autonomy level may be a higher level of autonomy. The autonomy levels may be represented on an ordinal scale (e.g., 1 to 6, or A to F). The robot may advance to a new autonomy level upon receiving instructions from the teleoperation system. The autonomy level may be evaluated and updated based on the robot's performance (e.g., successful execution of instructions, measurement of a benchmark, or achievement of a goal).

[0121] Updating the control model 414 can include a) identifying the instruction 412 as a preferred course of action in a given scenario (i.e., in response to an event and / or condition that triggered a request for the instruction and given data collected by the robot related to the selected candidate action), and / or b) assigning weights to the actions in the instruction for use in future evaluation of candidate actions in response to the event or condition.

[0122] As a more detailed example, for illustrative purposes only, the robot may send a request for instructions 410 as an implementation of act 320 of method 300. The request for instructions 410 may include: i) a set of candidate actions identified by the robot according to act 308 of method 300, and ii) corresponding data for each candidate action collected by the robot according to act 318 of method 300. The teleoperation system 404 may be configured to perform the following operations: Figure 5The method 500 shown (described later) receives a request for instructions 410 and generates instructions 412. The instructions 412 may include, among other things, a selection of a particular one of the candidate actions provided by the robot in the request for instructions 410. The robot may receive the instructions 412 according to action 324 of the method 300 and execute the instructions 412 according to action 328 of the method 300. For example, at 328 of the method 300, the robot may execute or implement the particular candidate action selected in the instructions 412.

[0123] In the foregoing example, the robot's control model 406 causes the robot to send a request 410 for an instruction as an implementation of act 320 of method 300 because the robot's control model 406 is unable (e.g., the robot does not have sufficient autonomy) to determine which candidate action (identified at 308) should be performed based on act 310 of method 300. That is, the exemplary robot's control model 406 is constructed such that, given a particular combination of work goals, candidate actions, and sensor readings encountered by the robot, the control model 406 (and therefore the robot) is unable to autonomously determine what action should be taken in order to achieve the goal or make the best progress toward achieving the goal. This uncertainty may be due in part to the robot (or control model 406) not having previously encountered this particular combination of work goals, candidate actions, and sensor readings. According to the present systems, devices, and methods, when the robot receives an instruction 412, the robot (or more specifically, the robot's control model 406) learns from the instruction 412 and updates (414) the control model 406 according to act 330 of method 300. The specific details of how the control model 406 is updated (e.g., what instructions, variables, parameters, algorithms, etc. are changed and in what manner) depend on the specific implementation of the control model 406 and may vary widely in different embodiments of the robot and / or robot system; however, in general, the updated control model 414 can replace or supplement the control model 406 and include instructions, variables, parameters, algorithms, etc. that are adapted (relative to the original instructions, variables, parameters, algorithms, etc. of the control model 406) to enable the robot to assign a higher ranking (e.g., probability) to the candidate action selected in the instructions 412 in subsequent situations (in which the robot encounters the same or similar combination of work objectives, candidate actions, and sensor readings), and, for example, to enable the robot to autonomously decide to perform such a candidate action at 310 of method 300 without consulting the teleoperation system 404.

[0124] Those skilled in the art will appreciate that in alternative embodiments, certain data streams of DFD 400 may be omitted and / or additional data streams may be added. Figure 4The triggering and timing of the data flows described by DFD 400 are for exemplary purposes only and may vary in alternative implementations. For example, teleoperation system 404 may provide instructions 412 at a time other than in response to request 410 for instructions from semi-autonomous robotic process 402.

[0125] In some embodiments, the teleoperation system 404 (or the leader of the teleoperation system) can learn from the requests for instructions 410 (i.e., can improve its own level of autonomy, ability to make decisions, and / or level of knowledge and understanding). For example, the requests for instructions 410 can include heretofore unexpected candidate actions and / or rankings of candidate actions identified by the robot. Similarly, in some embodiments, when the robot is in communication with the teleoperation system 404, other robots can learn from the requests for instructions 410 transmitted and / or the instructions 412 they participate in. That is, the control model of the second robot can be updated based on the updates made to the control model of the first robot.

[0126] In some embodiments, the teleoperation system 404 may send instructions to the second robot that may cause the second robot to update the control model and may cause the second robot to increase its level of autonomy. In some embodiments, the request for instructions 410 and the instructions 412 may be broadcast and received by one or more other robots. The other robots may increase their level of autonomy based on the broadcasted request and / or the broadcasted instructions.

[0127] In some embodiments, the request for instructions may be transmitted by the first robot to the second robot (e.g., via peer-to-peer communication). The second robot may have a higher level of autonomy than the first robot. The second robot may send instructions to the first robot.

[0128] Figure 5 According to the present systems, devices and methods, a robot (e.g., Figure 2 Flowchart of an exemplary method 500 for teleoperating a teleoperated semi-autonomous robot 200). Figure 5 The method 500 includes seven actions 502, 504, 506, 508, 510, 512 and 514. Those skilled in the art will appreciate that in alternative embodiments, the steps 502, 504, 506, 508, 510, 512 and 514 may be omitted. Figure 5 Certain actions of the embodiment and / or additional actions may be added. Those skilled in the art will also understand that the order of actions shown is shown for exemplary purposes only and may be changed in alternative embodiments.

[0129] At 502, in response to a start condition, method 500 starts. For example, the start condition may include a teleoperation system (e.g., Figure 1The system may be activated by a remote operation system 104), a command from another system, or a command from a leader of the remote operation system via a leader interface, or a signal from the robot.

[0130] At 504, the teleoperation system initializes the robot. In some embodiments, the teleoperation system initializes the robot by launching a target at the robot. At 506, the teleoperation system waits until it receives a request for instructions from the robot. In some embodiments, the teleoperation system can teleoperate multiple robots simultaneously. The request for instructions can be received asynchronously from multiple different robots. The request for instructions received from the robot initialized at 504 can include a set of candidate actions that can be performed by the robot, and a corresponding auxiliary data set for each candidate action in the set of candidate actions.

[0131] When the teleoperation system receives a request for instructions from the robot, the method proceeds to 508 where the teleoperation system analyzes the candidate actions and corresponding auxiliary data from the request. At 510, the teleoperation system determines if there is another candidate action to analyze. If so, the method 500 returns to 508. Otherwise, the method 500 proceeds to 512.

[0132] The analysis request may include analyzing at least a subset of the set of candidate actions and at least a subset of data of each corresponding auxiliary data set. The auxiliary data may include, for example, sensor data. The auxiliary data may include a score determined by the robot. The score may indicate the expected progress that the candidate action can achieve in facilitating the goal when performed by the robot. The score may include one or more probabilities, probability differences, and / or preference orders (also referred to as rankings in this application). The scores received by the teleoperation system from the robot may assist in the analysis of the teleoperation system. The scores may also be used to train a machine learning model or process of the teleoperation system, or otherwise provide input to a machine learning model or process of the teleoperation system. The analysis request may include determining the corresponding relative likelihood that each candidate action will result in facilitating the goal when performed by the robot.

[0133] At 508, analyzing the request may include a request by a facilitator (e.g. Figure 1 The instructions for transmitting to the robot can be analyzed by the leader via the leader interface (e.g. Figure 1 The instructions for transmission to the robot can be assembled without the intervention of the leader.

[0134] At 512, the teleoperation system transmits instructions to the robot. The instructions to the robot may be based at least in part on each respective relative likelihood and may result in an increase in the level of autonomy of the robot. The instructions may include instructions to cause the robot to: a) perform one or more candidate actions, b) exit standby mode, c) identify an auxiliary candidate action, and / or d) pause, sleep, or shut down. At 514, the method 500 ends.

[0135] Throughout this specification and the appended claims, the term "communicative" (as in "communicative coupling" and in variations such as "communicatively coupled") is generally used to refer to any engineering arrangement for transmitting and / or exchanging information. For example, communicative coupling can be achieved in the form of a variety of different media and / or communication paths, including but not limited to: conductive paths (e.g., conductive wires, conductive traces), magnetic paths (e.g., magnetic media), wireless signal transmission (e.g., radio frequency antennas) and / or optical paths (e.g., optical fibers). Exemplary communicative couplings include, but are not limited to: electrical coupling, magnetic coupling, radio frequency coupling and / or optical coupling.

[0136] Throughout this specification and the appended claims, infinitive verb forms are frequently used. Examples include, but are not limited to: "to encode," "to provide," "to store," and the like. Unless the specific context requires otherwise, such infinitive verb forms are used in an open, inclusive sense, i.e., as "to, at least, encode," "to, at least, provide," "to, at least, store," and the like.

[0137] This specification (including the drawings and summary) is not intended to be an exhaustive or limiting description of all embodiments and examples of the present systems, devices and methods. Those skilled in the art will appreciate that the various descriptions and drawings provided may be modified without departing from the spirit and scope of the present disclosure. In particular, the teachings herein are not intended to be limited by or to the illustrative examples of the computer systems and computing environments provided.

[0138] This specification provides various implementations and embodiments in the form of block diagrams, schematic diagrams, flow charts and examples. It will be understood by those skilled in the art that any function and / or operation in such block diagrams, schematic diagrams, flow charts or examples can be implemented individually and / or collectively by a wide range of hardware, software and / or firmware. For example, the various embodiments disclosed in whole or in part herein can be equivalently implemented in one or more of the following items: a dedicated integrated circuit (i.e., ASIC); a standard integrated circuit; a computer program executed by any number of computers (e.g., a program running on any number of computer systems); a program executed by any number of controllers (e.g., microcontrollers); and / or a program executed by any number of processors (e.g., microprocessors, central processing units, graphics processing units), and firmware and any combination of the foregoing.

[0139] Throughout this specification and the appended claims, a "memory" or "storage medium" is a processor-readable medium, which is an electronic, magnetic, optical, electromagnetic, infrared, semiconductor or other physical device or means that contains or stores processor data, data objects, logic, processor instructions and / or programs. When data, data objects, logic, processor instructions and / or programs are implemented as software and stored in a memory or storage medium, they can be stored in any suitable processor-readable medium for use by any suitable processor-related instruction execution system, device or device, such as a computer-based system, a system containing a processor, or other system that can retrieve data, data objects, logic, processor instructions and / or programs from a memory or storage medium and perform various actions or operations (i.e., processing steps) thereon and / or in response thereto. Thus, a "non-transitory processor-readable storage medium" can be any element that stores data, data objects, logic, processor instructions and / or programs for use by or in conjunction with an instruction execution system, device and / or device. As specific non-limiting examples, the processor readable medium may be: a portable computer floppy disk (magnetic, compact flash card, secure digital, etc.), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, EEPROM or flash memory), a portable compact disk read-only memory (CDROM), digital tape, and / or any other non-transitory medium.

[0140] The claims of the present disclosure are attached. The present disclosure is intended to support, enable and illustrate the claims, but is not intended to limit the scope of the claims to any specific implementation or example. In general, the claims should be interpreted to include all possible implementations and examples together with the full range of equivalents to which these claims are entitled.

Claims

1. A method for operating a robot, the method being performed by the robot, the method comprising: receiving a goal to be accomplished by the robot; identifying a set of candidate actions that can be performed by the robot to further the goal; For each candidate action in the set of candidate actions, collecting a corresponding auxiliary data set; transmitting a request for instructions to a teleoperation system communicatively coupled to the robot in response to determining that the robot cannot select a candidate action from the set of candidate actions to perform, the request for instructions comprising each candidate action from the set of candidate actions and each corresponding auxiliary data set; receiving the instruction from the teleoperation system; executing the instructions; and updating a control model of the robot based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and the instructions to increase a level of autonomy of the robot, The method further includes scoring each candidate action in the set of candidate actions, wherein scoring each candidate action in the set of candidate actions includes assigning a corresponding probability to each candidate action, and the corresponding probability is based at least in part on the corresponding progress expectation towards the goal that can be achieved by each candidate action.

2. The method according to claim 1, wherein: The identifying a set of candidate actions that can be performed by the robot includes identifying a set of candidate actions in response to detecting at least one of an event or a condition.

3. The method according to claim 2, wherein: Identifying a set of candidate actions in response to detecting at least one of an event or a condition includes identifying a set of candidate actions in response to detecting at least one of an external event or an external condition.

4. The method of claim 3, wherein the robot comprises one or more sensors, wherein: Identifying a set of candidate actions in response to detecting at least one of an external event or an external condition includes sensing the at least one of the external event or the external condition by the one or more sensors.

5. The method according to claim 1, wherein: The receiving of the goal to be completed by the robot includes: receiving the goal from the teleoperation system.

6. The method according to claim 1, wherein: Transmitting the request for instructions to the teleoperation system in response to determining that the robot cannot select a candidate action from the set of candidate actions to be performed includes: transmitting the request for instructions to the teleoperation system in response to determining that the robot cannot select a candidate action from the set of candidate actions to be performed to facilitate the goal.

7. The method according to claim 6, wherein: The robot is a member of a robot swarm, and the method also includes updating the control model of one or more other robots in the robot swarm at least in part based on the update made to the control model of the robot to increase the autonomy level of the one or more other robots.

8. The method of claim 1, further comprising ranking each candidate action in an order of preference based at least in part on the corresponding probability of each candidate action.

9. The method of claim 8, further comprising using at least one of the corresponding probability and the preference order of each candidate action to determine whether the robot can select a candidate action from the set of candidate actions to perform.

10. The method according to claim 9, wherein: Transmitting a request for an instruction to the teleoperation system, wherein the request for the instruction includes each candidate action in the set of candidate actions and each corresponding auxiliary data set includes: transmitting a request for an instruction to the teleoperation system, wherein the request for the instruction includes each corresponding probability and the preference order.

11. The method according to claim 9, wherein: The using at least one of the corresponding probability and the preference order of each candidate action to determine whether the robot can select a candidate action from the set of candidate actions to be performed includes: determining whether the corresponding probability meets a predetermined probability criterion.

12. The method according to claim 11, wherein: The determining whether the corresponding probability meets a predetermined probability standard includes: determining whether the corresponding probability exceeds a predetermined probability threshold.

13. The method according to claim 9, wherein: The use of at least one of the corresponding probability and the preference order of each candidate action to determine whether the robot can select a candidate action from the set of candidate actions to be performed includes: determining whether the difference between the first corresponding probability and the second corresponding probability meets a predetermined difference standard.

14. A method for operating a robot, the method being performed by the robot, the method comprising: receiving a goal to be accomplished by the robot; identifying a set of candidate actions that can be performed by the robot to further the goal; For each candidate action in the set of candidate actions, collecting a corresponding auxiliary data set; transmitting a request for instructions to a teleoperation system communicatively coupled to the robot in response to determining that the robot cannot select a candidate action from the set of candidate actions to perform, the request for instructions including each candidate action from the set of candidate actions and each corresponding auxiliary data set; receiving the instruction from the teleoperation system; executing the instructions; and updating a control model of the robot based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and the instructions to increase a level of autonomy of the robot, wherein the robot is a semi-autonomous robot characterized by a level of autonomy, the level of autonomy being determined at least in part by the control model, the method further comprising comparing the level of autonomy of the robot to a threshold level of autonomy, above which the robot is operable to autonomously select a candidate action from the set of candidate actions to be performed, wherein transmitting the request for the instruction to the teleoperation system comprises: transmitting the request for the instruction to the teleoperation system in response to determining that the level of autonomy of the robot is below the threshold level of autonomy; Wherein, the method further comprises determining the threshold level of autonomy, wherein determining the threshold level of autonomy comprises assigning a corresponding probability to each candidate action, the corresponding probability being based at least in part on a corresponding expectation of progress towards the goal that can be achieved by each candidate action.

15. The method of claim 14, further comprising, in response to determining that the level of autonomy of the robot meets or exceeds the threshold level of autonomy, performing one of the set of candidate actions. The method of claim 14 , further comprising determining the threshold level of autonomy.

17. The method of claim 14, further comprising receiving a goal to be accomplished by the robot, wherein: The identifying a set of candidate actions that can be performed by the robot includes identifying a set of candidate actions that can be performed by the robot to facilitate the goal.

18. The method according to claim 17, wherein: The receiving of a goal to be accomplished by the robot includes receiving the goal from the teleoperation system.

19. The method according to claim 14, wherein: The robot is a member of a robot swarm, and the method also includes updating the control model of one or more other robots in the robot swarm at least in part based on the update made to the control model of the robot to increase the autonomy level of the one or more other robots.

20. The method of claim 14, further comprising ranking each candidate action in an order of preference based at least in part on the corresponding probability.

21. The method according to claim 14, wherein: The determining that the level of autonomy of the robot is below the threshold level of autonomy comprises determining that the corresponding probability of each candidate action in the set of candidate actions fails to satisfy at least one predetermined criterion.

22. The method according to claim 1 or 14, wherein the robot comprises one or more sensors and a data repository, and for each candidate action in the set of candidate actions, the collecting of the corresponding auxiliary data set comprises at least one of the following items: collecting a first subset of the corresponding auxiliary data set by the one or more sensors or extracting a second subset of the corresponding auxiliary data set from the data repository.

23. The method according to claim 22, wherein: Collecting the first subset of the corresponding auxiliary data set by the one or more sensors includes collecting at least one of audio, image, video or environmental data.

24. The method of claim 1 or 14, wherein the robot comprises a communication subsystem, wherein: Transmitting a request for a command to a teleoperation system and receiving the command from the teleoperation system include: transmitting a request for the command by the communication subsystem and receiving the command by the communication subsystem.

25. The method of claim 1 or 14, wherein the robot comprises a controller, wherein: The executing the instruction and updating the control model includes: executing the instruction by the controller and updating the control model by the controller.

26. The method according to claim 1 or 14, wherein: Transmitting a request for an instruction to the teleoperation system includes transmitting a request for an instruction to the guided teleoperation system.

27. The method according to claim 1 or 14, wherein: Transmitting the request for instructions to the teleoperation system includes transmitting the request for instructions to the teleoperation system that is remotely located relative to the robot.

28. The method according to claim 1 or 14, wherein: Transmitting the request for the instruction to the teleoperation system includes: transmitting the request for the instruction to the teleoperation system via a data network.

29. The method according to claim 28, wherein: Transmitting a request for instructions to the remote operation system through a data network includes: transmitting a request for instructions to the remote operation system through a wireless data network.

30. The method according to claim 1 or 14, wherein: Transmitting a request for instructions to the teleoperation system includes transmitting a request to the teleoperation system to determine which candidate action the robot should perform.

31. The method according to claim 1 or 14, wherein: Transmitting a request for instructions to the teleoperation system includes entering a standby mode.

32. The method according to claim 1 or 14, wherein: The executing the instruction includes invoking at least one candidate action from the set of candidate actions.

33. The method of claim 32, wherein the robot comprises a motion actuator, wherein: Invoking the at least one candidate action includes causing the robot to perform movement via the movement actuator.

34. The method according to claim 1 or 14, wherein: The executing the instructions includes identifying one or more auxiliary actions that can be performed by the robot.

35. The method according to claim 1 or 14, wherein: Updating the control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and the instructions includes increasing the autonomy level of the robot from a first autonomy level to a second autonomy level, the second autonomy level being higher than the first autonomy level.

36. The method according to claim 1 or 14, wherein: Updating the control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and the instructions includes updating at least one of control instructions, control data, or control parameters.

37. The method according to claim 1 or 14, wherein: Updating the control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and the instructions includes updating at least one of a configuration setting, a control method, a strategy, a process, a classifier, or a rule.

38. A robot comprising: main body; a controller comprising at least one processor; at least one physically actuatable component mechanically coupled to the body and communicatively coupled to the controller; at least one sensor communicatively coupled to the controller, the at least one sensor operable to collect data; and at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to: identifying a set of candidate actions that can be performed by the robot; For each candidate action in the set of candidate actions, collecting a corresponding auxiliary data set; transmitting a request for instructions to a teleoperation system communicatively coupled to the robot, the request for instructions comprising each candidate action in the set of candidate actions and each corresponding auxiliary data set; receiving the instruction from the teleoperation system; executing the instructions; and updating a control model of the robot based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and the instructions to increase a level of autonomy of the robot, Wherein, the at least one non-transitory processor-readable storage medium also stores processor-executable instructions and / or data, which, when executed by at least one processor, cause the robot to score each candidate action in the set of candidate actions by assigning a corresponding probability to each candidate action, wherein the corresponding probability is based at least in part on the corresponding progress expectation towards the goal that can be achieved by each candidate action.

39. The robot according to claim 38, wherein: The robot is a humanoid robot, and the main body has the appearance of a human body.

40. The robot according to claim 38, wherein: The at least one physically actuatable component comprises a motion actuator.

41. The robot according to claim 38, wherein: The at least one sensor is operable to collect at least one of audio, image, video, or environmental data.

42. The robot according to claim 38, wherein: The at least one non-transitory processor-readable storage medium also stores processor-executable instructions and / or data, which, when executed by at least one processor, cause the robot to detect at least one of an event or condition through the at least one sensor, and wherein the processor-executable instructions and / or data, which, when executed by the at least one processor, cause the robot to identify a set of candidate actions that can be performed by the robot, also cause the robot to identify a set of candidate actions that can be performed by the robot in response to detecting at least one of the events or conditions.

43. The robot of claim 38, wherein: The at least one non-transitory processor-readable storage medium further stores processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to receive a goal to be accomplished by the robot, the goal being received from the teleoperation system; and The processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to identify a set of candidate actions that can be performed by the robot further cause the robot to identify a set of candidate actions that can be performed by the robot to facilitate the goal.

44. The robot according to claim 43, wherein: The processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to transmit a request for instructions to the teleoperation system also cause the robot to transmit the request for instructions to the teleoperation system in response to determining that the robot cannot select a candidate action from the set of candidate actions to be performed.

45. The robot according to claim 38, wherein: The robot is a member of a robot swarm, and based at least in part on the update made to the control model of the robot, one or more other robots in the robot swarm are able to update the control model of the one or more other robots to increase the autonomy level of the one or more other robots.

46. ​​The robot according to claim 38, wherein: The at least one non-transitory processor-readable storage medium also stores processor-executable instructions and / or data, which, when executed by at least one processor, cause the robot to rank each candidate action in a preference order, wherein the preference order is based at least in part on the corresponding probability of each candidate action.

47. The robot according to claim 46, wherein: The at least one non-transitory processor-readable storage medium also stores processor-executable instructions and / or data, which, when executed by at least one processor, cause the robot to use at least one of the corresponding probability and the preference order of each candidate action to determine whether the robot can select a candidate action from the set of candidate actions to be performed.

48. The robot according to claim 47, wherein: The processor executable instructions and / or data, which when executed by the at least one processor cause the robot to use the corresponding probability of each candidate action and at least one of the preference order to determine whether the robot can select a candidate action from the set of candidate actions to be performed, also cause the robot to determine whether the corresponding probability meets a predetermined probability criterion.

49. The robot according to claim 47, wherein: The processor executable instructions and / or data, which when executed by the at least one processor cause the robot to use the corresponding probability of each candidate action and at least one of the preference order to determine whether the robot can select a candidate action from the set of candidate actions to be performed, also cause the robot to determine whether a difference between a first corresponding probability and a second corresponding probability satisfies a predetermined difference criterion.

50. A robot comprising: main body; a controller comprising at least one processor; at least one physically actuatable component mechanically coupled to the body and communicatively coupled to the controller; at least one sensor communicatively coupled to the controller, the at least one sensor operable to collect data; and at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to: identifying a set of candidate actions that can be performed by the robot; For each candidate action in the set of candidate actions, collecting a corresponding auxiliary data set; transmitting a request for instructions to a teleoperation system communicatively coupled to the robot, the request for instructions comprising each candidate action in the set of candidate actions and each corresponding auxiliary data set; receiving the instruction from the teleoperation system; executing the instructions; and updating a control model of the robot based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and the instructions to increase a level of autonomy of the robot, The robot is a semi-autonomous robot characterized by a level of autonomy determined at least in part by the control model, wherein: The at least one non-transitory processor-readable storage medium further stores processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to compare the level of autonomy of the robot to a threshold level of autonomy, above which the robot is operable to autonomously select a candidate action from the set of candidate actions to be performed; processor-executable instructions and / or data that when executed by the at least one processor cause the robot to transmit a request for instructions to a teleoperation system further cause the robot to transmit the request for instructions to the teleoperation system in response to determining that the level of autonomy of the robot is below the threshold level of autonomy; The at least one non-transitory processor-readable storage medium further stores processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to determine the threshold level of autonomy; and The processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to determine the threshold level of autonomy also cause the robot to assign a respective probability to each candidate action, the respective probability being based at least in part on a respective expectation of progress toward a goal that can be achieved by each candidate action.

51. The robot according to claim 50, wherein: The at least one non-transitory processor-readable storage medium also stores processor-executable instructions and / or data, which, when executed by at least one processor, cause the robot to perform one of the candidate actions in the set of candidate actions in response to determining that the level of autonomy of the robot meets or exceeds the threshold level of autonomy.

52. The robot according to claim 50, wherein: The robot is a member of a robot swarm.

53. The robot according to claim 52, wherein: Based at least in part on the updates made to the control model of the robot, one or more other robots in the robot fleet can update the control model of the one or more other robots to increase the autonomy level of the one or more other robots.

54. The robot according to claim 50, wherein: The at least one non-transitory processor-readable storage medium also stores processor-executable instructions and / or data that, when executed by at least one processor, cause the robot to rank each candidate action in an order of preference based at least in part on the corresponding probabilities.

55. The robot according to claim 50, wherein: The processor executable instructions and / or data which, when executed by the at least one processor, cause the robot to determine that the level of autonomy of the robot is below the threshold level of autonomy also cause the robot to determine that the corresponding probability of each candidate action in the set of candidate actions fails to satisfy at least one criterion.

56. The robot according to claim 38 or 50, further comprising a data repository, wherein: The processor executable instructions and / or data that, when executed by the at least one processor, cause the robot to collect a corresponding auxiliary data set for each candidate action in the set of candidate actions also cause the robot to collect a first subset of the corresponding auxiliary data set through the one or more sensors and / or extract a second subset of the corresponding auxiliary data set from the data repository.

57. The robot according to claim 38 or 50, further comprising a communication subsystem, wherein: The processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to transmit a request for instructions to the teleoperation system also cause the robot to transmit a request for the instructions through the communication subsystem and receive the instructions through the communication subsystem.

58. The robot according to claim 38 or 50, wherein: The teleoperation system is a guided teleoperation system.

59. The robot according to claim 38 or 50, wherein: The teleoperation system is located at a remote location relative to the robot.

60. The robot according to claim 38 or 50, wherein: The processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to transmit a request for instructions to the teleoperation system also cause the robot to transmit a request for instructions to the teleoperation system over a data network.

61. The robot according to claim 38 or 50, wherein: The processor-executable instructions and / or data, which when executed by the at least one processor cause the robot to transmit a request for instructions to the teleoperation system, also cause the robot to enter a standby mode.

62. The robot according to claim 38 or 50, wherein: The processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to execute the instructions also cause the robot to invoke at least one candidate action from the set of candidate actions.

63. The robot of claim 62, wherein the at least one physically actuatable component comprises a motion actuator, wherein: The processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to invoke the at least one candidate action also cause the robot to perform movement via the movement actuator.

64. The robot according to claim 38 or 50, wherein: The processor-executable instructions and / or data which, when executed by the at least one processor, cause the robot to execute the instructions also cause the robot to identify one or more auxiliary actions that can be performed by the robot.

65. The robot according to claim 38 or 50, wherein: The processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to update the control model also cause the robot to increase an autonomy level of the robot from a first autonomy level to a second autonomy level, the second autonomy level being higher than the first autonomy level.

66. The robot according to claim 38 or 50, wherein: The processor executable instructions and / or data which, when executed by the at least one processor, cause the robot to update the control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set and the instructions, also cause the robot to update at least one of the control instructions, control data or control parameters.

67. The robot according to claim 38 or 50, wherein: The processor-executable instructions and / or data that, when executed by the at least one processor, cause the robot to update the control model based at least in part on each candidate action in the set of candidate actions, each corresponding auxiliary data set, and the instructions, also cause the robot to update at least one of a configuration setting, a control method, a strategy, a process, a classifier, or a rule.

Images