Virtual tools for supported tele-operations

Abstract

To provide a user interface for instructing a tele-operating system and the tele-operating system.SOLUTION: The user interface includes an output device configured to output at least one virtual tool representation of at least one tool to an operator of the tele-operating system, and an input device configured to obtain a selection instruction from the operator for selecting one of the at least one output virtual tool representation. The user interface further includes a control circuit configured to determine a task based on the selected virtual tool representation, control displaying the selected virtual tool representation to the operator via the output device, acquire an action instruction from the operator via the input device, interpret the acquired action instruction based on the determined task, and control the tele-operating system to perform an action using the at least one tool based on the interpreted action instruction and the selected virtual tool representation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the general field of virtual reality and interfaces for teleoperated systems. In particular, a user interface for a teleoperated system, a computer-implemented method for controlling a teleoperated system, and a corresponding program are disclosed. [Background technology]

[0002] Artificial intelligence (AI)-assisted teleoperation has a range of assistance levels: no assistance, moderate task assistance, and full robot autonomy. Current teleoperation systems apply virtual reality (VR) or augmented reality (AR) systems to their user interfaces (UIs), and their effectiveness depends on intuitive and efficient communication of the settings, preferences, and constraints of the assistance functions.

[0003] Teleoperation, sometimes called remote operation, refers to the operation of a system or machine located at a distance. Teleoperation is an example of a human-machine system. Teleoperated systems offer a range of autonomy, from manual control to full automatic piloting of autonomous devices (robots).

[0004] In teleoperation, the current state of the art involves robots equipped with general-purpose actuators and direct operator control of the actuators' movements. A typical example of a general-purpose actuator is a grasping manipulator. Nevertheless, direct operator control of the manipulator often provides the manipulator's movements, which can be inefficient and imprecise.

[0005] An alternative approach in the field of teleoperation is to equip the robot with specialized actuators that contain specific tool attachments. Alternatively, the robot is equipped with a general-purpose actuator that holds a specific tool, such as a wrench, and the operator controls the tool in a 1:1 manual control mode. While this alternative approach may be more efficient than using a robot with a general-purpose actuator, it does require the robot to be equipped with a specialized tool.

[0006] In practice, robots often have difficulty imposing constraints on physical tools such as torque wrenches, or it may not even be possible to impose constraints on physical tools.

[0007] Alternatively, robots may use specialized tools, such as power drills, which are less flexible and often much more expensive than general-purpose tools.

[0008] Generally, in AI-assisted teleoperation, the current state of the art places the burden on the operator to select the specific assistance mode to be performed by the robot, either explicitly from a text menu or by pressing a button or moving a slider in a VR user interface, which is not intuitive to use. As a specific example, the operator selects the final torque and rotation speed of a torque wrench using dials in a VR user interface.

[0009] U.S. Patent No. 9,272,418 (B1) discloses a user interface for a teleoperator that can teach a robot to learn capabilities. Each capability of the robot requires a set of constraints as user input from the operator. The user interface provides interface elements that assist in inputting the respective constraints for each capability of the robot. The operator must know what each capability does and what constraints should be applied to each capability. The operator selects the desired robot capability in the user interface and sets the desired constraints for the selected capability. The necessary constraints must be set in advance to successfully perform the intended operation. Summary of the Invention [Problem to be solved by the invention]

[0010] The described aspects of the remote control system suggest that an improved remote control system and an improved user interface for the remote control system would be desirable.

[0011] A user interface for a remote control system according to independent claim 1, a remote control system and a computer implemented method according to the corresponding independent claims address this problem among other issues. [Means for solving the problem]

[0012] According to a first aspect of the present disclosure, a user interface for instructing a teleoperated system to perform an operation includes an output device configured to output at least one virtual tool representation of at least one tool to an operator of the teleoperated system, and an input device configured to receive a selection command from the operator to select one of the output at least one virtual tool representation. The user interface includes a control circuit configured to determine a task based on the selected virtual tool representation, control display of the selected virtual tool representation to the operator via the output device, and obtain an operation command from the operator via the input device. The control circuit is configured to interpret the obtained operation command based on the determined task, and control the teleoperated system to perform an operation using the at least one tool based on the interpreted operation command and the selected virtual tool representation.

[0013] According to a second aspect, a remote control system includes a user interface according to the first aspect.

[0014] According to a third aspect, a computer program product comprises a non-transitory computer-readable medium having a computer-readable program embedded therein, which, when executed on a computing device, causes the computing device to perform the following steps: outputting at least one virtual tool representation of at least one tool to an operator of a teleoperation system via an output device; receiving, via an input device, a selection instruction from the operator to select one of the output at least one virtual tool representations; determining, by a control circuit, a task based on the selected virtual tool representation; displaying the selected virtual tool representation to the operator via the output device; receiving an operation instruction from the operator via the input device; interpreting the received operation instruction based on the determined task; and controlling the teleoperation system to perform an operation using the at least one tool based on the interpreted operation instruction and the selected virtual tool representation.

[0015] The dependent claims define advantageous embodiments of the user interface and the remote control system.

[0016] Aspects and implementations of the present disclosure are described in the following description of specific embodiments in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a schematic flow chart providing an overview of method steps for controlling a robot using a teleoperation system. [Figure 2] FIG. 2 is a block diagram illustrating a user interface for commanding the remote control system and elements of the remote control system. [Figure 3] 1A-1C illustrate some examples of virtual tools in an embodiment of the disclosed teleoperation system. DETAILED DESCRIPTION OF THE INVENTION

[0018] The figure descriptions use the same reference numbers for the same or corresponding elements in different figures. The figure descriptions omit detailed descriptions of the same reference numbers in different figures, where possible, without adversely affecting comprehensibility. The figures and the elements shown therein are not necessarily drawn to scale.

[0019] The term virtual tool refers to a tool metaphor, which is a representation of a tool (physical tool) in a user interface. A virtual tool refers to or relates to a physical tool that is familiar to a large group of potential operators. Therefore, its functionality and its operational constraints are intuitively clear to the operator. The tool metaphor has an iconic representation in the user interface and is further associated with typical parameters, such as cutting head dimensions, and constraints, such as constraining a circular saw to operate parallel to the work surface.

[0020] The expression physical equivalent tool (PET) refers to the actual physical tool that corresponds to the virtual tool.

[0021] The user interface according to the present disclosure is an intuitive interface that uses representations of actual tools as metaphors to allow the operator to select a teleoperated robotic action that provides a desired level of control over the teleoperation process. Selection provides the operator with a choice of level of control via the metaphorical interface. The present disclosure provides an intuitive user interface in VR or AR by visualizing virtual tools such as a wrench, a torque-controlled ratchet spanner, or a power drill. Selection of these virtual tools defines the operator's intention for assistance. The virtual tools are displayed and can be controlled by the operator in VR. On the physical side, the robot does not replace the tool but instead uses a standard tool to simulate the virtual tool. Thus, an operator skilled with a regular tool for directly manipulating an object, such as a standard wrench, can intuitively use the teleoperation system to control the robot.

[0022] The disclosed user interface provides an intuitive interface that uses a depiction of an actual tool in the form of a virtual tool representation as a metaphor for selecting a robotic operation. Unlike known approaches, this user interface does not focus on information about how a drill tool must be aligned to perform a functional operation, such as a drilling or screwing operation. Instead, by viewing at least one virtual tool representation corresponding to a physical tool and selecting a particular virtual tool representation, a user selects a particular function and a level of control over the function's execution by a teleoperation system. However, auxiliary information for function execution still needs to be obtained by the assistance system.

[0023] Selection aims to allow the operator to select a level of control through a metaphorical interface. Tool metaphors are an element of user interface design for graphical user interfaces, such as the brush metaphor in a painting program or the spray can metaphor. However, these tool metaphors only apply to controlling digital computers, not to controlling physical interactions with objects using telerobotics.

[0024] According to one embodiment, the user interface has a control circuit configured to determine task parameters and task constraints based on the selected virtual tool representation, and to interpret the obtained operation instructions based on the determined task, task parameters, and task constraints.

[0025] According to one embodiment, a user interface includes an output device configured to output a plurality of virtual tool representations of a plurality of tools to an operator, wherein actions performed by the robot mimic use of a tool corresponding to a selected virtual tool representation by using different tools from the plurality of tools.

[0026] In one embodiment of the user interface, each virtual tool representation comprises at least one adjustable tool constraint.

[0027] The adjustable tool constraints may include at least one of a direction constraint, a size constraint, a maximum speed setting, and a maximum thickness of material removed from the object.

[0028] In one embodiment of the user interface, at least one virtual tool representation includes an element configured to be manipulated by an operator to conform to an adjustable tool constraint.

[0029] A user interface according to one embodiment has at least one virtual tool representation including an operating element, in particular a virtual slide control or a rotation control, configured to be manipulated by an operator to numerically adapt adjustable tool constraints.

[0030] The user interface according to one embodiment is part of a virtual reality or augmented reality system.

[0031] In one embodiment of the user interface, the input device includes a pointer device.

[0032] In a second aspect of the present disclosure, a remote control system includes a user interface according to any one of the above embodiments.

[0033] A teleoperated system according to one embodiment includes a multipurpose physical tool configured to perform multiple operations corresponding to multiple tool operation capabilities.

[0034] According to one embodiment, the teleoperation system includes a specialized tool configured to perform an action corresponding to the operational capabilities of a selected virtual tool representation, and a control circuit configured to control the specialized tool based on the task constraints of the selected virtual tool representation to perform the action.

[0035] According to one embodiment, the teleoperated system is configured to operate in a direct control mode, and when operating in the direct control mode, the control circuitry is configured to determine whether controlling the teleoperated system based on the interpreted operation instructions to perform the operation would result in a violation of at least one tool constraint, and if it determines that the at least one tool constraint will be violated, the control circuitry is configured to output feedback information about the determined violation to an operator via an output device.

[0036] According to one embodiment, when operating in direct control mode, the teleoperated system is configured to determine whether controlling the teleoperated system based on the interpreted operation command to perform the operation would result in approaching a violation of at least one tool constraint.

[0037] In one embodiment, the teleoperation system is configured to generate and output feedback information, including a visual, audio, or tactile warning, upon determining an approaching violation of at least one tool constraint.

[0038] According to one embodiment of the teleoperation system, the teleoperation system is configured to stop operation of the tool or slow down the speed of the tool upon determining an approaching violation of at least one tool constraint.

[0039] A teleoperation system according to one embodiment is configured to predict a tool trajectory of a tool and project the predicted tool trajectory onto constraints associated with the tool.

[0040] According to one embodiment, the remote control system is configured to operate in an indirect control mode, in which the remote control system is configured to determine and store in a data storage a control command for controlling the remote control system based on the interpreted operation command to perform an operation, and the remote control system is configured to retrieve the stored control command and perform control of the remote control system based on the retrieved control command.

[0041] The remote control system according to one embodiment is configured to select a tool or tools to perform an action when operating in indirect control mode.

[0042] In one embodiment of the teleoperation system, when operating in the indirect control mode, the teleoperation system is configured to simulate the operation of the teleoperation system, in particular to simulate in real time the effects of operating at least one tool based on stored control commands, and to visualize the simulated effects via an output device.

[0043] The following description of the embodiment uses outputting multiple virtual tool representations of multiple tools to an operator via an output device of a user interface. The operator then selects one of the output virtual tool representations. In an alternative scenario also encompassed by this disclosure, an output device of a user interface outputs one virtual tool representation of one tool to an operator of a teleoperation system. The operator then selects the one output virtual tool representation by inputting a selection command.

[0044] FIG. 1 shows a schematic flow chart providing an overview of the method steps for controlling a robot 10 with a teleoperation system 1 shown in FIG.

[0045] The remote control system 1 is provided with a user interface for instructing the remote control system 1 to perform a particular function or an action or a series of actions.

[0046] The user interface includes an output device configured to output virtual tool representations of a plurality of tools to an operator of the remote operation system, and an input device configured to receive a selection command from the operator to select one of the output virtual tool representations.

[0047] The user interface comprises a control circuit 6 configured to determine tasks, task parameters, and task constraints based on a selected virtual tool representation. The control circuit may form part of a computing device that includes memory for storing data in a database 7. Some specific aspects of the physical implementation are described with reference to FIG.

[0048] The method for controlling a user interface, when executed by, for example, a control circuit 6 that is part of a computing device, may cause the computing device to perform step S1 of outputting virtual tool representations of a plurality of tools 9 to an operator of the remote operation system 1 via an output device.

[0049] The virtual tool representation may be associated with additional tool parameters and settings, such as the direction and magnitude of the tool constraint, or the maximum rotational speed setting, or the maximum feed rate, or the maximum thickness of material to be removed. Tool parameters may be visualized as variants of the virtual tool representation, for example, corresponding to a larger or smaller belt sander, or different grinding attachments for a multi-purpose tool.

[0050] Additional tool constraints may be defined by corresponding parts of the virtual tool representation that can be manipulated by the operator, for example, the size of the substrate for a virtual circular saw, or the diameter of a grinder tool.

[0051] Alternatively, the additional tool constraints may be set numerically by the operator using sliders or other conventional graphical user interface (GUI) elements output by the teleoperation system 1 via an output device.

[0052] In step S2, the operator selects one virtual tool representation from the displayed plurality of virtual tool representations. In step S2, the control circuit 6 executes the process of obtaining a selection command from the operator via the input device of the user interface to select one of the output plurality of virtual tool representations.

[0053] Step S2 is followed by step S3, in which the computer-implemented method determines a task, task parameters, and task constraints based on the selected virtual tool representation.

[0054] A task defines a goal that the robot 10 is intended to achieve in order to assist a human, in particular an operator, by performing an action or a sequence of actions using at least one actuator of the robot, including for example a tool 9 (physical tool 9).

[0055] Task parameters are typically numerical factors or similar observable factors that are a set of conditions for performing a task in the environment 13. Task parameters are quantities that have selectable values ​​for the particular circumstances of the task. In relation to one task parameter, other task parameters in the set of task parameters for a particular task may be set. Task parameters may include, for example, the direction and magnitude of a constraint, a setting for the maximum rotational speed of the rotary tool 9, or a maximum thickness of material to be removed from an object (workpiece).

[0056] In general, a task constraint is a constraint or boundary condition for a particular task. A task may be interpreted as an ordered set of constraints to be achieved by an actuator, specifically the tool 9 of the robot 10. In this example, the task constraint is set by selecting a particular virtual tool representation from multiple virtual tools. In one example, a specific task constraint associated with a grinding tool (grinder) is the diameter of the grinder.

[0057] In step S4, the computer-implemented method displays the representation of the selected virtual tool to the operator via an output device.

[0058] In step S5, the method obtains an operation command from an operator via an input device, and then the method interprets the obtained operation command based on the determined task parameters and task constraints.

[0059] Step S5 is followed by step S6, in which the method controls the teleoperation system to perform an operation using a tool (physical tool) corresponding to the selected virtual tool based on the interpreted operation command and the selected virtual tool representation.

[0060] In step S7, the method determines whether a new tool is needed. For example, the operator of the remote operation system 1 may want to use a new tool to continue the task. If it is determined that a new tool is needed (Yes), the method proceeds to step S1. The method then continues with a new processing loop by performing steps S1 to S6 again. If it is determined that a new tool is not needed (No), the processing of the method ends. Step S7 and the processing loop correspond to optional steps in the method shown in FIG. 1.

[0061] FIG. 2 shows a user interface for controlling the remote control system 1 and a block diagram showing elements of the remote control system 1.

[0062] The user interface may be part of a virtual reality (VR) or augmented reality (AR) system.

[0063] The remote control system 1 may operate in different control modes, including a direct control mode or an indirect control mode.

[0064] In the direct control mode, the teleoperation system exercises direct 1:1 control of the actual physical end effector of the robot 10. In the direct control mode, the teleoperation system 1 controls the teleoperation system 1, in particular the tool 9, directly based on determined motion commands from the operator. In the direct control mode, no simulation of the operation of the teleoperation system 1, in particular the tool control of the tool 9, is performed. No control commands are stored in the database 7 for later control of the tool. In the direct control mode, the teleoperation system 1 controls the tool 9 online. Characteristically, when operating in the direct control mode, the teleoperation system 1 determines whether controlling the teleoperation system 1 based on the interpreted motion commands to perform the operation would approach a violation of at least one tool constraint of the tool 9 corresponding to the selected virtual tool representation.

[0065] When operating in direct control mode, the teleoperation system 1 may determine whether the operator commands the selected virtual tool during operation of the tool 9 to perform an action that the physical tool 9 cannot perform due to tool constraints. For example, the operator commands the virtual tool representation to perform a tool trajectory that the associated physical tool cannot follow.

[0066] If it determines that such a tool constraint is violated, the teleoperation system 1 may output a respective warning by outputting a warning signal to the operator via an output device of the VR / AR interface 4. The warning signal may comprise a visual warning encoded as a periodic change in color or intensity (blinking) of the visual signal, or a respective audio signal.

[0067] Alternatively or additionally, if the teleoperation system 1 determines that a violation of such tool constraints is imminent, it may automatically reduce the feed rate or speed of the physical tool 9 or stop operation of the physical tool 9. The teleoperation system 1 may determine that a violation of such tool constraints is imminent if it determines that the difference between the predicted trajectory of the physical tool 9 generated based on commands obtained from the operator and the constraints of the physical tool 9 is less than a threshold value.

[0068] Alternatively or additionally, if the teleoperation system 1 determines that a violation of such tool constraints is at least imminent or has already occurred in direct control mode, it may temporarily disconnect the trajectory of the selected virtual tool from 1:1 control by the operator and project the trajectory of the virtual tool commanded by the operator onto the tool constraints defined by the physical tool 9 associated with the selected virtual tool. For example, instead of tilting the tool 9 relative to the surface of the object (workpiece), the teleoperation system 1 may maintain the path of the tool 9 perpendicular to the surface.

[0069] The direct control mode is advantageous in scenarios and applications where the robot 10 has the ability to perform its intended movements in at least approximate real time. As a specific example, the robot 10 is equipped with a tool 9 with a sander large enough to simulate a virtual belt sander, for example, if selected as the selected virtual tool representation by the operator.

[0070] In the indirect control mode, the teleoperation system 1 determines control commands for controlling the teleoperation system 1 based on the operation commands obtained from the operator and interpreted to perform an operation, and stores the control commands in the database 7. The teleoperation system 1 obtains the stored control commands from the database 7 and controls the teleoperation system 1 based on the obtained control commands. The indirect control mode is particularly suitable for applications in which the selected virtual tool has greater capabilities than the physical tool 9 available to the robot 10. This is the case, for example, when the virtual tool representation corresponds to a belt sander and the robot 10 uses a small rotary sander as the physical tool 9 to simulate the selected virtual tool.

[0071] In the indirect control mode, the teleoperation system 1 may select one or more tools 9 to perform the operation specified in the operation command obtained from the operator via the AR / VR interface 4. The teleoperation system 1 may select the most appropriate physical tool to perform the task commanded by the operator by selecting a virtual tool representation. The most appropriate physical tool 9 may be the physical tool 9 having a set of tool constraints determined to be closest to the tool constraints of the selected virtual tool.

[0072] Additionally or alternatively, while operating in indirect control mode, the teleoperation system 1 may provide assistance by simulating the commanded action or sequence of actions for operation, e.g., simulating in real time the effects of operating the tool 9 based on the commands, and visualizing the results of the simulation to the operator, e.g., via an output device.

[0073] The teleoperation system 1 includes an augmented reality / virtual reality interface 4 (AR / VR interface 4), a control circuit 6, and a database 7 stored in a memory configured to store the database.

[0074] The AR / VR interface 4 may operate as an augmented reality (AR) interface that seamlessly blends computer-generated content, e.g., computer-generated additional image data, utilizing one or more of the operator's senses with the real world of the environment perceived by the operator's senses. The AR interface enhances the operator's perception of the environment by overlaying additional computer-generated information. In the present disclosure, the control circuit 6 generates and outputs virtual tool representations of multiple tools to the operator of the teleoperation system 1, and the multiple virtual tool representations may be shown to the operator as overlaid additional information on an optical head-mounted display.

[0075] Alternatively, the AR / VR interface 4 may operate as a virtual reality (VR) interface that replaces the real environment with an artificial one, which the operator experiences through sensory stimuli provided by a computer. Actions taken by the operator at least partially determine what occurs in the environment. The VR interface may include hardware for operator pose tracking and a three-dimensional (3D) near-eye display of images and video.

[0076] The AR / VR interface 4 may include an output device for outputting output information 2 to a user (operator) and an input device for obtaining input information 3 from the operator. The output device of the AR / VR interface 4 may include a display, such as a computer monitor, or an image projector that projects an image onto a surface to visually display information to the operator.

[0077] The display device may include a wearable device, for example an optical head mounted display, that displays information in a hands-free format.

[0078] The output device outputs, inter alia, virtual tool representations corresponding to the plurality of tools 9 to the operator of the teleoperation system 1 .

[0079] The output device, among other things, displays a representation of the selected virtual tool to the operator.

[0080] The output device may further output feedback information when the teleoperation system 1 determines that a violation of at least one tool constraint is imminent.

[0081] The control circuit 6 may determine whether controlling the teleoperated system 1 based on the interpreted operation command to perform an operation would violate at least one tool constraint of the tool 9 when the teleoperated system is operating in direct control mode.

[0082] If the control circuitry 6 determines a violation of at least one tool constraint, the control circuitry 6 is configured to output feedback information about the determined violation to the operator via an output device of the AR / VR interface 4. The output feedback information may include a warning about the determined violation of the at least one tool constraint.

[0083] When the remote control system 1 is operating in the indirect control mode, the remote control system 1 simulates the operation of the remote control system 1 via the output device, in particular, simulates in real time the effect of operating the tool 9 based on the stored control commands, and visualizes the simulated effect via the output device.

[0084] The AR / VR interface 4 may include an input device for receiving a selection command from the operator to select one of the output virtual tool representations being presented to the operator.

[0085] The input device may include any device suitable for providing information, data, or control signals to an information processing system, such as a computer, and may include at least one of a keyboard, a mouse, a joystick, or a microphone.

[0086] In particular, the input device may include a pointing device (pointer).

[0087] Alternatively or additionally, the input device may include a gesture tracking system for tracking the gestures of the operator.

[0088] The control circuitry 6 may include at least one processor, signal processor, microprocessor, microcontroller (μC), application specific integrated circuit (ASIC), system on a chip (SoC), graphics processing unit (GPU), or combination of such integrated circuits (ICs).

[0089] The control circuit 6 may be implemented in a distributed manner, for example including the AR / VR interface 4, data processing resources of the robot 10 or one or more servers located remotely with respect to the robot 10 and connected via a communication network not explicitly shown in FIG. 2 .

[0090] The teleoperated system 1 further includes at least one sensor 5 for sensing a physical environment 13 in which the teleoperated system 1 is performing a task using a tool 9 (physical tool 9).

[0091] The sensors 5 may include multiple sensors forming a sensor suite that acquires sensor information 11 about the environment 13. The sensors 5 may include cameras that acquire images and video from the environment 13 in which the robot 10 is operating, in particular performing an action, series of actions or function under the control of the teleoperation system 1.

[0092] The teleoperation system 1 includes a tool control interface 8 that receives control signals from the control circuit 6 and controls a tool 9 that is performing an action 12, a series of actions, or a function on at least one physical object in an environment 13.

[0093] The tool control interface 8 and the tool 9 may form part of a device 10 (robot 10) that operates autonomously or semi-autonomously. The tool control interface 8 generates and outputs control signals for controlling at least one tool 9 that performs a function in the environment 13, for example by performing an action or series of actions on at least one object in the environment 13.

[0094] Preferably, tool 9 is a multi-function tool capable of performing several separate functions under the control of tool control interface 8 .

[0095] The physical tool 9 available to the robot 10 may include an arm of the robot 10 including an end effector (manipulator) disposed at the end of the arm. Thus, the end effector and arm, including, for example, a gripper, represent the tool 9 in this scenario. The application scenario may include, for example, the task of tightening a screw by using the gripper and the arm of the robot 10 as the respective physical tools 9. The operator may select a virtual representation of a torque wrench displayed by an output device. Selecting the virtual representation of the torque wrench gives the operator the possibility to specify the desired torque for tightening the screw. The arm and end effector of the robot 10 include built-in force-torque sensors, and the physical tool 9 then performs the commanded task of tightening the screw with the desired torque as the respective task parameter.

[0096] The robot 10 may use general-purpose or multi-purpose actuators to perform operations indirectly, for example, more slowly or repetitively than would the physical tool 9 corresponding to the selected virtual tool used in the virtual tool representation.

[0097] Alternatively or additionally, the robot 10 uses a special tool, e.g., a sander, for its operation, but instead of using physical guides used by the tool corresponding to the virtual tool representation, the control software of the robot actuators ensures task constraints set for the selected virtual tool representation. For example, the flat surface of a belt sander as physical tool 9 ensures parallel or flat application to the surface of the object serving as the workpiece.

[0098] The sensor 5 may be part of a device 10 (robot 10) that operates autonomously or semi-autonomously.

[0099] The robot 10 may be a service robot that assists humans in performing tasks in messy, dirty, dangerous, repetitive, or harsh environments. The robot 10 has a degree of autonomy, which, according to common convention, means "the ability to perform an intended task without human intervention based on current state and sensing." The degree of autonomy may range from partial autonomy, which includes human-robot interaction, to full autonomy, with no active human-robot intervention by an operator.

[0100] Artificial intelligence (AI)-assisted teleoperation spans a spectrum of assistance, from no assistance, through moderate task assistance, to full autonomy. User interfaces, including VR / AR interfaces, enable intuitive and efficient communication of settings, preferences, and constraints of the assistance functions provided by the robot 10.

[0101] An operator of the remotely operated system 1 intends to tighten a bolt. Depending on the particular task, the operator may intend to exert fine-grained control over the rotation angle of the bolt, for example 2.5 turns, or to rotate the bolt at a defined speed and duration, or to tighten the bolt to a predetermined torque value.

[0102] In the real world, without using the remote-controlled robot 10, the user selects the appropriate means to achieve these three goals:

[0103] The user may manually turn the bolt or use a wrench, thereby directly controlling the movement;

[0104] Alternatively, the user may use a power drill and affect the rotation speed and duration by pressing buttons on the power drill; or

[0105] Further alternatively, the user may use a spanning tool with a torque limiter to effectively rotate to a defined final torque.

[0106] In all three alternative cases, the appropriate means is an efficient choice with an appropriate level of control, high accuracy of goal achievement, and a familiar user interface for setting constraints, for example, with a torque setting dial. The user interface and teleoperation system 1 replicate the efficient and intuitive interface of a tool to control the teleoperation system 1 and robot 10 without actually requiring an actual physical tool. The use of a familiar and intuitive interface of a tool can reduce the training time required for an operator to use the teleoperation system.

[0107] FIG. 3 shows some examples of virtual tool representations and tools 9 suitable for embodiments of the disclosed teleoperation system.

[0108] The present disclosure provides an intuitive user interface that is particularly advantageous in virtual reality applications by visualizing a virtual tool such as a wrench, a torque-controlled ratchet spanner, or a power drill. By selecting a virtual tool, the operator communicates their intent with respect to artificial intelligence assistance via the selected tool. The user interface displays a representation of the virtual tool to the operator and allows the operator to control the selected virtual tool representation within the virtual environment. On the physical side, in the physical environment, the robot 10 simulates the selected virtual tool using a standard tool 9 for operation in the physical environment, rather than replacing its own tool 9 with a physical tool 9 corresponding to the selected virtual tool representation.

[0109] FIG. 3 shows three examples of virtual tool representations and corresponding robot 10 movements in a physical environment.

[0110] The "grinding," "polishing," and "routing" processes each use a respective virtual tool attachment on the virtual rotary tool 21. The virtual tool attachment may include spherical abrasive attachments 21.1, 21.2, 21.3, 21.4, or a cylindrical abrasive attachment, each having a specified diameter, for the multipurpose rotary tool 21. The robot may perform operations applying the physical tool 9 corresponding to the selected virtual rotary tool 21 directly following the trajectory performed by the operator, and may perform a specific cutting operation, for example, taking into account the depth and width of the groove defined by the virtual tool attachment with a specified diameter selected by the operator for the multipurpose rotary tool 21.

[0111] The operator's selection of the virtual tool representation of the belt sander 22 defines the process of creating flat areas on the surface of the part (workpiece), thereby obtaining a locally flat surface. By selecting the respective virtual abrasive belt 22.1 for the virtual belt sander 22, the operator may control, via the user interface, the roughness of the resulting flat surface produced by the robot on the physical part.

[0112] Selecting a virtual tool "router with specific tool bit 23" controls the robot to make a cut of a predetermined shape and depth in the part. The cut may be flat and at a 45-degree angle, for example, or may have a quarter-circle shape with a specified radius and follow a trajectory executed by the operator. The virtual trajectory of the virtual tool is constrained to be perpendicular to the surface, as would be done by a handheld router. The operator may set additional constraints by adjusting a virtual "ball-bearing bit" that defines an adjustable distance to the work surface at a height specific to the tool bit 23. The robot 10 performs the cutting action on the part based on the set distance.

[0113] The examples in Figure 3 are just a few example combinations of virtual tools and corresponding actions. A series of further examples are provided below, including but not limited to, the process of driving a nail.

[0114] The nailing process involves, for example, a virtual tool, a "nail gun." The corresponding physical actions performed by the robot 10 include placing a nail at a commanded location on the surface of the object and driving the nail into the object until the head of the nail is flush with the surface of the object.

[0115] Alternatively or additionally, the nailing process may involve a virtual tool, e.g., a "hammer." The corresponding physical action performed by the robot 10 includes placing a nail at a designated location on the surface of the object and gradually pushing or urging the nail into the object at a feed rate that depends on at least one of the size of the selected virtual tool "hammer," the speed of the virtual hammering gesture performed by the operator, and the amplitude of the virtual hammering gesture performed by the operator. The direction of the hammering gesture corresponds to the orientation of the nail relative to the surface of the object. The height of the final nail head above the surface of the object may vary between zero, meaning that the nail head is flush with the surface, and a value that depends on the total length of the nail.

[0116] The clamping process involves, for example, a virtual tool "clamp." The corresponding physical action performed by the robot 10 involves exerting pressure at a specified location on the surface of the object with a specified (maximum) force in a specific position and direction.

[0117] Alternatively or additionally, the clamping process may involve a virtual tool, such as a "corner clamp." The corresponding clamping process involves constraining two parts to maintain a specific angle between them, which angle may be set by an operator via a user interface.

[0118] The cutting process involves a virtual tool, e.g., a "knife," and the corresponding physical action performed by the robot 10 involves cutting the surface of the object essentially freehand and without constraints, following a trajectory performed by an operator, e.g., using a pointing device of a user interface.

[0119] Alternatively or additionally, the cutting process may involve a virtual tool, such as a "peeling knife." The corresponding cutting process may involve free cutting into the surface of the workpiece, constrained to remove a flake of material from the surface of the workpiece. The operator may set, via a user interface, the depth and / or size of the flake of material that the robot 10 removes from the workpiece.

[0120] Alternatively or additionally, the cutting step involves a virtual tool, e.g., a "circular saw," and a corresponding cutting step involves selecting a virtual tool representation of the "jigsaw" and then making a cut perpendicular to the otherwise unconstrained surface of the workpiece.

[0121] Alternatively or additionally, the cutting step involves, for example, a "jigsaw" virtual tool, and a corresponding cutting step involves selecting a "circular saw" virtual tool representation and then making a cut perpendicular to the surface of the workpiece, subject to the additional task constraint of a straight line.

[0122] The baking process involves a virtual tool, e.g., a "rolling pin." The corresponding physical action performed by the robot 10 involves flattening the dough to a specific height that can be adjusted by the operator via the user interface.

[0123] Alternatively or additionally, the baking process may involve a virtual tool, e.g., a "cookie cutter," and the corresponding physical process may involve an operator selecting a virtual tool representation of the "cookie cutter" and then cutting a particular shape into a flat dough rolled out on a flat surface.

[0124] The painting process involves selecting a virtual tool from a plurality of virtual tools, including instances of, for example, a "brush," a "paint roller," a "sprayer," a "pencil," a "crayon," etc. The corresponding physical action performed by the robot 10 involves physically painting or drawing on a surface according to a trajectory executed by the operator or based on specific settings made by the operator via a user interface. Unlike computer graphics software such as paint software, which uses a virtual brush to modify the display in a virtual environment, the robot 10 actually paints or draws the results of each of the operator's controls in the physical environment.

[0125] The greasing process may include various versions or variations of virtual tools such as an "oil dripper," a "grease gun," and a "grease brush." ​​The corresponding physical actions performed by the robot 10 include at least one of applying grease at a specific consistency, applying a specific amount of grease, and applying grease at a specific pressure to a surface area or grease nipple of the object.

[0126] The sorting process involves a virtual tool, e.g., a "sieve" in various versions, represented, e.g., by sieve inserts with respective diameters of sieve openings. The corresponding physical actions performed by the robot 10 involve sorting the objects according to their size, in particular into a first group of objects whose diameter is smaller than a selected size (size threshold) and a second group of objects whose diameter is larger than the selected size.

[0127] Another process performed by the robot 10 that the operator may control via the user interface includes vacuum cleaning. Selecting the "vacuum cleaning" virtual tool may include selecting one particular virtual attachment tool from multiple virtual attachment tools presented to the operator via the user interface. The robot 10 may perform the cleaning process by adjusting process parameters, such as the operating width and vacuum strength of the cleaning process, based on the selected virtual attachment tool.

[0128] The aforementioned examples are derived in particular from a workshop or home environment where the robot 10 is used in the respective teleoperation application. The described uses of the teleoperation system 1 include further application areas, for example, disaster recovery robots and underwater repair or installation by autonomous devices. The teleoperation system may also operate in extraterrestrial application scenarios, for example, in outer space. The method may prove advantageous in remote work applications in mechanical or electrical workshops or at on-site customer locations. Application areas include household chores and maintenance.

[0129] All of the features described above or shown in the figures may be combined with each other in any advantageous manner within the scope of the present disclosure. In the detailed description of the embodiments, numerous specific details have been presented to provide a thorough understanding of the invention as defined in the claims. It will be apparent that the claimed invention can be practiced without including all of these specific details.

[0130] In the specification and claims, the phrase "at least one of A and B" can be substituted for the phrase "A and / or B," and vice versa, since they are used interchangeably. The phrase "A and / or B" means "A, or B, or A and B."

Claims

1. A user interface for commanding a remote control system (1) to perform an action, comprising: an output device (4) configured to output at least one virtual tool representation (21, 22, 23) of at least one tool (9) to an operator of said teleoperation system (1); an input device (4) configured to receive a selection command from the operator to select one of the output at least one virtual tool representations (21, 22, 23); A control circuit (6) comprising: determining a task for the selected virtual tool representation (21, 22, 23); controlling the display of the selected virtual tool representation (21, 22, 23) to the operator via the output device (4); Acquiring an operation command from the operator via the input device (4); interpreting the acquired operation command based on the determined task; Controlling the teleoperation system (1) to perform an action using the at least one tool (9) based on the interpreted action command and the selected virtual tool representation (21, 22, 23). and a control circuit (6) configured to:

2. The control circuit determining task parameters and task constraints based on the selected virtual tool representation (21, 22, 23); Interpreting the obtained motion instructions based on the determined tasks, task parameters, and task constraints. The user interface of claim 1 , configured to:

3. the output device (4) is configured to output a plurality of virtual tool representations (21, 22, 23) of a plurality of tools (9) to the operator; 2. The user interface of claim 1, wherein the action mimics the use of a tool corresponding to the selected virtual tool representation by using a different tool from the plurality of tools.

4. each virtual tool representation (21, 22, 23) comprises at least one adjustable tool constraint; The user interface of claim 1 , wherein the adjustable tool constraints include at least one of a directional constraint, a size constraint, a maximum speed setting, and a maximum thickness of material removed from an object.

5. The user interface of claim 4 , wherein the at least one virtual tool representation (21, 22, 23) includes an element configured to be manipulated by the operator to conform to the adjustable tool constraint.

6. 5. The user interface of claim 4, wherein the at least one virtual tool representation (21, 22, 23) includes an operating element, in particular a virtual slide control or a rotation control, configured to be manipulated by the operator to numerically adapt adjustable tool constraints.

7. The user interface of claim 1 , wherein the user interface is part of a virtual reality or augmented reality system.

8. The user interface of claim 1 , wherein the input device (4) comprises a pointer device.

9. 1. A remote operation system including a user interface, the user interface comprising: an output device (4) configured to output at least one virtual tool representation (21, 22, 23) of at least one tool (9) to an operator of said teleoperation system; an input device (4) configured to receive a selection command from the operator to select one of the output at least one virtual tool representations (21, 22, 23); A control circuit (6) comprising: determining a task for the selected virtual tool representation (21, 22, 23); controlling the display of the selected virtual tool representation (21, 22, 23) to the operator via the output device (4); Acquiring an operation command from the operator via the input device (4); interpreting the acquired operation command based on the determined task; Controlling the teleoperation system to perform an action using the at least one tool (9) based on the interpreted action command and the selected virtual tool representation (21, 22, 23). and a control circuit (6) configured to:

10. 10. The teleoperated system of claim 9, wherein the teleoperated system includes a multipurpose physical tool (9) configured to perform multiple operations corresponding to multiple tool (9) operation capabilities.

11. the teleoperation system includes a specialized tool (9) configured to perform an action corresponding to the action capability of the selected virtual tool representation (21, 22, 23); 10. The teleoperation system of claim 9, wherein the control circuitry (6) is configured to control the specialized tool (9) based on task constraints of the selected virtual tool representation (21, 22, 23) to perform the operation.

12. the remote control system is configured to operate in a direct control mode; In the direct control mode, the control circuit (6) is configured to determine whether controlling the teleoperated system based on the interpreted operation command to perform the operation would violate at least one tool constraint; 10. The teleoperation system of claim 9, wherein, upon determining that the at least one tool constraint is violated, the control circuit (6) is configured to output feedback information about the determined violation to the operator via the output device (4).

13. the remote control system is configured to operate in a direct control mode; 10. The teleoperated system of claim 9, wherein in the direct control mode, the teleoperated system is configured to determine whether controlling the teleoperated system based on the interpreted operation command to perform the operation would result in approaching a violation of the at least one tool constraint.

14. 14. The teleoperation system of claim 13, wherein the teleoperation system is configured to generate and output feedback information comprising a visual, audio, or tactile warning upon determining an approaching violation of the at least one tool constraint.

15. 14. The teleoperation system of claim 13, wherein the teleoperation system is configured to stop operation of the tool (9) or reduce the speed of the tool (9) upon determining an approaching violation of the at least one tool constraint.

16. 10. The teleoperated system according to claim 9, wherein the teleoperated system is configured to predict a tool trajectory of the tool (9) and to project the predicted tool trajectory onto constraints associated with the tool (9).

17. the remote control system is configured to operate in an indirect control mode; In the indirect control mode, the remote control system is configured to determine and store in a data storage a control command for controlling the remote control system based on the interpreted operation command to perform the operation; The remote control system according to claim 9 , wherein the remote control system is configured to retrieve the stored control command and control the remote control system based on the retrieved control command.

18. In the indirect control mode, 18. The teleoperation system of claim 17, wherein the teleoperation system is configured to select one tool (9) or multiple tools (9) to perform the action.

19. In the indirect control mode, 18. The teleoperated system according to claim 17, wherein the teleoperated system is configured to simulate an operation of the teleoperated system, in particular to simulate in real time the effects of operating the at least one tool (9) based on the stored control instructions, and to visualize the simulated effects via the output device (4).

20. A non-transitory computer-readable medium having a computer-readable program embedded therein, the computer-readable program, when executed on a computing device, causing the computing device to: a step (S1) of outputting at least one virtual tool representation (21, 22, 23) of at least one tool (9) to an operator of the teleoperation system (1) via an output device (4); a step (S2) of receiving a selection command from the operator via an input device (4) to select one of the output at least one virtual tool representation (21, 22, 23); determining (S3) a task based on the selected virtual tool representation (21, 22, 23) by a control circuit (6); a step (S4) of displaying the selected virtual tool representation (21, 22, 23) to the operator via the output device (4); A step (S5) of receiving an operation command from the operator via the input device (4); interpreting the obtained operation command based on the determined task; controlling (S6) the teleoperation system (1) to perform an action using the at least one tool (9) based on the interpreted action command and the selected virtual tool representation (21, 22, 23); A non-transitory computer-readable medium for causing

Images