Virtual tools for assisted remote operation

Abstract

To provide a user interface for instructing a tele-operating system and the tele-operating system.SOLUTION: The user interface includes an output device configured to output at least one virtual tool representation of at least one tool to an operator of the tele-operating system, and an input device configured to obtain a selection instruction from the operator for selecting one of the at least one output virtual tool representation. The user interface further includes a control circuit configured to determine a task based on the selected virtual tool representation, control displaying the selected virtual tool representation to the operator via the output device, acquire an action instruction from the operator via the input device, interpret the acquired action instruction based on the determined task, and control the tele-operating system to perform an action using the at least one tool based on the interpreted action instruction and the selected virtual tool representation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the general field of virtual reality, and to interfaces for teleoperation systems. In particular, a user interface for a teleoperation system, a computer-implemented method for controlling a teleoperation system, and a corresponding program are disclosed. [Background Art]

[0002] In artificial intelligence (AI)-assisted teleoperation, there exists a spectrum of assistance ranging from no assistance, moderate task assistance, to full robot autonomy. Current teleoperation systems apply virtual reality (VR) or augmented reality (AR) systems to their user interfaces (UIs), and their efficiency relies on intuitive and efficient communication of assistance function settings, preferences, and constraints.

[0003] Teleoperation, sometimes referred to as remote operation, means the operation of a system or machine located at a distance. Teleoperation is an example of a human-machine system. Teleoperation systems provide a spectrum of autonomy ranging from manual control to full autopilot operation of autonomous devices (robots).

[0004] In teleoperation, the state of the art includes a robot provided with a general-purpose actuator, and an operator directly controlling the movement of the actuator. A typical example of a general-purpose actuator is a grasping manipulator. Nevertheless, in many cases direct control of the manipulator by an operator provides the movement of the manipulator, which is inefficient and inaccurate.

[0005] An alternative approach in the field of remote operation involves equipping the robot with specialized actuators that include unique tool attachments. Alternatively, the robot could be equipped with a general-purpose actuator that holds a specific tool, such as a wrench, and the operator controls that tool in a one-to-one manual control mode. While the efficiency of this alternative approach may be improved compared to using a robot equipped with a general-purpose actuator, this alternative approach requires the robot to be equipped with a specialized tool.

[0006] In practice, robots often find it difficult, or even impossible, to set constraints on physical tools such as torque wrenches.

[0007] As an alternative, robots may use specialized tools such as power drills, but these are less flexible and often far more expensive than general-purpose tools.

[0008] Generally, in AI-assisted remote control, the latest technologies burden the operator with selecting specific assistance modes performed by the robot, either explicitly from text menus or by pressing buttons or moving sliders in a VR user interface, and these modes are not intuitive to use. A concrete example is when an operator uses a dial in a VR user interface to select the final torque and rotational speed of a torque wrench.

[0009] U.S. Patent No. 9,272,418(B1) discloses a remote operator user interface that enables a robot to learn teaching capabilities. Each capability of the robot requires a set of constraints as user input from the operator. The user interface provides interface elements that assist in inputting each constraint for each capability of the robot. The operator must know what each capability does and what constraints should be given to each capability. The operator selects the desired robot capability in the user interface and sets the desired constraints for the selected capability. The necessary constraints must be pre-set in order for the intended operation to be performed successfully. [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] The described embodiments of the remote control system suggest that improved remote control systems and improved user interfaces for remote control systems are desirable.

[0011] The user interface for a remote control system according to independent claim 1, the remote control system and computer implementation method according to the corresponding independent claim address this particular problem among a variety of other problems. [Means for solving the problem]

[0012] According to a first aspect of the present disclosure, a user interface for commanding a remotely controlled system to perform an action comprises an output device configured to output at least one virtual tool representation of at least one tool to an operator of the remotely controlled system, and an input device configured to receive a selection command from the operator to select one of the output virtual tool representations. The user interface comprises a control circuit configured to determine a task based on the selected virtual tool representation, control the display of the selected virtual tool representation to the operator via the output device, and obtain an action command from the operator via the input device. The control circuit is configured to interpret the action command obtained based on the determined task and to control the remotely controlled system to perform an action using at least one tool based on the interpreted action command and the selected virtual tool representation.

[0013] According to the second embodiment, the remote control system includes a user interface according to the first embodiment.

[0014] According to a third embodiment, the computer program product is a non-temporary computer-readable medium containing a computer-readable program, wherein the computer-readable program, when executed on a computing device, causes the computing device to perform the following steps: outputting at least one virtual tool representation of at least one tool to an operator of a remote control system via an output device; receiving a selection command from the operator to select one of the outputted virtual tool representations via an input device; determining a task based on the selected virtual tool representation using a control circuit; displaying the selected virtual tool representation to the operator via an output device; receiving an operation command from the operator via an input device; interpreting the acquired operation command based on the determined task; and controlling the remote control system to perform an operation using at least one tool based on the interpreted operation command and the selected virtual tool representation, the non-temporary computer-readable medium.

[0015] The dependent claims define advantageous embodiments of a user interface and a remote control system.

[0016] The aspects and implementations of this disclosure will be described in relation to the attached drawings in the following description of specific embodiments. [Brief explanation of the drawing]

[0017] [Figure 1] This is a schematic flowchart that provides an overview of the steps involved in controlling a robot using a remote control system. [Figure 2] This is a block diagram showing the user interface for issuing commands to a remote control system and the elements of the remote control system. [Figure 3] This figure shows some examples of virtual tools in the disclosed embodiments of the remote control system. [Modes for carrying out the invention]

[0018] The diagram descriptions use the same reference numerals for the same or corresponding elements in different diagrams. Where possible, the diagram descriptions omit detailed explanations of the same reference numerals in different diagrams without negatively impacting comprehension. The diagrams and elements shown in them are not necessarily at a constant scale.

[0019] The term "virtual tool" refers to a metaphor for a tool, which is a representation of a tool (physical tool) in a user interface. A virtual tool refers to or relates to a physical tool that is well-known to a large group of potential operators. Therefore, its function and operational constraints are intuitively apparent to the operator. The tool metaphor has an iconographic representation within the user interface and is further associated with typical parameters, such as the dimensions of a cutting head, and constraints, such as restricting the circular saw to operate parallel to the work surface.

[0020] The term "physical equivalent tool (PET)" refers to an actual physical tool that corresponds to a virtual tool.

[0021] The user interface provided in this disclosure is an intuitive interface that uses a depiction of an actual tool as a metaphor to allow the operator to select the actions of a remotely operated robot that provide a desired level of control over a remotely operated process. The selection provides the operator with a choice of levels of control via the metaphorical interface. This disclosure provides an intuitive user interface in VR or AR by visualizing virtual tools such as a wrench, a torque-controlled ratchet spanner, or a power drill. By selecting these virtual tools, the operator defines their intent regarding assistance. The virtual tools are displayed and can be controlled by the operator in VR. On the physical side, the robot simulates the virtual tools using standard tools rather than exchanging tools. Thus, an operator skilled in using ordinary tools for directly manipulating objects, such as a standard wrench, can intuitively use the remote operation system to control the robot.

[0022] The disclosed user interface provides an intuitive interface that uses depictions of actual tools, in the form of virtual tool representations, as metaphors for selecting robotic actions. Unlike known methods, this user interface does not focus on information about how a drill tool must be positioned to functionally perform actions such as drilling or screwing. Instead, by looking at at least one virtual tool representation corresponding to a physical tool and selecting a particular virtual tool representation, the user selects a specific function and the level of control over its execution by the remote control system. Nevertheless, auxiliary information for function execution still needs to be obtained by the assistive system.

[0023] The selection aims to allow an operator to select a level of control via a metaphor interface. A tool metaphor is an element of user interface design for graphical user interfaces, for example, the brush metaphor or spray can metaphor in a drawing program. However, these tool metaphors are only applied for controlling digital computers, and are not applied for controlling physical interaction with objects using remote robotic technology.

[0024] According to one embodiment, the user interface comprises a control circuit configured to determine task parameters and task constraints based on a selected virtual tool representation, and to interpret an obtained motion command based on the determined task, task parameters, and task constraints.

[0025] The user interface according to one embodiment comprises an output device configured to output a plurality of virtual tool representations of a plurality of tools to an operator. An action performed and executed by the robot mimics the use of the tool corresponding to the selected virtual tool representation by using a different tool among the plurality of tools.

[0026] In one embodiment of the user interface, each virtual tool representation comprises at least one adjustable tool constraint.

[0027] The adjustable tool constraint may include at least one of a directional constraint, a size constraint, a maximum speed setting, and a maximum thickness of material removed from an object.

[0028] In one embodiment of the user interface, at least one virtual tool representation comprises an element configured to be operated by the operator to adjust the adjustable tool constraint.

[0029] A user interface according to one embodiment has at least one virtual tool representation, which includes an action element configured to be manipulated by an operator to numerically adapt adjustable tool constraints, in particular a virtual slide control or rotation control.

[0030] One embodiment of the user interface is part of a virtual reality system or an augmented reality system.

[0031] In one embodiment of the user interface, the input device includes a pointer device.

[0032] In a second aspect of this disclosure, the remote control system includes a user interface according to any one of the embodiments described above.

[0033] A remote control system according to one embodiment includes a multi-purpose physical tool configured to perform multiple actions corresponding to the operational capabilities of multiple tools.

[0034] According to one embodiment, the remote control system includes a special tool configured to perform actions corresponding to the operational capabilities of a selected virtual tool representation, and the control circuit is configured to control the special tool based on the task constraints of the selected virtual tool representation in order to perform the actions.

[0035] A remote control system according to one embodiment is configured to operate in direct control mode, and when operating in direct control mode, the control circuit is configured to determine whether controlling the remote control system based on an operation command interpreted to perform an operation would violate at least one tool constraint. If it is determined that at least one tool constraint is violated, the control circuit is configured to output feedback information about the determined violation to the operator via an output device.

[0036] A remote control system according to one embodiment is configured to determine, when operating in direct control mode, whether controlling the remote control system based on an action command interpreted to perform an action would approach a violation of at least one tool constraint.

[0037] A remote control system according to one embodiment is configured to generate and output feedback information, including a visual, audible, or tactile warning, when it detects an approach to a violation of at least one tool constraint.

[0038] According to one embodiment of a remote control system, the remote control system is configured to stop the operation of a tool or slow down the speed of a tool when it detects that it is approaching a violation of at least one tool constraint.

[0039] A remote control system according to one embodiment is configured to predict the tool trajectory of a tool and to project the predicted tool trajectory onto constraints associated with the tool.

[0040] A remote control system according to one embodiment is configured to operate in indirect control mode, in which the remote control system is configured to determine control commands for controlling the remote control system based on operation commands interpreted to perform an operation, and to store these commands in data storage. The remote control system is configured to retrieve the stored control commands and to control the remote control system based on the retrieved control commands.

[0041] A remote control system according to one embodiment is configured to select one or more tools to perform an action when operating in indirect control mode.

[0042] In one embodiment of a remote control system, the remote control system is configured to simulate the operation of the remote control system when operating in indirect control mode, in particular to simulate in real time the effect of operating at least one tool based on stored control commands, and to visualize the simulated effect via an output device.

[0043] The following description of embodiments uses a user interface output device to output multiple virtual tool representations of multiple tools to an operator. The operator then selects one of the outputted virtual tool representations. In an alternative scenario also covered by this disclosure, the user interface output device outputs one virtual tool representation of one tool to an operator of a remote control system. The operator then selects that one outputted virtual tool representation by entering a selection command.

[0044] Figure 1 shows a schematic flowchart that provides an overview of the steps for controlling the robot 10 with the remote control system 1 shown in Figure 2.

[0045] The remote control system 1 includes a user interface for instructing the remote control system 1 to perform a specific function or a single operation or a series of operations.

[0046] The user interface comprises an output device configured to output virtual tool representations of multiple tools to the operator of the remote control system, and an input device configured to receive a selection command from the operator to select one of the outputted virtual tool representations.

[0047] The user interface includes a control circuit 6 configured to determine tasks, task parameters, and task constraints based on the selected virtual tool representation. The control circuit may form part of a computing device that includes memory for storing data in a database 7. Several specific embodiments of the physical implementation are described with reference to Figure 2.

[0048] A method for controlling the user interface may be performed, for example, by a control circuit 6 which is part of a computing device, by causing the computing device to perform step S1, which outputs virtual tool representations of multiple tools 9 to the operator of the remote control system 1 via an output device.

[0049] The virtual tool representation may be associated with additional tool parameters and settings, such as the direction and magnitude of tool constraints, or the setting of the maximum rotational speed, or the maximum feed rate, or the maximum thickness of the material to be removed. The tool parameters may be visualized as variations of the virtual tool representation, for example, corresponding to larger or smaller belt sanders, or different grinding attachments for multi-purpose tools.

[0050] Additional tool constraints may be defined by the corresponding portion of the virtual tool representation that can be manipulated by the operator, for example, the size of the base plate of a virtual circular saw or the diameter of a grinder tool.

[0051] Alternatively, additional tool constraints may be numerically set by the operator using sliders or other conventional graphical user interface (GUI) elements output by the remote control system 1 via an output device.

[0052] In step S2, the operator selects one of the displayed virtual tool representations. In step S2, the control circuit 6 receives a selection command from the operator via the user interface input device to select one of the outputted virtual tool representations.

[0053] The computer implementation method then determines the task, task parameters, and task constraints based on the selected virtual tool representation in step S3, following step S2.

[0054] The task defines a goal that the robot 10 is intended to achieve in order to assist a human, in particular an operator, by performing one action, a series of actions, using at least one actuator of the robot, including, for example, tool 9 (physical tool 9).

[0055] Task parameters are generally numerical factors or similar observable factors that represent a set of conditions for performing a task in environment 13. Task parameters are quantities that have selectable values ​​for a particular situation of the task. In relation to one task parameter, other task parameters from a set of task parameters for a particular task may be set. Task parameters may include, for example, the direction and magnitude of constraints, the setting of the maximum rotational speed of the revolving tool 9, or the maximum thickness of material to be removed from the object (workpiece).

[0056] Generally, task constraints are constraints or boundary conditions for a particular task. A task may be interpreted as an ordered set of constraints that must be achieved by an actuator, particularly the tool 9 of the robot 10. In this example, task constraints are set by selecting a specific virtual tool representation from a group of virtual tools. In one example, a specific task constraint related to a grinding tool (grinder) is the diameter of the grinder.

[0057] In step S4, the computer implementation method displays the selected virtual tool representation to the operator via an output device.

[0058] In step S5, the method obtains an action command from the operator via an input device. The method then interprets the obtained action command based on the determined task parameters and task constraints.

[0059] In step S6, following step S5, the method controls the remotely operated system to perform an action using the tool (physical tool) corresponding to the selected virtual tool, based on the interpreted action command and the selected virtual tool representation.

[0060] In step S7, the method determines whether a new tool is needed. For example, the operator of remote control system 1 may try to use a new tool to continue the task. If it determines that a new tool is needed (yes), the method proceeds to step S1. The method then continues the new processing loop by repeating steps S1-S6. If it determines that a new tool is not needed (no), the method terminates its processing. Step S7 and the processing loop correspond to the optional steps of the method shown in Figure 1.

[0061] Figure 2 shows a user interface for controlling the remote control system 1 and a block diagram illustrating the elements of the remote control system 1.

[0062] The user interface may be part of a virtual reality (VR) or augmented reality (AR) system.

[0063] The remote control system 1 may operate in different control modes, including direct control mode or indirect control mode.

[0064] In direct control mode, the remote control system provides direct, one-to-one control of the actual physical end effector of the robot 10. In direct control mode, the remote control system 1 controls itself, in particular the tool 9, directly based on the determined motion commands from the operator. In direct control mode, no simulation is performed of the operation of the remote control system 1, in particular the tool control of tool 9. Control commands are not stored in the database 7 for subsequent tool control. In direct control mode, the remote control system 1 controls tool 9 online. A distinctive feature is that when operating in direct control mode, the remote control system 1 determines whether controlling itself based on the motion commands interpreted to perform the operation would approach a violation of at least one tool constraint of tool 9 corresponding to the selected virtual tool representation.

[0065] When operating in direct control mode, the remote control system 1 may determine whether the operator commands the selected virtual tool to perform an action that the physical tool 9 cannot perform due to tool constraints while the tool 9 is operating. For example, the operator commands the virtual tool representation to perform a tool trajectory that the associated physical tool cannot follow.

[0066] If such a violation of tool constraints is detected, the remote control system 1 may output a warning signal to the operator via the output device of the VR / AR interface 4. The warning signal may include a visual warning encoded as a periodic change (flashing) of the color or intensity of the visual signal, or an audio signal.

[0067] Alternatively or additionally, if the remote control system 1 determines that a violation of such tool constraints is imminent, it may automatically reduce the feed rate or speed of the physical tool 9 or stop the operation of the physical tool 9. The remote control system 1 may determine that a violation of such tool constraints is imminent if it determines that the difference between the predicted trajectory of the physical tool 9, generated based on commands received from the operator, and the constraints of the physical tool 9 is less than a threshold.

[0068] Alternatively or additionally, if the remote control system 1 determines in direct control mode that a violation of such tool constraints is at least imminent or has already occurred, it may temporarily detach the trajectory of the selected virtual tool from the operator's 1:1 control and project the operator-commanded trajectory of the virtual tool onto the tool constraints defined by the physical tool 9 associated with the selected virtual tool. For example, instead of tilting the tool 9 with respect to the surface of the object (workpiece), the remote control system 1 may maintain the path of the tool 9 perpendicular to the surface.

[0069] The direct control mode is advantageous in scenarios and applications of the robot 10 that require the ability to perform intended actions at least in near real-time. As a specific example, the robot 10 may be equipped with a tool 9 with a sander large enough to simulate a virtual belt sander, for example, when selected as a virtual tool representation by an operator.

[0070] In indirect control mode, the remote control system 1 determines control commands to control the remote control system 1 based on the operation commands received from the operator and interpreted to perform the operation, and stores them in the database 7. The remote control system 1 retrieves the stored control commands from the database 7 and controls the remote control system 1 based on the retrieved control commands. Indirect control mode is particularly suitable for applications where the selected virtual tool has higher capabilities than the physical tools 9 available to the robot 10. This is the case, for example, when the virtual tool representation corresponds to a belt sander, and the robot 10 simulates the selected virtual tool by using a small rotary sander as the physical tool 9.

[0071] In indirect control mode, the remote control system may select one or more tools 9 to perform an action specified in an action command received from the operator via the AR / VR interface 4. The remote control system 1 may select the most appropriate physical tool to perform the task commanded by the operator by selecting a virtual tool representation. The most appropriate physical tool 9 may be the physical tool 9 having a set of tool constraints that is determined to be closest to the tool constraints of the selected virtual tool.

[0072] As an addition or alternative, while operating in indirect control mode, the remote control system 1 may provide assistance by simulating the commanded action or sequence of actions for operation, for example, by simulating in real time the effect of operating the tool 9 based on the command, and visualize the results of the simulation to the operator, for example, via an output device.

[0073] The remote control system 1 includes an augmented reality / virtual reality interface 4 (AR / VR interface 4), a control circuit 6, and a database 7 stored in a memory configured for storing the database.

[0074] The AR / VR interface 4 may operate as an augmented reality (AR) interface that seamlessly blends computer-generated content, such as additional computer-generated image data, that utilizes one or more of the operator's senses into the real world within the environment perceived by the operator's senses. The AR interface enhances the operator's perception of the environment by superimposing additional computer-generated information. In this disclosure, the control circuit 6 generates virtual tool representations of multiple tools and outputs them to the operator of the remote control system 1, and the multiple virtual tool representations may be presented to the operator as additional information superimposed on an optical head-mounted display.

[0075] Alternatively, the AR / VR interface 4 may operate as a virtual reality (VR) interface that replaces the real environment with an artificial environment, which the operator experiences through computer-provided sensory stimuli. Actions taken by the operator determine, at least partially, what happens within the environment. The VR interface may include hardware for tracking the operator's posture, as well as a three-dimensional (3D) near-eye display of images and videos.

[0076] The AR / VR interface 4 may include an output device for outputting output information 2 to the user (operator) and an input device for obtaining input information 3 from the operator. The output device of the AR / VR interface 4 may include a display, such as a computer monitor, or an image projector that projects images onto a surface, in order to visually display the information to the operator.

[0077] The display device may include a wearable device, such as an optical head-mounted display that displays information in a hands-free manner.

[0078] The output device outputs virtual tool representations, particularly those corresponding to multiple tools 9, to the operator of the remote control system 1.

[0079] The output device, in particular, displays the selected virtual tool representation to the operator.

[0080] The output device may further output feedback information if the remote control system 1 determines that a violation of at least one tool constraint is likely to occur.

[0081] The control circuit 6 may determine whether controlling the remote control system 1 based on an operation command interpreted to perform an operation would violate at least one tool constraint of the tool 9 when the remote control system is operating in direct control mode.

[0082] If the control circuit 6 determines that at least one tool constraint has been violated, the control circuit 6 is configured to output feedback information about the determined violation to the operator via the output device of the AR / VR interface 4. The output feedback information may include a warning about the determined violation of at least one tool constraint.

[0083] When the remote control system 1 is operating in indirect control mode, the remote control system 1 simulates its operation via the output device, in particular, simulating in real time the effect of operating the tool 9 based on stored control commands, and visualizes the simulated effect via the output device.

[0084] The AR / VR interface 4 may include an input device for receiving a selection command from the operator to select one of the outputted virtual tool representations presented to the operator.

[0085] The input device may include any device suitable for providing information, data, or control signals to an information processing system, such as a computer, and may include at least one of a keyboard, mouse, joystick, or microphone.

[0086] In detail, the input device may include a pointing device (pointer).

[0087] Alternatively or additionally, the input device may include a gesture tracking system for tracking the operator's gestures.

[0088] The control circuit 6 may include at least one processor, signal processor, microprocessor, microcontroller (μC), application-specific integrated circuit (ASIC), system-on-a-chip (SoC), graphics processing unit (GPU), or a combination of such integrated circuits (ICs).

[0089] The control circuit 6 may be implemented in a distributed manner and may include, for example, an AR / VR interface 4, data processing resources of the robot 10 or one or more servers located remotely from the robot 10 and connected via a communication network not explicitly shown in Figure 2.

[0090] The remote control system 1 further includes at least one sensor 5 for sensing the physical environment 13 in which the remote control system 1 is performing a task using a tool 9 (physical tool 9).

[0091] Sensor 5 may include a plurality of sensors that form a sensor suite for acquiring sensor information 11 about the environment 13. Sensor 5 may include a camera that acquires images and videos from the environment 13 in which the robot 10 is operating, in particular under the control of the remote control system 1, while performing one action, a series of actions, or a function.

[0092] The remote control system 1 includes a tool control interface 8, which receives control signals from a control circuit 6 and controls a tool 9 performing one action 12, a series of actions, or a function on at least one physical object in the environment 13.

[0093] The tool control interface 8 and the tool 9 may form part of a device 10 (robot 10) that operates autonomously or semi-autonomously. The tool control interface 8 generates and outputs control signals for controlling at least one tool 9 that functions within the environment 13, for example, by performing one or a series of actions on at least one object in the environment 13.

[0094] Preferably, tool 9 is a multifunctional tool capable of performing several individual functions under the control of tool control interface 8.

[0095] The physical tools 9 available to the robot 10 may include the arms of the robot 10, which include end effectors (manipulators) positioned at the ends of the arms. Therefore, an end effector and arm, including a gripper, for example, corresponds to tool 9 in this scenario. This application scenario may include the task of tightening a screw, for example, by using a gripper and the arms of the robot 10 as respective physical tools 9. The operator may select a virtual representation of a torque wrench displayed by an output device. Selecting a virtual representation of a torque wrench gives the operator the possibility of specifying a desired torque for tightening the screw. The arms and end effectors of the robot 10 include built-in force torque sensors, and the physical tools 9 then perform the commanded action of tightening the screw with a desired torque as the respective task parameter.

[0096] The robot 10 may use general-purpose or multi-purpose actuators to perform operations indirectly, for example, more slowly or repetitively than the physical tool 9 corresponding to the selected virtual tool used in the virtual tool representation.

[0097] Alternatively or additionally, the robot 10 may use a special tool, such as a sander, for its operation, but instead of using the physical guides used by the tool corresponding to the virtual tool representation, the robot actuator's control software ensures that task constraints set for the selected virtual tool representation are maintained. For example, the flat surface of a belt sander as a physical tool 9 ensures parallel or flat application to the surface of an object that functions as a workpiece.

[0098] Sensor 5 may be part of a device 10 (robot 10) that operates autonomously or semi-autonomously.

[0099] Robot 10 may be a service robot that assists humans in performing tasks in cumbersome, dirty, dangerous, repetitive, or harsh environments. Robot 10 has a degree of autonomy, which, according to common convention, means "the ability to perform an intended task without human intervention, based on its current state and perceptions." The degree of autonomy can range from partial autonomy, which involves human-robot interaction, to full autonomy, which involves no active human-robot intervention by an operator.

[0100] In AI-assisted remote operation, the level of assistance ranges from no assistance to moderate task assistance and then to complete autonomy. User interfaces, including VR / AR interfaces, enable intuitive and efficient communication of settings, preferences, and constraints for the assistance functions provided by the robot 10.

[0101] The operator of the remote control system 1 intends to tighten a bolt. Depending on the specific task, the operator may intend to apply fine-grained control to the rotation angle of the bolt, for example, 2.5 turns, or to rotate the bolt at a specified speed and duration, or to tighten the bolt to a predetermined torque value.

[0102] In the real world, where the remotely operated robot 10 is not used, the user would choose the following appropriate means to achieve these three goals.

[0103] The user may manually turn the bolt or use a wrench to directly control its movement;

[0104] Alternatively, the user may use a power drill and control the rotation speed and duration by pressing a button on the power drill; or

[0105] Alternatively, the user may use a spanning tool equipped with a torque limiter to efficiently rotate the tool up to the specified final torque.

[0106] In all three alternative scenarios, the appropriate solution is an efficient choice that includes the right level of control, high precision in achieving the objective, and a well-known user interface for setting constraints, such as a torque setting dial. The user interface and remote control system 1 replicate the efficient and intuitive interface of the tool to control the remote control system 1 and the robot 10 without actually requiring the physical tool. The use of the tool's well-known and intuitive interface can reduce the training time required for the operator to use the remote control system.

[0107] Figure 3 shows several examples of virtual tool representations and tool 9 suitable for the disclosed embodiments of the remote control system.

[0108] This disclosure provides an intuitive user interface, particularly advantageous for virtual reality applications, by visualizing virtual tools such as wrenches, torque-controlled ratchet spanners, or power drills. The operator communicates their intentions regarding artificial intelligence assistance through the selected virtual tool by selecting the virtual tool. The user interface displays the virtual tool representation to the operator, allowing the operator to control the selected virtual tool representation within the virtual environment. On the physical side, in the physical environment, the robot 10 simulates the selected virtual tool using a standard tool 9 for operation in the physical environment, rather than replacing its own tool 9 with a physical tool 9 corresponding to the selected virtual tool representation.

[0109] Figure 3 shows three examples of the corresponding robot 10 movements in a virtual tool representation and a physical environment.

[0110] The processes of "grinding," "polishing," and "routing" use the respective virtual tool attachments on the virtual rotary tool 21. The virtual tool attachments may include spherical polishing attachments 21.1, 21.2, 21.3, 21.4, or cylindrical polishing attachments, each having a specified diameter for the multipurpose rotary tool 21. The robot performs actions to apply the physical tool 9 corresponding to the selected virtual rotary tool 21, directly following the trajectory performed by the operator, and may perform specific cutting operations, for example, taking into account the groove depth and width defined by the virtual tool attachment with a specified diameter selected by the operator for the multipurpose rotary tool 21.

[0111] The operator's selection of a virtual tool representation of the belt sander 22 defines a process for creating a planar area on the surface of the part (workpiece), thereby obtaining a locally flat surface. The operator may control the roughness of the resulting flat surface generated on the physical part by the robot via the user interface by selecting a virtual abrasive belt 22.1 for each virtual belt sander 22.

[0112] Selecting a virtual tool, such as a router with a specific tool bit 23, controls the robot to create a cut section of a predetermined shape and depth on the part. The cut section may be, for example, flat with a 45-degree angle, or a quarter-circle with a defined radius, following a trajectory performed by the operator. The virtual trajectory of the virtual tool is constrained to be perpendicular to the surface, as would be the case with a handheld router. The operator may set additional constraints by adjusting a virtual "ball bearing bit" that defines an adjustable distance to the work surface at a height specific to the tool bit 23. The robot 10 performs the cutting operation on the part based on the set distance.

[0113] The example in Figure 3 is merely one of several illustrative combinations of virtual tools and their corresponding actions. A series of further examples given below, though not limited to them, include the process of driving nails.

[0114] The nailing process includes, for example, a virtual tool called a "nail gun." The corresponding physical actions performed by the robot 10 include setting the nail at a designated position on the surface of the object and inserting the nail into the object until the nail head is level with the surface of the object.

[0115] Alternatively or additionally, the nailing process may include a virtual tool, for example, a "hammer." The corresponding physical action performed by the robot 10 includes setting the nail at a designated position on the surface of the object and gradually pushing or pressing the nail into the object at a feed rate corresponding to at least one of, for example, the size of the selected virtual tool "hammer," the speed of the virtual hammering gesture performed by the operator, and the amplitude of the virtual hammering gesture performed by the operator. The direction of the hammering gesture corresponds to the direction of the nail relative to the surface of the object. The final height of the nail head emerging from the surface of the object can vary between zero, meaning the nail head is at the same height as the surface, and a value corresponding to the total length of the nail.

[0116] The clamping process includes, for example, a virtual tool called a "clamp." The corresponding physical action performed by the robot 10 includes applying pressure with a specific (maximum) force at a designated location and in a specific direction on the surface of the object.

[0117] Alternatively or additionally, the clamping process may include a virtual tool, for example, a "corner clamp." The corresponding clamping process involves constraining two parts to maintain a specific angle between them, the angle of which may be set by the operator via a user interface.

[0118] The cutting process includes, for example, a virtual tool such as a "knife." The corresponding physical action performed by the robot 10 includes cutting the surface of an object freehand, essentially without constraints, along a trajectory, which is performed by an operator, for example, using a pointing device on a user interface.

[0119] Alternatively or additionally, the cutting process may include a virtual tool, for example, a “peeling knife.” The corresponding cutting process includes free cutting to the surface of the workpiece, constrained to remove a thin layer of material from the surface of the workpiece. The operator may set, via a user interface, at least one of the depth of material that the robot 10 removes from the workpiece and the size of the thin layer.

[0120] Alternatively or additionally, the cutting process may include, for example, a virtual tool such as a "circular saw." The corresponding cutting process involves selecting a virtual tool representation of a "hedge saw" and then creating a cut section perpendicular to the surface of the workpiece, which is otherwise unconstrained.

[0121] Alternatively or additionally, the cutting process may include a virtual tool such as a "scroll saw." The corresponding cutting process involves selecting a virtual tool representation of a "circular saw" and then creating a cut perpendicular to the surface of the workpiece, according to the further task constraint of a straight line.

[0122] The baking process includes, for example, a virtual tool such as a "rolling pin." The corresponding physical action performed by the robot 10 includes flattening the dough to a specific height that can be adjusted by the operator via a user interface.

[0123] Alternatively or additionally, the baking process may include a virtual tool, for example, a "cookie cutter." The corresponding physical process involves the operator selecting the virtual tool representation of the "cookie cutter" and then cutting out a specific shape from a flat piece of dough spread on a flat surface.

[0124] The painting process involves selecting a virtual tool from a set of virtual tools, including instances such as a "brush," "paint roller," "sprayer," "pencil," and "crayon." The corresponding physical actions performed by the robot 10 involve physically painting or drawing on the surface according to the trajectory performed by the operator or based on specific settings made by the operator via the user interface. Unlike computer graphics software with brushes in paint software that uses virtual brushes to change the screen display in a virtual environment, the robot 10 actually paints or draws the results of each of the operator's controls in a physical environment.

[0125] The greasing process includes, for example, various versions or modifications of virtual tools such as an "oil dripper," a "grease gun," and a "grease brush." ​​The corresponding physical actions performed by the robot 10 include at least one of applying grease of a specific consistency, applying a specific amount of grease, and applying grease to a surface area or grease nipple of an object with a specific pressure.

[0126] The sorting process includes, for example, a virtual tool of a "sieve" in various versions, indicated by sieve inserts having respective diameters for the sieve openings. The corresponding physical actions performed by the robot 10 include sorting the objects by their respective sizes, in particular, into a first group of objects whose diameter is smaller than a selected size (size threshold) and a second group of objects whose diameter is larger than the selected size.

[0127] Other processes performed by the robot 10, which can be controlled by the operator via a user interface, include cleaning with a vacuum cleaner. Selecting a virtual tool for the "vacuum cleaner" may include selecting one specific virtual attachment tool from a group of virtual attachment tools presented to the operator via the user interface. The robot 10 may perform the cleaning process by adjusting process parameters such as the operating range and vacuum strength of the cleaning process based on the selected virtual attachment tool.

[0128] The examples described above are particularly derived from workshop or home environments in which the robot 10 is used for its respective remote operation applications. The applications of the described remote operation system 1 include further application areas, such as disaster recovery robots and underwater repair or installation by autonomous devices. The remote operation system may also operate in extraterrestrial use scenarios, such as in outer space. The method may prove advantageous in remote work applications in workshops with mechanical or electrical systems, or at customer sites on-site. Application areas include household chores and maintenance.

[0129] All features described above or shown in the figures may be combined with each other in any advantageous way within the scope of this disclosure. The detailed description of the embodiments has provided numerous specific details to offer a complete understanding of the invention as defined in the claims. It is evident that the claimed invention can be carried out without including all of these specific details.

[0130] Since they are used interchangeably in the specification and claims, the expression "at least one of A and B" can be used as a substitute for the expression "A and / or B," and vice versa. The expression "A and / or B" means "A, or B, or A and B."

Claims

1. A user interface for commanding a remote control system (1) to perform an action, An output device (4) configured to output multiple virtual tool representations (21, 22, 23) of multiple tools (9) to the operator of the remote control system (1), An input device (4) configured to receive a selection command from the operator to select one of the multiple virtual tool representations (21, 22, 23) that have been output, Control circuit (6), Determine the task for the selected virtual tool representations (21, 22, 23), The output device (4) controls the display of the selected virtual tool representations (21, 22, 23) to the operator. The operator receives an operation command via the input device (4), Interpret the acquired operation command based on the determined task, Based on the interpreted operation commands and the selected virtual tool representations (21, 22, 23), the remote control system (1) is controlled to perform operations using at least one of the multiple tools (9). The control circuit (6) is configured as follows, The operation involves using a tool (9) corresponding to the selected virtual tool representation (21, 22, 23) in an approximate manner using a different tool (9) from among the multiple tools (9), wherein the other tool (9) is one that emulates the tool (9) corresponding to the selected virtual tool representation (21, 22, 23). User interface.

2. The aforementioned control circuit Based on the selected virtual tool representations (21, 22, 23), task parameters and task constraints are determined. Based on the determined task, task parameters, and task constraints, the acquired operation instructions are interpreted. The user interface according to claim 1, configured as described above.

3. Each of the plurality of virtual tool representations (21, 22, 23) comprises at least one adjustable tool constraint, The user interface according to claim 1, wherein the adjustable tool constraints include at least one of a directional constraint, a size constraint, a maximum speed setting, and a maximum thickness of material removed from the object.

4. The user interface according to claim 3, wherein the plurality of virtual tool representations (21, 22, 23) include elements configured to be manipulated by the operator to conform to the adjustable tool constraints.

5. The user interface according to claim 3, wherein the plurality of virtual tool representations (21, 22, 23) include operating elements, particularly virtual slide controls or rotation controls, configured to be operated by the operator to numerically fit the adjustable tool constraints.

6. The user interface according to claim 1, wherein the user interface is part of a virtual reality system or an augmented reality system.

7. The user interface according to claim 1, wherein the input device (4) includes a pointer device.

8. A remote control system including a user interface, wherein the user interface is An output device (4) configured to output multiple virtual tool representations (21, 22, 23) of multiple tools (9) to the operator of the remote control system, An input device (4) configured to receive a selection command from the operator to select one of the multiple virtual tool representations (21, 22, 23) that have been output, Control circuit (6), Determine the task for the selected virtual tool representations (21, 22, 23), The output device (4) controls the display of the selected virtual tool representations (21, 22, 23) to the operator. The operator receives an operation command via the input device (4), Interpret the acquired operation command based on the determined task, Based on the interpreted operation commands and the selected virtual tool representations (21, 22, 23), the remote control system is controlled to perform operations using at least one of the multiple tools (9). The control circuit (6) is configured as follows, The operation involves using a tool (9) corresponding to the selected virtual tool representation (21, 22, 23) in an approximate manner using a different tool (9) from among the multiple tools (9), wherein the other tool (9) is one that emulates the tool (9) corresponding to the selected virtual tool representation (21, 22, 23). Remote control system.

9. The remote control system according to claim 8, wherein the remote control system includes a multipurpose physical tool (9) configured to perform a plurality of operations corresponding to the operational capabilities of a plurality of tools (9).

10. The remote control system includes a special tool (9) configured to perform operations corresponding to the operational capabilities of the selected virtual tool representations (21, 22, 23), The remote control system according to claim 8, wherein the control circuit (6) is configured to control the special tool (9) based on the task constraints of the selected virtual tool representations (21, 22, 23) in order to perform the operation.

11. The remote control system is configured to operate in direct control mode. In the direct control mode, the control circuit (6) is configured to determine whether controlling the remote control system based on the interpreted operation command to perform the operation would violate at least one tool constraint. The remote control system according to claim 8, wherein if it determines that at least one of the tool constraints has been violated, the control circuit (6) is configured to output feedback information about the determined violation to the operator via the output device (4).

12. The remote control system is configured to operate in direct control mode. The remote control system according to claim 8, wherein in the direct control mode, the remote control system is configured to determine whether controlling the remote control system based on the interpreted operation command to perform the operation would bring it close to violating at least one tool constraint.

13. The remote control system according to claim 12, wherein when the remote control system determines that it is approaching a violation of the at least one tool constraint, it is configured to generate and output feedback information including a visual, audible, or tactile warning.

14. The remote control system according to claim 12, wherein when the remote control system determines that it is approaching a violation of at least one tool constraint, it is configured to stop the operation of the tool (9) or to reduce the speed of the tool (9).

15. The remote control system according to claim 8, wherein the remote control system is configured to predict the tool trajectory of the tool (9) and to project the predicted tool trajectory onto constraints associated with the tool (9).

16. The remote control system is configured to operate in indirect control mode. In the indirect control mode, the remote control system is configured to determine and store in data storage control commands for controlling the remote control system based on the interpreted operation commands in order to perform the operation. The remote control system according to claim 8, wherein the remote control system is configured to acquire the stored control commands and to control the remote control system based on the acquired control commands.

17. In the indirect control mode, The remote control system according to claim 16, wherein the remote control system is configured to select one or more tools (9) to perform the operation.

18. In the indirect control mode, The remote control system according to claim 16, wherein the remote control system is configured to simulate the operation of the remote control system, in particular to simulate in real time the effect of operating the at least one tool (9) based on the stored control commands, and to visualize the simulated effect via the output device (4).

19. A non-temporary computer-readable medium containing a computer-readable program, wherein, when executed on a computing device, the computer-readable program is transmitted to the computing device. Step (S1) of outputting multiple virtual tool representations (21, 22, 23) of multiple tools (9) to the operator of the remote control system (1) via the output device (4), Step (S2) involves obtaining a selection command from the operator via the input device (4) to select one of the multiple virtual tool representations (21, 22, 23) that have been output, The control circuit (6) performs the step (S3) of determining a task based on the selected virtual tool representations (21, 22, 23), Step (S4) of displaying the selected virtual tool representations (21, 22, 23) to the operator via the output device (4), Step (S5) of obtaining an operation command from the operator via the input device (4), A step of interpreting the acquired operation command based on the task determined above, Step (S6) of controlling the remote control system (1) to perform an operation using at least one of the multiple tools (9) based on the interpreted operation command and the selected virtual tool representation (21, 22, 23), Have them do it, The operation involves using a tool (9) corresponding to the selected virtual tool representation (21, 22, 23) in an approximate manner using a different tool (9) from among the multiple tools (9), wherein the other tool (9) is one that emulates the tool (9) corresponding to the selected virtual tool representation (21, 22, 23). A non-temporary computer-readable medium.

Images