Robot remote operation system, robot remote operation method and storage medium

Abstract

Provided are a robot remote operation system, a robot remote operation method, and a storage medium capable of instructing movement or rotation of a split robot by a simple operation. A robot remote operation system includes: a control device that controls a robot; and an HMI utilized by an operator when the robot is remotely operated. The HMI displays an image around the robot on a display, detects a line of sight of the operator using an eye tracker, estimates an operation intention of the operator, and transmits line of sight data indicating the detected line of sight and the like and operation data indicating the estimated operation intention of the operator to the control device. The control device determines a target position of the robot on the basis of the line-of-sight data and the operation data received from the HMI, and moves the robot to the target position.

Description

Technical Field

[0001] This invention relates to a robot remote operating system, a robot remote operation method, and a storage medium. Background Technology

[0002] The following technology is known: it remotely operates a robot by drawing a movement path by handwriting on a tablet terminal (see, for example, Patent Document 1). When instructing the movement location of the clone robot, a mouse or a VR (Virtual Reality) glasses controller is used to click on the location in the external image obtained from the clone robot.

[0003] Prior art literature

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2023-156710 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] The idea is to enable the avatar robot to independently understand its surroundings and automatically move to the location instructed by the operator while avoiding obstacles. However, even with such a mechanism, users can easily become bored with operating a mouse or VR headset controller, or feel stressed before becoming familiar with the controls. This stress is especially pronounced when operating multiple buttons or levers.

[0008] This invention was made in consideration of such circumstances, and one of its objectives is to provide a robot remote operating system, robot remote operation method, and storage medium that can instruct the movement or rotation of a clone robot through simple operation, thereby reducing the stress of operating the clone robot and creating the feeling of being able to control the clone robot according to the operator's will.

[0009] Solution for solving the problem

[0010] The robot remote operating system, robot remote operation method, and storage medium involved in this invention adopt the following structure.

[0011] (1) A first example of the present invention relates to a robot remote operating system, wherein the robot remote operating system comprises: a control device for controlling a robot; and a human-machine interface used by an operator when remotely operating the robot, the human-machine interface comprising: a first communication interface for communicating with the control device; a display for displaying images of the robot's surroundings captured by a camera; an eye tracker for detecting the operator's gaze and / or viewpoint while the operator visually recognizes the display showing the images; an estimation unit for estimating the operator's operating intention; and a first processing unit for processing gaze data. The eye-tracking data and operation data are sent to the control device via the first communication interface. The eye-tracking data represents the eye-tracking data and / or the viewpoint detected by the eye tracker. The operation data represents the operation intention of the operator as estimated by the estimation unit. The control device has: a second communication interface that communicates with the human-machine interface; and a second processing unit that, upon receiving the eye-tracking data and the operation data from the human-machine interface via the second communication interface, determines the target position of the robot based on the eye-tracking data and the operation data, and moves the robot to the target position.

[0012] (2) In the second example of the present invention, based on the first example, the second processing unit determines a first target position related to the travel direction of the robot and a second target position related to the rotation direction of the robot as the target position.

[0013] (3) In the third example of the present invention, based on the first or second example, the human-machine interface further has an input interface for inputting the operator's operation, and the second processing unit determines the position of the operator's viewpoint detected by the eye tracker as the target position of the robot when the operator inputs a specified operation to the input interface while visually recognizing the display showing the image.

[0014] (4) In the fourth example of the present invention, based on the third example, the second processing unit continuously updates the target position of the robot by linking with the position of the operator's viewpoint while inputting the prescribed operation to the input interface.

[0015] (5) The fifth example of the present invention is based on the third example, wherein the input interface includes a voice user interface or a switch interface, and the specified operation refers to inputting a specified speech into the voice user interface or operating the switch interface.

[0016] (6) In the sixth example of the present invention, based on the first or second example, the second processing unit determines the target position of the robot based on the change in the amount of movement of the operator's viewpoint detected by the eye tracker.

[0017] (7) In the seventh example of the present invention, based on the sixth example, the second processing unit determines the position where the change in the amount of movement of the viewpoint converges as the target position of the robot.

[0018] (8) An eighth embodiment of the present invention relates to a method for remotely operating a robot, which utilizes a control device for controlling a robot and a human-machine interface used by an operator when remotely operating the robot. The robot remote operation method causes the human-machine interface to perform the following processes: communicating with the control device; displaying an image of the robot's surroundings captured by a camera on a display; detecting the operator's gaze and / or viewpoint while the operator is visually recognizing the display showing the image; estimating the operator's operational intention; and sending gaze data and operational data to the control device, wherein the gaze data represents the detected gaze and / or viewpoint, and the operational data represents the estimated operational intention of the operator. The robot remote operation method causes the control device to perform the following processes: communicating with the human-machine interface; upon receiving the gaze data and operational data from the human-machine interface, determining a target position for the robot based on the gaze data and operational data; and moving the robot to the target position.

[0019] (9) A ninth embodiment of the present invention relates to a storage medium storing a program for causing a control device for controlling a robot and a human-machine interface used by an operator when remotely operating the robot to perform processing, wherein the program causes the human-machine interface to perform the following processing: communicating with the control device; displaying an image of the robot's surroundings captured by a camera on a display screen; detecting the operator's gaze and / or viewpoint while the operator is visually recognizing the display screen showing the image; estimating the operator's operational intention; and sending gaze data and operational data to the control device, wherein the gaze data is data representing the detected gaze and / or viewpoint, and the operational data is data representing the estimated operational intention of the operator, and the program causes the control device to perform the following processing: communicating with the human-machine interface; determining a target position of the robot based on the gaze data and the operational data upon receiving the gaze data and the operational data from the human-machine interface; and moving the robot to the target position.

[0020] [Invention Effects]

[0021] Based on the above examples, the movement or rotation of the clone robot can be instructed through simple operations, which can reduce the stress of operating the clone robot and create the feeling that the clone robot can be controlled according to the operator's will. Attached Figure Description

[0022] Figure 1 This diagram illustrates an example of the use of the robot remote operating system according to the first embodiment.

[0023] Figure 2 This is a functional structure diagram of the robot and control device involved in the first embodiment.

[0024] Figure 3 This is a functional structure diagram of the HMI involved in the first embodiment.

[0025] Figure 4 This is a flowchart illustrating a series of processes of the robot remote operating system involved in the first embodiment.

[0026] Figure 5 This diagram illustrates an example of what is shown from the operator's viewpoint on a wearable display when there are movement instructions related to the robot's direction of travel.

[0027] Figure 6 This is a diagram illustrating an example of the method for determining the location of the first target.

[0028] Figure 7 This diagram illustrates an example of what is shown from the operator's viewpoint on a wearable display when there is a movement indication related to the robot's rotation direction.

[0029] Figure 8 This is a diagram illustrating an example of a method for determining the location of a second target.

[0030] Figure 9 This is a flowchart illustrating a series of processes of the robot remote operating system involved in the second embodiment.

[0031] Figure 10 This is a diagram illustrating an example of the method for determining the location of the first target.

[0032] Figure 11 This is a diagram illustrating an example of a method for determining the location of a second target.

[0033] Figure 12 This is a diagram illustrating another example of the method for determining the location of the second target.

[0034] Figure 13 This is a diagram illustrating another example of the method for determining the location of the second target.

[0035] Explanation of reference numerals in the attached figures:

[0036] 1…Robot remote operating system, 2…Robot, 21…Camera, 22…Arm, 23…Hand, 24…Actuator, 25…Status sensor, 26…Processing unit, 100…Control device, 110…Communication interface, 120…Storage unit, 130…Processing unit, 132…Acquisition unit, 134…Decision unit, 136…Communication control unit, 200…HMI, 211…Communication interface, 212…Input interface, 213…Wearable display, 214…Eye tracking sensor, 215…Motion tracking sensor, 216…Storage unit, 220…Processing unit, 221…Acquisition unit, 222…Display control unit, 223…Communication control unit, NW…Network. Detailed Implementation

[0037] Hereinafter, embodiments of the robot remote operating system, robot remote operation method, and storage medium of the present invention will be described with reference to the accompanying drawings.

[0038] <First Implementation>

[0039] [Application Scenarios of Robot Remote Operating Systems]

[0040] Figure 1 This diagram illustrates an example of the application of the robot remote operating system 1 according to the first embodiment. For example... Figure 1 Thus, the robot remote operating system 1 includes, for example, a clone robot 2 (hereinafter referred to as robot 2) that is remotely operated by an operator (OP), a control device 100 for controlling robot 2, and an HMI (Human Machine Interface) 200 used by the operator (OP). They are connected via a network (NW). The network (NW) includes LAN (Local Area Network), WAN (Wide Area Network), etc.

[0041] Robot 2 may include, for example, a camera 21, an arm 22, a hand 23 (also known as a robotic arm or end effector), a body 30, wheels 31, and a head 32.

[0042] Robot 2 moves via wheels 31. On the body 30, for example at one end of an arm 22 mounted via joints on both sides, a cylindrical support extending upwards is provided on the upper surface. A head 32 is provided at the end of the support. A camera 21 is mounted on the head 32, for example. A hand 23 is mounted via joints to the other end of the arm 22. The hand 23 can be a multi-fingered hand with, for example, three or more fingers.

[0043] Camera 21 captures images of the external environment or workspace as observed by robot 2. For example, camera 21 may be a stereo camera. Camera 21 may capture images of a scene where an object is held or manipulated by hand 23, and send the image data of the scene to control device 100, or via control device 100 to HMI 200.

[0044] It should be noted that robot 2 is not limited to robots that move by wheels 31. It can also be a robot that moves by tracks, a robot that moves by bipedal walking, a robot that moves by quadrupedal walking, or a robot that moves by flying, such as a drone.

[0045] The control device 100 is typically mounted on the robot 2 and controls the robot 2 based on operating instructions received from the HMI 200. Alternatively, the control device 100 can be located at a location far from the robot 2 instead of being mounted on the robot 2, and remotely control the robot 2 via a network (NW).

[0046] HMI200 is, for example, an interface for remotely operating robot 2. For example, HMI200 includes an input interface 212 operated by operator OP, a wearable display 213 (also called a wearable device) worn by operator OP, etc.

[0047] Input interface 212 is used as a controller to move the robot 2 forward, backward, or rotate. Alternatively, input interface 212 can also be used as a controller to move the robot 2's hand 23 or arm 22. Input interface 212 can be a controller equipped with a switch interface 212a, such as a switch, button, or joystick, as shown in the figure. For example, the operator (OP) remotely operates the robot 2 by manually operating input interface 212 while checking the image displayed on wearable display 213.

[0048] Wearable display 213 is a display used to display still images and moving images (videos) captured by a camera 21 or similar device mounted on robot 2 under remote operation. For example, wearable display 213 can be glasses, headsets, head-mounted displays, smart glasses, etc., worn on the head. Wearable display 213 can be, for example, a device using VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), projection mapping, or other technologies. Wearable display 213 is an example of a "display".

[0049] [Functional Structure of Robot Remote Operating System]

[0050] Figure 2This is a functional structure diagram of the robot 2 and control device 100 according to the first embodiment. In addition to the camera 21, arm 22, and hand 23 mentioned above, the robot 2 also includes an actuator 24, a status sensor 25, and a processing unit 26.

[0051] The actuator 24 is controlled by the processing unit 26 to drive various parts of the robot 2 (arm 22, fingers F of hand 23, etc.). The actuator 24 includes, for example, an electromagnetic motor, gears, artificial muscles, etc.

[0052] The state sensor 25 is a sensor that detects the state of the robot 2 (e.g., joint angles, angular velocities, torque, etc.). The state sensor 25 may include, for example, a rotary encoder that detects the degree of rotation of the joints of the robot 2 and the hand 23, a tension sensor that detects the tension of the wire used to rotate the joints, a torque sensor that detects the torque applied to the joint axis, an accelerometer sensor that detects the posture of the robot 2, a gyroscope sensor, etc.

[0053] The processing unit 26 controls the actuator 24 based on information output from the control device 100. The processing unit 26 can be implemented by a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or the like. Alternatively, the processing unit 26 can be implemented by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or SOC (System on Chip), or it can be implemented through the coordinated cooperation of software and hardware.

[0054] The control device 100 includes, for example, a communication interface 110, a storage unit 120, and a processing unit 130.

[0055] Communication interface 110 communicates with external devices via a network NW, or with robot 2 via a communication line such as a bus. External devices may include, for example, an HMI 200. Communication interface 110 may include, for example, a wireless communication module including a receiver and transmitter, or a NIC (Network Interface Card). Communication interface 110 is an example of a "second communication interface".

[0056] Processing unit 130 may include, for example, an acquisition unit 132, a decision unit 134, and a communication control unit 136. Processing unit 130 is an example of a "second processing unit".

[0057] The aforementioned components of the processing unit 130 are implemented, for example, by executing programs or instructions stored in the storage unit 120 through a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). Some or all of these components may also be implemented by hardware such as an LSI, ASIC, or FPGA, or through the coordinated use of software and hardware.

[0058] The storage unit 120 may be implemented using, for example, an HDD (Hard Disc Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The storage unit 120 stores firmware, application programs, and the calculation results of the processing unit 130.

[0059] The acquisition unit 132 acquires image data from the robot 2 and HMI 200 via the communication interface 110, or acquires operation data, eye-tracking data, and motion tracking data from HMI 200.

[0060] Operation data is data that indicates what kind of operation exists on input interface 212 (i.e., data that indicates the content of the operation). In other words, operation data is data that indicates the operator's (OP's) intention to operate robot 2.

[0061] Eye-tracking data represents the gaze and / or viewpoint of an operator (OP) detected by eye-tracking sensor 214. Eye-tracking data is an example of "gaze data".

[0062] Motion tracking data represents the movement of the operator's head and neck detected by the motion tracking sensor 215.

[0063] Image data refers to images of the external environment or workspace observed by robot 2, which are visualized by camera 21 of robot 2. Image data can be not only images (moving images) but also still images.

[0064] The decision unit 134 determines the target position (hereinafter referred to as the target position) when the robot 2 moves forward or backward or rotates, based on the operation data and eye-tracking data obtained by the acquisition unit 132. Hereinafter, the target position determined when the robot 2 moves forward or backward will be referred to as the "first target position", and the target position determined when the robot 2 rotates (i.e., the rotation angle) will be referred to as the "second target position".

[0065] The communication control unit 136 sends control commands to the robot 2 via the communication interface 110 to move the robot 2 to a target position. When the processing unit 26 of the robot 2 receives the control commands from the control device 100, it controls the actuator 24 based on the control commands. As a result, the robot 2 moves to a first target position in the traveling direction or to a second target position in the rotational direction.

[0066] Figure 3 This is a functional structure diagram of the HMI200 according to the first embodiment. In addition to the input interface 212 and wearable display 213 mentioned above, the HMI200 may also include a communication interface 211, an eye-tracking sensor 214, a motion tracking sensor 215, a storage unit 216, a processing unit 220, etc.

[0067] Communication interface 211 communicates with robot 2 and control device 100 via network NW. Communication interface 211 may include, for example, a wireless communication module including a receiver and transmitter, a NIC, etc. Communication interface 211 is an example of a "first communication interface".

[0068] The input interface 212 may also include, or instead include, a voice user interface 212b in addition to the aforementioned switch interface 212a (switch, button, joystick, etc.). The voice user interface 212b includes a microphone, etc., which converts the voice spoken by the operator OP into commands for the robot 2 and outputs them.

[0069] Eye-tracking sensor 214 detects the operator's gaze and / or viewpoint. For example, eye-tracking sensor 214 may be integrated into wearable display 213. In this case, eye-tracking sensor 214 detects the operator's gaze and / or viewpoint when wearing wearable display 213 on their head. Eye-tracking sensor 214 is an example of an "eye tracker".

[0070] Motion tracking sensor 215 detects the movement, tilt, acceleration, etc. of the operator's head and / or neck. Similar to eye tracking sensor 214, motion tracking sensor 215 can also be integrated into wearable display 213. For example, motion tracking sensor 215 may include a gyroscope, accelerometer, magnetometer, etc.

[0071] The storage unit 216 may be implemented using, for example, an HDD (Hard Disc Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The storage unit 216 stores firmware, application programs, and the calculation results of the processing unit 220.

[0072] The processing unit 220 enables the wearable display 213 to display image data, or sends operation commands for the robot 2 input by the operator (OP) to the input interface 212 to the control device 100 via the communication interface 211. The processing unit 220 is an example of a "first processing unit".

[0073] The processing unit 220 includes, for example, an acquisition unit 221, a display control unit 222, and a communication control unit 223. These components can be implemented by executing programs or instructions stored in the storage unit 216 using a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or similar means. Alternatively, these components can be implemented in hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or SOC (System on Chip), or through a combination of software and hardware.

[0074] The acquisition unit 221 acquires image data from the robot 2 via the communication interface 211, or operation data from the input interface 212, or eye-tracking data from the eye-tracking sensor 214, or motion-tracking data from the motion-tracking sensor 215.

[0075] The display control unit 222 enables the wearable display 213 to display image data obtained from the robot 2.

[0076] The communication control unit 223 sends operation data, eye-tracking data, motion tracking data, etc. to the control device 100 via the communication interface 211.

[0077] [Processing flow of the robot remote operating system]

[0078] The following is a flowchart illustrating the process of a series of processes in the robot remote operating system 1 according to the first embodiment. Figure 4This is a flowchart illustrating a series of processes of the robot remote operating system 1 according to the first embodiment. The processes in this flowchart can, for example, be performed while the wearable display 213 is displaying image data.

[0079] First, the display control unit 222 of HMI200 determines whether a movement instruction related to the direction of travel of robot 2 is generated from operator OP while operator OP is visually recognizing image data of the external environment of robot 2 (step S100).

[0080] For example, when the operator (OP) speaks "Go," "Go back," or "Go backward" into the voice user interface 212b, the display control unit 222 determines that a movement instruction related to the robot 2's direction of travel has been generated from the operator (OP). Additionally, when the operator (OP) operates the switch interface 212a, the display control unit 222 can also determine that a movement instruction related to the robot 2's direction of travel has been generated from the operator (OP).

[0081] When the operator OP generates a movement instruction related to the direction of travel of the robot 2, the display control unit 222 causes the wearable display 213 to display the viewpoint of the operator OP detected by the eye-tracking sensor 214 (step S102).

[0082] Figure 5 This diagram illustrates an example of a movement instruction related to the direction of travel of the robot 2 being displayed at the viewpoint of the operator OP on the wearable display 213. As shown, when a movement instruction related to the direction of travel of the robot 2 is present, the operator OP's viewpoint is overlaid as an indicator P onto the image visually perceived by the operator OP.

[0083] Next, the acquisition unit 132 of the control device 100 acquires operation data and eye-tracking data from the HMI 200 via the communication interface 110, and the determination unit 134 determines whether there is a setting instruction for the first target position based on the operation data and eye-tracking data (step S104).

[0084] The first target position, as described above, is the target position determined when the robot 2 moves forward or backward. This setting instruction is the instruction requested by the operator (OP) to set the first target position.

[0085] For example, if the operator OP says "Set" or "Set" to the voice user interface 212b after a movement instruction related to the robot 2's direction of travel is given, the decision unit 134 determines that a setting instruction for a first target position has been generated from the operator OP. Alternatively, if the operator OP operates the switch interface 212a after a movement instruction related to the robot 2's direction of travel is given, the decision unit 134 determines that a setting instruction for a first target position has been generated from the operator OP. The operator OP's speech "Set" or "Set" to the voice user interface 212b is an example of a "prescribed operation," and the operator OP's operation of the switch interface 212a is another example of a "prescribed operation."

[0086] When a setting instruction for a first target position is generated from the operator OP, the decision unit 134 determines the viewpoint of the operator OP at the moment the setting instruction is generated as the first target position (step S106).

[0087] Figure 6 This diagram illustrates an example of a method for determining the first target position. For example, during the period when the operator (OP) continuously operates the switch interface 212a, the operator's viewpoint is displayed while changing as indicated by indicators P1, P2, and P3. At this time, for example, if a setting instruction for the first target position is generated when indicator P2 is displayed, the determination unit 134 determines the position of indicator P2 as the first target position.

[0088] Furthermore, the decision unit 134 can continuously update the first target position by linking with the indicator P representing the viewpoint of the operator OP during operation when a setting instruction for the first target position is input to the input interface 212. In the illustrated example, the first target position changes in the order of P1, P2, and P3.

[0089] On the other hand, if no setting instruction for the first target position is generated from the operator OP in the determination process of S104, the decision unit 134 does not make a new decision on the first target position, but maintains the first target position determined last time (step S108).

[0090] Next, the communication control unit 136 of the control device 100 sends a control command to the robot 2 via the communication interface 110 to make the robot 2 move forward or backward to the first target position. Upon receiving the control command, the processing unit 26 of the robot 2 controls the actuator 24 based on the control command, thereby making the robot 2 move forward or backward to the first target position (step S110).

[0091] On the other hand, if no movement instruction related to the travel direction of the robot 2 is generated from the operator OP during the determination process of S100, the display control unit 222 further determines whether a movement instruction related to the rotation direction of the robot 2 is generated from the operator OP while the operator OP is visually recognizing the image data of the external environment of the robot 2 (step S112).

[0092] For example, if the operator OP says "rotate" or "turn around" to the voice user interface 212b, the display control unit 222 determines that a movement instruction related to the rotation direction of the robot 2 has been generated from the operator OP. Alternatively, if the operator OP operates the switch interface 212a (but operates a button or switch different from the one operated when a movement instruction related to the direction of travel is given), the display control unit 222 determines that a movement instruction related to the rotation direction of the robot 2 has been generated from the operator OP.

[0093] When the operator OP generates a movement instruction related to the direction of travel of the robot 2, the display control unit 222 causes the wearable display 213 to display the viewpoint of the operator OP detected by the eye-tracking sensor 214 (step S114).

[0094] Figure 7 This diagram illustrates an example of a movement instruction related to the rotation direction of robot 2 being displayed at the viewpoint of the operator OP on the wearable display 213. As shown, when a movement instruction related to the rotation direction of robot 2 is present, similar to when a movement instruction related to the movement direction is present, the operator OP's viewpoint is superimposed as an indicator P onto the image visually perceived by the operator OP.

[0095] Next, the acquisition unit 132 of the control device 100 acquires operation data and eye-tracking data from the HMI 200 via the communication interface 110, and the determination unit 134 determines whether there is a setting instruction for the second target position based on the operation data and eye-tracking data (step S116).

[0096] The second target position, as described above, is the target position determined when the robot 2 is rotated. This setting instruction is the instruction requested by the operator (OP) to set the second target position.

[0097] For example, if the operator OP says "set" or "configure" to the voice user interface 212b after a movement instruction related to the rotation direction of robot 2 is given, the determination unit 134 determines that a setting instruction for a second target position has been generated from the operator OP. Alternatively, if the operator OP operates the switch interface 212a after a movement instruction related to the rotation direction of robot 2 is given, the determination unit 134 determines that a setting instruction for a second target position has been generated from the operator OP.

[0098] When a setting instruction for a second target position is generated from the operator OP, the determination unit 134 determines the viewpoint of the operator OP at the moment the setting instruction is generated as the second target position (step S118).

[0099] Figure 8 This diagram illustrates an example of a method for determining a second target position. For instance, during continuous operation of the switch interface 212a by the operator OP, the operator OP's viewpoint is displayed while indicators P1 and P2 change. At this time, for example, if a setting instruction for a second target position exists when indicator P2 is displayed, the determination unit 134 determines the position of indicator P2 as the second target position.

[0100] Furthermore, the decision unit 134 can continuously update the second target position by linking with the indicator P representing the operator's (OP's) viewpoint during operation when a setting instruction for the second target position is input to the input interface 212. In the illustrated example, the second target position changes in the order of P1 and P2.

[0101] On the other hand, if no setting instruction for the second target position is generated from the operator OP in the determination process of S116, the decision unit 134 does not make a new decision on the second target position, but maintains the second target position determined last time (step S120).

[0102] Next, the communication control unit 136 of the control device 100 sends a control command to the robot 2 via the communication interface 110 to rotate the robot 2 to the second target position (which can be read as the target rotation angle). Upon receiving the control command, the processing unit 26 of the robot 2 controls the actuator 24 based on the control command, thereby rotating the robot 2 to the second target position (step S122).

[0103] On the other hand, if no movement instruction related to the travel direction of robot 2 is generated from operator OP in the decision processing of S100, and no movement instruction related to the rotation direction of robot 2 is generated in the decision processing of S112, the communication control unit 136 of control device 100 sends a control command to robot 2 via communication interface 110 to stop robot 2 (a control command to prevent it from moving forward, backward, or rotating in any direction). Upon receiving the control command, the processing unit 26 of robot 2 controls actuator 24 based on the control command, thereby stopping robot 2 (step S124).

[0104] According to the first embodiment described above, the robot remote operating system 1 includes at least a control device 100 for controlling the robot 2 and an HMI 200 used by the operator OP.

[0105] HMI200 enables wearable display 213 to display image data of the robot 2's surroundings captured by camera 21. During the period when operator OP visually recognizes wearable display 213 displaying image data, eye-tracking sensor 214 is used to detect operator OP's gaze and / or viewpoint, and sends eye-tracking data (an example of "gaze data") and operation data to control device 100. The eye-tracking data represents the gaze and / or viewpoint detected by eye-tracking sensor 214, and the operation data represents the operation of operator OP input to input interface 212.

[0106] The control device 100 receives eye-tracking data and operational data from the HMI 200, and determines the target position (first target position, second target position) of the robot 2 based on this data, then moves the robot 2 to the target position. With this configuration, the operator (OP) can instruct the robot 2 to move or rotate through simple operations. As a result, the stress of operating the robot 2 is reduced, and a feeling of being able to control the robot 2 according to the operator's intentions is created.

[0107] <Second Implementation>

[0108] The second embodiment will now be described. In the second embodiment, the methods for determining the first target position and the second target position differ from those in the first embodiment described above. The description will focus on the differences from the first embodiment, omitting explanations of points common to the first embodiment. It should be noted that in the description of the second embodiment, the same reference numerals are used to describe the parts that are the same as in the first embodiment.

[0109] [Processing flow of the robot remote operating system]

[0110] The following uses flowcharts to illustrate the process of a series of processes in the robot remote operating system 1 according to the second embodiment. Figure 9 This is a flowchart illustrating a series of processes of the robot remote operating system 1 according to the second embodiment. The processes in this flowchart can, for example, be performed while image data is displayed on the wearable display 213.

[0111] First, the display control unit 222 of the HMI200 determines whether a movement instruction related to the direction of travel of the robot 2 is generated from the operator OP while the operator OP is visually recognizing the image data of the external environment of the robot 2 (step S200).

[0112] When the operator OP generates a movement instruction related to the direction of travel of the robot 2, the display control unit 222 causes the wearable display 213 to display the viewpoint of the operator OP detected by the eye-tracking sensor 214 (step S202).

[0113] Next, the acquisition unit 132 of the control device 100 acquires operation data and eye-tracking data from the HMI 200 via the communication interface 110, and the determination unit 134 determines, based on the operation data and eye-tracking data, whether there is a setting instruction for the first target position and whether the setting instruction continues for a certain period of time (step S204).

[0114] For example, if the operator OP says "tracking on" or something similar to the voice user interface 212b after a movement instruction related to the direction of travel of robot 2 is given, the decision unit 134 determines that there is a setting instruction for the first target position and that the setting instruction continues for a certain period of time. Furthermore, if the operator OP then says "tracking off" or something similar to the voice user interface 212b, the decision unit 134 determines that the duration of the setting instruction for the first target position has ended.

[0115] Furthermore, if the operator (OP) continues to operate the switch interface 212a after a movement instruction related to the direction of travel of the robot 2 is given, the determination unit 134 can also determine that there is a setting instruction for the first target position and that the setting instruction continues for a certain period of time (the period during which it is operated). Moreover, if the operator (OP) subsequently stops operating the switch interface 212a, the determination unit 134 determines that the continuity of the setting instruction for the first target position has ended.

[0116] When a setting instruction for a first target position is generated from the operator OP and continues, the determination unit 134 determines the first target position by coordinating with the viewpoint of the operator OP during the duration of the setting instruction (step S206).

[0117] Figure 10This diagram illustrates an example of a method for determining the position of a first target. For instance, the operator's (OP) viewpoint is displayed while transitioning between indicators P1 through P5. At this time, the operator OP says "Tracking on" to the voice user interface 212b and then "Tracking off." In this case, during the period from saying "Tracking on" to the voice user interface 212b to saying "Tracking off," the operator OP's viewpoint is tracked, and the position of the first target is determined in conjunction with the operator OP's viewpoint during this period. In the illustrated example, the tracking period includes P2, P3, and P4. Therefore, the first target position is first determined to be P2, then updated to P3, and finally updated to P4.

[0118] On the other hand, if no setting instruction for the first target position is generated from the operator OP in the determination process of S204, the decision unit 134 does not make a new decision on the first target position, but maintains the first target position determined last time (step S208).

[0119] Next, the communication control unit 136 of the control device 100 sends a control command to the robot 2 via the communication interface 110 to make the robot 2 move forward or backward to the first target position. Upon receiving the control command, the processing unit 26 of the robot 2 controls the actuator 24 based on the control command, thereby making the robot 2 move forward or backward to the first target position (step S210).

[0120] On the other hand, if no movement instruction related to the travel direction of the robot 2 is generated from the operator OP in the determination process of S200, the display control unit 222 further determines whether a movement instruction related to the rotation direction of the robot 2 is generated from the operator OP while the operator OP is visually recognizing the image data of the external environment of the robot 2 (step S212).

[0121] When the operator OP generates a movement instruction related to the rotation direction of the robot 2, the display control unit 222 causes the wearable display 213 to display the viewpoint of the operator OP detected by the eye-tracking sensor 214 (step S214).

[0122] Next, the acquisition unit 132 of the control device 100 acquires operation data and eye-tracking data from the HMI 200 via the communication interface 110, and the determination unit 134 determines, based on the operation data and eye-tracking data, whether there is a setting instruction for a second target position and whether the setting instruction continues for a certain period of time (step S216).

[0123] For example, if the operator OP says "tracking on" or something similar to the voice user interface 212b after a movement instruction related to the rotation direction of robot 2 is given, the decision unit 134 determines that there is a setting instruction for a second target position and that the setting instruction continues for a certain period of time. Furthermore, if the operator OP then says "tracking off" or something similar to the voice user interface 212b, the decision unit 134 determines that the duration of the setting instruction for the second target position has ended.

[0124] Furthermore, if the operator OP continues to operate the switch interface 212a after a movement instruction related to the rotation direction of robot 2 is given, the determination unit 134 can also determine that there is a setting instruction for a second target position, and that the setting instruction continues for a certain period of time (the period during which it is operated). Moreover, if the operator OP subsequently stops operating the switch interface 212a, the determination unit 134 determines that the continuity of the setting instruction for the second target position has ended.

[0125] When a setting instruction for a second target position is generated from the operator OP and continues, the determination unit 134 determines the second target position by linking with the viewpoint of the operator OP during the duration of the setting instruction (step S218).

[0126] Figure 11 This diagram illustrates an example of a method for determining the location of a second target. For instance, the operator's (OP) viewpoint is displayed while indicators P1 through P5 change. At this time, the operator OP says "Tracking on" to the voice user interface 212b and then "Tracking off." In this case, during the period from saying "Tracking on" to the voice user interface 212b to saying "Tracking off," the operator OP's viewpoint is tracked, and the location of the second target is determined in conjunction with the operator OP's viewpoint during this period. In the illustrated example, the tracking period includes P2, P3, and P4. Therefore, the second target location is first determined to be P2, then updated to P3, and finally updated to P4.

[0127] On the other hand, if no setting instruction for the second target position is generated from the operator OP in the determination process of S216, the decision unit 134 does not make a new decision on the second target position, but maintains the second target position determined last time (step S220).

[0128] Next, the communication control unit 136 of the control device 100 sends a control command to the robot 2 via the communication interface 110 to rotate the robot 2 to the second target position (which can be read as the target rotation angle). Upon receiving the control command, the processing unit 26 of the robot 2 controls the actuator 24 based on the control command, thereby causing the robot 2 to rotate to the second target position (step S222).

[0129] On the other hand, if no movement instruction related to the travel direction of robot 2 is generated from operator OP in the decision processing of S200, and no movement instruction related to the rotation direction of robot 2 is generated in the decision processing of S212, the communication control unit 136 of control device 100 sends a control command to robot 2 via communication interface 110 to stop robot 2 (a control command to prevent it from moving forward, backward, or rotating in any direction). Upon receiving the control command, the processing unit 26 of robot 2 controls actuator 24 based on the control command, thereby stopping robot 2 (step S224).

[0130] According to the second embodiment described above, similarly to the first embodiment, the operator (OP) can instruct the robot 2 to move or rotate through simple operations. As a result, the stress of operating the robot 2 can be reduced, and a feeling of being able to control the robot 2 according to the operator's (OP's) intentions can be created.

[0131] <Other Implementation Methods>

[0132] The following describes other implementation methods. In the above implementation methods, the setting instructions for determining the first target position and the second target position were described when the operator OP made a specific speech to the voice user interface 212b or operated the switch interface 212a, but it is not limited to this.

[0133] For example, the setting instruction for determining the first target position can be considered to be continuous from the time the movement instruction related to the direction of travel is given until the operator OP says "stop" to the voice user interface 212b. Alternatively, the setting instruction for determining the first target position can be considered to be continuous during the period after the movement instruction related to the direction of travel is given, while the operator OP continues to operate the switch interface 212a.

[0134] The same applies to the second target position. For example, the setting instruction for determining the second target position can be considered to continue from the time the movement instruction related to the rotation direction is given until the operator OP says "stop" to the voice user interface 212b. Alternatively, the setting instruction for determining the second target position can be considered to continue during the time the operator OP continues to operate the switch interface 212a after the movement instruction related to the rotation direction is given.

[0135] Figure 12This diagram illustrates another example of the method for determining the second target position. For example, the operator's (OP) viewpoint is displayed while changing as indicated by indicators P1 to P5. During the period from issuing a movement instruction related to the rotation direction to the operator OP saying "Stop" to the voice user interface 212b, the operator OP's viewpoint is tracked, and the second target position is determined in conjunction with the operator OP's viewpoint during this period. Furthermore, P4, the moment the word "Stop" is uttered, is determined as the final second target position.

[0136] The first target position and the second target position can also be determined automatically without the operator (OP) having to operate (i.e. without setting instructions). For example, the determination unit 134 can automatically determine the first target position and the second target position based on the change in the amount of movement of the operator's (OP's) gaze detected by the eye-tracking sensor 214.

[0137] Figure 13 This diagram illustrates another example of the method for determining the second target position. For example, when moving the viewpoint from the current P1 to the operator OP's intended Px, the following situation exists: when the viewpoint moves a small amount, it does not reach Px; on the other hand, when the viewpoint moves a large amount, it crosses Px. However, after a certain period of time, the viewpoint naturally converges to Px. Considering this situation, the determination unit 134 considers the position where the change in the viewpoint's movement converges as the position intended by the operator OP, and determines this position intended by the operator OP as the second target position.

[0138] Furthermore, while the above embodiments describe a robot 2 operated by the robot remote operating system 1 as a physical robot, this is not a limitation. For example, the robot 2 operated by the robot remote operating system 1 could also be a robot existing in a virtual space such as VR space, AR space, or MR space.

[0139] The above description illustrates specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention. For example, the display for remote operation of robot 2 is not limited to wearable display 213 worn by operator OP, but may also be a desktop or portable display device.

Claims

1. A robot teleoperation system, wherein the robot teleoperation system is provided with: a control device that controls a robot; and a human-machine interface that is used by an operator when teleoperating the robot, the human-machine interface is provided with: a first communication interface that communicates with the control device; a display that displays an image of surroundings of the robot taken by a camera; an eye tracker that detects a line of sight and / or a point of sight of the operator during a period in which the operator visually recognizes the display on which the image is displayed; an estimation section that estimates an operation intention of the operator; and a first processing section that transmits, to the control device via the first communication interface, line-of-sight data that is data indicating the line of sight and / or the point of sight detected by the eye tracker and operation data that is data indicating the operation intention of the operator estimated by the estimation section, the control device is provided with: a second communication interface that communicates with the human-machine interface; and a second processing section that, when the line-of-sight data and the operation data are received from the human-machine interface by the second communication interface, decides a target position of the robot on the basis of the line-of-sight data and the operation data and causes the robot to move to the target position.

2. The robot teleoperation system according to claim 1, wherein the second processing section decides, as the target position, a first target position that is related to a traveling direction of the robot and a second target position that is related to a rotating direction of the robot.

3. The robot teleoperation system according to claim 1, wherein the human-machine interface is further provided with an input interface that is inputted with an operation of the operator, the second processing section decides, as the target position of the robot, a position of the point of sight of the operator detected by the eye tracker when a prescribed operation is inputted to the input interface during a period in which the operator visually recognizes the display on which the image is displayed.

4. The robot teleoperation system according to claim 3, wherein the second processing section continuously updates the target position of the robot in association with the position of the point of sight of the operator during a period in which the prescribed operation is inputted to the input interface.

5. The robot teleoperation system according to claim 3, wherein the input interface includes a voice user interface or a switch interface, the prescribed operation refers to inputting a prescribed utterance to the voice user interface or operating the switch interface.

6. The robot teleoperation system according to claim 1, wherein the second processing section decides the target position of the robot on the basis of a change in an amount of movement of the point of sight of the operator detected by the eye tracker.

7. The robot teleoperation system according to claim 6, wherein the second processing section decides, as the target position of the robot, a position at which the change in the amount of movement of the point of sight converges. ​ ​ 8. A robot remote operation method which utilizes a control device that controls a robot, and a human-machine interface that is utilized by an operator when remotely operating the robot, wherein the robot remote operation method causes the human-machine interface to perform processing including: communicating with the control device; causing a display to display an image of surroundings of the robot taken by a camera; detecting a line of sight and / or a point of sight of the operator during a period in which the operator visually recognizes the display on which the image is displayed; inferring an operation intention of the operator; and transmitting line-of-sight data, which is data indicating the detected line of sight and / or point of sight, and operation data, which is data indicating the inferred operation intention of the operator, to the control device, the robot remote operation method causes the control device to perform processing including: communicating with the human-machine interface; deciding a target position of the robot based on the line-of-sight data and the operation data, in a case in which the line-of-sight data and the operation data are received from the human-machine interface; and causing the robot to move to the target position.

9. A storage medium that holds a program for causing a control device that controls a robot, and a human-machine interface that is utilized by an operator when remotely operating the robot, to perform processing, wherein the program is for causing the human-machine interface to perform processing including: communicating with the control device; causing a display to display an image of surroundings of the robot taken by a camera; detecting a line of sight and / or a point of sight of the operator during a period in which the operator visually recognizes the display on which the image is displayed; inferring an operation intention of the operator; and transmitting line-of-sight data, which is data indicating the detected line of sight and / or point of sight, and operation data, which is data indicating the inferred operation intention of the operator, to the control device, the program is for causing the control device to perform processing including: communicating with the human-machine interface; deciding a target position of the robot based on the line-of-sight data and the operation data, in a case in which the line-of-sight data and the operation data are received from the human-machine interface; and causing the robot to move to the target position. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Images