Robot remote operation system, remote operation control device, remote operation control method, and program

Abstract

To provide a robot remote operation system, a remote operation control device, a remote operation control method, and a program capable of restarting interrupted remote operation while satisfying both safety of an environment and operability of a human operator.SOLUTION: The robot remote operation system includes: a robot to be remotely operated; a first acquisition unit that acquires a state of the robot; an operation unit to which an operator inputs an operation command of the robot; a second acquisition unit that acquires an operation input value to the operation unit; and an operation command generation unit that generates an operation command from the operation input value. When a position based on the operation input value and a position of the state of the robot are different from each other, the operation command generation unit determines the different state as an offset, and generates the robot operation command so as to reduce the offset after receiving a command to reduce the state of the offset.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a robot remote operation system, a remote operation control device, a remote operation control method, and a program.

Background Art

[0002] In recent years, technologies for remotely operating robots have been developed. For example, Patent Document 1 discloses a technique for detecting an error when an error occurs in such a system and canceling the error to resume a command.

[0003] When remotely operating a robot, for example, there are a method in which an operator closely attaches a sensor such as an exoskeleton to the arm or hand and operates it, and a method in which the operator wears a data glove or the like on the hand and operates it. For example, when an operator remotely operates a robot, the operator may want to interrupt the work at will during a period such as a break time. When the method of closely attaching a sensor such as an exoskeleton to the arm or hand by the operator is used, the remote operation can be resumed in accordance with the state of the robot by reattaching it when resuming while maintaining the shape of a large number of attached sensors when the operator interrupts the remote operation. On the other hand, in the method of wearing a data glove or the like on the hand and operating it, the operator has to adjust to the state before resumption.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technology described in Patent Document 1, only a technology for automatically resuming a command at the time of an error is described, and when resuming work after interrupting the work at the operator's will, the work could not be resumed. Further, in the technology described in Patent Document 1, if the same processing as at the time of an error is performed, if the operation instructions before and after resumption are different, there is also a risk that the operator cannot perform remote operation or that the robot cannot be operated safely. Also, in the prior art of operating by closely attaching a sensor such as an exoskeleton to the arm or hand, wearing the exoskeleton was difficult, and convenience such as wearability was poor for the operator. Furthermore, in the prior art of operating by wearing a data glove or the like, it is troublesome and difficult for the operator to match the state of the joint angles of the robot before interruption, and if it cannot be matched, a deviation occurs between the instructions of the robot and the operator, making it difficult to work and there is also a risk that the robot cannot be operated safely.

[0006] The present invention has been made in view of the above problems, and an object thereof is to provide a robot remote operation system, a remote operation control device, a remote operation control method, and a program that can resume an interrupted remote operation while achieving both environmental safety and ease of operation by a human operator.

Means for Solving the Problems

[0007] (1) To achieve the above object, a robot remote operation system according to an aspect of the present invention includes a robot to be remotely operated, a first acquisition unit that acquires the state of the robot, an operation unit through which an operator inputs an operation command for the robot, a second acquisition unit that acquires an operation input value of the operation unit, and an operation command generation unit that generates an operation command from the operation input value. When the position based on the operation input value and the position of the state of the robot are different, the operation command generation unit determines different states as an offset, and generates a robot operation command so as to reduce the offset after receiving a command to reduce the state of the offset.

[0008] (2) In the robot remote operation system according to one aspect of the present invention as described in (1) above, when there is a deviation greater than or equal to a threshold value between the position based on the operation input value and the position of the state of the robot, the operation command generation unit may not output a robot operation command to the robot until the position based on the operation input value and the position of the state of the robot are within the threshold value.

[0009] (3) In the robot remote operation system according to one aspect of the present invention as described in (2) above, the operation command generation unit may output the robot operation command to the robot when the deviation between the position based on the operation input value and the position of the state of the robot is less than the threshold value.

[0010] (4) In the robot remote operation system according to one aspect of the present invention among any one of (1) to (3) above, further comprising a state determination unit that determines the state of the robot, and when the state determination unit determines that the robot is in a state where it does not interfere with the environment, it may output an instruction to reduce the offset state. It may be like this.

[0011] (5) In the robot remote operation system according to one aspect of the present invention among any one of (1) to (4) above, when the operation command generation unit acquires an instruction from the operator to reduce the offset, it may set the offset to zero.

[0012] (6) In the robot remote operation system according to one aspect of the present invention among any one of (1) to (5) above, further comprising a reduction command output unit that issues a command to reduce the offset state, and the operation command generation unit may generate the robot operation command by leaving or reducing the offset according to the output of the reduction command output unit.

[0013] (7) In the robot remote operation system according to any one of the above (1) to (6) aspects of the present invention, when the position based on the operation input value is different from the position of the state of the robot, an image generation unit that generates and presents a guide image for moving the difference between the position based on the operation input value and the position of the state of the robot to be less than a threshold value is further provided, and the operation command generation unit changes the position and posture of the operation unit according to the guide image so that the operator makes the difference between the position based on the operation input value and the position of the state of the robot less than the threshold value, and when an instruction to reduce the offset is obtained from the operator, the robot operation command for setting the offset to zero may be output to the robot to change the position and posture of the robot.

[0014] (8) To achieve the above object, a remote operation control device according to an aspect of the present invention is a remote operation control device for remotely operating a robot, including a first acquisition unit that acquires the state of the robot, an operation unit through which an operator inputs an operation command for the robot, a second acquisition unit that acquires an operation input value of the operation unit, and an operation command generation unit that generates an operation command from the operation input value. When the position based on the operation input value is different from the position of the state of the robot, the operation command generation unit determines a different state as an offset, and generates a robot operation command to eliminate the offset after receiving an instruction to reduce the state of the offset.

[0015] (9) To achieve the above object, a remote operation control method according to an aspect of the present invention is a remote operation control method for remotely operating a robot, in which a first acquisition unit acquires the state of the robot, a second acquisition unit acquires an operation input value through which an operator inputs an operation command for the robot, an operation command generation unit generates an operation command from the operation input value, and when the position based on the operation input value is different from the position of the state of the robot, the operation command generation unit determines a different state as an offset, and generates a robot operation command to eliminate the offset after receiving an instruction to reduce the state of the offset.

[0016] (10) To achieve the above object, a program according to an aspect of the present invention causes a computer of a remote operation control device that remotely operates a robot to acquire the state of the robot, acquire an operation input value input by an operator for an operation command of the robot, generate an operation command from the operation input value, and when the position based on the operation input value is different from the position of the state of the robot, determine a different state as an offset, and generate a robot operation command to eliminate the offset after receiving a command to reduce the state of the offset.

Advantages of the Invention

[0017] According to the above (1) to (10), by reducing the offset, it is possible to achieve both environmental safety and ease of operation for a human operator, and resume the interrupted remote operation. According to the above (1) to (10), by safely reducing the offset, it is possible to prevent causing coordinate confusion for the operator Us.

Brief Description of the Drawings

[0018]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each member is appropriately changed in order to make each member recognizable. In all the drawings for explaining the embodiments, those having the same function are denoted by the same reference numerals, and repeated explanations are omitted. In addition, “based on XX” as used in the present application means “based on at least XX”, and includes cases based on other elements in addition to XX. Also, “based on XX” is not limited to the case of directly using XX, and includes cases based on those obtained by performing calculations or processing on XX. “XX” is an arbitrary element (for example, arbitrary information).

[0020] [Overview of Remote Operation and Object to be Operated] First, an overview of remote operation and the object to be operated will be described. FIG. 1 is a diagram for explaining an overview of remote operation of a robot and an object to be operated. As shown in FIG. 1, in the remote operation space, an operator Us wears, for example, an HMD (head-mounted display) 4 on the head and operation units 5 (5L, 5R) such as data gloves on the hands. An environmental sensor 3 is installed in the robot work space. Note that the environmental sensor 3 may be attached to the robot 2. The robot 2 includes a remote operation control device 6, end effectors 21 (first end effector 21L, second end effector 21R), and hands 211 (211L, 211R).

[0021] The target object obj is composed of a plurality of objects. In the example of FIG. 1, the target object obj is, for example, a plastic bottle or a bottle, and includes a body body and a lid cap. The operator Us remotely operates the robot 2 to manipulate the target object obj by, for example, moving the hand or finger wearing the operation unit 5 while looking at the image displayed on the HMD 4. An example of the operation content is, as shown in FIG. 1, attaching and closing the lid cap to the body, or opening the lid cap from the body, etc.

[0022] Alternatively, as shown in FIG. 2, pick up the screw which is the first target object obj1, move it above the second target object obj2 which is the object to attach the screw, and then tighten the screw to the second target object obj2 using a driver. FIG. 2 is a diagram showing an example of the work by remote operation.

[0023] In the following embodiments, when temporarily interrupting and then resuming the remote operation, resume it in the operation mode that has been used so far. The operation modes include, for example, a free movement mode and a constrained mode. The free movement mode is, for example, a mode in which the hand 211 etc. is not constrained and can be freely moved according to the instruction of the operator Us. The constrained mode is, for example, a mode constrained in some dimensions (for example, the z-axis direction).

[0024] [Configuration of Robot Remote Operation System] Next, a configuration example of the robot remote operation system 1 will be described. FIG. 3 is a diagram showing a configuration example of the robot remote operation system according to the present embodiment. As shown in FIG. 3, the robot remote operation system 1 includes, for example, a robot 2, an environment sensor 3, an HMD 4, an operation unit 5, and a remote operation control device 6.

[0025] The robot 2 includes, for example, a first end effector 21L, a second end effector 21R, and a communication unit 22. The first end effector 21L includes, for example, a hand 211L, an actuator 212L, and a sensor 213L. The second end effector 21R includes, for example, a hand 211R, an actuator 212R, and a sensor 213R. Note that the robot 2 may include a communication unit, a power supply unit, a body, legs, a head, etc. not shown in the figure. In the following description, when the first end effector 21L and the second end effector 21R are not distinguished, they are also referred to as the end effector 21. Similarly, when the hand 211L and the hand 211R are not distinguished, they are also referred to as the hand 211. When the actuator 212L and the actuator 212R are not distinguished, they are also referred to as the actuator 212. When the sensor 213L and the sensor 213R are not distinguished, they are also referred to as the sensor 213. The robot 2 transmits and receives various information to and from the remote operation control device 6 via a wired or wireless network NW.

[0026] The environment sensor 3 includes, for example, a sensor 31 and a communication unit 32. Note that the environment sensor 3 also includes a power supply unit (not shown) and the like. The environment sensor 3 transmits and receives various information to and from the remote operation control device 6 via a wired or wireless network NW.

[0027] The HMD 4 includes, for example, an image display unit 41, a gaze detection unit 42, and a communication unit 43. Note that the HMD 4 also includes a power supply unit (not shown) and the like. The HMD 4 transmits and receives various information to and from the remote operation control device 6 via a wired or wireless network NW.

[0028] The operation unit 5 includes, for example, a sensor 51, a trigger SW 52, and a communication unit 53. The operation unit 5 transmits information to the remote operation control device 6 via a wired or wireless network NW.

[0029] The remote operation control device 6 includes, for example, a first acquisition unit 61, a second acquisition unit 62, an offset detection unit 63, an operation command generation unit 64, a state determination unit 65, a reduction command output unit 66, a drive circuit 67, an image generation unit 68, an output unit 69, a storage unit 70, and a threshold setting unit 71. Note that the remote operation control device 6 also includes a power supply unit (not shown) and the like. The remote operation control device 6 transmits and receives various information to and from the robot 2 and the HMD 4 via a wired or wireless network NW. The remote operation control device 6 receives various information from the environment sensor 3 and the operation unit 5 via a wired or wireless network NW.

[0030] Note that the configuration example shown in FIG. 3 is just an example and is not limited to this.

[0031] [Functions of Each Device in the Robot Remote Operation System] Next, the functions of each device in the robot remote operation system 1 will be described with reference to FIG. 3. (Robot 2) The hand 211 includes, for example, a plurality of finger portions. Each finger portion has a joint. Note that the hand 211 may be a gripper or the like. The actuator 212 is attached to each joint. The sensor 213 is, for example, a six-axis sensor attached to a joint, a tactile sensor attached to a finger portion, or the like. Note that the six-axis sensor detects forces in three axes (x, y, z) and moments in three axes (α, β, γ). The communication unit 22 transmits the detection value detected by the sensor 213 to the remote operation control device 6. The communication unit 22 receives a control signal or a control instruction output by the remote operation control device 6. Note that the data output by the robot 2 includes identification information that can identify the robot 2. Also, the data acquired by the robot 2 includes identification information that can identify that it is addressed to the robot 2. Note that the robot 2 may be provided with a drive circuit that drives the actuator 212.

[0032] (Environmental Sensor 3) As shown in FIG. 1, the environmental sensor 3 is installed, for example, in the robot working space. The sensor 31 is, for example, an RGB-D camera that acquires RGB (red, green, blue) information and depth information. Note that the information is acquired, for example, at a predetermined time interval. The communication unit 32 outputs the detection value detected by the sensor 31 to the remote operation control device 6. Note that the data output by the environmental sensor 3 includes identification information that can identify the environmental sensor 3.

[0033] (HMD 4) The image display unit 41 displays the image data acquired by the communication unit 43 from the remote operation control device 6. The line-of-sight detection unit 42 detects the line-of-sight direction and movement of the operator Us. The communication unit 43 transmits the line-of-sight information detected by the line-of-sight detection unit 42 to the remote operation control device 6. The communication unit 43 receives the image data output by the remote operation control device 6. Note that the data output by the HMD 4 includes identification information that can identify the HMD 4. Also, the data acquired by the HMD 4 includes identification information that can identify that it is addressed to the HMD 4.

[0034] (Operation unit 5) The operation unit 5 is, for example, a data glove. The sensor 51 detects the finger joint angles and wrist joint angle of the operator Us's hand. The trigger SW 52 is, for example, a mechanical switch. The trigger SW 52 detects the operation of the operator Us. Note that the operator Us selects and inputs which operation it is, for example, when starting an operation, interrupting an operation, resuming an operation (returning to an operation), or ending an operation, by pressing a dedicated switch or a predetermined number of times of the switch. Note that the operator Us may select and input the above by performing a predetermined gesture. The communication unit 53 outputs to the remote operation control device 6 the detection value detected by the sensor 51 and an operation trigger signal indicating the detection result detected by the trigger SW 52. Note that the data output by the operation unit 5 includes identification information that can identify the HMD 4.

[0035] (Remote operation control device 6) The first acquisition unit 61 acquires, for example, an offset reduction instruction for reducing the offset from the operation unit 5. The first acquisition unit 61 acquires, for example, an operation trigger signal from the operation unit 5. The first acquisition unit 61 acquires the first sensor value detected by the sensor 213 from the robot 2. The first acquisition unit 61 acquires the first sensor value detected by the environment sensor 3.

[0036] The second acquisition unit 62 acquires the second sensor value detected by the sensor 51 of the operation unit 5. The second acquisition unit 62 acquires the second sensor value detected by the line-of-sight detection unit 42 of the HMD 4.

[0037] The offset detection unit 63 detects, for example, the difference between the position and posture of the hand 211 of the end effector 21 of the robot 2 and the position and posture of the hand of the operator Us. The detection of the difference in position and posture is performed, for example, in the robot coordinate system. For coordinate system conversion, refer to, for example, Japanese Patent Application No. 2023-031054. The offset detection unit 63 compares the detected offset amount with the threshold value stored in the storage unit 70, and when the offset amount is less than the threshold value, determines it as the current offset amount. Note that the detection method example will be described later.

[0038] The operation command generation unit 64 determines whether or not it has received a command to reduce the offset state from the reduction command output unit 66. When the operation command generation unit 64 has received a command to reduce the offset state from the reduction command output unit 66, it sets the offset to 0 or a predetermined amount, and generates a robot operation command based on the second sensor value acquired by the second acquisition unit 62. When the operation command generation unit 64 has not received a command to reduce the offset state from the reduction command output unit 66, it generates a robot operation command based on the second sensor value acquired by the second acquisition unit 62 while leaving the offset amount as it is.

[0039] The state determination unit 65 determines whether or not the robot is in a state of not interfering with the operation environment. The operation environment is an object in the robot work space, and is, for example, an operation object, a table, a tool, a wall, etc. When the robot is in a state of not interfering with the operation environment, the state determination unit 65 outputs information indicating a command to reduce the offset amount to the reduction command output unit 66. When the robot is not in a state of not interfering with the operation environment, the state determination unit 65 does not output information indicating a command to reduce the offset amount to the reduction command output unit 66.

[0040] The reduction command output unit 66 outputs a command to reduce the offset amount to the operation command generation unit 64 and the image generation unit 68 according to the offset reduction instruction acquired by the first acquisition unit 61. Further, the reduction command output unit 66 compares the offset amount with the threshold value stored in the storage unit 70 and determines whether to output a command to reduce the offset to the operation command generation unit 64 and the image generation unit 68. For example, the reduction command output unit 66 outputs information to the operation command generation unit 64 to prevent the operation command from being sent to the robot 2 until the position based on the operation input value and the position of the state of the robot 2 are less than the threshold value.

[0041] The drive circuit 67 generates a drive signal for controlling the robot 2 based on the control command generated by the operation command generation unit 64. When the robot 2 is provided with the drive circuit 67, the remote operation control device 6 may not be provided with the drive circuit 67. Alternatively, the remote operation control device 6 and the robot 2 may each be provided with a part of the drive circuit 67.

[0042] The image generation unit 68 generates an image to be provided to the HMD 4 based on the image captured by the environment sensor 3 and the output of the reduction command output unit 66. Regarding the method of generating the image displayed on the HMD 4 and examples of the image, for example, the method described in Japanese Patent Application No. 2022-156322 may also be used. The image generation unit 68 generates a guide image so that the offset amount is less than the threshold value. After the offset amount becomes less than the threshold value, the image generation unit 68 generates a presentation image when further reducing the offset amount. Examples of the image presented to the HMD 4 will be described later.

[0043] The output unit 69 outputs the image data generated by the image generation unit 68 to the HMD 4. The output unit 69 outputs the drive signal output by the drive circuit 67 or the operation command generated by the operation command generation unit 64 to the robot 2.

[0044] The storage unit 70 stores, for example, programs used by each part of the remote operation control device 6, mathematical formulas, threshold values described later, identification information of the robot 2, identification information of the environment sensor 3, identification information of the HMD 4, identification information of the operation unit 5, and the like.

[0045] The threshold setting unit 71 sets a threshold value and stores the set threshold value in the storage unit 70. Note that the number of threshold values is not limited to one, and may be plural, for example, according to the application. The method for setting the threshold value will be described later.

[0046] [Offset Detection] Next, the offset detection method will be described. FIG. 4 is a diagram for explaining the offset detection method. Note that the coordinate system is the robot coordinate system. In FIG. 4, a gripper is shown as an example of the hand 211, but the hand 211 may have three or more finger parts. When the operator Us performs remote operation, the operation unit 5 and the HMD 4 are worn in advance, and calibration with the hand 211 of the robot 2 is performed.

[0047] In FIG. 4, the image g101 is an image showing the current state (position and posture) of the hand 211 of the robot 2. The image g102 is the state of the hand 211 of the robot 2 based on the operation command virtually generated by the operation command generation unit 64 based on the detection value acquired from the operation unit 5, and is also referred to as a "VR marker" in the following description.

[0048] First, the offset detection unit 63 uses the detection value acquired by the second acquisition unit 62 from the sensor 51 of the operation unit 5 to obtain the position and posture of the fingertip from the angle of the finger joint, for example, by forward kinematics calculation of the finger (see, for example, Reference 1). Note that the position and posture of this fingertip are based on the instructed position and posture of the operation unit 5 of the operator Us. Next, the offset detection unit 63 converts the obtained position and posture of the fingertip into the robot coordinate system. Next, the offset detection unit 63 calculates the difference between the position and orientation of the fingertip of the hand 211 of the current robot 2 and the position and orientation of the fingertip converted into the robot coordinate system. For example, Roll, Pitch, and Yaw for each of the x, y, and z axes are calculated using the rotation matrix components and the translation matrix components (see, for example, Reference 2). Hereinafter, in the embodiment, the "difference between the position and orientation of the fingertip of the hand 211 of the current robot 2 and the position and orientation of the fingertip converted into the robot coordinate system" is referred to as the "offset state" or the "offset amount".

[0049] Reference 1; Masaharu Takano, Toshiaki Nagashima, et al., "Grasping and Manipulating Movements by the Finger Pad", The Robotics Society of Japan, Transactions of the Robotics Society of Japan Vol.14 No.1, p83~90, 1996 Reference 2; Tokuo Tsuji, Hoshisei Baba, et al., "Evaluation of Grasping Stability Based on Potential Field Allowable External Force Energy with Flexible Fingertips and Joint Variations", The Robotics Society of Japan, The 33rd Annual Conference of the Robotics Society of Japan, RSJ2015AC1G3-02, 2015

[0050] Next, the offset detection unit 63 outputs information indicating the calculated offset amount to the image generation unit 68 and the operation command generation unit 64.

[0051] Note that the above-described method for calculating the offset is an example, and other methods may be used.

[0052] [Threshold] Next, an example of how to determine the threshold value will be described. The threshold value may be determined according to the application. (I) From the viewpoint of not disturbing the operator Us, if it is desired to start in a state where the deviation between the end effector 21 of the actual robot 2 and the operation unit 5 is small, a small threshold value is set. (II) If the operator Us wants to resume the work of the hand of the robot 2 at a high position while keeping his / her own hand low, the position threshold value is set large so that the resumption can be made with a large positional offset. (III) Depending on at least one of the task type (high place, low place), the age of the operator, the weight of the operator, the length of the arm of the end effector 21 of the robot 2, etc., the threshold setting unit 71 automatically determines the threshold value. In this case, the storage unit 70 stores the threshold values associated with each task in advance.

[0053] Note that the above setting examples of the threshold values in (I) to (III) are just examples, and other settings may be made according to the work content, the preferences of the operator Us, etc. For example, the threshold setting unit 71 may prepare and maintain low sensitivity (large threshold), normal sensitivity (normal threshold), high sensitivity (small threshold), etc., and select and use them from the settings presented by the operator Us. Note that the threshold value is preferably within a range where the operator Us will not be confused even when operating with an offset amount at restart.

[0054] [Prompt Image and Guide Image] Next, an example of the image presented on the HMD 4 and the guide image will be described. FIG. 5 is a diagram showing an example of the image presented on the HMD and the guide. In each of the images g210 to g240, the reference sign g201 indicates the position and posture of the current hand 211 of the robot 2, and the reference sign g202 indicates the virtual position and posture of the hand of the robot 2 based on the instruction of the operation unit 5 of the operator Us.

[0055] The image g210 is a diagram showing the first state. The image g211 is an example of a guide image that guides (advises) how the operation unit 5 of the operator Us should be moved to match the position and posture of the current hand 211 of the robot 2. In the first state example, it shows that the operation unit 5 of the operator Us is rotated clockwise by about 45 degrees and further moved downward.

[0056] The image g220 is a diagram showing the second state. The image g221 is an example of a guide image that guides how the operation unit 5 of the operator Us should be moved to match the position and posture of the current hand 211 of the robot 2. In the second state example, it shows that the operation unit 5 of the operator Us is moved diagonally downward to the left.

[0057] The image g230 is a diagram showing the third state. The image g231 is an example of a guide image that guides how the operation unit 5 of the operator Us should be moved to match the current position and posture of the hand 211 of the robot 2. In the example of the third state, it shows that the operation unit 5 of the operator Us is rotated about 90 degrees clockwise and further moved upward.

[0058] The image g240 is a diagram showing the fourth state. The fourth state is, for example, a state in which, as a result of moving the operation unit 5 of the operator Us according to the guides of the first to third states, the position and posture of the current hand 211 of the robot 2 and the virtual position and posture of the hand of the robot 2 based on the instruction of the operation unit 5 of the operator Us are within a threshold value. The image g221 indicates that the position and posture of the operation unit 5 are appropriate (e.g., "Good!"). Note that during remote operation, the image g202 showing the virtual position and posture may not be displayed, or it may be displayed or erased according to the instruction of the operator Us.

[0059] Note that in the example described with reference to FIG. 5, an example of presenting a guide image on the HMD 4 is shown, but it is not limited to this. The guide may use sound, or may use sound as well. For example, when approaching the threshold value, the feeling of the sound ringing may be shortened. Or, the guide may be a voice guide.

[0060] [First Process] Next, an example of the first process performed by the remote operation control device 6 will be described. FIG. 6 is a flowchart of the first processing procedure performed by the remote operation control device according to the present embodiment. Note that the following processing is repeated by acquiring each sensor value, for example, every several hundred msec.

[0061] (Step S1) The first acquisition unit 61 determines, for example, whether an operation trigger signal has been acquired from the operation unit 5. When the first acquisition unit 61 has acquired the operation trigger signal (Step S1; YES), the process proceeds to the process of Step S2. When the first acquisition unit 61 has not acquired the operation trigger signal (Step S1; NO), the process of Step S1 is repeated.

[0062] (Step S2) The first acquisition unit 61 acquires the first sensor value detected by the sensor 213 from the robot 2. The second acquisition unit 62 acquires the second sensor value detected by the sensor 51 of the operation unit 5. The offset detection unit 63 calculates the position and orientation of the VR marker using the second sensor value acquired by the second acquisition unit 62. The offset detection unit 63 calculates the current position and orientation of, for example, the hand 211 of the robot 2 using the first sensor value acquired by the first acquisition unit 61.

[0063] (Step S3) The offset detection unit 63 calculates and detects the offset amount between the position and orientation of the VR marker and the current position and orientation of the hand 211 of the robot 2. The offset detection unit 63 obtains the offset amount, for example, by subtracting (the current position and orientation of the hand 211 of the robot 2) from (the position and orientation of the VR marker).

[0064] (Step S4) The offset detection unit 63 determines whether the obtained offset amount is less than the threshold value. When the offset amount is less than the threshold value (Step S4; YES), the offset detection unit 63 proceeds to the process of Step S5. When the offset amount is greater than or equal to the threshold value (Step S4; NO), the offset detection unit 63 proceeds to the process of Step S5. Note that the offset amount includes information indicating the offset amount of the orientation and the offset amount of the position, respectively.

[0065] (Step S5) The image generation unit 68 uses the offset amount of the posture and the offset amount of the position obtained by the offset detection unit 63 to generate a guide image such as that in FIG. 5 so as to easily align the position and posture of the operation unit 5. The image generation unit 68 generates an output image by, for example, superimposing the generated guide image on the image to be displayed on the HMD 4. The output unit 69 outputs the generated output image with the guide image to the HMD 4. After the process, the image generation unit 68 returns to the process of Step S2.

[0066] (Step S6) The offset detection unit 63 determines the offset amount that was less than the threshold value in Step S4 as the current offset amount.

[0067] (Step S7) After the offset detection unit 63 determines the offset amount, the operation command generation unit 64 generates a robot operation command based on the second sensor value acquired by the second acquisition unit 62 while maintaining the offset state (offset amount).

[0068] (Step S8) The output unit 69 resumes the remote operation by outputting a drive signal based on the robot operation command generated by the operation command generation unit 64 to the robot 2.

[0069] As described above, in this embodiment, after acquiring the operation trigger, instead of immediately generating a robot operation command based on the detected value and resuming the remote operation, the robot operation command is not transmitted to the robot 2 until the offset amount becomes less than the threshold value, and the remote operation is not resumed.

[0070] In this embodiment, the reason for leaving the offset amount less than the threshold value is that although a smaller offset amount is easier for the operator Us to operate, it is difficult and time-consuming to completely align when the threshold value is, for example, 0. By resuming with a certain offset amount remaining as in this embodiment, the confusion of the operator Us can be reduced, and it is possible to easily align and quickly resume the work.

[0071] However, depending on the operator Us or the task, even if the offset amount is less than the threshold value, the operator Us may feel that it is difficult to perform the actual operation. Therefore, in this embodiment, in addition to the above processing, as follows, the offset is set to 0 in response to the instruction of the operator Us (second processing), and the remote operation control device 6 automatically reduces the offset amount (third processing).

[0072] [Images presented during the second and third processes] Next, an example of the image presented on the HMD 4 during the second and third processes will be described. FIG. 7 is a diagram showing an example of the image presented on the HMD during the second and third processes.

[0073] Image g310 is a diagram showing the first state. In the first state example, it shows that the operation is performed while leaving the offset.

[0074] Image g320 is a diagram showing the second state. This example is an example in which the offset amount is set to 0, for example, in response to an instruction from the operator Us. Image g321 is an example of an image showing that an offset deletion instruction is received in response to the result of the operator Us operating the operation unit 5. In the second state example, it shows that the offset amount is set to 0 as shown by the arrow g322. Note that the arrow g322 may or may not be displayed on the HMD 4.

[0075] Image g330 is a diagram showing the third state. This example is an example in which the remote operation control device 6 automatically reduces the offset amount. Image g331 is an example of an image presenting that the offset amount is automatically reduced. In the third state example, it shows that the offset amount is reduced by a predetermined amount as shown by the arrow g332. Note that the arrow g332 may or may not be displayed on the HMD 4.

[0076] The image g340 is a diagram showing the fourth state. The fourth state represents the position based on the instruction of the operator Us after setting the offset amount to 0 in the second state and the position of the hand 221 of the robot 2. Or, in the third state, it represents the position based on the instruction of the operator Us after reducing the offset amount by a predetermined amount and the position of the hand 221 of the robot 2. Note that in the image g340, the difference between the positions is exaggerated for easy understanding when illustrated. Also, during remote operation, the image g202 showing virtual positions and postures may not be displayed, or may be displayed or erased according to the instruction of the operator Us.

[0077] Note that in the example described with reference to FIG. 7, an example of presenting an image on the HMD4 is shown, but it is not limited to this. The guide may be a voice guide.

[0078] [Second Process] Next, an example of the second processing procedure performed by the remote operation control device 6 will be described. The following processing is performed after the first processing (steps S1 to S8) described with reference to FIG. 6. FIG. 8 is a flowchart of the second processing procedure performed by the remote operation control device according to the present embodiment. The following processing is repeated, for example, by acquiring each sensor value every several hundred msec.

[0079] (Step S101) The reduction command output unit 66 determines whether the first acquisition unit 61 has acquired an offset reduction instruction from the operation unit 5. When the reduction command output unit 66 has acquired an offset reduction instruction (step S101; YES), it proceeds to the processing of step S102. When the reduction command output unit 66 has not acquired an offset reduction instruction (step S101; NO), it proceeds to the processing of step S103.

[0080] (Step S102) The operation command generation unit 64 sets the offset amount determined by the offset detection unit 63 to, for example, 0. After the processing, the operation command generation unit 64 proceeds to the processing of step S104.

[0081] (Step S103) The operation command generation unit 64 retains the offset amount. After the process, the operation command generation unit 64 proceeds to the process of step S104.

[0082] (Step S104) After step S102, the operation command generation unit 64 uses the second sensor value acquired by the second acquisition unit 62 to generate a robot operation command with the offset amount set to 0. Alternatively, after step S103, the operation command generation unit 64 uses the second sensor value acquired by the second acquisition unit 62 to generate a robot operation command while retaining the offset amount.

[0083] (Step S105) The output unit 69 outputs a drive signal based on the robot operation command generated by the operation command generation unit 64 to the robot 2. Alternatively, the output unit 69 outputs the robot operation instruction generated by the operation command generation unit 64 to the robot 2. Note that this drive signal or robot operation instruction includes an operation instruction to set the position and posture of, for example, the hand 211 of the robot 2 to an offset amount of 0 when the offset amount is set to 0.

[0084] Note that the timing at which the operator Us issues an offset reduction instruction is, for example, when the operation is resumed and while viewing the image displayed on the HMD4, when it is determined that there is no risk of interference between the hand 211 of the robot 2 and an object (target object, table, tool, wall, etc.) around the robot 2. In this way, in the second process, when an offset reduction command is received, the remaining offset amount is reduced or eliminated by moving the robot 2.

[0085] [Third Process] Next, an example of the third processing procedure performed by the remote operation control device 6 will be described. Note that the following process is performed after the first process (steps S1 to S8) described with reference to FIG. 6. FIG. 9 is a flowchart of the third processing procedure performed by the remote operation control device according to the present embodiment. Note that the following process is repeated, for example, by acquiring each sensor value every several hundred msec.

[0086] (Step S201) The state determination unit 65 calculates, for example, the virtual position and orientation of the hand 211 at the time of its operation instruction based on the operation input value acquired by the second acquisition unit 62. Subsequently, the state determination unit 65 uses the detection data acquired by the first acquisition unit 61 from the environment sensor 3 to detect the position of an object within a predetermined range including, for example, the reachable range of the hand 211 in the robot work space by performing well-known image processing. Subsequently, the state determination unit 65 determines the relative relationship (e.g., the distance between each other) between the calculated virtual position and orientation and the position of the object within the predetermined range including the reachable range of the hand 211 detected from the detection data of the environment sensor 3 in the robot work space.

[0087] (Step S202) The state determination unit 65 determines whether there is a possibility of interference between the hand 211 and the environment based on the relative relationship obtained in Step S201. If there is a possibility of interference between the hand 211 and the environment (Step S202; NG), the state determination unit 65 proceeds to the process of Step S204. If there is no possibility of interference between the hand 211 and the environment (Step S202; OK), the state determination unit 65 proceeds to the process of Step S203.

[0088] (Step S203) Since there is no risk of drying, the operation command generation unit 64 sets the offset amount determined by the offset detection unit 63 to be reduced by a predetermined amount (e.g., 5%). After the process, the operation command generation unit 64 proceeds to the process of Step S205.

[0089] (Step S204) Since there is a risk of interference, the operation command generation unit 64 leaves the offset amount as it is. After the process, the operation command generation unit 64 proceeds to the process of Step S205.

[0090] (Step S205) After Step S203, the operation command generation unit 64 uses the second sensor value acquired by the second acquisition unit 62 to reduce the offset amount by a predetermined amount and generate a robot operation command. Or, after Step S204, the operation command generation unit 64 uses the second sensor value acquired by the second acquisition unit 62 to leave the offset amount as it is and generate a robot operation command.

[0091] (Step S206) The output unit 69 outputs a drive signal based on the robot operation command generated by the operation command generation unit 64 to the robot 2. Alternatively, the output unit 69 outputs the robot operation instruction generated by the operation command generation unit 64 to the robot 2. Note that this drive signal or robot operation instruction includes an operation instruction to reduce the offset amount of, for example, the position and posture of the hand 211 of the robot 2 by a predetermined amount when reducing the offset amount by a predetermined amount.

[0092] As described above, in this embodiment, it is determined whether to reduce or leave the offset amount according to the relative relationship between the robot 2 and the environment. And in this embodiment, when reducing, a robot control command with the reduced offset amount is generated, and the adjusted (or unadjusted) robot operation command is transmitted to the robot 2 to move the robot 2 to reduce the offset amount.

[0093] Note that in the example described with reference to FIG. 9, an example of reducing the offset value by a predetermined value, for example, 5% has been described, but it is not limited to this. The offset amount to be reduced may be associated with a predetermined amount to be reduced for each type of task and stored in the storage unit 70. Also, the predetermined amount is not limited to %, and may be, for example, a length in mm, or large, medium, or small representing a degree. Also, in the example of FIG. 9, the state determination unit 65 may set the offset amount to 0. Furthermore, the state determination unit 65 may be configured to select a predetermined amount step by step according to the proximity of the distance between the hand 211 of the robot 2 and an object in the environment or the like.

[0094] In the above-described example, an example of obtaining the relative relationship between the hand 211 and the environment has been described, but the present invention is not limited thereto. The state determination unit 65 may also estimate the virtual posture of the robot 2 using the position of the fingertip, the angle of the finger, the angle of the wrist, etc. acquired from the operation unit 5 for other parts of the robot 2, such as the legs, the body, and the head. The estimation may be performed, for example, by inputting the "position of the fingertip, the angle of the finger, the angle of the wrist" acquired from the operation unit 5 and the "teacher data" into the model, and outputting the posture of the robot. Then, the learning is repeated until the difference between the "teacher data" and the output data falls within a predetermined range to train the model. Then, the state determination unit 65 may estimate the virtual posture of the robot (including the joint angles of the fingers, the joint angles of the wrists, the joint angles of the arms, the joint angles of the arms, the inclination and position of the body) using the model trained in this way.

[0095] In addition, in each of the above-described embodiments, an offset amount may be set and left. The reason for this is that, for example, when the hand of the operator Us remains in a low position and the position of the hand 211 of the robot 2 is high, it may be easier to perform remote operation by setting an offset amount. In this way, the remote operation control device 6 may increase the threshold value or deliberately leave an offset amount according to the type of task, the working environment, and the intention of the operator Us.

[0096] Also, in the examples of FIGS. 8 and 9, an example of performing the second process or the third process after the first process has been described, but the present invention is not limited thereto. The second process and the third process may be performed without the first process. In this case, in the second process, after performing steps S1 to S3, the processes after step S101 may be performed. And in the third process, after performing steps S1 to S3, the processes after step S201 may be performed.

[0097] In the above-described example, for the sake of simplicity, only one hand 211 was used for the description. However, it is not limited to this. As shown in FIG. 1, when remotely operating a robot 2 having both arms, when remotely operating both arms, the offsets of both the first end effector 21L and the second end effector 21R may be detected simultaneously, or the offsets may be detected one by one in order. Note that whether to perform the detection simultaneously or in order may be selected, for example, by an operator Us operating the operation unit 5.

[0098] Also, in the above-described example, a hand or an end effector was used as an example for the description. However, it is not limited to this. For example, when remotely operating a walkable biped robot, for example, even for the legs, the offset may be reduced or deleted in response to an offset deletion instruction as described above or based on the relative relationship with the environment.

[0099] As described above, in the present embodiment, when the operator Us resumes the operation, first, a guide image is displayed and the operator Us is made to operate so that the offset amount becomes equal to or less than the threshold value. And in the present embodiment, after resuming, in response to an offset reduction instruction from the operator Us, the robot 2 is moved to delete the offset amount or set it to 0. Alternatively, in the present embodiment, the robot 2 is moved to delete the offset amount or set it to 0 according to the relative relationship between the robot 2 and an object in the environment of the robot 2.

[0100] Thereby, according to the present embodiment, when resuming the operation after interrupting the operation during the remote operation, it is possible to prevent the operation from being resumed with a large offset amount between the current state of the robot 2 (for example, the position and posture of the hand 211) and the state of the operation unit 5. Thereby, according to the present embodiment, it is possible to prevent the operator Us from causing confusion in the operation due to a large offset amount and a large deviation between the operation instruction and the operation of the robot 2. And according to the present embodiment, after resuming, the robot 2 can be appropriately remotely operated, so that it is possible to reconcile the environmental safety and the ease of operation of the human operator and resume the interrupted remote operation.

[0101] Also, according to this embodiment, the offset remaining after resumption is automatically reduced according to the instruction of the operator Us or according to the relative relationship between the robot 2 and the environment. Thereby, according to this embodiment, at the time of resumption, remote operation can be performed safely and with a reduced offset amount.

[0102] As described above, in each of the above-described embodiments, if a certain condition is satisfied from the offset state, the robot receives a command to cancel the offset. The certain condition includes those based on an operation command from the operator and those cancelled when the system determines that there is no risk of interfering with the surrounding environment. Also, when canceling the offset state, it may be gradually reduced. Thereby, according to each of the above-described embodiments, it is possible to safely reduce the offset to zero or deliberately leave it without causing coordinate confusion of the operator Us.

[0103] Note that a program for realizing all or part of the functions of the remote operation control device 6 in the present invention is recorded on a computer-readable recording medium, and the program recorded on this recording medium is read into a computer system and executed, whereby all or part of the processing performed by the remote operation control device 6 may be performed. Here, the “computer system” is assumed to include hardware such as an OS and peripheral devices. Also, the “computer system” is assumed to include a WWW system having a homepage providing environment (or display environment). Further, the “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” also includes a volatile memory (RAM) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and holds the program for a certain period of time. Alternatively, some or all of these components may be implemented by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), SOC (System On Chip), or may be implemented by the cooperation of software and hardware.

[0104] Also, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) like the Internet or a communication line (communication wire) like a telephone line. Further, the above program may be for realizing a part of the aforementioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the aforementioned functions in combination with a program already recorded in the computer system.

[0105] As described above, the embodiments for implementing the present invention have been described using the embodiments. However, the present invention is not limited to such embodiments, and various modifications and substitutions can be made without departing from the gist of the present invention.

Description of Reference Numerals

[0106] 1... Robot remote operation system, 2... Robot, 3... Environmental sensor, 4... HMD, 5... Operation unit, 6... Remote operation control device, 21... End effector, 21L... First end effector, 21R... Second end effector, 211, 211L, 211R... Hand, 212, 212L, 212R... Actuator, 213, 213, 213L, 213R... Sensor, 22... Communication unit, 31... Sensor, 32... Communication unit, 41... Image display unit, 42... Line-of-sight detection unit, 43... Communication unit, 51... Sensor, 52... Trigger SW, 53... Communication unit, 61... First acquisition unit, 62... Second acquisition unit, 63... Offset detection unit, 64... Operation command generation unit, 65... State determination unit, 66... Reduction command output unit, 67... Drive circuit, 68... Image generation unit, 69... Output unit, 70... Memory unit, 71... Threshold setting unit, NW... Network

Claims

1. A robot that is remotely operated, a first acquisition unit that acquires the state of the robot, an operation unit through which an operator inputs an operation command for the robot, a second acquisition unit that acquires the operation input value of the operation unit, an operation command generation unit that generates an operation command from the operation input value, and comprising when the position based on the operation input value and the position of the state of the robot are different, the operation command generation unit determines a different state as an offset, and generates a robot operation command so as to reduce the offset after receiving a command to reduce the state of the offset. A robot remote operation system.

2. When there is a deviation greater than or equal to a threshold value between the position based on the operation input value and the position of the state of the robot, the operation command generation unit does not output a robot operation command to the robot until the position based on the operation input value and the position of the state of the robot are within the threshold value. The robot remote operation system according to Claim 1.

3. When the position based on the operation input value and the position of the state of the robot are less than the threshold value, the operation command generation unit outputs the robot operation command to the robot. The robot remote operation system according to Claim 2.

4. further comprising a state determination unit that determines the state of the robot, when the state determination unit determines that the robot is in a state where it does not interfere with the environment, the state determination unit outputs an instruction to reduce the offset state. The robot remote operation system according to Claim 1 or Claim 2.

5. When the operation command generation unit acquires an instruction from the operator to reduce the offset, the operation command generation unit sets the offset to zero. The robot remote operation system according to Claim 1 or Claim 2.

6. further comprising a reduction command output unit that outputs a command to reduce the offset state, the operation command generation unit generates the robot operation command by leaving the offset or reducing the offset in response to the output of the reduction command output unit. The robot remote operation system according to Claim 1 or Claim 2.

7. when the position based on the operation input value and the position of the state of the robot are different, further comprising an image generation unit that generates and presents a guide image for moving the position based on the operation input value and the position of the state of the robot so that the difference therebetween is less than the threshold value. When the operation instruction generation unit changes the position and posture of the operation unit according to the guide image such that the difference between the position based on the operation input value and the position of the robot state is less than the threshold value, and when an instruction to reduce the offset is obtained from the operator, the operation instruction generation unit outputs a robot operation instruction that sets the offset to zero to change the position and posture of the robot. The robot remote operation system according to claim 1 or claim 2.

8. A remote operation control device for remotely operating a robot, a first acquisition unit that acquires the state of the robot; an operation unit through which an operator inputs an operation instruction for the robot; a second acquisition unit that acquires an operation input value of the operation unit; an operation instruction generation unit that generates an operation instruction from the operation input value; and includes When the position based on the operation input value and the position of the robot state are different, the operation instruction generation unit determines a different state as an offset, and after receiving an instruction to reduce the state of the offset, generates a robot operation instruction to eliminate the offset. Remote operation control device.

9. A remote operation control method for remotely operating a robot, wherein a first acquisition unit acquires the state of the robot, a second acquisition unit acquires an operation input value through which an operator inputs an operation instruction for the robot, an operation instruction generation unit generates an operation instruction from the operation input value, and when the position based on the operation input value and the position of the robot state are different, the operation instruction generation unit determines a different state as an offset, and after receiving an instruction to reduce the state of the offset, generates a robot operation instruction to eliminate the offset. Remote operation control method.

10. Causing a computer of a remote operation control device for remotely operating a robot to acquire the state of the robot, acquire an operation input value through which an operator inputs an operation instruction for the robot, generate an operation instruction from the operation input value, when the position based on the operation input value and the position of the robot state are different, determine a different state as an offset, and after receiving an instruction to reduce the state of the offset, generate a robot operation instruction to eliminate the offset. Program.

Images