Robot remote operation system, remote operation control device, remote operation control method, and program

Abstract

To provide a robot remote operation system, a remote operation control device, a remote operation control method, and a program capable of restarting interrupted remote operation while satisfying both safety of an environment and operability of a human operator.SOLUTION: The robot remote operation system includes: a robot to be remotely operated; a first acquisition unit that acquires a state of the robot; an operation unit to which an operator inputs an operation command of the robot; a second acquisition unit that acquires an operation input value to the operation unit; and an operation command generation unit that generates an operation command from the operation input value. When a position based on the operation input value and a position of the state of the robot are different from each other, the operation command generation unit determines the different state as an offset, and generates the robot operation command, with the offset state maintained.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a robot remote operation system, a remote operation control device, a remote operation control method, and a program.

Background Art

[0002] In recent years, technologies for remotely operating robots have been developed. For example, Patent Document 1 discloses a technique for detecting an error when an error occurs in such a system and releasing the error to resume commands.

[0003] When remotely operating a robot, for example, there are a method in which an operator closely attaches a sensor such as an exoskeleton to the arm or hand and operates it, and a method in which the operator wears a data glove or the like on the hand and operates it. For example, when an operator remotely operates a robot, the operator may want to interrupt the work at will, for example, during a break time. When the method of closely attaching a sensor such as an exoskeleton to the arm or hand by the operator is used, the remote operation can be resumed in accordance with the state of the robot by reattaching it when resuming while maintaining the shape of a large number of attached sensors when the operator interrupts the remote operation. On the other hand, in the method of wearing a data glove or the like on the hand and operating it, the operator needs to adjust to the state before resuming.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technology described in Patent Document 1, only a technology for automatically resuming a command at the time of an error is described, and when resuming work after interrupting the work at the operator's discretion, the work could not be resumed. Further, in the technology described in Patent Document 1, if the same processing as at the time of an error is performed, if the operation instructions before and after resumption are different, there is also a risk that the operator cannot perform remote operation or that the robot cannot be operated safely. In addition, in the prior art in which sensors such as exoskeletons are closely attached to the arms and hands for operation, it is very difficult to wear the exoskeleton, and the convenience such as wearability is poor for the operator. Furthermore, in the prior art in which a data glove or the like is worn for operation, it is troublesome and difficult for the operator to match the state of the joint angles of the robot before interruption, and if it cannot be matched, a deviation occurs between the instructions of the robot and the operator, making it difficult to work and there is also a risk that the robot cannot be operated safely.

[0006] The present invention has been made in view of the above problems, and an object thereof is to provide a robot remote operation system, a remote operation control device, a remote operation control method, and a program that can resume an interrupted remote operation while achieving both environmental safety and ease of operation for a human operator.

Means for Solving the Problems

[0007] (1) To achieve the above object, a robot remote operation system according to an aspect of the present invention includes a robot to be remotely operated, a first acquisition unit that acquires the state of the robot, an operation unit through which an operator inputs an operation command for the robot, a second acquisition unit that acquires an operation input value of the operation unit, and an operation command generation unit that generates an operation command from the operation input value. When the position based on the operation input value is different from the position of the state of the robot, the operation command generation unit determines different states as an offset and generates a robot operation command while maintaining the offset state.

[0008] (2) In the robot remote operation system according to one aspect of the present invention as described in (1) above, when there is a deviation of a threshold value or more between the position based on the operation input value and the position of the state of the robot, the operation command generation unit may not output the robot operation command to the robot until the position based on the operation input value and the position of the state of the robot are within the threshold value.

[0009] (3) In the robot remote operation system according to one aspect of the present invention as described in (2) above, when the difference between the position based on the operation input value and the position of the state of the robot becomes less than the threshold value, the operation command generation unit may output the robot operation command to the robot.

[0010] (4) The robot remote operation system according to one aspect of the present invention as described in (2) or (3) above further includes a threshold setting unit that sets the threshold value. When the operator wants to resume work at a high place with the robot's hand while keeping their own hand low, the threshold setting unit may set a large threshold value for the position.

[0011] (5) The robot remote operation system according to one aspect of the present invention as described in (2) or (3) above further includes a threshold setting unit that sets the threshold value. The threshold setting unit may select the threshold value according to at least one of the type of task, the age of the operator, the weight of the operator, and the length of the arm of the robot.

[0012] (6) The robot remote operation system according to one aspect of the present invention as described in (2) or (3) above further includes a threshold setting unit that sets the threshold value. The threshold setting unit may present a plurality of the preset threshold values to the operator and set the threshold value selected by the operator from among the plurality of threshold values.

[0013] (7) In the robot remote operation system according to any one of the above (2) to (6) aspects of the present invention, an image generation unit that generates an image to be presented to the operator using the state of the robot acquired by the first acquisition unit and the operation command of the robot acquired by the second acquisition unit is further provided. The image generation unit may generate a guide image indicating a change direction that needs to be changed so that the difference between the information acquired by the second acquisition unit and the position information of the state of the robot becomes a position less than the threshold value, and cause the generated guide image to be presented to the operator.

[0014] (8) To achieve the above object, a remote operation control device according to an aspect of the present invention is a remote operation control device for remotely operating a robot, comprising a first acquisition unit that acquires the state of the robot, a second acquisition unit that acquires an operation input value in which an operator inputs an operation command of the robot, and an operation command generation unit that generates an operation command from the operation input value acquired by the second acquisition unit. When the position based on the operation input value and the position of the state of the robot are different, the operation command generation unit determines a different state as an offset and generates a robot operation command while maintaining the offset state.

[0015] (9) To achieve the above object, a remote operation control method according to an aspect of the present invention is a remote operation control method for remotely operating a robot, in which a first acquisition unit acquires the state of the robot, a second acquisition unit acquires an operation input value in which an operator inputs an operation command of the robot, an operation command generation unit generates an operation command from the operation input value, and when the position based on the operation input value and the position of the state of the robot are different, the operation command generation unit determines a different state as an offset and generates a robot operation command while maintaining the offset state.

[0016] (10)To achieve the above object, a program according to an aspect of the present invention causes a computer of a remote operation control device that remotely operates a robot to acquire the state of the robot, acquire an operation input value in which an operator inputs an operation command for the robot, generate an operation command from the operation input value, and when a position based on the operation input value is different from the position of the state of the robot, determine a different state as an offset, and generate a robot operation command while maintaining the offset state.

Effects of the Invention

[0017] According to the above (1) to (10), it is possible to resume the interrupted remote operation while achieving both environmental safety and ease of operation for a human operator. According to the above (4), it is possible to resume while maintaining a large positional offset. According to the above (5), a threshold value can be set according to the type of task or the like. According to the above (6), a threshold value according to the preference of the operator can be set. According to the above (7), while the operator looks at the guide, the state (for example, position and posture) of the operation unit can be appropriately brought close to the state of the robot.

Brief Description of the Drawings

[0018]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each member is appropriately changed in order to make each member recognizable in size. In all the drawings for explaining the embodiments, those having the same function are denoted by the same reference numerals, and repeated explanations are omitted. In addition, “based on XX” as used in the present application means “based on at least XX”, and includes cases based on another element in addition to XX. Also, “based on XX” is not limited to the case of directly using XX, and includes cases based on something obtained by performing operations or processing on XX. “XX” is an arbitrary element (for example, arbitrary information).

[0020] [Overview of Remote Operation and Object to be Operated] First, an overview of the remote operation and the object to be operated will be described. FIG. 1 is a diagram for explaining an overview of the remote operation of a robot and an object to be operated. As shown in FIG. 1, in the remote operation space, an operator Us wears, for example, an HMD (head-mounted display) 4 on the head and wears operation units 5 (5L, 5R) such as data gloves on the hands. An environmental sensor 3 is installed in the robot work space. Note that the environmental sensor 3 may be attached to the robot 2. The robot 2 includes a remote operation control device 6, end effectors 21 (first end effector 21L, second end effector 21R), and hands 211 (211L, 211R).

[0021] The object to be operated obj is composed of a plurality of objects. In the example of FIG. 1, the object to be operated obj is, for example, a plastic bottle or a bottle, and includes a body body and a lid cap. The operator Us remotely operates the robot 2 to operate the object to be operated obj by moving the hand or finger wearing the operation unit 5 while looking at the image displayed on the HMD 4, for example. An example of the operation content is, as shown in FIG. 1, attaching and closing the lid cap to the body, or opening the lid cap from the body, etc.

[0022] Or, as shown in FIG. 2, pick up a screw which is the first target object obj1, move it above the second target object obj2 which is the object to attach the screw to, and then tighten the screw to the second target object obj2 using a driver. FIG. 2 is a diagram showing an example of the work by remote operation.

[0023] In the following embodiments, when temporarily interrupting and then resuming the remote operation, resume it in the operation mode that has been used so far. The operation modes are, for example, a free movement mode and a restraint mode. The free movement mode is a mode in which, for example, the hand 211 etc. is not restrained and can be freely moved according to the instruction of the operator Us. The restraint mode is a mode in which, for example, it is restrained in a part of dimensions (for example, the z-axis direction).

[0024] [Configuration of Robot Remote Operation System] Next, a configuration example of the robot remote operation system 1 will be described. FIG. 3 is a diagram showing a configuration example of the robot remote operation system according to the present embodiment. As shown in FIG. 3, the robot remote operation system 1 includes, for example, a robot 2, an environmental sensor 3, an HMD 4, an operation unit 5, and a remote operation control device 6.

[0025] The robot 2 includes, for example, a first end effector 21L, a second end effector 21R, and a communication unit 22. The first end effector 21L includes, for example, a hand 211L, an actuator 212L, and a sensor 213L. The second end effector 21R includes, for example, a hand 211R, an actuator 212R, and a sensor 213R. Note that the robot 2 may include a communication unit, a power supply unit, a body, legs, a head, etc. not shown in the figure. In the following description, when the first end effector 21L and the second end effector 21R are not distinguished, they are also referred to as the end effector 21. Similarly, when the hand 211L and the hand 211R are not distinguished, they are also referred to as the hand 211, when the actuator 212L and the actuator 212R are not distinguished, they are also referred to as the actuator 212, and when the sensor 213L and the sensor 213R are not distinguished, they are also referred to as the sensor 213. The robot 2 transmits and receives various information to and from the remote control device 6 via a wired or wireless network NW.

[0026] The environment sensor 3 includes, for example, a sensor 31 and a communication unit 32. Note that the environment sensor 3 also includes a power supply unit (not shown) and the like. The environment sensor 3 transmits and receives various information to and from the remote control device 6 via a wired or wireless network NW.

[0027] The HMD 4 includes, for example, an image display unit 41, a gaze detection unit 42, and a communication unit 43. Note that the HMD 4 also includes a power supply unit (not shown) and the like. The HMD 4 transmits and receives various information to and from the remote control device 6 via a wired or wireless network NW.

[0028] The operation unit 5 includes, for example, a sensor 51, a trigger SW 52, and a communication unit 53. The operation unit 5 transmits information to the remote control device 6 via a wired or wireless network NW.

[0029] The remote control device 6 includes, for example, a first acquisition unit 61, a second acquisition unit 62, an offset detection unit 63, an operation command generation unit 64, a drive circuit 65, an image generation unit 66, a threshold setting unit 67, an output unit 68, and a storage unit 69. Note that the remote control device 6 also includes a power supply unit (not shown) and the like. The remote control device 6 transmits and receives various information to and from the robot 2 and the HMD 4 via a wired or wireless network NW. The remote control device 6 receives various information from the environment sensor 3 and the operation unit 5 via a wired or wireless network NW.

[0030] Note that the configuration example shown in FIG. 3 is just an example and is not limited to this.

[0031] [Functions of Each Device in the Robot Remote Operation System] Next, with reference to FIG. 3, the functions of each device in the robot remote operation system 1 will be described. (Robot 2) The hand 211 includes, for example, a plurality of finger portions. Each finger portion has a joint. Note that the hand 211 may be a gripper or the like. The actuator 212 is attached to each joint. The sensor 213 is, for example, a six-axis sensor attached to a joint, a tactile sensor attached to a finger portion, or the like. The six-axis sensor detects forces in three axes (x, y, z) and moments in three axes (α, β, γ). The communication unit 22 transmits the detection value detected by the sensor 213 to the remote operation control device 6. The communication unit 22 receives a control signal or a control instruction output by the remote operation control device 6. Note that the data output by the robot 2 includes identification information that can identify the robot 2. Also, the data acquired by the robot 2 includes identification information that can identify that it is addressed to the robot 2. Note that the robot 2 may be provided with a drive circuit that drives the actuator 212.

[0032] (Environmental Sensor 3) As shown in FIG. 1, the environmental sensor 3 is installed, for example, in the robot work space. The sensor 31 is, for example, an RGB-D camera that acquires RGB (red, green, blue) information and depth information. Note that the acquisition of information is performed, for example, at a predetermined time interval. The communication unit 32 outputs the detection value detected by the sensor 31 to the remote operation control device 6. Note that the data output by the environmental sensor 3 includes identification information that can identify the environmental sensor 3.

[0033] (HMD 4) The image display unit 41 displays the image data acquired by the communication unit 43 from the remote operation control device 6. The line-of-sight detection unit 42 detects the line-of-sight direction and movement of the operator Us. The communication unit 43 transmits the line-of-sight information detected by the line-of-sight detection unit 42 to the remote operation control device 6. The communication unit 43 receives the image data output by the remote operation control device 6. Note that the data output by the HMD 4 includes identification information that can identify the HMD 4. Also, the data acquired by the HMD 4 includes identification information that can identify that it is addressed to the HMD 4.

[0034] (Operation unit 5) The operation unit 5 is, for example, a data glove. The sensor 51 detects the finger joint angles and wrist joint angles of the operator Us's hand. The trigger SW 52 is, for example, a mechanical switch. The trigger SW 52 detects the operation of the operator Us. Note that the operator Us selects and inputs which operation it is, for example, when starting an operation, interrupting an operation, resuming an operation (returning to an operation), or ending an operation, by pressing a dedicated switch or a predetermined number of times of the switch. Note that the operator Us may select and input the above by performing a predetermined gesture, for example. The communication unit 53 outputs to the remote operation control device 6 the detection value detected by the sensor 51 and the operation trigger signal indicating the detection result detected by the trigger SW 52. Note that the data output by the operation unit 5 includes identification information that can identify the HMD 4.

[0035] (Remote operation control device 6) The first acquisition unit 61 acquires, for example, an operation trigger signal from the operation unit 5. The first acquisition unit 61 acquires the first sensor value detected by the sensor 213 from the robot 2. The first acquisition unit 61 acquires the first sensor value detected by the environmental sensor 3.

[0036] The second acquisition unit 62 acquires the second sensor value detected by the sensor 51 of the operation unit 5. The second acquisition unit 62 acquires the second sensor value detected by the line-of-sight detection unit 42 of the HMD 4.

[0037] The offset detection unit 63 detects, for example, the difference between the position and orientation of the hand 211 of the end effector 21 of the robot 2 and the position and orientation of the hand of the operator Us. Note that the detection of the difference in position and orientation is performed, for example, in the robot coordinate system. For coordinate system conversion, refer to, for example, Japanese Patent Application No. 2023-031054. The offset detection unit 63 compares the detected offset amount with the threshold value stored in the storage unit 69, and determines the current offset amount when the offset amount is less than the threshold value. Note that the detection method example will be described later.

[0038] After the offset detection unit 63 determines the offset amount, the operation command generation unit 64 generates a robot operation command (also simply referred to as an "operation instruction") based on the second sensor value acquired by the second acquisition unit 62.

[0039] The drive circuit 65 generates a drive signal for controlling the robot 2 based on the robot control command generated by the operation command generation unit 64. Note that when the robot 2 is provided with the drive circuit 65, the remote operation control device 6 does not necessarily need to be provided with the drive circuit 65. Alternatively, the remote operation control device 6 and the robot 2 may each be provided with a part of the drive circuit 65.

[0040] The image generation unit 66 generates an image to be provided to the HMD 4 based on the image captured by the environment sensor 3 and the detection value detected by the offset detection unit 63. Note that for the method of generating the image displayed on the HMD 4 and examples of the image, for example, the method described in Japanese Patent Application No. 2022-156322 may also be used. Examples of the image presented to the HMD 4 will be described later.

[0041] The threshold setting unit 67 sets a threshold value and stores the set threshold value in the storage unit 69. Note that the number of threshold values is not limited to one, and may be plural, for example, according to the application. The method of setting the threshold value will be described later.

[0042] The output unit 68 outputs the image data generated by the image generation unit 66 to the HMD 4. The output unit 68 outputs the drive signal output by the drive circuit 65 or the operation command generated by the operation command generation unit 64 to the robot 2.

[0043] The memory unit 69 stores, for example, programs, mathematical formulas, threshold values, identification information of the robot 2, identification information of the environment sensor 3, identification information of the HMD 4, identification information of the operation unit 5, etc. used by each part of the remote operation control device 6.

[0044] [Offset Detection] Next, the offset detection method will be described. FIG. 4 is a diagram for explaining the offset detection method. The coordinate system is the robot coordinate system. In FIG. 4, a gripper is shown as an example of the hand 211, but the hand 211 may have three or more finger parts. When the operator Us performs remote operation, the operation unit 5 and the HMD 4 are worn in advance, and calibration is performed with the hand 211 of the robot 2.

[0045] In FIG. 4, the image g101 is an image showing the current state (position and posture) of the hand 211 of the robot 2. The image g102 is the state of the hand 211 of the robot 2 based on the operation command virtually generated by the operation command generation unit 64 based on the detection value acquired from the operation unit 5, and is also referred to as a "VR marker" in the following description.

[0046] First, the offset detection unit 63 uses the detection value acquired by the second acquisition unit 62 from the sensor 51 of the operation unit 5 to obtain the position and posture of the fingertip from the angle of the finger joint by, for example, forward kinematics calculation of the finger (see, for example, Reference 1). Note that the position and posture of this fingertip are based on the instructed position and posture of the operation unit 5 of the operator Us. Next, the offset detection unit 63 converts the obtained position and posture of the fingertip into the robot coordinate system. Next, the offset detection unit 63 calculates the difference between the current position and orientation of the fingertip of the hand 211 of the robot 2 and the position and orientation of the fingertip converted into the robot coordinate system. For example, Roll, Pitch, and Yaw for each of the x, y, and z axes are calculated using the rotation matrix components and the translation matrix components (see, for example, Reference 2). Hereinafter, in the embodiment, the "difference between the current position and orientation of the fingertip of the hand 211 of the robot 2 and the position and orientation of the fingertip converted into the robot coordinate system" is referred to as the "offset state" or the "offset amount".

[0047] Reference 1; Masaharu Takano, Toshiaki Nagashima, et al., "Grasping and Manipulating Kinematics by the Finger Pad", The Robotics Society of Japan, Transactions of the Robotics Society of Japan Vol.14 No.1, p83~90, 1996 Reference 2; Tokuo Tsuji, Hoshisei Baba, et al., "Evaluation of Grasping Stability Based on Potential Field Allowable External Force Energy with Flexible Fingertips and Joint Variations", The Robotics Society of Japan, The 33rd Annual Conference of the Robotics Society of Japan, RSJ2015AC1G3-02, 2015

[0048] Next, the offset detection unit 63 compares the calculated offset with the threshold value stored in the storage unit 69 and determines whether the offset is less than the threshold value. The offset detection unit 63 outputs the determined result to the image generation unit 66 and the operation command generation unit 64.

[0049] Note that the above-described method for calculating the offset is an example, and other methods may be used.

[0050] [Threshold value] Next, an example of how to determine the threshold value will be described. The threshold value may be determined according to the application. (I) From the viewpoint of not confusing the operator Us, when it is desired to start in a state where the deviation between the end effector 21 of the actual robot 2 and the operation unit 5 is small, a small value is set as the threshold value. (II) When the operator Us wants to resume the operation of the hand of the robot 2 at a high position while keeping their own hand low, the position threshold value is set large so that the operation can be resumed with a large positional offset. (III) Depending on at least one of the task type (high place, low place), the age of the operator, the weight of the operator, the length of the arm of the end effector 21 of the robot 2, etc., the threshold setting unit 67 automatically determines the threshold value. In this case, the storage unit 69 stores the threshold values associated with each task in advance.

[0051] Note that the above setting examples (I) to (III) of the threshold value are just examples, and other settings may be made according to the work content, the preferences of the operator Us, etc. For example, the threshold setting unit 67 may prepare and maintain low sensitivity (large threshold value), normal sensitivity (normal threshold value), high sensitivity (small threshold value), etc., and select and use them from the settings presented by the operator Us. Note that the threshold value is preferably within a range where the operator Us will not be confused even when operating with an offset amount at restart.

[0052] [Prompt Image] Next, an example of the image presented on the HMD 4 will be described. FIG. 5 is a diagram showing an example of the image presented on the HMD. In each of the images g210 to g240, the reference numeral g201 indicates the position and posture of the current hand 211 of the robot 2, and the reference numeral g202 indicates the virtual position and posture of the hand of the robot 2 based on the instruction of the operation unit 5 of the operator Us.

[0053] The image g210 is a diagram showing the first state. The image g211 is an example of a guide image that guides (advises) how the operation unit 5 of the operator Us should be moved to match the position and posture of the current hand 211 of the robot 2. In the first state example, it shows that the operation unit 5 of the operator Us is rotated clockwise by about 45 degrees and further moved downward.

[0054] The image g220 is a diagram showing the second state. The image g221 is an example of a guide image that guides how the operation unit 5 of the operator Us should be moved to match the position and posture of the current hand 211 of the robot 2. In the second state example, it shows that the operation unit 5 of the operator Us is moved diagonally downward to the left.

[0055] The image g230 is a diagram showing the third state. The image g231 is an example of a guide image that guides how the operation unit 5 of the operator Us should be moved to match the current position and posture of the hand 211 of the robot 2. In the example of the third state, it shows that the operation unit 5 of the operator Us is rotated clockwise by about 90 degrees and further moved upward.

[0056] The image g240 is a diagram showing the fourth state. The fourth state is, for example, a state in which, as a result of moving the operation unit 5 of the operator Us according to the guides of the first to third states, the current position and posture of the hand 211 of the robot 2 and the virtual position and posture of the hand of the robot 2 based on the instruction of the operation unit 5 of the operator Us are within a threshold value. The image g221 indicates that the position and posture of the operation unit 5 are appropriate (e.g., "Good!"). Note that during remote operation, the image g202 showing the virtual position and posture may not be displayed, or it may be displayed or erased according to the instruction of the operator Us.

[0057] Note that in the example described with reference to FIG. 5, an example of presenting a guide image on the HMD 4 is shown, but it is not limited to this. The guide may use sound, or may use sound as well. For example, when approaching the threshold value, the feeling of the sound ringing may be shortened. Or, the guide may be a voice guide.

[0058] [Processing Procedure] Next, an example of the processing procedure performed by the remote operation control device 6 will be described. FIG. 6 is a flowchart of the processing procedure performed by the remote operation control device according to the present embodiment. Note that the following processing is repeated by acquiring each sensor value, for example, every several hundred msec.

[0059] (Step S1) The first acquisition unit 61 determines, for example, whether an operation trigger signal has been acquired from the operation unit 5. When the first acquisition unit 61 has acquired the operation trigger signal (Step S1; YES), the process proceeds to the process of Step S2. When the first acquisition unit 61 has not acquired the operation trigger signal (Step S1; NO), the process of Step S1 is repeated.

[0060] (Step S2) The first acquisition unit 61 acquires the first sensor value detected by the sensor 213 from the robot 2. The second acquisition unit 62 acquires the second sensor value detected by the sensor 51 of the operation unit 5. The offset detection unit 63 calculates the position and orientation of the VR marker using the second sensor value acquired by the second acquisition unit 62. The offset detection unit 63 calculates the current position and orientation of, for example, the hand 211 of the robot 2 using the first sensor value acquired by the first acquisition unit 61.

[0061] (Step S3) The offset detection unit 63 calculates and detects the offset amount between the position and orientation of the VR marker and the current position and orientation of the hand 211 of the robot 2. The offset detection unit 63 obtains the offset amount, for example, by subtracting (the current position and orientation of the hand 211 of the robot 2) from (the position and orientation of the VR marker).

[0062] (Step S4) The offset detection unit 63 determines whether or not the obtained offset amount is less than the threshold value. If the offset amount is less than the threshold value (Step S4; YES), the offset detection unit 63 proceeds to the process of Step S5. If the offset amount is greater than or equal to the threshold value (Step S4; NO), the offset detection unit 63 proceeds to the process of Step S5. Note that the offset amount includes information indicating the offset amount of the orientation and the offset amount of the position, respectively.

[0063] (Step S5) The image generation unit 66 generates a guide image such as that in FIG. 5, for example, using the offset amount of the orientation and the offset amount of the position obtained by the offset detection unit 63 so as to facilitate adjusting the position and orientation of the operation unit 5. The image generation unit 66 generates an output image by, for example, superimposing the generated guide image on the image to be displayed on the HMD 4. The output unit 68 outputs the generated output image with the guide image to the HMD 4. After the process, the image generation unit 66 returns to the process of Step S2.

[0064] (Step S6) The offset detection unit 63 determines the offset amount that became less than the threshold value in Step S4 as the current offset amount.

[0065] (Step S7) After the offset detection unit 63 determines the offset amount, the operation command generation unit 64 generates a robot operation command based on the second sensor value acquired by the second acquisition unit 62 while maintaining the offset state (offset amount).

[0066] (Step S8) The output unit 68 resumes the remote operation by outputting a drive signal based on the robot operation command generated by the operation command generation unit 64 to the robot 2.

[0067] In this way, in the present embodiment, after acquiring the operation trigger, instead of immediately generating a robot operation command based on the detected value and resuming the remote operation, the robot operation command is not transmitted to the robot 2 until the offset amount becomes less than the threshold value, and the remote operation is not resumed.

[0068] In the present embodiment, the reason for leaving the offset amount less than the threshold value is that although a smaller offset amount is easier for the operator Us to operate, it is difficult and time-consuming to completely match when the threshold value is, for example, 0. By resuming with a certain offset amount remaining as in the present embodiment, confusion of the operator Us can be reduced, and it can be easily adjusted, and the work can be resumed quickly.

[0069] In each of the above-described embodiments, an offset amount may be set and left. The reason for this is that, for example, when the hand of the operator Us is in a low position and the position of the hand 211 of the robot 2 is high, it may be easier to perform remote operation by setting an offset amount. In this way, the remote operation control device 6 may increase the threshold value or deliberately leave an offset amount according to the type of task, the working environment, and the intention of the operator Us.

[0070] In the above example described with reference to FIGS. 4 and 5, for simplicity of explanation, only one hand 211 was used for the description, but it is not limited to this. As shown in FIG. 1, when remotely operating a robot 2 having both arms, when remotely operating both arms, the offsets of both the first end effector 21L and the second end effector 21R may be detected simultaneously, or the offsets may be detected one by one in order. Note that whether to perform simultaneously or in order may be selected, for example, by the operator Us operating the operation unit 5.

[0071] Further, in the above example, the hand and the end effector were used as examples for the description, but it is not limited to this. For example, when remotely operating a biped robot capable of walking, for example, regarding the legs, the operation may be resumed after the offset amount becomes less than the threshold value as described above. In this case, the first threshold value for the offset amount of the hand and the second threshold value for the legs may be the same or different.

[0072] As described above, in the present embodiment, when the operator Us resumes the operation, the operation is not started until the offset amount between the position and posture of the operation unit 5 and, for example, the position and posture of the hand 211 of the current robot 2 becomes less than the threshold value, and the operation is resumed, that is, resumed, after the offset amount becomes less than the threshold value. Further, in the present embodiment, before resuming the operation, a guide image as shown in FIG. 5 is displayed, for example, so as to easily align the position and posture of the operation unit 5.

[0073] Thereby, according to the present embodiment, when resuming the operation after interrupting the operation during the remote operation, it is possible to prevent the operation from being resumed while the offset amount between the state of the current robot 2 (for example, the position and posture of the hand 211) and the state of the operation unit 5 is large. Thereby, according to the present embodiment, it is possible to prevent the operator Us from causing a large deviation between the operation instruction from the operation unit 5 and the operation of the robot 2 due to a large offset amount, which may cause confusion in the operation. And according to the present embodiment, after resuming, the robot 2 can be appropriately remotely operated, so that it is possible to achieve both environmental safety and ease of operation for the human operator, and resume the interrupted remote operation. Further, according to the present embodiment, the operator can appropriately approximate the state (e.g., position and orientation) of the operation unit to the state of the robot while looking at the guide.

[0074] Here, if there is an object or the like around the robot 2 and the hand 211 is moved to adjust the offset amount, there is a possibility of interfering with the object or the like in such an environment. On the other hand, in the present embodiment, when making the offset amount less than the threshold value, the operation unit 5 is moved and adjusted instead of the robot 2, so that such interference can be avoided and restarted safely.

[0075] Note that a program for realizing all or part of the functions of the remote operation control device 6 in the present invention may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed to perform all or part of the processing performed by the remote operation control device 6. Here, the "computer system" is assumed to include hardware such as an OS and peripheral devices. Also, the "computer system" includes a WWW system having a homepage providing environment (or display environment). Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in a computer system. Furthermore, the "computer-readable recording medium" includes a volatile memory (RAM) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and that holds the program for a certain period of time. Alternatively, some or all of these components may be implemented by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or SOC (System On Chip), or may be implemented by cooperation between software and hardware.

[0076] Also, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) like the Internet or a communication line (communication wire) like a telephone line. Further, the above program may be for realizing a part of the functions described above. Furthermore, it may be a so-called difference file (difference program) that can realize the functions described above in combination with a program already recorded in the computer system.

[0077] As described above, the embodiments for implementing the present invention have been described using embodiments, but the present invention is not limited to such embodiments, and various modifications and substitutions can be made without departing from the gist of the present invention.

Description of Reference Numerals

[0078] 1... Robot remote operation system, 2... Robot, 3... Environmental sensor, 4... HMD, 5... Operation unit, 6... Remote operation control device, 21... End effector, 21L... First end effector, 21R... Second end effector, 211, 211L, 211R... Hand, 212, 212L, 212R... Actuator, 213, 213, 213L, 213R... Sensor, 22... Communication unit, 31... Sensor, 32... Communication unit, 41... Image display unit, 42... Line-of-sight detection unit, 43... Communication unit, 51... Sensor, 52... Trigger SW, 53... Communication unit, 61... First acquisition unit, 62... Second acquisition unit, 63... Offset detection unit, 64... Operation command generation unit, 65... Drive circuit, 66... Image generation unit, 67... Threshold setting unit, 68... Output unit, 69... Memory unit, NW... Network

Claims

1. A robot that is remotely operated, a first acquisition unit that acquires the state of the robot, an operation unit for an operator to input an operation command for the robot, a second acquisition unit that acquires an operation input value of the operation unit, an operation command generation unit that generates an operation command from the operation input value, comprising: when the position based on the operation input value and the position of the state of the robot are different, the operation command generation unit determines a different state as an offset and generates a robot operation command while maintaining the offset state. A robot remote operation system.

2. When there is a deviation of a threshold value or more between the position based on the operation input value and the position of the state of the robot, the operation command generation unit does not output the robot operation command to the robot until the position based on the operation input value and the position of the state of the robot are within the threshold value. The robot remote operation system according to Claim 1.

3. When the difference between the position based on the operation input value and the position of the state of the robot becomes less than the threshold value, the operation command generation unit outputs the robot operation command to the robot. The robot remote operation system according to Claim 2.

4. Further comprising a threshold setting unit that sets the threshold value, when the operator wants to resume work at a high place with the robot's hand while keeping their own hand low, the threshold setting unit sets a large threshold value for the position. The robot remote operation system according to Claim 2 or Claim 3.

5. Further comprising a threshold setting unit that sets the threshold value, the threshold setting unit selects the threshold value according to at least one of the type of task, the age of the operator, the weight of the operator, and the length of the arm of the robot. The robot remote operation system according to Claim 2 or Claim 3.

6. Further comprising a threshold setting unit that sets the threshold value, the threshold setting unit presents a plurality of preset threshold values to the operator and sets the threshold value selected by the operator from among the plurality of threshold values. The robot remote operation system according to Claim 2 or Claim 3.

7. Further comprising an image generation unit that generates an image to be presented to the operator using the state of the robot acquired by the first acquisition unit and the operation command of the robot acquired by the second acquisition unit. The image generation unit generates a guidance image indicating a change direction that needs to be changed so that the difference between the information acquired by the second acquisition unit and the position information of the state of the robot becomes a position less than the threshold value, and causes the generated guidance image to be presented to the operator. The robot remote operation system according to claim 2 or claim 3.

8. A remote operation control device for remotely operating a robot, a first acquisition unit that acquires the state of the robot; a second acquisition unit that acquires an operation input value input by an operator for an operation command of the robot; an operation command generation unit that generates an operation command from the operation input value acquired by the second acquisition unit; comprising: when the position based on the operation input value and the position of the state of the robot are different, the operation command generation unit determines a different state as an offset, and generates a robot operation command while maintaining the offset state. Remote operation control device.

9. A remote operation control method for remotely operating a robot, wherein a first acquisition unit acquires the state of the robot, a second acquisition unit acquires an operation input value input by an operator for an operation command of the robot, an operation command generation unit generates an operation command from the operation input value, and when the position based on the operation input value and the position of the state of the robot are different, the operation command generation unit determines a different state as an offset, and generates a robot operation command while maintaining the offset state. Remote operation control method.

10. Causing a computer of a remote operation control device for remotely operating a robot to acquire the state of the robot, acquire an operation input value input by an operator for an operation command of the robot, generate an operation command from the operation input value, and when the position based on the operation input value and the position of the state of the robot are different, determine a different state as an offset, and generate a robot operation command while maintaining the offset state. Program.

Images