Robot remote operation control device, robot remote operation control method, and program

Abstract

To provide a robot remote operation control device, a robot remote operation control method, and a program that are able to reduce a sense of incongruity with respect to an operation of an operator.SOLUTION: A robot remote operation control device for a robot remote operation control for an operator to remotely operate a robot, comprises: a spatial information sensor configured to acquire spatial information of a space in which the robot is present; an image output unit configured to present, to the operator, a spatial reproduction image in which the space where the robot is present is reproduced from the spatial information; a robot posture calculation unit configured to calculate a posture of the robot, based on a three-dimensional model of the robot and based on the spatial information; a spatial reproduction image drawing unit that calculates how the robot is seen from a visual point of the operator, based on the spatial information and the information calculated by the robot posture calculation unit, and draws, as a spatial reproduction image, an image in which a space blocked by a body of the robot is removed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a robot remote operation control device, a robot remote operation control method, and a program. [Background technology]

[0002] In the case of a typical robot, a camera is provided on the head to capture images of what is in front of it (see, for example, Patent Document 1). When such a robot is remotely controlled, a fisheye camera attached to the robot's head is used to present the operator with an image that is close to the operator's viewpoint (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-46350 [Patent Document 2] Patent No. 6940879 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology described in Patent Document 2 and the like, when an image mounted on the robot's head is used to control the robot while it is operating, the robot's body may block the operator's view, making it impossible to operate objects normally.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a robot remote operation control device, a robot remote operation control method, and a program that can reduce the sense of discomfort felt by the operator when operating the robot. [Means for solving the problem]

[0006] (1) In order to achieve the above object, one embodiment of the present invention provides a robot remote operation control device that, in a robot remote operation control where an operator remotely controls a robot, includes: a spatial information sensor that acquires spatial information of a space in which the robot exists; an image output unit that presents to the operator a spatial reproduction image that reproduces the space in which the robot exists from the spatial information; a robot posture calculation unit that calculates the posture of the robot based on a three-dimensional model of the robot and the spatial information; and a spatial reproduction image drawing unit that calculates how the robot appears from the operator's viewpoint based on the spatial information and the information calculated by the robot posture calculation unit, and draws an image in which spatial occlusion by the robot's body is eliminated as a spatial reproduction image.

[0007] (2) In order to achieve the above object, one embodiment of the present invention provides a robot remote operation control device that, in a robot remote operation control where an operator remotely controls a robot, includes: a spatial information sensor that acquires spatial information of a space in which the robot exists; an image output unit that presents to the operator a spatial reproduction image that reproduces the space in which the robot exists from the spatial information; a robot posture calculation unit that calculates the posture of the robot based on a three-dimensional model of the robot and the spatial information, and calculates how the robot appears from the operator's viewpoint; and a spatial reproduction image drawing unit that calculates how the robot appears from the operator's viewpoint from the spatial information and the information calculated by the robot posture calculation unit, and draws an image from any viewpoint of the operator as a spatial reproduction image.

[0008] (3) Furthermore, a robot remote operation control device according to one aspect of the present invention is the robot remote operation control device described in (1) above, in which the spatial reproduction image drawing unit calculates how the robot appears from the operator's viewpoint based on the spatial information and the information calculated by the robot posture calculation unit, determines whether the robot's body is blocking the operator's view, and if the robot's body is blocking the operator's view, draws an image from any viewpoint of the operator as a spatial reproduction image.

[0009] (4) Furthermore, a robot remote operation control device according to one aspect of the present invention is a robot remote operation control device described in any one of (1) to (3) above, further comprising an information acquisition unit that acquires operator status information of the status of an operator operating the robot, and the robot posture calculation unit calculates the posture of the robot based on a three-dimensional model of the robot, the spatial information sensor, and the operator status information.

[0010] (5) Furthermore, a robot remote control control device according to one aspect of the present invention is a robot remote control control device described in any one of (1) to (4) above, in which the spatial information sensor has a known position, acquires data in a time series, and assigns a timestamp.

[0011] (6) Furthermore, a robot remote control control device according to one aspect of the present invention is a robot remote control control device described in any one of (1) to (5) above, in which the spatial reproduction image drawing unit calculates how the robot appears from the operator's viewpoint using a model learned using the spatial information and an image from the operator's viewpoint, which is training data.

[0012] (7) Furthermore, a robot remote operation control device according to one aspect of the present invention is the robot remote operation control device described in (4) above, further comprising an operation amount determination unit that determines the operation amount of the robot based on the operator state information.

[0013] (8) In order to achieve the above object, one aspect of the present invention provides a robot remote operation control method, in which a robot remote operation control device, in which an operator remotely controls a robot, acquires spatial information of a space in which the robot exists, calculates the posture of the robot based on a three-dimensional model of the robot and the spatial information, calculates how the robot appears from the operator's viewpoint based on the spatial information and the calculated posture of the robot, and renders an image in which spatial occlusion by the robot's body has been eliminated as a spatial reproduction image and presents it to the operator.

[0014] (9) In order to achieve the above object, one aspect of the present invention provides a robot remote operation control method, in which a robot remote operation control device, in which an operator remotely controls a robot, acquires spatial information of a space in which the robot exists, calculates the posture of the robot based on a three-dimensional model of the robot and the spatial information, calculates how the robot appears from the operator's viewpoint, calculates how the robot appears from the operator's viewpoint based on the spatial information and the calculated posture of the robot, and renders an image from any viewpoint of the operator as a spatial reproduction image and presents it to the operator.

[0015] (10) In order to achieve the above object, a program according to one aspect of the present invention is a program that causes a computer of a robot remote operation control device, which is used by an operator to remotely control a robot, to acquire spatial information of the space in which the robot exists, calculate the posture of the robot based on a three-dimensional model of the robot and the spatial information, calculate how the robot appears from the operator's viewpoint based on the spatial information and the calculated posture of the robot, and render an image in which spatial occlusion by the robot's body has been eliminated as a spatial reproduction image and present it to the operator.

[0016] (11) In order to achieve the above object, one aspect of the present invention provides a program that causes a computer of a robot remote control control device, which is used by an operator to remotely control a robot, to acquire spatial information of the space in which the robot exists, calculate the posture of the robot based on a three-dimensional model of the robot and the spatial information, calculate how the robot appears from the operator's viewpoint, calculate how the robot appears from the operator's viewpoint based on the spatial information and the calculated posture of the robot, and render an image from any viewpoint of the operator as a spatial reproduction image and present it to the operator. [Effects of the Invention]

[0017] According to the above (1) to (11), by eliminating the obstruction caused by the robot's body, it is possible to reduce the discomfort felt by the operator when operating the robot. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a diagram illustrating an example of the configuration of a robot remote operation control system according to an embodiment. [Figure 2] 1A and 1B are diagrams illustrating an example of a robot according to an embodiment and an example of a location where a camera is installed. [Figure 3] FIG. 10 is a diagram illustrating an example of a data format of joint angle information according to the embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a data format of image information according to the embodiment. [Figure 5] 10A to 10C are diagrams for explaining an example of processing performed by a spatial reproduction image rendering unit according to the embodiment. [Figure 6] 10A to 10C are diagrams illustrating an example of input and output of a model included in a spatial reproduction image rendering unit according to the embodiment. [Figure 7] 1 is a flowchart of a processing procedure performed by a robot remote operation control system according to an embodiment. [Figure 8] FIG. 10 is a diagram showing an example of an image in which the arm of the robot blocks the line of sight of the operator. [Figure 9] FIG. 10 is an image diagram showing an example of an image in which the robot's arm blocks the operator's line of sight. [Figure 10] FIG. 10 is an image diagram of an example of an image synthesized so as to erase the occluded portion. [Figure 11] FIG. 10 is a diagram showing an example of an image of how the robot appears when its body is removed. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0020] [overview] In this embodiment, when a robot is remotely operated, even if the robot's body (for example, a hand or end effector) blocks the target object being worked on, an image that appears unblocked is provided to the operator. The image provided is synthesized using known information such as the robot's posture, three-dimensional model information of the robot, and information on the installation position of the imaging device. According to this embodiment, by eliminating the obstruction caused by the robot's body, it is possible to reduce the sense of discomfort felt by the operator when operating the robot.

[0021] [Configuration of a robot remote control system] An example of the configuration of a robot remote operation control system will be described. 1 is a diagram showing an example of the configuration of a robot remote operation control system according to this embodiment. As shown in Fig. 1, the robot remote operation control system 9 includes, for example, a robot 1, a second image capturing unit 2 (spatial information sensor), an HMD 3, an operation detection unit 4, a robot remote operation control device 5, and a robot model storage unit 6.

[0022] The robot 1 includes, for example, a first image capturing unit 11 (spatial information sensor), a sensor 12 (spatial information sensor), an arm 13, an end effector 14, and a driving unit 15.

[0023] The robot remote operation control device 5 includes, for example, an acquisition unit 51, an operation amount determination unit 52, a robot posture calculation unit 53, a space reproduction image drawing unit 54, a control unit 55, a memory unit 56, and an image output unit 57. The acquisition unit 51 includes, for example, an image acquisition unit 511, a sensor information acquisition unit 512, and an operator information acquisition unit 513 (information acquisition unit).

[0024] The robot 1 is, for example, any one of a single-arm robot, a double-arm robot, a multi-arm robot, an arm robot, and the like.

[0025] The first image capturing unit 11 is, for example, a CCD (Charge Coupled Device) image capturing device or a CMOS (Complementary Metal-Oxide-Semiconductor) image capturing device. The first image capturing unit 11 includes, for example, a fisheye lens or a wide-angle lens. The first image capturing unit 11 is attached, for example, to the head or end effector 14 of the robot 1. The first image capturing unit 11 includes, for example, an RGB camera that captures RGB (red-green-blue) images and a depth camera that captures depth images. The images captured by the first image capturing unit 11 are spatial information of the space in which the robot exists. The position of the first image capturing unit 11 attached to the robot 1 is known and is stored, for example, in the robot model storage unit 6. The first image capturing unit 11 acquires data in chronological order and assigns a timestamp to the data.

[0026] The sensor 12 is, for example, a six-axis sensor. The sensor 12 detects the joint angles of each joint of the arm 13 and each joint of the end effector 14. The sensor 12 is a joint torque sensor attached to the joints of the robot 1. In this way, the sensor 12 detects sensor information used by the robot 1. The first image capturing unit 11 may also be one of the sensors 12. The sensor 12 acquires data in chronological order and assigns a timestamp to the data.

[0027] The arm 13 has an end effector 14 connected to its tip.

[0028] The end effector 14 has at least two fingers, but may have three or more fingers.

[0029] The second image capturing unit 2 is installed, for example, in the working environment of the robot 1. The second image capturing unit 2 is equipped, for example, with an RGB camera that captures RGB (red-green-blue) images and a depth camera that captures depth images. The images captured by the second image capturing unit 2 are spatial information of the space in which the robot exists. The installation position of the second image capturing unit 2 is known and is stored, for example, in the robot model storage unit 6. The second image capturing unit 2 acquires data in chronological order and assigns a timestamp to it.

[0030] The HMD 3 is, for example, a head-mounted display worn by the operator on the head. The HMD 3 includes, for example, an image display unit 31 that presents an image to the operator, a gaze detection unit 32 that detects the gaze of the operator, and a communication unit (not shown) that transmits and receives information to and from the robot remote operation control device 5.

[0031] The operation detection unit 4 detects an operation instruction from an operator. The operation detection unit 4 is, for example, a data glove.

[0032] The robot model storage unit 6 stores, for example, a three-dimensional shape model of the robot 1 (including dimensions, etc.).

[0033] The robot remote operation control device 5 uses spatial information of the space in which the robot exists, operator status information indicating the status of the operator operating the robot 1, and a robot model to draw an image in which spatial occlusion caused by the body of the robot 1 has been eliminated as a spatial reproduction image.

[0034] The image acquisition unit 511 acquires the image captured by the first image capture unit 11 and the image captured by the second image capture unit 2.

[0035] The sensor information acquisition unit 512 acquires, from the sensor 12, the joint angles of each joint of the end effector 14 detected by the sensor 12. The sensor information acquisition unit 512 may also acquire, as sensor information, images captured by the imaging units (first imaging unit 11, second imaging unit 2). Alternatively, the sensor information acquisition unit 512 may acquire, as sensor information, operator state information detected by the operation detection unit 4.

[0036] The operator information acquisition unit 513 acquires operator status information on the status of the operator operating the robot 1, for example, from the operation detection unit 4. The operator status information indicates how the operator wants the robot 1 to operate, or how the operator wants the robot 1 to operate. The operator information acquisition unit 513 may also acquire gaze information detected by the gaze detection unit 32.

[0037] The operation amount determination unit 52 determines the operation amount of the robot 1 based on the operator state information.

[0038] The robot posture calculation unit 53 calculates the posture of the robot using the three-dimensional model of the robot 1 and the information acquired by the sensor information acquisition unit 512. Alternatively, the robot posture calculation unit 53 calculates the posture of the robot using the three-dimensional model of the robot 1, the information acquired by the sensor information acquisition unit 512, and the information acquired by the operator information acquisition unit 513.

[0039] The spatial reproduction image drawing unit 54 calculates how the robot appears from the operator's viewpoint based on the spatial information and the information calculated by the robot posture calculation unit 53, and determines whether the body of the robot 1 is blocking the operator's view. If the body of the robot 1 is blocking the operator's view, the spatial reproduction image drawing unit 54 draws an image in which the spatial occlusion caused by the body of the robot 1 has been removed, as a spatial reproduction image, based on the spatial information and the information calculated by the robot posture calculation unit 53. The spatial reproduction image drawing unit 54 provides the generated image to the HMD 3.

[0040] The control unit 55 controls the movement of the robot 1 based on the operation amount determined by the operation amount determination unit 52.

[0041] The storage unit 56 stores, for example, values, programs, etc. required for controlling the robot 1. The storage unit 56 temporarily stores acquired information.

[0042] The image output unit 57 outputs a space reproduction image, which reproduces the space in which the robot exists from the space information, to the HMD 3 so that the operator can see it.

[0043] [Robot configuration example] First, an example of a robot and an example of a camera installation location will be described. FIG. 2 is a diagram showing an example of a robot according to this embodiment and an example of the installation location of a camera. The robot 1 includes, for example, a body 16, a head 17, arms 13 (13L, 13R), and hands (end effectors) 14 (14L, 14R). In the following description, the end effectors 14 are also referred to as hands 14. The hand 14 has first image capturing units 11a and 11b attached to the wrist on the back side of the palm, for example. The head 17 has first image capturing units 11c and 11d. In addition, a second imaging unit 2 is installed in the working space of the robot 1.

[0044] The first imaging units 11c and 11d on the head of the robot 1 may be the second imaging unit 2. In the example shown in FIG. 2, the hand 14 has five fingers, but the number of fingers may be two or more. Also, in FIG. 2, an example of a robot with two arms is shown, but it may have one arm. Also, the number of image capturing units is an example and is not limited to this. Also, the image capturing unit may be at least one of the palm, fingertips, and wrist, or multiple units may be provided on the hand 14.

[0045] [Data format to be acquired] Next, an example of the format of data acquired by the robot remote operation control device 5 will be described. 3 is a diagram showing an example of a data format of joint angle information according to this embodiment. As shown in FIG. 3, a timestamp is associated with the joint angle information. 4 is a diagram showing an example of the data format of image information according to this embodiment. As shown in FIG. 4, a timestamp is associated with the image information. The joint angle information and image information are stored in the storage unit 56, for example, temporarily.

[0046] [Example of processing performed by the manipulated variable determination unit] Next, an example of the processing performed by the manipulated variable determiner 52 will be described. The operation amount determination unit 52 determines the operation amount of the robot 1 based on the operator state information acquired by the operator information acquisition unit 513. Note that the operation amount determination unit 52 may, for example, estimate the target object, the position of the target object, the shape of the target object, etc., using the image acquired by the image acquisition unit 511, and determine the operation amount of the robot 1 using the estimated information about the target object as well. Furthermore, the operation amount determination unit 52 may estimate taxonomy information (see, for example, Reference 1) related to the task based on the operator state information acquired by the operator information acquisition unit 513, and determine the operation amount of the robot 1 using the estimated taxonomy information as well.

[0047] Reference 1; Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77

[0048] [Example of processing performed by the robot posture calculation unit] Next, an example of the processing performed by the robot posture calculation unit 53 will be described. The robot posture calculation unit 53 calculates the posture of the robot 1, for example, at predetermined time intervals, using the joint angle information (including timestamps) acquired by the sensor information acquisition unit 512 and the three-dimensional shape model stored in the robot model storage unit 6. Note that the robot posture calculation unit 53 calculates the posture of the arm 13 and the end effector 14 depending on the task, and does not necessarily need to calculate the posture of the body 16, head 17, etc.

[0049] [Example of processing performed by the spatial reproduction image rendering unit] Next, an example of the processing performed by the spatial reproduction image drawing unit 54 will be described. 5 is a diagram illustrating an example of processing performed by the spatial reproduction image rendering unit according to this embodiment. The spatial reproduction image rendering unit 54 renders, as a spatial reproduction image, an image in which spatial occlusion by the body of the robot 1 has been eliminated, based on the spatial information and the robot posture information. More specifically, the spatial reproduction image rendering unit 54 first calculates how the robot appears from the operator's viewpoint. If the calculated appearance of the robot from the operator's viewpoint obstructs the operator's view, the spatial reproduction image rendering unit 54 uses the acquired image to convert it into an image in which the view is not obstructed, and presents it.

[0050] The spatial reproduction image drawing unit 54 may be provided with a trained model. Fig. 6 is a diagram showing an example of input and output of a model provided in the spatial reproduction image drawing unit according to this embodiment. The model 541 may be trained by inputting image information, joint angle information, and robot model information in advance, and outputting a provided image using the provided image as training data. The spatial reproduction image rendering unit 54 may input image information, joint angle information, and robot model information to the trained model during operation, and output a provided image. The model may be configured using a network such as a DNN (Deep Neural Network) or an RNN (Recurrent Neural Network).

[0051] [Example of processing procedure] Next, a description will be given of an example of the processing procedure performed by the robot remote operation control system 9. Fig. 7 is a flowchart of the processing procedure performed by the robot remote operation control system according to this embodiment.

[0052] (Step S1) The image acquisition unit 511 acquires the image captured by the first image capture unit 11 and the image captured by the second image capture unit 2.

[0053] (Step S2) The sensor information acquisition unit 512 acquires the sensor information (including joint angle information) detected by the sensor 12.

[0054] (Step S3) The operator information acquisition unit 513 acquires operator state information from the HMD 3 and the operation detection unit 4.

[0055] (Step S4) The robot posture calculation unit 53 calculates the posture of the robot 1 based on the information acquired by the sensor information acquisition unit 512. The robot posture calculation unit 53 may also calculate the posture of the robot 1 based on the operator state information. Alternatively, the robot posture calculation unit 53 may calculate the posture of the robot 1 based on the information acquired by the sensor information acquisition unit 512 and the operator state information.

[0056] (Step S5) The space reproduction image drawing unit 54 uses the acquired information and the three-dimensional shape model of the robot to calculate how the robot will look from the operator's viewpoint.

[0057] (Step S6) The spatial reproduction image drawing unit 54 determines whether the body of the robot 1 (arm 13, end effector 14, head 17, body, etc.) is blocking the operator's field of view in the calculated view of the robot from the operator's viewpoint. If the body of the robot 1 is blocking the creator's field of view (Step S6; YES), the spatial reproduction image drawing unit 54 proceeds to processing of Step S7. If the body of the robot 1 is not blocking the creator's field of view (Step S6; NO), the spatial reproduction image drawing unit 54 proceeds to processing of Step S9.

[0058] (Step S7) The spatial reproduction image drawing unit 54 converts the acquired image into an image from the operator's viewpoint. Note that the spatial reproduction image drawing unit 54 may convert the acquired image into an image from the operator's viewpoint using, for example, a trained model.

[0059] (Step S8) The spatial reproduction image drawing unit 54 synthesizes an image from the operator's viewpoint so as to eliminate occlusion by the robot 1's body by synthesizing an image converted from the acquired image into an image of how it appears from the operator's viewpoint.

[0060] (Step S9) When the body of the robot 1 does not obstruct the operator's view, the spatial reproduction image drawing unit 54 provides the operator with an image of how it appears from the operator's viewpoint, generated based on the acquired image, via the image output unit 57. When the body of the robot 1 obstructs the operator's view, the spatial reproduction image drawing unit 54 provides the operator with an image of how it appears from the operator's viewpoint, generated based on the acquired image and converted into an image of how it appears from the operator's viewpoint, via the image output unit 57.

[0061] 7 is an example, and is not limited to this. For example, the processing of steps S1 to S3 is not limited to this order, and may be performed simultaneously in parallel.

[0062] [Method for determining whether the operator's view is obstructed] Here, an example of a method for determining whether or not the operator's field of vision is obstructed will be described. The robot remote operation control device 5 can grasp the position and state of the target object from the captured image before grasping the target object. In addition, the robot remote operation control device 5 can grasp the posture of the robot from the information acquired by the sensor information acquisition unit 512 and the three-dimensional model of the robot 1 stored in the robot model storage unit 6.

[0063] The robot remote operation control device 5 creates an image of the operator's field of view using this information, a three-dimensional model of the robot 1, and the captured image. Based on information on the position of the target object and the positions and postures of the arm 13 and end effector 14 of the robot 1, the robot remote operation control device 5 determines whether the body of the robot 1 blocks the operator's field of view in the created image of the operator's field of view.

[0064] Fig. 8 is a diagram showing an example of an image in which the arm of a robot blocks the line of sight of an operator. Fig. 8 is an example of an image drawn using a three-dimensional model of a robot, showing a state in which the arm and hand of robot 1 are performing a task. For example, this is an example of a state in which the arm or hand blocks the line of sight of an operator by blocking a target object.

[0065] 9 is an image diagram showing an example of an image in which the robot's arm blocks the operator's line of sight. In the example of FIG. 9, the target object obj is blocked by the right hand 14R. Even if an image in which the operator's line of sight is blocked is provided in this way, the operator is unable to give a gripping operation instruction or finds it difficult to give a gripping operation instruction because the target object obj is blocked.

[0066] For this reason, in this embodiment, when the operator's line of sight is blocked in this way, an image is synthesized based on the acquired image, converted by the generated robot 1 into an image that appears from the operator's viewpoint so as to erase the blocked portion. FIG. 10 is an illustration of an example of an image synthesized so as to erase the blocked portion. As shown in FIG. 10, the robot remote operation control device 5 generates an image in which the portion of the target object obj that is blocked by a palm or the like is not blocked, using the acquired image and a three-dimensional model of the robot 1. Note that the robot remote operation control device 5 may also generate an image in which, for example, the blocking portion of the palm is made semi-transparent so that the target object obj can be seen.

[0067] As described above, in this embodiment, detailed three-dimensional shape information of the known robot 1, information on the joint angles of the robot 1 with the acquired timestamp, and the acquired timestamp-attached image are used to generate and synthesize an image in which any obstructions to the operator's line of sight have been eliminated.

[0068] As a result, according to this embodiment, by eliminating the obstruction caused by the robot's body, it is possible to reduce the discomfort felt by the operator when operating the robot.

[0069] Furthermore, since the body shape of the robot 1 is different from that of the operator, the view is obstructed by the arms of the robot 1 in the image captured by the first image capturing unit 11 mounted on the head of the robot 1, which can cause an uncomfortable feeling when operating the robot. According to this embodiment, such an uncomfortable feeling can be reduced.

[0070] [Variations] In the above example, an example of providing an image from the operator's viewpoint has been described, but the image to be provided is not limited to this. Since the robot remote operation control device 5 already knows the position and orientation information of the image capturing units (first image capturing unit 11, second image capturing unit 2), it is also possible to synthesize and provide an image from any viewpoint using the acquired information. For example, the robot remote operation control device 5 can also provide an image of how the robot 1 looks when its body is hidden, as shown in FIG. 11.

[0071] Fig. 11 shows an example of an image with the robot's body erased. In the example of Fig. 11, for example, some objects placed on a table and the arms other than the fingers on the right hand and the gripper on the left hand have been erased.

[0072] As described above, in the modified example, an image from any viewpoint is generated using detailed three-dimensional shape information of the known robot 1, information on the angles of each joint of the robot 1 with the acquired timestamp, and the acquired image with the timestamp.

[0073] It should be noted that the provision of an image from an arbitrary viewpoint may be performed, for example, by the operator based on the line-of-sight detection result of the HMD 3 or the operation detection unit 4.

[0074] A program for implementing all or part of the functions of the robot remote operation control device 5 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the robot remote operation control device 5. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a website provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0075] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0076] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0077] 9...Robot remote operation control system, 1...Robot, 2...Second image capturing unit, 3...HMD, 4...Operation detection unit, 5...Robot remote operation control device, 6...Robot model memory unit, 11...First image capturing unit, 12...Sensor, 13...Arm, 14...End effector, 15...Drive unit, 31...Image display unit, 32...Gaze detection unit, 51...Acquisition unit, 52...Operation amount determination unit, 53...Robot posture calculation unit, 54...Spatial reproduction image drawing unit, 55...Control unit, 56...Memory unit, 511...Image acquisition unit, 512...Sensor information acquisition unit, 513...Operator information acquisition unit, 541...Model

Claims

1. In robot remote control, an operator remotely controls a robot. a spatial information sensor for acquiring spatial information of a space in which the robot exists; an image acquisition unit that acquires an image captured by an imaging unit that captures an image of a space where the robot exists; an image output unit that presents to the operator a spatial reproduction image in which the space in which the robot exists is reproduced from the spatial information; a robot posture calculation unit that calculates a posture of the robot based on the three-dimensional model of the robot and the spatial information; a spatial reproduction image drawing unit that calculates how the robot appears from the operator's viewpoint based on the spatial information and the information calculated by the robot posture calculation unit, determines whether the body of the robot is blocking the operator's field of view based on the appearance of the robot, removes the spatial occlusion caused by the body of the robot when it is determined that the field of view is blocked, and draws an image converted from the image acquired by the image acquisition unit as a spatial reproduction image, the spatial reproduction image rendering unit generates an image in which the parts of the robot's body that are blocking the view are semi-transparent. Robot remote control device.

2. the spatial reproduction image drawing unit calculates how the robot appears from the operator's viewpoint based on the spatial information and the information calculated by the robot posture calculation unit, determines whether the body of the robot is blocking the operator's field of view, and, if the body of the robot is blocking the operator's field of view, draws an image from an arbitrary viewpoint of the operator as a spatial reproduction image. The robot remote control device according to claim 1 .

3. an information acquisition unit that acquires operator status information about a status of an operator operating the robot; the robot posture calculation unit calculates the posture of the robot based on the three-dimensional model of the robot, the spatial information sensor, and the operator state information; The robot remote control device according to claim 1 or 2.

4. The spatial information sensor has a known position, acquires data in a time series, and assigns a time stamp to the data. The robot remote control device according to claim 1 .

5. the spatial reproduction image rendering unit calculates how the robot appears from the operator's viewpoint using a model trained using the spatial information and an image from the operator's viewpoint, which is training data; The robot remote control device according to claim 1 .

6. further comprising an operation amount determination unit that determines an operation amount of the robot based on the operator state information; The robot remote control device according to claim 3 .

7. A robot remote control control device that allows an operator to remotely control a robot, acquiring spatial information of the space in which the robot exists; acquiring an image captured by an imaging unit that captures an image of a space in which the robot exists; calculating a posture of the robot based on the three-dimensional model of the robot and the spatial information; From the spatial information and the calculated information of the posture of the robot, how the robot appears from the viewpoint of the operator is calculated, and based on how the robot appears, it is determined whether the body of the robot is blocking the field of view of the operator, and if it is determined that the field of view is blocked, the spatial occlusion caused by the body of the robot is eliminated and converted into an image captured by the imaging unit, and further an image is generated in which the part of the body of the robot that is blocking the field of view is made semi-transparent, and the image is rendered as a spatial reproduction image and presented to the operator. A method for remotely controlling a robot.

8. The computer of the robot remote control control device, which allows the operator to remotely control the robot, acquiring spatial information of a space in which the robot exists; acquiring an image captured by an imaging unit that captures an image of a space in which the robot exists; calculating a posture of the robot based on the three-dimensional model of the robot and the spatial information; From the spatial information and the calculated information of the posture of the robot, how the robot appears from the viewpoint of the operator is calculated, and based on how the robot appears, it is determined whether the body of the robot is blocking the field of view of the operator, and if it is determined that the field of view is blocked, the spatial occlusion caused by the body of the robot is eliminated and converted into an image captured by the imaging unit, and further an image is generated in which the part of the body of the robot that is blocking the field of view is made semi-transparent, and the image is rendered as a spatial reproduction image and presented to the operator. program.

Images