ROBOT REMOTE OPERATION CONTROL DEVICE, ROBOT REMOTE OPERATION CONTROL SYSTEM, ROBOT REMOTE OPERATION CONTROL METHOD, AND PROGRAM

Abstract

To provide a robot remote operation control device, a robot remote operation control system, a robot remote operation control method and a program which can grasp visual line information accurately.SOLUTION: A robot remote operation control device, which performs robot remote operation control by which movements of an operator are recognized and the movements of the operator are transmitted to a robot to operate the robot, comprises: an information obtaining part that obtains a visual line vector showing visual line information on the operator; and a visual line information processing part that synchronizes a coordinate system of the obtained visual line vector with a coordinate system of a virtual space displayed on an image display device that displays an image of an environment around the robot which is in a field of view of the operator.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a program. [Background technology]

[0002] A control device that allows a user to assist in the operation of a robot has been proposed. As such a control device, for example, a control device has been proposed having a first information acquisition unit that acquires first user posture information indicating the posture of a first user operating the robot, a second information acquisition unit that acquires pre-change posture information indicating a pre-change posture that is a posture of the robot before the posture of the robot is changed based on the first user posture information, and a determination unit that determines a target posture different from the posture of the first user as the posture of the robot based on the pre-change posture information and the first user posture information acquired by the first information acquisition unit at the time when the robot is in the pre-change posture indicated by the pre-change posture information (see Patent Document 1). In the system described in Patent Document 1, the posture of the robot is changed to a posture corresponding to the posture detected by a device worn by the operator.

[0003] In such a system, when estimating an object of interest located away from the operator using gaze information, it is necessary to convert the gaze vector obtained from the gaze detection device into coordinates in the robot's world. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6476358 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technology, it was difficult to convert the gaze vector obtained from the gaze detection device into the robot world, making it impossible to accurately capture gaze information. When estimating the gaze object based on gaze, the gaze may move to the robot itself or the system status displayed on the head-mounted display, or the gaze may be fixed on a wall or hand out of concern for collision with the environment, which can lead to errors in estimating the manipulated object.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a program that can accurately capture gaze information. [Means for solving the problem]

[0007] (1) In order to achieve the above-mentioned object, a robot remote operation control device according to one embodiment of the present invention, in a robot remote operation that recognizes an operator's movements and transmits the operator's movements to a robot to operate the robot, comprises an information acquisition unit that acquires a gaze vector, which is the operator's gaze information, and a gaze information processing unit that synchronizes a coordinate system of the acquired gaze vector with a coordinate system of a virtual space displayed on an image display device that displays an image of the robot environment in the operator's field of view.

[0008] (2) In addition, in a robot remote operation control device according to one embodiment of the present invention, the gaze information processing unit may detect a first gaze point by approximating a gripping unit of the robot with a sphere and detecting an intersection between the gaze vector and the sphere, and may exclude the detected first gaze point from the objects that are candidates for operation by the operator.

[0009] (3) In addition, in a robot remote operation control device according to one embodiment of the present invention, the gaze information processing unit may detect a second gaze point by detecting an intersection between an object that is a candidate for operation by the operator and the gaze vector.

[0010] (4) In addition, in a robot remote operation control device according to one embodiment of the present invention, the gaze information processing unit may detect a third gaze point by detecting an intersection between a substrate on which an object to be operated by the operator is located and the gaze vector, and may exclude the detected third gaze point from the objects to be operated by the operator.

[0011] (5) In addition, in a robot remote operation control device according to one embodiment of the present invention, the gaze information processing unit may detect a fourth gaze point by detecting an intersection between the gaze vector and a display position of the robot state presented to the operator, and may exclude the detected fourth gaze point from the objects to be operated by the operator.

[0012] (6) In the robot remote operation control device according to one aspect of the present invention, the line of sight information processing unit may approximate the virtual space with a sphere.

[0013] (7) In order to achieve the above-mentioned object, a robot remote operation control system according to one embodiment of the present invention, in which a robot remote operation recognizes an operator's movements and transmits the operator's movements to a robot to operate the robot, comprises the robot remote operation control device described in any one of (1) to (6), a gripping unit for gripping an object, an environmental sensor installed on the robot or the robot's surrounding environment for detecting a robot environment sensor value, an operator sensor for detecting the operator's movements as an operator sensor value, and an image display device for displaying an image of the robot environment within the operator's field of view.

[0014] (8) In order to achieve the above-mentioned object, a robot remote operation control method according to one embodiment of the present invention is a robot remote operation method for recognizing an operator's movements and transmitting the operator's movements to a robot to operate the robot, in which an information acquisition unit acquires a gaze vector, which is the operator's gaze information, and a gaze information processing unit synchronizes a coordinate system of the acquired gaze vector with a coordinate system of a virtual space displayed on an image display device that displays an image of the robot environment within the operator's field of view.

[0015] (9) In order to achieve the above-mentioned object, a program according to one embodiment of the present invention, in a remote robot operation that recognizes an operator's movements and transmits the operator's movements to a robot to operate the robot, causes a computer to acquire a gaze vector, which is the operator's line of sight, and synchronizes the coordinate system of the acquired gaze vector with the coordinate system of a virtual space displayed on an image display device that displays an image of the robot environment within the operator's field of view. Effect of the Invention

[0016] According to (1) to (9), line-of-sight information can be accurately captured, and the accuracy of identifying a target object can be improved. [Brief description of the drawings]

[0017] [Figure 1] 1 is a diagram showing an overview of a robot remote operation control system according to an embodiment and an overview of operations thereof; [Diagram 2] 1 is a block diagram showing an example of the configuration of a robot remote operation control system according to an embodiment; [Diagram 3] FIG. 1 is a diagram showing an example of a state in which an operator wears an HMD and a controller. [Figure 4] FIG. 13 is a diagram for explaining an example of an image displayed on the HMD. [Diagram 5] FIG. 4 is a diagram showing a processing procedure of the robot remote operation control device according to the embodiment. [Figure 6] 4 is a flowchart of a processing procedure of a robot remote operation control device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component is of a recognizable size.

[0019] [overview] First, an overview of the operations and processes performed by the robot remote operation control system will be given. FIG. 1 is a diagram showing an overview of a robot remote operation control system 1 according to this embodiment and an overview of the work. As shown in FIG. 1, an operator Us wears, for example, an HMD (head mounted display) 5 and a controller 6. For example, an environmental sensor 7 (7a, 7b) is installed in the work environment. The environmental sensor 7 may be attached to the robot 2. The robot 2 also includes a gripping unit 222 (222a, 222b). The environmental sensor 7 (7a, 7b) includes, for example, an RGB camera and a depth sensor, as described later. The operator Us remotely controls the robot 2 by moving the hand or finger wearing the controller 6 while looking at an image displayed on the HMD 5. In this embodiment, the coordinate system of the operator's line of sight is linked to the coordinate system of the HMD, and the line of sight vector is aligned with the coordinate system of the virtual space. In this embodiment, a collision determination is then performed based on the coordinates and size of an object in the virtual space, thereby accurately capturing the operator's line of sight information, and using the line of sight information to estimate the target object.

[0020] [Example of a robot remote control system configuration] Next, a configuration example of the robot remote operation control system 1 will be described. 2 is a block diagram showing an example of the configuration of a robot remote operation control system 1 according to this embodiment. As shown in FIG. 2, the robot remote operation control system 1 includes a robot 2, a robot remote operation control device 3, an HMD 5 (image display device), a controller 6, and an environmental sensor 7.

[0021] The robot 2 includes, for example, a control unit 21, a driving unit 22, a sound collection unit 23, a memory unit 25, a power source 26, and a sensor 27. The robot remote operation control device 3 includes, for example, an information acquisition unit 31, a line of sight information processing unit 33, an intention estimation unit , a control command generation unit 35, a robot state image creation unit , a transmission unit 37, and a memory unit . The line of sight information processing unit 33 includes a coordinate synchronization unit 331 and a collision determination unit 332 .

[0022] The HMD 5 includes, for example, an image display unit 51, a gaze detection unit 52 (eye tracker device, operator sensor), a control unit 54, and a communication unit 55. The HMD 5 may include, for example, a sensor that detects the movement of the operator's gaze.

[0023] The controller 6 includes, for example, a sensor 61 (operator sensor), a control unit 62, a communication unit 63, and a feedback means 64.

[0024] The environmental sensor 7 includes, for example, an imaging device 71, a sensor 72, an object position detection unit 73, and a communication unit 74.

[0025] The robot remote operation control device 3 and the HMD 5 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the controller 6 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the environmental sensor 7 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the robot 2 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the HMD 5 may be directly connected without a network. The robot remote operation control device 3 and the controller 6 may be directly connected without a network. The robot remote operation control device 3 and the environmental sensor 7 may be directly connected without a network. The robot remote operation control device 3 and the robot 2 may be directly connected without a network.

[0026] [Examples of functions for a robot remote control system] Next, a functional example of the robot remote operation control system will be described with reference to FIG. The HMD 5 displays the robot status image received from the robot remote operation control device 3. The HMD 5 detects the movement of the operator's line of sight, and transmits the detected line of sight information (operator sensor value) to the robot remote operation control device 3.

[0027] The image display unit 51 displays the robot status image received from the robot remote control device 3 under the control of the control unit 54.

[0028] The gaze detection unit 52 detects the gaze of the operator, and outputs the detected gaze information to the control unit 54. Note that the gaze information is a gaze vector.

[0029] The control unit 54 transmits the gaze information detected by the gaze detection unit 52 to the robot remote operation control device 3 via the communication unit 55. The control unit 54 causes the image display unit 51 to display the robot state image transmitted by the robot remote operation control device 3.

[0030] The communication unit 55 receives the robot status image transmitted by the robot remote operation control device 3, and outputs the received robot status image to the control unit 54. The communication unit 55 transmits line of sight information to the robot remote operation control device 3 in response to the control of the control unit 54.

[0031] The controller 6 is, for example, a tactile data glove, and is worn on the hand of the operator. The controller 6 detects the direction, the movements of each finger, and the hand using a sensor 61, and transmits the detected hand movement information (operator sensor value) to the robot remote operation control device 3.

[0032] The sensor 61 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 61 includes a plurality of sensors, and tracks the movement of each finger using, for example, two sensors. The sensor 61 detects the movement of each finger and the movement of the hand, and outputs the detected hand movement information (operator sensor value) to the control unit 62.

[0033] The control unit 62 transmits the hand movement information detected by the sensor 61 to the robot remote operation control device 3 via the communication unit 63. The control unit 62 controls the feedback means 64 based on the feedback information.

[0034] The communication unit 63 transmits the operator action information to the robot remote operation control device 3 in response to the control of the control unit 62. The communication unit 63 acquires the feedback information transmitted by the robot remote operation control device 3, and outputs the acquired feedback information to the control unit 62.

[0035] The feedback means 64 feeds back feedback information to the operator in response to the control of the control unit 62. The feedback means 64 feeds back sensations to the operator in response to the feedback information, for example, by means for applying vibrations (not shown), means for applying air pressure (not shown), means for restricting hand movement (not shown), means for making the operator feel temperature (not shown), means for making the operator feel hardness or softness (not shown), etc., which are attached to the gripping unit 222 of the robot 2.

[0036] The environmental sensor 7 is installed in a position where it can capture and detect, for example, the work of the robot 2. The environmental sensor 7 may be provided in the robot 2 or may be attached to the robot 2. Alternatively, there may be multiple environmental sensors 7, which may be installed in the work environment as shown in FIG. 1 and also attached to the robot 2. The environmental sensor 7 detects object position information based on the captured image and the detection result detected by the sensor, and transmits the detected object position information to the robot remote operation control device 3.

[0037] The image capturing device 71 is, for example, an RGB camera. The image capturing device 71 outputs a captured image to the object position detection unit 73. In the environmental sensor 7, the positional relationship between the image capturing device 71 and the sensor 72 is known.

[0038] The sensor 72 is, for example, a depth sensor. The sensor 72 outputs a detection result to the object position detection unit 73. The image capturing device 71 and the sensor 72 may be distance sensors.

[0039] The object position detection unit 73 detects the three-dimensional position, size, shape, etc. of the target object in the captured image by a known method based on the captured image and the detection result detected by the sensor. The object position detection unit 73 estimates the position of the object by performing image processing (edge ​​detection, binarization processing, feature extraction, image enhancement processing, image extraction, pattern matching processing, etc.) on the image captured by the image capture device 71 with reference to a pattern matching model stored in the object position detection unit 73. When multiple objects are detected from the captured image, the object position detection unit 73 detects the position of each object. The object position detection unit 73 transmits the detected object position information to the robot remote operation control device 3 via the communication unit 74.

[0040] The communication unit 74 transmits the object position information to the robot remote operation control device 3. Note that the data transmitted by the environment sensor 7 may be, for example, a point cloud having position information.

[0041] When the robot 2 is not remotely operated, the behavior of the robot 2 is controlled according to the control of the control unit 21. When the robot 2 is remotely operated, the behavior of the robot 2 is controlled according to the grasping plan information generated by the robot remote operation control device 3.

[0042] The control unit 21 controls the driving unit 22 based on a control command output by the robot remote operation control device 3. The control unit 21 may perform voice recognition processing (such as speech period detection, sound source separation, sound source localization, noise suppression, and sound source identification) on the acoustic signal collected by the sound collection unit 23. The control unit 21 generates feedback information and transmits the generated feedback information to the controller 6 via the robot remote operation control device 3.

[0043] The driving unit 22 drives each part (the gripping unit 222, arms, fingers, legs, head, torso, waist, etc.) of the robot 2 under the control of the control unit 21. The driving unit 22 includes, for example, an actuator, a gear, an artificial muscle, etc.

[0044] The sound collection unit 23 is, for example, a microphone array having a plurality of microphones. The sound collection unit 23 outputs a collected acoustic signal to the control unit 21. The sound collection unit 23 may have a voice recognition processing function. In this case, the sound collection unit 23 outputs a voice recognition result to the control unit 21.

[0045] The storage unit 25 stores, for example, a program, thresholds, etc. used by the control unit 21 for control, and temporarily stores voice recognition results, image processing results, control commands, etc. The storage unit 25 may also be the storage unit 38. Alternatively, the storage unit 38 may also function as the storage unit 25.

[0046] The power source 26 supplies power to each component of the robot 2. The power source 26 may include, for example, a rechargeable battery or a charging circuit.

[0047] The sensor 27 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, an encoder for each joint, etc. The sensor 27 is attached to each joint, the head, etc. of the robot 2. The sensor 27 outputs the detection result to the control unit 21, the intention estimation unit 34, the control command generation unit 35, and the robot state image creation unit 36.

[0048] The robot remote operation control device 3 estimates the object intended to be operated by the operator based on the gaze information detected by the HMD 5, the hand movement information detected by the controller 6, and the object position information detected by the environmental sensor 7, and generates a control command for the robot 2.

[0049] The information acquisition unit 31 acquires gaze information from the HMD 5, acquires hand movement information from the controller 6, and acquires object position information from the environment sensor 7. The information acquisition unit 31 outputs the acquired gaze information, hand movement information, and object position information to the gaze information processing unit 33 and the intention estimation unit 34.

[0050] The coordinate synchronization unit 331 of the line-of-sight information processing unit 33 links the coordinate system with the coordinate system of the HMD, and matches (synchronizes) the coordinate system of the line-of-sight vector with the coordinate system of the virtual space of the image displayed on the HMD 5. A collision determination unit 332 of the line-of-sight information processing unit 33 performs collision determination based on the coordinates and size of an object in the virtual space. The processing of the line of sight information processing unit 33 will be described later.

[0051] Based on the gaze information processed by the gaze information processing unit 33 and the hand gesture information and object position information acquired by the information acquisition unit 31, the intention estimation unit 34 estimates information about the target object intended by the operator (such as the name of the target object, the position of the target object, and the size of the target object) by, for example, inputting it into a trained estimation model or by calculating a probability distribution.

[0052] The control command generation unit 35 generates a control command for grasping, for example, an object, based on the result estimated by the intention estimation unit 34, the detection result detected by the sensor 27, and the object position information detected by the environmental sensor 7. The control command generation unit 35 outputs the generated control command information to the control unit 21.

[0053] The robot state image creation unit 36 ​​creates a robot state image to be displayed on the HMD 5 based on the control command information generated by the control command generation unit 35 .

[0054] The transmission unit 37 transmits the robot state image created by the robot state image creation unit 36 ​​to the HMD 5. The transmission unit 37 acquires feedback information output by the robot 2, and transmits the acquired feedback information to the controller 6.

[0055] The memory unit 38 stores conversion formulas and predetermined values ​​used by the line of sight information processing unit 33. The memory unit 38 stores the positional relationship between the image capture device 71 and the sensor 72 of the environment sensor 7. The memory unit 38 stores information on the target to be assisted for each task, that is, information on the degree of freedom that the operator should control and the controllable range. The memory unit 38 stores programs used to control the robot remote operation control device 3. The programs may be stored in the cloud or on a network.

[0056] [Example of an operator wearing HMD5 and controller 6] Next, an example of a state in which the operator wears the HMD 5 and the controller 6 will be described. Fig. 3 is a diagram showing an example of a state in which an operator wears the HMD 5 and the controller 6. In the example of Fig. 3, the operator Us wears the controller 6a on his left hand, the controller 6b on his right hand, and the HMD 5 on his head. Note that the HMD 5 and the controller 6 shown in Fig. 3 are merely examples, and the wearing method, shape, and the like are not limited to these.

[0057] [Synchronization of the line of sight vector of line of sight detection unit 52 with the virtual space] Next, synchronization between the line-of-sight vector of line-of-sight detection unit 52 and the virtual space will be described. The coordinate synchronization unit 331 determines the center of the controller 6 worn by the operator as the reference position for the operator's operation in the virtual space. The coordinate synchronization unit 331 sets the orientation of the HMD 5 worn by the operator as the reference posture for the operator's operation in the virtual space. The coordinate synchronization unit 331 sets the center of the gripping unit 222 of the robot 2 as the reference position and posture of the robot operation in the virtual space. The coordinate synchronization unit 331 inputs to the robot 2 the relative position and orientation of the device from the reference point of the operator. The coordinate synchronization unit 331 determines the relative position and orientation of the HMD 5 from the reference point of the operator as the viewpoint of the operator in the virtual space. Through these processes, the coordinate synchronization unit 331 synchronizes the coordinates of the line of sight vector with the coordinates of the virtual space displayed on the HMD 5.

[0058] In aligning the operator's operating environment with the robot's operating environment, the coordinate synchronization unit 331 converts the coordinates of sensors such as the controller 6 worn by the operator and the HMD 5 into the coordinate system of the robot's operating environment. At this time, the coordinate synchronization unit 331 also performs calibration to correct the position of the image of the virtual operating environment displayed on the HMD 5. This process is the alignment of the operator's operating environment with the robot's operating environment.

[0059] When aligning the coordinates of the gaze acquisition device and the virtual space, the coordinate synchronization unit 331 uses the coordinate system of the modified robot operating environment of the HMD 5 and converts the gaze vector (local coordinates) acquired from the gaze detection unit 52 provided in the HMD 5 into a vector in the robot coordinate system.

[0060] Here, an example of how to align the gaze detection unit 52 with the coordinates of the virtual space will be described. First, the information acquisition unit 31 acquires the 5-coordinates of the HMD in the robot operating environment (right-handed coordinate system, map coordinates (coordinates of the robot operating environment)). Next, the information acquisition unit 31 acquires a gaze vector from the gaze detection unit 52 (left-handed coordinate system, local coordinates). Next, the coordinate synchronization unit 331 converts the gaze vector of the gaze detection unit 52 into the right-handed coordinate system (simply multiplying by a fixed value). Next, the coordinate synchronization unit 331 sets the coordinate position of the 5-HMD as the origin of the gaze detection unit 52.

[0061] [Example of image displayed on HMD5] Next, an example of an image displayed on the HMD 5 will be described. FIG. 4 is a diagram for explaining an example of an image displayed on the HMD 5. As shown in FIG. As feedback to the operator, the status of the gripper 222 of the robot 2, the robot remote control device 3, and the robot 2 may be displayed on the robot status image displayed on the HMD 5. In this case, if the operator's gaze is directed in that direction, the vector of the gaze may adversely affect the accuracy of estimation of the object to be operated.

[0062] As shown in the example of Fig. 4, factors that may adversely affect the accuracy of estimation of the target object include the image g11 of the hand of the operator Us, and the status image g12 of the robot remote control device 3 and the robot 2. The status image g11 includes information such as the task currently being executed. 4, there are two operation candidate objects (object obj1, object obj2). The probability that the object obj1 is the operator's intention is 1 / 4, and the probability that the object obj2 is the operator's intention is 3 / 4. It should be noted that the line-of-sight information processing unit 33 approximates the virtual space to a sphere.

[0063] [Collision detection] Next, the collision determination with the line of sight vector will be described with reference to FIG. The collision determination unit 332 approximates the gripping unit 222 of the robot 2 in the virtual space with a sphere. The collision determination unit 332 then considers the line of sight that collides with the sphere g21 as noise and does not use it in the estimation, thereby preventing an increase in the probability that the operator will be interested in an object overlapping with the gripping unit 222. Note that an object overlapping with the gripping unit 222 is, for example, a cup behind the gripping unit 222.

[0064] The collision determination unit 332 narrows down the target objects to be estimated according to the task. The intention estimation unit 34 excludes, for example, a table (board) that cannot be grasped from the target objects. Furthermore, the collision determination unit 332 also excludes the line of sight when it crosses the status displayed on the HMD 5 (because the display position is fixed and therefore crossing determination is applicable) from the estimation target.

[0065] [Processing Procedure] Next, the processing procedure of the robot remote operation control device 3 will be described with reference to FIG. 5 and FIG. Fig. 5 is a diagram showing a processing procedure of the robot remote operation control device 3 according to this embodiment. Fig. 6 is a flowchart showing a processing procedure of the robot remote operation control device 3 according to this embodiment.

[0066] (Step S1 ) The information acquisition unit 31 acquires line-of-sight information (line-of-sight vector) from the HMD 5 , acquires hand movement information from the controller 6 , and acquires object position information from the environment sensor 7 .

[0067] (Step S2) Coordinate synchronization unit 331 aligns the coordinates of the line of sight vector with the coordinates of virtual space g21.

[0068] (Step S3) The collision determination unit 332 approximates the grip unit 222 of the robot 2 with a sphere. Next, the collision determination unit 332 detects the intersection of the line of sight g22 on the grip unit 222 approximated with a sphere in the virtual space, thereby detecting the gaze point of the operator.

[0069] (Step S4) The collision determination unit 332 detects the intersection of the lines of sight g22 on the objects (object obj1, object obj2) in the virtual space, thereby detecting the gaze point of the operator.

[0070] (Step S5) The collision determination unit 332 detects the operator's gaze point by detecting an intersection of lines of sight on, for example, a table Tab on which an object is placed in the virtual space.

[0071] (Step S6) The collision determination unit 332 detects the gaze point of the operator by detecting the intersection of the lines of sight on a status image (status image of the robot 2) displayed in the virtual space on the HMD 5. The collision determination unit 332 stores the position of the status image displayed on the HMD 5.

[0072] (Step S7) The collision determination unit 332 detects the gaze point of the operator by detecting an intersection of the line of sight g22 on the sphere g21 of the working space (for example, a room) in the virtual space.

[0073] (Step S8) The collision determination unit 332 removes intersections that are noise. For example, the collision determination unit 332 removes, as noise, the intersections of the lines of sight on the object, the intersections of the lines of sight on the table Tab on which the object is placed, the intersections of the lines of sight on the status image, and the intersections of the lines of sight on the sphere of the working space.

[0074] (Step S9) The intention estimation unit 34 estimates an intended object, which is a target object for which the operator is going to perform work, based on the result of processing by the line-of-sight information processing unit 33 and the acquired sensor value.

[0075] (Step S10) The control command generating unit 35 generates a control command for, for example, gripping the intended object based on the result of estimation by the intention estimating unit 34 and the sensor value.

[0076] The robot remote operation control device 3 repeats the above process until the robot 2 grasps, for example, the object intended by the operator.

[0077] In FIG. 6, the line-of-sight information processing unit 33 may perform the processes of steps S3 to S6 in parallel, or in a time-division manner, and the processing order may be different. Furthermore, the intersection points with the line of sight are not limited to those mentioned above, and if there are other intersection points, each of the intersection points is detected.

[0078] 5, the working space is assumed to be spherical and the intersection points are detected, but the present invention is not limited to this. The gripping part 222 of the robot 2 is approximated by a sphere, but since a model can be reproduced on the HMD 5, it is possible to approximate the actual shape.

[0079] As described above, in this embodiment, the coordinates of the line of sight acquisition device and the virtual space are matched. Then, in this embodiment, the intersection with the line of sight in the processing space is detected, and the intersection that becomes noise is excluded. Also, in this embodiment, the gripping part 222 of the robot 2 is approximated by a sphere.

[0080] As a result, according to this embodiment, it is possible to accurately capture line-of-sight information and improve the accuracy of identifying a target object.

[0081] The intention estimation unit 34 may predict in advance the future trajectory of the hand intended by the operator, based on the operator state information and state information of the robot 2.

[0082] In addition, since the coordinate system is different between the environment in which the operator operates and the robot operating environment, the robot remote operation control device 3 may be configured to calibrate the operator's operating environment and the robot operating environment, for example, when the robot 2 is started up.

[0083] In addition, when grasping, the robot remote operation control device 3 may determine the grasping position by correcting the error in the grasping position at the time of grasping based on the grasping force of the robot 2 and the frictional force between the object and the grasping portion 222, etc.

[0084] Moreover, the above-mentioned robot 2 may be, for example, a bipedal robot, a stationary reception robot, or a working robot.

[0085] In the above example, the robot 2 is remotely controlled to grasp the object, but the present invention is not limited to this.

[0086] In the above example, the operator wears the HMD 5, but this is not limiting. Detection of gaze information and provision of a robot state image to the operator may be achieved by, for example, a combination of a sensor and an image display device.

[0087] In addition, a program for realizing all or part of the functions of the robot remote operation control device 3 in the present invention may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform all or part of the processing performed by the robot remote operation control device 3. In addition, the term "computer system" as used herein includes hardware such as an OS and peripheral devices. In addition, the term "computer system" also includes a system built on a local network and a system built on the cloud. In addition, the term "computer-readable recording medium" refers to portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into a computer system. In addition, the term "computer-readable recording medium" also refers to a storage device that holds a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0088] The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be for realizing part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0089] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0090] 1...robot remote operation control system, 2...robot, 3...robot remote operation control device, 5...HMD, 6...controller, 7...environmental sensor, 21...controller, 22...driving unit, 23...sound collection unit, 25...storage unit, 26...power supply, 27...sensor, 222, 222a, 222b...gripping unit, 31...information acquisition unit, 33...gaze information processing unit, 331...coordinate synchronization unit, 332...collision determination unit, 34...intention estimation unit, 35...control command generation unit, 36...robot state image creation unit, 37...transmission unit, 38...storage unit, 51...image display unit, 52...gaze detection unit, 54...control unit, 55...communication unit, 61...sensor, 62...control unit, 63...communication unit, 64...feedback means, 71...imaging device, 72...sensor, 73...object position detection unit, 74...communication unit

Claims

1. In a robot remote control system, a motion of an operator is recognized and the motion of the operator is transmitted to a robot to control the robot, an information acquisition unit that acquires a line-of-sight vector in a local coordinate system of a space in which the operator operates the robot based on line-of-sight information of the operator; a line-of-sight information processing unit that synchronizes the local coordinate system with a coordinate system of a virtual space in an image of a robot environment in the field of view of the operator that is displayed on an image display device, the local coordinate system has a coordinate position of the image display device as its origin, The line of sight information processing unit is a gripping unit of the robot is approximated by a sphere, and a first gaze point is detected by detecting an intersection between the line of sight vector and the sphere; excluding the detected first gaze point from objects that are candidates for operation by the operator; Robot remote control device.

2. In a robot remote control system, a motion of an operator is recognized and the motion of the operator is transmitted to a robot to control the robot, an information acquisition unit that acquires a line-of-sight vector in a local coordinate system of a space in which the operator operates the robot based on line-of-sight information of the operator; a line-of-sight information processing unit that synchronizes the local coordinate system with a coordinate system of a virtual space in an image of a robot environment in the field of view of the operator that is displayed on an image display device, the local coordinate system has a coordinate position of the image display device as its origin, The line of sight information processing unit is detecting a fourth gaze point by detecting an intersection between a display position of the state of the robot presented to the operator and the line of sight vector; excluding the detected fourth gaze point from objects that are candidates for operation by the operator; Robot remote control device.

3. The line of sight information processing unit is detecting a second gaze point by detecting an intersection between an object that is a candidate for operation by the operator and the line of sight vector; The robot remote control device according to claim 1 or 2.

4. The line of sight information processing unit is Detecting a third gaze point by detecting an intersection point between a surface on which an object to be operated by the operator is placed and the line of sight vector; excluding the detected third gaze point from objects that are candidates for operation by the operator; The robot remote operation control device according to any one of claims 1 to 3.

5. The line of sight information processing unit approximates the virtual space with a sphere. The robot remote operation control device according to any one of claims 1 to 4.

6. In a robot remote control system, a motion of an operator is recognized and the motion of the operator is transmitted to a robot to control the robot, The robot remote operation control device according to any one of claims 1 to 5, A gripping unit that grips an object; An environmental sensor that is installed on the robot or in the surrounding environment of the robot and detects image information and depth information of the surrounding environment of the robot; an operator sensor that detects line of sight information of the operator and hand movement information of the operator as an operator sensor value; a visual display device that displays an image of the robot environment in the operator's field of view; A robot remote operation control system comprising:

7. In a robot remote control system, a motion of an operator is recognized and the motion of the operator is transmitted to a robot to control the robot, an information acquisition unit acquires a line-of-sight vector in a local coordinate system of a space in which the operator operates the robot based on line-of-sight information of the operator; a line-of-sight information processing unit synchronizes the local coordinate system with a coordinate system of a virtual space in an image of a robot environment in the field of view of the operator displayed on an image display device; the local coordinate system has a coordinate position of the image display device as its origin, the line-of-sight information processing unit approximates a gripping unit of the robot with a sphere, detects a first gaze point by detecting an intersection between the line-of-sight vector and the sphere, and excludes the detected first gaze point from objects that are candidates for operation by the operator; A method for remotely controlling a robot.

8. In a robot remote control system, a motion of an operator is recognized and the motion of the operator is transmitted to a robot to control the robot, an information acquisition unit acquires a line-of-sight vector in a local coordinate system of a space in which the operator operates the robot based on line-of-sight information of the operator; a line-of-sight information processing unit synchronizes the local coordinate system with a coordinate system of a virtual space in an image of a robot environment in the field of view of the operator displayed on an image display device; the local coordinate system has a coordinate position of the image display device as its origin, the line-of-sight information processing unit detects a fourth gaze point by detecting an intersection between a display position of the state of the robot presented to the operator and the line-of-sight vector, and excludes the detected fourth gaze point from objects that are candidates for operation by the operator. A method for remotely controlling a robot.

9. In a robot remote control system, a motion of an operator is recognized and the motion of the operator is transmitted to a robot to control the robot, On the computer, acquiring a line-of-sight vector in a local coordinate system of a space in which the operator operates the robot based on line-of-sight information of the operator; Synchronizing the local coordinate system with a coordinate system of a virtual space in an image of a robot environment in the operator's field of view displayed on an image display device; the local coordinate system has a coordinate position of the image display device as its origin, a gripping unit of the robot is approximated by a sphere, and a first gaze point is detected by detecting an intersection between the line of sight vector and the sphere; Excluding the detected first gaze point from objects that are candidates for operation by the operator; program.

10. In a robot remote control system, a motion of an operator is recognized and the motion of the operator is transmitted to a robot to control the robot, On the computer, acquiring a line-of-sight vector in a local coordinate system of a space in which the operator operates the robot based on line-of-sight information of the operator; Synchronizing the local coordinate system with a coordinate system of a virtual space in an image of a robot environment in the operator's field of view displayed on an image display device; the local coordinate system has a coordinate position of the image display device as its origin, detecting a fourth gaze point by detecting an intersection between a display position of the state of the robot presented to the operator and the line of sight vector; Excluding the detected fourth gaze point from objects that are candidates for operation by the operator; program.

Images