Robot remote operation control device, robot remote operation control system, robot remote operation control method, and program

Abstract

To provide a robot remote operation control device, a robot remote operation control system, a robot remote operation control method and a program which enable even a user who is not skilled in operating a robot to make the robot perform work with high accuracy.SOLUTION: A robot remote operation control device, which performs robot remote operation control by which an operator is enabled to remotely operate a robot which can grip an object, comprises: an information obtaining part that obtains operator state information on a state of an operator who operates the robot; an intention estimating part that estimates an intention of motion that the operator intends to make the robot perform on the basis of the operator state information; and a gripping method determining part that determines a method of gripping the object based on the estimated intention of the motion of the operator.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a program. [Background technology]

[0002] A control device that allows a user to assist in operating a robot has been proposed. For example, one such control device has a first information acquisition unit that acquires first user posture information that indicates the posture of a first user operating the robot, a second information acquisition unit that acquires pre-change posture information that indicates a pre-change posture that is the posture of the robot before the posture of the robot is changed based on the first user posture information, and a determination unit that determines a target posture different from the posture of the first user as the posture of the robot based on the pre-change posture information and the first user posture information acquired by the first information acquisition unit at the time when the robot is in the pre-change posture indicated by the pre-change posture information (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6476358 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional technology, when making a robot perform a specified task, it is sometimes difficult for a user who is not familiar with operating a robot to make the robot perform the task with high accuracy compared to a user who is familiar with operating a robot.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a program that allow even a user who is not familiar with operating a robot to have the robot perform tasks with high precision. [Means for solving the problem]

[0006] (1) In order to achieve the above object, a robot remote operation control device according to one embodiment of the present invention is a robot remote operation control device in which an operator remotely controls a robot capable of grasping an object, and includes an information acquisition unit that acquires operator status information on the status of the operator operating the robot, an intention estimation unit that estimates the operational intention that the operator is intending the robot to perform based on the operator status information, and a grasping method determination unit that determines a method of grasping the object based on the estimated operational intention of the operator.

[0007] (2) In addition, in a robot remote operation control device according to one aspect of the present invention, the intention estimation unit may classify the posture of the operator based on operator state information, thereby determining a classification of the posture of the robot and estimating the operator's intention to operate.

[0008] (3) In addition, in a robot remote operation control device according to one aspect of the present invention, the intention estimation unit may estimate the operator's intention to operate by estimating at least one of the manner in which the object to be grasped is held and the object to be grasped based on the operator state information.

[0009] (4) Furthermore, in a robot remote operation control device according to one aspect of the present invention, the intention estimation unit may estimate a manner of grasping an object to be grasped based on the operator state information, and estimate the object to be grasped based on the estimated manner of grasping the object to be grasped, thereby estimating the operator's intention to perform the action.

[0010] (5) Furthermore, in a robot remote operation control device according to one aspect of the present invention, the intention estimation unit may estimate a manner of grasping an object to be grasped based on the operator state information, and estimate the object to be grasped based on the estimated manner of grasping the object to be grasped, thereby estimating the operator's intention to perform the action.

[0011] (6) In addition, in a robot remote operation control device according to one aspect of the present invention, the operator status information may be at least one of the operator's line of sight information, the operator's arm movement information, and the operator's head movement information.

[0012] (7) In addition, in a robot remote operation control device according to one aspect of the present invention, the information acquisition unit may acquire position information of the object, and the grasping method determination unit may estimate the object to be grasped using the acquired position information of the object.

[0013] (8) In addition, in a robot remote operation control device according to one aspect of the present invention, the gripping method determination unit may acquire position information of a gripping unit provided on the robot and correct the position information of the gripping unit based on operator status information.

[0014] (9) Furthermore, in a robot remote operation control device according to one aspect of the present invention, a robot state image creation unit may be further provided, wherein the intention estimation unit acquires information about the object based on an image captured by a photographing device, and the robot state image creation unit generates an image to be provided to the operator based on the information about the object, position information of the gripping unit, the operator state information, and the corrected position information of the gripping unit.

[0015] (10) In order to achieve the above object, a robot remote operation control system according to one embodiment of the present invention comprises a robot having a gripping unit for gripping the object and a detection unit for detecting position information of the gripping unit, the robot remote operation control device described in any one of (1) to (6) above, an environmental sensor for detecting position information of the object, and a sensor for detecting operator status information of the status of an operator operating the robot.

[0016] (11) In order to achieve the above object, one aspect of the present invention provides a robot remote operation control method in which an operator remotely controls a robot capable of grasping an object, wherein an information acquisition unit acquires operator state information on the state of the operator operating the robot, an intention estimation unit estimates at least one of an object to be grasped and a grasping method based on the operator state information, and a grasping method determination unit determines a grasping method for the object based on the estimation result.

[0017] (12) In order to achieve the above object, a program according to one embodiment of the present invention is a program for controlling a robot remotely, in which an operator remotely controls a robot capable of grasping an object. The program causes a computer to acquire operator status information on the status of the operator operating the robot, estimate at least one of an object to be grasped and a grasping method based on the operator status information, and determine a grasping method for the object based on the estimation result. [Effects of the Invention]

[0018] According to (1) to (12), the target object can be picked up without the operator having to perform precise positioning, so even a user who is not familiar with operating a robot can have the robot perform the task with high precision. According to (2) to (5), the intention of the operator can be estimated with high accuracy by estimating the operator's intention based on the movement of the operator's arm including the hand and fingers. According to (8), the position information of the gripping part is corrected based on the actual position of the robot's gripping part and the state of the operator, so that the target object can be picked up without the operator having to perform accurate alignment. According to (9), an image based on the corrected position information of the gripping part can be provided to the operator, making it easier for the operator to remotely control the robot. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a diagram illustrating an overview of a robot remote operation control system according to an embodiment and an overview of operations. FIG. [Figure 2] 1 is a block diagram illustrating an example of the configuration of a robot remote operation control system according to an embodiment. [Figure 3] FIG. 10 is a diagram showing an example of a state in which an operator wears an HMD and a controller. [Figure 4] FIG. 2 is a diagram illustrating an example of a processing procedure of the robot and the robot remote operation control device according to the embodiment. [Figure 5] FIG. 10 is a diagram showing an example of a state in which three objects are placed on a table and the operator is trying to make the robot grasp the object obj3 with the left hand. [Figure 6] 10 is a flowchart of an example of processing performed by the robot remote operation control device according to the embodiment. [Figure 7] FIG. 10 is a diagram showing an example of a robot state image displayed on an HMD according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized.

[0021] [overview] First, an overview of the operations and processes performed by the robot remote operation control system will be explained. FIG. 1 illustrates an overview of a robot remote operation control system 1 according to this embodiment and an overview of the operation. As shown in FIG. 1, an operator Us wears, for example, an HMD (head-mounted display) 5 and controllers 6 (6a, 6b). Environmental sensors 7a and 7b are installed in the workspace. The environmental sensor 7 may be attached to the robot 2. The robot 2 also includes grippers 222 (222a, 222b). The environmental sensors 7a and 7b include, for example, an RGB camera and a depth sensor, as described below. The operator Us remotely controls the robot 2 by moving the hand or fingers wearing the controller 6 while viewing the image displayed on the HMD 5. In the example of FIG. 1, the operator Us remotely controls the robot 2 to grasp a plastic bottle obj on a table Tb. Note that, during remote operation, the operator Us cannot directly see the movements of the robot 2, but can indirectly see the image from the robot 2's side through the HMD 5. In this embodiment, the robot remote operation control device 3 provided in the robot 2 acquires information on the status of the operator operating the robot 2 (operator status information), estimates the object to be grasped and the grasping method based on the acquired operator status information, and determines the object grasping method based on the estimation.

[0022] [Example of a robot remote control system configuration] Next, a configuration example of the robot remote operation control system 1 will be described. 2 is a block diagram showing an example of the configuration of a robot remote operation control system 1 according to this embodiment. As shown in FIG. 2, the robot remote operation control system 1 includes a robot 2, a robot remote operation control device 3, an HMD 5, a controller 6, and an environmental sensor 7.

[0023] The robot 2 includes, for example, a control unit 21, a driving unit 22, a sound collection unit 23, a memory unit 25, a power source 26, and a sensor 27. The robot remote operation control device 3 includes, for example, an information acquisition unit 31, an intention estimation unit 33, a gripping method determination unit 34, a robot state image creation unit 35, a transmission unit 36, and a storage unit 37.

[0024] The HMD 5 includes, for example, an image display unit 51, a gaze detection unit 52, a sensor 53, a control unit 54, and a communication unit 55. The controller 6 includes, for example, a sensor 61 , a control unit 62 , a communication unit 63 , and a feedback means 64 .

[0025] The environment sensor 7 includes, for example, an imaging device 71, a sensor 72, an object position detection unit 73, and a communication unit 74.

[0026] The robot remote operation control device 3 and the HMD 5 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the controller 6 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the environmental sensor 7 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the robot 2 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the HMD 5 may be connected directly without a network. The robot remote operation control device 3 and the controller 6 may be connected directly without a network. The robot remote operation control device 3 and the environmental sensor 7 may be connected directly without a network. The robot remote operation control device 3 and the robot 2 may be connected directly without a network.

[0027] [Example of functions of a robot remote control system] Next, an example of the functions of the robot remote operation control system will be described with reference to FIG. The HMD 5 displays the robot status image received from the robot remote operation control device 3. The HMD 5 detects the operator's line of sight movement, head movement, etc., and transmits the detected operator status information to the robot remote operation control device 3.

[0028] The image display unit 51 displays the robot status image received from the robot remote operation control device 3 in accordance with the control of the control unit 54.

[0029] The line-of-sight detection unit 52 detects the line of sight of the operator, and outputs the detected line-of-sight information (operator sensor value) to the control unit .

[0030] The sensor 53 is, for example, an acceleration sensor, a gyroscope, or the like, which detects the movement and tilt of the operator's head, and outputs the detected head movement information (operator sensor value) to the control unit .

[0031] The control unit 54 transmits the gaze information detected by the gaze detection unit 52 and the head movement information detected by the sensor 53 to the robot remote operation control device 3 via the communication unit 55. The control unit 54 also causes the image display unit 51 to display the robot status image transmitted by the robot remote operation control device 3.

[0032] The communication unit 55 receives the robot status image transmitted by the robot remote operation control device 3, and outputs the received robot status image to the control unit 54. The communication unit 55 transmits gaze information and head movement information to the robot remote operation control device 3 in accordance with the control of the control unit 54.

[0033] The controller 6 is, for example, a tactile data glove that is worn on the hand of the operator. The controller 6 detects the direction, the movements of each finger, and the movements of the hand using sensors 61, and transmits the detected operator status information to the robot remote operation control device 3.

[0034] The sensor 61 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 61 includes a plurality of sensors, and tracks the movement of each finger using, for example, two sensors. The sensor 61 detects operator arm information (operator sensor value, operator state information) that is information on the posture and position of the operator's arm, such as the direction, the movement of each finger, and the movement of the hand, and outputs the detected operator arm information to the control unit 62. The operator arm information includes information covering the entire human arm, such as hand position and posture information, angle information of each finger, position and posture information of the elbow, and information tracking the movement of each part.

[0035] The control unit 62 transmits the operator arm information to the robot remote operation control device 3 via the communication unit 63. The control unit 62 controls the feedback means 64 based on the feedback information.

[0036] The communication unit 63 transmits the line of sight information and the operator's arm information to the robot remote operation control device 3 in accordance with the control of the control unit 62. The communication unit 63 acquires the feedback information transmitted by the robot remote operation control device 3 and outputs the acquired feedback information to the control unit 62.

[0037] The feedback means 64 feeds back feedback information to the operator in accordance with the control of the control unit 62. In accordance with the feedback information, the feedback means 64 feeds back sensations to the operator by, for example, a means (not shown) that applies vibration attached to the gripping unit 222 of the robot 2, a means (not shown) that applies air pressure, a means (not shown) that restricts hand movement, a means (not shown) that makes the operator feel temperature, or a means (not shown) that makes the operator feel hardness or softness.

[0038] The environmental sensor 7 is installed in a position where it can capture and detect, for example, the work of the robot 2. The environmental sensor 7 may be provided in the robot 2 or may be attached to the robot 2. Alternatively, there may be multiple environmental sensors 7, and they may be installed in the work environment as shown in FIG. 1 and also attached to the robot 2. The environmental sensor 7 transmits object position information (environmental sensor value), captured images (environmental sensor value), and detected sensor values ​​(environmental sensor value) to the robot remote operation control device 3. The environmental sensor 7 may be a motion capture device, and may detect object position information by motion capture. Alternatively, a GPS receiver (not shown) equipped with a position information transmission unit may be attached to the object. In this case, the GPS receiver may transmit position information to the robot remote operation control device 3.

[0039] The image capturing device 71 is, for example, an RGB camera. In the environment sensor 7, the positional relationship between the image capturing device 71 and the sensor 72 is known.

[0040] The sensor 72 is, for example, a depth sensor. The image capturing device 71 and the sensor 72 may be distance sensors.

[0041] The object position detection unit 73 detects the three-dimensional position, size, shape, etc. of the target object in the captured image using a known method based on the captured image and the detection results detected by the sensor. The object position detection unit 73 estimates the position of the object by performing image processing (edge ​​detection, binarization processing, feature extraction, image enhancement processing, image extraction, pattern matching processing, etc.) on the image captured by the image capture device 71 with reference to a pattern matching model stored in the object position detection unit 73. If multiple objects are detected in the captured image, the object position detection unit 73 detects the position of each object. The object position detection unit 73 transmits the detected object position information (environmental sensor value), the captured image (environmental sensor value), and the sensor value (environmental sensor value) to the robot remote operation control device 3 via the communication unit 74.

[0042] The communication unit 74 transmits the object position information to the robot remote operation control device 3. The communication unit 74 transmits the object position information (environment sensor value), the captured image (environment sensor value), and the sensor value (environment sensor value) to the robot remote operation control device 3.

[0043] When the robot 2 is not remotely controlled, its behavior is controlled according to the control of the control unit 21. When the robot 2 is remotely controlled, its behavior is controlled according to the grasping plan information generated by the robot remote operation control device 3.

[0044] The control unit 21 controls the driving unit 22 based on the gripping method information output by the robot remote operation control device 3. The control unit 21 performs voice recognition processing (such as speech interval detection, sound source separation, sound source localization, noise suppression, and sound source identification) on the acoustic signal collected by the sound collection unit 23. If the voice recognition result includes an operation instruction for the robot, the control unit 21 may control the operation of the robot 2 based on the voice operation instruction. The control unit 21 performs image processing (such as edge detection, binarization, feature extraction, image enhancement, image extraction, and pattern matching) on ​​the image captured by the environmental sensor 7 based on the information stored in the memory unit 25. Note that the data transmitted by the environmental sensor 7 may be, for example, a point cloud having position information. The control unit 21 extracts information about the object (object information) from the captured image by image processing. The object information includes, for example, the name and position of the object. The control unit 21 controls the driving unit 22 based on the program stored in the memory unit 25, the voice recognition result, and the image processing result. The control unit 21 outputs the motion status information of the robot 2 to the robot status image creation unit 35. The control unit 21 generates feedback information and transmits the generated feedback information to the controller 6 via the robot remote operation control device 3.

[0045] The driving unit 22 drives each part (arms, fingers, legs, head, torso, waist, etc.) of the robot 2 according to the control of the control unit 21. The driving unit 22 includes, for example, an actuator, a gear, an artificial muscle, etc.

[0046] The sound collection unit 23 is, for example, a microphone array including a plurality of microphones. The sound collection unit 23 outputs the collected acoustic signal to the control unit 21. The sound collection unit 23 may have a voice recognition processing function. In this case, the sound collection unit 23 outputs the voice recognition result to the control unit 21.

[0047] The storage unit 25 stores, for example, a program, thresholds, etc. used for control by the control unit 21. The storage unit 37 may also function as the storage unit 25. Alternatively, the storage unit 37 may also function as the storage unit 25.

[0048] The power supply 26 supplies power to each part of the robot 2. The power supply 26 may include, for example, a rechargeable battery or a charging circuit.

[0049] The sensor 27 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, an encoder for each joint, etc. The sensor 27 is attached to each joint, the head, etc. of the robot 2. The sensor 27 outputs the detection result to the control unit 21, the intention estimation unit 33, the grasping method determination unit 34, and the robot state image creation unit 35.

[0050] The information acquisition unit 31 acquires gaze information and head movement information from the HMD 5, acquires operator arm information from the controller 6, and acquires environmental sensor values ​​(object position information, sensor values, and images) from the environmental sensor 7, and outputs the acquired operator state information to the intention estimation unit 33 and the robot state image creation unit 35.

[0051] The intention estimation unit 33 estimates the operator's intention based on the information acquired by the information acquisition unit 31. The intention estimation unit 33 estimates the operator's intention using at least one of gaze information, operator arm information, and head movement information. The intention estimation unit 33 may also estimate the operator's intention using environmental sensor values. The operator's intention will be described later.

[0052] The gripping method determination unit 34 determines a gripping method for the object based on the intention of the action estimated by the intention estimation unit 33, the detection result detected by the sensor 27, and the result of image processing of the image captured by the image capturing device 71. The gripping method determination unit 34 outputs the determined gripping method information to the control unit 21.

[0053] The robot status image creation unit 35 performs image processing (edge ​​detection, binarization, feature extraction, image enhancement, image extraction, clustering processing, etc.) on the image captured by the image capture device 71. The robot status image creation unit 35 estimates the position and movement of the robot 2's hands and the movement of the operator's hands based on the gripping method information estimated by the gripping method determination unit 34, the image processing results, and the motion status information of the robot 2 output by the control unit 21, and creates a robot status image to be displayed on the HMD 5 based on the estimation results. The robot status image may also include system status information indicating the system status, such as information on the process the robot remote operation control device 3 is about to perform and error information.

[0054] The transmission unit 36 ​​transmits the robot state image created by the robot state image creation unit 35 to the HMD 5. The transmission unit 36 ​​acquires feedback information output by the robot 2, and transmits the acquired feedback information to the controller 6.

[0055] The storage unit 37 stores templates used by the intention estimation unit 33 during estimation, trained models used for estimation, etc. The storage unit 37 also temporarily stores voice recognition results, image processing results, gripping method information, etc. The storage unit 37 stores model images to be compared in the pattern matching process of image processing.

[0056] [Example of an operator wearing HMD5 and controller 6] Next, an example of a state in which the operator wears the HMD 5 and the controller 6 will be described. Fig. 3 is a diagram showing an example of a state in which an operator wears the HMD 5 and the controller 6. In the example of Fig. 3, the operator Us wears the controller 6a on his left hand, the controller 6b on his right hand, and the HMD 5 on his head. Note that the HMD 5 and the controller 6 shown in Fig. 3 are merely examples, and the wearing method, shape, etc. are not limited to these.

[0057] [Operator status information] Next, the operator status information acquired by the information acquisition unit 31 will be further described. The operator state information is information that indicates the state of the operator, and includes information on the operator's line of sight, information on the movement and position of the operator's fingers, and information on the movement and position of the operator's hands. The HMD 5 detects the operator's line of sight. The controller 6 detects information about the movement and position of the operator's fingers and the movement and position of the operator's hand.

[0058] [Example of information estimated by the intention estimation unit 33] Next, an example of information estimated by the intention estimation unit 33 will be described. The intention estimation unit 33 estimates the operator's intention based on the acquired operator state information. The operator's intention is, for example, the purpose of the task that the operator wants the robot 2 to perform, the content of the task that the operator wants the robot 2 to perform, and the movements of the hands and fingers at each time. The intention estimation unit 33 classifies the posture of the arm including the gripper 222 of the robot 2 by classifying the posture of the operator's arm based on the operator sensor value of the controller 6. The intention estimation unit 33 estimates the operator's intention of the task that the operator wants the robot to perform based on the classification result. The intention estimation unit 33 estimates, for example, the way an object is held and the object that the operator wants to grip as the operator's intention. The purpose of the task is, for example, gripping an object, moving an object, etc. The content of the task is, for example, gripping and lifting an object, gripping and moving an object, etc.

[0059] The intention estimation unit 33 estimates the operator's intention for the action by, for example, the GRASP Taxonomy method (see, for example, Reference 1). In this embodiment, the operator's state is classified by classifying the posture of the operator or the robot 2, i.e., the grasping posture, using, for example, a grasp taxonomy method, and the operator's intention is estimated. The intention estimation unit 33 estimates the operator's intention, for example, by inputting the operator's state information into a trained model stored in the storage unit 37. In this embodiment, the intention estimation is performed by classifying the grasping posture, and thus the operator's intention can be estimated with high accuracy. Note that other methods may be used to classify the grasping posture.

[0060] Reference 1; Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human Grasp Types” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb. 2016), IEEE, p66-77

[0061] The intention estimation unit 33 may also perform an integrated estimation using the gaze and arm movements. In this case, the intention estimation unit 33 may input gaze information, hand movement information, and position information of an object on a table into a trained model to estimate the operator's intention.

[0062] The intention estimation unit 33 first estimates the object to be grasped based on, for example, operator state information. The intention estimation unit 33 estimates the object to be grasped based on, for example, gaze information. Next, the intention estimation unit 33 estimates the posture of the operator's hand based on the estimated object to be grasped.

[0063] Alternatively, the intention estimation unit 33 may first estimate the hand posture of the operator based on, for example, the operator state information. Next, the intention estimation unit 33 estimates an object to be grasped from the estimated hand posture of the operator. For example, if three objects are placed on a table, the intention estimation unit 33 estimates which of the three objects is a candidate for grasping based on the hand posture.

[0064] Furthermore, the intention estimation unit 33 may estimate in advance the future trajectory of the hand intended by the operator, based on the operator state information and the state information of the robot 2.

[0065] In addition, the intention estimation unit 33 may also use the detection results detected by the sensor 27, the results of image processing of the image captured by the environmental sensor 7, etc. to estimate the object to be grasped and the position of the object.

[0066] Furthermore, since the coordinate systems are different between the environment in which the operator operates and the robot operating environment, calibration of the operator operating environment and the robot operating environment may be performed when the robot 2 is started, for example.

[0067] In addition, when grasping, the robot remote operation control device 3 may determine the grasping position based on the grasping force of the robot 2 and the frictional force between the object and the grasping part, taking into account the error in the grasping position during grasping.

[0068] [Processing example of robot 2 and robot remote control device 3] Next, an example of processing by the robot 2 and the robot remote operation control device 3 will be described. FIG. 4 is a diagram showing an example of a processing procedure of the robot 2 and the robot remote operation control device 3 according to this embodiment.

[0069] (Step S1) The information acquisition unit 31 acquires line of sight information (operator sensor value) and head movement information (operator sensor value) from the HMD 5, and acquires operator arm information (operator sensor value) from the controller 6.

[0070] (Step S2) The information acquisition unit 31 acquires an environment sensor value from the environment sensor 7.

[0071] (Step S3) The intention estimation unit 33 estimates the operator's intention, such as the task content and the object to be grasped, based on the acquired operator sensor value. The intention estimation unit 33 estimates the operator's intention using at least one of gaze information, operator arm information, and head movement information. The intention estimation unit 33 may also estimate the operator's intention using environmental sensor values. Subsequently, the grasping method determination unit 34 calculates a remote operation command for the robot 2 based on the estimation result.

[0072] (Step S4) The control unit 21 calculates a drive command value for stable gripping based on the remote operation command value calculated by the robot remote operation control device 3.

[0073] (Step S5) The control unit 21 controls the drive unit 22 using the drive command value to drive the gripping unit and the like of the robot 2. After the process, the control unit 21 returns to the process of step S1.

[0074] The processing procedure shown in FIG. 4 is an example, and the robot 2 and the robot remote operation control device 3 may perform the above-described processing in parallel.

[0075] [Estimation results, work information] Next, an example of the estimation result and the work information will be described with reference to FIGS. FIG. 5 shows an example of a state in which three objects obj1 to obj3 are placed on a table, and the operator is trying to make the robot 2 grasp the object obj3 with the left hand. In such a case, the robot remote operation control device 3 needs to estimate which of the objects obj1 to obj3 the operator wants the robot 2 to grasp. Note that the robot remote operation control device 3 needs to estimate whether the operator is trying to grasp the object with his right hand or his left hand.

[0076] Here, the reason why it is necessary to estimate the operator's intention beforehand will be explained. In the case of remote control, the world seen by the operator through the HMD 5 differs from the real world seen with the operator's own eyes. Furthermore, even if the operator issues an instruction via the controller 6, the operator is not actually grasping an object, resulting in a different situation perception from that in the real world. Furthermore, delays occur between the operator's instruction and the robot 2's movement due to communication time, calculation time, and the like. Furthermore, due to differences in the physical structures of the operator and the robot (mainly the hand), even if the operator commands the movement of the fingers that can be grasped and the robot accurately traces it, this does not necessarily mean that the robot can actually grasp the object. To address this issue, in this embodiment, the operator's intention is estimated and the operator's movement is converted into an appropriate movement for the robot.

[0077] As described above, conventional remote control systems have difficulty recognizing the real-world situation and have been unable to successfully pick up objects. In conventional remote control systems, for example, the operator must gradually bring the gripper 22a of the robot 2 closer to the object and accurately align it to grasp it.

[0078] In contrast, in this embodiment, the operator's intention to operate is estimated and the operation of the robot 2 is controlled based on the estimated results, making it possible for the operator to pick an object without performing accurate positioning.

[0079] Next, an example of intention estimation processing and correction processing will be described. FIG. 6 is a flowchart of an example of processing performed by the robot remote operation control device 3 according to this embodiment.

[0080] (Step S101) The intention estimation unit 33 uses the acquired environmental sensor values ​​to perform environmental recognition, such as recognizing that three objects obj1 to obj3 are placed on a table.

[0081] When an operator remotely controls the robot 2 to grasp an object obj3, the operator generally directs his or her gaze toward the object obj3 to be grasped. Therefore, the intention estimation unit 33 estimates that the target object is object obj3 based on gaze information included in the operator state information acquired from the HMD 5. The intention estimation unit 33 may also use head direction and tilt information included in the operator state information to make the estimation. When there are multiple objects, the intention estimation unit 33 calculates the probability that each object is the target object (reach object probability). The intention estimation unit 33 calculates the probability based on, for example, gaze information, the distance between the estimated target object and the gripping portion of the robot 2, the position and movement (trajectory) of the controller 6, etc.

[0082] (Step S102) The intention estimation unit 33 compares the arm positions and movements (hand positions, hand movements (trajectories), finger positions, finger movements (trajectories), arm positions, arm movements (trajectories)) and head positions and movements included in the acquired operator state information with templates stored in the storage unit 37, and classifies the movements to estimate the operator's intention to grasp the object obj3 with the left hand and the holding manner (grasping method). For example, the gripping method determination unit 34 determines the gripping method by referring to the templates stored in the storage unit 37. Note that the gripping method determination unit 34 may select the gripping method by inputting it into a trained model stored in the storage unit 37, for example. Note that the intention estimation unit 33 estimates the operator's intention to grasp the object obj3 with the left hand using at least one of the gaze information, the operator's arm information, and the head movement information. Note that the intention estimation unit 33 may also use environmental sensor values ​​to estimate the intention.

[0083] (Step S103) The gripping method determination unit 34 determines a gripping method for the robot 2 based on the estimated operator's intention to move. (Step S104) The gripping method determination unit 34 calculates the amount of deviation between the position of the operator's hand and fingers and the position of the gripping unit of the robot. The memory unit 37 stores, for example, a delay time measured in advance, which is the time it takes for the drive unit 22 to operate after an instruction is given. The gripping method determination unit 34 calculates the amount of deviation using, for example, the delay time stored in the memory unit 37. Next, the gripping method determination unit 34 corrects the amount of deviation between the position of the operator's hand and fingers and the position of the gripping unit of the robot. The gripping method determination unit 34 calculates the current operation target value based on the sampling time of the robot control.

[0084] (Step S106) The robot status image creation unit 35 creates a robot status image to be displayed on the HMD 5 based on the results recognized and estimated by the intention estimation unit 33 and the results calculated by the gripping method determination unit 34. The robot status image also includes information about the processing that the robot remote operation control device 3 is about to perform, system status information, etc.

[0085] FIG. 7 is a diagram showing an example of a robot state image displayed on the HMD 5 according to this embodiment. Images g11 to g13 correspond to objects obj1 to obj3 placed on the table. In this case, the reach object probability is assumed to be 0.077 for image g11, 0.230 for image g12, and 0.693 for image g13. Image g21 shows the actual position of the gripper of robot 2. Image g22 represents the position input by the operator using the controller 6. Image g23 shows the corrected command position of the gripper of robot 2.

[0086] 7, shape data (e.g., CAD (Computer Aided Design) data) of the gripping portion of the robot 2 is stored in the storage unit 37. The robot state image creation unit 35 uses the shape data of the gripping portion of the robot 2 to generate an image of the gripping portion of the robot 2. The robot state image creation unit 35 creates a robot state image such as that shown in FIG. 7 using, for example, a technique such as SLAM (Simultaneous Localization and Mapping).

[0087] In this way, the operator can visually check the actual position of the gripper of the robot 2 (image g21), the position that the operator is inputting (image g22), and the corrected position of the gripper of the robot 2 (image g23), which is helpful for the operator. In this embodiment, the robot's operation is corrected based on the operator's intention, and the process that the robot remote operation control device 3 is about to perform is presented to the operator as visual information, for example, so that remote operation can be performed smoothly. As a result, according to this embodiment, the position information of the gripper is corrected based on the actual position of the robot's gripper and the state of the operator, so that the target object can be picked up without the operator having to perform accurate positioning.

[0088] In this manner, in this embodiment, the following steps I to V are carried out for remote control. I. Object Recognition II. Intention estimation (e.g., grasping object and taxonomy estimation from gaze and operator's hand trajectory) III. Motion correction (e.g., correcting the robot's hand trajectory to a position where it can grasp, selecting the grasping method) IV. Stable Grasping (Control of the Grasping Unit to Stably Grasp Using the Selected Grasping Method) V. Present the robot model, recognition results, information about the processing that the robot remote operation control device 3 is about to perform, information about the system status, etc. on the HMD.

[0089] Here, the motion correction will be further explained. The grasping method determination unit 34 determines, for example, the contact points of the fingers of the robot 2 with the object that can stably grasp the object without dropping it, based on the selected motion classification, the object shape, estimated physical parameters such as the friction and weight of the object, and constraints such as the torque that can be output by the robot 2. Then, the grasping method determination unit 34 performs a corrective motion using, for example, the joint angles calculated from these as target values.

[0090] Next, stable gripping will be described. When the robot operates according to the target values, the grasping method determination unit 34 controls, for example, the finger joint angles and torques in real time so as to eliminate errors between the target values / parameter estimates and the values ​​observed by the sensor 27 of the robot 2. As a result, according to this embodiment, the robot can grasp stably and continuously without dropping the object.

[0091] As a result, according to this embodiment, the target object can be picked up without the operator having to perform accurate positioning. As a result, according to this embodiment, even a user who is not familiar with operating the robot 2 can have the robot perform a task with high accuracy.

[0092] In the above example, the robot 2 is provided with the robot remote operation control device 3, but this is not limiting. The robot 2 does not have to be provided with the robot remote operation control device 3, and it may be an external device to the robot 2. In this case, the robot 2 and the robot remote operation control device 3 may transmit and receive various information. Alternatively, the robot 2 may be provided with some of the functional units of the robot remote operation control device 3, and the external device may be provided with the other functional units.

[0093] Furthermore, the robot 2 described above may be, for example, a bipedal robot, a stationary reception robot, or a working robot.

[0094] In the above example, the robot 2 is remotely controlled to grasp the object, but this is not limiting. For example, if the robot 2 is a bipedal robot, an operator may remotely control the walking of the robot 2 by wearing controllers on the feet. In this case, the robot 2 may detect object information such as an obstacle by image processing, and the operator may remotely control the robot 2 to walk while avoiding the obstacle.

[0095] In the above example, the operator wears the HMD 5, but this is not limiting. Detection of gaze information and provision of a robot status image to the operator may be achieved by, for example, a combination of a sensor and an image display device.

[0096] In addition, a program for implementing all or part of the functions of the robot 2 and all or part of the functions of the robot remote operation control device 3 in the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the robot 2 and all or part of the processing performed by the robot remote operation control device 3. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes systems built on local networks and cloud-based systems. The term "computer-readable recording medium" refers to portable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain programs for a certain period of time, such as volatile memory (RAM) within computer systems that function as servers or clients when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0097] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0098] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0099] 1...Robot remote operation control system, 2...Robot, 3...Robot remote operation control device, 5...HMD, 6...Controller, 7...Environment sensor, 21...Controller, 22...Driver, 23...Sound collection unit, 25...Memory unit, 26...Power supply, 27...Sensor, 31...Information acquisition unit, 33...Intention estimation unit, 34...Grip method determination unit, 35...Robot state image creation unit, 36...Transmitter, 37...Memory unit, 51...Image display unit, 52...Gaze detection unit, 53...Sensor, 54...Controller, 55...Communication unit, 61...Sensor, 62...Controller, 63...Communication unit, 64...Feedback means, 71...Capturing device, 72...Sensor, 73...Object position detection unit, 74...Communication unit

Claims

1. In a robot remote control system in which an operator remotely controls a robot capable of grasping an object, an information acquisition unit that acquires operator status information about a status of the operator operating the robot; an intention estimation unit that estimates an intention of an action that the operator is intending the robot to perform based on the operator state information; a grasping method determination unit that determines a method of grasping the object based on the estimated intention of the operator, The intention estimation unit classifying a gripping posture of the operator into a predetermined gripping posture classification based on a hand shape of the operator, and estimating an intention of the operator's action based on the gripping posture classification result; estimating, as the action intention, a future trajectory of the operator's hand intended by the operator based on the operator state information and robot state information representing a state of the robot; The gripping method determination unit calculating contact points between the object and the fingers of the robot based on the classification result of the grasping posture, the shape of the object, physical parameters of the object, and torque that can be output by the robot; calculating a joint angle of the finger based on the contact point; correcting the motion of the fingers of the robot operated by the operator based on the motion intention and the joint angles; Robot remote control device.

2. the intention estimation unit classifies a posture of the operator based on the operator state information, thereby determining a classification of a posture of the robot and estimating an intention of the operator. The robot remote control device according to claim 1 .

3. the intention estimation unit estimates an intention of the operator by estimating at least one of a way of holding an object to be grasped and the object to be grasped based on the operator state information; The robot remote control device according to claim 1 .

4. the intention estimation unit estimates an object to be grasped based on the operator state information, and estimates a holding manner of the object to be grasped related to the estimated object, thereby estimating an intention of the operator. The robot remote control device according to claim 1 .

5. the intention estimation unit estimates a manner of gripping an object to be gripped based on the operator state information, and estimates the object to be gripped based on the estimated manner of gripping the object to be gripped, thereby estimating the operator's intention to perform a function. The robot remote control device according to claim 1 .

6. the operator state information is at least one of line of sight information of the operator, arm movement information of the operator, and head movement information of the operator; The robot remote operation control device according to any one of claims 1 to 5.

7. the information acquisition unit acquires position information of the object; The grasping method determination unit estimates an object to be grasped and a grasping method for the object using the acquired position information of the object. The robot remote operation control device according to any one of claims 1 to 6.

8. the gripping method determination unit acquires position information of a gripping unit provided in the robot, and corrects the position information of the gripping unit based on the operator state information. The robot remote operation control device according to any one of claims 1 to 7.

9. a robot state image creation unit, the intention estimation unit acquires information about the object based on an image captured by an image capture device; the robot state image creation unit generates an image to be provided to the operator based on information about the object, position information of the gripping unit, the operator state information, and the corrected position information of the gripping unit. The robot remote operation control device according to claim 8.

10. a gripping unit that grips the object; a detection unit that detects position information of the gripping unit; a robot comprising: The robot remote operation control device according to any one of claims 1 to 9, an environmental sensor that detects position information of the object; a sensor for detecting operator status information of a status of the operator operating the robot; A robot remote operation control system comprising:

11. In a robot remote control system in which an operator remotely controls a robot capable of grasping an object, an information acquisition unit acquires operator status information on the status of an operator operating the robot; an intention estimation unit that estimates an intention of an action that the operator is intending the robot to perform based on the operator state information; a grasping method determination unit determines a grasping method for the object based on the estimated intention of the operator; the intention estimation unit classifies a grip posture of the operator into a grip posture classification determined in advance based on a shape of the operator's hand, and estimates an intention of the operator based on a classification result of the grip posture; the intention estimation unit estimates, as the action intention, a future trajectory of the operator's hand intended by the operator, based on the operator state information and robot state information representing a state of the robot; the grasping method determination unit calculates contact points between the object and the fingers of the robot based on the grasping posture classification result, the shape of the object, physical parameters of the object, and torque that can be output by the robot; the grasping method determination unit calculates the joint angles of the fingers based on the contact points; the grasping method determination unit corrects the motion of the fingers of the robot operated by the operator based on the motion intention and the joint angle. A method for remotely controlling a robot.

12. In a robot remote control system in which an operator remotely controls a robot capable of grasping an object, On the computer, acquiring operator status information of the operator operating the robot; Inferring an intention of the operator that the operator is intending the robot to perform based on the operator state information; determining a gripping method for the object based on the estimated intention of the operator; classifying the gripping posture of the operator into a gripping posture classification that is predetermined based on the shape of the operator's hand; Inferring the operator's intention to operate based on the classification result of the gripping posture; a future trajectory of the operator's hand intended by the operator is estimated as the action intention based on the operator state information and robot state information representing a state of the robot; calculating contact points between the object and the fingers of the robot based on the classification result of the grasping posture, the shape of the object, physical parameters of the object, and torque that can be output by the robot; Calculating the joint angles of the fingers based on the contact points; correcting the motion of the fingers of the robot operated by the operator based on the motion intention and the joint angles; Program for.

Images