ROBOT REMOTE OPERATION CONTROL DEVICE, ROBOT REMOTE OPERATION CONTROL SYSTEM, ROBOT REMOTE OPERATION CONTROL METHOD, AND PROGRAM

Abstract

To provide a robot remote operation control device, a robot remote operation control system, a robot remote operation control method and a program which enable an operator to easily perform work.SOLUTION: A robot remote operation control device, which performs robot remote operation control by which movements of an operator are recognized and the movements of the operator are transmitted to a robot to operate the robot, comprises: an intention estimating part that estimates motion of the operator on the basis of a robot environment sensor value obtained by an environment sensor installed in the robot or in an environment around the robot and an operator sensor value showing the motion of the operator obtained by an operator sensor; and a control command generating part that generates a control command while reducing a degree of freedom of motion of the operator, by generating a control command suitable for degree of freedom of a portion of the motion of the operator, on the basis of the estimated motion of the operator.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a program. [Background technology]

[0002] A control device that allows a user to assist in the operation of a robot has been proposed. As such a control device, for example, a control device has been proposed having a first information acquisition unit that acquires first user posture information indicating the posture of a first user operating the robot, a second information acquisition unit that acquires pre-change posture information indicating a pre-change posture that is a posture of the robot before the posture of the robot is changed based on the first user posture information, and a determination unit that determines a target posture different from the posture of the first user as the posture of the robot based on the pre-change posture information and the first user posture information acquired by the first information acquisition unit at the time when the robot is in the pre-change posture indicated by the pre-change posture information (see Patent Document 1). In the system described in Patent Document 1, the posture of the robot is changed to a posture corresponding to the posture detected by a device worn by the operator.

[0003] To perform various tasks using a remote-controlled robot, it is necessary to accurately command and control the six-degree-of-freedom hand target (position and posture) in three-dimensional space. For example, when making a robot open a plastic bottle when the vertical direction of the bottle is not perpendicular to the table, the operator who remotely controls the robot instructs the robot to turn the lid while aligning the gripping part with the vertical inclination of the plastic bottle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6476358 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technology, in order to perform a variety of tasks using a remote-controlled robot, the operator is required to accurately grasp the status of the robot and objects in the remote location, as well as to have advanced operating techniques, which limits the tasks that can be performed and reduces efficiency. For this reason, with conventional technology, it was difficult for an operator to determine and accurately control the target values ​​of six degrees of freedom (position and orientation) in space during remote operation and perform the task.

[0006] The present invention has been made in consideration of the above problems, and aims to provide a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a program that make it easier for an operator to perform work. [Means for solving the problem]

[0007] (1) In order to achieve the above object, a robot remote operation control device according to one embodiment of the present invention, in a robot remote operation that recognizes an operator's movement and transmits the operator's movement to a robot to operate the robot, includes an intention estimation unit that estimates an operator's movement based on a robot environment sensor value obtained by an environmental sensor installed on the robot or in the robot's surrounding environment and an operator sensor value, which is the operator's movement obtained by an operator sensor, and a control command generation unit that generates a control command by reducing the degree of freedom of the operator's movement by generating an appropriate control command for some of the operator's movement based on the estimated operator's movement.

[0008] (2) In addition, in a robot remote operation control device according to one aspect of the present invention, the control command generation unit may limit the degrees of freedom that the operator should control or the controllable range, and provide operational assistance for the limited degrees of freedom among the operational instructions given by the operator to the robot.

[0009] (3) Furthermore, in a robot remote operation control device according to one aspect of the present invention, the control command generation unit does not reduce the degree of freedom of the operator's movements when the distance between a gripping unit provided on the robot and a target object to be operated by the operator is outside a predetermined range, and reduces the degree of freedom of the operator's movements when the distance between a gripping unit provided on the robot and a target object to be operated by the operator is within a predetermined range. The robot environment sensor value may include captured image information and depth information.

[0010] (4) In addition, in a robot remote operation control device according to one aspect of the present invention, the intention estimation unit may input the robot environment sensor value and the operator sensor value to a learned intention estimation model to estimate a movement of the operator.

[0011] (5) In addition, in a robot remote operation control device according to one embodiment of the present invention, the operator sensor value may be at least one of the operator's gaze information and operator arm information, which is information regarding the posture or position of the operator's arm.

[0012] (6) In the robot remote operation control device according to one aspect of the present invention, the robot environment sensor value may include captured image information and depth information.

[0013] (7) In order to achieve the above object, a robot remote operation control system according to one embodiment of the present invention comprises a gripping unit for gripping an object, a robot remote operation control device described in any one of (1) to (6) above for robot remote operation that recognizes an operator's movements and transmits the operator's movements to a robot to operate the robot, an environmental sensor installed on the robot or in the robot's surrounding environment for detecting a robot environmental sensor value, and an operator sensor for detecting the operator's movements as an operator sensor value.

[0014] (8) In order to achieve the above-mentioned object, a robot remote operation control method according to one embodiment of the present invention is a robot remote operation method for recognizing an operator's movements and transmitting the operator's movements to a robot to operate the robot, in which an intention estimation unit estimates the operator's movements based on a robot environment sensor value obtained by an environmental sensor installed on the robot or in the robot's surrounding environment and an operator sensor value, which is the operator's movements obtained by an operator sensor, and a control command generation unit generates appropriate control commands for some of the degrees of freedom of the operator's movements based on the estimated operator's movements, thereby generating control commands by reducing the degrees of freedom of the operator's movements.

[0015] (9) In order to achieve the above-mentioned object, a program according to one embodiment of the present invention, in a robot remote operation that recognizes an operator's movements and transmits the operator's movements to a robot to operate the robot, estimates the operator's movements based on robot environment sensor values ​​obtained by an environmental sensor installed on the robot or in the robot's surrounding environment, and operator sensor values, which are the operator's movements obtained by an operator sensor, and generates appropriate control commands for some of the degrees of freedom of the operator's movements based on the estimated operator's movements, thereby generating control commands by reducing the degrees of freedom of the operator's movements. Effect of the Invention

[0016] According to (1) to (9), by substituting the generation of some of the control target values ​​of the six degrees of freedom depending on the situation, the degrees of freedom that the operator should control are limited, making it easier for the operator to perform the work. [Brief description of the drawings]

[0017] [Figure 1] 1 is a diagram showing an overview of a robot remote operation control system according to an embodiment and an overview of operations thereof; [Diagram 2] 1 is a block diagram showing an example of the configuration of a robot remote operation control system according to an embodiment; [Diagram 3]FIG. 1 is a diagram showing an example of a state in which an operator wears an HMD and a controller. [Figure 4] FIG. 4 is a diagram showing an overview of intention estimation and control command generation processing according to the embodiment. [Diagram 5] FIG. 13 is a diagram showing a case where the operator intends to open the cap of a plastic bottle. [Figure 6] FIG. 13 is a diagram showing a case where the operator intends to grab a box. [Figure 7] FIG. 4 is a diagram illustrating an example of information stored in a storage unit according to the embodiment. [Figure 8] FIG. 13 is a diagram showing an example of correction of a hand target of a grasping position with the assistance of a robot remote operation control device. [Figure 9] 10 is a flowchart of an example of a processing procedure of a robot and a robot remote operation control device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component is of a recognizable size.

[0019] [overview] First, an overview of the operations and processes performed by the robot remote operation control system will be given. FIG. 1 is a diagram showing an overview of a robot remote operation control system 1 according to this embodiment and an overview of the work. As shown in FIG. 1, an operator Us wears, for example, an HMD (head mounted display) 5 and a controller 6. Environmental sensors 7 (7a, 7b) are installed in the work environment. An environmental sensor 7c may be attached to the robot 2. The environmental sensors 7 (7a, 7b) include, for example, an RGB camera and a depth sensor, as described later. The operator Us remotely controls the robot 2 by moving the hand or fingers wearing the controller 6 while looking at an image displayed on the HMD 5. In the example of FIG. 1, the operator Us remotely controls the robot 2 to grip a plastic bottle obj on a table Tb and, for example, open the cap of the plastic bottle. In the remote operation, the operator Us cannot directly see the operation of the robot 2, but can indirectly see the image on the robot 2 side through the HMD 5. In this embodiment, the operator's intention (information about the target object, work content) is inferred based on information acquired by the controller 6 worn by the operator and information acquired by the environmental sensor 7, and the operation is supported by limiting the degree of freedom that the operator should control based on the inferred result.

[0020] [Example of a robot remote control system configuration] Next, a configuration example of the robot remote operation control system 1 will be described. 2 is a block diagram showing an example of the configuration of a robot remote operation control system 1 according to this embodiment. As shown in FIG. 2, the robot remote operation control system 1 includes a robot 2, a robot remote operation control device 3, an HMD 5, a controller 6, and an environmental sensor 7.

[0021] The robot 2 includes, for example, a control unit 21, a driving unit 22, a sound collection unit 23, a memory unit 25, a power source 26, and a sensor 27. The robot remote operation control device 3 includes, for example, an information acquisition unit 31, an intention estimation unit 33, a control command generation unit 34, a robot state image creation unit 35, a transmission unit 36, and a storage unit 37.

[0022] The HMD 5 includes, for example, an image display unit 51, a line-of-sight detection unit 52 (operator sensor), a control unit 54, and a communication unit 55. The HMD 5 may include, for example, a sensor that detects the inclination of the operator's head.

[0023] The controller 6 includes, for example, a sensor 61 (operator sensor), a control unit 62, a communication unit 63, and a feedback means 64.

[0024] The environmental sensor 7 includes, for example, an imaging device 71, a sensor 72, and a communication unit 73.

[0025] The robot remote operation control device 3 and the HMD 5 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the controller 6 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the environmental sensor 7 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the robot 2 are connected, for example, via a wireless or wired network. The robot remote operation control device 3 and the HMD 5 may be directly connected without a network. The robot remote operation control device 3 and the controller 6 may be directly connected without a network. The robot remote operation control device 3 and the environmental sensor 7 may be directly connected without a network. The robot remote operation control device 3 and the robot 2 may be directly connected without a network.

[0026] [Examples of functions for a robot remote control system] Next, a functional example of the robot remote operation control system will be described with reference to FIG. The HMD 5 displays the robot status image received from the robot remote operation control device 3. The HMD 5 detects the movement of the operator's line of sight, and transmits the detected line of sight information (operator sensor value) to the robot remote operation control device 3.

[0027] The image display unit 51 displays the robot status image received from the robot remote control device 3 under the control of the control unit 54.

[0028] The gaze detection unit 52 detects the gaze of the operator, and outputs the detected gaze information to the control unit 54.

[0029] The control unit 54 transmits the gaze information based on the gaze information detected by the gaze detection unit 52 to the robot remote operation control device 3 via the communication unit 55. The control unit 54 causes the image display unit 51 to display the robot status image transmitted by the robot remote operation control device 3.

[0030] The communication unit 55 receives the robot status image transmitted by the robot remote operation control device 3, and outputs the received robot status image to the control unit 54. The communication unit 55 transmits operator status information to the robot remote operation control device 3 in response to the control of the control unit 54.

[0031] The controller 6 is, for example, a tactile data glove, and is worn on the hand of the operator. The controller 6 detects operator arm information, which is information on the posture and position of the operator's arm, such as the direction, the movement of each finger, and the movement of the hand, by a sensor 61, and transmits the detected operator arm information (operator sensor value) to the robot remote operation control device 3. The operator arm information (operator sensor value) includes information covering the entire human arm, such as hand position and posture information, angle information of each finger, position and posture information of the elbow, and information tracking the movement of each part.

[0032] The sensor 61 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 61 includes a plurality of sensors, and tracks the movement of each finger using, for example, two sensors. The sensor 61 detects the operator's arm information and outputs the detected operator's arm information to the control unit 62.

[0033] The control unit 62 transmits operator arm information based on the detection result detected by the sensor 61 to the robot remote operation control device 3 via the communication unit 63. The control unit 62 controls the feedback means 64 based on the feedback information.

[0034] The communication unit 63 transmits the operator arm information to the robot remote operation control device 3 in response to the control of the control unit 62. The communication unit 63 acquires the feedback information transmitted by the robot remote operation control device 3, and outputs the acquired feedback information to the control unit 62.

[0035] The feedback means 64 feeds back feedback information to the operator in response to the control of the control unit 62. The feedback means 64 feeds back sensations to the operator in response to the feedback information, for example, by means for applying vibrations (not shown), means for applying air pressure (not shown), means for restricting hand movement (not shown), means for making the operator feel temperature (not shown), means for making the operator feel hardness or softness (not shown), etc., which are attached to the gripping unit 222 of the robot 2.

[0036] The environmental sensor 7 is installed in a position where it can capture and detect, for example, the work of the robot 2. The environmental sensor 7 may be provided in the robot 2 or may be attached to the robot 2. Alternatively, there may be multiple environmental sensors 7, and they may be installed in the work environment as shown in FIG. 1 and also attached to the robot 2. The environmental sensor 7 transmits the captured images and the detected detection results (environmental sensor values) to the robot remote operation control device 3.

[0037] The photographing device 71 is, for example, an RGB camera. The photographing device 71 transmits a photographed image to the robot remote control device 3 via the communication unit 73. In the environment sensor 7, the positional relationship between the photographing device 71 and the sensor 72 is known.

[0038] The sensor 72 is, for example, a depth sensor. The sensor 72 transmits the detected sensor value to the robot remote control device 3 via the communication unit 73. The image capturing device 71 and the sensor 72 may be distance sensors.

[0039] The communication unit 73 transmits the captured image and the sensor value detected by the sensor 72 as environmental sensor information to the robot remote operation control device 3. The environmental sensor 7 may detect position information of an object using the captured image and the sensor value, and transmit the detection result as environmental sensor information to the robot remote operation control device 3. The data transmitted by the environmental sensor 7 may be, for example, a point cloud having position information.

[0040] When the robot 2 is not remotely operated, the behavior of the robot 2 is controlled according to the control of the control unit 21. When the robot 2 is remotely operated, the behavior of the robot 2 is controlled according to the grasping plan information generated by the robot remote operation control device 3.

[0041] The control unit 21 controls the driving unit 22 based on a control command output by the robot remote operation control device 3. Based on the information stored in the storage unit 25, the control unit 21 acquires the three-dimensional position, size, shape, and the like of the target object in the captured image by using the image captured by the environment sensor 7 and the detection result, using a known method. The control unit 21 may perform voice recognition processing (such as speech period detection, sound source separation, sound source localization, noise suppression, and sound source identification) on the acoustic signal collected by the sound collection unit 23. When the result of the voice recognition includes an operation instruction for the robot, the control unit 21 may control the operation of the robot 2 based on the operation instruction by voice. The control unit 21 generates feedback information and transmits the generated feedback information to the controller 6 via the robot remote operation control device 3.

[0042] The driving unit 22 drives each part (arms, fingers, legs, head, torso, waist, etc.) of the robot 2 according to the control of the control unit 21. The driving unit 22 includes, for example, an actuator, a gear, an artificial muscle, etc.

[0043] The sound collection unit 23 is, for example, a microphone array having a plurality of microphones. The sound collection unit 23 outputs a collected acoustic signal to the control unit 21. The sound collection unit 23 may have a voice recognition processing function. In this case, the sound collection unit 23 outputs a voice recognition result to the control unit 21.

[0044] The storage unit 25 stores, for example, a program, a threshold value, etc. used by the control unit 21 for control, and temporarily stores a voice recognition result, an image processing result, a control command, etc. The storage unit 25 may also be the storage unit 37. Alternatively, the storage unit 37 may also function as the storage unit 25.

[0045] The power source 26 supplies power to each component of the robot 2. The power source 26 may include, for example, a rechargeable battery or a charging circuit.

[0046] The sensor 27 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, an encoder for each joint, etc. The sensor 27 is attached to each joint, the head, etc. of the robot 2. The sensor 27 outputs the detection result to the control unit 21, the intention estimation unit 33, the control command generation unit 34, and the robot state image creation unit 35.

[0047] The robot remote operation control device 3 estimates the operator's intention to operate based on the operator detection value detected by the operator sensor (gaze detection unit 52 of the HMD 5, sensor 61 of the controller 6) and the environmental sensor value detected by the environmental sensor 7, and generates a control command for the robot 2.

[0048] The information acquisition unit 31 acquires gaze information from the HMD 5 and acquires operator arm information from the controller 6, and outputs the acquired gaze information and operator arm information to the intention estimation unit 33 and the robot state image creation unit 35. The information acquisition unit 31 acquires environmental sensor information from the environmental sensor 7, and outputs the acquired environmental sensor information to the intention estimation unit 33 and the robot state image creation unit 35. The information acquisition unit 31 acquires detection results detected by the sensor 27, and outputs the acquired detection results to the intention estimation unit 33 and the robot state image creation unit 35.

[0049] The intention estimation unit 33 estimates information about the target object (robot environment sensor value) (such as the name of the target object, the position of the target object, and the vertical inclination of the target object) based on the environmental sensor information acquired by the information acquisition unit 31. The intention estimation unit 33 performs image processing (such as edge detection, binarization processing, feature extraction, image enhancement processing, image extraction, and pattern matching processing) on ​​the image captured by the image capture device 71 of the environmental sensor 7 based on the information stored in the storage unit 37, for example, to estimate the name of the target object. The intention estimation unit 33 estimates the position and vertical inclination angle of the target object based on the image processing result and the detection result of the depth sensor. The intention estimation unit 33 also estimates the target object using, for example, the gaze information of the operator detected by the HMD 5. The intention estimation unit 33 estimates the operator's intention to operate based on the gaze information and operator arm information acquired by the information acquisition unit 31, and information about the target object. The operator's intention and the estimation method will be described later.

[0050] The control command generation unit 34 generates a control command for grasping an object based on the result estimated by the intention estimation unit 33, the detection result detected by the sensor 27, and the image captured by the environment sensor 7 and the detection result. As described later, the control command generation unit 34 generates a control command by restricting the degree of freedom to be controlled by the operator and the controllable range, that is, by reducing the degree of freedom of the operator's movements. The control command generation unit 34 outputs the determined control command information to the control unit 21.

[0051] The robot state image creation unit 35 creates a robot state image to be displayed on the HMD 5 based on the control command information generated by the control command generation unit 34.

[0052] The transmission unit 36 ​​transmits the robot state image created by the robot state image creation unit 35 to the HMD 5. The transmission unit 36 ​​acquires feedback information output by the robot 2, and transmits the acquired feedback information to the controller 6.

[0053] The storage unit 37 stores the positional relationship between the image capture device 71 and the sensor 72 of the environmental sensor 7. The storage unit 37 stores information on the target to be assisted for each task, i.e., the degree of freedom that the operator should control and the controllable range. The storage unit 37 stores model images to be compared in the pattern matching process of the image processing. The storage unit 37 stores programs used to control the robot remote operation control device 3. The programs may be stored in the cloud or on a network.

[0054] [Example of an operator wearing HMD5 and controller 6] Next, an example of a state in which the operator wears the HMD 5 and the controller 6 will be described. Fig. 3 is a diagram showing an example of a state in which an operator wears the HMD 5 and the controller 6. In the example of Fig. 3, the operator Us wears the controller 6a on his left hand, the controller 6b on his right hand, and the HMD 5 on his head. Note that the HMD 5 and the controller 6 shown in Fig. 3 are merely examples, and the wearing method, shape, and the like are not limited to these.

[0055] [Examples of intention estimation and control command generation] Next, an overview of the intention estimation and control command generation process will be described. FIG. 4 is a diagram showing an overview of the intention estimation and control command generation process according to this embodiment. As shown in FIG. 4, the robot remote control device 3 acquires line of sight information from the HMD 5 , operator arm information from the controller 6 , environmental sensor information from the environmental sensor 7 , and the detection results detected by the sensor 27 .

[0056] The information obtained from the robot 2 is information such as detection values ​​of motor encoders provided at each joint of the robot 2. The information obtained from the environmental sensor 7 includes an image captured by the RGB camera, a detection value detected by the depth sensor, and the like.

[0057] The intention estimation unit 33 estimates the operator's intention to perform a motion based on the acquired information and the information stored in the storage unit 37. The operator's intention is, for example, an object to be operated, operation contents, etc. The operation contents are, for example, opening the cap of a plastic bottle, etc.

[0058] The control command generation unit 34 limits the degree of freedom that the operator should control and the controllable range according to the target object and the work content based on the information acquired as a result of the estimation by the intention estimation unit 33. Then, the control command generation unit 34 generates a control command based on the result of the estimation by the intention estimation unit 33.

[0059] [Explanation of examples of limitations on the degree of freedom that the operator should control and the controllable range] Next, examples of limiting the degrees of freedom that the operator should control and the controllable range will be described with reference to Fig. 5 and Fig. 6. Note that the xyz axes in Fig. 5 and Fig. 6 are the xyz axes in the robot world. The robot remote operation control device 3 of this embodiment limits the degrees of freedom and corrects the hand target as follows, based on the xyz axes in the robot world.

[0060] (First operation example) FIG. 5 is a diagram showing a case where the operator intends to open the cap of a plastic bottle. The intention estimation unit 33 estimates the target object Obj1 based on the tracking result of the operator's hand, the operator's line of sight information, and the image and detection result captured by the environmental sensor 7. As a result, the intention estimation unit 33 estimates that the operation target Obj1 is a "plastic bottle" based on the acquired information. Furthermore, the intention estimation unit 33 detects, for example, the vertical direction of the plastic bottle and the inclination of the vertical direction of the plastic bottle with respect to, for example, the z-axis direction, based on the image and detection result captured by the environmental sensor 7.

[0061] Next, the intention estimation unit 33 estimates the operation content based on the tracking result of the operator's hand, the operator's line of sight information, and the image and detection result captured by the environment sensor 7. As a result, the intention estimation unit 33 estimates that the operation content is the action of "opening the cap of a plastic bottle" based on the acquired information.

[0062] In the case of Fig. 5, in the conventional method, the operator had to remotely control the gripper 221 of the robot 2 so as to orient the gripper 221 vertically to the PET bottle and to open the lid. Thus, the conventional method had a problem in that the operator himself was required to have a high level of operating skill.

[0063] In contrast, in this embodiment, the robot remote control device 3 substitutes for the generation of some of the control target values ​​of the six degrees of freedom depending on the situation, thereby limiting the degrees of freedom that the operator should control. In the example of Fig. 5, the robot remote control device 3 controls and assists the gripper 221 of the robot 2 so that the gripper 221 stays vertically upward. This allows the operator to concentrate on giving instructions to rotate the cap without worrying about the inclination of the plastic bottle.

[0064] (Second operation example) FIG. 6 is a diagram showing a case where the operator intends to grab a box. The intention estimation unit 33 estimates that the operation object Obj2 is a "box" based on the acquired information. Next, the intention estimation unit 33 estimates, based on the acquired information, that the operation content is the action of "grabbing a box."

[0065] In the case of Fig. 6, the robot remote operation control device 3 assists, for example, the position of the operation target Obj2 on the xy plane and the rotation around the xyz axes by controlling the gripper 221. This allows the operator to concentrate on instructing the translational motion of the z-axis position.

[0066] Note that the target object, the work content, the assistance content (degree of freedom), etc. shown in Figures 5 and 6 are merely examples, and the target object, the work content, and the assistance content are not limited to these. The assistance content may be anything that corresponds to the target object and the work content.

[0067] Here, an example of information stored in the storage unit 37 and used for assisting with work will be described. Fig. 7 is a diagram showing an example of information stored in the storage unit 37 according to this embodiment. As shown in Fig. 7, the storage unit 37 stores a target object, a task content, a degree of freedom restricted for the operator (a degree of freedom reinforced by the robot 2), and a degree of freedom that is operable by the operator, in association with each other. Note that the example shown in Fig. 7 is just one example, and other information may also be stored in association with each other.

[0068] In addition, since the coordinate system is different between the environment in which the operator operates and the robot operating environment, for example, calibration of the operator operating environment and the robot operating environment may be performed when the robot 2 is started.

[0069] In addition, when grasping, the robot remote operation control device 3 may determine the grasping position based on the grasping force of the robot 2 and the frictional force between the object and the grasping part, etc., taking into account the error in the grasping position at the time of grasping.

[0070] [Correction of the hand target for the grasping position] Next, an example of correcting the hand target of the gripping position with the assistance of the robot remote operation control device 3 will be described. FIG. 8 is a diagram showing an example of correction of the hand target of the gripping position by the assistance of the robot remote operation control device 3. In the example of FIG. 8, the shape of the target object g21 to be gripped is an approximately rectangular parallelepiped. When providing assistance by restricting the degree of freedom so that the object is easy to grip in response to the operator's input g11, the robot remote operation control device 3 changes the hand target to a position that is easy to grip (position after correction g12). For example, when the vertical direction of the target object is inclined with respect to the z-axis direction as shown in FIG. 5, the hand target is changed by correcting the angle of the z-axis direction of the operator's input to match the vertical direction of the plastic bottle, and generating a control command for the gripping unit 221. More specifically, first, the control command generating unit 34 estimates a pattern to be assisted for each target object. The estimation method is, for example, by matching with a database stored in the storage unit 37 or machine learning. After estimating the assistance pattern, the control command generating unit 34 performs vector calculation (cross product and inner product) between the coordinate system of the target object and the hand coordinate system of the robot to correct the direction. In addition, the control command generator 34 may perform correction and generate command values ​​by an end-to-end machine learning method that directly generates corrected command values ​​from sensor information, including correction of direction by vector calculation. As a result, according to this embodiment, the robot remote operation control device 3 estimates the target object and the work content, and corrects the hand target to a position that is easy to grasp, making it easier for the operator to give instructions.

[0071] The intention estimation unit 33 or the storage unit 37 may be provided with an intention estimation model that inputs, for example, gaze information, operator arm information, environmental sensor information, and detection results detected by the sensors of the robot 2, and learns the target object and operation content as teacher data. The intention estimation model may be provided on a cloud. The intention estimation unit 33 may input the acquired information to the intention estimation model to estimate the intention of the operator. Alternatively, the intention estimation unit 33 may input the acquired information to the intention estimation model to estimate at least one of the target object and the operation content.

[0072] Alternatively, the intention estimation unit 33 may estimate the intention of the operator by probabilistic inference using the acquired information. Alternatively, the intention estimation unit 33 may estimate at least one of the target object and the work content by probabilistic inference using the acquired information.

[0073] [Example of processing procedure for robot 2 and robot remote control device 3] Next, an example of a processing procedure of the robot 2 and the robot remote operation control device 3 will be described. FIG. 9 is a flowchart of an example of a processing procedure of the robot 2 and the robot remote operation control device 3 according to this embodiment.

[0074] (Step S1 ) The information acquisition unit 31 acquires line-of-sight information from the HMD 5 and acquires operator arm information from the controller 6 .

[0075] (Step S2) The information acquisition unit 31 acquires environmental sensor information from the environmental sensor 7.

[0076] (Step S3) The intention estimation unit 33 estimates the intention of the operator, including the object to be grasped and the work content, based on the acquired line of sight information, operator arm information, and environmental sensor information.

[0077] (Step S4) The control command generating unit 34 determines the degree of freedom to be assisted by the robot remote control device 3, i.e., the restriction on the degree of freedom of the operator's motion instructions, based on the estimated operator's intention and the information stored in the storage unit 37. Note that the control command generating unit 34 generates control commands appropriate for some of the degrees of freedom of the operator's motion, thereby generating control commands with reduced degrees of freedom of the operator's motion.

[0078] (Step S5) The control command generator 34 corrects the hand position of the target to be grasped based on the arm movement information included in the acquired operator arm information and the restricted degree of freedom determined in step S4.

[0079] (Step S6) The robot state image creation unit 35 creates a robot state image to be displayed on the HMD 5 based on the operator arm information acquired by the information acquisition unit 31, the result estimated by the intention estimation unit 33, and the grip position corrected by the control command generation unit 34.

[0080] (Step S7) The control command generating unit 34 generates a control command based on the corrected hand position of the target to be grasped.

[0081] (Step S8) Based on the control command generated by the control command generating unit 34, the control unit 21 controls the driving unit 22 to drive the gripping unit and the like of the robot 2. After the process, the control unit 21 returns to the process of step S1.

[0082] The processing procedure shown in FIG. 9 is an example, and the robot remote operation control device 3 may perform the above-mentioned processes in parallel.

[0083] The image displayed on the image display unit 51 of the HMD 5 is, for example, an image as shown in FIG. 8. The robot state image does not have to include the image of the operator's input g11. The robot state image creation unit 35 may present the operator with a restriction on the degree of freedom and the content of the robot's assistance, for example, by presenting an assistance pattern in the form of characters or displaying a rectangle or an arrow on the HMD 5. This allows the operator, when giving a work instruction while looking at the robot state image displayed on the HMD 5, to concentrate on giving a work instruction to open the lid, for example, when opening a lid of a PET bottle, without worrying about aligning the gripping part with the vertical direction of the PET bottle.

[0084] As described above, in this embodiment, the robot remote operation control device 3 determines the work content and the target object through the sensor information installed on the robot 2, the environment, and the operator, and substitutes for the generation of some of the control target values ​​of the six degrees of freedom depending on the situation, thereby limiting the degrees of freedom that the operator should control. That is, in this embodiment, the degrees of freedom that the operator should control are reduced by automatically generating appropriate control commands for some of the degrees of freedom.

[0085] As a result, according to this embodiment, in a remote-controlled robot, it is possible to simultaneously reduce the burden on the operator, diversify the tasks, and increase the efficiency.

[0086] The above-mentioned examples of limiting the degree of freedom are merely examples, and are not intended to be limiting. Limiting the degree of freedom does not mean that the operator cannot completely control the degree of freedom, but also includes limiting the operable area. For example, the robot remote control device 3 may not restrict the degrees of freedom when the distance from the target object is outside a predetermined range, but may restrict the degrees of freedom when the distance from the target object is within a predetermined range. The predetermined range is, for example, a range of ±1 m on the translational x-axis.

[0087] In the above example, the target object is grasped by remote control, but the present invention is not limited to this example. Even if the robot is to perform other tasks, the robot remote control device 3 may limit the degree of freedom depending on the task. In addition, when there are multiple tasks, the robot remote operation control device 3 may change the restriction on the degree of freedom each time the task changes. For example, when multiple objects are placed on a table and a PET bottle is to be grasped from among the objects and then the lid of the PET bottle is to be opened, the robot remote operation control device 3 sets a restriction on the first degree of freedom for the stage of grasping and a restriction on the second degree of freedom for the stage of opening the lid.

[0088] Furthermore, in the case of a task performed using both grippers, the robot remote operation control device 3 limits the degree of freedom and corrects the hand target for each gripper.

[0089] Moreover, the above-mentioned robot 2 may be, for example, a bipedal robot, a stationary reception robot, or a working robot.

[0090] In the above example, the robot 2 is remotely controlled to grasp the object, but the present invention is not limited to this.

[0091] In the above example, the operator wears the HMD 5, but this is not limiting. Detection of gaze information and provision of a robot state image to the operator may be achieved by, for example, a combination of a sensor and an image display device.

[0092] In addition, a program for realizing all or part of the functions of the robot 2 and all or part of the functions of the robot remote operation control device 3 in the present invention may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform all or part of the processing performed by the robot 2 and all or part of the processing performed by the robot remote operation control device 3. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes systems built on a local network and systems built on the cloud. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" also includes those that hold a program for a certain period of time, such as volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0093] The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be for realizing part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0094] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0095] 1...robot remote operation control system, 2...robot, 3...robot remote operation control device, 5...HMD, 6...controller, 7...environmental sensor, 21...controller, 22...driving unit, 23...sound collection unit, 25...storage unit, 26...power supply, 27...sensor, 221...grasping unit, 31...information acquisition unit, 33...intention estimation unit, 34...control command generation unit, 35...robot state image creation unit, 36...transmission unit, 37...storage unit, 51...image display unit, 52...gaze detection unit, 54...control unit, 55...communication unit, 61...sensor, 62...control unit, 63...communication unit, 64...feedback means, 71...imaging device, 72...sensor, 73...communication unit

Claims

1. A robot remote control system for controlling a robot having a hand, which recognizes a motion of an operator and transmits the motion of the operator to the robot having a plurality of degrees of freedom with respect to the hand, comprises: an intention estimation unit that estimates a motion of the operator based on a robot environment sensor value obtained by an environmental sensor installed in the robot or in the surrounding environment of the robot, and an operator sensor value that is the motion of the operator obtained by an operator sensor; a control command generating unit that generates a control command for causing the robot to operate an object using the hand based on the estimated motion of the operator; Equipped with the control command generation unit replaces the control command for a part of the plurality of degrees of freedom with the control command having a predetermined content corresponding to the estimated motion of the operator, thereby limiting the part of the plurality of degrees of freedom that is operated by the motion of the operator. Robot remote control device.

2. The intention estimation unit, inputting the robot environment sensor value and the operator sensor value into a trained intention estimation model to estimate a motion of the operator; The robot remote operation control device according to claim 1 .

3. The operator sensor value is at least one of gaze information of the operator and operator arm information which is information regarding a posture or a position of the arm of the operator. The robot remote control device according to claim 1 or 2.

4. The robot environment sensor value includes captured image information and depth information. The robot remote operation control device according to any one of claims 1 to 3.

5. A robot remote control system for controlling a robot having a hand, which recognizes a motion of an operator and transmits the motion of the operator to the robot having a plurality of degrees of freedom with respect to the hand, comprises: The robot remote operation control device according to any one of claims 1 to 4, A gripping unit that grips an object; An environmental sensor that is installed on the robot or in the surrounding environment of the robot and detects a robot environmental sensor value; an operator sensor that detects a movement of the operator as an operator sensor value; A robot remote operation control system comprising:

6. A robot remote control system for controlling a robot having a hand, which recognizes a motion of an operator and transmits the motion of the operator to the robot having a plurality of degrees of freedom with respect to the hand, comprises: an intention estimation unit estimates a motion of the operator based on a robot environment sensor value obtained by an environment sensor installed in the robot or in a surrounding environment of the robot and an operator sensor value which is a motion of the operator obtained by an operator sensor; a control command generating unit generating a control command for causing the robot to operate an object using the hand, based on the estimated motion of the operator; the control command generating unit replaces the control command for a part of the plurality of degrees of freedom with the control command having a predetermined content corresponding to the estimated motion of the operator, thereby limiting the part of the plurality of degrees of freedom that is operated by the motion of the operator. A method for remotely controlling a robot.

7. A robot remote control system for controlling a robot having a hand, which recognizes a motion of an operator and transmits the motion of the operator to the robot having a plurality of degrees of freedom with respect to the hand, comprises: On the computer, A motion of the operator is estimated based on a robot environment sensor value obtained by an environment sensor installed in the robot or in the surrounding environment of the robot, and an operator sensor value which is a motion of the operator obtained by an operator sensor; generating a control command for causing the robot to operate an object using the hand based on the estimated motion of the operator; By replacing the control command for a part of the plurality of degrees of freedom with the control command having a predetermined content corresponding to the estimated motion of the operator, the degree of freedom that is a part of the plurality of degrees of freedom and is operated by the motion of the operator is limited. Program for.

Images