Remote Control System

Abstract

To provide a remote control system that can reflect movement of an operator's hand.SOLUTION: A remote control system in which an operator remotely operates a robot having an end effector that is capable of gripping and manipulating an object includes: an acquisition part acquiring information on a state of the operator operating the robot; an end effector capable of performing multiple types of gripping methods; a gripping method table in which the gripping methods are stored; and a gripping method selector selecting the gripping method from the gripping method table based on a joint flexion angle of the operator obtained from operator information of the operation information acquisition part.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a remote control system. [Background technology]

[0002] A control device that allows a user to assist in operating a robot has been proposed. For example, one such control device has a first information acquisition unit that acquires first user posture information that indicates the posture of a first user operating the robot, a second information acquisition unit that acquires pre-change posture information that indicates a pre-change posture that is the posture of the robot before the posture of the robot is changed based on the first user posture information, and a determination unit that determines a target posture different from the posture of the first user as the posture of the robot based on the pre-change posture information and the first user posture information acquired by the first information acquisition unit at the time when the robot is in the pre-change posture indicated by the pre-change posture information (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6476358 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, even if an attempt is made to reflect the operator's hand movements, it has been difficult to grasp the human hand movements.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a remote control system that can reflect the hand movements of an operator. [Means for solving the problem]

[0006] (1) In order to achieve the above object, a remote operation system according to one aspect of the present invention is a remote operation system in which an operator remotely operates a robot having an end effector capable of grasping an object, the remote operation system including: an acquisition unit that acquires information on the state of the operator operating the robot; an end effector that can perform a plurality of types of grasping methods; a grasping method table that stores the grasping methods; Recording Tokube The above was acquired by operator of the state Obtained from information The aforementioned Operator's Fingers a grasping method selection unit that selects the grasping method from the grasping method table based on a joint flexion angle; a grasping method determination unit that estimates a hand shape using a detection value detected by a sensor included in a controller worn by the operator, compares the estimated shape with information on grasping and target object stored in a database, and performs a cross-check to evaluate whether the selected grasping method is appropriate; Equipped with.

[0007] (2) In addition, in a remote control system according to one aspect of the present invention, learning data obtained by learning the characteristics of the operator's joint flexion angles may be stored for each grasping method stored in the grasping method table.

[0008] (3) In order to achieve the above object, a remote operation system according to one aspect of the present invention is a remote operation system in which an operator remotely operates a robot having an end effector capable of grasping an object, the remote operation system including: an acquisition unit that acquires information on the state of the operator operating the robot; an end effector that can perform a plurality of types of grasping methods; a grasping method table that stores the grasping methods; The object to be grasped by the end effector an intention estimation unit that compares the size of the target object with the size of the target object and estimates the intention of the operator; recommendation The operator specified of a gripping method selection unit that selects the gripping method from the gripping method table based on an intention. The intention estimation unit determines that the object is not an intended object that the operator is trying to grasp when the distance between the fingers is narrower than the size of the object, and determines that the object is an intended object that the operator is trying to grasp when the distance between the fingers is wider than the size of the object. . [Effects of the Invention]

[0009] According to (1) to (3), the movement of the operator's hand can be reflected. [Brief explanation of the drawings]

[0010] [Figure 1]1 is a diagram for explaining an overview of a remote control system according to an embodiment and an overview of operations thereof; [Figure 2] 1 is a diagram illustrating an example of the configuration of a remote control device according to a first embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of a model learning process. [Figure 4] FIG. 1 is a diagram illustrating an example of an online processing process. [Figure 5] FIG. 1 is a diagram illustrating an example of a taxonomy class. [Figure 6] FIG. 10 is a diagram illustrating an example of the configuration of a remote control device according to a second embodiment. [Figure 7] FIG. 10 is a diagram for explaining a geometric method, which is a second method according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0012] First Embodiment FIG. 1 is a diagram illustrating an overview of a remote control system and an overview of a work performed according to this embodiment. As shown in FIG. 1, an operator Us wears, for example, an HMD (head-mounted display) 501 and controllers 502 (502L, 502R) (data gloves). Environmental sensors 300 (300a, 300b) are installed in the workspace. The environmental sensors 300 may be attached to the robot 1. The robot 1 also includes hands 5 (5L, 5R). The environmental sensors 300 include, for example, an RGB camera and a depth sensor. The operator Us remotely controls the robot 1 by moving the hand or fingers wearing the controller 502 while viewing an image displayed on the HMD 501. In the example of FIG. 1, the operator Us remotely controls the robot 1 to grasp a plastic bottle obj on a table Tb. In remote control, the operator Us cannot directly see the movements of the robot 1, but can indirectly see the image of the robot 1 on the HMD 501.

[0013] In this embodiment, we build a method that takes the gestures being performed by a human as direct input to identify the optimal grasp for achieving the human's long-term intention. Human hand shapes have distinct characteristics depending on the type of grasp being performed. In this embodiment, these characteristics are used to train an ML model that classifies human intentions into a set of grasp classes. The ML model refers to the model artifact created during the model training process. In this embodiment, for each class in this fixed set, examples of humans performing gestures in realistic ways are collected, and a supervised learning algorithm is trained on this data to identify the correct class. This is done to maintain human autonomy by clarifying the number of possible classes and to naturally create a way for humans to choose among these classes.

[0014] (Configuration of remote control device) Next, a configuration example of the remote control device will be described. Fig. 2 is a diagram showing a configuration example of the remote control device according to this embodiment. As shown in FIG. 2, the remote control system 7 includes, for example, a remote control device 2, an environmental sensor 300, an HMD 501, a controller 502, and a robot 1.

[0015] The remote control device 2 includes an acquisition unit 21, a learning unit 22, a storage unit 23, a DB 24, a selection unit 25, an evaluation unit 26, a gripping method determination unit 27, a drive command generation unit 28, and a drive unit 29.

[0016] The remote control device 2 is connected to, for example, an environmental sensor 300, a controller 502, and a robot 1 via wire or wirelessly. H The MD501 may also be connected by wire or wirelessly.

[0017] The controller 502 includes a sensor 5021. The sensor 5021 includes a six-axis sensor, a gyro sensor, a position sensor, etc. The sensor may also include a force sensor.

[0018] The HMD 501 may include, for example, a line-of-sight detection sensor that detects the line of sight. Note that the remote control system 7 does not necessarily have to include the HMD 501.

[0019] The acquisition unit 21 includes an environmental sensor 300, an HMD 501, and Beauty The sensor values ​​detected by the sensors are obtained from the controller 502.

[0020] The learning unit 22 uses the sensor values ​​acquired by the acquisition unit 21 to learn the model 242, and stores the learned model 241 (grasping method table) in the DB 24 or updates it.

[0021] The storage unit 23 stores various values, thresholds, formulas, programs, etc. required for processing.

[0022] DB24 is a database. DB24 stores a model 241. DB24 stores information about grasping and target objects.

[0023] The selection unit 25 detects the hand shape based on the acquired sensor value. The selection unit 25 inputs information indicating the detected hand shape into the trained model and selects a class indicating the gripping method.

[0024] The evaluation unit 26 evaluates the reliability of the selected class.

[0025] The gripping method determination unit 27 determines the optimal gripping method (class) based on the evaluation result. For example, the gripping method determination unit 27 estimates and selects the taxonomy (see Reference 1) of the work that the operator is about to perform through the above process.

[0026] The drive command generating unit 28 generates a drive command based on the acquired sensor values ​​(including the position, shape, size, hand position, finger position, etc. of the target object) and the determined gripping method.

[0027] The driving unit 29 drives the end effector 5 of the robot 1 in accordance with the driving command generated by the driving command generating unit .

[0028] Reference 1; Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human Grasp Types” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb. 2016), IEEE, p66-77

[0029] (Model training) FIG. 3 is a diagram illustrating an example of a model learning process. (Step S1) When creating a model, in order to collect data to be used for learning, the worker wears data gloves on his / her hands and performs tasks for each taxonomy, for example. The tasks involve the worker wearing the data gloves on his / her hands, for example, grabbing a bottle, pinching the mouth of a bottle, or holding a teacup from above. The learning unit 22 acquires data (sensor values ​​of the data gloves) from the tasks performed by the worker in chronological order. The collected data may be data for each taxonomy that has been created in advance.

[0030] (Step S2) The learning unit 22 detects the state of the hand by detecting information on the joint flexion angles, such as the bending angles of the operator's fingers, based on the sensor values ​​detected by the sensors included in the controller 502. The learning unit 22 may also remove unnecessary data from the acquired data depending on the usage situation and application.

[0031] (Step S3) The learning unit 22 stores information such as the bending angle of the operator's finger as a data set.

[0032] (Step S4) The learning unit 22 uses this data set and the teacher data (classes) to train the model 241. In this way, the learning unit causes the model 241 to learn to correctly identify classes.

[0033] (Online processing) FIG. 4 is a diagram illustrating an example of an online processing process. (Step S11) During actual work, the acquisition unit 21 acquires sensor values ​​of the data glove (controller 502). The acquisition unit 21 also performs detection, for example, at predetermined time intervals. For example, the selection unit 25 detects the state of the operator's hand by detecting information such as the bending angle of the operator's fingers based on the sensor values ​​detected by the sensor provided in the controller 502. The selection unit 25 may delete any unnecessary data from the acquired data.

[0034] (Step S12) The selection unit 25 estimates the gripping method (class) using a trained model based on the acquired sensor value and the detected result. For example, the remote control device 2 estimates the gripping method (class) by inputting the bending angle of the operator's fingers into the trained model 241. In other words, the selection unit 25 estimates whether the gripping method (class) is that taxonomy.

[0035] (Step S13) Selection The selection unit 25 outputs information indicating the estimated class to the gripping method determination unit 27.

[0036] (Step S14) The gripping method determination unit 27 performs a cross-check using the DB 24, which stores information about grips and target objects. In the cross-check, the hand shape may be estimated using a sensor value detected by a sensor provided in the controller 502, and the estimated shape may be compared with the information stored in the DB 24 to evaluate whether the gripping method is appropriate.

[0037] (Step S15) The gripping method determination unit 27 outputs the optimal class determined by the processing of step S14 to the drive command generation unit .

[0038] Here, we explain why the gripping method and taxonomy class can be determined from the hand shape. FIG. 5 is a diagram showing an example of a taxonomy class. There are multiple taxonomy classes, as shown in Figure 5. However, as shown in Figure 5, the shape of the fingers and the hand shape differ for each class. Therefore, it is possible to determine which class a person belongs to by the hand shape, such as the angle of the fingers.

[0039] As described above, in this embodiment, a model for classifying the class of a target object to be grasped is trained in advance using the shape of the worker's hand (e.g., the angle of the fingers) based on actual work data and training data. Then, in this embodiment, a grasping method is selected by inputting information indicating the shape of the worker's hand during work into the trained model.

[0040] As a result, according to this embodiment, the gripping method that is the operator's intention during operation can be appropriately selected, and the operator's hand movements can be reflected.

[0041] In addition, when the robot 1 has two arms, the remote control device 2 uses the sensor values ​​acquired when the operator works using both hands during learning. of The model 241 is trained using the trained model 241. Then, during work, the remote control device 2 uses the trained model 241 to determine a gripping method, including work performed with both arms.

[0042] The DB 24 and the model 241 may be located on the cloud or may be connected via a network.

[0043] Second Embodiment Prediction of human intentions can be achieved by utilizing various behavioral cues that humans exhibit during a task, such as changes in the shape of a human hand in anticipation of grasping a specific object for a specific purpose. Recent systems prefer to collect huge amounts of data and train deep learning models. These systems are not explicitly trained to mathematically interpret the shape of a human hand. They also tend to ignore the shape of a human hand, as it is difficult to capture in camera images due to many naturally occurring occlusions.

[0044] For this reason, in this embodiment, the shape of the operator's hand is detected using sensor values ​​provided in the controller 502. Furthermore, in this embodiment, human grip data is mathematically modeled. Furthermore, in this embodiment, a wearable finger motion detection device (for example, a data glove (controller 502)) equipped with a sensor is used, as in the first embodiment.

[0045] (Configuration of remote control device) Next, a configuration example of the remote control device will be described. Fig. 6 is a diagram showing a configuration example of the remote control device according to this embodiment. 6, the remote control system 7A includes, for example, a remote operation device 2A, an environmental sensor 300, an HMD 501, a controller 502, and a robot 1. Note that the remote control system 7A does not necessarily have to include the HMD 501.

[0046] The remote control device 2A includes an acquisition unit 21, a learning unit 22, a storage unit 23, a DB 24A, a selection unit 25, an evaluation unit 26, a gripping method determination unit 27, a drive command generation unit 28, a drive unit 29, an intention estimation unit 30, and measurement It has a section 31. DB24A also includes models 241, 242, and 243.

[0047] The remote control device 2A is connected to, for example, an environmental sensor 300, a controller 502, and a robot 1 by wire or wirelessly. H The MD501 may also be connected by wire or wirelessly.

[0048] The intention estimation unit 30 estimates the intention of the operator using a model 242 that can output a successful grasping pose for a given target object. The intention estimation unit 30 estimates the intention of the operator using the results of measurement by the measurement unit 31. The intention estimation unit 30 estimates the intention of the operator using a trained model 243.

[0049] The measurement unit 31 measures, for example, the distance between the thumb and other fingers based on the sensor value detected by the sensor provided in the data glove (controller 502). The measurement unit 31 also obtains the shape, size, and position of the target object based on the position information of each feature point of the target object included in the sensor value detected by the environment sensor 300.

[0050] (Processing content) In this embodiment, the controller 502 uses the following three methods to model the data collected by the environmental sensor 300 and estimate the operator's intention.

[0051] 1. First method: Back-computation based method The operator's intention is estimated using a model that can output the successful grasping pose for a given object ((Model 242) Reference: graspit: https: / / graspit-simulator.github.io / ).

[0052] 2. Second method: Geometric method Since humans always grasp objects with their thumbs, the system calculates the intention by comparing the distance between the thumb and other fingers with the dimensions of the object.

[0053] 3. Third method: Deep learning method By creating a dataset of when an operator wears a data glove (controller 502) and successfully grasps a target object, a deep learning model (model 243) for predicting intentions is created, and the intention is estimated using this model.

[0054] In this way, by effectively modeling the information about a human finger, this embodiment can improve the efficiency of all other human intention prediction models that did not take this information into account. As a result, this embodiment can accurately estimate the operator's task intention, i.e., which target object the operator is trying to touch.

[0055] (geometric method) Here, the second method, the geometric method, will be described. 7 is a diagram for explaining the geometric method, which is the second method according to this embodiment. The model in FIG. 7 represents the finger of a worker.

[0056] Image g101 represents the distance between two fingers (e.g., thumb and index finger) during grasping. Image g102 shows the case of gripping with three fingers (for example, thumb, index finger, and middle finger). When gripping with three fingers, the space created by the three fingers can be represented by triangle g103. This triangle g103 can be approximated, like triangle g104, with its base being the position F1 of the first finger and the position F2 of the second finger, and its vertex being the position T of the third finger.

[0057] In the second method, the distance between the modeled fingers is measured and compared with the size of the candidate target object to determine whether the candidate target object is the object the operator is trying to grasp. For example, if the distance between the fingers is narrower than the size of the candidate target object, it can be assumed that the object is not the intended object. Alternatively, if the distance between the fingers is wider than the size of the candidate target object, it can be assumed that the object is the intended object.

[0058] In this embodiment, human grip data is mathematically modeled, and a wearable finger motion detection device is used instead of a photographing device.

[0059] As described above, according to this embodiment, by effectively modeling human finger information, it is possible to improve the efficiency of all other human intention prediction models that do not take this information into account.

[0060] In addition, a program for realizing some or all of the functions of the remote control device 2 (or 2A) of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the remote control device 2 (or 2A). Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0061] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0062] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0063] 7, 7A... Remote operation system, 2, 2A... Remote operation device, 300... Environmental sensor, 501... HMD, 502... Controller, 1... Robot, 21... Acquisition unit, 22... Learning unit, 23... Memory unit, 24, 24A... DB, 25... Selection unit, 26... Evaluation unit, 27... Grasping method determination unit, 28... Drive command generation unit, 29... Drive unit, 30... Intention estimation unit, 31... measurement Parts, 241, 242, 243...Models, 5, 5L, 5R...End effector

Claims

1. In a remote control system in which an operator remotely controls a robot having an end effector capable of grasping and manipulating an object, an acquisition unit that acquires information about the state of an operator operating the robot; an end effector capable of performing multiple types of gripping methods; a gripping method table in which the gripping methods are stored; a gripping method selection unit that selects the gripping method from the gripping method table based on the flexion angle of the finger joints of the operator's hand obtained from the information on the state of the operator acquired by the acquisition unit; a grasping method determination unit that estimates a hand shape using a detection value detected by a sensor included in a controller worn by the operator, compares the estimated shape with information on grasping and target object stored in a database, and performs a cross-check to evaluate whether the selected grasping method is appropriate; A remote control system comprising:

2. For each of the gripping methods stored in the gripping method table, learning data obtained by learning feature amounts of the flexion angles of the finger joints of the operator's hand is stored. The remote control system according to claim 1 .

3. In a remote control system in which an operator remotely controls a robot having an end effector capable of grasping and manipulating an object, an acquisition unit that acquires information about the state of an operator operating the robot; an end effector capable of performing multiple types of gripping methods; a gripping method table in which the gripping methods are stored; an intention estimation unit that compares a distance between the tip of the thumb and the tips of the other fingers of the operator with a dimension of a target object to be grasped by the end effector, and estimates the intention of the operator; a gripping method selection unit that selects the gripping method from the gripping method table based on the estimated intention of the operator, The intention estimation unit determines that the object is not an intended object that the operator is trying to grasp when the distance between the fingers is narrower than the size of the object, and determines that the object is an intended object that the operator is trying to grasp when the distance between the fingers is wider than the size of the object. Remote control system.

Images