Remote operation assistance system and remote operation assistance method

Abstract

To provide a remote operation auxiliary system and a remote operation auxiliary method which can assist diversified teleoperations to perform various operations with respect to a large variety of objects.SOLUTION: A remote operation auxiliary system comprises: a behavior acquisition part which acquires information regarding behavior of an operator of end effector; a target object which is a target to be operated by the end effector based on the information acquired by the behavior acquisition part; an intention understanding part which estimates a task which is an operation method of the target object; an environmental status determination part which acquires environment information of the environment where the end effector is operated; and an operation amount determination part which determines operation amount of the end effector based on the information acquired by the intention understanding part and the environmental status determination part. The operation amount determination part estimates an operation distance to the target object based on the information of the target object and the behavior of the operator, determines operation status of the end effector regarding the target object according to the estimated operation distance, and changes input mode of auxiliary regarding remote operation according to the operation status.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a remote operation assistance system and a remote operation assistance method.

Background Art

[0002] Techniques for remotely operating and controlling a robot have been proposed (see, for example, Patent Document 1). In such remote operation, for example, the robot is remotely operated according to an instruction from a sensor worn by the operator. The robot needs to perform various operations on various objects according to the remote operation instruction.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the technique described in Patent Document 1 cannot assist in diverse teleoperation tasks that perform various operations on various objects.

[0005] The present invention has been made in view of the above problems, and an object thereof is to provide a remote operation assistance system and a remote operation assistance method that can assist in diverse teleoperation tasks that perform various operations on various objects.

Means for Solving the Problems

[0006] (1) To achieve the above object, a remote operation assistance system according to an aspect of the present invention is a remote operation assistance system for remotely operating at least an end effector, including an operation acquisition unit that acquires information regarding the operation of an operator who operates at least the end effector, a target object that is an object to be operated by the end effector using the information obtained by the operation acquisition unit, an intention understanding unit that estimates a task, which is a method of operating the target object, and an environmental situation determination unit that acquires environmental information of the environment in which the end effector is operated. The remote operation assistance system further includes an operation amount determination unit that acquires information from the intention understanding unit and the environmental situation determination unit and determines an operation amount of the end effector from the acquired information. The operation amount determination unit estimates an operation distance from the information of the target object and the operation of the operator to the target object, determines an operation state of the end effector with respect to the target object according to the estimated operation distance, and changes an input mode of assistance for remote control according to the operation state.

[0007] (2) Further, in the remote operation assistance system according to (1) of an aspect of the present invention, the operation state is divided into a distant state in which the distance from the target object is far, a peripheral state in which the position with respect to the target object is close, and an operation state in which the target object is operated. The input mode of the assistance includes a distant mode in which the operation of the operator is directly used when the distance from the target object is far, a peripheral mode in which the operator is guided to the working position of the target object when the position with respect to the target object is close, and an operation mode in which the operation on the target object is assisted when the target object is in the operation state.

[0008] (3) Also, in the remote operation assistance system according to (2) of one aspect of the present invention, the peripheral mode includes a state of approaching the object with the hand (Approach Object), a state of moving to a position where the object can be grasped (Snap to Object), a state of adjusting the final grasping position according to the operator's instruction in accordance with the surface of the object (Align with Object), and a state of moving away from the vicinity of the object (Unsnap to Object). The operation amount determination unit switches the transition between the four states according to the work content and the work state.

[0009] (4) Also, in the remote operation assistance system according to (3) of one aspect of the present invention, the operation amount determination unit inputs the fingertip command value of the operator acquired by the motion acquisition unit, the intention information estimated by the intention understanding unit, the environment information acquired by the environment situation determination unit, and the teacher data which is the work content, and uses the learned model to switch the transition between the four states according to the work content and the work state.

[0010] (5) To achieve the above object, a remote operation assistance method according to one aspect of the present invention is a remote operation assistance system for remotely operating at least an end effector, and includes a motion acquisition unit that acquires information regarding the operation of an operator who operates at least the end effector, an object to be operated by the end effector using the information obtained by the motion acquisition unit, an intention understanding unit that estimates a task which is a method of operating the object, an environment situation determination unit that acquires environment information of the environment in which the end effector is operated, and an operation amount determination unit that acquires information of the intention understanding unit and the environment situation determination unit and determines the operation amount of the end effector from the acquired information. The operation amount determination unit estimates the operation distance from the object information and the operator's motion to the object, determines the operation state of the end effector with respect to the object according to the estimated operation distance, and changes the input mode of assistance for remote control according to the operation state.

Effect of the Invention

[0011] (1) to (5) can assist in diverse teleoperation tasks that perform diverse operations on various objects.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Modes for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scales of the respective members are appropriately changed in order to make the respective members recognizable in size. In all the drawings for explaining the embodiments, those having the same function are denoted by the same reference numerals, and repeated explanations are omitted. In addition, "based on XX" as used in the present application means "based on at least XX", and includes cases based on other elements in addition to XX. Further, "based on XX" is not limited to the case of directly using XX, and includes cases based on those obtained by performing operations or processing on XX. "XX" is an arbitrary element (for example, arbitrary information).

[0014] <Remote operation assistance system> FIG. 1 is a diagram showing a configuration example of a remote operation assistance system according to the present embodiment. As shown in FIG. 1, the remote operation assistance system 1 includes an operation acquisition unit 2, an intention understanding unit 3, an environmental situation determination unit 4, an operation amount determination unit 5, and a robot 6. The operation amount determination unit 5 includes, for example, an operation distance estimation unit 51, an operation state determination unit 52, and a state determination unit 53. The robot 6 includes, for example, an arm 61 and an end effector 62. In FIG. 1, the robot controller 7 is omitted from the illustration.

[0015] The operation acquisition unit 2 acquires at least information regarding the operation of the operator who operates the end effector 62. The operation acquisition unit 2 is, for example, a data glove.

[0016] The intention understanding unit 3 uses the information acquired by the operation acquisition unit 2 to estimate the target object to be operated by the end effector 62 and the task that is the operation method of the target object. Note that the intention understanding unit 3 performs intention estimation using, for example, the method of Japanese Patent Application No. 2021-058952. Further, the intention understanding unit 3 may estimate the target object based on, for example, the position information of the operator's fingertips and the line-of-sight information of the operator. Further, the estimated task is, for example, the taxonomy of the work that the operator is about to perform (see Reference 1).

[0017] Reference 1: Thomas Feix, Javier Romero, et al., "The GRASP Taxonomy of Human Grasp Types", IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb. 2016), IEEE, p66-77

[0018] The environmental condition determination unit 4 acquires environmental information of the environment in which the end effector 62 is operated. The environmental condition determination unit 4 may be, for example, an imaging device installed in the working environment or an imaging device attached to the robot 6. The environmental condition determination unit 4 acquires the shape, size, position, etc. of the target object using the RGB (red, green, blue) information and depth information included in the captured image.

[0019] The operation amount determination unit 5 acquires the information of the intention understanding unit 3 and the environmental condition determination unit 4, and determines the operation amount of the end effector 62 from the acquired information.

[0020] The operation distance estimation unit 51 estimates the operation distance to the target object from the information of the target object and the operator's movement. The operation distance to the target object may be, for example, the distance within the screen visible to the operator through the HMD or the actual distance on the robot 6 side.

[0021] The operation state determination unit 52 determines the operation state of the end effector with respect to the target object according to the operation distance estimated by the operation distance estimation unit 51. The state is, for example, the state where the operation is started, during operation, the grasped state, etc.

[0022] The state determination unit 53 changes the input mode of assistance for remote control according to the operation state.

[0023] The robot 6 is, for example, any one of a single-arm robot, a double-arm robot, and a multi-arm robot. The arm 61 has an end effector 62 connected to its tip. The end effector 62 includes at least two fingers. The number of fingers included in the end effector 62 may be three or more.

[0024] <Overview of remote operation> FIG. 2 is a diagram showing an overview of the remote operation according to the present embodiment. Note that, although the robot 6 shown in FIG. 2 shows an example including a double bowl, a head, and a body, the configuration, shape, etc. of the robot 6 are not limited to this. As shown in FIG. 2, the operator Us wears, for example, an HMD (head-mounted display) 2c and data gloves 2a, 2b. An environmental situation determination unit 4a (environmental sensor) is installed around the robot 6 or the robot 6, and an environmental sensor 4b is also installed in the working environment. Note that the environmental situation determination unit 4 (4a) (environmental sensor) may be attached to the robot 6. Further, the robot 6 includes end effectors 62 (62a, 62b). The operator Us remotely operates the robot 6 by moving the hands and fingers wearing the data gloves 2a, 2b while looking at the image displayed on the HMD 2c.

[0025] <Example of processing procedure> Next, an example of the processing procedure of the remote operation assistance system 1 will be described. FIG. 3 is a flowchart showing an example of the processing procedure of the remote operation assistance system according to the present embodiment.

[0026] (Step S1) The motion acquisition unit 2 acquires information regarding the motion of at least the operator who operates the end effector 62.

[0027] (Step S2) The intention understanding unit 3 uses the information acquired by the motion acquisition unit 2 to estimate the target object to be operated by the end effector 62 and the task that is the operation method of the target object.

[0028] (Step S3) The environmental situation determination unit 4 acquires environmental information of the environment in which the end effector 62 is operated.

[0029] (Step S4) The operation distance estimation unit 51 estimates the operation distance to the target object from the information of the target object and the motion of the operator.

[0030] (Step S5) The operation state determination unit 52 classifies the operation distance estimated by the operation distance estimation unit 51, that is, the distance between the human fingertip command value and the target object, into being greater than or equal to a first distance (a far state where the distance from the object is far), less than the first distance and greater than or equal to a second distance (a peripheral state where the position of the object is close), and less than the second distance (an operation state for operating the object). Here, being greater than or equal to the first distance is, for example, a state where the distance between the human fingertip command value and the target object is far. Being less than the first distance and greater than or equal to the second distance is, for example, a state where the human fingertip command value can be regarded as being around the target object. Being less than the second distance is, for example, a state where the distance between the human fingertip command value and the target object is close. Note that the first distance is, for example, 20 cm in the case of the actual distance on the robot 6 side. The second distance is, for example, 5 cm in the case of the actual distance on the robot 6 side. Note that the length of the distance is just an example, and it may be a distance corresponding to the target object, for example.

[0031] (Step S6) When the operation state determination unit 52 determines that the distance is greater than or equal to the first distance, it proceeds to the process in Step S7 (the first mode). When the operation state determination unit 52 determines that the distance is less than the first distance and greater than or equal to the second distance, it proceeds to the process in Step S8 (the second mode). When the operation state determination unit 52 determines that the distance is less than the second distance, it proceeds to the process in Step S9 (the third mode).

[0032] (Step S7) The state determination unit 53 outputs the acquired human fingertip command value as it is to the robot 6 as a command value without modifying the human fingertip command value.

[0033] (Step S8) The state determination unit 53 modifies the human fingertip command value. More specifically, the state determination unit 53 moves the command value to the robot 6 to a position where work can be performed while avoiding a collision with the target object.

[0034] (Step S9) The state determination unit 53 modifies the human fingertip command value. More specifically, while ensuring that the distance between the command value to the robot 6 and the target object remains within a range where work can be immediately performed, it accepts instructions for starting and ending the work operation.

[0035] As described above, in this embodiment, the auxiliary input mode is changed according to the operation state (including the distance from the object). The auxiliary input mode is the remote mode (step S7) that directly uses the operator's actions when the distance from the object is far (the remote state). The auxiliary input mode is the peripheral mode (step S8) that guides to the working position of the target object when the position of the object is close (the peripheral state). The auxiliary input mode is the operation mode (step S9) that assists the operation on the target object when in the operation state of operating the object. Note that the corrections in steps S8 and S9 are performed using the models learned in each mode.

[0036] Note that the remote operation assistance system 1 repeats the above processing until one operation (for example, start of gripping, gripping, leaving after gripping, etc.) is completed.

[0037] [First Embodiment] In this embodiment, an example in which the second mode has four states will be described. FIG. 4 is a diagram for explaining the correction of the human fingertip command value in the second mode according to this embodiment. As shown in FIG. 4, the operation amount determination unit 5 corrects the input human fingertip command value using parameters. The fingertip command value represents the human hand state and includes, for example, the position and orientation of the human wrist and the finger joint angle information detected by the sensors provided in the data glove. The parameters are the results detected by the environmental situation determination unit 4 and are, for example, information about the target object such as the position information of the target object and the shape of the target object.

[0038] The second mode can be divided into four states as shown in FIG. 4, for example. "Approach Object (parameter)" is an instruction to bring the hand closer to the object and is used when there is a certain distance. "Snap to Object (parameter)" is an instruction to move to a position where the object can be grasped and is used when moving to the vicinity of an object where work can be started immediately. "Align with Object (parameter)" is an instruction to adjust the final grasping position according to a person's instruction, for example, in alignment with the surface of the target object, and is used for adjusting the working point along the surface of the target object. Further, "unsnap from object (parameter)", which will be described later, is used when moving away from the vicinity of the target object.

[0039] However, since the transition criteria between the above modes and states are determined by the distance between the hand command value of the operator and the object of interest, a module for separately estimating the target object and its detailed information (position, orientation, shape, etc.) is required. At the same time, detailed parameters are required to determine the specifications of each mode and state, and these parameters are also determined by the above target object and its information.

[0040] In this way, in the present embodiment, when in the peripheral state close to the object, the work is performed by transitioning through the four states (Approach Object, Snap to Object, Align with Object, and unsnap from object) of the second mode.

[0041] In the above example, four examples of the states of the second mode are described as shown in FIG. 4, but the number of states of the second mode may be three or less, or five or more. Also, in the above example, an example of dividing the distance between the hand command value of a person and the target object into three is described, but it is not limited to this. The distance between the hand command value of a person and the target object may be divided into two, or four or more.

[0042] (In the case of gripping work) Here, a processing example performed by the operation amount determination unit 5 during the gripping operation will be described. FIG. 5 is a diagram showing a processing example during the gripping operation according to the present embodiment. Note that Interaction with Object505 is an instruction by the operator's intervention on the support by operating the fingertips of the data glove, for example. For example, when the gripping position is displaced, the operator instructs to correct the displacement with the fingertips of the data glove. Or, when the operator wants to grip with the fingertips when the work progresses, the operator operates the fingertips of the data glove to close (or open, etc.) the fingers. And Align with Object503 corrects based on the intervention when there is such an intervention of the operator's instruction.

[0043] Assume that the condition before the operation is that the distance between the end effector 62 and the object is slightly separated. Therefore, the human fingertip command value is in a state of not gripping the target object and not touching the target object, with the hands empty (g11).

[0044] In this state, since there is a slight distance between the target object and the finger part of the end effector 62, the finger part is brought closer to the target object (g12).

[0045] In this operation, the process is performed without the support of Approach Object501 (g13), and the finger part is brought closer to the target object within a predetermined range (g14).

[0046] Next, Snap to Object502 activates the support operation and moves to a position where it can be safely gripped (for example, so as not to collide with the target object) (g15).

[0047] Next, in response to an instruction for the person to adjust the final gripping position, Align with Object503 moves the finger part along the surface of the target object (g16).

[0048] Next, Interaction with Object505 closes the finger part of the end effector 62 (g17). Thereby, the end effector 62 grips the target object.

[0049] In the gripping operation, after gripping, it is necessary to separate the finger part from the target object. Therefore, Unsnap to Object 504 removes the finger part from the target object and moves it away from the target object (g18).

[0050] Note that for each of the above processes, a robot command in which the human hand tip command value is corrected is output to the robot 6.

[0051] (In the case of the placing operation) Next, an example of the process performed by the operation amount determination unit 5 during the placing operation will be described. FIG. 6 is a diagram showing an example of the process during the placing operation according to the present embodiment. Assume that the work content is to place the target object being gripped on the table.

[0052] The finger part of the end effector 62 is gripping the target object (g31).

[0053] In this state, the end effector 62 that is still gripping is being brought closer to the table, but there is still a slight distance between the table and the end effector 62 (g32).

[0054] In this operation, the process is performed without the support of Approach Object 501 (g33), and the end effector 62 is brought closer to the table within a predetermined range (g34).

[0055] Next, Snap to Object 502 activates the support operation and moves to a position where it can be safely placed (for example, so that the target object or the end effector 62 does not collide with the table) (g35).

[0056] Next, Align with Object 503 finely adjusts the position of the end effector 62 along the surface of the table in response to an instruction from a human to finely adjust to the final placement position (g36).

[0057] Next, Interaction with Object 505 opens the finger part of the end effector 62 (g37). As a result, the target object is placed on the table.

[0058] Even in the placement operation, after placing the target object, since it is necessary to separate the end effector 62 from the target object, Unsnap to Object 504 separates the end effector 62 from the target object (g38).

[0059] (In the case of the operation of tightening a screw) Next, a processing example performed by the operation amount determination unit 5 during the operation of tightening a screw will be described. FIG. 7 is a diagram showing a processing example during the operation of tightening a screw according to the present embodiment. Assume that the work content is to grip a wrench in advance and tighten a hexagon screw with the wrench.

[0060] The finger portion of the end effector 62 grips the wrench (g51).

[0061] In this state, the end effector 62 gripping the wrench is approaching the screw, but there is a slight distance between the end effector 62 and the screw (g52).

[0062] In this operation, the process is performed without the support of Approach Object 501 (g53), and the end effector 62 is brought closer to the screw within a predetermined range (g54).

[0063] Next, Snap to Object 502 activates the support operation and moves the end effector 62 to a position where the operation of tightening the screw can be started (g55).

[0064] Next, Align with Object 503 aligns the wrench along the surface of the screw, and in response to an instruction from a person to finely adjust to the final working position, Align with Object 503 finely adjusts the position of the end effector 62 (g56).

[0065] Next, Interaction with Object 505 controls the operation of the end effector 62 to perform screw tightening (g57).

[0066] After the screwing is completed, Unsnap to Object504 causes the end effector 62 to move away from the screw (g58).

[0067] In the above-described example, the operation of tightening the screw was described, but the same applies to the operation of loosening the screw.

[0068] Also, in each operation described with reference to FIGS. 5 to 7, the operation distance estimation unit 51 estimates the operation distance from the information of the target object and the operator's movement to the target object. The operation state determination unit 52 determines the operation state of the end effector with respect to the target object according to the operation distance estimated by the operation distance estimation unit 51. The state determination unit 53 changes the input mode of assistance for remote operation according to the operation state. In addition, the processing of each state and the transition between states in the operations using FIGS. 5 to 7 are learned by inputting, for example, the fingertip command value, the estimated intention information, the recognized environment information, and the teacher data into the learning model in advance.

[0069] Note that, as described above, the operation state is divided into a far state where the distance from the target object is far, a peripheral state where the position of the target object is close, and an operation state where the target object is operated. Also, the input mode of assistance is a remote mode that directly uses the operator's movement when in the far state where the distance from the target object is far, a peripheral mode that guides to the working position of the target object when in the peripheral state where the position of the target object is close, and an operation mode that assists the operation on the target object when in the operation state where the target object is operated.

[0070] Note that the peripheral mode is divided into a state of approaching the hand to the target object (Approach Object), a state of moving to a position where the target object can be grasped (Snap to Object), a state of adjusting the final grasping position according to the operator's instruction in accordance with the surface of the target object (Align with Object), and a state of moving away from the vicinity of the target object (Unsnap to Object). Then, the operation amount determination unit 5 switches the transition between the four states according to the work content and the work state.

[0071] Here, an operation example by an end effector according to a comparative example will be described. FIG. 8 is a diagram for explaining an operation example by an end effector according to a comparative example. For example, assume that an object obj at location A on a desk is picked (grasped and lifted) (g900) and placed (moved and released) (g910, g911) to location B. When the robot hand moves away from the object obj, it may move freely according to the operator's manual input. However, when approaching, it is required by the support function to approach a position where it does not collide with the object obj and can be grasped, but this has not been done in the prior art.

[0072] On the other hand, in the present embodiment, the operation amount determination unit 5 corrects the human hand tip command value or changes the correction method according to the distance between the human hand tip command value and the target object as described above. Thus, according to the present embodiment, it is possible to approach a position where it does not collide with the object obj and can be grasped when approaching.

[0073] Thereby, according to the present embodiment, the second mode for correcting the hand tip command value is established with a unified mechanism. Also, according to the present embodiment, it is possible to cope with various tasks in a general-purpose mode and a plurality of states regardless of the type of task.

[0074] FIG. 9 is a diagram for explaining another operation example by an end effector according to a comparative example. The operation in FIG. 9 is an operation of tightening a hexagonal screw with a wrench. In such an operation, the wrench being held may move freely according to the operator's manual input during normal times. However, when approaching the hexagonal screw which is the target object to be worked on, it is preferable that the support function works, but this has not been done in the prior art.

[0075] On the other hand, in the present embodiment, when approaching the hexagonal screw which is the target object to be worked on, the support function works, and it is possible to smoothly snap to a position where the hexagonal screw can be tightened.

[0076] [Second Embodiment] In this embodiment, an example in which the second mode has nine states will be described. FIG. 10 is a diagram for explaining the outline of the remote operation assistance system according to this embodiment. In this embodiment, the operator wears, for example, an HMD and a data glove like reference numeral g501. The data glove detects the movement of the operator's hand and fingers. Further, the movement of the fingers is detected by, for example, continuous buttons provided in the data glove. For this reason, the state of the fingers is represented by a continuous value [0, 1]. Also, the rendered scene is presented to the operator by the HMD. The operator can freely walk around the scene in the VR. Reference numeral g501 represents the working space of the robot, and reference numeral g503 shows an example of an image displayed on the HDM. Reference numeral g502 represents the working space of the robot 6.

[0077] The remote operation assistance system 1A processes in combination with inputs such as the state of the operator's fingers, the operator's line of sight, the state of the robot, and environmental information, and generates a command to the robot. The remote operation assistance system 1A estimates the operator's intention and modifies the operator's input command according to the situation.

[0078] Also in this embodiment, the current hand state of the operator may not be sent to the robot as a target, and the hand state may be processed by the operation amount determination unit 5A. In this embodiment, the hand state is variously changed according to the current action state (for example, grasping an object, approaching), and the achievement of the operator's goal is supported.

[0079] Further, the framework of this embodiment is composed of an intention understanding unit that estimates the operator's intention, an environment determination unit that determines the environmental situation, an operation amount determination unit 5A that generates and corrects the target command of the robot based on the current action state, and various modules that provide feedback to the operator.

[0080] Note that the intention understanding unit estimates the operator's intention by tracking the operator's hand gesture and line of sight. For example, the intention understanding unit causes, for example, a hidden Markov model to learn to infer the most likely object of interest from the operator's line-of-sight pattern and the trajectory of hand movement in a VR environment, and estimates the operator's intention using the learned model.

[0081] In addition, the remote operation assistance system 1A has tactile and visual feedback functions. As a result, in a remote operation environment, a two-way communication channel is established between the operator and the robot system through feedback connections of all modalities. Through this feedback connection, the operator can not only send commands to the remote operation assistance system 1A, but rather can interact with the remote operation assistance system 1A.

[0082] Note that in order to assist a human operator in achieving an operation goal, it is important for the operator to always maintain a sense of control, that is, a sense of agency, over the system. In this case, it is useful for the operator not to even notice that the remote operation assistance system 1A is operating. For this purpose, the support provided by the remote operation assistance system 1A should always be subtle.

[0083] In the following description, a pick-and-place task will be described in which the robot grasps a target object, moves the grasped target object to a desired position, and places it at the desired position. FIG. 11 is a diagram showing a configuration example of the remote operation assistance system according to the present embodiment. As shown in FIG. 11, the remote operation assistance system 1A includes an operation acquisition unit 2, an intention understanding unit 3, an environmental situation determination unit 4, an operation amount determination unit 5A, a robot 6, a robot controller 7, and a feedback device 8. Note that the remote operation assistance system 1 described in the first embodiment may also include a feedback device 8.

[0084] The operation amount determination unit 5A includes, for example, an operation state machine 55, a virtual environment simulator 56, and a controller simulator 57.

[0085] Based on the current input, the operating state machine 55 that determines how to convert the operator's input into the target hand state of the robot 6 is activated. Then, the controller simulator 57 uses a virtual copy of the robot controller 7 to check whether the end effector of the robot can safely reach the target object. If it is determined that it can reach safely, the target object is sent to the actual robot controller 7. And the end effector moves to the target object. Further, for example, a feedback signal to the operator is generated when the robot 6 collides or when it cannot reach the target object.

[0086] (Scene understanding and intention estimation) Since the operation task requires physical interaction with the target object in the environment, it is first necessary to be able to detect the target object in the scene. Further, it is necessary to infer the operation intention of the operator. For this reason, the intention understanding unit 3 infers the operation intention of the operator, and the environmental situation judgment unit 4 detects the target object in the scene.

[0087] In object detection, for example, the database of object models is referred to. The database is, for example, the object model of the YCB-video dataset. Each object is accompanied by a mesh and primitive geometric representations (cylinder, sphere, box, cone). Note that since collision detection using the mesh is too costly, the primitive shape is used for collision detection. For object detection and pose estimation, for example, the scene is photographed with two stationary calibrated RGB-D cameras. The images are segmented by an instance of MASK R-CNN (Reference 2) for each camera. Point cloud data is generated for each segment and fused between the cameras. After feature extraction, global registration (Reference 3) and fine registration (Reference 4) are performed. Finally, the results are filtered using pose confidence estimation (similar to visible surface inconsistency in Reference 5).

[0088] Reference 2; K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask r-cnn”, in IEEE International Conference on Computer Vision, 2017. Reference 3; Q.-Y. Zhou, J. Park, and V. Koltun, “Fast global registration”, in European Conference on Computer Vision, 2016. Reference 4; Y. Chen and G. Medioni, “Object modelling by registration of multiple range images”, Image and Vision Computing, vol. 10, no. 3, pp. 145-155, 1992. Reference 5; T. Homas, J. Matas, and S. Obdrzalek, “On evaluation of 6d object pose estimation,” in European Conference on Computer Vision, 2016.

[0089] The detection of the operator's intention infers, for example, by tracking the operator's hand movements and line of sight, what actions (pick, place, move) the operator wants to perform on which object. For example, in the case of picking, the teleoperation assistance system 1A estimates the target object to be grasped and the most likely grasp that the operator wants to perform on the target object. In the case of placing, the teleoperation assistance system 1A estimates the most likely placement location and several parameters required to place the object. Note that the parameters may be some details regarding the exact placement location, placement area, and how the operator wants to place the target object (e.g., upside down, sideways, etc.).

[0090] (Estimation of grasping and placement postures) The grasping posture determines how the robot 6 grasps an object. The placement posture determines how a given object is placed on top of other objects. In order to calculate such postures, in this embodiment, a given object is decomposed into a set of manifolds. For example, a cylindrical object can be decomposed into two circular manifolds representing the bottom and top surfaces of the object and a cylindrical manifold representing the side surface of the object. This embodiment has derived functions that map different types of manifolds to each other. These functions can be used for calculating the grasping posture and the placement posture. For example, by mapping the circular manifold of the bottom surface of a cylindrical object to a rectangular parallelepiped manifold representing the top surface of a table, it is possible to generate a placement posture that determines how the cylindrical object is placed on the table. The palm of the robot 6's hand and the tool center point are also modeled as manifolds in the same way. Therefore, the same method can be used for calculating the grasping posture and the placement posture.

[0091] In shared autonomous teleoperation, the current grasping posture is determined by the current (virtual) position of the operator's hand and the object of interest indicated by the intention estimator. The placement posture is calculated when the object is grasped and depends on the relative posture between the grasped object and the placement position indicated by the intention estimator. Even if the system has already determined the grasping and placement postures, appropriate new postures on the corresponding manifolds can be found from the movement of the operator's hand. Therefore, the operator can perform fine adjustments without worrying about the consistency of the grasping posture and the placement posture. Thus, according to this embodiment, even when a remote operation assistance system 1A controls part of the operation of the robot 6, a sense of agency can be realized.

[0092] (Action) In this embodiment, the overall task of picking and placing the target object is decomposed into different (sub) tasks. The support provided by the remote operation assistance system 1A to the operator depends on and changes with the current sub task. In this embodiment, switching is performed between different sub tasks as shown in FIG. 12 based on the operator's intention, the distance between the hand of the robot 6 and the object, and the feasibility of the operator's current operation. FIG. 12 is a diagram showing examples of different sub tasks according to this embodiment. Note that the sub tasks are used in the second mode of the first embodiment.

[0093] (1)Approach Object When at least one object is detected in the scene and the system receives a pick intention for that object, the "approach object" operation extends the hand in an arbitrary direction. The remote operation assistance system 1A overrides those commands only when the robot 6 is about to reach its joint or speed limits. In that case, the remote operation assistance system 1A stops the robot 6 or slightly corrects the trajectory of the robot 6 to prevent reaching the limits. In the background, a grasping posture for the given object is calculated from the current hand posture of the robot 6. The grasping posture is updated, for example, at about 40 Hz.

[0094] (2)Snap to Object When the distance between the hand of the robot 6 and the calculated grasping posture becomes equal to or less than a predefined threshold, the remote operation assistance system 1A takes over control and automatically moves the end effector of the robot 6 to the grasping posture. During that short time (usually less than 1 second), the movement of the operator's hand does not affect the movement of the robot 6. The reason for performing such an operation is that it has been observed that it is difficult for the operator to accurately judge the relative distance between the hand and the object because there is no sense of depth.

[0095] (3)Align with Object Even after the end effector reaches the grasping posture, the operator can move the end effector around the target object within the null space of the stable grasping posture provided by the manifold used in the calculation of the grasping posture. For example, when snapped onto the side surface of a cylinder, the end effector can move the vertical axis up and down. It can also move around the circumference of the cylinder, but the movement near the roll angle is blocked to avoid collision with the object. Therefore, the operator can move the end effector to finely adjust the desired grasping posture and can operate while maintaining their initiative even when assisted with positioning. At this time, the robot 6 moves to the calculated grasping posture without colliding with the object. Note that the grasping posture also includes the desired finger joint arrangement. However, this finger arrangement is ignored until the operator actually starts grasping the object.

[0096] (4) Grasp Object When the operator is satisfied with the posture of the end effector and presses the button on the data glove, the fingers of the hand of the robot 6 are closed. In this operation, the end effector does not move. Since a feedforward controller is used, it can be simply assumed that the operator has grasped the target object in order to complete the operation instruction when feeling that the object has been grasped when the fingers are almost completely closed.

[0097] (5) Align with Surface After grasping the object, the operator can move the object on the current support surface (such as a table). Minor movements in the normal direction of the surface and rotations other than around the normal of the surface are ignored, and the object is kept in contact with the support surface. Similar to Align with Object, the teleoperation assistance system 1A can freely adjust the posture on the surface while keeping the target object in contact with the support surface. When the operator moves the data glove away from the support surface, the teleoperation assistance system 1A lifts the target object and switches to another operation.

[0098] (6) Unsnap from Surface When the operator virtually separates the hand from the support surface, the hand of the robot 6 can autonomously lift the object without colliding with other objects.

[0099] (7)Approach Surface After lifting the object, the teleoperation assistance system 1A returns to a state where it can freely move around the object in the scene again. Therefore, this mode is similar to Approach Object, except that the robot 6 holds the object with its hand.

[0100] (8)Snap to Surface When approaching the estimated installation position of the object, the teleoperation assistance system 1A takes control again and automatically arranges the held object to be stably placed on the surface. After snapping, the teleoperation assistance system 1A switches back to Align with Surface, and the operator can move the object on the support plane to finely adjust the placement posture, or open the fingers of the robot 6 to release it from the object (or adjust the hand accordingly).

[0101] (9)Release Object In this state, the robot 6 can simply open its fingers, and the end effector cannot be moved.

[0102] <Simulator> In this embodiment, before transmitting the target posture to the actual robot 6, first, whether the robot 6 can actually reach that posture is simulated. The controller simulator 57 calculates the joint speed using the decomposed motion rate controller, multiplies it by the step size, and adds it to the current joint angle of the robot 6. The controller simulator 57 checks whether the newly calculated posture of the robot 6 is within the limits of collisions or joints, and whether it has already reached the target target posture. The controller simulator 57 continues to execute those steps until the operation is considered successful, infeasible, or until the time limit (for example, 500 ms) is reached. The advantage of this controller simulator 57 is that it is fast enough to calculate in real time. Note that since the simulation is continuously executed, it has been confirmed that sufficient accuracy can be obtained. The same controller used to calculate the control commands for the actual robot 6 is used for the simulation.

[0103] <Feedback signal> Tactile feedback is used to warn the operator when the movement of the robot 6 is changed to avoid collisions of the robot 6, such as physical limits (for example, joint limits), self-collisions, or collisions with objects in the scene. Note that the feedback signal is transmitted to the data glove. The data glove is equipped with, for example, a vibrator, and transmits feedback information to the operator according to the feedback signal.

[0104] <Verification result> FIG. 13 is a diagram showing an example of a state in which a robot is working by remote control according to this embodiment. In the example of FIG. 13, the robot 6 grips a plastic bottle, which is an object to be manipulated, with its hand based on remote control, and rotates and opens the lid with a gripper.

[0105] Note that the robot 6 may have a gripper as an end effector with a single arm as shown in FIG. 10, or may have two arms as shown in FIG. 13, with a multi-fingered hand on one arm and a gripper on the other arm. As described above, in this embodiment, verification was performed using the robot 6 as shown in FIG. 10 and the robot 6 as shown in FIG. 13.

[0106] As described above, in this embodiment, a shared autonomous remote operation framework based on virtual reality is used to support a human operator during object interaction such as pick-and-place tasks.

[0107] Thereby, in this embodiment as well, it is possible to assist in diverse teleoperation tasks that perform diverse operations on diverse objects.

[0108] Note that a program for realizing all or part of the functions of the operation amount determination unit 5 (or 5A) in the present invention may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed to perform all or part of the processing performed by the operation amount determination unit 5 (or 5A). Here, the "computer system" shall include hardware such as an OS and peripheral devices. Further, the "computer system" shall also include a WWW system having a homepage providing environment (or display environment). Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, etc., and a storage device such as a hard disk built into a computer system. Furthermore, the "computer-readable recording medium" also includes a volatile memory (RAM) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and that holds the program for a certain period of time.

[0109] Also, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) like the Internet or a communication line (communication wire) like a telephone line. Further, the above program may be for realizing a part of the functions described above. Furthermore, it may be a so-called difference file (difference program) that can realize the functions described above in combination with a program already recorded in the computer system.

[0110] As described above, the embodiments for carrying out the present invention have been described using the embodiments. However, the present invention is not limited to such embodiments at all, and various modifications and substitutions can be made without departing from the gist of the present invention.

Explanation of Reference Numerals

[0111] 1, 1A... Remote operation assistance system, 2... Motion acquisition unit, 3... Intention understanding unit, 4... Environmental situation determination unit, 5, 5A... Operation amount determination unit, 6... Robot, 51... Operation distance estimation unit, 52... Operation state determination unit, 53... State determination unit, 61... Arm, 62... End effector

Claims

1. A remote operation assistance system for remotely operating at least an end effector, comprising: an operation acquisition unit that acquires information regarding the operation of an operator who operates at least the end effector; an intention understanding unit that estimates a target object to be operated by the end effector using the information obtained by the operation acquisition unit; an environmental situation determination unit that acquires environmental information of the environment in which the end effector is operated; an operation amount determination unit that acquires information from the intention understanding unit and the environmental situation determination unit and determines the operation amount of the end effector from the acquired information; The operation amount determination unit: estimates an operation distance from the information of the target object and the operation of the operator to the target object, determines the operation state of the end effector with respect to the target object according to the estimated operation distance, and changes the input mode of assistance for remote control according to the operation state; The operation state: a far state where the distance from the target object is far; a peripheral state where the position from the target object is close; is divided into an operation state of operating the target object; The input mode of the assistance: a far mode that directly uses the operation of the operator when the distance from the target object is far; a peripheral mode that guides to the working position of the target object when the position from the target object is close; an operation mode that assists the operation on the target object when in the operation state of operating the target object. Remote operation assistance system.

2. The peripheral mode: a state of approaching the hand to the target object (Approach Object); a state of moving to a position where the target object can be grasped (Snap to Object); a state of adjusting the final grasping position according to the instruction of the operator in accordance with the surface of the target object (Align with Object); a state of leaving from the vicinity of the target object (Unsnap to Object); and is divided into The operation amount determination unit: switches the transition between the four states according to the work content and the working state. The remote operation assistance system according to Claim 1.

3. The operation amount determination unit: uses a model that inputs and learns the hand command value of the operator acquired by the operation acquisition unit, the intention information estimated by the intention understanding unit, the environmental information acquired by the environmental situation determination unit, and the teacher data that is the work content, and switches the transition between the four states according to the work content and the working state. The remote operation assistance system according to claim 2.

4. A remote operation assistance system for remotely operating at least an end effector, an operation acquisition unit that acquires information regarding the operation of an operator who operates at least the end effector, an intention understanding unit that estimates a target object to be operated by the end effector using the information obtained by the operation acquisition unit, an environmental situation determination unit that acquires environmental information of the environment in which the end effector is operated, an operation amount determination unit that acquires information from the intention understanding unit and the environmental situation determination unit and determines the operation amount of the end effector from the acquired information, and the operation amount determination unit estimates an operation distance from the information of the target object and the operation of the operator to the target object, determines an operation state of the end effector with respect to the target object according to the estimated operation distance, and changes an input mode of assistance for remote control according to the operation state, the operation state is divided into a distant state where the distance from the target object is far, a peripheral state where the position from the target object is close, and an operation state of operating the target object, the input mode of the assistance is a distant mode that directly uses the operation of the operator when the distance from the target object is far, a peripheral mode that guides to the working position of the target object when the position from the target object is close, and an operation mode that assists the operation on the target object when in the operation state of operating the target object. Remote operation assistance method.

Images