System and method for controlling robot avatars by multiple users

Abstract

This system 1 for control of a robot avatar by a plurality of persons comprises: one robot avatar 2 that includes a control object 22, the operation of which is controlled; an input device 3 for generating, on the basis of instruction operations performed by each of a plurality of operators OP, a plurality of items of input information for causing the robot avatar 2 to execute a task; a first generation unit 512 for generating a plurality of operation commands for causing the control object 22 to operate, on the basis of the plurality of items of input information, so that the robot avatar 2 operates in accordance with the plurality of instruction operations; an operation control unit 61 for controlling the operation of the control object 22 on the basis of the plurality of operation commands; and a plurality of movement state presentation devices 41 worn by each of the plurality of operators OP, each of the movement state presentation devices 41 presenting a tactile stimulus so that each operator OP can ascertain the movement state associated with the instruction operations of the other operators OP than themselves during the use of the input device 3.

Description

[Technical field]

[0001] The present invention relates to a system and method for controlling a robot avatar by multiple people. [Background technology]

[0002] In recent years, robot avatar technology, in which a human being reflects his / her own movements in a robot and controls the robot as his / her avatar, has been attracting attention. When this robot avatar technology is applied, for example, an operator can remotely control a robot placed in a remote location and receive various information (visual information, auditory information, tactile information) acquired by the robot (robot avatar). In other words, the operator can experience being in the place where the robot is placed while controlling the robot, without actually going there. Furthermore, the application of the robot avatar technology is expected in cases where a robot is made to perform a task in a special environment where humans cannot enter or the like (Non-Patent Document 1).

[0003] Known methods for controlling a robot avatar include, for example, a method based on motion information of an operator measured by motion capture or the like (Non-Patent Document 2) and a method using a controller (Non-Patent Document 3).

[0004] Currently, research is being conducted on sharing one avatar body among multiple people based on the concept of cybernetic avatars, which build a new relationship between people and the body. Related research has targeted virtual avatars in cyberspace, attempting to control one avatar (virtual avatar) by sharing the arm movements of two operators (Non-Patent Document 4). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] O. Khatib, X. Yeh, G. Brantner, B. Soe, B. Kim, S. Ganguly, HS Stuart, S. Wang, M. Cutkosky, A. Edsinger, P. Mullins, M. Barham, C. Voolstra, K. Salama, M. L'Hour, and V. Creuze & Automation Magazine, Vol. 23, pp. 20-29,

Outdoor Tool2

Outdoor Tools3

Outdoor Tools 4

[0006] (Problem to be solved by the invention) Conventionally, in the field of robot avatar technology, the control of one robot (robot avatar) by multiple operators has not been considered at all.

[0007] Unlike virtual avatars in cyberspace, which can be redone any number of times, robot avatars perform tasks in the real world (real space), and therefore require higher operational capabilities than virtual avatars. Summary of the Invention

[0008] An object of the present invention is to provide a robot avatar control system that can be used by multiple people and has excellent operability.

[0009] (Means for solving the problem) The means for solving the above problems are as follows. <1> A robot avatar control system for multiple operators, comprising: a robot avatar including a control object whose movement is controlled; an input device that generates a plurality of pieces of input information based on instruction movements made by a plurality of operators to cause the robot avatar to perform a task; a first generation unit that generates a plurality of movement commands for moving the control object based on the plurality of pieces of input information so that the robot avatar moves in accordance with the plurality of instruction movements; a movement control unit that controls the movement of the control object based on the plurality of movement commands; and a plurality of movement state presentation devices that are worn by the plurality of operators respectively and that each presents a tactile stimulus so that when the input device is used, each operator can grasp the movement state related to the instruction movements of the other operators other than himself / herself.

[0010] <2> a second generation unit that generates motion information corresponding to the motion state based on the input information or based on information on a motion state of the control object that operates in response to the instruction motion, and the motion state presentation device presents the tactile stimulus based on the motion information related to the other operator other than the operator himself / herself. <1> A robot avatar control system for multiple people as described above.

[0011] <3> The second generation unit generates the motion information by using a physical quantity related to the motion state. <2> A robot avatar control system for multiple people as described above.

[0012] <4> The motion state presentation device has a vibrator that presents vibration as the tactile stimulus to the operator. <1> ~ <3> 13. A robot avatar control system for multiple people according to any one of the above.

[0013] <5> The plurality of operators are assigned different control objects or different control purposes, and the first generation unit generates a plurality of the operation commands for the plurality of operators to control the different control objects or for the plurality of operators to control for the different control purposes based on the plurality of pieces of input information. <1> ~ <4> 13. A robot avatar control system for multiple people according to any one of the above.

[0014] <6> a ratio of contribution to control of the specific control object is determined for each of the multiple operators, and the first generation unit generates the multiple operation commands for the multiple operators to control the specific control object according to the ratio based on the multiple pieces of input information. <1> ~ <4> 13. A robot avatar control system for multiple people according to any one of the above.

[0015] <7> The robot avatar has an action unit operable to act on an object, and a body unit movable while holding the action unit. <1> ~ <6> 13. A robot avatar control system for multiple people according to any one of the above.

[0016] <8> a detection sensor attached to the action unit and detecting a physical action that the action unit receives from the object side when the action unit acts on the object; and an action unit information presentation device attached to each of the operators and presenting a tactile stimulus corresponding to a detection result of the detection sensor to each operator so that the action that the action unit receives from the object side can be shared among the operators when the action unit is in operation. <7> A robot avatar control system for multiple people as described above.

[0017] <9> The instruction action performed by the operator on the input device is a three-dimensional action of moving a part of the operator's body three-dimensionally. <1> ~ <8> 13. A robot avatar control system for multiple people according to any one of the above.

[0018] <10> The robot avatar is provided with a display device that displays an image of the robot avatar to the operator so that the operator can perform the instruction action without seeing the actual robot avatar. <1> ~ <9> 13. A robot avatar control system for multiple people according to any one of the above.

[0019] <11> A control method in which a single robot avatar including a control object whose movement is controlled is controlled by multiple operators, the control method comprising: an input information generation process in which an input device generates multiple pieces of input information based on instruction movements made by each of the multiple operators to cause the robot avatar to perform a task; a movement command generation process in which an input device generates multiple movement commands for moving the control object based on the multiple pieces of input information so that the robot avatar moves in accordance with the multiple instruction movements; a movement control process in which the movement of the control object is controlled based on the multiple movement commands; and a tactile stimulus presentation process in which multiple movement state presentation devices worn by each of the multiple operators respectively present tactile stimuli to the multiple operators so that each operator can grasp the movement state related to the instruction movements of the other operators when using the input device.

[0020] <12> a motion information generating step of generating motion information corresponding to the motion state based on the input information or based on information regarding a motion state of the control object that operates in response to the instruction motion, and in the tactile stimulus presenting step, presenting the tactile stimulus based on the motion information regarding the other operator other than the operator himself / herself. <11> The control method described in

[0021] <13> a plurality of operators are assigned with different control objects or different control purposes, and the operation command step generates a plurality of operation commands for the plurality of operators to control the different control objects or for the plurality of operators to control with the different control purposes based on a plurality of pieces of input information. <11> or <12> The control method described in

[0022] <14> a ratio of contribution to the control of the specific control object is determined for each of the multiple operators, and the operation command step generates a plurality of the operation commands for the multiple operators to control the specific control object according to the ratio based on the multiple pieces of input information. <11> or <12> The control method described in

[0023] <15> The control object is the action part and the main body part. <7> A robot avatar control system for multiple people as described above.

[0024] <16> The control object is a part of the main body and another part of the main body. <7> A robot avatar control system for multiple people as described above.

[0025] <17> The control object is a part of the action unit and another part of the action unit. <7> A robot avatar control system for multiple people as described above.

[0026] <18> The control object is the main body, and the control purpose is position control of the action unit and attitude control of the action unit. <7> A robot avatar control system for multiple people as described above.

[0027] (Effects of the Invention) According to the present invention, it is possible to provide a robot avatar control system that can be used by multiple people and has excellent operability. [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 is an explanatory diagram showing the overall configuration of a robot avatar control system for multiple users according to a first embodiment. [Diagram 2] Illustration of a robot avatar [Diagram 3] FIG. 1 is an explanatory diagram showing an input device and an information presentation device worn by a first operator; [Figure 4] FIG. 13 is an explanatory diagram showing an input device and an information presentation device worn by a second operator. [Diagram 5] FIG. 11 is an explanatory diagram showing another input device attached to a second operator. [Figure 6] A diagram showing the hardware configuration of an operating computer. [Figure 7]FIG. 1 is an explanatory diagram showing the relationship between input information input by each operator and feedback information returned to each operator in the control system of the first embodiment. [Figure 8] FIG. 13 is an explanatory diagram showing the overall configuration of a control system according to a second embodiment. [Figure 9] FIG. 11 is an explanatory diagram showing the relationship between input information input by each operator and feedback information returned to each operator in the control system of the third embodiment. [Figure 10] FIG. 13 is an explanatory diagram showing the relationship between input information input by each operator and feedback information returned to each operator in the control system of the fourth embodiment. [Figure 11] FIG. 13 is a schematic explanatory diagram showing an action unit of the robot avatar of the fourth embodiment; [Figure 12] FIG. 13 is an explanatory diagram showing the relationship between input information input by each operator and feedback information returned to each operator in the control system of the fifth embodiment. [Figure 13] FIG. 13 is an explanatory diagram showing the relationship between input information input by each operator and feedback information returned to each operator in the control system of the sixth embodiment. [Figure 14] FIG. 13 is an explanatory diagram showing the relationship between input information input by each operator and feedback information returned to each operator in the control system of the seventh embodiment. [Figure 15] FIG. 13 is an explanatory diagram showing the relationship between input information input by each operator and feedback information returned to each operator in the control system of the eighth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] <Embodiment 1> A robot avatar control system 1 operated by multiple operators according to a first embodiment of the present invention will be described with reference to Figs. 1 to 7. Fig. 1 is an explanatory diagram showing an overall configuration of the robot avatar control system 1 operated by multiple operators according to the first embodiment. The robot avatar control system 1 operated by multiple operators according to the present embodiment is a system intended for multiple operators OP to jointly operate one robot avatar 2 while sharing the operation among themselves and understanding each other's operating conditions. In this specification, for convenience of explanation, the "robot avatar control system operated by multiple operators" may be simply referred to as a "control system".

[0030] Needless to say, each operator OP can grasp (perceive) its own operation status by itself. However, it is difficult or impossible to grasp the operation status of other operator OPs other than itself. For example, when each operator OP operates the robot avatar 2 from a place distant from each other, the operator OPs cannot visually confirm and cannot grasp each other's operation status. In addition, since the operator OP concentrates on operating the robot avatar 2, it is difficult for the operator OP to grasp the operation status of the other operator OPs even if the other operator OPs are nearby. Note that, although it may be possible to grasp the operation status of the other operator OPs from the state of the movement of the robot avatar 2, in this case, it is not possible to instantly grasp the operation status of the other operator OPs. Moreover, when the movement of the robot avatar 2 is very slight, it is difficult to visually confirm such movement.

[0031] In the control system 1 of this embodiment, when each operator OP operates the robot avatar 2, a tactile stimulus is presented to each operator OP so that the operator OP can intuitively and quickly grasp the motion state (one example of a motion state related to the instruction motion) corresponding to the instruction motion of the other operator OP. That is, each operator OP can grasp the motion state (operation status) corresponding to the instruction motion of the other operator OP from the presence or absence of a tactile stimulus (for example, a vibration stimulus) and a change (change in strength, change in type, etc.). Even if there are multiple operators OP other than the operator OP, the motion states (operation status) corresponding to the instruction motions of the multiple operators OP can be distinguished and grasped by appropriately setting the type of tactile stimulus, the pattern in which the tactile stimulus is presented, etc. In addition, in another embodiment, as described later, the motion information related to the instruction motion of the operator OP may be the motion state of the control object that operates in response to the instruction motion of the operator OP.

[0032] Here, an example will be described in which two operators OP operate one robot avatar 2. For ease of explanation, in order to distinguish between the two operators OP, one will be referred to as a "first operator OP1" and the other as a "second operator OP2."

[0033] FIG. 2 is an explanatory diagram of the robot avatar 2. The robot avatar 2 is a robot used as an alter ego (avatar) to reflect the movement of the operator OP for the purpose of performing a predetermined task. The robot avatar 2 moves according to the movement (instruction movement) performed by each operator OP for operation. In the case of this embodiment, the robot avatar 2 performs a movement similar to the movement (instruction movement) performed by each operator OP (that is, a movement in accordance with the movement of each operator OP). Moreover, as described above, when each operator OP operates the robot avatar 2, the operator OP can intuitively grasp the motion state corresponding to the instruction movement of the other operator OP other than the operator OP (one example of the motion state related to the instruction movement). Therefore, each operator OP can operate one robot avatar 2 while feeling as if each alter ego (avatar) was fused into one. The control system 1 will be described in detail below.

[0034] The control system 1 includes a robot avatar 2, an input device 3, and an information presentation device 4.

[0035] The robot avatar 2 mainly comprises an action unit (end effector) 21 operable to act on an object, and a main body unit 22 capable of moving (moving) while holding the action unit 21. The robot avatar 2 of this embodiment comprises a robot arm 2 as shown in FIG. 2. The robot arm 2 is a multi-jointed seven-degree-of-freedom robot arm ("xArm7", manufactured by UFACTORY), and comprises an arm unit 22 as the main body unit 22, and an action unit 21 that is held by the arm unit 22 and includes a gripper unit (gripper) 21a capable of gripping an object. In this embodiment, an "xArm Gripper" (manufactured by UFACTORY) was used as the gripper unit 21a.

[0036] The arm section 22 is used with its base section 22a fixed on a predetermined stage (not shown). The arm section 22 has a plurality of link sections 22b and a plurality of joint sections 22c connecting the link sections 22b to each other. The arm section 22 also includes a plurality of motors (drive sections) for rotating the link sections 22b fixed to the joint sections 22c in a predetermined direction. The arm section 22 can move three-dimensionally while holding the action section 21 by controlling the rotational drive of each of these motors. The arm section 22 also includes a plurality of encoders for detecting the position (angle) of each drive shaft (rotation shaft) of each motor.

[0037] The object (control object) whose operation is controlled by the operation of the first operator OP1 is the arm section 22, and the object (control object) whose operation is controlled by the operation of the second operator OP2 is also the arm section 22. However, the first operator OP1 and the second operator OP2 have different purposes (control purposes) for controlling the operation of the arm section 22 (control object). The control purpose of the operation of the arm section 22 assigned to the first operator OP1 is to control the position of the action unit 21 to a desired position (i.e., position control of the action unit 21). In contrast, the control purpose of the operation of the arm section 22 assigned to the second operator OP2 is to control the posture of the action unit 21 to a desired posture (i.e., posture control of the action unit 21). That is, in the control system 1 of this embodiment, the role of operating the arm section 22 is shared between the first operator OP1 and the second operator OP2. Such a role sharing is effective, for example, when the working posture of each operator OP during task execution is important. Role allocation is used to isolate behaviors (killer factors) that are critical to task success.

[0038] The position (three-dimensional position) of the action unit 21 (predetermined portion R) held by the arm unit 22 is expressed using a three-dimensional coordinate system (three-dimensional coordinate system for avatar) set for the robot arm 2. A three-dimensional coordinate system with an origin located at a predetermined location is set for the robot arm 2, and the three-dimensional position of the action unit 21 is expressed using the values ​​(x, y, z) of the x-axis, y-axis, and z-axis in the three-dimensional coordinate system.

[0039] In addition, the posture (three-dimensional posture) of the action unit 21 (specific part R) held by the arm unit 22 is expressed using a rotation angle (roll angle) around the x-axis, a rotation angle (yaw angle) around the z-axis, and a rotation angle (pitch angle) around the y-axis of the above-mentioned three-dimensional coordinate system.

[0040] The operation of the arm unit 22 is controlled so that the position and posture of the action unit 21 at the predetermined site R are the predetermined position and posture instructed by each operator OP. Specifically, the position (three-dimensional position) of the action unit 21 at the predetermined site R is controlled based on three-dimensional position information corresponding to the instruction operation of the first operator OP1. Also, the posture (three-dimensional posture) of the action unit 21 at the predetermined site R is controlled based on three-dimensional posture information corresponding to the instruction operation of the second operator OP2.

[0041] The action unit 21 includes a gripper 21a for gripping an object. The gripper 21a includes two fingers 21b that are movable when gripping an object. The two fingers 21b are arranged facing each other, separated from each other. The gripper 21a can pinch an object between the facing fingers 21b. The action unit 21 includes a drive unit (motor, etc.) for driving each finger 21b of the gripper 21a. When the gripper 21a of the action unit 21 grips an object, the fingers 21b move toward each other (closing operation). On the other hand, when the gripper 21a releases the object that it has been gripping, the fingers 21b move away from each other (opening operation).

[0042] The operation of the gripping portion 21a (finger portion 21b) of the action portion 21 is performed by controlling the drive portion (motor, etc.) provided in the action portion 21. In the case of this embodiment, the gripping portion 21a of the action portion 21 is operated only by the second operator OP2, as described later.

[0043] In addition, a detection sensor 23 is attached to the inside of the gripping portion 21a to detect a physical action (e.g., force, vibration, heat, etc.) that the action portion 21 receives from the object side when the action portion 21 acts on the object. Here, as an example of the detection sensor 23, a force sensor (thin film pressure sensor, "RP-C10-ST", manufactured by Xuuyuu Co., Ltd.) is used to detect a force that is received from the object side when the gripping portion 21a (fingers 21b) of the action portion 21 grips the object.

[0044] The input device 3 is a device used when the operator OP operates the robot avatar 2. The input device 3 generates input information based on an instruction action (three-dimensional action) performed by the operator OP. The "instruction action" performed by the operator OP is the movement of a part of the body (e.g., an arm, a leg, a head, a finger, etc.) by the operator OP in order to make the robot avatar 2 execute a task. In particular, in this specification, an instruction action by an action similar to the action to be made by the robot avatar 2 is called a "three-dimensional action". For example, an instruction action when moving the arm part 22 in accordance with the movement of the operator OP's hand (arm) and an instruction action when opening and closing the grip part 21a in accordance with the bending and straightening of the operator OP's fingers correspond to the "three-dimensional action".

[0045] The control system 1 of this embodiment uses two types of input devices 3. Specifically, as the input devices 3, a first input device 31 for operating the movement of the arm portion 22 of the robot arm (robot avatar) 2 and a second input device 32 for operating the movement of the grip portion 21a (finger portion 21b) of the action portion 21 are used.

[0046] FIG. 3 is an explanatory diagram showing an input device 3 and an information presentation device 4 worn by a first operator OP1, FIG. 4 is an explanatory diagram showing an input device 3 and an information presentation device 4 worn by a second operator OP2, and FIG. 5 is an explanatory diagram showing another input device 3 worn by the second operator OP2.

[0047] The first input device 31 is a device that uses motion capture. In the first input device 31, an image (photographed image) corresponding to an instruction action (three-dimensional action) from the first operator OP1 and the second operator OP2 is generated as input information (input information generating step). Here, the first input device 31 that uses optical motion capture will be described as an example.

[0048] As is well known, motion capture is a technique in which a plurality of markers 311 attached to predetermined parts of an operator OP are photographed by a plurality of cameras (imaging devices) 312 at different angles, and the movements of the markers 311 are measured based on the photographed images obtained by the cameras 312 by the principle of triangulation. In the case of this embodiment, "OptiTrack Prime 13W" (NaturalPoint, Inc.) was used as the motion capture. Eight cameras 312 were used, and the resolution of the cameras 312 was 1280×1024 pixels, the frame rate was 240frs, and the lens was 3.5 mm F2.4. The field of view of the cameras 312 was 82° (horizontal direction) and 70° (vertical direction).

[0049] As shown in FIG. 3, a tool 313 for attaching a plurality of (four) markers 311 used as part of the first input device 31 is attached to the back OP1a of the right hand of the first operator OP1. The tool 313 includes a flat plate-shaped mounting portion 313a to be placed on the back of the hand, and a plurality of mounting rods 313b each extending in a different direction from the periphery of the mounting portion 313a outward. A marker 311 is attached to the tip of each of the four mounting rods 313b. The marker 311 is made of a sphere coated with paint for reflecting infrared light irradiated from the camera 312. The tool 313 is fixed in place by using a band 314 that is wrapped around the mounting portion 313a so that it is sandwiched between the back of the right hand and the back of the right hand.

[0050] 4, like the first operator OP1, an implement 313 for attaching a plurality of (four) markers 311 used as part of the first input device 31 is attached to the back OP2a of the right hand of the second operator OP2. A marker 311 is attached to each tip of four attachment rods 313b extending outward from the periphery of a placement portion 313a of the implement 313. The three-dimensional arrangement of the four markers 311 used by the second operator OP2 is set to be different from the three-dimensional arrangement of the four markers 311 used by the first operator OP1 so that the movement of the first operator OP1 can be distinguished from the movement of the second operator OP2.

[0051] A three-dimensional coordinate system (three-dimensional coordinate system for the operators) with the origin located at a predetermined position is set for the first operator OP1 and the second operator OP2. The three-dimensional coordinate system for the operators corresponds to the three-dimensional coordinate system for the robot avatar 2 described above.

[0052] The position (three-dimensional position) of the back of the right hand OP1a of the first operator OP1 (hereinafter sometimes referred to as the "first rigid body") is expressed using the values ​​(x', y', z') of the x-axis, y-axis, and z-axis in the three-dimensional coordinate system for the operator. In addition, the posture (three-dimensional posture) of the back of the right hand OP2a of the second operator OP2 (hereinafter sometimes referred to as the "second rigid body") is expressed using the rotation angle (roll angle) around the x-axis, the rotation angle (yaw angle) around the z-axis, and the rotation angle (pitch angle) around the y-axis in the three-dimensional coordinate system for the operator.

[0053] The movements of multiple markers 311 attached to the right hand of the first operator OP1 are captured by a camera 312, and the captured image (input information) is processed by an image analysis unit 511 provided in an operating computer 5 described later, whereby three-dimensional position information (part of the input information) of the first rigid body is obtained from the captured image.

[0054] In addition, the movements of multiple markers 311 attached to the right hand of the second operator OP2 are captured by a camera 312, and the captured image (input information) is processed by an image analysis unit 511 provided in the operating computer 5 described later, whereby three-dimensional posture information (part of the input information) of the second rigid body is obtained from the captured image.

[0055] Each piece of input information corresponding to each instruction operation of the first operator OP1 and the second operator OP2 is obtained as a time-series photographed image by the first input device 31. In the control system of this embodiment, the motion capture is mainly composed of a marker 311, a camera 312, and an image analysis unit 511.

[0056] As described above, the second input device 32 is a device for controlling the movement of the grip portion 21a (finger portion 21b) of the action portion 21. The second input device 32 in this embodiment is made up of a bending sensor 32, and is used by being attached to the index finger OP2b of the left hand of the second operator OP2. The movement of the grip portion 21a (finger portion 21b) is operated only by the second operator OP2.

[0057] The main body 320 of the bending sensor 32 ("FS-L-0055-253-ST", manufactured by Spectra Symbol) has an elongated shape that fits the index finger OP2b as a whole, and the output resistance value changes depending on the bending angle. The main body 320 of the bending sensor 32 is attached to the index finger OP2b by using two ring-shaped attachment members 321, 322.

[0058] The bending sensor 32 has a pair of electrode patterns, and a change in the output resistance value of the bending sensor 32 (input information) is output as a signal to the outside through a pair of electrode terminals 32a, 32b connected to the electrode patterns. Signal lines 323 are connected to the electrode terminals 32a, 32b, respectively.

[0059] When the index finger OP2b of the left hand of the second operator OP2 is gradually bent from a straightened state, the opposing finger portions 21b of the grip portion 21a of the operating unit 21 move toward each other. In contrast, when the index finger OP2b of the left hand is gradually extended from a bent state to a straight state, the opposing finger portions 21b of the grip portion 21a of the operating unit 21 move away from each other.

[0060] By means of such a second input device 32, an output signal (change in the output resistance value of the bending sensor 32) corresponding to the movement of the gripping portion 21a of the action portion 21 is generated as input information for operating the action portion 21.

[0061] The information presentation device 4 is a device that is worn by an operator OP and presents a tactile stimulus to the operator OP when the operator OP operates the robot avatar 2. The control system 1 of this embodiment uses two types of information presentation devices 4. Specifically, the information presentation device 4 includes a motion state presentation device 41 and an action part information presentation device 42.

[0062] The motion state presentation device 41 is a device that is worn by each of the multiple operators OP and presents a tactile stimulus to each operator OP so that the multiple operators OP can grasp a motion state (an example of a motion state related to an instruction action) corresponding to an instruction action of the other operators OP other than themselves, so that the multiple operators OP can cooperate with each other to make the robot avatar 2 execute a task when using the first input device 31. In the case of this embodiment, one motion state presentation device 41 is worn by each of the first operator OP1 and the second operator OP2. The motion state presentation device 41 is generally shaped like a wristwatch.

[0063] The motion state presentation device 41 worn by the first operator OP1 presents a tactile stimulus to the first operator OP1 so that the first operator OP1 can grasp a motion state corresponding to an instruction action of the second operator OP2 when the first operator OP1 uses the first input device 31. In contrast, the motion state presentation device 41 worn by the second operator OP2 presents a tactile stimulus to the second operator OP2 so that the second operator OP2 can grasp a motion state corresponding to an instruction action of the first operator OP1 when the second operator OP2 uses the first input device 31.

[0064] In this embodiment, a vibrator 411 that presents vibration (an example of a tactile stimulus) to an operator OP is used as the exercise state presentation device 41. The vibrator 411 is attached to each of two operators OP (i.e., a first operator OP1 and a second operator OP2).

[0065] 3, a vibrator 411 as the exercise state presentation device 41 is attached using a band 412 so as to be in close contact with the outer skin OP1c near the right wrist of the first operator OP1. The band 412 is wrapped around the right wrist of the first operator OP1 and attached so that the vibrator 411 is sandwiched between the band 412 and the skin OP1c.

[0066] The transducer 411 attached to the first operator OP1 presents a vibration stimulus (tactile stimulus) based on the motion information of the second operator OP2 corresponding to the instruction motion (three-dimensional motion) of the second operator OP2. The transducer 411 vibrates upon receiving a tactile vibration signal corresponding to the motion information (tactile stimulus presenting step).

[0067] 4, a vibrator 411 as the exercise state presentation device 41 is attached using a band 412 so as to be in close contact with the outer skin OP2c near the right wrist of the second operator OP2. The band 412 is wrapped around the right wrist of the second operator OP2 and attached so that the vibrator 411 is sandwiched between the band 412 and the skin OP2c.

[0068] The transducer 411 attached to the second operator OP2 presents a vibration stimulus (tactile stimulus) based on the motion information of the first operator OP1 corresponding to the instruction motion (three-dimensional motion) of the first operator OP1. The transducer 411 vibrates upon receiving a tactile vibration signal corresponding to the motion information (tactile stimulus presenting step).

[0069] The action unit information presentation device 42 is worn by multiple operators OP, and presents tactile stimulation corresponding to the detection results of the detection sensor 23 to each operator OP so that the operators OP can share the physical action that the action unit 21 receives from the object side when the action unit 21 is operating.

[0070] In this embodiment, one action unit information presentation device 42 is attached to each of the first operator OP1 and the second operator OP2. By receiving the tactile stimuli presented by the action unit information presentation device 42, the first operator OP1 and the second operator OP2 can simultaneously grasp the physical action (e.g., the force received when the gripper 21a grips an object) detected by the action unit 21 through the detection sensor 23 when the action unit 21 of the robot avatar (robot arm) 2 acts on an object (e.g., when the gripper 21a grips an object). The same tactile stimuli are presented to the first operator OP1 and the second operator OP2 by each action unit information presentation device 42.

[0071] The action part information presenting device 42 of this embodiment is configured to present a pressure stimulus (an example of a tactile stimulus) to the right forearm of the operator OP.

[0072] For example, as shown in FIG. 3, the action part information presentation device 42 is attached to the right forearm (part of the upper arm side from the wrist) OP1d of the first operator OP1. The action part information presentation device 42 mainly includes an annular tightening part 421 made of a rubber band and an adjustment part 422 that adjusts the diameter of the tightening part 421. The right forearm OP1d of the first operator OP1 is passed through the annular tightening part 421. The adjustment part 422 includes a servo motor (DC servo motor) that is driven by receiving a drive signal corresponding to the detection result of the detection sensor 23. A disk-shaped winding part 424 is fixed to the output shaft 423 of the servo motor, and a part of the annular tightening part 42 is fixed to the periphery of the winding part 424. When the servo motor is driven and the output shaft 423 rotates in a predetermined direction, the winding part 424 rotates so that a part of the tightening part 42 is wound up and reduced in diameter. Furthermore, when the output shaft 423 rotates in the reverse direction, the winding portion 424 rotates so that the tightening portion 42 that has been wound up is returned and expanded to its original diameter.

[0073] The adjustment unit 422 is fixed onto a support plate 425 having a rectangular shape in a plan view. In addition, the support plate 425 is provided with a wearing band 426 that is attached by wrapping it around the right forearm OP1d of the first operator OP1.

[0074] The servo motor of the adjustment unit 422 drives the tightening unit 42 to reduce or increase the diameter when a drive signal corresponding to the detection result of the detection sensor 23 is supplied. The adjustment unit 422 is connected to a first microcomputer 7, which will be described later, via a cable 427.

[0075] 4, an action part information presentation device 42 similar to that for the first operator OP1 described above is also attached to the right forearm (part of the upper arm side from the wrist) OP2d of the second operator OP2. Note that the adjustment unit 422 provided in the action part information presentation device 42 for the second operator OP2 is connected to the second microcomputer 8 described later via a cable 427.

[0076] The control system 1 further includes an operating computer 5, a controller 6, a first microcomputer 7, a second microcomputer 8, and the like.

[0077] An operating computer (OP computer) 5 centrally controls the entire system. Fig. 6 is an explanatory diagram showing the hardware configuration of the operating computer 5. As shown in Fig. 6, the OP computer 5 is made up of a CPU (Central Processing Unit) 51, a RAM (Random Access Memory) 52, a ROM (Read Only Memory) 53, a storage unit 54, a display unit 55, a communication unit 56, an input unit 57, a timer unit 58, etc.

[0078] The CPU 51 of the OP computer 5 reads out various programs stored in the storage unit 54, expands them in a work area of ​​the RAM 52, and executes various processes described below in accordance with the expanded programs. The storage unit 54 stores programs executed by the CPU 51 as appropriate, data required for various processes, and the like. The storage unit 54 is composed of a physical drive such as a memory or a hard disk drive. The clock unit 58 is composed of, for example, a timer IC, a quartz oscillator, or a clock module, and has a function of clocking the current time. The CPU 51 acquires the current time from the clock unit 58 as appropriate as necessary.

[0079] The display unit 55 is, for example, a liquid crystal display, and displays necessary messages and the like to the administrator who operates the OP computer 5. The input unit 57 is a user interface, and is made up of a keyboard, a pointing device, and the like, and is used when the administrator inputs information such as various data and commands into the OP computer 5.

[0080] The communication unit 56 is a communication interface, and has a function of transmitting information to other devices and a function of receiving information from other devices. The communication unit 56 in this embodiment has both a wireless communication function and a wired communication function. The communication unit 56 communicates with the first input device 31 (camera 312), the transmitter 416 for the first operator OP1, the transmitter 416 for the second operator OP2, the controller 6, etc. via wired communication. The communication unit 56 also communicates with the first microcomputer 7, the second microcomputer 8, etc. via wireless communication.

[0081] The OP computer 5 also includes a control unit 510 configured with a CPU 51 etc. The control unit 510 further includes an image analysis unit 511, an arm unit command generation unit 512, a motion information generation unit 513, a motion information supply unit 514, an action unit command generation unit 515, and a shared information generation unit 516.

[0082] The image analysis unit 511 executes a process of extracting input information corresponding to the instruction operation of each operator OP from the captured image acquired by the camera 312 of the first input device 31 for each operator OP. When the captured image captured by the camera 312 of the first input device 31 is transmitted to the OP computer 5, the image analysis unit 511 generates three-dimensional position information of the first rigid body in time series based on the captured image. In addition, the image analysis unit 511 generates three-dimensional posture information of the second rigid body in time series based on the same captured image. The sampling frequency of the captured image (image data) is, for example, 100 to 150 [Hz].

[0083] The arm section command generating unit (first generating unit) 512 executes a process of generating a plurality of operation commands for operating the arm section 22 (control target) based on a plurality of input information (three-dimensional position information of the first rigid body and three-dimensional posture information of the second rigid body) so that the robot arm 2 operates in accordance with the instruction actions (three-dimensional actions) respectively performed by a plurality of operators OP (i.e., a first operator OP1 and a second operator OP2).

[0084] The arm section command generating unit 512 generates an operation command for controlling the operation of the arm section 22 based on the input information (three-dimensional position information of the first rigid body) corresponding to the second operator OP2 so that the three-dimensional position of the action section 21 moves in accordance with the movement of the right hand (three-dimensional movement) regarding the three-dimensional position of the first operator OP1 (operation command generating process).

[0085] In addition, the arm section command generating unit 512 generates an operation command for controlling the operation of the arm section 22 based on the input information (three-dimensional posture information of the second rigid body) corresponding to the second operator OP2 so that the three-dimensional posture of the action section 21 moves in accordance with the movement of the right hand (three-dimensional movement) related to the three-dimensional posture of the second operator OP2 (operation command generating process).

[0086] In this way, the arm command generating unit 512 generates a plurality of operation commands for the first operator OP1 and the second operator OP2 to control the operation of the arm unit 22 for mutually different control purposes.

[0087] The multiple operation commands generated by the arm command generating unit 512 are transmitted to the controller 6.

[0088] The movement information generation unit (second generation unit) 513 executes a process of generating multiple pieces of movement information (tactile vibration signals) corresponding to the three-dimensional movements of each operator OP based on the multiple pieces of input information (three-dimensional position information of the first rigid body and three-dimensional posture information of the second rigid body) generated by the image analysis unit 511 (movement information generation process).

[0089] The motion information generating unit 513 generates the motion information by using physical quantities related to the three-dimensional motion of the corresponding operator OP. The physical quantities include, for example, speed, acceleration, jerk, amount of change in position, amount of change in posture, difference value between the position of the first rigid body and the position of the second rigid body, difference value between the posture of the first rigid body and the posture of the second rigid body, etc.

[0090] The motion information generating unit 513 of this embodiment generates motion information indicating the motion state of the first operator OP1 during a three-dimensional motion from the three-dimensional position information of the first rigid body. For example, the motion information generating unit 513 obtains the norm (scalar quantity) of the time change of the position as the velocity information v [mm / s] of the first rigid body based on the three-dimensional position information of the first rigid body, and sets it as the motion information (tactile vibration signal) of the first operator OP1.

[0091] Further, the motion information generating unit 513 generates motion information indicating the motion state of the second operator OP2 during a three-dimensional motion from the three-dimensional posture information of the second rigid body. For example, the motion information generating unit 513 obtains the norm (scalar quantity) of the time change of the posture as the rotational angular velocity information ω [rad / s] of the second rigid body based on the three-dimensional posture information of the second rigid body, and regards it as the motion information (tactile vibration signal) of the second operator.

[0092] The motion information supply unit 514 executes a process of allocating and supplying motion information to each of the motion state presentation devices 41 so that each of the motion information presentation devices 41 can present a tactile stimulus. The motion information supply unit 514 selects motion information to be supplied to each of the motion state presentation devices 41 from the multiple motion information generated by the motion information generation unit 513 so that each operator OP can grasp the three-dimensional motion of other operators OP other than itself.

[0093] In this embodiment, the exercise status presentation device 41 worn by the first operator OP1 is assigned the exercise information of the second operator OP2, and the exercise status presentation device 41 worn by the second operator OP2 is assigned the exercise information of the first operator OP1.

[0094] Furthermore, the motion information supply unit 514 executes a process of supplying the motion information of the second operator OP2 to the transmitter 416 for the first operator OP1, and also supplies the motion information of the first operator OP1 to the transmitter 416 for the second operator OP2.

[0095] When the motion information (tactile vibration signal) of the second operator OP2 is supplied to the transmitter 416 for the first operator OP1, the motion information is modulated in the transmitter 416 and sent to the receiver 415 for the first operator OP1 by wireless communication. The motion information received by the receiver 415 is demodulated and then sent to the vibration amplifier 414, where the signal is amplified, and then supplied to the transducer 411 of the motion state presentation device 41 for the first operator OP1 via the cable (signal line) 413.

[0096] The vibrations were generated by amplitude modulating (AM modulating) the carrier wave (sine wave) with a frequency of 200 [Hz] according to the change in each value. In other embodiments, the vibrations may be generated by modulating with other modulation methods (FM modulation, etc.).

[0097] Furthermore, when the motion information (tactile vibration signal) of the first operator OP1 is supplied to the transmitter 416 for the second operator OP2, the motion information is modulated in the transmitter 416 and sent to the receiver 415 for the second operator OP2 by wireless communication. The motion information received by the receiver 415 is demodulated and then sent to the vibration amplifier 414, where the signal is amplified, and then supplied to the transducer 411 of the motion state presentation device 41 for the second operator OP2 via the cable (signal line) 413.

[0098] The action unit command generating unit 515 executes a process of generating an operation command for operating the action unit 21 (the gripping unit 21a) based on the input information input by the second operator OP2 using the second input device (bending sensor) 32. The operation command generated by the action unit command generating unit 515 is transmitted to the controller 6. The input information of the second input device 32 is supplied from the second microcomputer 8.

[0099] The shared information generating unit 516 executes a process of generating a drive signal for driving the action part information presenting device 42 based on the detection result (action part information) of the detection sensor 23 supplied from the controller 6 (action part information acquiring unit 63) side. The drive signal generated by the shared information generating unit 516 is transmitted to the first microcomputer 7 and the second microcomputer 8 by wireless communication.

[0100] The first microcomputer 7 is a microcontroller (e.g., "ESP32" manufactured by Espressif Systems) equipped with wireless communication and wired communication functions, and is composed of a CPU, a memory, a communication unit, etc. The first microcomputer 7 functions as a drive control unit 71 that controls the drive of the adjustment unit (servo motor) 422 of the action part information presentation device 42 for the first operator OP1. When a drive signal generated by the shared information generation unit 516 of the OP computer 5 is supplied to the first microcomputer 7, the drive control unit 71 drives the action part information presentation device 42 for the first operator OP1 based on the drive signal.

[0101] The second microcomputer 8, like the first microcomputer 7, is a microcontroller (e.g., "ESP32" manufactured by Espressif Systems) equipped with wireless communication and wired communication functions, and is composed of a CPU, a memory, a communication unit, etc. The second microcomputer 8 functions as a drive control unit 82 that controls the drive of an adjustment unit (servo motor) 422 of an action part information presentation device 42 for the second operator OP2. When a drive signal generated by the shared information generation unit 516 of the OP computer 5 is supplied to the second microcomputer 8, the drive control unit 81 drives the action part information presentation device 42 for the second operator OP2 based on the drive signal.

[0102] The second microcomputer 8 also functions as a bending information acquisition unit 82 that acquires input information (changes in the output resistance value of the bending sensor 32) generated by the second input device (bending sensor) 32. The input information of the second input device acquired by the bending information acquisition unit 82 is transmitted to the OP computer 5 by wireless communication.

[0103] The controller 6 mainly controls the movement of the robot avatar (robot arm) 2. The controller 6 is composed of a CPU, a memory, a communication unit, etc. The controller 6 includes an arm unit movement control unit 61, an action unit movement control unit 62, and an action unit information acquisition unit 63, all of which are composed of the CPU, etc.

[0104] The arm section operation control section 61 executes a process of controlling the operation of the arm section 22, which is the object to be controlled, based on a plurality of operation commands generated by the arm section command generation section 512 (operation control process). In the arm section command generation section 512, a plurality of operation commands are generated for the first operator OP1 and the second operator OP2 to control the operation of the arm section 22 for mutually different control purposes, and based on these operation commands, the arm section operation control section 61 controls the operation of the arm section 22. Note that the plurality of operation commands (data) are time-series aligned with each other.

[0105] Specifically, the arm section operation control unit 61 controls the operation of the arm section 22 based on an operation command generated in response to the control objective of the first operator OP1 (position control of the action unit 21), and at the same time controls the operation of the arm section 22 based on an operation command generated in response to the control objective of the second operator OP2 (posture control of the action unit 21).

[0106] As a result, the position control of the action unit 21 is performed only by the instruction action of the first operator OP1, and the attitude control of the action unit 21 is performed only by the instruction action of the second operator OP2.

[0107] The action unit operation control unit 62 executes a process of controlling the operation of the action unit 21 (the gripping unit 21a) based on the operation command generated by the action unit command generation unit 515. As a result, the open / closed state of the gripping unit 21a in the action unit 21 is controlled according to the degree of bending of the index finger OP2b of the left hand of the second operator OP2 (pointing operation, three-dimensional operation).

[0108] The action section information acquisition unit 63 executes a process of acquiring the detection result of the detection sensor 23. The detection result of the detection sensor 23 acquired by the action section information acquisition unit 63 is transmitted to the OP computer 5 (supply information generation unit).

[0109] Here, in the control system 1 of this embodiment, when controlling the movement of the robot avatar 2, the input information input by the instruction action of each operator OP and the feedback information returned to each operator OP will be described with reference to Fig. 7 etc. Fig. 7 is an explanatory diagram showing the relationship between the input information input by each operator OP and the feedback information returned to each operator OP in the control system 1 of embodiment 1.

[0110] A control object whose operation is controlled by the operations of the first operator OP1 and the second operator O2 is the arm unit 22. The control objective for the first operator OP1 to control the operation of the arm unit 22 is position control of the action unit 21, and the control objective for the second operator OP2 to control the operation of the arm unit 22 is posture control of the action unit 21. Another control object whose operation is controlled by the operation of the second operator OP2 is the gripping unit 21a of the action unit 21.

[0111] Under these conditions, when the first operator OP1 and the second operator OP2 control the movement of the robot avatar (robot arm) 2, the information (input information) input to the robot avatar 2 (control unit 510 of the OP computer 5) is as follows:

[0112] A first operator OP1 inputs position information of a first rigid body for operating the arm unit 22 via the first input device 31 for the purpose of controlling the position of the action unit 21. A second operator OP2 inputs posture information of a second rigid body for operating the arm unit 22 via the first input device 31 for the purpose of controlling the posture of the action unit 21.

[0113] Furthermore, the second operator OP2 inputs information (opening / closing information) for operating the gripping portion 21a of the action portion 21 via the second input device 32 for the purpose of controlling the opening and closing of the gripping portion 21a of the action portion 21.

[0114] In response to this, when controlling the movement of the robot avatar 2, the information (feedback information) returned from the robot avatar 2 side (the control unit 510 of the OP computer 5) to the first operator OP1 and the second operator OP2 is as follows:

[0115] The motion information of the second operator OP2 is returned to the first operator OP1 as a vibration stimulus presented by the motion state presentation device 41 worn by the first operator OP1. The motion information of the first operator OP1 is returned to the second operator OP2 as a vibration stimulus presented by the motion state presentation device 41 worn by the second operator OP2. Therefore, the first operator OP1 and the second operator OP2 can operate the motion of the robot avatar 2 while intuitively and quickly grasping the motion state corresponding to the instruction motion of the other operator. Therefore, each operator OP can grasp, for example, each other's subtle instruction motion, and it is possible to cause the robot avatar 2 to execute a task with high accuracy compared to a case without feedback information.

[0116] Furthermore, information when the gripping unit 21a grips an external object (grip state information of the gripping unit) is returned to the first operator OP1 and the second operator OP2 as a pressure stimulus presented by each action unit information presentation device 42 attached to each operator OP. Therefore, while operating the robot avatar 2, the first operator OP1 and the second operator OP2 can intuitively and quickly feel the force that the gripping unit 21a receives from the object side (the physical action that the action unit 21 receives) as a pressure stimulus presented by the action unit information presentation device 42.

[0117] 〔test〕 Using the control system 1 of this embodiment, the robot avatar 2 was experimentally made to execute the following tasks 1 and 2. For comparison, a case in which all operations (position and posture control of the action part, and opening and closing control of the grip part) were performed by only one operator OP was also verified.

[0118] (Task 1) With a wall placed vertically diagonally in front of and to the left of the robot arm 2, the robot arm 2 was made to execute a task (hole passing) of passing a block held by the gripping portion 21a of the action portion 21 from right to left through a hole penetrating the left-right direction in the center of the wall.

[0119] (Task 2) The robot was made to carry a ring-attached rod with the gripping part 21a of the action part 21 from the start to the goal of a course formed by connecting pieces of timber together in a frame shape, with the timber passing through the inside of the ring.

[0120] If one operator OP were to perform all the operations in Task 1 and Task 2, the operator OP would be required to assume an unreasonable posture in terms of the range of motion of the joints. For example, if one operator OP were to perform Task 1, the operator would need to move his / her arm horizontally without changing his / her posture after adjusting the direction of the block, making it extremely difficult to perform the task. Similarly, if one operator OP were to perform Task 2, the arm he / she is controlling would reach the limit of its range of motion when turning the second corner from the starting position, and he / she would be unable to perform the task any further. Although there are individual differences in the flexibility of joints, the task would still require the operator OP to maintain an unreasonable posture to perform the task.

[0121] In contrast, when the operations are shared by two operators OP as described above to perform tasks 1 and 2, in either case, the operations can be easily performed without forcing each operator OP into an unnatural posture. The operators OP can concentrate only on the operations that are shared with them. For example, the first operator OP1 can concentrate on accurately moving only the position of the first rigid body without worrying about the posture. Also, the second operator OP2 can concentrate on accurately moving only the posture of the second rigid body without worrying about the position. Also, by sharing the operations in this way, the state (position, posture) of the rigid bodies (first rigid body, second rigid body) of each operator OP can be controlled not only by the movement of the wrist, but also by the movement of the entire arm centered on the shoulder, or by the movement of the whole body, such as changing the direction of the entire body.

[0122] In particular, in the control system 1 of this embodiment, the operators OP can feel the movement information corresponding to the instruction movements of the other operator OP as tactile stimulation through the movement state presentation device 41, so that the operators OP can easily coordinate their operations when performing a task, and furthermore, feel as if their arms are integrated with the robot avatar (robot arm) 2.

[0123] In addition, each operator OP feels as if he or she is actually holding a block or the like by receiving a pressure stimulus from the action part information presentation device 42. Therefore, the reliability of the operation is improved.

[0124] In the control system 1 of this embodiment, the tasks that the robot avatar 2 is caused to execute are not limited to the above tasks 1 and 2.

[0125] <Embodiment 2> Next, a control system 1A according to a second embodiment will be described with reference to FIG. 8. FIG. 8 is an explanatory diagram showing the overall configuration of the control system 1A according to the second embodiment. The control system 1A of this embodiment is a system that aims to allow multiple (two) operators OP to simultaneously operate one robot avatar while sharing the operation with each other and grasping each other's operation status (movement corresponding to the instruction operation), as in the first embodiment. In the control system 1A of this embodiment, as in the first embodiment, the first operator OP1 is in charge of position control of the action unit 21, and the second operator OP2 is in charge of posture control of the action unit 21 and opening and closing control of the action unit 21 (the gripper 21a). Therefore, various processing contents executed in the control unit 510 of the control system 1A of this embodiment are basically the same as those in the first embodiment.

[0126] In the control system 1A of this embodiment, when controlling the movement of the robot avatar 2, the input information input by each operator OP and the feedback information returned to each operator OP are the same as those in the first embodiment (see FIG. 7).

[0127] In this embodiment, a space S1 where a first operator OP1 is, a space S2 where a second operator OP2 is, and a space S3 where a robot avatar (robot arm) 2 is installed are far away from each other. That is, each operator OP is in a situation where he or she cannot directly check the other operators OP other than himself or the robot avatar 2 with his or her eyes or the like. The control system 1A of this embodiment is used when a plurality of (two) operators OP remotely operate one robot avatar 2 under such a situation.

[0128] In addition, among the configurations of the control system 1A of the second embodiment, the configurations similar to those of the control system 1 of the first embodiment are denoted in FIG. 8 by the same reference numerals as those of the first embodiment, and detailed description thereof will be omitted as appropriate.

[0129] In this embodiment, a first input device 31 using motion capture is assigned to each of the first operator OP1 and the second operator OP2. That is, two first input devices 31 are used in the control system 1A. A plurality of (eight) cameras 313 are used as each of the first input devices 31. As in the first embodiment, a plurality of markers 311 for motion capture are attached to the right hand of the first operator OP1 and the right hand of the second operator OP2 using a predetermined tool 313. Also, a second input device (bending sensor) 32 similar to the first embodiment is attached to the left hand of the second operator OP2.

[0130] The photographed image taken by the first input device 31 for the first operator OP1 is sent to the first computer CP1 for the first operator OP1 connected to the first input device 31. The first computer CP1 transmits the received photographed image to the OP computer 5 using the communication line 14. Examples of the communication line 14 include the Internet, an Ethernet (registered trademark) line, a public line, and a dedicated line. The photographed image taken by the first input device 31 for the second operator OP2 is transmitted to the second computer CP2 for the second operator OP2 connected to the first input device 31. The second computer CP2 transmits the received photographed image to the OP computer 5 using the communication line 14. Both the first computer CP1 and the second computer CP2 have the same hardware configuration as the OP computer 5, and execute various processes.

[0131] When captured images corresponding to each operator OP are supplied to the OP computer 5 from each first input device 31, the image analysis unit 511 generates three-dimensional position information of the first rigid body over time and three-dimensional posture information of the second rigid body over time from those captured images.

[0132] The input information (change in the output resistance value of the bending sensor 32) generated by the second input device (bending sensor) 32 is acquired by a bending information acquisition unit 82 of the second microcomputer 8, and then transmitted to the second computer CP2 by wireless communication. After that, the second computer CP2 transmits the input information of the second input device to the OP computer 5 via the communication line 14.

[0133] The motion information supply unit 514 supplies the motion information of the second operator OP2 to the first computer CP1 via the communication line 14. The first computer CP1 supplies the motion information of the second operator OP2 to the transmitter 416 for the first operator OP1. In addition, the motion information supply unit 514 supplies the motion information of the first operator OP1 to the second computer CP2 via the communication line 14. The second computer CP2 supplies the motion information of the first operator OP1 to the transmitter 416 for the second operator.

[0134] The drive signal generated by the shared information generation unit 516 is transmitted to the first computer CP1 and the second computer CP2 via the communication line 14. Thereafter, the first computer CP1 transmits the drive signal to the first microcomputer 7 by wireless communication, and the second computer CP2 transmits the drive signal to the second microcomputer 8 by wireless communication.

[0135] The control system 1A also includes display devices 11 and 12 that display images of the robot avatar 2 to each operator OP so that each operator OP can perform three-dimensional movements without seeing the actual robot avatar 2. The display devices 11 and 12 are, for example, head-mounted displays, liquid crystal displays, etc. The display control of the display device 11 for the first operator OP1 is performed by a first computer CP1, and the display control of the display device 12 for the second operator OP2 is performed by a second computer CP2. An image of the robot avatar 2 captured by a camera (imaging device) 13 is displayed on each of the display devices 11 and 12. The camera 13 is installed in the space S3 to capture the image of the robot avatar 2. The image captured by the camera 13 is sent to each of the display devices 11 and 12 via the OP computer 5 and a communication line 14.

[0136] By using the control system 1A of this embodiment, even if multiple (two) operators OP and robot avatars 2 are apart from each other, the multiple (two) operators OP can share the operation with each other and understand each other's operation status (movement state corresponding to the instruction action), and simultaneously (jointly) operate one robot avatar.

[0137] <Embodiment 3> Next, a control system 1B according to a third embodiment will be described with reference to Fig. 9. Fig. 9 is an explanatory diagram showing the relationship between input information input by each operator OP and feedback information returned to each operator OP in the control system 1B of the third embodiment. In this embodiment, one robot avatar 2 is operated by two operators OP (a first operator OP1 and a second operator OP2).

[0138] The control system 1B of this embodiment, like the first embodiment, causes a robot avatar (robot arm) 2 to execute a task, and includes various components required for this (OP computer, controller, etc.) In this embodiment, differences from the first embodiment will be described (the same applies to the other embodiments following this embodiment).

[0139] The control object whose operation is controlled by the operation of the first operator OP1 is the arm section 22, and the control object whose operation is controlled by the operation of the second operator OP2 is the gripping section 21a of the action section 21. In this embodiment, the control objects of the first operator OP1 and the second operator OP2 are different from each other. The arm section 22 is operated only by the first operator OP1, and the gripping section 21a of the action section 21 is operated only by the second operator OP2. The control object for the first operator OP1 to control the operation of the arm section 22 is the position control and attitude control (position and attitude control) of the action section 21, and the control object for the second operator OP2 to control the operation of the gripping section 21a is the opening and closing control of the gripping section 21a. Such a division of roles is effective, for example, when the operation of the action section 21 (the gripping section 21a) needs to be performed carefully (when the gripping section 21a grips a soft object, etc.).

[0140] Under such conditions, when the first operator OP1 and the second operator OP2 control the movement of the robot avatar (robot arm) 2, the information (input information) input to the robot avatar 2 side (control unit of the OP computer) is as follows:

[0141] A first operator OP1 inputs position information and orientation information of a first rigid body for operating the arm unit 22 via a first input device 31 (camera 312) using motion capture for the purpose of controlling the position and orientation of the action unit 21. Also, a second operator OP2 inputs information (opening / closing information) for operating the gripping unit 21a via a second input device (bending sensor) 32 for the purpose of controlling the opening and closing of the gripping unit 21a.

[0142] In response to this, when controlling the movement of the robot avatar 2, the information (feedback information) returned from the robot avatar 2 side (control section of the OP computer) to the first operator OP1 and the second operator OP2 is as follows:

[0143] The motion information of the second operator OP2 is returned to the first operator OP1 as a vibration stimulus presented by the motion state presentation device 41 worn by the first operator OP1. In this case, the motion information of the second operator OP2 is generated, for example, in the control unit of the OP computer based on a change in the output resistance value obtained from the second input device 32. In addition, the motion information of the first operator OP1 is returned to the second operator OP2 as a vibration stimulus presented by the motion state presentation device 41 worn by the second operator OP2. Therefore, the first operator OP1 and the second operator OP2 can intuitively and quickly grasp the motion state corresponding to the instruction motion of the other operator and control the motion of the robot avatar 2.

[0144] Furthermore, information when the gripping unit 21a grips an external object (grip state information of the gripping unit) is returned to the first operator OP1 and the second operator OP2 as a pressure stimulus presented by each action unit information presentation device 42 attached to each operator OP. Therefore, while operating the robot avatar 2, the first operator OP1 and the second operator OP2 can intuitively and quickly share the force that the gripping unit 21a receives from the object side (the physical action that the action unit 21 receives) as a pressure stimulus presented by the action unit information presentation device 42.

[0145] <Embodiment 4> Next, a control system 1C according to the fourth embodiment will be described with reference to Figs. 10 and 11. Fig. 10 is an explanatory diagram showing the relationship between input information input by each operator OP and feedback information returned to each operator OP in the control system 1C of the fourth embodiment, and Fig. 11 is an explanatory diagram showing a schematic representation of an action unit 21C provided in a robot avatar 2C of the fourth embodiment. In this embodiment, one robot avatar 2C is operated by two operators OP (a first operator OP1 and a second operator OP2). The robot avatar (robot arm) 2C of this embodiment differs from that of the first embodiment in the type of action unit 21C. The action unit 21C is made of a five-fingered robot hand capable of independently controlling the movement of each finger unit. The action unit 21C includes a first finger unit 211 corresponding to the thumb, a second finger unit 212 corresponding to the index finger, a third finger unit 213 corresponding to the middle finger, a fourth finger unit 214 corresponding to the ring finger, and a fifth finger unit 215 corresponding to the little finger. The arm portion 22 is the same as in the first embodiment.

[0146] The control objects whose movements are controlled by the operation of the first operator OP1 are the arm portion 22, the first finger portion 211, the second finger portion 212, and the third finger portion 213. The control objects whose movements are controlled by the operation of the second operator OP2 are the fourth finger portion 214 and the fifth finger portion 215.

[0147] The control purpose of the first operator OP1 controlling the operation of the arm unit 22 is to control the position and attitude (position and attitude control) of the action unit 21C. The control purpose of the first operator OP1 controlling the operation of each finger unit (the first finger unit 211, the second finger unit 212, and the third finger unit 213) is to switch between a contact state in which each finger unit contacts an external object and a non-contact state in which each finger unit does not contact an external object. The control purpose of the second operator OP2 controlling the operation of each finger unit (the fourth finger unit 214 and the fifth finger unit 215) is to perform contact / non-contact control to switch between a contact state in which each finger unit contacts an external object and a non-contact state in which each finger unit does not contact an external object. Such division of roles is effective, for example, when five fingers need to be controlled independently (such as when playing a musical instrument).

[0148] In this embodiment, the first operator OP1 wears one second input device (bending sensor) 32 on each of the first finger (thumb), second finger (index finger), and third finger (middle finger). The second input device 32 worn on the first finger of the first operator OP1 is used to operate the first finger portion 211, the second input device 32 worn on the second finger is used to operate the second finger portion 212, and the second input device 32 worn on the third finger is used to operate the third finger portion 213. The second operator OP2 wears one second input device (bending sensor) 32 on each of the fourth finger (ring finger) and fifth finger (little finger). The second input device 32 worn on the fourth finger of the second operator OP2 is used to operate the fourth finger portion 214, and the second input device 32 worn on the fifth finger is used to operate the fifth finger portion.

[0149] Furthermore, a vibration sensor (detection sensor) 23C for detecting vibration is attached to the inner side (the side that comes into contact with an object) of each finger of the action portion 21C.

[0150] Under such conditions, when the first operator OP1 and the second operator OP2 control the movement of the robot avatar (robot arm) 2C, the information (input information) input to the robot avatar 2C side (control unit of the OP computer) is as follows:

[0151] The first operator OP1 inputs position information and orientation information of the first rigid body for operating the arm unit 22 via the first input device 31 (camera 312) using motion capture for the purpose of controlling the position and orientation of the action unit 21C. Also, the first operator OP1 inputs information (contact and non-contact information) for operating each finger unit via each second input device (bending sensor) 32 attached to the first operator OP1 for the purpose of contact and non-contact control of each finger unit (first finger unit 211, second finger unit 212, and third finger unit 213). Also, the second operator OP2 inputs information (contact and non-contact information) for operating each finger unit via each second input device (bending sensor) 32 attached to the second operator OP2 for the purpose of contact and non-contact control of each finger unit (fourth finger unit 214 and fifth finger unit 215).

[0152] In response to this, when controlling the movement of the robot avatar 2C, the information (feedback information) returned from the robot avatar 2C side (control section of the OP computer) to the first operator OP1 and the second operator OP2 is as follows:

[0153] The first operator OP1 receives a vibration stimulus presented by a motion state presentation device (vibrator) 41C3 attached to the first operator OP1, which represents a motion state corresponding to each of the pointing actions of the fourth and fifth fingers of the second operator OP2. The motion state presentation device (vibrator) 41C3 presents a vibration stimulus to the first operator OP1 so that the motion information of the two fingers (the fourth and fifth fingers) can be distinguished.

[0154] Moreover, the second operator OP2 receives the motion information 1 (motion information of the back of the hand) corresponding to the instruction action of the first operator OP1 using the first input device 31 as a vibration stimulus presented by the motion state presentation device (vibrator) 41C1 worn by the second operator OP2. Furthermore, the second operator OP2 receives the motion information 2 corresponding to each instruction action of the first finger, second finger, and third finger of the first operator OP1 as a vibration stimulus presented by the motion state presentation device (vibrator) 41C2 worn by the second operator OP2. This motion state presentation device (vibrator) 41C2 presents a vibration stimulus to the first operator OP1 so that the motion states of the three fingers (first finger, second finger, and third finger) can be distinguished. Therefore, the first operator OP1 and the second operator OP2 can intuitively and quickly grasp the motion states corresponding to the instruction actions of the other party and control the motion of the robot avatar 2C.

[0155] In addition, the first operator OP1 and the second operator OP2 receive information (contact information of each finger) detected by a plurality of vibration sensors 23C attached to each finger of the action unit 21C as vibration stimuli from a vibrator serving as an action unit information presentation device 42C attached to each operator OP. Therefore, the first operator OP1 and the second operator OP2 can intuitively and quickly share the vibrations received by each finger as vibration stimuli presented by the action unit information presentation device 42C while operating the robot avatar 2C. The vibrator used in the action unit information presentation device 42C presents vibration stimuli to each operator OP so that the vibrations received by each finger can be distinguished.

[0156] <Embodiment 5> Next, a control system 1D according to the fifth embodiment will be described with reference to Fig. 12. Fig. 12 is an explanatory diagram showing the relationship between input information input by each operator OP and feedback information returned to each operator OP in the control system 1D of the fifth embodiment. In this embodiment, one robot avatar 2 is operated by two operators OP (a first operator OP1 and a second operator OP2).

[0157] The control object whose operation is controlled by the operation of the first operator OP1 is the arm section 22, and the control object whose operation is controlled by the operation of the second operator OP2 is also the arm section 22. The control objective of the first operator OP1 to control the operation of the arm section 22 is the position control of the action section 21 in the x-axis and y-axis in a three-dimensional coordinate system (xy position control). The control objective of the second operator OP2 to control the operation of the arm section 22 is the attitude control of the action section 21 and the position control of the z-axis in a three-dimensional coordinate system (z position control) in the position control of the action section 21. Such a division of roles is effective, for example, when the position control in the height direction (z position control) is important among the position control of the action section 21 (such as when a pen is held by the grip section 21a of the action section 21 and a character is written on the surface of a board placed on a desk). The operation of the grip section 21a of the action section 21 is performed by the second operator OP2, as in the first embodiment.

[0158] Under such conditions, when the first operator OP1 and the second operator OP2 control the movement of the robot avatar (robot arm) 2, the information (input information) input to the robot avatar 2 side (control unit of the OP computer) is as follows:

[0159] A first operator OP1 inputs xy position information of a first rigid body for operating the arm unit 22 via a first input device 31 (camera 312) using motion capture for the purpose of controlling the xy position of the action unit 21. Also, a second operator OP2 inputs z position information and posture information of a second rigid body for operating the arm unit 22 via the first input device 31 (camera 312) using motion capture for the purpose of controlling the posture and z position of the action unit 21.

[0160] In response to this, when controlling the movement of the robot avatar 2, the information (feedback information) returned from the robot avatar 2 side (control section of the OP computer) to the first operator OP1 and the second operator OP2 is as follows:

[0161] The motion information of the second operator OP2 is returned to the first operator OP1 as a vibration stimulus presented by the motion state presentation device 41 worn by the first operator OP1. The motion information of the first operator OP1 is returned to the second operator OP2 as a vibration stimulus presented by the motion state presentation device 41 worn by the second operator OP2. Therefore, the first operator OP1 and the second operator OP2 can operate the motion of the robot avatar 2 while intuitively and quickly grasping the motion state corresponding to the instruction motion of the other operator.

[0162] Furthermore, information when the gripping unit 21a grips an external object (grip state information of the gripping unit) is returned to the first operator OP1 and the second operator OP2 as a pressure stimulus presented by each action unit information presentation device 42 attached to each operator OP. Therefore, while operating the robot avatar 2, the first operator OP1 and the second operator OP2 can intuitively and quickly share the force that the gripping unit 21a receives from the object side (the physical action that the action unit 21 receives) as a pressure stimulus presented by the action unit information presentation device 42.

[0163] <Embodiment 6> Next, a control system 1E according to a sixth embodiment will be described with reference to Fig. 13. Fig. 13 is an explanatory diagram showing the relationship between input information input by each operator OP and feedback information returned to each operator OP in the control system 1E of the sixth embodiment. In this embodiment, one robot avatar 2 is operated by three operators OP (a first operator OP1, a second operator OP2, and a third operator OP3).

[0164] The control object whose operation is controlled by the operation of the first operator OP1 is the arm section 22, and the control object whose operation is controlled by the operation of the second operator OP2 is also the arm section 22. And the control object whose operation is controlled by the operation of the third operator OP3 is the gripping section 21a of the action section 21. The control objective for the first operator OP1 to control the operation of the arm section 22 is position control of the action section 21. The control objective for the second operator OP2 to control the operation of the arm section 22 is attitude control of the action section 21. And the control objective for the third operator OP3 to control the operation of the arm section 22 is opening and closing control of the gripping section 21a. Such a division of roles is effective, for example, when attitude control of the action section 21 is important and the action section 21 is to be carefully acted on an external object.

[0165] Under such conditions, when each operator OP controls the movement of the robot avatar (robot arm) 2, the information (input information) input to the robot avatar 2 side (control unit of the OP computer) is as follows:

[0166] A first operator OP1 inputs position information of a first rigid body for operating the arm unit 22 via a first input device 31 (camera 312) using motion capture for the purpose of controlling the position of the action unit 21. A second operator OP2 inputs posture information of a second rigid body for operating the arm unit 22 via the first input device 31 (camera 312) using motion capture for the purpose of controlling the posture of the action unit 21. A third operator OP3 inputs information (opening / closing information) for operating the grip unit 21a via a second input device (bending sensor) 32 for the purpose of controlling the opening / closing of the grip unit 21a.

[0167] In response to this, when controlling the movement of the robot avatar 2, the information (feedback information) returned from the robot avatar 2 side (control section of the OP computer) to each operator OP is as follows:

[0168] Two types of motion information, the motion information of the second operator OP2 and the motion information (grasping motion information) of the third operator OP3, are returned to the first operator OP1 as vibration stimuli presented by the motion state presentation device 41 worn by the first operator OP1. The motion state presentation device 41 for the first operator OP1 presents two types of vibration stimuli so that the motion information of each operator can be distinguished. In addition, two types of motion information, the motion information of the first operator OP1 and the motion information (grasping motion information) of the third operator are returned to the second operator OP2 as vibration stimuli presented by the motion state presentation device 41 worn by the second operator OP2. The motion state presentation device 41 for the second operator OP2 presents two types of vibration stimuli so that the motion information of each operator can be distinguished. In addition, two types of motion information, the motion information of the first operator OP2 and the motion information of the second operator OP2, are returned to the third operator OP3 as vibration stimuli presented by the motion state presentation device 41 worn by the third operator OP3. The motion state presentation device 41 for the third operator OP3 presents two types of vibration stimuli so that the motion information of each operator can be distinguished. Therefore, each of the three operators OP can intuitively and quickly grasp the motion state corresponding to the instruction motion of the other operator and control the motion of the robot avatar 2.

[0169] Furthermore, information when the gripping unit 21a grips an external object (grip state information of the gripping unit based on the detection result of the detection sensor 23) is returned to each of the three operators OP as a pressure stimulus presented by each action unit information presentation device 42 attached to each operator OP. Therefore, while operating the robot avatar 2, each of the three operators OP can intuitively and quickly share the force that the gripping unit 21a receives from the object side (the physical action that the action unit 21 receives) as a pressure stimulus presented by the action unit information presentation device 42.

[0170] <Embodiment 7> Next, the control system 1F according to Embodiment 7 will be described with reference to FIG. 14. FIG. 14 is an explanatory diagram showing the relationship between the input information input by each operator OP and the feedback information returned to each operator OP in the control system 1F of Embodiment 7. In the present embodiment, one robot avatar 2 is operated by two operators OP (first operator OP1, second operator OP2).

[0171] The control targets whose operations are controlled by the operation of the first operator OP1 and the control targets whose operations are controlled by the operation of the second operator OP2 are both the arm unit 22. Also, the control purposes for which the first operator OP1 controls the operation of the arm unit 22 and the control purposes for which the second operator OP2 controls the operation of the arm unit 22 are both the position and posture control of the working unit. Note that the opening and closing control of the gripping part 21a of the working part 21 is performed by the second operator OP2.

[0172] In the case of the present embodiment, for a plurality of operators OP (first operator OP1, second operator OP2), the ratios (control ratios) contributing to the control of the arm unit 22, which is a specific control target, are each predetermined.

[0173] For example, regarding the control of the operation of the arm unit 22, when the sum of the control ratios of each operator OP is α%, the control ratio of the first operator OP1 is αr%, and the control ratio of the second operator OP2 is α(1 - r)% (where 0 < r < 1). The respective values of α and r are set as appropriate.

[0174] The arm unit command generation unit (first generation unit) of the present embodiment executes a process of generating a plurality of operation commands for each operator OP to control the arm unit 22 according to each control ratio, based on a plurality of input information corresponding to a plurality of instructed operations so that the robot avatar 2 operates in accordance with the instructed operations performed by the plurality of operators OP.

[0175] In this embodiment, in the arm command generating unit (control unit of the OP computer), an input value based on a first operator OP1 is processed as a value of α%, and an input value based on a second operator OP2 is processed as a value of α(1-r)%. The input value of each operator OP is, for example, composed of the displacement amount (displacement amount of position coordinate data, displacement amount of rotation data) of the position and orientation information of each rigid body (first rigid body, second rigid body) corresponding to each operator OP.

[0176] For example, when α is set to 100 and r is set to 0.5, the control ratio of each of the first operator OP1 and the second operator OP2 is 50%. When α is set to 100 and r is set to 0.3, the control ratio of the first operator OP1 is 30%, and the control ratio of the second operator OP2 is 70%. When α is set to 200 and r is set to 0.5, the control ratio of each of the first operator OP1 and the second operator OP2 is 100%.

[0177] In this way, by determining for each operator OP the control ratio that contributes to the control of the control target (arm unit 22), the instructed actions of multiple operators OP can be combined in any ratio regarding the operation of the arm unit 22.

[0178] Under such conditions, when the first operator OP1 and the second operator OP2 control the movement of the robot avatar (robot arm) 2, the information (input information) input to the robot avatar 2 side (control unit of the OP computer) is as follows:

[0179] A first operator OP1 inputs position information and orientation information of a first rigid body for operating the arm unit 22 via a first input device 31 (camera 312) using motion capture for the purpose of controlling the position and orientation of the action unit 21. A second operator OP2 inputs position information and orientation information of a second rigid body for operating the arm unit 22 via the first input device 31 (camera 312) using motion capture for the purpose of controlling the position and orientation of the action unit 21.

[0180] Further, the second operator OP2 inputs information (opening / closing information) for operating the gripper 21a via the second input device (bending sensor) 32 for the purpose of controlling the opening / closing of the gripper 21a.

[0181] In response to this, when controlling the movement of the robot avatar 2, the information (feedback information) returned from the robot avatar 2 side (control section of the OP computer) to the first operator OP1 and the second operator OP2 is as follows:

[0182] The motion information of the second operator OP2 is returned to the first operator OP1 as a vibration stimulus presented by the motion state presentation device 41 worn by the first operator OP1. The motion information of the first operator OP1 is returned to the second operator OP2 as a vibration stimulus presented by the motion state presentation device 41 worn by the second operator OP2.

[0183] The motion information generating unit (second generating unit) included in the OP computer of this embodiment executes a process of generating a plurality of motion information (tactile vibration signals) corresponding to the instruction motion of each operator OP based on a plurality of input information (three-dimensional position and orientation information of the first rigid body and the second rigid body) generated by the image analyzing unit 511. Here, for each rigid body (first rigid body, second rigid body), the velocity is calculated from the time change of the position and orientation, and the scalar amount is used as the motion information of each operator OP. In addition, a sine wave of 200 [Hz] is amplitude-modulated in response to the value (scalar amount) obtained here, and the resulting value is presented as vibration (vibration stimulus) corresponding to the motion information using the motion state presenting device 41 of the corresponding operator OP. Therefore, the first operator OP1 and the second operator OP2 can intuitively and quickly grasp the motion state corresponding to the instruction motion of the other side and control the motion of the robot avatar 2.

[0184] Furthermore, information when the gripping unit 21a grips an external object (grip state information of the gripping unit based on the detection result of the detection sensor 23) is returned to the first operator OP1 and the second operator OP2 as a pressure stimulus presented by each action unit information presentation device 42 attached to each operator OP. Therefore, while operating the robot avatar 2, the first operator OP1 and the second operator OP2 can intuitively and quickly share the force that the gripping unit 21a receives from the object side (the physical action that the action unit 21 receives) as a pressure stimulus presented by the action unit information presentation device 42.

[0185] <Embodiment 8> Next, a control system 1G according to an eighth embodiment will be described with reference to Fig. 15. Fig. 15 is an explanatory diagram showing the relationship between input information input by each operator OP and feedback information returned to each operator OP in the control system 1G of the eighth embodiment. In this embodiment, one robot avatar 2 is operated by two operators OP (a first operator OP1 and a second operator OP2).

[0186] The control system 1G of this embodiment is the same as that of the seventh embodiment in terms of the control content of the arm unit 22 of the robot avatar 2, and the control ratio contributing to the control of the arm unit 22 is determined in advance for each operator OP. Since the control content of the arm unit 22 is the same as that of the seventh embodiment, a description thereof will be omitted.

[0187] In this embodiment, the first operator OP1 and the second operator OP2 both control the action part 21. The control purpose of the first operator OP1 and the second operator OP2 is also common to the opening and closing control of the grip part 21a. Each operator OP uses the second input device (bending sensor) 32 to instruct the opening and closing operation of the grip part 21a.

[0188] For each operator OP, a control ratio contributing to the opening / closing control of the gripping part 21a is determined in advance. Regarding the opening / closing control of the gripping part 21a, when the total of the control ratios of the respective operators OP is α%, the control ratio of the first operator OP1 is αr%, and the control ratio of the second operator OP2 is α(1 - r)% (where 0 < r < 1). Note that the values of α and r may be the same as those of the arm part 22 or different from them.

[0189] The control unit of the OP operator in this embodiment executes a process of generating a plurality of operation commands for each operator OP to control the gripping part 21a according to each control ratio, based on a plurality of input information corresponding to a plurality of instruction operations so that the robot avatar 2 operates in accordance with the instruction operations performed by a plurality of operators OP respectively.

[0190] In the control unit of the OP operator in this embodiment, regarding the opening / closing control of the gripping part 21a, the input value based on the first operator OP1 is processed as a value of αr%, and the input value based on the second operator OP2 is processed as a value of α(1 - r)%. The input value of each operator OP consists of, for example, the amount of change in the output resistance value from each second input device 32 corresponding to each operator OP.

[0191] In this way, if a control ratio contributing to the opening / closing control of the control target (the gripping part 21a of the acting part 21) is determined for each operator OP, the instruction operations of a plurality of operators OP regarding the opening / closing operation of the gripping part 21a can be fused at an arbitrary ratio.

[0192] Under such conditions, when the first operator OP1 and the second operator OP2 control the operation of the robot avatar (robot arm) 2, the information (input information) input to the robot avatar 2 side (the control unit of the OP computer) is as follows.

[0193] A first operator OP1 inputs position information and orientation information of a first rigid body for operating the arm unit 22 via a first input device 31 (camera 312) using motion capture for the purpose of controlling the position and orientation of the action unit 21. A second operator OP2 inputs position information and orientation information of a second rigid body for operating the arm unit 22 via the first input device 31 (camera 312) using motion capture for the purpose of controlling the position and orientation of the action unit 21.

[0194] Moreover, for the purpose of controlling the opening and closing of the gripper 21a, the first operator OP1 inputs information (opening / closing information) for operating the gripper 21a via the second input device (bending sensor) 32. Moreover, for the purpose of controlling the opening and closing of the gripper 21a, the second operator OP2 inputs information (opening / closing information) for operating the gripper 21a via the second input device (bending sensor) 32.

[0195] In response to this, when controlling the movement of the robot avatar 2, the information (feedback information) returned from the robot avatar 2 side (control section of the OP computer) to the first operator OP1 and the second operator OP2 is as follows:

[0196] The motion information of the second operator OP2 is returned to the first operator OP1 as a vibration stimulus presented by the motion state presentation device 41 worn by the first operator OP1. The motion information of the first operator OP1 is returned to the second operator OP2 as a vibration stimulus presented by the motion state presentation device 41 worn by the second operator OP2. Therefore, the first operator OP1 and the second operator OP2 can operate the motion of the robot avatar 2 while intuitively and quickly grasping the motion state corresponding to the instruction motion of the other operator.

[0197] Furthermore, information when the gripping unit 21a grips an external object (grip state information of the gripping unit based on the detection result of the detection sensor 23) is returned to the first operator OP1 and the second operator OP2 as a pressure stimulus presented by each action unit information presentation device 42 attached to each operator OP. Therefore, while operating the robot avatar 2, the first operator OP1 and the second operator OP2 can intuitively and quickly share the force that the gripping unit 21a receives from the object side (the physical action that the action unit 21 receives) as a pressure stimulus presented by the action unit information presentation device 42.

[0198] <Other embodiments> The present invention is not limited to the embodiments described above and illustrated in the drawings, and the following embodiments, for example, are also included within the technical scope of the present invention.

[0199] (1) When motion capture is used as an input device for operating a robot avatar, it is not limited to the optical motion capture exemplified in embodiment 1. In other embodiments, other types of motion capture, such as magnetic, mechanical, inertial sensor, and image recognition, may be used as long as the object of the present invention is not impaired. In addition, the optical motion capture may be, for example, a type in which an operator holds a specific controller in his / her hand and captures light (infrared light) from an infrared LED provided on the controller with a camera.

[0200] (2) As an input device for operating a robot avatar, for example, a glove-type sensor worn on the operator's hand (high-performance data glove, "CyberGlove", manufactured by CyberGlove Systems) may be used.

[0201] (3) In the above embodiment 1 etc., the operation of the action part (gripping part) was performed using a bending sensor, but in other embodiments, the operation of the action part (gripping part) may be performed using other input devices such as motion capture.

[0202] (4) In cases where an operator has a physical disability, the parts of the body that the operator can move and the range (distance) that the operator can move may be limited. For this reason, the input device may be a mechanical operation device (push-type switch, joystick, etc.), keyboard, eye tracker, etc. that can be operated by the operator with limited movements.

[0203] (5) In another embodiment, the motion state related to the instruction action of the operator may be the motion state (motion state) of a control object that operates in response to the instruction action of the operator. For example, a detection device that detects the position of each part of a robot avatar, such as an encoder provided on a robot avatar (robot arm), can be used to acquire information (detection signal from the detection device) on the motion state of a control object (e.g., an arm part) that operates in response to the instruction action of the operator. In this case, based on the acquired information, the control unit of the OP computer generates motion information corresponding to the motion state related to the instruction action of the operator (motion state of a control object that operates in response to the instruction action of the operator). Furthermore, the control unit of the OP computer feeds back the motion information to a motion state presentation device of another operator other than the operator. [Explanation of symbols]

[0204] 1...robot avatar control system, 2...robot avatar (robot arm), 21...action unit, 21a...grasping unit, 21b...finger unit, 22...main body unit (arm unit), 3...input device, 311...marker, 312...camera, 31...first input device, 32...second input device, 4...information presentation device, 41...motion state presentation device, 411...vibrator, 42...action unit information presentation device, 5...operating computer (OP computer), 510...control unit, 6...controller, 7...first microcomputer, 8...second microcomputer

Claims

1. A robot avatar including a control target whose movement is to be controlled; an input device that generates a plurality of pieces of input information based on instruction actions performed by a plurality of operators to cause the robot avatar to execute a task, the input device being attached to each of the plurality of operators; a first generator that generates a plurality of motion commands for moving the control target based on the plurality of pieces of input information such that the robot avatar moves in accordance with the plurality of instruction motions; an operation control unit that controls an operation of the control target based on the plurality of operation commands; a plurality of motion state presentation devices that are attached to the plurality of operators separately from the input device and present a tactile stimulus to each of the operators so that each operator can grasp a motion state related to the instruction action of the other operators when using the input device; A robot avatar control system for multiple operators, in which each operator can feel the tactile stimulus presented by the motion state presentation device separately from the instruction action performed on the input device.

2. a second generator that generates motion information corresponding to the motion state based on the input information or based on information on a motion state of the control object that operates in response to the instruction action, The motion information is information related to at least one of a position and a posture, The robot avatar control system for multiple operators according to claim 1 , wherein the motion state presentation device presents the tactile stimulation based on the motion information relating to other operators than the operator himself / herself.

3. The second generator generates the motion information by using a physical quantity related to the motion state, The exercise information includes: When the physical quantity is a time change amount of the position, the physical quantity is velocity information, 3. A robot avatar control system for multiple operators according to claim 2, wherein the physical quantity is rotational angular velocity information when the physical quantity is a time change amount of the posture.

4. The robot avatar control system for multiple operators according to any one of claims 1 to 3, wherein the motion state presentation device has a vibrator that presents a vibration stimulus as the tactile stimulus to the operator.

5. A plurality of operators are assigned different control objects or different control objectives, The robot avatar control system for multiple operators described in any one of claims 1 to 3, wherein the first generation unit generates a plurality of the action commands based on a plurality of the input information for the multiple operators to control different control objects, or for the multiple operators to control for different control purposes.

6. A ratio of contribution to the control of a specific control target is determined for each of the multiple operators, The robot avatar control system for multiple operators described in any one of claims 1 to 3, wherein the first generation unit generates a plurality of the action commands based on a plurality of the input information for a plurality of the operators to control a specific one of the control objects according to the ratio.

7. A robot avatar control system for multiple people described in any one of claims 1 to 3, wherein the robot avatar has an action part operable to act on an object, and a main body part capable of moving while holding the action part.

8. a detection sensor attached to the action portion and configured to detect a physical action received by the action portion from the object when the action portion acts on the object; an action unit information presentation device that is attached to each of the multiple operators and presents a tactile stimulus corresponding to a detection result of the detection sensor to each operator so that the action that the action unit receives from the object side can be shared among the operators when the action unit is in operation; each of the operators can feel the tactile stimulation presented by the action unit information presentation device separately from the instruction action performed on the input device; The robot avatar control system for multiple operators according to claim 7 , wherein the same tactile stimulus is presented to each of the operators by each of the action part information presentation devices.

9. A robot avatar control system for multiple people as described in any one of claims 1 to 3, wherein the instruction action performed by the operator on the input device comprises a three-dimensional action of moving a part of the operator's body in three dimensions.

10. A multi-person robot avatar control system as described in any one of claims 1 to 3, further comprising a display device that displays an image of the robot avatar to the operator so that the operator can perform the instruction action without seeing the actual robot avatar.

11. A control method for controlling a single robot avatar including a control target whose movement is controlled by a plurality of operators, comprising the steps of: an input information generating step of generating a plurality of pieces of input information by input devices respectively attached to the plurality of operators based on instruction actions performed by the plurality of operators to cause the robot avatar to execute a task; a motion command generating step of generating a plurality of motion commands for moving the control target based on the plurality of pieces of input information so that the robot avatar moves in accordance with the plurality of instruction motions; a motion control step of controlling a motion of the control target based on the plurality of motion commands; a tactile stimulus presentation step of presenting a tactile stimulus to each of a plurality of motion state presentation devices, which are respectively attached to the plurality of operators separately from the input device, so that each operator can grasp a motion state related to the instruction motion of the other operators when using the input device; A control method in which, in the tactile stimulus presentation step, each operator can feel the tactile stimulus presented from the motion state presentation device separately from the instruction action performed on the input device.

12. a motion information generating step of generating motion information corresponding to the motion state based on the input information or based on information regarding a motion state of the control object that operates in response to the instruction motion, The motion information is information related to at least one of a position and a posture, The control method according to claim 11 , wherein in the tactile stimulus presenting step, the tactile stimulus is presented based on the motion information relating to another operator other than the operator himself / herself.

13. The plurality of operators are assigned different control objects or different control objectives, 13. The control method according to claim 11 or 12, wherein the operation command generation step generates a plurality of operation commands for a plurality of operators to control different control objects based on a plurality of pieces of input information, or for a plurality of operators to control for different control purposes.

14. A ratio of contribution to the control of a specific control target is determined for each of the multiple operators, 13. The control method according to claim 11, wherein the operation command generating step generates a plurality of the operation commands for a plurality of the operators to control specific ones of the control targets according to the ratios, based on a plurality of the input information.

Images