Control system and control method for robot avatar cooperation operation by multiple persons

Abstract

To provide a control system for robot avatar cooperation operation by multiple persons which is excellent in operability.SOLUTION: A control system for robot avatar cooperation by multiple persons includes: one robot avatar 2 having a plurality of control objects 22, 23; a plurality of operation means 3 for operating each action of the plurality of control objects by multiple operators OP; an input information generation part 511 for generating input information corresponding to the operation action; an action planning part 611 for generating operation planning information of the control objects; a delay setting part 612 for setting a predetermined delay time; a motion information generation part 512 for generating motion information on the operation action; motion information presentation parts 4, 41 for presenting the motion information on the operation action by an arbitrary operator to residual operators; and an action instruction part 71 for operating the corresponding control object on the basis of the action planning information after passage of delay time, so that the motion information on the operation action by the arbitrary operator is previously presented to the residual operators, before the corresponding control object in response to the operation action is operated.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a control system and a control method for cooperative operation of a robot avatar by multiple people. [Background technology]

[0002] In recent years, robot avatar technology, in which a human being reflects his or her own movements in a robot and controls the robot as the human's alter-ego (avatar), has been attracting attention. Among these, technology in which multiple operators work together to operate one robot avatar has particularly attracted attention (see Patent Document 1). With such technology, for example, a robot arm (robot avatar) with degrees of freedom exceeding those of a human being can be operated by multiple people working together to enable the movement of the robot arm independent of the range of motion of the human being. For example, motion capture or the like is used for the operation of the robot avatar by the operator. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2023 / 037966 Summary of the Invention [Problem to be solved by the invention]

[0004] In order for multiple operators to smoothly operate a single robot avatar in cooperation with each other, it is important that the movements of the other person (the other person) feel as if they are their own body, just as if they are the movements of the other person (the other person) that they are operating.

[0005] In conventional robot avatar control systems, for example, when any one operator operates a robot avatar, a vibration stimulus is generated in response and fed back to the other operators, thereby promoting cooperation between operators.

[0006] In the case of a conventional robot avatar control system, the timing at which the robot avatar moves due to the operation of an operator is the same as the timing at which the vibration stimulus generated in response to the operation is fed back to other operators other than the operator. Therefore, before the robot avatar moves due to the operation of the operator, the other operators cannot predict the movement (motion) of the robot avatar in advance.

[0007] An object of the present invention is to provide a control system etc. for cooperative operation of robot avatars by multiple people with excellent operability. [Means for solving the problem]

[0008] The means for solving the above problems are as follows. <1> A robot avatar having a plurality of control objects, each of which is operable; a plurality of operation means each assigned to a plurality of operators so that the plurality of operators can share the task of operating the plurality of control objects, and each operating the plurality of control objects; an input information generating unit that generates, for each of the operation means, input information corresponding to an operation action performed by the operator when operating the operation means; an operation planning unit that generates operation plan information for the corresponding control object based on the input information so that the corresponding control object operates in accordance with the operation operation; a delay setting unit that sets a predetermined delay time for the operation plan information; a motion information generating unit configured to generate motion information related to the corresponding operation motion for each of the operation means; an exercise information presentation unit that presents the exercise information related to the operation action by a given operator to other operators other than the given operator so that the operators can understand each other's operation actions; and an action command unit that operates the corresponding control object based on the action plan information after the delay time has elapsed so that the motion information regarding the operation action by the arbitrary operator is presented to the remaining operators in advance before the corresponding control object operates in accordance with the operation action by the arbitrary operator.

[0009] <2> The delay time is 150 ms or more and 300 ms or less. <1> A control system for cooperative operation of a robot avatar by multiple people as described above.

[0010] <3> the motion information presenting unit presents the motion information related to the operation motion by an arbitrary operator to remaining operators other than the arbitrary operator within 150 ms from the start of the operation motion. <1> or <2> A control system for cooperative operation of a robot avatar by multiple people as described above.

[0011] <4> the motion information presentation unit has a vibrator that presents a tactile stimulus corresponding to the motion information to the operator; <1> ~ <3> 2. A control system for cooperative operation of a robot avatar by multiple people, comprising:

[0012] <5> the exercise information presentation unit performs lighting drive corresponding to the exercise information or displays an image corresponding to the exercise information so that the operator can visually recognize the exercise information. <1> ~ <4> 2. A control system for cooperative operation of a robot avatar by multiple people, comprising:

[0013] <6> The motion information generating unit generates the motion information by using a physical quantity related to the operation motion. <1> ~ <5> 2. A control system for cooperative operation of a robot avatar by multiple people, comprising:

[0014] <7> The object to be controlled includes a robot arm with multiple degrees of freedom. <1> ~ <6> by multiple people described in any one of A control system for collaborative operation of robot avatars.

[0015] <8> A control method for allowing multiple people to cooperatively operate a robot avatar having multiple control objects, each of which is operable, comprising the steps of: an operation process in which a plurality of operation means respectively assigned to a plurality of operators are operated to operate the plurality of control objects in a shared manner by the plurality of operators; an input information generating step of generating, for each of the operation means, input information corresponding to an operation action performed by the operator when operating the operation means; an action planning step of generating action plan information for the corresponding control object based on the input information so that the corresponding control object moves in accordance with the operation action; a delay setting step of setting a predetermined delay time for the operation plan information; a motion information generating step of generating motion information relating to the corresponding operation motion for each of the operation means; an exercise information presenting step of presenting the exercise information related to the operation action by a given operator to remaining operators other than the given operator so that the operators can understand each other's operation actions; and an operation command process of operating the corresponding control object based on the operation plan information after the delay time has elapsed so that the motion information regarding the operation action by the arbitrary operator is presented to the remaining operators in advance before the corresponding control object operates in accordance with the operation action by the arbitrary operator.

[0016] <9> The delay time is 150 ms or more and 300 ms or less. <8> The control method described in

[0017] <10> In the motion information presenting step, the motion information regarding the operation motion by an arbitrary operator is presented to remaining operators other than the arbitrary operator within 150 ms from the start of the operation motion. <8> or <9> The control method described in Effect of the Invention

[0018] According to the present invention, it is possible to provide a control system and the like that allows multiple people to cooperatively operate a robot avatar with excellent operability. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is an explanatory diagram showing the overall configuration of a control system for cooperative operation of robot avatars by multiple people according to a first embodiment. [Diagram 2] Illustration of a robot avatar [Diagram 3] FIG. 1 is an explanatory diagram showing an operation means and a tactile stimulation motion information presentation unit worn by a first operator. [Figure 4] FIG. 13 is an explanatory diagram showing the operation means, the grip operation means, and the tactile sense stimulation exercise information presentation unit that are attached to the second operator. [Diagram 5] FIG. 1 is an explanatory diagram showing a hardware configuration of a first computer. [Figure 6] An explanatory diagram of the exercise information display unit for lighting up, which is attached to the arm at the tip side. [Figure 7] FIG. 13 is an explanatory diagram showing the overall configuration of a control system according to a second embodiment. [Figure 8] FIG. 13 is an explanatory diagram showing the overall configuration of a control system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] <Embodiment 1> A control system 1 for cooperative operation of a robot avatar by multiple people according to a first embodiment of the present invention will be described with reference to Figs. 1 to 6. Fig. 1 is an explanatory diagram showing an overall configuration of the control system 1 for cooperative operation of a robot avatar by multiple people according to the first embodiment. The control system 1 for cooperative operation of a robot avatar by multiple people according to this embodiment is a system intended for multiple operators OP to cooperate and jointly operate one robot avatar 2 while sharing the operation among themselves and grasping each other's operation information. In this specification, the "control system for cooperative operation of a robot avatar by multiple people" may be simply referred to as the "control system".

[0021] Although each operator OP can grasp (perceive) its own operation status, it is usually difficult for the operator OP to grasp the operation status of other operator OPs. For example, when each operator OP operates the robot avatar 2 from a location distant from each other, the operator OPs cannot visually confirm and cannot grasp each other's operation status. In addition, since the operator OP concentrates on operating the robot avatar 2, it is difficult for the operator OP to grasp the operation status of the other operator OPs even if the other operator OPs are nearby.

[0022] Therefore, in the control system 1 of this embodiment, in order to enable the operators OP to reliably understand each other's operating actions, motion information regarding the operating action of any operator OP is presented to the remaining operators OP other than the arbitrary operator OP as visual information such as tactile stimulation (e.g., vibration stimulation), lighting display, image display, etc.

[0023] In particular, in the control system 1 of this embodiment, before the control target of the corresponding robot avatar 2 moves in accordance with the operation action of any operator OP, motion information related to the operation action by that arbitrary operator OP is presented to the remaining operators OP in advance. In the case of this embodiment, when any operator OP performs an operation action, the motion information related to the operation action is presented (feedback) to the remaining operators OP immediately (for example, within less than 150 ms from the start of the operation action) via the vibrator 411 and the LEDs 42a and 42b, but the motion of the control target of the corresponding robot avatar 2 does not start immediately, but starts with a slight delay from the presentation of the motion information. In short, the remaining operators OP can share the motion information during the delay time before the control target of the robot avatar 2 starts to move, so that they can predict the motion of the control target in advance. Therefore, in the control system 1 of this embodiment, the operators OP can operate the control target they are in charge of while knowing each other's operation actions (motion of the control target) in advance, so that they can easily operate in cooperation with each other.

[0024] In addition, even if there are three or more operators OP, the operation actions of the other operators OP can be distinguished and understood by appropriately setting the type of lighting display, the type of tactile stimulation, etc. The motion information can be understood, for example, from the presence or absence of tactile stimulation, changes in tactile stimulation (changes in strength, change in type, etc.), the presence or absence of lighting, the lighting pattern (differences in lighting time and extinguishing time, etc.), changes in lighting color, etc.

[0025] Here, an example will be described in which two operators OP operate one robot avatar 2. For ease of explanation, in order to distinguish between the two operators OP, one will be referred to as a "first operator OP1" and the other as a "second operator OP2."

[0026] FIG. 2 is an explanatory diagram of the robot avatar 2. The robot avatar 2 is a robot used as an alter ego (avatar) to reflect the movement of the operator OP for the purpose of performing a predetermined task. The robot avatar 2 moves according to the movement (operation movement) performed by each operator OP for operation. In the case of this embodiment, the robot avatar 2 performs a movement similar to the movement (operation movement) performed by each operator OP (that is, a movement in accordance with the movement of each operator OP). Moreover, as described above, when each operator OP operates the robot avatar 2, the operator OP can grasp the operation movement of the other operator OP other than himself in advance during the delay time before the control target of the corresponding robot avatar 2 moves. Therefore, each operator OP can cooperate and operate one robot avatar 2 in cooperation with each other while feeling as if each alter ego (avatar) was fused into one. The control system 1 will be described in detail below.

[0027] The control system 1 of this embodiment includes a robot avatar 2, an operation means 3, a gripping portion operation means 31, and a tactile sense stimulation motion information presentation unit 4.

[0028] The robot avatar 2 is composed of a multi-degree-of-freedom robot arm as shown in FIG. 2. The robot avatar 2 mainly comprises a tip arm (an example of a controlled object) 22 having a gripper 21 attached to its tip, and a base arm (an example of a controlled object) 23. The tip arm 22 is composed of a four-degree-of-freedom robot arm ("mikata Arm", manufactured by Robotis), and the base arm 23 is composed of a seven-degree-of-freedom robot arm ("xArm7", manufactured by UFACTORY). A predetermined joint is attached to the tip of the base arm 23, and the tip arm 22 is connected to the tip side of the base arm 23 via the joint.

[0029] The root side arm 23 is used with its base 23a fixed on a predetermined stage (not shown). The root side arm 23 has a plurality of link parts and a plurality of joint parts connecting the link parts to each other. The root side arm 23 also has a plurality of motors (drive parts) for rotating the link parts fixed to the joint parts in a predetermined direction. The root side arm 23 can move three-dimensionally while holding the tip side arm 22 by controlling each of these rotation drives. The root side arm 23 is provided with an encoder for detecting the position (angle) of each drive shaft (rotation shaft) of each motor.

[0030] The tip side arm 22 is used with the gripper 21 attached to its tip side and with the root side connected to the tip side of the root side arm 23. The tip side arm 22 has a plurality of link parts and a plurality of joint parts connecting these link parts to each other. The tip side arm 22 also has a plurality of motors (drive parts) for rotating the link parts fixed to the joint parts in a predetermined direction. The tip side arm 22 can move three-dimensionally while holding the gripper 21 by controlling each of these rotation drives. The tip side arm 22 is provided with an encoder for detecting the position (angle) of each drive shaft (rotation shaft) of each motor.

[0031] The movable control object, the operation of which is controlled by the operation of the first operator OP1, is the root side arm 23. The first operator OP1 operates the root side arm 23 so that its position and attitude are at a desired position and attitude. The position (three-dimensional position) of the predetermined part 23R set on the tip side of the root side arm 23 is expressed using a three-dimensional coordinate system (root side three-dimensional coordinate system) set for the root side arm 23. A three-dimensional coordinate system with an origin located at a predetermined position is set for the root side arm 23, and the three-dimensional position of the predetermined part 23R of the root side arm 23 is expressed using the values ​​(x, y, z) of the x-axis, y-axis, and z-axis in the three-dimensional coordinate system. The attitude (three-dimensional attitude) of the predetermined part 23R of the root side arm 23 is expressed using the rotation angle (roll angle) around the x-axis, the rotation angle (yaw angle) around the z-axis, and the rotation angle (pitch angle) around the y-axis of the three-dimensional coordinate system.

[0032] The movement of the root side arm 23 is controlled so that the position and posture of a predetermined portion 23R of the root side arm 23 becomes a predetermined position and posture instructed by the first operator OP1. The position (three-dimensional position) of the predetermined portion 23R is controlled based on three-dimensional position information corresponding to the operation movement of the first operator OP1, and the posture (three-dimensional posture) of the predetermined portion 23R is controlled based on three-dimensional posture information corresponding to the operation movement of the first operator OP1. The three-dimensional position information and three-dimensional posture information are collectively referred to as "three-dimensional position / posture information."

[0033] On the other hand, the movable control object whose operation is controlled by the operation of the second operator OP2 is the tip side arm 22. The tip side arm 22 is operated by the second operator OP2 so that the position and posture of the tip side arm 22 are the desired position and posture. The position (three-dimensional position) of the predetermined part 22R set on the tip side of the tip side arm 22 is expressed using a three-dimensional coordinate system (tip side three-dimensional coordinate system) set for the tip side arm 22. A three-dimensional coordinate system with an origin located at a predetermined position is set for the tip side arm 22, and the three-dimensional position of the predetermined part 22R of the tip side arm 22 is expressed using each value (x, y, z) of the x-axis, y-axis, and z-axis in the three-dimensional coordinate system. In addition, the posture (three-dimensional posture) of the predetermined part 22R of the tip side arm 22 is expressed using the rotation angle (roll angle) around the x-axis, the rotation angle (yaw angle) around the z-axis, and the rotation angle (pitch angle) around the y-axis of the above three-dimensional coordinate system.

[0034] The operation of the tip-side arm 22 is controlled so that the position and posture of a predetermined part 22R of the tip-side arm 22 becomes a predetermined position and posture instructed by the second operator OP2. The position (three-dimensional position) of the predetermined part 22R is controlled based on three-dimensional position information corresponding to the operation movement of the second operator OP2, and the posture (three-dimensional posture) of the predetermined part 22R is controlled based on three-dimensional posture information corresponding to the operation movement of the second operator OP2.

[0035] In this embodiment, the second operator OP2 also performs operations for opening and closing the gripper 21 at the tip of the tip-side arm 22.

[0036] The gripping unit 21 has a function of gripping an object. The gripping unit 21 has a plurality of (three) finger portions 21a that are movable when gripping an object. The plurality of finger portions 21a are divided into two and one, and are arranged facing each other, and an object can be pinched between the facing finger portions 21a. The gripping unit 21 has a drive portion (motor, etc.) for driving each finger portion 21a, and the gripping unit 21 opens and closes by controlling the drive portion (motor, etc.).

[0037] The operation means 3 is a device used by the operator OP when operating the robot avatar 2. The operator OP uses the operation means 3 to perform an operation action (three-dimensional action) for moving a controlled object. The "operation action" performed by the operator OP is an action in which the operator OP moves a part of the body (for example, an arm, a leg, a head, a finger, etc.) for moving a controlled object of the robot avatar 2.

[0038] In the control system 1 of this embodiment, each of the multiple operators OP is assigned one operating means 3 so that the multiple operators OP can share the task of operating the operations of the multiple control objects. The multiple operators OP use the multiple operating means 3 to operate the operations of the multiple control objects, respectively.

[0039] In the control system 1 of this embodiment, a plurality of operating means 3 are assigned to a plurality of operators OP so that the plurality of operators OP can share the task of operating a plurality of control objects, and are each operated to operate the plurality of control objects (operation process).

[0040] 3 is an explanatory diagram showing the operation means 3 and the tactile stimulation motion information presentation unit 4 attached to the first operator OP1. The operation means 3 shown in FIG. 3 is an operation means for optical motion capture, and is used to operate the movement of the base side arm 23.

[0041] As is well known, motion capture is a technique in which a plurality of markers 311 provided on an operating means 3 worn on a predetermined part of an operator OP are photographed by a plurality of cameras (imaging devices) 312 at different angles, and the movements of the markers 311 are measured by the principle of triangulation based on the photographed images obtained by the cameras 312. In the case of this embodiment, "OptiTrack Prime 13W" (manufactured by Acuity Corporation) was used as the motion capture.

[0042] As shown in FIG. 3, the operation means 3 is attached to the back OP1a of the right hand of the first operator OP1. The operation means 3 includes a plurality of (four) markers 311 and a tool 313 for attaching them. The tool 313 includes a flat plate-shaped mounting portion 313a to be placed on the back of the hand, and a plurality of mounting rods 313b each extending in a different direction from the periphery of the mounting portion 313a outward. The markers 311 are attached to the tips of the four mounting rods 313b. The markers 311 are spheres coated with paint for reflecting infrared light irradiated from the camera 312. The tool 313 is fixed in place by a band 314 that is wrapped around the mounting portion 313a so that it is sandwiched between the mounting portion 313a and the back of the right hand, preventing it from shifting out of position.

[0043] A three-dimensional coordinate system (three-dimensional coordinate system for the first operator) with the origin located at a predetermined position is set for the first operator OP1, and this three-dimensional coordinate system corresponds to the three-dimensional coordinate system for the root side arm 23.

[0044] The position (three-dimensional position) of the back of the right hand OP1a of the first operator OP1 (hereinafter sometimes referred to as the "first rigid body") is expressed using the values ​​(x', y', z') of the x-axis, y-axis, and z-axis in the three-dimensional coordinate system for the first operator OP1. In addition, the posture (three-dimensional posture) of the back of the hand OP1a of the first operator OP1 is expressed using the rotation angle (roll angle) around the x-axis, the rotation angle (yaw angle) around the z-axis, and the rotation angle (pitch angle) around the y-axis of the three-dimensional coordinate system for the first operator OP1.

[0045] The movements of multiple markers 311 attached to the right hand of a first operator OP1 are captured by a camera 312. The captured images are then processed by an image analysis unit (input information generation unit) 511 provided in a first computer 5, which will be described later. From the captured images, three-dimensional position and orientation information (input information) of the first rigid body is obtained in time series.

[0046] 4 is an explanatory diagram showing the operation means 3 attached to the second operator OP2, the gripping unit operation means 31, and the tactile stimulation motion information presentation unit 4. The operation means 3 shown in FIG. 4 is an operation means for optical motion capture, and is used to operate the movement of the distal end arm 22.

[0047] As shown in Fig. 4, the second operator OP2 is also fitted with the operating means 3 on the back OP2a of his right hand, similar to that of the first operator OP1. The operating means 3 includes a plurality of (four) markers 311 and an instrument 313 for attaching them. The instrument 313 includes four attachment rods 313b extending outward from the periphery of a placement section 313a, and each of the four attachment rods 313b has a marker 311 attached to its tip. In order to distinguish between the operating action of the first operator OP1 and the operating action of the second operator OP2, the three-dimensional arrangement of the four markers 311 used by the second operator OP2 is set to be different from the three-dimensional arrangement of the four markers 311 used by the first operator OP1.

[0048] A three-dimensional coordinate system (three-dimensional coordinate system for the second operator) with the origin located at a predetermined position is set for the second operator OP2, and this three-dimensional coordinate system corresponds to the three-dimensional coordinate system for the tip side arm 22.

[0049] The position (three-dimensional position) of the back of the right hand OP2a of the second operator OP2 (hereinafter sometimes referred to as the "second rigid body") is expressed using the values ​​(x', y', z') of the x-axis, y-axis, and z-axis in the three-dimensional coordinate system for the second operator OP2. In addition, the posture (three-dimensional posture) of the back of the hand OP2a of the second operator OP2 is expressed using the rotation angle (roll angle) around the x-axis, the rotation angle (yaw angle) around the z-axis, and the rotation angle (pitch angle) around the y-axis in the three-dimensional coordinate system for the second operator OP2.

[0050] In addition, the movements of multiple markers 311 attached to the right hand of the second operator OP2 are captured by a camera 312, and the captured images are processed by an image analysis unit (input information generation unit) 511 provided in the first computer 5 described later, whereby three-dimensional position and orientation information (input information) of the second rigid body is obtained in time series from the captured images.

[0051] As shown in FIG. 4, the second operator OP2 uses the gripper operation means 31 to operate the opening and closing of the gripper 21 (fingers 21a) at the tip of the tip-side arm 22. The gripper operation means 31 adjusts the opening and closing of the gripper 21 according to the distance (spacing) between the long-hand portion 31a applied to the index finger and the long-hand portion 31b applied to the thumb while pinching it between the index finger and the thumb. One end of the long-hand portion 31a and one end of the long-hand portion 31b are pivotally supported in a mutually rotatable state, and a position sensor (e.g., a potentiometer) is provided at the pivotally supported portion. A signal line (not shown) is connected to the electrode terminal of the position sensor. The distance between the opposing fingers 21a of the gripper 21 is adjusted according to the distance between the long-hand portion 31a and the long-hand portion 31b. From the grip operation means 31, an output (input information) of a position sensor according to the distance between the elongated portion 31a and the elongated portion 31b is transmitted to a microcomputer 8, which will be described later.

[0052] When the second operator OP2 operates the grip operation means 31 so as to narrow the distance between the index finger and thumb of the right hand, the opposing finger portions 21a of the grip 21 move closer to each other. In contrast, when the second operator OP2 operates the grip operation means 31 so as to widen the distance between the index finger and thumb of the right hand, the opposing finger portions 21a of the grip 21 move away from each other.

[0053] The tactile stimulation motion information presentation unit (motion information presentation unit) 4 presents motion information regarding the operation motion by any operator OP as a tactile stimulation to the remaining operators OP other than the arbitrary operator OP so that the operators OP can understand each other's operation motions (motion information presentation process).

[0054] The vibrator 411 is a device that is attached to a plurality of (all) operators OP and presents motion information related to an operation action by an arbitrary operator OP to the remaining operators OP other than the arbitrary operator OP as a tactile stimulus so that the operators OP can understand each other's operation actions. In the case of this embodiment, the vibrator 411 is attached to each of the right wrists of the first operator OP1 and the second operator OP2.

[0055] The vibrator 411 attached to the first operator OP1 presents motion information related to the operating movement of the second operator OP2 to the first operator OP1 as a vibration stimulus (an example of a tactile stimulus) so that the second operator OP2 can understand the operating movement of the second operator OP2 when the second operator OP2 uses the operating means 3.

[0056] 3, the vibrator 411 is attached to the first operator OP1 using a band 412 so as to be in close contact with the outer skin OP1c near the right wrist. The band 412 is wrapped around the right wrist of the first operator OP1 and attached so that the vibrator 411 is sandwiched between the band 412 and the skin OP1c. When the second operator OP2 uses the operating means 3, the vibrator 411 attached to the first operator OP1 immediately (almost simultaneously with the operating action) receives motion information (tactile vibration signal) related to the operating action of the second operator OP2 and vibrates.

[0057] In response to this, the vibrator 411 attached to the second operator OP2 presents motion information related to the operation movement of the first operator OP1 to the second operator OP2 as a vibration stimulus (an example of a tactile stimulus) so that the second operator OP2 can understand the operation movement of the first operator OP1 when the first operator OP1 uses the operating means 3.

[0058] 4, the vibrator 411 is attached to the second operator OP2 using a band 412 so as to be in close contact with the outer skin OP2c near the right wrist of the second operator OP2. The band 412 is wrapped around the right wrist of the second operator OP2 and attached so that the vibrator 411 is sandwiched between the band 412 and the skin OP2c. When the first operator OP1 uses the operating means 3, the vibrator 411 attached to the second operator OP2 immediately (almost simultaneously with the operating action) receives motion information (tactile vibration signal) related to the operating action of the first operator OP1 and vibrates.

[0059] The control system 1 further includes a first computer 5, a second computer 6, a controller 7, a microcomputer 8, and the like.

[0060] Fig. 5 is an explanatory diagram showing a hardware configuration of the first computer 5. As shown in Fig. 5, the first computer 5 is made up of a CPU (Central Processing Unit) 51, a RAM (Random Access Memory) 52, a ROM (Read Only Memory) 53, a storage unit 54, a display unit 55, a communication unit 56, an input unit 57, a timer unit 58, etc.

[0061] The CPU 51 of the first computer 5 reads out various programs stored in the storage unit 54, expands them in a work area of ​​the RAM 52, and executes various processes described below in accordance with the expanded programs. The storage unit 54 stores programs executed by the CPU 51 as appropriate, data required for various processes, and the like. The storage unit 54 is composed of a physical drive such as a memory or a hard disk drive. The clock unit 58 is composed of, for example, a timer IC, a quartz oscillator, or a clock module, and has a function of clocking the current time. The CPU 51 acquires the current time from the clock unit 58 as appropriate as necessary.

[0062] The display unit 55 is, for example, a liquid crystal display, and displays necessary messages and the like to the administrator who operates the first computer 5. The input unit 57 is a user interface, and is made up of a keyboard, a pointing device, and the like, and is used when the administrator inputs information such as various data and commands into the first computer 5.

[0063] The communication unit 56 is a communication interface and has a function of transmitting information to other devices and a function of receiving information from other devices. The communication unit 56 in this embodiment has both a wireless communication function and a wired communication function. The communication unit 56 performs wired communication with the camera 312 and the like. The communication unit 56 also performs wireless communication with the microcomputer 8 and the like.

[0064] The first computer 5 also includes a control unit 510 configured with the CPU 51 etc. The control unit 510 further includes an image analysis unit 511, a motion information generation unit 512, a motion information supply unit 513, and a grip information supply unit 514.

[0065] The image analysis unit (input information generation unit) 511 executes a process of extracting input information (three-dimensional position and orientation information) corresponding to the operation action of each operator OP from the captured image acquired by the camera 312 for each operator OP. When the captured image captured by the camera 312 is transmitted to the first computer 5, a first image analysis unit 511a for the first operator OP1 of the image analysis unit 511 generates three-dimensional position and orientation information of the time series of the first rigid body based on the captured image. Also, a second image analysis unit 511b for the second operator OP2 of the image analysis unit 511 generates three-dimensional position and orientation information of the time series of the second rigid body based on the captured image. The sampling frequency of the captured image (image data) is, for example, 100 to 250 [Hz].

[0066] The image analysis unit (input information generation unit) 511 generates, for each operation means 3, input information corresponding to the operation action performed when the operator OP operates the operation means 3 (input information generation step).

[0067] The time-series three-dimensional position / posture information of the first rigid body and the time-series three-dimensional position / posture information of the second rigid body generated by the image analysis unit (input information generation unit) 511 are supplied to the motion information generation unit 512 and a motion planning unit 611 of the second computer 6, which will be described later.

[0068] The motion information generating unit 512 executes a process of generating motion information related to the corresponding operation motion for each operating means 3 (motion information generating step). Specifically, the first motion information generating unit 512a for the first operator OP1 of the motion information generating unit 512 executes a process of generating motion information (tactile vibration signal, lighting control signal) corresponding to the operation motion of the first operator OP1 based on the three-dimensional position and orientation information (input information) of the first rigid body generated by the first image analyzing unit 511a of the image analyzing unit 511. Also, the second motion information generating unit 512b for the second operator OP2 of the motion information generating unit 512 executes a process of generating motion information (tactile vibration signal, lighting control signal) corresponding to the operation motion of the second operator OP2 based on the three-dimensional position and orientation information (input information) of the second rigid body generated by the second image analyzing unit 511b of the image analyzing unit 511.

[0069] The motion information generating unit 512 generates the motion information by using physical quantities related to the operation motion of the corresponding operator OP. Examples of the physical quantities include speed, acceleration, jerk, amount of change in position, amount of change in posture, difference value between the position of the first rigid body and the position of the second rigid body, difference value between the posture of the first rigid body and the posture of the second rigid body, etc.

[0070] For example, the first movement information generation unit 512a calculates the norm (scalar quantity) of the change in position over time as velocity information v [mm / s] of the first rigid body based on the three-dimensional position and orientation information of the first rigid body, and based on that, generates movement information of the first operator OP1 as a tactile vibration signal and movement information of the first operator OP1 as a lighting control signal.

[0071] In addition, the second movement information generation unit 512b calculates the norm (scalar quantity) of the change in position over time as velocity information v [mm / s] of the first rigid body based on the three-dimensional position and orientation information of the second rigid body, and based on that, generates movement information of the second operator OP2 as a tactile vibration signal and movement information of the second operator OP2 as a lighting control signal.

[0072] The motion information supply unit 513 supplies motion information (tactile vibration signals) corresponding to the operation of the first operator OP1 to the transducer for the second operator OP2. The motion information supply unit 513 also supplies motion information (tactile vibration signals) corresponding to the operation of the second operator OP2 to the transducer 411 for the first operator OP1.

[0073] For example, the motion information supply unit 513 supplies motion information corresponding to the operation motion of the first operator OP1 to the DA converter 415 for the second operator OP2. When motion information (tactile vibration signal) corresponding to the operation motion of the first operator OP1 is supplied to the DA converter 415 for the second operator OP2, the motion information is converted into an electric signal in the DA converter 415 and sent to the vibration amplifier 414 for the second operator OP2 by a wired connection. After the signal is amplified by the vibration amplifier 414, it is supplied to the transducer 411 for the second operator OP2 via the cable (signal line) 413.

[0074] The vibrations that are amplitude modulated (AM modulated) according to the change in the value of a carrier wave (sine wave) with a frequency of 200 [Hz] are supplied as the motion information (tactile vibration signal) to the vibrator 411. In other embodiments, the vibrations that are modulated by other modulation methods (FM modulation, etc.) may be supplied as the motion information (tactile vibration signal).

[0075] In addition, when motion information (tactile vibration signal) corresponding to the operation action of the second operator OP2 is supplied to the DA converter 415 for the first operator OP1, the motion information is converted into an electric signal in the DA converter 415 and sent to the vibration amplifier 414 for the first operator OP1 by a wired connection. After the signal is amplified by the vibration amplifier 414, it is supplied to the transducer 411 for the first operator OP1 via a cable (signal line) 413.

[0076] The motion information supply unit 513 also supplies motion information (lighting control signal) corresponding to the operation of the first operator OP1 to the motion information presentation unit 41B for lighting display for the second operator OP2. The motion information supply unit 513 also supplies motion information (lighting control signal) corresponding to the operation of the second operator OP2 to the motion information presentation unit 41A for lighting display for the first operator OP1.

[0077] 6 is an explanatory diagram of the motion information presentation unit 41 for lighting display attached to the distal end arm 22. The motion information presentation unit 41 for lighting display is composed of a motion information presentation unit 41A for lighting display for the first operator OP1 and a motion information presentation unit 41B for lighting display for the second operator OP2.

[0078] The motion information presentation unit for lighting display (motion information presentation unit) 41 presents motion information (lighting control signal) regarding the operation action by any operator OP to the remaining operators OP other than the arbitrary operator OP as a lighting display (visual information) so that the operators OP can understand each other's operation actions (motion information presentation process).

[0079] The exercise information presentation unit 41A for lighting display and the exercise information presentation unit 41B for lighting display each have three LEDs 42a, 42b. Each LED 42a, 42b is arranged on the upper side 22a, the right side 22b, and the left side 22c of the distal end arm 22, one for the first operator OP1 and one for the second operator OP2. In addition, on the right side 22b and the left side 22c, as shown in FIG. 6, it is preferable that the exercise information presentation unit 41A for lighting display to be referred to by the first operator OP1 is arranged on the front side (front side), and the exercise information presentation unit 41B for lighting display to be referred to by the second operator OP2 is arranged on the rear side. In this way, by arranging the motion information presenting unit 41A for lighting display and the motion information presenting unit 41B for lighting display at positions (arrangement order) corresponding to the control objects (tip arm 22, base arm 23) for which each operator OP is in charge, each operator OP can easily intuitively grasp the motion information of the control objects (tip arm 22, base arm 23) operated by the other operator. Note that in other embodiments, the motion information presenting unit 41A for lighting display and the motion information presenting unit 41B for lighting display may be similarly arranged at positions (arrangement order) corresponding to the control objects (tip arm 22, base arm 23) for which each operator OP is in charge on the upper side 22a or lower side of the tip arm 22.

[0080] For example, the motion information (lighting control signal) corresponding to the operation of the first operator OP1 is supplied to the motion information presenting unit 41B for lighting display so that the LED 42b (e.g., the LED 42b arranged on the upper side 22a) arranged in the direction in which the base side arm 23 moves (e.g., upward) among the three LEDs 42b provided in the motion information presenting unit 41B for lighting display for the second operator OP2 is lit. The LED 42b of the motion information presenting unit 41B for lighting display for the second operator OP2 is lit for a predetermined time substantially simultaneously with the operation of the first operator OP1. Therefore, the second operator OP2 can grasp (predict) in advance the direction in which the base side arm 23 moves by the lighting of the LED 42b.

[0081] In addition, the motion information (lighting control signal) corresponding to the operation motion of the second operator OP2 is supplied to the motion information presentation unit 41A for lighting display so that the LED 42a (e.g., the LED 42a arranged on the left side surface 22c) arranged in the direction in which the tip side arm 22 moves (e.g., the left direction) among the three LEDs 42a provided in the motion information presentation unit 41A for lighting display for the first operator OP1 is lit. The LED 42a of the motion information presentation unit 41A for lighting display for the first operator OP1 is lit for a predetermined time substantially simultaneously with the operation motion of the second operator OP2. Therefore, the first operator OP1 can grasp (predict) in advance the direction in which the tip side arm 22 moves by the lighting of the LED 42a.

[0082] The microcomputer 8 is a microcontroller (e.g., "ESP32" manufactured by Espressif Systems) equipped with wireless communication and wired communication functions, and is composed of a CPU, a memory, a communication unit, etc. The microcomputer 8 is equipped with a gripper input information generation unit 81, which generates gripper input information corresponding to a change amount of a position sensor (e.g., a potentiometer) of the gripper operation means 31 based on the operation action of the gripper operation means 31 by the second operator OP2. The gripper input information generated by the gripper input information generation unit 81 is transmitted to the first computer 5 by wireless communication. The gripper input information transmitted to the first computer 5 is transmitted to the second computer 6 via the gripper information supply unit 514 of the first computer 5.

[0083] The second computer 6 has the same hardware configuration as the above-mentioned first computer 5. The second computer 6 is composed of a CPU, RAM, ROM, a storage unit, a display unit, a communication unit, an input unit, a timer unit, etc. The second computer 6 also has a control unit 610 composed of a CPU, etc. The control unit 610 further has a motion planning unit 611, a delay setting unit 612, a gripper motion planning unit 613, and a gripper delay setting unit 614.

[0084] The motion planning unit 611 executes a process of generating motion plan information for a corresponding control target based on input information so that the corresponding control target moves in accordance with the operation motion.

[0085] Specifically, the first motion planning unit 611a for the first operator OP1 of the motion planning unit 611 generates motion planning information (hereinafter sometimes referred to as "first motion planning information") for the root side arm 23, which is the corresponding control object, based on the three-dimensional position / posture information of the first rigid body, so that the root side arm 23, which is the corresponding control object, moves in accordance with the operating motion of the first operator OP1.

[0086] Further, a second motion planning unit 611b for the second operator OP2 of the motion planning unit 611 generates motion planning information (hereinafter sometimes referred to as "second motion planning information") for the tip arm 22, which is the corresponding control target, based on the three-dimensional position / posture information of the second rigid body so that the tip arm 22, which is the corresponding control target, moves in accordance with the operating motion of the second operator OP2.

[0087] In this way, the motion planning unit 611 of this embodiment generates motion plan information for a corresponding control object based on input information so that the corresponding control object moves in accordance with the operation motion (motion planning process).

[0088] The delay setting unit 612 executes a process of setting a predetermined delay time for the operation plan information (delay setting step).

[0089] Specifically, a first delay setting unit 612a for the first operator OP1 of the delay setting unit 612 sets a predetermined delay time for the first operation plan information. Also, a second delay setting unit 612b for the second operator OP2 of the delay setting unit 612 sets a predetermined delay time for the second operation plan information.

[0090] The predetermined delay time that the first delay setting unit 612a sets for the first operation plan information is equal to or greater than 150 ms and equal to or less than 300 ms.

[0091] Moreover, the predetermined set time that the second delay setting unit 612b sets for the second operation plan information is not less than 150 ms and not more than 300 ms.

[0092] In this embodiment, the delay time set for the first operation plan information and the delay time set for the second operation plan information are set to the same value (time). In other embodiments, these delay times may be different values ​​(times).

[0093] The first operation plan information generated by the first operation plan unit 611a and the delay time set for the first operation plan information are transmitted to the controller 7. In addition, the second operation plan information generated by the second operation plan unit 611b and the delay time set for the second operation plan information are transmitted to the controller 7.

[0094] The gripper operation planning unit 613 generates operation planning information for the gripper 21, which is the corresponding controlled object, based on input information from the gripper operation means 31 so that the gripper 21, which is the corresponding controlled object, operates in accordance with the operation action using the gripper operation means 31 (hereinafter, "operation planning information for the gripper").

[0095] The gripping unit delay setting unit 614 executes a process of setting a predetermined delay time for the gripping unit operation plan information. In the present embodiment, the delay time set by the gripping unit delay setting unit 614 is the same value (time) as the delay time set by the above-mentioned second delay setting unit 612b.

[0096] The gripper motion plan information generated by the gripper motion plan section 613 and the delay time set for the gripper motion plan information are transmitted to the controller 7.

[0097] The controller 7 mainly controls the movement of the robot avatar (robot arm) 2. The controller 7 is composed of a CPU, a memory, a communication unit, etc. The controller 7 includes an operation command unit 71 composed of the CPU, etc., and a gripper operation command unit 72.

[0098] The operation command unit 71 executes a process of operating the corresponding control object based on the operation plan information after a delay time has elapsed so that, before the corresponding control object operates in accordance with the operation operation by any operator OP, motion information regarding the operation operation by any operator OP is presented to the remaining operators OP in advance (operation command process).

[0099] For example, the first operation command unit 71a for the first operator OP1 of the operation command unit 71 executes a process of operating the root side arm 23 based on the first operation plan information after a delay time set for the first operation plan information has elapsed so that before the root side arm 23 operates, movement information regarding the operation movement by the first operator OP1 is presented to the second operator OP2 in advance via the tactile stimulation movement information presentation unit 4 (vibrator 411) for the second operator OP2 and the light-up display movement information presentation unit 41B for the second operator OP2 in accordance with the operation movement by the first operator OP1 using the operating means 3.

[0100] In addition, the second operation command unit 71b for the second operator OP2 of the operation command unit 71 executes a process of operating the tip-side arm 22 based on the second operation plan information after a delay time set for the second operation plan information has elapsed so that before the tip-side arm 22 operates, movement information regarding the operation movement by the second operator OP2 is presented to the first operator OP1 in advance via the tactile stimulation movement information presentation unit 4 (vibrator 411) for the first operator OP1 and the light-up display movement information presentation unit 41A for the first operator OP1 in accordance with the operation movement by the second operator OP2 using the operating means 3.

[0101] The gripper operation command unit 72 executes a process of operating the gripper 21 based on the gripper operation plan information after a delay time set for the gripper operation plan information has elapsed so that the gripper 21 operates in accordance with the operation of the gripper operation means 31 by the second operator OP2. As a result, the open / closed state of the gripper 21 is controlled in accordance with the opening / closing operation of the gripper operation means 31 operated by the second operator OP2.

[0102] In the control system 1 of the present embodiment as described above, for example, when the first operator OP1 performs an operation on the root side arm 23 using the operating means 3, the operation of the root side arm 23 corresponding to the operation starts after a predetermined delay time has elapsed. In response to this, the motion information regarding the operation is immediately (for example, within 150 ms from the start of the operation) supplied to the transducer 411 for the second operator OP2 as a haptic vibration signal, and is also supplied to the motion information presentation unit 41B (LED 42b) for the second operator OP2 for lighting display as a lighting control signal. Therefore, the second operator OP2 can predict the operation of the first operator OP1 during the delay time by the vibration stimulus from the transducer 411 and the lighting drive of the corresponding LED 42b before the root side arm 23 operates.

[0103] In the control system 1 of this embodiment, for example, when the second operator OP2 performs an operation on the distal end arm 22 using the operating means 3, the operation of the distal end arm 22 corresponding to the operation starts after a predetermined delay time has elapsed. In response to this, the motion information regarding the operation is immediately (for example, within 150 ms from the start of the operation) supplied to the transducer 411 for the first operator OP1 as a tactile vibration signal, and is also supplied to the motion information presentation unit 41A (LED 42a) for the first operator OP1 for lighting display as a lighting control signal. Therefore, the first operator OP1 can predict the operation of the second operator OP2 during the delay time by the vibration stimulus from the transducer 411 and the lighting drive of the corresponding LED 42a before the distal end arm 22 operates.

[0104] As described above, the control system 1 for cooperative operation of a robot avatar by multiple people in this embodiment can share motion information regarding each other's operating movements during the delay time, making it easier for people to perform cooperative operations with each other, and providing excellent operability of the robot avatar 2.

[0105] <Embodiment 2> Next, a control system 1A according to a second embodiment will be described with reference to FIG. 7. FIG. 7 is an explanatory diagram showing the overall configuration of the control system 1A according to the second embodiment. The control system 1A of this embodiment is a system intended for a plurality of (two) operators OP to cooperate and jointly operate one robot avatar 2 while sharing the operation among themselves and grasping each other's operation status, as in the first embodiment. In the control system 1A of this embodiment, as in the first embodiment, the first operator OP1 is in charge of operating the base side arm 23, and the second operator OP2 is in charge of operating the tip side arm 22 and the gripper 21. Therefore, in the control system 1A of this embodiment, various processing contents executed by the control unit 510, the control unit 610, etc. are basically the same as in the first embodiment.

[0106] In this embodiment, the space S1 where the first operator OP1 and the second operator OP2 are located and the space S2 where the robot avatar (robot arm) 2 is installed are far away from each other. That is, each operator OP is in a situation where they cannot directly check the robot avatar 2 with their eyes, etc. The control system 1A of this embodiment is used when a plurality of (two) operators OP remotely operate one robot avatar 2 under such a situation, etc.

[0107] In addition, among the configurations of the control system 1A of the second embodiment, the configurations similar to those of the control system 1 of the first embodiment are denoted in FIG. 7 by the same reference numerals as those of the first embodiment, and detailed description thereof will be omitted as appropriate.

[0108] In this embodiment, a camera 9 is installed in the space S2 to capture an image of the robot avatar 2. The captured image (video) of the robot avatar 2 captured by the camera 9 is transmitted to the second computer 6, and then transmitted to the first computer 5 installed in the space S1 using a communication line 14. Examples of the communication line 14 include the Internet, an Ethernet (registered trademark) line, a public line, and a dedicated line. The captured image transmitted to the first computer 5 is displayed on a display device (e.g., a liquid crystal display panel) 43 connected to the first computer 5 under the control of a display control unit 515.

[0109] Also, the image analysis unit (input information generation unit) 511 executes a process of generating, for each operator OP, input information (three-dimensional position and orientation information) corresponding to the operation motion of each operator OP from the captured image acquired by the camera 312, as in the first embodiment. The time-series three-dimensional position and orientation information of the first rigid body and the time-series three-dimensional position and orientation information of the second rigid body generated by the image analysis unit (input information generation unit) 511 are supplied to a motion information generation unit 512. Note that, in order to operate each control target of the robot avatar 2, the time-series three-dimensional position and orientation information of the first rigid body and the time-series three-dimensional position and orientation information of the second rigid body are transmitted to the control unit 610 of the second computer 6 via the communication line 14.

[0110] The motion information generation unit 512 (first motion information generation unit 512a) executes a process of generating motion information (tactile vibration signal, display control signal) corresponding to the operation motion of the first operator OP1 based on the three-dimensional position and posture information of the first rigid body.

[0111] In addition, the motion information generation unit 512 (second motion information generation unit 512b) executes a process of generating motion information (tactile vibration signal, display control signal) corresponding to the operation motion of the second operator OP2 based on the three-dimensional position and posture information of the second rigid body.

[0112] The motion information supply unit 513 immediately (for example, within 150 ms from the start of the operation motion) supplies motion information (tactile vibration signal) corresponding to the operation motion of the first operator OP1 to the transducer 411 for the second operator OP2, as in the first embodiment. Also, the motion information supply unit 513 immediately (for example, within 150 ms from the start of the operation motion) supplies motion information (tactile vibration signal) corresponding to the operation motion of the second operator OP2 to the transducer 411 for the first operator OP1.

[0113] In addition, the exercise information supply unit 513 supplies exercise information (display control signal) corresponding to the operation motion of the first operator OP1 to the image display exercise information presentation unit 141B for the second operator OP2 for the display device 43. In addition, the exercise information supply unit 513 supplies exercise information (display control signal) corresponding to the operation motion of the second operator OP2 to the image display exercise information presentation unit 141A for the first operator OP1.

[0114] The motion information presentation unit for image display 141 of this embodiment is composed of a motion information presentation unit for image display 141A for the first operator OP1 and a motion information presentation unit for image display 141B for the second operator OP2. In this embodiment, the motion information presentation unit for image display 141A and the motion information presentation unit for image display 141B each consist of a display image (e.g., an arrow image) that is shown on the display device 43 and indicates the direction in which the controlled object moves. This display image is displayed, for example, around the image of the robot avatar 2 displayed on the display screen of the display device 43 (e.g., in the corner of the display screen).

[0115] When the second operator OP2 uses the gripper operation means 31, gripper input information based on the operation action is generated by a gripper input information generating unit 81 of the microcomputer 8. The generated gripper input information is transmitted to the first computer 5, and then transmitted to the second computer 6 via the communication line 14 under the control of a gripper information supplying unit 514.

[0116] In the control system 1A of this embodiment, for example, when the first operator OP1 performs an operation on the root side arm 23 using the operating means 3, the operation of the root side arm 23 corresponding to the operation starts after a predetermined delay time has elapsed. In response to this, the motion information regarding the operation is immediately (for example, within 150 ms from the start of the operation) supplied to the transducer 411 for the second operator OP2 as a tactile vibration signal, and is also supplied to the image display motion information presenting unit 141B for the second operator OP2 as a display control signal. Therefore, during the delay time, the second operator OP2 can predict (understand) the operation of the first operator OP1 by the vibration stimulus from the transducer 411 and the display image of the corresponding image display motion information presenting unit 141B before the root side arm 23 operates. In this case, the second operator OP2 can operate the tip side arm 22 and the gripping unit 21, which are the control targets, after understanding the operation of the first operator OP1 in advance.

[0117] In the control system 1A of this embodiment, for example, when the second operator OP2 uses the operating means 3 to perform an operation on the tip-side arm 22, the operation of the tip-side arm 22 corresponding to the operation starts after a predetermined delay time has elapsed. In response to this, the motion information regarding the operation is immediately (for example, within 150 ms from the start of the operation) supplied to the transducer 411 for the first operator OP1 as a haptic vibration signal, and is also supplied to the image display motion information presenting unit 141A for the first operator OP1 as a display control signal. Therefore, during the delay time, the first operator OP1 can predict (understand) the operation of the second operator OP2 by the vibration stimulus from the transducer 411 and the corresponding image display motion information presenting unit 141A before the tip-side arm 22 moves. In this case, the first operator OP1 can operate the base side arm 23, which is the control target, after understanding the operation of the second operator OP2 in advance.

[0118] As described above, in the control system 1A for cooperative operation of a robot avatar by multiple people in this embodiment, even if the space S1 in which each operator OP is located and the space S2 in which the robot avatar 2 is installed are separated from each other, they can share motion information regarding each other's operating movements during the delay time, making it easier for them to perform cooperative operations with each other and providing excellent operability of the robot avatar 2.

[0119] <Embodiment 3> Next, a control system 1B according to a third embodiment will be described with reference to FIG. 8. FIG. 8 is an explanatory diagram showing the overall configuration of the control system 1B according to the third embodiment. As in the first embodiment, the control system 1B of this embodiment is also a system intended for a plurality of (two) operators OP to cooperate and jointly operate one robot avatar 2 while sharing the operation among themselves and grasping each other's operation status. Also, in the control system 1B of this embodiment, as in the first embodiment, a first operator OP1 is in charge of operating the base side arm 23, and a second operator OP2 is in charge of operating the tip side arm 22 and the gripper 21.

[0120] In this embodiment, a space S11 where a first operator OP1 is, a space S12 where a second operator OP2 is, and a space S13 where a robot avatar (robot arm) 2 is installed are far away from each other. That is, the operators OP are in a situation where they cannot directly check each other with their eyes, and each operator OP is in a situation where they cannot directly check the robot avatar 2 with their eyes. The control system 1B of this embodiment is used when a plurality of (two) operators OP remotely operate one robot avatar 2 under such a situation.

[0121] Among the configurations of the control system 1B of embodiment 3, configurations similar to those of the control system 1 of embodiment 1 or the control system 1A of embodiment 2 are given the same symbols in Figure 8 as those of embodiment 1 or embodiment 2, and detailed descriptions thereof will be omitted as appropriate.

[0122] In this embodiment, as in embodiment 2, a camera 9 for photographing the robot avatar 2 is installed in space S12, and the image (video) of the robot avatar 2 captured by the camera 9 is transmitted to a second computer 6, and then transmitted via a communication line 14 to a first computer 5 installed in space S11 and a third computer 10 installed in space S12.

[0123] The captured image (video) of the robot avatar 2 sent to the first computer 5 is displayed on a first display device (e.g., a liquid crystal display panel) 45 connected to the first computer 5 under the control of a display control unit 515 of the first computer 5.

[0124] In addition, the captured image (video) of the robot avatar 2 sent to the third computer 10 is displayed on a second display device (e.g., a liquid crystal display panel) 46 connected to the third computer 10 under the control of a display control unit 515A configured by the control unit 510A of the third computer 10.

[0125] In this embodiment, a set of motion capture cameras 312 is assigned to each of the first operator OP1 and the second operator OP2. The set of cameras 312 consists of multiple cameras (8 cameras).

[0126] An image analysis unit (input information generation unit) 511 constituted by the control unit 510 of the first computer 5 executes a process of generating input information (three-dimensional position and orientation information) corresponding to the operation motion of the first operator OP1 from a photographed image acquired by the camera 312 for the first operator OP1. The image analysis unit (input information generation unit) 511 of the first computer 5 generates time-series three-dimensional position and orientation information of the first rigid body, which is supplied to a motion information generation unit 512A of the third computer 10 via the communication line 14 for feedback to the second operator OP2. In addition, in order to operate the base side arm 23, the time-series three-dimensional position and orientation information of the first rigid body is transmitted to the control unit 610 of the second computer 6 via the communication line 14.

[0127] The motion information generation unit 512A of the third computer 10 executes processing to generate motion information (tactile vibration signal, display control signal) corresponding to the operation motion of the first operator OP1 based on the three-dimensional position and orientation information of the first rigid body, for use as feedback to the second operator OP2.

[0128] The motion information supply unit 513A of the third computer 10 immediately supplies motion information (tactile vibration signal) corresponding to the operation motion of the first operator OP1 to the transducer 411 for the second operator OP2 (e.g., within less than 150 ms from the start of the operation motion).

[0129] In addition, the exercise information supply unit 513A of the third computer 10 supplies exercise information (display control signal) corresponding to the operation action of the first operator OP1 to the image display exercise information presenting unit 241B for the second operator OP2 for the second display device 46.

[0130] In response to this, an image analysis unit (input information generation unit) 511A configured by a control unit 510A of the third computer 10 executes a process of generating input information (three-dimensional position and orientation information) corresponding to the operation motion of the second operator OP2 from a photographed image acquired by the camera 312 for the second operator OP2. The image analysis unit (input information generation unit) 511A of the third computer 10 generates three-dimensional position and orientation information of the second rigid body in time series, which is supplied to a motion information generation unit 512 of the first computer 5 via the communication line 14 for feedback to the first operator OP1. In addition, in order to operate the tip-side arm 22, the three-dimensional position and orientation information of the second rigid body in time series is transmitted to a control unit 610 of the second computer 6 via the communication line 14.

[0131] The motion information generation unit 512 of the first computer 5 executes processing to generate motion information (tactile vibration signal, display control signal) corresponding to the operation motion of the second operator OP2 based on the three-dimensional position and orientation information of the second rigid body, for use as feedback to the first operator OP1.

[0132] The motion information supply unit 513 of the first computer 5 immediately supplies motion information (tactile vibration signal) corresponding to the operational movement of the second operator OP2 to the transducer 411 for the first operator OP1 (e.g., within less than 150 ms from the start of the operational movement).

[0133] In addition, the exercise information supply unit 513 of the first computer 5 supplies exercise information (display control signal) corresponding to the operation action of the second operator OP2 to the image display exercise information presenter 241A for the first operator OP1 for the first display device 45.

[0134] In this embodiment, the image display motion information presenting unit 241A is made up of a display image (e.g., an arrow image) showing the moving direction of the control object (tip arm 22) operated by the second operator OP2, which is displayed on the first display device 45. That is, the display image of the image display motion information presenting unit 241A is displayed on the display screen of the first display device 45 together with the image of the robot avatar 2 captured by the camera 9. The image display motion information presenting unit 241B is made up of a display image (e.g., an arrow image) showing the moving direction of the control object (root arm 23) operated by the first operator OP1, which is displayed on the second display device 46. That is, the display image of the image display motion information presenting unit 241B is displayed on the display screen of the second display device 46 together with the image of the robot avatar 2 captured by the camera 9.

[0135] When the second operator OP2 uses the gripper operation means 31, gripper input information based on the operation is generated by a gripper input information generating unit 81 of the microcomputer 8. The generated gripper input information is transmitted to the third computer 5, and then transmitted to the second computer 6 via the communication line 14 under the control of a gripper information supplying unit 514.

[0136] In the control system 1B of this embodiment, for example, when the first operator OP1 performs an operation on the root side arm 23 using the operating means 3, the operation of the root side arm 23 corresponding to the operation starts after a predetermined delay time has elapsed. In response to this, the motion information regarding the operation is immediately (for example, within 150 ms from the start of the operation) supplied to the transducer 411 for the second operator OP2 as a tactile vibration signal, and is also supplied to the image display motion information presenting unit 241B for the second operator OP2 as a display control signal. Therefore, during the delay time, the second operator OP2 can predict (understand) the operation of the first operator OP1 by the vibration stimulus from the transducer 411 and the display image of the corresponding image display motion information presenting unit 141B before the root side arm 23 operates. In this case, the second operator OP2 can operate the tip side arm 22 and the gripping unit 21, which are the control targets, after understanding the operation of the first operator OP1 in advance.

[0137] In the control system 1A of this embodiment, for example, when the second operator OP2 uses the operating means 3 to perform an operation on the tip-side arm 22, the operation of the tip-side arm 22 corresponding to the operation starts after a predetermined delay time has elapsed. In response to this, the motion information regarding the operation is immediately (for example, within 150 ms from the start of the operation) supplied to the transducer 411 for the first operator OP1 as a haptic vibration signal, and is also supplied to the image display motion information presenting unit 241A for the first operator OP1 as a display control signal. Therefore, during the delay time, the first operator OP1 can predict (understand) the operation of the second operator OP2 by the vibration stimulus from the transducer 411 and the display image of the corresponding image display motion information presenting unit 241A before the tip-side arm 22 operates. In this case, the first operator OP1 can operate the base side arm 23, which is the control target, after understanding the operation of the second operator OP2 in advance.

[0138] As described above, in the control system 1B of this embodiment for cooperative operation of a robot avatar by multiple people, even if the space S11 where the first operator OP1 is, the space S12 where the second operator OP2 is, and the space S13 where the robot avatar 2 is installed are separated from each other, they can share motion information regarding each other's operating actions during the delay time, making it easy to perform cooperative operations with each other and providing excellent operability of the robot avatar 2.

[0139] <Other embodiments> The present invention is not limited to the embodiments described above and illustrated in the drawings, and the following embodiments, for example, are also included within the technical scope of the present invention.

[0140] (1) In the above embodiment 1 etc., visual information such as vibration stimulation, lighting display, image display, etc. was presented to each operator OP as movement information related to the operation movement. However, the present invention is not limited to this. For example, auditory information such as sound may be presented to each operator OP.

[0141] (2) In the above embodiment 1 etc., two types of motion information related to the operating movement were presented to each operator OP (vibration stimulation, light display, etc.), but the present invention is not limited to this. For example, one type of information may be fed back to each operator OP, or three or more types of information may be fed back to each operator OP.

[0142] (3) When motion capture is used as an operation means for operating a robot avatar, it is not limited to the optical motion capture exemplified in embodiment 1. In other embodiments, other types of motion capture, such as magnetic, mechanical, inertial sensor, and image recognition, may be used as long as the object of the present invention is not impaired. In addition, the optical motion capture may be, for example, a type in which an operator holds a specific controller in his / her hand and captures light (infrared light) from an infrared LED provided on the controller with a camera.

[0143] (4) As an operating means for operating the robot avatar, for example, a glove-type sensor worn on the operator's hand (high-performance data glove, "CyberGlove", manufactured by CyberGlove Systems) may be used.

[0144] (5) In the above embodiment 1 etc., the grip portion was operated using a grip portion operation means 31 having a position sensor (e.g., a potentiometer). However, in other embodiments, the grip portion may be operated using other operation means such as motion capture.

[0145] (6) In cases where an operator has a physical disability, the parts of the body that the operator can move and the range (distance) that the operator can move may be limited. For this reason, the operation means may be a mechanical operation device (push-type switch, joystick, etc.), keyboard, eye tracker, etc. that can be operated by the operator with limited movement.

[0146] (7) In another embodiment, motion information regarding the manipulation action of the gripping portion may be fed back to another operator as vibration stimulation or the like.

[0147] (8) For example, a virtual display of movement information (e.g., an image of an arrow displayed in a virtual space) may be superimposed on the real space in which the robot avatar 2 is installed using AR (Augmented Reality) technology, allowing operators OP to share each other's movement information through the virtual display.

[0148] (9) In the above-mentioned embodiment 1 and the like, a vibrator (tactile stimulation motion information presentation unit) for grasping the operation motion (motion of the tip side arm 22) of the second operator OP is attached to the wrist (right wrist) of the first operator OP, and a vibrator (tactile stimulation motion information presentation unit) for grasping the operation motion (motion of the base side arm 23) of the first operator OP is attached to the wrist (right wrist) of the second operator OP, but the present invention is not limited to this. For example, a vibrator for grasping the operation motion (motion of the tip side arm 22) of the second operator OP may be attached to the wrist of the first operator OP, and a vibrator for grasping the operation motion (motion of the base side arm 23) of the first operator OP may be attached to the upper arm (right upper arm) of the second operator OP. In other words, a vibrator may be assigned to a position corresponding to the control object (tip side arm 22, base side arm 23) that each operator OP is in charge of. By assigning a tactile stimulation motion information presentation unit (vibrator, etc.) to such a position, the operators OP can more easily grasp the operation motion of the other party. [Explanation of symbols]

[0149] 1...robot avatar control system, 2...robot avatar (robot arm), 21...grasping unit, 22...tip arm (control target), 23...base arm, 3...operation means, 311...marker, 312...camera, 4...tactile stimulation motion information presentation unit (motion information presentation unit), 411...vibrator, 41...lighting display motion information presentation unit (motion information presentation unit), 5...first computer, 6...second computer, 7...controller, 8...microcomputer

Claims

1. A robot avatar having a plurality of control objects, each of which is operable; a plurality of operation means each assigned to a plurality of operators so that the plurality of operators can share the task of operating the plurality of control objects, and each operating the plurality of control objects; an input information generating unit that generates, for each of the operation means, input information corresponding to an operation action performed by the operator when operating the operation means; an operation planning unit that generates operation plan information for the corresponding control object based on the input information so that the corresponding control object operates in accordance with the operation operation; a delay setting unit that sets a predetermined delay time for the operation plan information; a motion information generating unit configured to generate motion information related to the corresponding operation motion for each of the operation means; an exercise information presentation unit that presents the exercise information related to the operation action by a given operator to other operators other than the given operator so that the operators can understand each other's operation actions; and an action command unit that operates the corresponding control object based on the action plan information after the delay time has elapsed so that the motion information regarding the operation action by the arbitrary operator is presented to the remaining operators in advance before the corresponding control object operates in accordance with the operation action by the arbitrary operator.

2. 2. The control system for cooperative operation of robot avatars by multiple operators according to claim 1, wherein the delay time is between 150 ms and 300 ms.

3. The control system for collaborative operation of robot avatars by multiple people as described in claim 1 or claim 2, wherein the movement information presentation unit presents the movement information regarding the operation motion by any operator to the remaining operators other than the any operator within less than 150 ms from the start of the operation motion.

4. 3. The control system for cooperative operation of robot avatars by a plurality of operators according to claim 1, wherein the motion information presentation unit has a vibrator that presents a tactile stimulus corresponding to the motion information to the operator.

5. The control system for collaborative operation of robot avatars by multiple people as described in claim 1 or claim 2, wherein the movement information presentation unit drives lighting corresponding to the movement information or displays an image corresponding to the movement information so that the operator can see it.

6. 3. The control system for cooperative operation of robot avatars by a plurality of operators according to claim 1, wherein the motion information generating unit generates the motion information by utilizing a physical quantity related to the operation motion.

7. 3. The control system for cooperative operation of robot avatars by a plurality of operators according to claim 1, wherein the controlled object includes a robot arm with multiple degrees of freedom.

8. A control method for allowing a plurality of people to cooperatively operate a single robot avatar having a plurality of control objects, each of which is operable, comprising the steps of: an operation process in which a plurality of operation means respectively assigned to a plurality of operators are operated to operate the plurality of control objects in a shared manner by the plurality of operators; an input information generating step of generating, for each of the operation means, input information corresponding to an operation action performed by the operator when operating the operation means; an action planning step of generating action plan information for the corresponding control object based on the input information so that the corresponding control object moves in accordance with the operation action; a delay setting step of setting a predetermined delay time for the operation plan information; a motion information generating step of generating motion information relating to the corresponding operation motion for each of the operation means; an exercise information presenting step of presenting the exercise information related to the operation action by a given operator to remaining operators other than the given operator so that the operators can understand each other's operation actions; and an operation command process of operating the corresponding control object based on the operation plan information after the delay time has elapsed so that the motion information regarding the operation action by the arbitrary operator is presented to the remaining operators in advance before the corresponding control object operates in accordance with the operation action by the arbitrary operator.

9. The control method according to claim 8 , wherein the delay time is not less than 150 ms and not more than 300 ms.

10. The control method according to claim 8 or claim 9, wherein in the motion information presenting step, the motion information regarding the operation motion by a given operator is presented to remaining operators other than the given operator within less than 150 ms from the start of the operation motion.

Images