Robot remote operation control system, robot remote operation control device, robot remote operation control method, and program

Abstract

This robot remote operation control system comprises: three or more end effectors in a robot; a movement mechanism that moves each of the end effectors; a first control unit that controls an operation of at least one of the end effectors in accordance with motion by an operator; an operation designation unit that determines an action start point of the end effector on the basis of the motion by the operator and designates an operation from the action start point of the end effector; a path generation unit that generates a restricted path of the end effector from the operation which has been designated by the operation designation unit; and a second control unit that operates, on the path, at least one end effector among end effectors which are not controlled by the first control unit.

Description

Robot remote operation control system, robot remote operation control device, robot remote operation control method, and program

[0001] The present invention relates to a robot remote operation control system, a robot remote operation control device, a robot remote operation control method, and a program. This application claims priority to Japanese Patent Application Nos. 2024-046088 and 2024-046095, filed on March 22, 2024, the contents of which are incorporated herein by reference.

[0002] For example, there is technology in which robots with three or more arms work together to assemble on a factory line, or in which medical robots perform surgery by remotely switching between multiple arms (see, for example, Patent Document 1).

[0003] Furthermore, for example, when remotely controlling a robot arm, depending on the object to be controlled, it may be necessary to use three or more robot arms for remote control (see, for example, Patent Document 1). One example is the Da Vinci surgical robot, which has three arms and one camera, and the arms are controlled by two hand controllers operated by stepping on foot pedals.

[0004] When you want to remotely control an object around it, it can be difficult to do so with just one viewpoint and two arms due to occlusion and lack of reachability. In such cases, you can install additional cameras and arms and switch between the image and the control arm.

[0005] Japanese Patent Application Laid-Open No. 2019-107721

[0006] However, conventional systems that control multiple arms fully automatically have great difficulty in manipulating flexible objects. Furthermore, with conventional systems, switching between arms remotely results in confusion and time loss, and the system cannot handle tasks that require the simultaneous movement of three or more arms. Thus, with conventional systems, remote operation of three or more arms, each with an end effector, is difficult and inefficient.

[0007] Furthermore, with conventional technology, simply switching the view would result in the left and right directions being reversed when trying to operate an arm based on a video image seen from behind, making remote control difficult. Furthermore, switching can confuse the operator about which way to move the arm. Thus, with conventional technology, when there are three or more arms, intuitive operation is often difficult, making remote control difficult.

[0008] The aspects of the present invention have been made in consideration of the above-mentioned problems, and one of the objects is to provide a robot remote operation control system, a robot remote operation control device, a robot remote operation control method, and a program that can improve workability compared to conventional methods when remotely operating three or more arms, each having an end effector.

[0009] The aspects of the present invention have been made in consideration of the above-mentioned problems, and one of the objects is to provide a robot remote operation control system, a robot remote operation control device, a robot remote operation control method, and a program that can be operated intuitively and make remote control easy in the case of three or more arms.

[0010] In order to solve the above problems, the present invention employs the following aspects: (1) A robot remote operation control system according to one aspect of the present invention is a robot remote operation control system that recognizes a movement of an operator and transmits the movement of the operator to a robot to operate the robot, the robot including three or more end effectors, a movement mechanism that moves each of the end effectors, a first control unit that controls the movement of at least one of the end effectors in accordance with the movement of the operator, a movement designation unit that determines a movement starting point of the end effector based on the movement of the operator and designates a movement of the end effector from the movement starting point, a trajectory generation unit that generates a constrained trajectory of the end effector from the movement designated by the movement designation unit, and a second control unit that operates at least one of the end effectors not controlled by the first control unit on the trajectory.

[0011] (2) In the above aspect (1), when there are two gripping points at which the object to be operated is grasped based on the movement of the operator, the operation designation unit may set one of the two gripping points as the action starting point and the other of the two gripping points as the action end point, and the trajectory generation unit may set a constrained trajectory of the end effector between the action starting point and the action end point.

[0012] (3) In the above aspect (1) or (2), the trajectory generation unit may generate a constrained trajectory by preventing movement in at least one of a plurality of axial directions based on the target object and the work content estimated based on the movement of the operator.

[0013] (4) In any one of the above aspects (1) to (3), an intention estimation unit may be provided that estimates a target object and a task content based on the movement of the operator, and the first control unit and the second control unit may control the end effectors to be controlled, respectively, using the estimation results obtained by the intention estimation unit.

[0014] (5) In the above aspect (4), the first control unit may be configured to control the operation of the end effector so as to support the operation of the operator based on the operation intention estimated by the intention estimation unit.

[0015] (6) A robot remote operation control device according to one aspect of the present invention is a remote operation control device that remotely controls a robot having three or more end effectors and movement mechanisms that move each of the end effectors, and includes: a first control unit that controls the operation of at least one of the end effectors in accordance with the movement of an operator; an operation designation unit that determines a starting point of action for the end effector based on the movement of the operator and designates the operation of the end effector from the starting point of action; a trajectory generation unit that generates a constrained trajectory for the end effector from the operation designated by the operation designation unit; and a second control unit that operates at least one of the end effectors that are not controlled by the first control unit on the trajectory.

[0016] (7) A robot remote control control method according to one aspect of the present invention is a control method for a remote control device that remotely controls a robot having three or more end effectors and a movement mechanism for moving each of the end effectors, the robot remote control control method comprising: a first control unit that controls the movement of at least one of the end effectors in accordance with the movement of an operator; a movement designation unit that determines a starting point of action for the end effector based on the movement of the operator and designates the movement of the end effector from the starting point of action; a trajectory generation unit that generates a constrained trajectory for the end effector from the movement designated by the movement designation unit; and a second control unit that operates at least one of the end effectors that is not controlled by the first control unit on the trajectory.

[0017] (8) A program according to one aspect of the present invention is a program that causes a computer of a remote control device that remotely controls a robot having three or more end effectors and movement mechanisms that move each of the end effectors to control the movement of at least one of the end effectors in accordance with the movement of an operator, determines a starting point of action for the end effector based on the movement of the operator, specifies the movement of the end effector from the starting point of action, generates a constrained trajectory for the end effector from the specified movement, and moves at least one of the end effectors other than the end effector controlled in accordance with the movement of the operator on the trajectory.

[0018] (9) A robot remote operation control system according to one aspect of the present invention is a robot remote operation control system that recognizes the movements of an operator, transmits the movements of the operator to a robot, and operates the robot. The robot includes a plurality of end effectors and a movement mechanism that moves the end effectors. The robot remote operation control system also includes a plurality of environmental sensors that acquire information about the surrounding environment, a first selection unit that selects the end effector that transmits the movements of the operator, a second selection unit that selects the environmental sensor in accordance with the end effector selected by the first selection unit, and a setting unit that, when the end effector is changed, switches a coordinate system based on the environmental sensors so that the movement direction of the end effector and the movement of the operator are aligned.

[0019] (10) In the above aspect (9), the number of selected end effectors may be two, and the environmental sensor may be installed between the two end effectors and at a position that acquires information about the operable area of ​​the two selected end effectors.

[0020] (11) In the above aspect (9) or (10), the second selection unit may select the environmental sensor in accordance with the operator's hand operating and viewpoint, and the setting unit may set the coordinate system in accordance with the operator's hand operating and viewpoint.

[0021] (12) In any one of the above aspects (9) to (11), the present invention may further include a control unit that calculates the hand coordinates of the end effector based on the coordinate system switched by the setting unit and controls the end effector.

[0022] (13) In the above aspect (12), an intention estimation unit may be provided that estimates a target object and a task content based on the movement of the operator, and the control unit may control each of the end effectors to be controlled using the estimation results obtained by the intention estimation unit.

[0023] (14) In the above aspect (13), the control unit may be configured to control the operation of the end effector so as to support the operation of the operator based on the operation intention estimated by the intention estimation unit.

[0024] (15) A robot remote operation control device according to one aspect of the present invention is a control device that recognizes the movements of an operator and transmits the movements of the operator to a robot having a plurality of end effectors and a movement mechanism for moving the end effectors to operate the robot, and is equipped with a plurality of environmental sensors that acquire information about the surrounding environment, a first selection unit that selects the end effectors that transmit the movements of the operator, a second selection unit that selects the environmental sensor depending on the end effector selected by the first selection unit, and a setting unit that switches a coordinate system based on the environmental sensor when the end effector is changed so that the movement direction of the end effector and the movement of the operator are aligned.

[0025] (16) A robot remote operation control method according to one aspect of the present invention is a control method for a control device that recognizes the movements of an operator and transmits the movements of the operator to a robot having a plurality of end effectors and a movement mechanism for moving the end effectors to operate the robot, wherein an acquisition unit acquires information about the surrounding environment acquired by a plurality of environmental sensors, a first selection unit selects the end effector that transmits the movements of the operator, a second selection unit selects the environmental sensor according to the end effector selected by the first selection unit, and a setting unit switches a coordinate system based on the environmental sensor when the end effector is changed.

[0026] (17) A program according to one aspect of the present invention is a program that recognizes the movements of an operator, transmits the movements of the operator to a robot having a plurality of end effectors and a movement mechanism for moving the end effectors, and causes a computer of a control device that operates the robot to acquire information about the surrounding environment obtained by a plurality of environmental sensors, selects the end effector that will transmit the movements of the operator, selects the environmental sensor according to the selected end effector, and, when the end effector is changed, switches a coordinate system based on the environmental sensor so that the movement direction of the end effector and the movement of the operator are aligned.

[0027] According to the above aspects (1) to (8), when three or more arms each having an end effector are remotely controlled, workability can be improved compared to conventional methods.

[0028] According to the above aspects (9) to (17), in the case of three or more arms, it is possible to intuitively operate the device and to easily remotely control it.

[0029] 1 is a diagram showing a schematic configuration example of a robot remote operation control system according to an embodiment. FIG. 2 is a diagram showing a configuration example of a robot remote operation control system according to an embodiment. FIG. 3 is a flowchart of processing of a remote operation control device according to an embodiment. FIG. 4 is a diagram for explaining a first example. FIG. 5 is a flowchart of processing in the first example. FIG. 6 is a diagram for explaining a second example. FIG. 7 is a flowchart of processing in the second example. FIG. 8 is a diagram showing a schematic configuration example of a robot remote operation control system according to an embodiment. FIG. 9 is a diagram showing an example of switching of a robot, a coordinate system, and an environmental sensor according to an embodiment. FIG. 10 is a diagram showing an example of a viewpoint image by a first environmental sensor before switching. FIG. 11 is a diagram showing an example after switching according to an embodiment. FIG. 12 is a diagram showing an example after switching in the prior art. FIG. 13 is a flowchart of processing of a remote operation control device according to an embodiment.

[0030] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component has been appropriately changed so that each component can be recognized. In all drawings used to explain the embodiments, components having the same function are designated by the same reference numerals, and repeated explanations will be omitted. In addition, "based on XX" in this application means "based on at least XX" and includes cases where the component is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where the component is based on XX after calculation or processing. "XX" is any element (for example, any information).

[0031] [Overview] Fig. 1 is a diagram showing a schematic configuration example of a robot remote operation control system according to this embodiment. The robot remote operation control system 1 includes, for example, a first robot 2-1, a second robot 2-2, a third robot 2-3, a remote operation control device 3 (robot remote operation control device), and an environmental sensor 4. The nth robot 2-n (n is an integer between 1 and 3) includes, for example, a movement mechanism 21-n and an end effector 22-n. Note that one robot 2 may include three movement mechanisms and three end effectors. Alternatively, the first robot may include two movement mechanisms and two end effectors, and the second robot may include one movement mechanism and one end effector. Furthermore, the number of robots each having a combination of a movement mechanism and an end effector may be four or more.

[0032] In this embodiment, for example, in a combination of three sets of arms and end effectors, two sets are controlled based on remote control instructions, and the remaining set is automatically controlled based on the operation content and operation intention. This allows this embodiment to efficiently perform tasks that are difficult to automate.

[0033] [Configuration of Robot Remote Operation Control System] Fig. 2 is a diagram showing an example of the configuration of a robot remote operation control system according to this embodiment. As shown in Fig. 2, the robot remote operation control system 1 includes, for example, a first robot 2-1, a second robot 2-2, a third robot 2-3, ..., a remote operation control device 3, an environmental sensor 4, an operation input unit 5, and an image display device 6.

[0034] The robots 2 (first robot 2-1, ..., nth robot-n) each include, for example, a moving mechanism 21 (21-1, ..., 21-n), an end effector 22 (22-1, ..., 22-n), a sensor 23 (23-1, ..., 23-n), an actuator 24 (24-1, ..., 24-n), a drive unit 25 (25-1, ..., 25-n), and a communication unit 26 (26-1, ..., 26-n). Note that the configurations of the robots 2-n may be the same or different. The remote operation control device 3 includes, for example, an acquisition unit 31, an action designation unit 32, a trajectory generation unit 33, a first control unit 34, a second control unit 35, a memory unit 36, a communication unit 37, an image generation unit 38, and an intention estimation unit 39. The environmental sensor 4 includes, for example, an imaging device 41 and a communication unit 42. The robot 2, the remote control device 3, the environmental sensor 4, the operation input unit 5, and the image display device 6 are equipped with a power supply and the like (not shown).

[0035] The operation input unit 5 detects operation input information from the operator and outputs it to the remote operation control device 3. The operation input unit 5 is, for example, a data glove. The operation input information includes information such as the position of each hand, the angle of the finger joints, the position of the wrist, and the angle of the wrist joint.

[0036] The image display device 6 displays the display image required for remote operation generated by the image generation unit 38 of the remote operation control device 3. The image display device 6 is, for example, an HMD (head mounted display). The image display device 6 may also include a line of sight detection device that detects the line of sight of the operator.

[0037] (Environment Sensor) The image capturing device 41 is, for example, an RGB (red-green-blue)-D image capturing device that can also acquire depth D information. The image capturing device 41 may also be, for example, an RGB image capturing device and a distance sensor. The communication unit 42 outputs environmental data (image data, distance data) captured by the image capturing device 41 to the remote control device 3.

[0038] (Robot) The moving mechanism 21 is, for example, an arm of the robot 2, and one end is connected to the end effector 22 via a joint, and the other end is connected to the body or a base, etc., via a joint. The moving mechanism 21 may also include, for example, a cart. Each joint is equipped with a sensor 23 and an actuator.

[0039] The end effector 22 has, for example, two or more fingers. The end effector 22 may be, for example, a multi-fingered hand having three or more fingers, or may be a gripper or the like. The fingers have joints, and each joint has a sensor 23 and an actuator.

[0040] The sensor 23 is, for example, an encoder, a force sensor, a pressure sensor, a six-axis sensor, or the like, and is attached to each joint of the moving mechanism 21 and the end effector 22 .

[0041] The actuator 24 is driven in response to the output of the drive unit 25 and is attached to each joint of the moving mechanism 21 and the end effector 22 .

[0042] The driving unit 25 drives the actuator 24 in response to a control instruction or an operation instruction from the remote control device 3, thereby causing the movement mechanism 21 and the end effector 22 to perform operations. Note that the control instruction is an instruction based on an instruction from the operator. Also, the operation instruction is an instruction to automatically operate the movement mechanism 21 and the end effector 22 in accordance with the trajectories of the movement mechanism 21 and the end effector 22 that are operated based on the operation instruction.

[0043] The communication unit 26 adds identification information for identifying the robot 2 to the detection value detected by the sensor 23 and outputs the result to the remote control device 3. The communication unit 26 acquires a control instruction or an operation instruction from the remote control device 3.

[0044] (Remote Operation Control Device) The acquisition unit 31 acquires operation input information from the operation input unit 5. The acquisition unit 31 acquires environmental data from the environmental sensor 4. The acquisition unit 31 outputs the detection values ​​detected by each sensor 23 from each robot 2.

[0045] The intention estimation unit 39 estimates, for example, the object to be worked on and the intended work of the operator based on the operation input information and the environmental data. Note that the intention estimation is performed using, for example, a method described in Japanese Patent Laid-Open No. 2022-157101.

[0046] The action designation unit 32 determines the action starting point of the end effector 22 based on the operation input information, and designates the action from the action starting point of the end effector 22. Note that the action designation unit 32 may also use the result of estimation by the intention estimation unit 39 to determine the action starting point of the end effector 22 and designate the action from the action starting point of the end effector 22.

[0047] The trajectory generation unit 33 generates a constrained trajectory for the end effector 22 using the motion specified by the motion specification unit 32. The constraints are imposed, for example, by fixing the motion in at least one of the x-, y-, and z-axis directions or at least one of the rotational axis directions (roll, pitch, and yaw). This allows for stable grasping of an object and stable work. The constraints on the moving mechanism 21 and the end effector 22 may be imposed based on the estimated work content and the object being investigated, for example, using the method described in Japanese Patent Application No. 2023-219221.

[0048] The first control unit 34 determines the robot whose operation is to be controlled in accordance with the operation input information, for example, based on the operation input information and the estimated task content and intention. For example, when performing a task using two sets of movement mechanisms 21 and end effectors 22, the first control unit 34 determines the robot whose operation is to be controlled in accordance with the operation input information based on the distance between the object to be operated and the end effector 22, the positional relationship between the end effector 22 and its surroundings (walls and other objects), etc. The first control unit 34 generates a first operation command based on the operation input information and the estimated task content and intention, and outputs the generated first operation command to the determined robot 2 via the communication unit 37. Note that the first control unit 34 may control the robot 2 to support (assist) the operator's operation based on the estimated operation intention, for example, using a method described in Japanese Patent Application No. 2023-045616.

[0049] The second control unit 35 generates a second operation instruction to operate at least one end effector 22 other than the robot 2 determined by the first control unit 34 on the trajectory generated by the trajectory generation unit 33. The second control unit 35 generates the second operation instruction based on, for example, the task content and operation intention of the operator estimated by the intention estimation unit 39, and the operation target object.

[0050] The storage unit 36 ​​stores programs, mathematical formulas, thresholds, predetermined values, identification information of the robot 2, etc. used by each unit of the remote control control device 3.

[0051] The communication unit 37 outputs the first operation instruction generated by the first control unit 34 to, for example, the first robot 2-1 and the second robot 2-2. The communication unit 37 outputs the second operation instruction generated by the second control unit 35 to, for example, the third robot 2-3.

[0052] The image generating unit 38 generates a display image required for remote control to be displayed on the image display device 6 using the environmental data.

[0053] [Processing Procedure] Next, an example of the processing procedure will be described. Fig. 3 is a flowchart showing the processing of the remote control device in this embodiment.

[0054] (Step S1 ) The acquisition unit 31 acquires operation input information from the operation input unit 5 .

[0055] (Step S2) The acquisition unit 31 acquires environmental data from the environmental sensor 4.

[0056] (Step S3) The intention estimation unit 39 estimates, for example, the object to be worked on and the work content intended by the operator based on the operation input information and the environmental data.

[0057] (Step S4) The first control unit 34 determines the robot 2 to be controlled by the first operation instruction, for example, based on the operation input information and the result of estimation by the intention estimation unit 39. The second control unit 35 determines the robot 2 to be controlled by the second operation instruction from among the robots 2 other than the robot 2 determined by the first control unit 34, for example, based on the result of estimation by the intention estimation unit 39.

[0058] (Step S5) The operation designation unit 32, for example, based on the operation input information, determines a gripping point of the end effector 22 relative to the object. Based on the determined gripping point, the operation designation unit 32 determines a starting point of action of the end effector 22 controlled by the second operation instruction, and designates an operation of the end effector 22 from the starting point of action.

[0059] (Step S6) The trajectory generating unit 33 uses the motion designated by the motion designating unit 32 to generate a constrained trajectory of the end effector 22 controlled by the second motion instruction.

[0060] (Step S7) The first control unit 34 generates the determined first operation instruction for the robot 2 based on the input operation information and the estimated operation intention, etc.

[0061] (Step S8) The second control unit 35 generates a second operation instruction for at least one end effector 22 other than the one determined by the first control unit 34 based on the trajectory generated by the trajectory generation unit 33.

[0062] (Step S9) The first control unit 34 outputs the first operation instruction to the robot 2 that it controls via the communication unit 37. The second control unit 35 outputs the second operation instruction to the robot 2 that it controls via the communication unit 37.

[0063] The first control unit 34 and the second control unit 35 may switch between the robot 2 controlled by the first operation instruction and the robot controlled by the second operation instruction during work.

[0064] (First Example) A first example will be described with reference to FIG. 4. FIG. 4 is a diagram for explaining the first example. In FIG. 4, reference symbol g11 denotes a first object, and reference symbol g12 denotes a second object. The second object is, for example, a string that binds the first object g11. In this example, the second robot 2-2 and the third robot 2-3 are controlled by a first operation instruction in accordance with operation input information, and the operation of the first robot 2-1 is automatically controlled by a second operation instruction. That is, in the first example, two robots are remotely controlled, and the remaining robot is automatically controlled.

[0065] In this example, the intention estimation unit 39 estimates that the task is to cut the object g11 with the tool g15. The grip point may be input or selected by the operator.

[0066] The processing procedure of the work in Fig. 4 will be described with reference to Fig. 4 and Fig. 5. Fig. 5 is a flowchart of the processing in the first embodiment.

[0067] (Step S11) The operator operates the operation input units 5 of both hands to input an input to grasp the object g11. Next, based on the operation input information, the first control unit 34 determines the robots to be used for the task as the first robot 2-1 and the second robot 2-2, because the task is performed using two end effectors 22. Next, based on the operation input information, the first control unit 34 determines the grasping points (points g12 and g13 in FIG. 4) of the first object g11, and grasps the object g11. The intention estimation unit 39 estimates the operation intention and the object based on the operation input information and the acquired environmental information.

[0068] (Step S12) The first control unit 34 pulls the object g11 to both sides with an appropriate force at the position where it is grasped using the support function, based on, for example, the operation input information, the estimated operation intention, and the first object.

[0069] (Step S13) The first control unit 34 generates a first operation instruction to move the first object g11 to a position where it is easy to cut the object g11 while pulling the object g11 to both sides, and controls the corresponding first robot 2-1 and second robot 2-2.

[0070] (Step S14) After the object g11 has moved to a position where it is easy to cut, the action specifying unit 32 determines the start point g12 (action starting point) and end point g13 (action end point) of the trajectory from the grip point. Note that the grip point, the start point, and the end point may be the same or different. In this way, when there are two grip points, the action specifying unit 32 may set one of the two grip points as the start point and the other of the two grip points as the end point.

[0071] (Step S15) The trajectory generating unit 33 generates a trajectory g14 from the start point g12 and end point g13 determined by the operation specifying unit 32. Next, the second control unit 35 generates a second operation instruction based on the generated trajectory g14 and controls the corresponding operation of the robot 2. Specifically, the second control unit 35 controls the end effector 22-1 of the first robot 2-1 to grasp a tool (for example, a cutter or a knife) g15 and cut the object g11.

[0072] The information about the operation target object may be stored in advance in the storage unit 36, or may be selected or input by operating the operation input unit 5 or the like. Alternatively, the first control unit 34 or the like may acquire the information about the operation target object based on the detection value of the sensor 23 provided in the end effector 22.

[0073] Second Example A second example will be described with reference to FIG. 6. FIG. 6 is a diagram for explaining the second example. The example in FIG. 6 is an example of a task in which three robots 2 are used to unfold an object and place it over another object, for example. In this case, the operator first operates the operation input units 5 of both hands to determine the gripping points of the two robots 2, and then operates the operation input unit 5 of one hand to determine the gripping point of the remaining robot 2, and then performs the operation.

[0074] The processing procedure of the work in Fig. 6 will be described with reference to Fig. 6 and Fig. 7. Fig. 7 is a flowchart of the processing in the second embodiment.

[0075] (Step S21) The operator operates the operation input units 5 with both hands to input an instruction to grasp the object g21 using, for example, markers g22 and g23 as landmarks. The operator operates the operation input unit 5 with one hand to input an instruction to grasp the object g21 using, for example, the markers as landmarks. Based on this, for example, the first control unit 34 controls the first robot 2-1 and the second robot 2-2 based on the operation input information to grasp both sides of the object g21. Furthermore, the second control unit 35 controls the third robot 2-3 based on the operation input information to grasp the object g21. Note that the first control unit 34 and the second control unit 35 may automatically grasp three points on the object g21 using the markers as landmarks. The intention estimation unit 39 estimates the operation intention and the object based on the operation input information and the acquired environmental information.

[0076] (Step S22) After the processing of step S21, the first control unit 34 controls, for example, the first robot 2-1 and the second robot 2-2 based on operation input information from the operator so that they move along trajectories restricted to the direction in which they pull the object g22. In other words, the two robots 2 are controlled based on remote operation.

[0077] (Step S23) Simultaneously with the processing of step S22, the second control unit 35 controls, for example, the third robot 2-3 to move in a direction in which the object g21 is pulled and spread.

[0078] (Step S24) The operator operates the operation input units 5 with both hands, for example, to place the object g22 over another object. In response to this, the first control unit 34 controls the first robot 2-1 and the second robot 2-2 to place them over the other object. Furthermore, the second control unit 35 controls the third robot 2-3 to operate on a trajectory that will place it over the other object.

[0079] 4 to 6 are merely examples and are not intended to be limiting. For example, if there are four or more robots 2, the first control unit 34 and the second control unit 35 may select a robot 2 to be used for each task, and perform the task using, for example, three of the four robots 2.

[0080] As described above, in this embodiment, for example, two sets of moving mechanisms 21 and end effectors are remotely controlled in accordance with instructions from an operator, and the remaining set of moving mechanisms 21 and end effectors are remotely controlled, so that they operate according to the trajectory of movement.

[0081] As a result, according to this embodiment, when remotely operating three or more arms each having an end effector, workability can be improved compared to conventional methods. Also, according to this embodiment, tasks for which automatic trajectory generation is difficult, such as flexible object manipulation, can be performed by a human remotely, and by automatically generating motions for constrained workpieces, the reproducibility of operations can be improved.

[0082] 8 is a diagram showing a schematic configuration example of a robot remote operation control system according to another embodiment. The robot remote operation control system 101 includes, for example, a first robot 102-1, a second robot 102-2, a third robot 102-3, a remote operation control device 103 (robot remote operation control device), a first environmental sensor 104-1, a second environmental sensor 104-2, and a third environmental sensor 104-3.

[0083] The n-th robot 2-n (n is an integer from 1 to 3) includes, for example, a moving mechanism 121-n (e.g., an arm) and an end effector 122-n. Note that one robot 102 may include three moving mechanisms and three end effectors. Alternatively, the first robot may include two moving mechanisms and two end effectors, and the second robot may include one moving mechanism and one end effector.

[0084] In the example of Fig. 8, the operator operates the input operation unit (Fig. 9) to operate the robot 102, thereby causing the robot 102 to perform a task on an object obj to be operated. In operation, the operator switches the robot 102 to be used. In such a case, in this embodiment, for example, in a combination of three sets of movement mechanisms and end effectors, not only are the movement mechanisms and environmental sensors to be operated switched, but the coordinates of the operator are also switched in accordance with the object to be operated.

[0085] [Configuration of Robot Remote Operation Control System] Fig. 9 is a diagram showing an example of the configuration of a robot remote operation control system according to this embodiment. As shown in Fig. 9, the robot remote operation control system 101 includes, for example, a first robot 102-1, a second robot 102-2, a third robot 102-3, ..., a remote operation control device 103, an environment sensor 104, an operation input unit 105, and an image display device 106.

[0086] The robots 102 (first robot 102-1, ..., n-th robot-n) include, for example, moving mechanisms 121 (121-1, ..., 121-n), end effectors 122 (122-1, ..., 122-n), sensors 123 (123-1, ..., 123-n), actuators 124 (124-1, ..., 124-n), drive units 125 (125-1, ..., 125-n), and communication units 126 (126-1, ..., 126-n). Note that the configurations of the robots 102-n may be the same or different. The remote operation control device 103 includes, for example, an acquisition unit 131, a first selection unit 132, a second selection unit 133, a setting unit 134, a control unit 135, a storage unit 136, a communication unit 137, an image generation unit 138, and an intention estimation unit 139. The robot 102, the remote operation control device 103, the environment sensor 104, the operation input unit 105, and the image display device 106 are each equipped with a power supply and the like (not shown).

[0087] The operation input unit 105 detects operation input information from the operator and outputs it to the remote operation control device 103. The operation input unit 105 is, for example, a controller equipped with operation buttons used in games, etc. The operation input information includes, for example, a selection instruction for selecting the robot 102 to be operated, an operation instruction for the robot 102, and information indicating the viewpoint that the operator wants to see. Note that the operation input unit 105 may be multiple, for example, one for operating the first robot (e.g., a controller), one for operating the second robot (e.g., a controller), and one for switching between robots to be used (e.g., a foot pedal).

[0088] The image display device 106 displays a display image required for remote operation generated by the image generation unit 138 of the remote operation control device 103. The image display device 106 is, for example, an HMD (head mounted display). The image display device 106 may also include a gaze detection device that detects the gaze of the operator.

[0089] (Environmental Sensor) The environmental sensor 104 acquires information about the surrounding environment. The environmental sensor 104 is, for example, an RGB (red, green, blue)-D imaging device that can also acquire depth information D. The environmental sensor 104 may also include an RGB imaging device and a distance sensor. The environmental sensor 104 assigns identification information that identifies the environmental sensor 104 to the surrounding environment information and outputs the information to the remote operation control device 103. The number of environmental sensors 104 may be two or more, for example, as long as the viewpoint can be switched, and may be the same as or different from the number of robots. When two robots 102 are selected, the environmental sensor 104 is installed, for example, between the two robots 102, at a position where it acquires information about the operable areas of the two selected end effectors. The installation position of the environmental sensor 104 is not limited to between the two robots 102, but may be, for example, directly behind the moving mechanism 121.

[0090] (Robot) The moving mechanism 121 is, for example, an arm of the robot 102, and one end is connected to the end effector 122 via a joint, and the other end is connected to the body or a base, etc., via a joint. The moving mechanism 121 may include, for example, a cart. Each joint is equipped with a sensor 123 and an actuator.

[0091] The end effector 122 has, for example, two or more fingers. The end effector 122 may be, for example, a multi-fingered hand having three or more fingers, or may be a gripper or the like. The fingers have joints, and each joint has a sensor 123 and an actuator.

[0092] The sensor 123 is, for example, an encoder, a force sensor, a pressure sensor, a 106-axis sensor, or the like, and is attached to each joint of the moving mechanism 121 and the end effector 122 .

[0093] The actuator 124 is driven in response to the output of the drive unit 125 and is attached to each joint of the moving mechanism 121 and the end effector 122 .

[0094] The driving unit 125 drives the actuator 124 in response to a control instruction or an operation instruction from the remote control device 103, thereby causing the movement mechanism 121 and the end effector 122 to perform operations. Note that the control instruction is an instruction based on an instruction from the operator. Also, the operation instruction is an instruction to automatically operate the movement mechanism 121 and the end effector 122 in accordance with the trajectories of the movement mechanism 121 and the end effector 122 that are operated based on the operation instruction.

[0095] The communication unit 126 adds identification information for identifying the robot 102 to the detection value detected by the sensor 123 and outputs the result to the remote control device 103. The communication unit 126 acquires a control instruction or an operation instruction from the remote control device 103.

[0096] (Remote Operation Control Device) The acquisition unit 131 acquires operation input information from the operation input unit 105. The acquisition unit 131 acquires environmental data from the environmental sensor 104. The acquisition unit 131 outputs detection values ​​detected by each sensor 123 from each robot 102.

[0097] The intention estimation unit 139 estimates, for example, the object to be worked on and the intended work of the operator based on the operation input information and the environmental data. Note that the intention estimation is performed using, for example, a method described in Japanese Patent Laid-Open No. 2022-157101.

[0098] The first selection unit 132 selects the robot 102 (the moving mechanism 121 and the end effector 122) based on the operation input information. The number of robots 102 to be selected may be one or two.

[0099] The second selection unit 133 selects the environmental sensor 104 according to the robot 102 selected by the first selection unit 132. Note that the second selection unit 133 may select the environmental sensor 104 based on information indicating a viewpoint that the operator desires to view, which is included in the operation input information. Alternatively, for example, when the operator operates the robot 102 with one hand, the second selection unit 133 may select the environmental sensor 104 installed on the left side of the robot 102 when the operator operates the robot 102 with the right hand, and may select the environmental sensor 104 installed on the right side of the robot 102 when the operator operates the robot 102 with the left hand. In other words, the second setting unit 133 may select the environmental sensor 104 according to the operator's operating hand and viewpoint.

[0100] When the selected robot 102 is changed, the setting unit 134 changes the operation coordinate system for operation based on, for example, the correspondence stored in the storage unit 136. For example, in FIG. 8 , when the first environmental sensor 104-1 installed between the first robot 102-1 and the third robot 102-3 is selected as the desired viewpoint, the setting unit 134 selects the coordinate system of the first environmental sensor 104-1 as the operation coordinate system based on the selected two robots, the first robot 102-1 and the third robot 102-3. That is, the setting unit 134 converts the coordinates of the operation input unit 105 operated by the operator who controls the robot 102 so that the movement direction of the robot 102 and the operation direction coincide with each other. Alternatively, the setting unit 134 may set the operation coordinate system to the coordinate system of the selected environmental sensor 104 in accordance with the operator's operating hand and viewpoint.

[0101] The control unit 135 acquires information about the surrounding environment from the environmental sensor 104 selected by the second selection unit 133. The control unit 135 generates an operation instruction for each robot 102 selected by the first selection unit 132 based on the input operation information. Note that the control unit 135 may perform control to support (assist) the operation of the operator based on the estimated operation intention, for example, using a method described in Japanese Patent Application No. 2023-045616.

[0102] The storage unit 136 stores programs, mathematical expressions, thresholds, predetermined values, identification information of the robots 102, identification information of the environmental sensors 104, etc. used by each unit of the remote operation control device 103. The storage unit 136 also stores a combination of the robots 102 to be selected and a combination of the environmental sensors 104 and coordinate systems selected accordingly. The storage unit 136 stores the environmental sensors and the operation coordinate systems in association with the viewpoints to be viewed.

[0103] The communication unit 137 outputs to the selected robot the operation instruction generated by the control unit 135. The communication unit 137 acquires an image from the selected environmental sensor 104.

[0104] The image generation unit 138 generates a display image required for remote control to be displayed on the image display device 106 using image data included in the captured information on the surrounding environment, for example.

[0105] [Example of Switching Between Robots, Coordinate Systems, and Environmental Sensors] Figure 10 is a diagram showing an example of switching between robots, coordinate systems, and environmental sensors in this embodiment. The example in Figure 10 is an example in which the first robot 102-1 and the third robot 102-3 are operated before switching, and the first robot 102-1 and the second robot 102-2 are operated after switching. When operating the first robot 102-1 and the third robot 102-3, as shown in the dashed-line rectangle g101 in Figure 8, the operation can be intuitively performed from the viewpoint of the first environmental sensor 104-1, so the first environmental sensor 104-1 is selected, and Σ based on the coordinate system of the first environmental sensor 104-1 is used as the operation coordinate system. ｃａｍＡ When operating the first robot 102-1 and the second robot 102-2, as shown in the dashed square g102 in FIG. 8, the operation can be intuitively performed from the viewpoint of the second environmental sensor 104-2, so the second environmental sensor 104-2 is selected, and Σ based on the coordinate system of the second environmental sensor 104-2 is used as the operation coordinate system. ｃａｍＢ is selected.

[0106] 10 is an example, and the combination is not limited to this. Furthermore, the number of robots 102 to be investigated may be one. Even in this case, the remote control device 103 switches the environment sensor 104 and the operation coordinate system in accordance with the switching of the robot 102.

[0107] 11 is a diagram showing an example of a viewpoint image captured by the first environmental sensor before switching. In this case, as in the example of FIG. 10, the first robot 102-1 and the third robot 102-3 are operated to perform tasks, and the operation coordinate system Σ ｃａｍＡ is selected. Operation coordinate system Σ ｃａｍＡ The coordinates of the hand of the first robot 102-1 as seen from ｃａｍＡ In addition, the operation coordinate system Σ ｃａｍＡ The coordinates of the hand of the third robot 102-3 as seen from the ｃａｍＡ is.

[0108] 12 is a diagram showing an example after switching according to this embodiment. The image g121 is an example of a viewpoint image of the second environment sensor 104-2 after switching. After switching, the first robot 102-1 and the second robot 102-2 are operated to perform tasks, and the operation coordinate system Σ ｃａｍＢ is selected. Operation coordinate system Σ ｃａｍＢ The coordinates of the hand of the first robot 102-1 as seen from ｃａｍＢ In addition, the operation coordinate system Σ ｃａｍＢ The coordinates of the hand of the second robot 102-2 as seen from the ｃａｍＢ is.

[0109] Reference symbol g122 is an image diagram of the operation direction by the operation input unit 105 and the movement direction of the robot 102. According to this embodiment, as shown by reference symbol g122, the operation input and the movement direction of the robot 102 are the same, so the operator can intuitively operate the robot 102.

[0110] FIG. 13 is a diagram showing an example of the state after switching in the prior art. The image g131 is an example of a viewpoint image of the second environment sensor 104-2 after switching. After switching, the first robot 102-1 and the second robot 102-2 are operated to perform work. In the prior art, the operation coordinate system is not switched, so the operation coordinate system after switching is also ΣｃａｍＡ Therefore, the operation coordinate system Σ ｃａｍＡ The coordinates of the hand of the first robot 102-1 as seen from ｃａｍＡ In addition, the operation coordinate system Σ ｃａｍＡ The coordinates of the hand of the second robot 102-2 as seen from the ｃａｍＢ is.

[0111] Reference symbol g132 is an image diagram of the operation direction by the operation input unit 105 and the movement direction of the robot 102. In the conventional technology, even if the robot 102 and the environmental sensor are switched, the operation coordinate system is not switched. Therefore, in the conventional technology, the operation input and the movement direction of the robot 102 differ as shown by reference symbol g132, making it difficult for the operator to intuitively operate the robot.

[0112] [Processing Procedure] Next, an example of the processing procedure will be described. Fig. 14 is a flowchart showing the processing of the remote control control device in this embodiment.

[0113] (Step S101) The acquisition unit 131 acquires operation input information from the operation input unit 105. The acquisition unit 131 extracts an instruction to select the robot 102 to be used for the operation, which is included in the operation instruction information.

[0114] (Step S102 ) The first selection unit 132 selects a robot 102 based on the selection instruction for the robot 102 .

[0115] (Step S103 ) The second selection unit 133 selects an environmental sensor 104 in accordance with the robot 102 selected by the first selection unit 132 .

[0116] (Step S104) The setting unit 134 sets an operation coordinate system for operation based on the correspondence stored in the storage unit 136, for example.

[0117] (Step S105) The control unit 135 calculates the coordinates of the selected hand of the robot 102 based on the input operation information, the estimated operation intention, and the determined operation coordinate system, and generates an operation instruction.

[0118] (Step S106) The control unit 135 outputs an operation instruction to the robot 102 that it controls via the communication unit 137.

[0119] (Step S107) The first selection unit 132 determines whether or not a selection instruction to switch the set robot 102 has been acquired. If a selection instruction to switch the set robot 102 has been acquired (Step S107; YES), the first selection unit 132 proceeds to the processing of Step S108. If a selection instruction to switch the set robot 102 has not been acquired (Step S107; NO), the first selection unit 132 returns to the processing of Step S105.

[0120] (Step S108) The first selection unit 132 switches and selects the robot 102 based on the selection instruction of the switched robot 102.

[0121] (Step S109 ) The second selection unit 133 switches and selects the environmental sensor 104 depending on the robot 102 switched and selected by the first selection unit 132 .

[0122] (Step S110) The setting unit 134 changes the operation coordinate system to be operated based on the correspondence stored in the storage unit 136, for example.

[0123] (Step S111) The control unit 135 calculates the coordinates of the selected hand of the robot 102 based on the input operation information, the estimated operation intention, and the switched operation coordinate system, and generates an operation instruction.

[0124] (Step S112) The control unit 135 outputs an operation instruction to the robot 102 that it controls via the communication unit 137.

[0125] 14 are merely examples, and are not limiting. Other processes may be performed, and some processes may be performed in parallel.

[0126] In the above example, the environmental sensor 104 is switched in accordance with the switching of the robot 102 to be used, but this is not limiting. The viewpoint may be switched in accordance with the switching of the robot 102. For example, the viewpoint may be changed by changing the tilt angle or pan angle of the environmental sensor before and after the switching. Alternatively, if the environmental sensor 104 is capable of capturing a wide range, an image of a range based on a first viewpoint may be cut out from the entire image before the switching, and an image of a range based on a second viewpoint may be cut out from the entire image after the switching.

[0127] Furthermore, the switching pattern of the robots 102 is not limited to the above-described patterns, and may be a pattern of switching from two robots 102 to one robot 102, or from one robot 102 to two robots 102, etc. Furthermore, the number of robots 102 that are combinations of moving mechanisms 121 (arms) and end effectors 122 may be four or more.

[0128] However, with conventional technology, if the above switching is simply performed, for example, when trying to operate a moving mechanism (arm) based on an image seen from behind, the left and right and front and back will be reversed, making remote control difficult. Also, with conventional technology, switching can confuse the operator as to which direction the arm will move when moved in either direction.

[0129] In contrast to this, in this embodiment, not only are the robot 102 and the environmental sensor 104 to be operated switched, but the operator coordinates are also switched in accordance with the operation target. As a result, according to this embodiment, by switching the operation coordinate system, the direction of the input to the robot 102 and the input to the operation input unit 105 coincides, thereby realizing intuitive operation and facilitating remote control.

[0130] The robot remote operation control system 101 of this embodiment can be applied to, for example, factories, hospitals, etc.

[0131] In addition, a program for implementing some or all of the functions of the remote control device 3, 103 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the remote control device 3, 103. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. Alternatively, some or all of these components may be realized by LSI (Large Scale Integration) hardware (including circuitry) such as an ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or SOC (System On Chip), or may be realized by a combination of software and hardware.

[0132] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0133] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0134] 1...Robot remote operation control system, 2-1...First robot, 2-2...Second robot, 22-3...Third robot, 2 (2-1, 2-2, 2-3, ...)...Robot, 3...Remote operation control device, 4...Environment sensor, 5...Operation input unit, 6...Image display device, NW...Network, 21, 21-1, ..., 21-n...Moving mechanism, 22, 22-1, ..., 22-n...End effector, 23, 2 3-1, ..., 23-n... sensors, 24, 24-1, ..., 24-n... actuators, 25, 25-1, ..., 25-n... drive units, 26, 26-1, ..., 26-n... communication units, 31... acquisition unit, 32... operation designation unit, 33... trajectory generation unit, 34... first control unit, 35... second control unit, 36... storage unit, 37... communication unit, 38... image generation unit, 39... intention estimation unit, 41... imaging device, 42... communication unit

[0135] REFERENCE SIGNS LIST 101...robot remote operation control system, 102-1...first robot, 102-2...second robot, 122-3...third robot, 102 (102-1, 102-2, 102-3, ...)...robot, 103...remote operation control device, 104-1...first environment sensor, 104-2...second environment sensor, 104-3...third environment sensor, 104 (104-1, 104-2, 104-3, ...)...environment sensor, 105...operation input unit, 106...image display device, NW...network, 121, 121-1, ... , 121-n...movement mechanism, 122, 122-1, ..., 122-n...end effector, 123, 123-1, ..., 123-n...sensor, 124, 124-1, ..., 124-n...actuator, 125, 125-1, ..., 125-n...drive unit, 126, 126-1, ..., 126-n...communication unit, 131...acquisition unit, 132...first selection unit, 133...second selection unit, 134...setting unit, 135...control unit, 136...storage unit, 137...communication unit, 138...image generation unit, 139...intention estimation unit

Claims

1. A robot remote operation control system that recognizes the movements of an operator and communicates the movements of the operator to a robot to operate the robot, wherein the robot comprises: three or more end effectors; a movement mechanism that moves each of the end effectors; a first control unit that controls the movement of at least one of the end effectors in accordance with the movement of the operator; a movement designation unit that determines a starting point of action for the end effector based on the movement of the operator and designates the movement of the end effector from the starting point of action; a trajectory generation unit that generates a constrained trajectory for the end effector from the movement designated by the movement designation unit; and a second control unit that operates at least one of the end effectors not controlled by the first control unit on the trajectory.

2. The robot remote operation control system of claim 1, wherein, when there are two gripping points at which the object to be operated is grasped based on the operator's movement, the operation designation unit sets one of the two gripping points as the action starting point and the other of the two gripping points as the action end point, and the trajectory generation unit sets the space between the action starting point and the action end point as a constrained trajectory of the end effector.

3. A robot remote operation control system as described in claim 1 or claim 2, wherein the trajectory generation unit generates a constrained trajectory by preventing movement in at least one of multiple axial directions based on the target object and work content estimated based on the operator's movements.

4. A robot remote operation control system as described in claim 1 or claim 2, further comprising an intention estimation unit that estimates a target object and a task content based on the movement of the operator, and wherein the first control unit and the second control unit use the estimation results estimated by the intention estimation unit to control each of the end effectors to be controlled.

5. A robot remote operation control system as described in claim 4, wherein the first control unit controls the operation of the end effector so as to support the operation of the operator based on the operation intention estimated by the intention estimation unit.

6. A remote operation control device that remotely controls a robot having three or more end effectors and a movement mechanism for moving each of the end effectors, comprising: a first control unit that controls the movement of at least one of the end effectors in accordance with the movement of an operator; a movement designation unit that determines a starting point of action for the end effector based on the movement of the operator and designates the movement of the end effector from the starting point of action; a trajectory generation unit that generates a constrained trajectory for the end effector from the movement designated by the movement designation unit; and a second control unit that operates at least one of the end effectors not controlled by the first control unit on the trajectory.

7. A control method for a remote control device that remotely controls a robot having three or more end effectors and a movement mechanism for moving each of the end effectors, comprising: a first control unit that controls the movement of at least one of the end effectors in accordance with the movement of an operator; a movement designation unit that determines a starting point of action for the end effector based on the movement of the operator and designates the movement of the end effector from the starting point of action; a trajectory generation unit that generates a constrained trajectory for the end effector from the movement designated by the movement designation unit; and a second control unit that operates at least one of the end effectors not controlled by the first control unit on the trajectory.

8. A program that causes a computer of a remote control device that remotely controls a robot having three or more end effectors and a movement mechanism for moving each of the end effectors to control the movement of at least one of the end effectors in accordance with the movement of an operator, determines a starting point of action for the end effector based on the movement of the operator, specifies the movement of the end effector from the starting point of action, generates a constrained trajectory for the end effector from the specified movement, and moves at least one of the end effectors other than the end effector controlled in accordance with the movement of the operator on the trajectory.

9. A robot remote operation control system that recognizes the movements of an operator and transmits the movements of the operator to a robot to operate the robot, wherein the robot comprises a plurality of end effectors and a movement mechanism that moves the end effectors; a plurality of environmental sensors that acquire information about the surrounding environment; a first selection unit that selects the end effector that transmits the movements of the operator; a second selection unit that selects the environmental sensor in accordance with the end effector selected by the first selection unit; and a setting unit that, when the end effector is changed, switches a coordinate system based on the environmental sensors so that the movement direction of the end effector and the movement of the operator coincide.

10. A robot remote operation control system according to claim 9, wherein the number of selected end effectors is two, and the environmental sensor is installed between the two end effectors and at a position that acquires information about the operable area of ​​the two selected end effectors.

11. A robot remote operation control system as described in claim 9 or claim 10, wherein the second selection unit selects the environmental sensor in accordance with the operator's operating hand and viewpoint, and the setting unit sets the coordinate system in accordance with the operator's operating hand and viewpoint.

12. A robot remote operation control system according to claim 9 or claim 10, further comprising a control unit that calculates the hand coordinates of the end effector based on the coordinate system switched by the setting unit and controls the end effector.

13. A robot remote operation control system as described in claim 12, further comprising an intention estimation unit that estimates a target object and a task content based on the movement of the operator, and the control unit controls each of the end effectors to be controlled using the estimation results obtained by the intention estimation unit.

14. A robot remote operation control system according to claim 13, wherein the control unit controls the operation of the end effector so as to support the operation of the operator based on the operation intention estimated by the intention estimation unit.

15. A control device for remotely operating a robot that recognizes the movements of an operator and transmits the movements of the operator to a robot having a plurality of end effectors and a movement mechanism for moving the end effectors to operate the robot, comprising: a plurality of environmental sensors that acquire information about the surrounding environment; a first selection unit that selects the end effector that transmits the movements of the operator; a second selection unit that selects the environmental sensor in accordance with the end effector selected by the first selection unit; and a setting unit that, when the end effector is changed, switches the coordinate system based on the environmental sensor so that the movement direction of the end effector and the movement of the operator coincide.

16. A control method for a control device that recognizes the movements of an operator and transmits the movements of the operator to a robot having a plurality of end effectors and a movement mechanism for moving the end effectors to operate the robot, wherein an acquisition unit acquires information about the surrounding environment acquired by a plurality of environmental sensors, a first selection unit selects the end effector that transmits the movements of the operator, a second selection unit selects the environmental sensor according to the end effector selected by the first selection unit, and a setting unit switches the coordinate system based on the environmental sensor when the end effector is changed.

17. A program that causes a computer of a control device that recognizes the movements of an operator and transmits the movements of the operator to a robot having multiple end effectors and a movement mechanism for moving the end effectors to operate the robot, to acquire information about the surrounding environment obtained by multiple environmental sensors, to select the end effector that will transmit the movements of the operator, to select the environmental sensor according to the selected end effector, and, when the end effector is changed, to switch the coordinate system based on the environmental sensor so that the movement direction of the end effector and the movement of the operator are aligned.

Images