Robot remote operation control system, robot remote operation control device, robot remote operation control method, program

Abstract

To provide a robot remote operation control system, a robot remote operation control device, a robot remote operation control method, and a program which can intuitively operate a robot having three or more arms to facilitate remote control.SOLUTION: In a robot remote operation control system, a robot comprises a plurality of end effectors and a movement mechanism that moves the end effectors, and further comprises: a plurality of environment sensors that obtain information on a peripheral environment; a first selecting part that selects the end effector that transmits movement of an operator; a second selecting part that selects the environment sensor in accordance with the end effector selected by the first selecting part; and a setting part that switches a coordinate based on the environment sensor so that a moving direction of the end effector matches the movement of the operator, when changing the end effector.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a robot remote operation control system, a robot remote operation control device, a robot remote operation control method, and a program. [Background technology]

[0002] For example, when remotely controlling a robot arm, it may be necessary to use three or more robot arms depending on the object being controlled (see, for example, Patent Document 1). One example is the Da Vinci surgical robot. Da Vinci has three arms and one camera, and the arms are controlled by two hand controllers operated by stepping on foot pedals.

[0003] When you want to remotely control an object around it, it can be difficult to do so with one viewpoint and two arms due to occlusion and lack of reachability. In such cases, you can install additional cameras and arms and switch between the image and the control arm. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-107721 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technology, simply switching the view would result in the left and right directions being reversed when trying to operate an arm based on a video image seen from behind, making remote control difficult. Furthermore, switching can confuse the operator about which way to move the arm. Thus, with conventional technology, when there are three or more arms, intuitive operation is often difficult, making remote control difficult.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a robot remote operation control system, a robot remote operation control device, a robot remote operation control method, and a program that can be operated intuitively and make remote control easy in the case of three or more arms. [Means for solving the problem]

[0007] (1) In order to achieve the above-mentioned object, a robot remote operation control system according to one embodiment of the present invention is a robot remote operation control system that recognizes the movements of an operator, transmits the movements of the operator to a robot, and operates the robot, wherein the robot is equipped with a plurality of end effectors and a movement mechanism that moves the end effectors, and is equipped with a plurality of environmental sensors that acquire information about the surrounding environment, a first selection unit that selects the end effector that transmits the movements of the operator, a second selection unit that selects the environmental sensor in accordance with the end effector selected by the first selection unit, and a setting unit that, when the end effector is changed, switches a coordinate system based on the environmental sensors so that the movement direction of the end effector and the movement of the operator are aligned.

[0008] (2) In the robot remote operation control system according to one aspect of (1) above, the number of selected end effectors may be two, and the environmental sensor may be installed between the two end effectors at a position where it can acquire information about the operable area of ​​the two selected end effectors.

[0009] (3) In the robot remote operation control system according to one aspect of (1) or (2) above, the second selection unit may select the environmental sensor in accordance with the operator's operating hand and viewpoint, and the setting unit may set the coordinate system in accordance with the operator's operating hand and viewpoint.

[0010] (4) In the robot remote operation control system according to any one of the above (1) to (3), the system may further include a control unit that calculates the hand coordinates of the end effector based on the coordinate system switched by the setting unit and controls the end effector.

[0011] (5) In a robot remote operation control system according to one aspect of (4) above, an intention estimation unit may be provided that estimates a target object and a task content based on the movement of the operator, and the control unit may control each of the end effectors to be controlled using the estimation results obtained by the intention estimation unit.

[0012] (6) In the robot remote operation control system according to one aspect of (5) above, the control unit may be configured to control the operation of the end effector to support the operation of the operator based on the operation intention estimated by the intention estimation unit.

[0013] (7) In order to achieve the above object, a robot remote operation control device according to one embodiment of the present invention is a control device that recognizes the movements of an operator and transmits the movements of the operator to a robot having a plurality of end effectors and a movement mechanism for moving the end effectors to operate the robot, and is equipped with a plurality of environmental sensors that acquire information about the surrounding environment, a first selection unit that selects the end effectors that transmit the movements of the operator, a second selection unit that selects the environmental sensor depending on the end effector selected by the first selection unit, and a setting unit that switches a coordinate system based on the environmental sensor when the end effector is changed so that the movement direction of the end effector and the movement of the operator are aligned.

[0014] (8) In order to achieve the above-mentioned object, a robot remote operation control method according to one embodiment of the present invention is a control method for a control device that recognizes the movements of an operator and transmits the movements of the operator to a robot having a plurality of end effectors and a movement mechanism for moving the end effectors to operate the robot, wherein an acquisition unit acquires information about the surrounding environment acquired by a plurality of environmental sensors, a first selection unit selects the end effector that transmits the movements of the operator, a second selection unit selects the environmental sensor according to the end effector selected by the first selection unit, and a setting unit switches a coordinate system based on the environmental sensor when the end effector is changed.

[0015] (9) In order to achieve the above object, one aspect of the present invention provides a program that recognizes the movements of an operator, transmits the movements of the operator to a robot having a plurality of end effectors and a movement mechanism for moving the end effectors, and causes a computer of a control device that controls the robot to acquire information about the surrounding environment obtained by a plurality of environmental sensors, selects the end effector that will transmit the movements of the operator, selects the environmental sensor according to the selected end effector, and, when the end effector is changed, switches a coordinate system based on the environmental sensor so that the movement direction of the end effector and the movement of the operator are aligned. [Effects of the Invention]

[0016] According to the above (1) to (9), in the case of three or more arms, it is possible to intuitively operate the robot and to easily control it remotely. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram illustrating a schematic configuration example of a robot remote operation control system according to an embodiment. [Figure 2] 1 is a diagram illustrating an example of the configuration of a robot remote operation control system according to an embodiment. [Figure 3]10A and 10B are diagrams illustrating examples of switching between a robot, a coordinate system, and an environmental sensor in an embodiment. [Figure 4] FIG. 10 is a diagram showing an example of a viewpoint image captured by a first environmental sensor before switching. [Figure 5] FIG. 10 is a diagram illustrating an example after switching according to the embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of a state after switching in the prior art. [Figure 7] 4 is a flowchart of a process of the remote control control device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0019] [overview] FIG. 1 is a diagram showing an example of a schematic configuration of a robot remote operation control system according to this embodiment. The robot remote operation control system 1 includes, for example, a first robot 2-1, a second robot 2-2, a third robot 2-3, a remote operation control device 3 (robot remote operation control device), a first environmental sensor 4-1, a second environmental sensor 4-2, and a third environmental sensor 4-3.

[0020] The n-th robot 2-n (n is an integer from 1 to 3) includes, for example, a moving mechanism 21-n (e.g., an arm) and an end effector 22-n. Note that one robot 2 may include three moving mechanisms and three end effectors. Alternatively, the first robot may include two moving mechanisms and two end effectors, and the second robot may include one moving mechanism and one end effector.

[0021] In the example of Fig. 1, the operator operates the input operation unit (Fig. 2) to operate the robot 2, thereby causing the robot 2 to perform a task on an operation target object obj. In operation, the operator switches the robot 2 to be used. In such a case, in this embodiment, for example, in a combination of three sets of moving mechanisms and end effectors, not only are the moving mechanisms and environmental sensors to be operated switched, but the operator's coordinates are also switched according to the operation targets.

[0022] [Configuration of a robot remote control system] Fig. 2 is a diagram showing an example of the configuration of a robot remote operation control system according to this embodiment. As shown in Fig. 2, the robot remote operation control system 1 includes, for example, a first robot 2-1, a second robot 2-2, a third robot 2-3, ..., a remote operation control device 3, an environmental sensor 4, an operation input unit 5, and an image display device 6.

[0023] The robots 2 (first robot 2-1, ..., n-th robot-n) include, for example, moving mechanisms 21 (21-1, ..., 21-n), end effectors 22 (22-1, ..., 22-n), sensors 23 (23-1, ..., 23-n), actuators 24 (24-1, ..., 24-n), drive units 25 (25-1, ..., 25-n), and communication units 26 (26-1, ..., 26-n). Note that the configurations of the robots 2-n may be the same or different. The remote control control device 3 includes, for example, an acquisition unit 31, a first selection unit 32, a second selection unit 33, a setting unit 34, a control unit 35, a memory unit 36, a communication unit 37, an image generation unit 38, and an intention estimation unit 39. The robot 2, the remote control device 3, the environmental sensor 4, the operation input unit 5, and the image display device 6 are equipped with a power supply and the like (not shown).

[0024] The operation input unit 5 detects operation input information from the operator and outputs it to the remote operation control device 3. The operation input unit 5 is, for example, a controller equipped with operation buttons used in games and the like. The operation input information includes, for example, a selection instruction to select the robot 2 to be used for operation, an operation instruction for the robot 2, and information indicating the viewpoint that the operator wants to see. Note that the operation input unit 5 may be multiple, for example, one for operating the first robot (for example, a controller), one for operating the second robot (for example, a controller), and one for switching between the robots to be used (for example, a foot pedal).

[0025] The image display device 6 displays the display image required for remote operation generated by the image generation unit 38 of the remote operation control device 3. The image display device 6 is, for example, an HMD (head mounted display). The image display device 6 may also include a line of sight detection device that detects the line of sight of the operator.

[0026] (environmental sensor) The environmental sensor 4 acquires information about the surrounding environment. The environmental sensor 4 is, for example, an RGB (red, green, blue)-D imaging device that can also acquire depth information D. The environmental sensor 4 may also include an RGB imaging device and a distance sensor. The environmental sensor 4 assigns identification information that identifies the device to the information about the surrounding environment and outputs the information to the remote operation control device 3. The number of environmental sensors 4 may be two or more as long as it allows for switching of viewpoints, and may be the same as or different from the number of robots. When two robots 2 are selected, the environmental sensor 4 is installed, for example, between the two robots 2, at a position where it acquires information about the operable areas of the two selected end effectors. The installation position of the environmental sensor 4 is not limited to between the two robots 2, but may be, for example, directly behind the moving mechanism 21.

[0027] (robot) The moving mechanism 21 is, for example, an arm of the robot 2, with one end connected to the end effector 22 via a joint and the other end connected to the body or a base via a joint. The moving mechanism 21 may also include, for example, a cart. Each joint is equipped with a sensor 23 and an actuator.

[0028] The end effector 22 has, for example, two or more fingers. The end effector 22 may be, for example, a multi-fingered hand having three or more fingers, or may be a gripper or the like. The fingers have joints, and each joint has a sensor 23 and an actuator.

[0029] The sensor 23 is, for example, an encoder, a force sensor, a pressure sensor, a six-axis sensor, or the like, and is attached to each joint of the moving mechanism 21 and the end effector 22.

[0030] The actuator 24 is driven in response to the output of the drive unit 25 and is attached to each joint of the moving mechanism 21 and the end effector 22 .

[0031] The driving unit 25 drives the actuator 24 in response to a control instruction or an operation instruction from the remote control device 3, thereby causing the movement mechanism 21 and the end effector 22 to perform an operation. Note that the control instruction is an instruction based on an instruction from the operator. Also, the operation instruction is an instruction to automatically operate the movement mechanism 21 and the end effector 22 according to the trajectory of the movement mechanism 21 and the end effector 22 that are operated based on the operation instruction.

[0032] The communication unit 26 adds identification information for identifying the robot 2 to the detection value detected by the sensor 23 and outputs the result to the remote control device 3. The communication unit 26 acquires a control instruction or an operation instruction from the remote control device 3.

[0033] (Remote control device) The acquisition unit 31 acquires operation input information from the operation input unit 5. The acquisition unit 31 acquires environmental data from the environmental sensor 4. The acquisition unit 31 outputs the detection values ​​detected by each sensor 23 from each robot 2.

[0034] The intention estimation unit 39 estimates, for example, the object to be worked on and the intended work of the operator based on the operation input information and the environmental data. Note that the intention estimation is performed using, for example, a method described in Japanese Patent Laid-Open No. 2022-157101.

[0035] The first selection unit 32 selects the robot 2 (the moving mechanism 21, the end effector 22) based on the operation input information. Note that the number of robots 2 to be selected may be one or two.

[0036] The second selection unit 33 selects an environmental sensor 4 in accordance with the robot 2 selected by the first selection unit 32. The second selection unit 33 may select an environmental sensor 4 based on information indicating a viewpoint that the operator desires to view, which is included in the operation input information. Alternatively, for example, when the operator operates the robot 2 with one hand, the second selection unit 33 may select an environmental sensor 4 installed on the left side of the robot 2 when the operator operates the robot 2 with the right hand, and may select an environmental sensor 4 installed on the right side of the robot 2 when the operator operates the robot 2 with the left hand. In other words, the second setting unit 33 may select an environmental sensor 4 in accordance with the operator's operating hand and viewpoint.

[0037] When the selected robot 2 is changed, the setting unit 34 changes the operation coordinate system for operation based on, for example, the correspondence stored in the storage unit 36. For example, when the first environmental sensor 4-1 installed between the first robot 2-1 and the third robot 2-3 in FIG. 1 is selected as the desired viewpoint, the setting unit 34 selects the coordinate system of the first environmental sensor 4-1 as the operation coordinate system based on the selected two robots, the first robot 2-1 and the third robot 2-3. That is, the setting unit 34 converts the coordinates of the operation input unit 5 operated by the operator who controls the robot 2 so that the movement direction of the robot 2 matches the operation direction. Alternatively, the setting unit 34 may set the operation coordinate system to the coordinate system of the selected environmental sensor 4 in accordance with the operator's operating hand and viewpoint.

[0038] The control unit 35 acquires information about the surrounding environment from the environmental sensor 4 selected by the second selection unit 33. The control unit 35 generates operation instructions for each robot 2 selected by the first selection unit 32 based on the input operation information. Note that the control unit 35 may perform control to support (assist) the operator's operation based on the estimated operation intention, for example, using a method described in Japanese Patent Application No. 2023-045616.

[0039] The storage unit 36 ​​stores programs, mathematical formulas, thresholds, predetermined values, identification information of the robots 2, identification information of the environmental sensors 4, etc. used by each unit of the remote control device 3. The storage unit 36 ​​also stores the combination of robots 2 to be selected and the combination of the environmental sensors 4 and coordinate systems selected accordingly. The storage unit 36 ​​stores the environmental sensors and the operation coordinate systems in association with the viewpoints to be viewed.

[0040] The communication unit 37 outputs to the selected robot the operation instruction generated by the control unit 35. The communication unit 37 acquires an image from the selected environmental sensor 4.

[0041] The image generating unit 38 generates a display image required for remote control to be displayed on the image display device 6 using image data included in the captured information on the surrounding environment, for example.

[0042] [Example of switching between robots, coordinate systems, and environmental sensors] 3 is a diagram showing an example of switching between robots, coordinate systems, and environmental sensors in this embodiment. The example in Fig. 3 shows an example in which the first robot 2-1 and the third robot 2-3 are operated before switching, and the first robot 2-1 and the second robot 2-2 are operated after switching. When operating the first robot 2-1 and the third robot 2-3, as shown by the dashed square g1 in Fig. 1, the operation can be intuitively performed from the viewpoint of the first environmental sensor 4-1, so the first environmental sensor 4-1 is selected, and the Σ camA is selected. When operating the first robot 2-1 and the second robot 2-2, as shown by the dashed square g2 in Figure 1, the operation can be intuitively performed from the viewpoint of the second environmental sensor 4-2, so the second environmental sensor 4-2 is selected, and the Σ camB is selected.

[0043] 3 is an example and is not limited to this. Furthermore, the number of robots 2 to be investigated may be one. Even in this case, the remote control device 3 switches the environment sensor 4 and the operation coordinate system in accordance with the switching of the robot 2.

[0044] 4 is a diagram showing an example of a viewpoint image captured by the first environmental sensor before switching. In this case, as in the example of FIG. 3, the first robot 2-1 and the third robot 2-3 are operated to perform tasks, and the operation coordinate system Σ camA is selected. Operation coordinate system Σ camA The coordinates of the first robot 2-1's hand as seen from the camA In addition, the operation coordinate system Σ camA The coordinates of the third robot 2-3's hand as seen from the camA is.

[0045] 5 is a diagram showing an example of a post-switching image according to this embodiment. An image g21 is an example of a viewpoint image of the second environmental sensor 4-2 after switching. After the switching, the first robot 2-1 and the second robot 2-2 are operated to perform the task, and the operation coordinate system Σ camB is selected. Operation coordinate system Σ camB The coordinates of the first robot 2-1's hand as seen from the camB In addition, the operation coordinate system Σ camB The coordinates of the hand of the second robot 2-2 as seen from the camB is.

[0046] Reference symbol g22 is an image diagram of the operation direction by the operation input unit 5 and the movement direction of the robot 2. According to this embodiment, as shown by reference symbol g22, the operation input and the movement direction of the robot 2 are the same, so the operator can intuitively operate it.

[0047] 6 is a diagram showing an example of a post-switching example of the prior art. The image g31 is an example of a viewpoint image of the second environmental sensor 4-2 after switching. After the switching, the first robot 2-1 and the second robot 2-2 are operated to perform the work. In the conventional technology, the operation coordinate system is not switched, so the operation coordinate system after the switching is also Σ camA It remains as it is. Therefore, the operation coordinate system Σ camA The coordinates of the first robot 2-1's hand as seen from the camA In addition, the operation coordinate system Σ camA The coordinates of the hand of the second robot 2-2 as seen from the camB is.

[0048] Reference symbol g32 is an image diagram of the operation direction by the operation input unit 5 and the movement direction of the robot 2. In the prior art, even if the robot 2 is switched and the environmental sensor is switched, the operation coordinate system is not switched. For this reason, in the prior art, as shown by reference symbol g32, the operation input and the movement direction of the robot 2 differ, making it difficult for the operator to intuitively operate the robot.

[0049] [Processing Procedure] Next, an example of the processing procedure will be described with reference to Fig. 7, which is a flowchart showing the processing of the remote control device in this embodiment.

[0050] (Step S1) The acquisition unit 31 acquires operation input information from the operation input unit 5. The acquisition unit 31 extracts an instruction to select the robot 2 to be used for the operation, which is included in the operation instruction information.

[0051] (Step S2) The first selection unit 32 selects a robot 2 based on the selection instruction for the robot 2.

[0052] (Step S3) The second selection unit 33 selects an environment sensor 4 according to the robot 2 selected by the first selection unit 32.

[0053] (Step S4) The setting unit 34 sets an operation coordinate system for operation based on the correspondence stored in the storage unit 36, for example.

[0054] (Step S5) The control unit 35 calculates the coordinates of the selected hand of the robot 2 based on the input operation information, the estimated operation intention, and the determined operation coordinate system, and generates an operation instruction.

[0055] (Step S6) The control unit 35 outputs an operation instruction to the robot 2 to be controlled via the communication unit 37.

[0056] (Step S7) The first selection unit 32 determines whether or not a selection instruction to switch the set robot 2 has been acquired. If a selection instruction to switch the set robot 2 has been acquired (Step S7; YES), the first selection unit 32 proceeds to the processing of Step S8. If a selection instruction to switch the set robot 2 has not been acquired (Step S7; NO), the first selection unit 32 returns to the processing of Step S5.

[0057] (Step S8) The first selection unit 32 switches and selects the robot 2 based on the selection instruction of the switched robot 2.

[0058] (Step S9) The second selection unit 33 switches and selects the environmental sensor 4 according to the robot 2 switched and selected by the first selection unit 32.

[0059] (Step S10) The setting unit 34 changes the operation coordinate system to be operated based on the correspondence stored in the storage unit 36, for example.

[0060] (Step S11) The control unit 35 calculates the coordinates of the selected hand of the robot 2 in the input operation information, the estimated operation intention, and the switched operation coordinate system, and generates an operation instruction.

[0061] (Step S12) The control unit 35 outputs an operation instruction to the robot 2 to be controlled via the communication unit 37.

[0062] 7 are merely examples, and are not limiting. Other processes may be performed, and some processes may be performed in parallel.

[0063] In the above example, the environmental sensor 4 is switched in accordance with the switching of the robot 2 to be used, but this is not limiting. The viewpoint may be switched in accordance with the switching of the robot 2, and for example, the viewpoint may be changed by changing the tilt angle or pan angle of the environmental sensor before and after the switching. Alternatively, if the environmental sensor 4 is capable of capturing a wide range, an image of a range based on a first viewpoint may be cut out from the entire image before the switching, and an image of a range based on a second viewpoint may be cut out from the entire image after the switching.

[0064] Furthermore, the switching pattern of the robots 2 is not limited to the above-mentioned patterns, and may be a pattern of switching from two robots 2 to one robot 2, or from one robot 2 to two robots 2, etc. Furthermore, the number of robots 2 that are combinations of moving mechanisms 21 (arms) and end effectors 22 may be four or more.

[0065] However, with conventional technology, if the above switching is simply performed, for example, when trying to operate a moving mechanism (arm) based on an image seen from the back, the left and right and front and back will be reversed, making remote control difficult. Also, with conventional technology, switching can confuse the operator as to which direction the arm will move when moved in either direction.

[0066] In contrast to this, in this embodiment, not only are the robot 2 and the environmental sensor 4 that are the operation targets switched, but the operator coordinates are also switched in accordance with the operation targets. As a result, according to this embodiment, by switching the operation coordinate system, the direction of the input of the robot 2 and the input of the operation input unit 5 are matched, thereby realizing intuitive operation and facilitating remote control.

[0067] The robot remote operation control system 1 of this embodiment can be applied to, for example, factories, hospitals, and the like.

[0068] A program for implementing some or all of the functions of the remote control device 3 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the remote control device 3. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. Alternatively, some or all of these components may be realized by LSI (Large Scale Integration) hardware (including circuitry) such as an ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or SOC (System On Chip), or may be realized by a combination of software and hardware.

[0069] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0070] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0071] 1...robot remote operation control system, 2-1...first robot, 2-2...second robot, 22-3...third robot, 2 (2-1, 2-2, 2-3,...)...robot, 3...remote operation control device, 4-1...first environment sensor, 4-2...second environment sensor, 4-3...third environment sensor, 4 (4-1, 4-2, 4-3,...)...environment sensor, 5...operation input unit, 6...image display device, NW...network, 21, 21-1,..., 21-n ...movement mechanism, 22, 22-1,..., 22-n...end effector, 23, 23-1,..., 23-n...sensor, 24, 24-1,..., 24-n...actuator, 25, 25-1,..., 25-n...drive unit, 26, 26-1,..., 26-n...communication unit, 31...acquisition unit, 32...first selection unit, 33...second selection unit, 34...setting unit, 35...control unit, 36...storage unit, 37...communication unit, 38...image generation unit, 39...intention estimation unit

Claims

1. A robot remote operation control system that recognizes an operator's movement and transmits the operator's movement to a robot to operate the robot, the robot includes a plurality of end effectors and a movement mechanism that moves the end effectors; a plurality of environmental sensors for acquiring information about the surrounding environment; a first selection unit that selects the end effector that transmits the movement of the operator; a second selection unit that selects the environmental sensor in accordance with the end effector selected by the first selection unit; a setting unit that switches a coordinate system based on the environmental sensor when the end effector is changed so that a moving direction of the end effector coincides with a movement of the operator; A robot remote operation control system comprising:

2. The number of selected end effectors is two, The environmental sensor is installed between the two end effectors and is installed at a position where information on an operable area by the two selected end effectors is acquired. The robot remote operation control system according to claim 1 .

3. the second selection unit selects the environmental sensor in accordance with the operator's hand and viewpoint; the setting unit sets the coordinate system in accordance with the operator's hand operating and viewpoint.

3. The robot remote operation control system according to claim 1 or 2.

4. a control unit that calculates a hand coordinate of the end effector based on the coordinate system switched by the setting unit and controls the end effector, 3. The robot remote operation control system according to claim 1 or 2.

5. an intention estimation unit that estimates a target object and a task content based on the movement of the operator; the control unit controls each of the end effectors to be controlled using the estimation result obtained by the intention estimation unit.

5. The robot remote operation control system according to claim 4.

6. the control unit controls the operation of the end effector to support the operation of the operator, based on the operation intention estimated by the intention estimation unit. The robot remote operation control system according to claim 5.

7. A control device that recognizes a movement of an operator and transmits the movement of the operator to a robot having a plurality of end effectors and a movement mechanism that moves the end effectors, thereby operating the robot, a plurality of environmental sensors for acquiring information about the surrounding environment; a first selection unit that selects the end effector that transmits the movement of the operator; a second selection unit that selects the environmental sensor in accordance with the end effector selected by the first selection unit; a setting unit that switches a coordinate system based on the environmental sensor when the end effector is changed so that a moving direction of the end effector coincides with a movement of the operator; A robot remote control device comprising:

8. A control method for a control device that recognizes a movement of an operator and transmits the movement of the operator to a robot having a plurality of end effectors and a movement mechanism that moves the end effectors to operate the robot, comprising: an acquisition unit acquires information about the surrounding environment acquired by a plurality of environmental sensors; a first selection unit selecting the end effector that transmits the movement of the operator; a second selection unit selecting the environmental sensor in accordance with the end effector selected by the first selection unit; a setting unit that switches a coordinate system based on the environmental sensor when the end effector is changed; A method for remotely controlling a robot.

9. a computer of a control device that recognizes a movement of an operator and transmits the movement of the operator to a robot having a plurality of end effectors and a movement mechanism that moves the end effectors, thereby operating the robot; Acquire information about the surrounding environment acquired by a plurality of environmental sensors; selecting the end effector that transmits the operator's movement; selecting the environmental sensor in response to the selected end effector; When the end effector is changed, the coordinate system based on the environmental sensor is switched so that the movement direction of the end effector coincides with the movement of the operator. program.

Images