Remote control system, remote control device, remote control method, and program

Abstract

This remote control system remotely controls an end effector which has a finger portion with joints, and comprises: a first detection unit for detecting a fingertip position of an operator; a second detection unit for detecting a joint angle of a finger of the operator; a determination unit for determining the operation content of the operator; a priority decision unit for deciding whether priority should be given to the fingertip position of the operator or the joint angle of the operator when calculating a joint angle of the finger portion of the end effector, in accordance with the operation content determined by the determination unit; a calculation unit for performing a joint angle optimization calculation including the priority determined by the priority decision unit; and a control unit for outputting the joint angle calculated by the calculation unit as a control command to the end effector.

Description

Remote control system, remote control device, remote control method, and program

[0001] This application claims priority to Japanese Patent Application No. 2024-044918, filed March 21, 2024, the contents of which are incorporated herein by reference.

[0002] Systems are being developed in which an operator remotely controls a robot to perform a task. The robot performing the task may have, for example, an arm and a robotic hand. Furthermore, the robotic hand may have two or more fingers. In such systems, it has been difficult to operate an object with a multi-fingered hand in the same way as a human hand. This is because the shapes and degrees of freedom of a robotic hand and a human hand are different, and simply mapping the movements of a human hand to a robotic hand as inputs does not allow for operation. For this reason, it has been proposed to appropriately map the shape of a human hand to a robotic hand (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2023-131033

[0004] However, when manipulating objects with a human hand and a robot, there is no comprehensive mapping method that matches multiple taxonomies. A taxonomy, for example, is a classification of grasping. Therefore, with conventional technology, the operability of a multi-fingered robotic hand can be poor.

[0005] The aspects of the present invention have been made in consideration of the above-mentioned problems, and aim to provide a remote control system, a remote control device, a remote control method, and a program that can improve the operation of a robot's multi-fingered hand.

[0006] In order to solve the above problems, the present invention employs the following aspects: (1) A remote control system according to one aspect of the present invention is a control system in which an operator remotely operates an end effector having fingers with joints, the remote control system including: a first detection unit that detects a fingertip position of the operator; a second detection unit that detects a joint angle of the finger of the operator; a determination unit that determines an operation content of the operator; a priority determination unit that determines whether a priority should be assigned to the fingertip position of the operator or a joint angle of the operator when calculating a joint angle of the finger of the end effector according to the operation content determined by the determination unit; a calculation unit that performs an optimization calculation of the joint angle including the priority determined by the priority determination unit; and a control unit that outputs the joint angle calculated by the calculation unit to the end effector as a control command.

[0007] (2) In the above aspect (1), the calculation unit may calculate a first cost function regarding a relationship between the end effector and the positions of the fingertips of each of the operators based on the priority, and the control unit may perform an optimization calculation based on the calculated first cost function to calculate joint angles of the fingers of the end effector.

[0008] (3) In the above aspect (1), the calculation unit may calculate a second cost function regarding the relationship between the end effector and angles of each joint of each finger of the operator based on the priority, and the control unit may perform optimization calculations based on the calculated second cost function to calculate joint angles of the finger portions of the end effector.

[0009] (4) In the above aspect (3), the calculation unit may calculate the second cost function by multiplying either the joint angle of the finger portion of the end effector or the joint angle of the finger of the operator by a weight related to the bending of the end effector and the finger of the operator.

[0010] (5) In any one of the above aspects (2) to (4), the first detection unit may detect distances between the fingers of the operator, the priority determination unit may calculate a third cost function related to the distances between the end effector and each of the fingers of the operator, and the control unit may perform optimization calculations using the third cost function to calculate joint angles of the fingers of the end effector.

[0011] (6) In any one of the above aspects (1) to (5), the first detection unit may detect the position of the operator's fingertip based on the distance from the center of the operator's palm to the fingertip, and the shape of the operator's hand and fingers at the time of detecting the position of the operator's fingertip may be a predetermined shape when gripping.

[0012] (7) A remote control device according to one aspect of the present invention is a device for remotely operating an end effector having fingers with joints when an operator remotely operates the end effector, and includes: a first detection unit that detects the position of the operator's fingertip; a second detection unit that detects the joint angles of the operator's fingers; a determination unit that determines the operation content of the operator; a priority determination unit that determines whether the priority to be given when calculating the joint angles of the fingers of the end effector is the position of the operator's fingertip or the joint angle of the operator, depending on the operation content determined by the determination unit; a calculation unit that performs optimization calculation of the joint angle including the priority determined by the priority determination unit; and a control unit that outputs the joint angle calculated by the calculation unit to the end effector as a control command.

[0013] (8) A remote control method according to one aspect of the present invention is a method for remotely operating an end effector having fingers with joints when an operator remotely operates the end effector, the remote control method including: a first detection unit detecting a fingertip position of the operator; a second detection unit detecting a joint angle of the operator's finger; a determination unit determining an operation content of the operator; a priority determination unit determining whether a priority should be assigned to the fingertip position of the operator or the joint angle of the operator when calculating a joint angle of the finger of the end effector according to the determined operation content; a calculation unit performing an optimization calculation of the joint angle including the determined priority; and a control unit outputting the calculated joint angle to the end effector as a control command.

[0014] (9) A program according to one aspect of the present invention is a program that, when an operator remotely controls an end effector having fingers with joints, causes a computer of a remote control device that remotely controls the end effector to detect the positions of the operator's fingertips, detect the joint angles of the operator's fingers, determine the operation content of the operator, determine whether the priority to be given to the position of the operator's fingertip or the joint angle of the operator when calculating the joint angle of the finger of the end effector based on the determined operation content, perform an optimization calculation of the joint angle including the determined priority, and output the calculated joint angle to the end effector as a control command.

[0015] According to the aspects of the present invention, the operation of a multi-fingered hand of a robot can be improved.

[0016] FIG. 1 is a diagram for explaining an overview of remote control and operation of a robot. FIG. 2 is a diagram showing examples of classification of grasps according to the GRASP classification method. FIG. 3 is a diagram showing an example of a task using fingertips and an example of a task using the finger pads and palm. FIG. 4 is a diagram showing an example of the configuration of a remote control system according to the present embodiment. FIG. 5 is a diagram showing an example of information stored in a storage unit. FIG. 6 is a diagram showing a first example of mapping of fingertip positions. FIG. 7 is a diagram showing a second example of mapping of fingertip positions. FIG. 8 is a diagram showing an example of mapping of joint angles. FIG. 9 is a flowchart of the processing procedure of a remote control device according to the embodiment.

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component has been appropriately changed so that each component can be recognized. In all drawings used to explain the embodiments, components having the same function are designated by the same reference numerals, and repeated explanations will be omitted. In addition, "based on XX" in this application means "based on at least XX" and includes cases where the component is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where the component is based on XX after calculation or processing. "XX" is any element (for example, any information).

[0018] [Outline of Remote Control and Control Body] First, an overview of remote control and operation will be described. FIG. 1 is a diagram for explaining an overview of remote control and operation of a robot. As shown in FIG. 1, in a remote operation space, an operator Us, for example, wears an image display device 4 on his / her head and an operation input unit 5 (5L, 5R) on his / her hand. An environmental sensor 3 is installed in the robot workspace. Note that the environmental sensor 3 may be attached to the robot 2. The robot 2 also includes an arm 21 (21L, 21R) and an end effector 22 (22L, 22R) (robot hand). Note that the end effector 22 has two or more fingers.

[0019] In the example of Fig. 1 , the target object obj is, for example, a plastic bottle or a glass bottle, and includes a body and a lid. The operator Us remotely controls the robot 2 to manipulate the target object obj by, for example, moving the hand or fingers wearing the operation input unit 5 while viewing an image displayed on the HMD 4. In Fig. 1 , an example of the operation is to attach and close the lid cap to the body, or to open the lid cap from the body. In such a task, the robot 2 needs to grasp the body with its palm and pinch the cap with its fingers.

[0020] [Taxonomy] Next, we will explain the taxonomy of grasping. In object grasping and manipulation, grasps are classified according to the required task, the distribution and size of the grasp records (see, for example, Reference 1). Figure 2 is a diagram showing an example of grasp classification using the GRASP classification method. As shown in Figure 2, grasps are classified in the column direction according to allocation to power grasp, intermediate grasp, and precision grasp, interpersonal relationship, and virtual finger allocation. In addition, column classification is performed according to the position of the thumb, and the thumb is abducted or adducted. Note that the grasp classification of the taxonomy shown in Figure 2 is an example and is not limited to this.

[0021] Reference 1: Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77

[0022] FIG. 3 shows examples of tasks using fingertips and tasks using the finger pads and palm. The image designated by reference symbol g10 is an example of a task using fingertips. For example, a screw, which is a first target object obj1, is picked up with the fingertips and moved to a second target object obj2 to which the screw is to be attached. Such object manipulation performed at points using the fingertips is classified as Palmar Pinch (No. 9 in FIG. 2 ) in taxonomy. The image designated by reference symbol g20 is an example of a task using the finger pads and palm. The object object obj, which is a sphere, is grasped with the finger pads and grasped with the palm. Such object manipulation performed at surfaces using the finger pads is classified as Tip Pinch (No. 24 in FIG. 2 ) in taxonomy. In Palmar Pinch and Tip Pinch, the human joint angles are approximately the same, but the object manipulation methods are different.

[0023] For this reason, in this embodiment, the mapping method to be applied is switched depending on the work state, work content, and the like.

[0024] [Remote Control System] Next, an example configuration of the remote control system 1 will be described. FIG. 4 is a diagram showing an example configuration of the remote control system according to this embodiment. The remote control system 1 includes, for example, a robot 2, an environmental sensor 3, an image display device 4, an operation input unit 5, a remote control device 6, and a measurement sensor 7. The robot 2 includes, for example, an arm 21, an end effector 22, a sensor 23, a drive unit 24, and a communication unit 25. The image display device 4 includes, for example, a display unit 41 and a gaze detection unit 42. The operation input unit 5 includes, for example, an operation detection unit 51. The remote control device 6 includes, for example, an acquisition unit 61, a measurement unit 62 (first detection unit, second detection unit), a judgment unit 63, a priority determination unit 64, a calculation unit 65, a control unit 66, an image generation unit 67, an output unit 68, and a memory unit 69.

[0025] The environmental sensor 3 is, for example, an RGB (red, green, blue)-D imaging device that can also acquire depth D information. The environmental sensor 3 is installed in the robot workspace. The environmental sensor 3 may also be installed in the remote control space. The environmental sensor 3 may also be a combination of a distance sensor and an RGB imaging device. The environmental sensor 3 is connected to the remote control device 6 via a wired or wireless network. The environmental sensor 3 includes a communication unit (not shown) and outputs acquired data to the remote control device 6.

[0026] The image display device 4 is, for example, an HMD (head mounted display). The image display device 4 and the remote control device 6 are connected via a wired or wireless network. The display unit 41 displays images required for remote operation in the robot workspace output by the remote control device 6. The gaze detection unit 42 detects the operator's gaze. Note that the image display device 4 does not necessarily have to include the gaze detection unit 42. The image display device 4 acquires images from the remote control device 6 and outputs the detected gaze information to the remote control device 6.

[0027] The operation input unit 5 and the remote control device 6 are connected via a wired or wireless network. The operation detection unit 51 is, for example, a data glove, and detects the movement and position of the operator's hand and fingers. The operation input unit 5 also includes a communication unit (not shown) and outputs detected data to the remote control device 6.

[0028] The measurement sensor 7 is, for example, an RGB (red, green, blue)-D imaging device that can also acquire depth D information, but the measurement sensor 7 may also be a combination of a distance sensor and an RGB imaging device. The measurement sensor 7 is installed, for example, in the remote control space. The measurement sensor 7 measures data about the operator's hand (such as finger length and fingertip spacing). The measurement sensor 7 is connected to the remote control device 6 via a wired or wireless network. The measurement sensor 74 includes a communication unit (not shown) and outputs the measured measurement data to the remote control device 6.

[0029] (Robot) The robot 2 includes at least an arm 21 and an end effector 22. The robot 2 may be a double-armed robot, or a walking bipedal robot, and may include a head, a body, etc. The robot 2 is connected to a remote control device 6 via a wired or wireless network.

[0030] One end of the arm 21 is connected to, for example, the shoulder via a joint, and the other end is connected to the end effector 22 via a joint.

[0031] The end effector 22 is a multi-fingered hand equipped with two or more fingers 221 (221-1, ..., 221-n (n is an integer of 2 or more)). In the following example, an example with five fingers 221 will be described, but the number of fingers 221 is not limited to this and may be two or more. Each finger 221 has at least two joints.

[0032] The sensor 23 is attached to each joint of the arm 21 and the end effector 22. The sensor 23 is, for example, an encoder, an acceleration sensor, a force sensor, a six-axis sensor, or the like.

[0033] The driving unit 24 drives the arm 21 and the end effector 22 in response to a control command received from the remote control device 6. An actuator is attached to each joint of the arm 21 and the end effector 22.

[0034] The communication unit 25 receives a control command from the remote control device 6 and outputs the detection value of the sensor 23 to the remote control device 6 .

[0035] (Remote Control Device) The remote control device 6 operates the robot 2 using information acquired from the environmental sensor 3 , the operation input unit 5 and the measurement sensor 7 .

[0036] The acquisition unit 61 acquires measurement data from the measurement sensor 7. The acquisition unit 61 acquires data detected by the sensor 23 from the robot 2. The acquisition unit 61 acquires data detected from the environmental sensor 3. The acquisition unit 61 acquires information related to the detected operation from the operation input unit 5.

[0037] The measurement unit 62 acquires measurement data, for example, before the start of remote operation, and uses the acquired measurement data to measure the length of the operator's fingers, the distance between the fingertips when the hand is open, etc. Note that these measurements may be performed in advance and stored in the storage unit 69. Furthermore, during measurement, the operator, for example, extends his or her fingers and faces the palm side toward the measurement sensor 7 to perform the measurement. Note that the length of the fingers to be measured, etc. will be described later. The measurement unit 62 measures the distance between the operator's fingers using the measurement values ​​of the measurement sensor 7, and measures the fingertip positions of each finger.

[0038] The determination unit 63 determines the work content that the operator is about to perform. The work content may be, for example, delicate work using the fingertips, work using the finger pads and palm, etc. The determination unit 63 estimates the operation target and the operation intention based on the information acquired from the environmental sensor 3 and the operation instruction acquired from the operation input unit 5, for example, using a method described in Japanese Patent Application Laid-Open No. 2022-157101.

[0039] The priority determination unit 64 determines whether to give priority to the operator's fingertip position or the operator's joint angle when calculating the joint angle of the finger of the end effector 22 according to the determination result of the determination unit 63 .

[0040] For example, in the case of a task using the fingertips, the calculation unit 65 calculates weights (e.g., cost functions) related to the fingertip positions from the fingertip positions of the operator. For example, in the case of a task using the finger pads and palms, the calculation unit 65 calculates weights (e.g., cost functions) related to the joint angles from the joint angles of the operator's fingers. Note that the weights may be calculated by the priority determination unit 64 according to the priorities. The calculation unit 65 performs optimization calculations to calculate the joint angles for the robot 2.

[0041] The control unit 66 generates a control command using the joint angles of the robot 2 calculated by the calculation unit 65, and outputs the generated control command to the robot 2 via the output unit 68. Note that the control unit 66 may perform control to support the operation of the operator based on the estimated operation intention, for example, using the method described in Japanese Patent Application No. 2023-045616.

[0042] The image generation unit 67 generates an image required for remote control to be displayed on the image display device 4, for example, by using data acquired from the environmental sensor 3. The image required for remote control is, for example, an image including a finger and a target object. The image generation unit 67 outputs the generated image data to the image display device 4 via the output unit 68.

[0043] The output unit 68 outputs the image data generated by the image generation unit 67 to the image display device 4. The output unit 68 outputs the control command generated by the control unit 66 to the robot 2.

[0044] The storage unit 69 stores programs, thresholds, formulas, etc. used by each unit of the remote control device 6. The storage unit 69 stores mapping data in association with the tree-specific information, as shown in Fig. 5. Fig. 5 is a diagram showing an example of information stored in the storage unit.

[0045] 4 is an example, and is not limited to this. For example, each device has a power supply.

[0046] [Mapping] Next, mapping will be described. FIG. 6 is a diagram showing a first example of mapping of fingertip positions. In the example of FIG. 6, the fingertip positions are represented as distances from the origin O of the palm. The origin O of the palm position may be arbitrarily set by the measurement unit 62, or may be set by the measurement unit 62 by determining the center of the palm. The shape of the operator's hand during measurement as shown in FIG. 6 may be determined in advance and displayed, for example, on the image display device 4. The hand shape during measurement is, for example, the shape of the hand when grasping an object, with the thumb and index finger facing each other. In FIG. 6, drOI (I = 1, ..., 5) is the distance from the origin of the robot's palm to each finger. dhOI (I = 1, ..., 5) is the distance from the origin of the operator's palm to each finger.

[0047] The shape and size of the hand of the operator and the robot 2 are different. Therefore, the measurement unit 62 measures dhOI relating to the length of the operator's finger and drOI relating to the length of the robot 2's finger using the measurement values ​​of the measurement sensor 7. Then, the measurement unit 62 calculates the first cost function Cost ｄｏ The measurement unit 62 may calculate the first cost function by calculating the difference between the distance from the palm origin of the robot 2 to each finger and the distance from the palm origin of the operator's finger, and then summing up the differences between the fingers. Alternatively, the measurement unit 62 may calculate the ratio between the distance from the palm origin of the robot 2 to each finger and the distance from the palm origin of the operator's finger, and then averaging the ratios between the fingers.

[0048] FIG. 7 is a diagram illustrating a second example of mapping fingertip positions. The example in FIG. 7 is an example of measuring the distance between each finger. In FIG. 7, drTI (I = 1, ..., 5) is the distance between each finger of the robot. dhTI (I = 1, ..., 5) is the distance between each finger of the operator. Note that in the example in FIG. 7, the distances between each finger are shown as the distances from the tip of the thumb to the tip of the index finger, the tip of the thumb to the tip of the middle finger, the tip of the thumb to the tip of the ring finger, the tip of the thumb to the tip of the little finger, the tip of the middle finger to the tip of the index finger, the tip of the ring finger to the tip of the middle finger, and the tip of the little finger to the tip of the ring finger. However, different fingers may be used as the reference for measuring the distance. For example, "from the tip of the middle finger to the tip of the index finger" may be "from the tip of the index finger to the tip of the middle finger," as long as it is the distance between the fingertips.

[0049] The measurement unit 62 measures dhTI, which is the distance between the fingers of the operator, and drTI of the robot 2, using the measurement values ​​of the measurement sensor 7. Then, the measurement unit 62 calculates a third cost function Cost ｄｆ The measurement unit 62 may calculate the third cost function by calculating the difference between the distance between each finger of the robot 2 and the distance between each finger of the operator, and summing up the differences between the distances between each finger. Alternatively, the measurement unit 62 may calculate the ratio between the distance between each finger of the robot 2 and the distance between each finger of the operator, and averaging the ratio of the distances between each finger.

[0050] FIG. 8 is a diagram showing an example of mapping of joint angles. The example in FIG. 8 shows the shape of the hand and fingers when pinching an object with the thumb and index finger. The shape of the operator's hand during measurement as in FIG. 8 may be determined in advance and displayed on the image display device 4, for example. The measurement unit 62 uses the results measured by the measurement sensor 7 to determine the angle of each joint of each finger. For example, the joint angle θ ｈＩ０ is the angle of the third joint of the operator's index finger, and the joint angle θ ｈＩ１ is the angle of the second joint of the operator's index finger, and the joint angle θ ｈＩ３ is the angle of the first joint of the operator's index finger. ｒＩ０ is the angle of the third joint of the robot's index finger, and the joint angle θ ｒＩ１is the angle of the second joint of the robot's index finger, and the joint angle θ ｒＩ３ is the angle of the first joint of the robot's index finger.

[0051] The measurement unit 62 measures each joint angle of the operator and each joint angle of the robot 2 using the measurement values ​​of the measurement sensor 7. Then, the measurement unit 62 calculates a second cost function Cost ｆｓ The measurement unit 62 may calculate the second cost function by calculating the difference between the joint angle of each finger of the robot 2 and the joint angle of each finger of the operator, and then summing up the differences between the joint angles of each finger. Alternatively, the measurement unit 62 may calculate the second cost function by calculating the ratio between the joint angle of each finger of the robot 2 and the joint angle of each finger of the operator, and then averaging the ratios between the joint angles of each finger. The measurement unit 62 may calculate the second cost function by multiplying the joint angles of the robot 2 or the joint angles of the operator by a weight that indicates a difference in ease of bending the fingers of the robot 2 and the operator.

[0052] [Processing Procedure] Next, an example of a processing procedure when performing a task by remote control will be described. In the following examples, examples of tasks that involve "tasks using fingertips" and "tasks using finger pads and palms" will be described. Fig. 9 is a flowchart of the processing procedure of the remote control device according to this embodiment.

[0053] (Step S1) The measurement unit 62 measures the distance between the operator's fingers using the measurement values ​​of the measurement sensor 7, and measures the fingertip position of each finger. The measurement unit 62 measures each joint angle of each finger using the measurement values ​​of the measurement sensor 7. Note that these processes may be performed in advance and stored in the storage unit 69. The measurement unit 62 may also measure the fingertip positions and joint angles of the robot 2 when measuring the operator's fingertip positions and joint angles, or may measure them in advance and store them in the storage unit 69. Next, an example of a processing procedure for a task performed by remote control will be described. Note that the following examples will explain examples of tasks involving "tasks using fingertips" and "tasks using finger pads and palms." FIG. 9 is a flowchart of the processing procedure of the remote control device according to this embodiment.

[0054] (Step S2) The determination unit 63 estimates and acquires the work content using the data acquired from the environmental sensor 3. The work content may be selected or input by the operator by operating the operation input unit 5 or the like. In this case, the acquisition unit 61 may acquire the selected or input work content. For example, the selection may be made by displaying work content candidates on the display unit 41 of the image display device 4 and selecting by line of sight. Alternatively, the operator may input or select the work content by operating a keyboard (not shown) or the like connected to the remote control device 6.

[0055] (Step S3) The determination unit 63 determines whether the work content is, for example, a "work using fingertips" or a "work using finger pads and palms." If the work content is a "work using fingertips," the determination unit 63 proceeds to processing in step S4, and if the work content is a "work using finger pads and palms," the determination unit 63 proceeds to processing in step S5.

[0056] (Step S4) The priority determination unit 64 determines the fingertip positions of the operator as the priority when calculating the joint angles of the fingers of the end effector, based on the determination result of the determination unit 63. Next, the calculation unit 65 calculates weights (e.g., a second cost function) related to the fingertip positions from the fingertip positions of the operator. Alternatively, the calculation unit 65 calculates weights (e.g., a third cost function) related to the distance between the fingers of the operator. After this processing, the calculation unit 65 proceeds to the processing of step S6.

[0057] (Step S5) The priority determination unit 64 determines the priority of the joint angles of the operator when calculating the joint angles of the fingers of the end effector according to the determination result of the determination unit 63. Next, the calculation unit 65 calculates weights (e.g., a second cost function) related to the joint angles from the joint angles of the fingers of the operator. After this processing, the calculation unit 65 proceeds to the processing of step S6.

[0058] (Step S6) The calculation unit 65 uses the weights obtained in step S6 or step S7 to perform, for example, optimization calculations to calculate the joint angles of the robot 2.

[0059] (Step S7) The control unit 66 generates a control command for the robot 2 based on the calculated joint angles.

[0060] (Step S8) The control unit 66 outputs the generated control command to the robot 2 via the output unit 68 to control the operation of the robot 2.

[0061] As described above, in this embodiment, the joint angles of the robot 2 are determined by calculating, for example, weights for an evaluation function that prioritizes the operator's fingertip position or weights for an evaluation function that prioritizes the operator's finger joint angles, depending on the task content. The weights are, for example, for the first to third cost functions, respectively, and are, for example, α, β, and γ in the following equation (1). The calculation unit 65 changes and controls the dominant cost function by changing these weights α, β, and γ depending on the task content. The calculation unit 65 then performs optimization calculations to minimize the sum (cost) of the weighted cost functions, thereby determining the joint angles of each finger of the robot 2. For example, in the case of a pinching task using the fingertips, the weight β or γ, or the weights β and γ, are changed. Furthermore, in the case of a task using the palm, the weight α is changed.

[0062]

[0063] The processes of steps S2 to S8 among the above processes are performed, for example, at predetermined time intervals, and the weight used when the work content is switched is changed. The process of step S1 may also be performed at predetermined time intervals.

[0064] In the above example, the weight is switched depending on whether the task is "task using fingertips" or "task using finger pads and palms," but this is not limiting. The task is not limited to "task using fingertips" and "task using finger pads and palms," and other task content as shown in FIG. 2 may be used, and the number of tasks to be switched is not limited to two, but may be three or more.

[0065] As described above, in this embodiment, the weights of the fingertip positions and joint angles are changed when calculating the joint angles of the robot 2 depending on the operation content.

[0066] In the prior art, since the fingertip positions and taxonomies roughly match, grasping using the fingertips, such as Tip Pinch, was possible, but grasping using the finger pads or palm was difficult. In contrast, according to the present embodiment, object manipulation using the fingertips and object manipulation using the finger pads and palm can be simultaneously achieved. Furthermore, according to the present embodiment, object manipulation using the fingertips and object manipulation using the finger pads and palm can be simultaneously achieved, thereby improving the operability of a robot's multi-fingered hand.

[0067] In the above example, the robot 2 has one arm, but it may have two arms. In such a case, the arm and end effector to be used may be selected depending on the task and controlled using the above-described method, or each arm may be controlled using the above-described method so that the two end effectors work in coordination.

[0068] A program for implementing all or part of the functions of the remote control device 6 of the present invention may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed to perform all or part of the processing performed by the remote control device 6. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. It also includes a WWW system equipped with a website provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as floppy disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. Alternatively, some or all of these components may be realized by hardware (including circuitry) using LSIs (Large Scale Integration) such as ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), GPUs (Graphics Processing Units), and SOCs (System On Chips), or may be realized by a combination of software and hardware.

[0069] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0070] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0071] 1...Remote control system, 2...Robot, 3...Environment sensor, 4...Image display device, 5...Operation input unit, 6...Remote control device, 7...Measurement sensor, 21...Arm, 22...End effector, 23...Sensor, 24...Drive unit, 25...Communication unit, 41...Display unit, 42...Gaze detection unit, 51...Operation detection unit, 61...Acquisition unit, 62...Measurement unit, 63...Judgment unit, 64...Priority determination unit, 65...Calculation unit, 66...Control unit, 67...Image generation unit, 68...Output unit, 69...Storage unit

Claims

1. A control system in which an operator remotely controls an end effector having fingers with joints, comprising: a first detection unit that detects the position of the operator's fingertip; a second detection unit that detects the joint angles of the operator's fingers; a judgment unit that judges the operation content of the operator; a priority determination unit that determines whether the priority to be given when calculating the joint angles of the fingers of the end effector is the position of the operator's fingertip or the joint angles of the operator according to the operation content judged by the judgment unit; a calculation unit that performs optimization calculations of the joint angles including the priorities determined by the priority determination unit; and a control unit that outputs the joint angles calculated by the calculation unit to the end effector as control commands.

2. The remote control system according to claim 1, wherein the calculation unit calculates a first cost function relating to the relationship between the end effector and the positions of the fingertips of each of the operators based on the priority, and the control unit performs optimization calculations based on the calculated first cost function to calculate the joint angles of the fingers of the end effector.

3. The remote control system according to claim 1, wherein the calculation unit calculates a second cost function relating to the relationship between the angle of each joint of the end effector and each of the operators' fingers based on the priority, and the control unit performs optimization calculations based on the calculated second cost function to calculate the joint angles of the fingers of the end effector.

4. The remote control system according to claim 3, wherein the calculation unit calculates the second cost function by multiplying either the joint angle of the finger portion of the end effector or the joint angle of the operator's finger by a weight related to the bending of the end effector and the operator's finger.

5. A remote control system as described in claim 2 or claim 3, wherein the first detection unit detects the distance between each of the operator's fingers, the priority determination unit calculates a third cost function relating to the distance between the end effector and each of the operator's fingers, and the control unit performs optimization calculations using the third cost function as well to calculate the joint angles of the fingers of the end effector.

6. A remote control system as described in claim 1 or claim 2, wherein the first detection unit detects the position of the operator's fingertip based on the distance from the center of the operator's palm to the fingertip, and the shape of the operator's hand and fingers when detecting the position of the operator's fingertip is a predetermined shape when gripping.

7. A remote control device for remotely operating an end effector having fingers with joints when the end effector is remotely operated by an operator, the remote control device comprising: a first detection unit that detects the position of the operator's fingertip; a second detection unit that detects the joint angles of the operator's fingers; a judgment unit that judges the operation content of the operator; a priority determination unit that determines whether the priority to be given when calculating the joint angles of the fingers of the end effector is the position of the operator's fingertip or the joint angle of the operator according to the operation content judged by the judgment unit; a calculation unit that performs optimization calculation of the joint angle including the priority determined by the priority determination unit; and a control unit that outputs the joint angle calculated by the calculation unit to the end effector as a control command.

8. A method for remotely operating an end effector having fingers with joints when an operator remotely operates the end effector, comprising: a first detection unit detects the position of the operator's fingertip; a second detection unit detects the joint angles of the operator's fingers; a judgment unit judges the operation content of the operator; a priority determination unit determines whether the priority to be given when calculating the joint angles of the fingers of the end effector is the position of the operator's fingertip or the joint angles of the operator according to the judged operation content; a calculation unit performs optimization calculation of the joint angles including the determined priority; and a control unit outputs the calculated joint angles to the end effector as control commands.

9. A program that, when an operator remotely controls an end effector having fingers with joints, causes a computer of a remote control device that remotely controls the end effector to detect the position of the operator's fingertip, detect the joint angles of the operator's fingers, determine the operation content of the operator, determine whether the priority to be given when calculating the joint angles of the fingers of the end effector should be the position of the operator's fingertip or the joint angle of the operator according to the determined operation content, perform an optimization calculation of the joint angle including the determined priority, and output the calculated joint angle to the end effector as a control command.

Images