Remote control system, remote control device, remote control method, and program

Abstract

To provide a remote control system, a remote control device, a remote control method that can improve operations of a multi-finger hand of a robot.SOLUTION: A remote control system is a control system in which an operator performs a remote operation of an end effector with a finger section having a joint. The remote control system includes: a first detection section that detects a fingertip position of the operator; a second detection section that detects a joint angle of a finger of the operator; a determination section that determines operation content of the operator; a priority determination section that determines which of the fingertip position of the operator or the joint angle of the operator is set as a priority when calculating the joint angle of the finger section of the end effector according to the operation content determined by the determination section; a calculation section that performs optimization calculation of the joint angle including the priority determined by the priority determination section; and a control section that outputs the joint angle calculated by the calculation section as a control command to the end effector.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a remote control system, a remote control device, a remote control method, and a program. [Background technology]

[0002] Systems are being developed that allow operators to remotely control robots to perform tasks. The robots that perform tasks have, for example, an arm and a robotic hand. Furthermore, the robotic hand may have two or more fingers. In such systems, it has been difficult to operate an object with a multi-fingered hand in the same way as a human hand. This is because the shapes and degrees of freedom of a robotic hand and a human hand are different, and therefore operation is not possible even if the movements of a human hand are simply input and mapped to the robotic hand. For this reason, it has been proposed to appropriately map the shape of a human hand onto a robotic hand (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-131033 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when manipulating objects with a human hand and a robot, there is no comprehensive mapping method that is compatible with multiple taxonomies. A taxonomy, for example, is a classification of grasping. For this reason, conventional techniques have sometimes resulted in poor operability for multi-fingered robotic hands.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a remote control system, a remote control device, a remote control method, and a program that can improve the operation of a robot's multi-fingered hand. [Means for solving the problem]

[0006] (1) In order to achieve the above object, a remote control system according to one aspect of the present invention is a control system in which an operator remotely controls an end effector having fingers with joints, the remote control system including: a first detection unit that detects the position of the operator's fingertip; a second detection unit that detects the joint angles of the operator's fingers; a determination unit that determines the operation content of the operator; a priority determination unit that determines whether the priority to be given when calculating the joint angles of the fingers of the end effector is the position of the operator's fingertip or the joint angles of the operator, depending on the operation content determined by the determination unit; a calculation unit that performs optimization calculation of the joint angles including the priority determined by the priority determination unit; and a control unit that outputs the joint angles calculated by the calculation unit to the end effector as control commands.

[0007] (2) In the remote control system according to one aspect of (1) above, the calculation unit may calculate a first cost function relating to a relationship between the end effector and the positions of the fingertips of each of the operators based on the priority, and the control unit may perform an optimization calculation based on the calculated first cost function to calculate joint angles of the fingers of the end effector.

[0008] (3) In the remote control system according to one aspect of (1) above, the calculation unit may calculate a second cost function relating to the relationship between the angle of each joint of the end effector and the angle of each joint of each finger of the operator based on the priority, and the control unit may perform an optimization calculation based on the calculated second cost function to calculate the joint angles of the finger parts of the end effector.

[0009] (4) In the remote control system according to one aspect of (3) above, the calculation unit may calculate the second cost function by multiplying either the joint angle of the finger portion of the end effector or the joint angle of the operator's finger by a weight related to the bending of the end effector and the operator's finger.

[0010] (5) In the remote control system according to any one of the above (2) to (4), the first detection unit may detect the distance between each of the fingers of the operator, the priority determination unit may calculate a third cost function relating to the distance between the end effector and each of the fingers of the operator, and the control unit may perform an optimization calculation using the third cost function to calculate the joint angles of the fingers of the end effector.

[0011] (6) In a remote control system according to any one of the above aspects (1) to (5), the first detection unit may detect the position of the operator's fingertip based on the distance from the center of the operator's palm to the fingertip, and the shape of the operator's hand and fingers at the time of detecting the position of the operator's fingertip may be a predetermined shape when gripping.

[0012] (7) In order to achieve the above object, a remote control device according to one embodiment of the present invention is a device for remotely operating an end effector having fingers with joints when an operator remotely operates the end effector, and includes: a first detection unit that detects the position of the operator's fingertip; a second detection unit that detects the joint angles of the operator's fingers; a determination unit that determines the operation content of the operator; a priority determination unit that determines whether the priority to be given when calculating the joint angles of the fingers of the end effector is the position of the operator's fingertip or the joint angle of the operator, depending on the operation content determined by the determination unit; a calculation unit that performs optimization calculation of the joint angle including the priority determined by the priority determination unit; and a control unit that outputs the joint angle calculated by the calculation unit to the end effector as a control command.

[0013] (8) In order to achieve the above object, a remote control method according to one aspect of the present invention is a method for remotely operating an end effector having fingers with joints when an operator remotely operates the end effector, the remote control method including: a first detection unit detecting a fingertip position of the operator; a second detection unit detecting a joint angle of the operator's finger; a determination unit determining an operation content of the operator; a priority determination unit determining whether to prioritize the fingertip position of the operator or the joint angle of the operator when calculating a joint angle of the finger of the end effector according to the determined operation content; a calculation unit performing an optimization calculation of the joint angle including the determined priority; and a control unit outputting the calculated joint angle to the end effector as a control command.

[0014] (9) In order to achieve the above object, one aspect of the present invention provides a program that, when an operator remotely controls an end effector having fingers with joints, causes a computer of a remote control device that remotely controls the end effector to detect the positions of the operator's fingertips, detect the joint angles of the operator's fingers, determine the operation content of the operator, determine whether the priority should be the position of the operator's fingertip or the joint angle of the operator when calculating the joint angle of the finger of the end effector based on the determined operation content, perform an optimization calculation of the joint angle including the determined priority, and output the calculated joint angle to the end effector as a control command. [Effects of the Invention]

[0015] According to the above (1) to (9), the operation of the multi-fingered hand of the robot can be improved. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram for explaining an overview of remote control and operation of a robot. [Figure 2] 1A and 1B are diagrams showing examples of grasp classification according to the GRASP classification method. [Figure 3] 1A and 1B are diagrams showing examples of work using fingertips and examples of work using finger pads and the palm. [Figure 4] 1 is a diagram illustrating an example of the configuration of a remote control system according to an embodiment of the present invention. [Figure 5] FIG. 4 is a diagram illustrating an example of information stored in a storage unit. [Figure 6] FIG. 10 is a diagram showing a first mapping example of fingertip positions. [Figure 7] FIG. 10 is a diagram showing a second mapping example of fingertip positions. [Figure 8] FIG. 10 is a diagram illustrating an example of mapping of joint angles. [Figure 9] 10 is a flowchart of a processing procedure of a remote control device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0018] [Remote control and operating unit overview] First, an overview of remote control and operation will be given. FIG. 1 is a diagram for explaining an overview of remote control and operation of a robot. As shown in FIG. 1, in a remote operation space, an operator Us wears, for example, an image display device 4 on his head and an operation input unit 5 (5L, 5R) on his hand. An environmental sensor 3 is installed in the robot workspace. Note that the environmental sensor 3 may be attached to the robot 2. The robot 2 also includes an arm 21 (21L, 21R) and an end effector 22 (22L, 22R) (robot hand). Note that the end effector 22 has two or more fingers.

[0019] In the example of FIG. 1, the target object obj is, for example, a plastic bottle or a glass bottle, and has a body and a lid. The operator Us remotely controls the robot 2 to operate the target object obj by, for example, moving the hand or fingers wearing the operation input unit 5 while viewing an image displayed on the HMD 4. In Fig. 1, an example of the operation is to attach and close a lid cap to the body body, or to open the lid cap from the body body. In such a task, the robot 2 needs to hold the body body in the palm and pinch the cap with the fingers.

[0020] [Taxonomy] Next, we will explain the taxonomy of grasping. In object grasping and manipulation, grasps are classified according to the required task, the distribution of grasps, and their size (see, for example, Reference 1). Figure 2 shows an example of grasp classification using the GRASP classification method. As shown in Figure 2, grasps are classified in columns according to their assignment to power grasp, intermediate grasp, and precision grasp, as well as their interpersonal and virtual finger assignments. Column classification is also performed according to the position of the thumb, which can be abducted or adducted. Note that the grasp classification of the taxonomy shown in Figure 2 is an example and is not limited to this.

[0021] Reference 1; Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77

[0022] FIG. 3 shows examples of tasks using fingertips and tasks using the pads of the fingers and the palm. The image with the code g10 is an example of a task using the fingertips, for example, picking up a screw, which is a first object obj1, with the fingertips and moving it onto a second object obj2, which is the object to attach the screw to. This kind of object manipulation using the fingertips is classified as a Palmar Pinch (No. 9 in Figure 2) in the taxonomy. The image with code g20 is an example of a task using the finger pads and palm, in which the target object obj, a sphere, is grasped with the finger pads and is being grasped with the palm. This type of object manipulation using the finger pads is classified as a tip pinch (No. 24 in Figure 2) in the taxonomy. In the Palmar Pinch and Tip Pinch, the joint angles of humans are almost the same, but the methods of object manipulation are different.

[0023] For this reason, in this embodiment, the mapping method to be applied is switched depending on the work state, work content, and the like.

[0024] [Remote control system] Next, a description will be given of an example of the configuration of the remote control system 1. Fig. 4 is a diagram showing an example of the configuration of the remote control system according to this embodiment. The remote control system 1 includes, for example, a robot 2, an environmental sensor 3, an image display device 4, an operation input unit 5, a remote control device 6, and a measurement sensor 7. The robot 2 includes, for example, an arm 21, an end effector 22, a sensor 23, a driving unit 24, and a communication unit 25. The image display device 4 includes, for example, a display unit 41 and a line-of-sight detection unit . The operation input unit 5 includes an operation detection unit 51, for example. The remote control device 6 includes, for example, an acquisition unit 61, a measurement unit 62 (first detection unit, second detection unit), a judgment unit 63, a priority determination unit 64, a calculation unit 65, a control unit 66, an image generation unit 67, an output unit 68, and a memory unit 69.

[0025] The environmental sensor 3 is, for example, an RGB (red, green, blue)-D imaging device that can also acquire depth D information. The environmental sensor 3 is installed in the robot workspace. The environmental sensor 3 may also be installed in the remote control space. The environmental sensor 3 may also be a combination of a distance sensor and an RGB imaging device. The environmental sensor 3 is connected to the remote control device 6 via a wired or wireless network. The environmental sensor 3 has a communication unit (not shown) and outputs acquired data to the remote control device 6.

[0026] The image display device 4 is, for example, an HMD (head mounted display). The image display device 4 and the remote control device 6 are connected via a wired or wireless network. The display unit 41 displays images output by the remote control device 6 that are necessary for remote control in the robot workspace. The line-of-sight detection unit detects the line of sight of the operator. Note that the image display device 4 does not necessarily have to include the line-of-sight detection unit . The image display device 4 acquires an image from the remote control device 6 and outputs the detected line-of-sight information to the remote control device 6.

[0027] The operation input unit 5 and the remote control device 6 are connected via a wired or wireless network. The operation detection unit 51 is, for example, a data glove, and detects the movement and position of the operator's hand and fingers. The operation input unit 5 includes a communication unit (not shown) and outputs the detected data to the remote control device 6.

[0028] The measurement sensor 7 is, for example, an RGB (red, green, blue)-D imaging device that can also acquire depth D information, but the measurement sensor 7 may also be a combination of a distance sensor and an RGB imaging device. The measurement sensor 7 is installed, for example, in the remote control space. The measurement sensor 7 measures data about the operator's hand (finger length, spacing between fingertips, etc.). The measurement sensor 7 is connected to the remote control device 6 via a wired or wireless network. The measurement sensor 74 includes a communication unit (not shown) and outputs the measured measurement data to the remote control device 6.

[0029] (robot) The robot 2 includes at least an arm 21 and an end effector 22. The robot 2 may be a double-armed robot, or a walking biped robot, and may include a head, a body, etc. The robot 2 is connected to a remote control device 6 via a wired or wireless network.

[0030] One end of the arm 21 is connected to, for example, the shoulder via a joint, and the other end is connected to the end effector 22 via a joint.

[0031] The end effector 22 is a multi-fingered hand equipped with two or more fingers 221 (221-1, ..., 221-n (n is an integer of 2 or more)). In the following example, an example with five fingers 221 will be described, but the number of fingers 221 is not limited to this and may be two or more. Each finger 221 has at least two joints.

[0032] The sensor 23 is attached to each joint of the arm 21 and the end effector 22. The sensor 23 is, for example, an encoder, an acceleration sensor, a force sensor, a six-axis sensor, or the like.

[0033] The driving unit 24 drives the arm 21 and the end effector 22 in response to a control command received from the remote control device 6. An actuator is attached to each joint of the arm 21 and the end effector 22.

[0034] The communication unit 25 receives a control command from the remote control device 6 and outputs the detection value of the sensor 23 to the remote control device 6.

[0035] (Remote control device) The remote control device 6 operates the robot 2 using information acquired from the environmental sensor 3, the operation input unit 5, and the measurement sensor 7.

[0036] The acquisition unit 61 acquires measurement data from the measurement sensor 7. The acquisition unit 61 acquires data detected by the sensor 23 from the robot 2. The acquisition unit 61 acquires data detected from the environmental sensor 3. The acquisition unit 61 acquires information related to the detected operation from the operation input unit 5.

[0037] The measurement unit 62 acquires measurement data, for example, before the start of remote operation, and uses the acquired measurement data to measure the length of the operator's fingers, the distance between the fingertips when the hand is open, and so on. Note that these measurements may be performed in advance and stored in the storage unit 69. Furthermore, during measurement, the operator, for example, extends his or her fingers and faces the palm side toward the measurement sensor 7 to perform the measurement. Note that the length of the fingers to be measured and so on will be described later. The measurement unit 62 measures the distance between the operator's fingers using the measurement values ​​of the measurement sensor 7, and measures the fingertip positions of each finger.

[0038] The determination unit 63 determines the work content that the operator is about to perform. The work content is, for example, delicate work using the fingertips, work using the finger pads and palm, etc. The determination unit 63 estimates the operation target and the operation intention based on the information acquired from the environmental sensor 3 and the operation instruction acquired from the operation input unit 5, for example, using a method described in JP 2022-157101 A.

[0039] The priority determination unit 64 determines whether the priority to be given when calculating the joint angles of the fingers of the end effector 22 is the fingertip position of the operator or the joint angle of the operator, depending on the determination result of the determination unit 63 .

[0040] For example, in the case of a task using the fingertips, the calculation unit 65 calculates weights (for example, cost functions) related to the fingertip positions from the fingertip positions of the operator. For example, in the case of a task using the pads of the fingers and the palm, the calculation unit 65 calculates weights (for example, cost functions) related to the joint angles from the joint angles of the fingers of the operator. Note that the weights may be calculated by the priority determination unit 64 according to the priorities. The calculation unit 65 performs optimization calculations to calculate the joint angles for the robot 2.

[0041] The control unit 66 generates a control command using the joint angle of the robot 2 calculated by the calculation unit 65, and outputs the generated control command to the robot 2 via the output unit 68. Note that the control unit 66 may perform control to support the operation of the operator based on the estimated operation intention, for example, using the method described in Japanese Patent Application No. 2023-045616.

[0042] The image generation unit 67 generates an image required for remote control to be displayed on the image display device 4, for example, by using data acquired from the environmental sensor 3. The image required for remote control is, for example, an image including a finger and a target object. The image generation unit 67 outputs the generated image data to the image display device 4 via the output unit 68.

[0043] The output unit 68 outputs the image data generated by the image generation unit 67 to the image display device 4. The output unit 68 outputs the control command generated by the control unit 66 to the robot 2.

[0044] The storage unit 69 stores programs, thresholds, formulas, etc. used by each unit of the remote control device 6. The storage unit 69 stores mapping data in association with the tree-specific information, as shown in Fig. 5. Fig. 5 is a diagram showing an example of information stored in the storage unit.

[0045] 4 is an example, and is not limited to this. For example, each device has a power supply.

[0046] [mapping] Next, mapping will be described. FIG. 6 is a diagram showing a first mapping example of fingertip positions. In the example of FIG. 6, the fingertip positions are represented by the distance from the origin O of the palm. The origin O of the palm position may be arbitrarily set by the measurement unit 62, or may be set by the measurement unit 62 by determining the center of the palm. The shape of the operator's hand during measurement as shown in FIG. 6 may be determined in advance and displayed, for example, on the image display device 4. The hand shape during measurement is, for example, the shape of the hand when grasping an object, with the thumb and index finger facing each other. In FIG. 6, drOI (I = 1, ..., 5) is the distance from the origin of the robot's palm to each finger. dhOI (I = 1, ..., 5) is the distance from the origin of the operator's palm to each finger.

[0047] The shape and size of the hand of the operator and the robot 2 are different. Therefore, the measurement unit 62 measures dhOI relating to the length of the operator's finger and drOI relating to the length of the robot 2's finger using the measurement values ​​of the measurement sensor 7. Then, the measurement unit 62 calculates the first cost function Cost do The measurement unit 62 may calculate the first cost function by calculating the difference between the distance from the palm origin of the robot 2 to each finger and the distance from the palm origin of the operator's finger, and then summing up the differences between the fingers. Alternatively, the measurement unit 62 may calculate the ratio between the distance from the palm origin of the robot 2 to each finger and the distance from the palm origin of the operator's finger, and then averaging the ratios between the fingers.

[0048] FIG. 7 is a diagram showing a second example of mapping fingertip positions. The example in FIG. 7 is an example of measuring the distance between each finger. In FIG. 7, drTI (I=1, . . . , 5) is the distance between each finger of the robot. dhTI (I=1, . . . , 5) is the distance between each finger of the operator. Note that in the example in FIG. 7, the distances between each finger are shown as the distances from the tip of the thumb to the tip of the index finger, from the tip of the thumb to the tip of the middle finger, from the tip of the thumb to the tip of the ring finger, from the tip of the thumb to the tip of the little finger, from the tip of the middle finger to the tip of the index finger, from the tip of the ring finger to the tip of the middle finger, and from the tip of the little finger to the tip of the ring finger. However, different fingers may be used as the reference for measuring the distance. For example, "from the tip of the middle finger to the tip of the index finger" may be "from the tip of the index finger to the tip of the middle finger," as long as it is the distance between the fingertips.

[0049] The measurement unit 62 measures dhTI, which is the distance between the fingers of the operator, and drTI of the robot 2, using the measurement values ​​of the measurement sensor 7. Then, the measurement unit 62 calculates a third cost function Cost df The measurement unit 62 may calculate the third cost function by calculating the difference between the distance between each finger of the robot 2 and the distance between each finger of the operator, and summing up the differences between the distances between each finger. Alternatively, the measurement unit 62 may calculate the ratio between the distance between each finger of the robot 2 and the distance between each finger of the operator, and then averaging the ratio of the distances between each finger.

[0050] Fig. 8 is a diagram showing an example of mapping of joint angles. The example in Fig. 8 shows the shape of the hand and fingers when pinching an object with the thumb and index finger. Note that the shape of the operator's hand during measurement as shown in Fig. 8 may be determined in advance and displayed on the image display device 4, for example. The measurement unit 62 calculates the angle of each joint of each finger using the results of measurement by the measurement sensor 7. For example, the joint angle θ hI0 is the angle of the third joint of the operator's index finger, and the joint angle θ hI1 is the angle of the second joint of the operator's index finger, and the joint angle θ hI3 is the angle of the first joint of the operator's index finger. Also, the joint angle θ rI0is the angle of the third joint of the robot's index finger, and the joint angle θ rI1 is the angle of the second joint of the robot's index finger, and the joint angle θ rI3 is the angle of the first joint of the robot's index finger.

[0051] The measurement unit 62 measures each joint angle of the operator and each joint angle of the robot 2 using the measurement values ​​of the measurement sensor 7. Then, the measurement unit 62 calculates a second cost function Cost fs The measurement unit 62 may calculate the second cost function by calculating the difference between the joint angle of each finger of the robot 2 and the joint angle of each finger of the operator, and then summing up the differences between the joint angles of each finger. Alternatively, the measurement unit 62 may calculate the second cost function by calculating the ratio between the joint angle of each finger of the robot 2 and the joint angle of each finger of the operator, and then averaging the ratios between the joint angles of each finger. The measurement unit 62 may calculate the second cost function by multiplying the joint angles of the robot 2 or the joint angles of the operator by a weight indicating a difference in ease of bending the fingers of the robot 2 and the operator.

[0052] [Processing Procedure] Next, an example of a processing procedure when performing a task by remote control will be described. In the following examples, examples of tasks that involve "tasks using fingertips" and "tasks using finger pads and palms" will be described. Fig. 9 is a flowchart of the processing procedure of the remote control device according to this embodiment.

[0053] (Step S1) The measurement unit 62 measures the distance between the operator's fingers using the measurement values ​​of the measurement sensor 7, and measures the fingertip position of each finger. The measurement unit 62 measures each joint angle of each finger using the measurement values ​​of the measurement sensor 7. Note that these processes may be performed in advance and stored in the storage unit 69. Furthermore, the measurement unit 62 may measure the fingertip positions and joint angles of the robot 2 when measuring the fingertip positions and joint angles of the operator, or may measure them in advance and store them in the storage unit 69. Next, an example of a processing procedure when performing a task by remote control will be described. In the following examples, examples of tasks that involve "tasks using fingertips" and "tasks using finger pads and palms" will be described. Fig. 9 is a flowchart of the processing procedure of the remote control device according to this embodiment.

[0054] (Step S2) The determination unit 63 estimates and acquires the work content using the data acquired from the environmental sensor 3. The work content may be selected or input by the operator by operating the operation input unit 5 or the like. In this case, the acquisition unit 61 may acquire the selected or input work content. For example, the selection may be made by displaying work content candidates on the display unit 41 of the image display device 4 and selecting by line of sight. Alternatively, the operator may input or select the work content by operating a keyboard or the like (not shown) connected to the remote control device 6.

[0055] (Step S3) The determination unit 63 determines whether the work content is, for example, "work using fingertips" or "work using finger pads and palms." If the work content is "work using fingertips," the determination unit 63 proceeds to processing in step S4, and if the work content is "work using finger pads and palms," the determination unit 63 proceeds to processing in step S5.

[0056] (Step S4) The priority determination unit 64 determines the priority for calculating the joint angles of the fingers of the end effector to be the fingertip positions of the operator, depending on the determination result of the determination unit 63. Subsequently, the calculation unit 65 calculates weights (e.g., a second cost function) related to the fingertip positions from the fingertip positions of the operator. Alternatively, the calculation unit 65 calculates weights (e.g., a third cost function) related to the distance between the fingers of the operator. After the process, the calculation unit 65 proceeds to the process of step S6.

[0057] (Step S5) The priority determination unit 64 determines the priority of the joint angles of the operator when calculating the joint angles of the fingers of the end effector according to the determination result of the determination unit 63. Subsequently, the calculation unit 65 calculates weights (e.g., a second cost function) related to the joint angles from the joint angles of the fingers of the operator. After the process, the calculation unit 65 proceeds to the process of step S6.

[0058] (Step S6) The calculation unit 65 uses the weights obtained in step S6 or step S7 to calculate the joint angles of the robot 2 by, for example, performing optimization calculations.

[0059] (Step S7) The control unit 66 generates a control command for the robot 2 based on the calculated joint angles.

[0060] (Step S8) The control unit 66 outputs the generated control command to the robot 2 via the output unit 68 to control the operation of the robot 2.

[0061] As described above, in this embodiment, the joint angles of the robot 2 are determined by calculating, for example, weights for an evaluation function that prioritizes the operator's fingertip position or weights for an evaluation function that prioritizes the operator's finger joint angles, depending on the task content. The weights are, for example, for the first to third cost functions, and are, for example, α, β, and γ in the following equation (1). The calculation unit 65 changes and controls the dominant cost function by changing these weights α, β, and γ depending on the task content. The calculation unit 65 then performs optimization calculations to minimize the sum (cost) of the weighted cost functions, thereby determining the joint angles of each finger of the robot 2. For example, in the case of pinching using the fingertips, the weight β or γ, or the weights β and γ, are changed. Furthermore, in the case of a task using the palm, the weight α is changed.

[0062]

number

[0063] Among the above-described processes, the processes of steps S2 to S8 are performed, for example, at predetermined time intervals, and the weight used when the work content is switched is changed. The process of step S1 may also be performed at predetermined time intervals.

[0064] In the above example, the weight is switched depending on whether the task is "task using fingertips" or "task using finger pads and palms," but this is not limiting. The task is not limited to "task using fingertips" and "task using finger pads and palms," and may be other task content as shown in Figure 2, and the number of tasks to be switched is not limited to two, but may be three or more.

[0065] As described above, in this embodiment, the weights of the fingertip positions and joint angles are changed when calculating the joint angles of the robot 2 depending on the operation content.

[0066] In the prior art, since the fingertip positions and taxonomies roughly match, grasping using the fingertips like a tip pinch was possible, but grasping using the finger pads or palm was difficult. In contrast, according to this embodiment, object manipulation using the fingertips and object manipulation using the finger pads and palm can both be performed. Furthermore, according to this embodiment, object manipulation using the fingertips and object manipulation using the finger pads and palm can be performed simultaneously, thereby improving the operability of the robot's multi-fingered hand.

[0067] In the above example, the robot 2 has one arm, but it may also be a double-armed robot with two arms. In such a case, the arm and end effector to be used may be selected depending on the task and controlled using the above-mentioned method, or each arm may be controlled using the above-mentioned method so that the two end effectors work in coordination.

[0068] A program for implementing all or part of the functions of the remote control device 6 of the present invention may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed to perform all or part of the processing performed by the remote control device 6. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. Alternatively, some or all of these components may be realized by hardware (including circuitry) using LSI (Large Scale Integration) such as ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or SOC (System On Chip), or may be realized by a combination of software and hardware.

[0069] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0070] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0071] 1... remote control system, 2... robot, 3... environmental sensor, 4... image display device, 5... operation input unit, 6... remote control device, 7... measurement sensor, 21... arm, 22... end effector, 23... sensor, 24... drive unit, 25... communication unit, 41... display unit, 42... gaze detection unit, 51... operation detection unit, 61... acquisition unit, 62... measurement unit, 63... judgment unit, 64... priority determination unit, 65... calculation unit, 66... ​​control unit, 67... image generation unit, 68... output unit, 69... memory unit

Claims

1. A control system in which an operator remotely controls an end effector having jointed fingers, a first detection unit that detects a fingertip position of the operator; a second detection unit that detects a joint angle of the operator's finger; a determination unit for determining the operation content of the operator; a priority determination unit that determines whether to assign a priority to a fingertip position of the operator or a joint angle of the operator when calculating a joint angle of a finger of the end effector, depending on the operation content determined by the determination unit; a calculation unit that performs optimization calculation of the joint angles including the priorities determined by the priority determination unit; a control unit that outputs the joint angle calculated by the calculation unit as a control command to an end effector; A remote control system comprising:

2. The calculation unit calculating a first cost function relating to a relationship between the end effector and a fingertip position of each of the operators based on the priority; The control unit calculating joint angles of the finger portions of the end effector by performing optimization calculations based on the calculated first cost function; The remote control system of claim 1 .

3. The calculation unit calculating a second cost function relating to a relationship between the end effector and an angle of each joint of each finger of the operator based on the priority; The control unit calculating joint angles of the finger portions of the end effector by performing optimization calculations based on the calculated second cost function; The remote control system of claim 1 .

4. The calculation unit calculating the second cost function by multiplying either the joint angle of the finger portion of the end effector or the joint angle of the finger of the operator by weights related to bending of the end effector and the finger of the operator; The remote control system of claim 3 .

5. the first detection unit detects a distance between the fingers of the operator, the priority determination unit calculates a third cost function related to a distance between the end effector and each of the fingers of the operator; The control unit performing an optimization calculation using the third cost function to calculate joint angles of the fingers of the end effector; 4. The remote control system according to claim 2 or 3.

6. the first detection unit detects a position of a fingertip of the operator based on a distance from a center of a palm of the operator to the fingertip; The shape of the operator's hand and fingers at the time of detecting the fingertip position of the operator is a predetermined shape when gripping.

3. The remote control system according to claim 1 or 2.

7. 1. A device for remotely operating an end effector having jointed fingers when an operator remotely operates the end effector, comprising: a first detection unit that detects a fingertip position of the operator; a second detection unit that detects a joint angle of the operator's finger; a determination unit for determining the operation content of the operator; a priority determination unit that determines whether to assign a priority to a fingertip position of the operator or a joint angle of the operator when calculating a joint angle of a finger of the end effector, depending on the operation content determined by the determination unit; a calculation unit that performs optimization calculation of the joint angles including the priorities determined by the priority determination unit; a control unit that outputs the joint angle calculated by the calculation unit as a control command to an end effector; A remote control device comprising:

8. A method for remotely operating an end effector having jointed fingers when an operator remotely operates the end effector, comprising: a first detection unit detecting a position of a fingertip of the operator; a second detection unit that detects a joint angle of a finger of the operator; a determination unit that determines the operation content of the operator; a priority determination unit determines whether to assign a priority to the fingertip position of the operator or the joint angle of the operator when calculating a joint angle of a finger of the end effector, according to the determined operation content; a calculation unit performing an optimization calculation of the joint angle including the determined priority; a control unit that outputs the calculated joint angle to the end effector as a control command; Remote control method.

9. When an operator remotely controls an end effector having fingers with joints, a computer of a remote control device that remotely controls the end effector is Detecting the position of the operator's fingertip; Detecting the joint angles of the fingers of the operator; determining the operation content of the operator; determining whether to give priority to the fingertip position of the operator or the joint angle of the operator when calculating the joint angle of the finger of the end effector according to the determined operation content; performing optimization calculation of joint angles including the determined priorities; outputting the calculated joint angle to an end effector as a control command; program.

Images