Remote Control System

Abstract

To provide a remote control system that can perform detailed operation of finger tips from a distant place.SOLUTION: A remote control system includes: a coordinate origin setting unit for setting a coordinate origin of an end effector coordinating with an operator; an operator's joint angle information acquisition unit for acquiring information on a joint angle of the operator's finger; an operator finger length measurement unit for measuring a length of each link of an operator's finger; an operator's fingertip position calculation unit for calculating a tip position of each operator's finger when being viewed from a local coordinate origin of an operator model from the information on operator's fingertip position acquired by the operator's joint angle information acquisition unit and the operator's finger length measurement unit; and an end effector joint angle lead-out unit for leading-out a joint angle of the end effector from the operator's fingertip position and the local coordinate origin of the end effector with inverse kinematics.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates to a remote control system. [Background technology]

[0002] In the past, in the case of task-executing robots such as industrial robots, the position of the object operated by the end effector was often fixed, and most types of robots were designed and executed at the level of each joint according to a predetermined task (see, for example, Patent Document 1). In the case of robots that require precise balance control, such as robots that recognize the position of the object from a camera image and correct the position of the hand depending on the situation, or robots that operate in an unrestricted environment, such as walking robots, it is necessary to use IK (Inverse Kinematics) to calculate the angle of each joint required to control the end effector to that position, relative to the position of the end effector as seen from the origin position.

[0003] Furthermore, when considering contact with the environment or physical interaction with people, such as with a humanoid robot, it is desirable to control the torque of each joint. In this case, the position of the end effector and the value of the force generated by the end effector must be calculated as the torque command value for each joint using IK with the Jacobian matrix.

[0004] When controlling a robot using IK or ID (Inverse Dynamics), the problem is that the computational processing time is greater than when the movement of each joint is directly designed. Furthermore, with multi-joint robots, computational processing becomes even heavier, and problems such as singularity issues and inability to determine posture due to redundant degrees of freedom can also be a concern. Humanoid robots and multi-fingered hands have more joints than industrial robots, making computational time a major issue during implementation. In the case of humanoid robots, the base is not fixed, but the position of the robot's waist can be considered the origin of the robot's local coordinate system as viewed from the global coordinate system. Geometrically, calculations can be performed as a branching structure consisting of five branches, with the waist as the origin: the head, right arm, left arm, right leg, and left leg.

[0005] Furthermore, the control of multi-fingered hands can be considered as a branched structure similar to that of humanoid robots. In the case of multi-fingered hands, calculations can be performed using a geometric structure consisting of branches for the five fingers (thumb, index finger, middle finger, ring finger, and little finger) with the wrist position as the origin of the local coordinate system, and the tips of each finger as the end effector. When controlling the tip position of each finger, for example, when controlling the spatial position of the index finger, calculating IK as a connected geometric structure consisting of Fingertip-3Y-2Y-1Y-1Z and Wrist (XYZ) + Elbow (Y) + Shoulder (XYZ) would be computationally inefficient and would make it impossible to find an IK solution. In this case, it is preferable to control the wrist position on the arm side, set the wrist position as the local origin of the multi-fingered hand, and then calculate the joint angles of each finger using IK from the position of each finger.

[0006] Furthermore, in conventional direct remote control of multi-fingered hands, a framework is configured in which the posture angles of the operator's fingers are estimated using, for example, a data glove, and the detected angle information of each joint is used as an angle command value for each joint of the robot (see, for example, Patent Document 2). If the positions and rotation axis configuration of a human joint, and the link lengths of each finger and the configuration of the robot are the same, then even if the robot's hand is moved based on the angle information of each human joint, the positions of the fingertips will remain the same as those of a human. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2020-163511 [Patent Document 2] Japanese Patent Application Laid-Open No. 2006-167867 Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the case of teleoperation, the finger structures of an actual operator and a robot are different, and the finger sizes of each operator also vary. Furthermore, while glove-type devices used in virtual reality (VR) are often used to detect the posture (joint angles) of a human's fingers, no device exists that can accurately detect the angles of a human's joints with approximately 20 DOF (Degrees of Freedom). For this reason, even if a robot is instructed to hold its fingers in a position similar to that of a human pinching something with its index finger and thumb, the posture cannot be detected properly, and even if it is detected, the robot does not adopt the same pinching posture. Thus, with conventional technology, precise remote control of fingertips is difficult to achieve in direct teleoperation.

[0009] The present invention has been made in consideration of the above-mentioned problems, and has as its object to provide a remote control system that allows detailed remote control with fingertips and also allows intuitive operation. [Means for solving the problem]

[0010] (1) In order to achieve the above object, a remote control system according to one aspect of the present invention is a control system in which an operator remotely operates an end effector having finger portions, and includes: a coordinate origin setting unit that sets a coordinate origin of the operator and the end effector; an operator joint angle information acquisition unit that acquires information on joint angles of the operator's fingers; an operator finger length measurement unit that measures the length of each link of the operator's fingers; an operator fingertip position calculation unit that calculates the position of the tip of each finger of the operator as viewed from the local coordinate origin of an operator model based on the information acquired by the operator joint angle information acquisition unit and the operator finger length measurement unit; and an end effector joint angle derivation unit that derives the joint angles of the finger portions that constitute the end effector from the operator's fingertip positions and the local coordinate origin of the end effector using inverse kinematics.

[0011] (2) In addition, in a remote control system according to one aspect of the present invention, the operator finger length measurement unit may model the geometric structure of the operator's joint positions and link lengths, and measure the operator's joint positions and the lengths of each link of the fingers based on the length of a first link of one of the operator's five fingers.

[0012] (3) In addition, in a remote control system according to one aspect of the present invention, the end effector joint angle derivation unit may generate a command value for a second joint of a finger of the end effector based on the measured length of each link of the operator and the calculated position of the fingertip of the operator.

[0013] (4) Furthermore, in a remote control system according to one aspect of the present invention, a position correction unit may be provided that measures the difference in position between the center point of the operator's fingertip and the center point of the finger of the end effector, and corrects the tip position of each finger as viewed from the local coordinate origin of the operator model as an offset for a fingertip position command.

[0014] (5) Furthermore, a remote control system according to one aspect of the present invention may include a position correction unit that measures a difference between a target point of the operator's fingertip and the position of the fingertip of a model of the operator's hand, and corrects the tip position of each finger as viewed from the local coordinate origin of the operator model as an offset of a fingertip position command.

[0015] (6) Furthermore, a remote control system according to one aspect of the present invention may include a position correction unit that measures a difference in position between a target point of the operator's fingertip and a target point of a finger of the end effector, and corrects the tip position of each finger as viewed from the local coordinate origin of the operator model as an offset for a fingertip position command. In the case of a pinching motion using the thumb and index finger, the center point between the positions of the tips of both fingers may be set as the target position. Also, the center point between the positions of the tips of both end effectors may be set as the target position.

[0016] (7) Furthermore, in a remote control system according to one aspect of the present invention, a mode selection unit may be provided that can select, for each finger, a mode that derives the joint angle of the end effector and a mode that estimates the posture of the operator's hand based on sensor information of a robot equipped with the end effector and moves the robot.

[0017] (8) In the remote control system according to an aspect of the present invention, the mode selection unit may switch modes according to a taxonomy or according to a speed of the operator's fingertip. [Effects of the Invention]

[0018] According to (1) to (8), detailed remote control of the fingertips is possible, and at the same time, intuitive operation is possible. [Brief explanation of the drawings]

[0019] [Figure 1] 10A and 10B are diagrams for explaining modeling of the geometric structure of the joint positions and link lengths of the operator in the first processing according to the embodiment. [Figure 2]FIG. 1 is a diagram illustrating an example of a system for measuring the finger size of each operator according to an embodiment. [Figure 3] FIG. 10 is a diagram showing an example of the actually measured size of an operator's finger. [Figure 4] FIG. 10 is a diagram for explaining calculation of a fingertip position according to an embodiment. [Figure 5] FIG. 10 is a diagram for explaining an example of matching between a human hand model, a robot hand model, and positions. [Figure 6] 10A and 10B are diagrams for explaining derivation of a joint angle of an end effector by IK according to an embodiment. [Figure 7] FIG. 1 is a diagram illustrating an example of a taxonomy. [Figure 8] FIG. 10 is a diagram for explaining the use of a command value for a second joint of a finger of the end effector. [Figure 9] FIG. 10 is a diagram for explaining how to realize a pinch operation. [Figure 10] FIG. 10 is a diagram for explaining calibration of a human posture and a human model posture using a data glove. [Figure 11] FIG. 10 is a diagram for explaining calibration of a human posture and a human model posture using a data glove. [Figure 12] FIG. 10 is a diagram illustrating an example of a verification result obtained by simulation. [Figure 13] FIG. 1 is a diagram illustrating an example of the configuration of an end effector control device according to an embodiment. [Figure 14] 1A and 1B are diagrams illustrating an example of the configuration of an end effector according to an embodiment. [Figure 15] 10 is a flowchart of a finger size measurement and model creation process according to an embodiment. [Figure 16] FIG. 10 is a diagram illustrating a processing procedure performed by a calibration unit according to the embodiment. [Figure 17] 10A and 10B are diagrams for explaining the operation of remotely pinching a pull tab with the end effector. [Figure 18] 10 is a flowchart of a processing procedure performed by an IK processing unit according to the embodiment. [Figure 19]10 is a flowchart of a processing procedure performed in an IK processing block according to the embodiment. [Figure 20] 10 is a flowchart of an IK processing procedure for the thumb according to the embodiment. [Figure 21] FIG. 10 is a diagram illustrating a coordinate system in FK processing. [Figure 22] 10A and 10B are diagrams illustrating a reference point and a base position of a thumb of a human model, and a reference point and a base position of a thumb in a robot coordinate system. [Figure 23] 10A and 10B are diagrams for explaining the relationship between a reference point and the base of the thumb of a human model, and between a reference point and the base of the thumb in robot coordinates. [Figure 24] FIG. 10 is a diagram showing an example of transition of links up to the fingertip when the base positions of the thumbs are aligned. [Figure 25] FIG. 10 is a diagram showing an example of transition of links up to the fingertip when the base positions of the thumbs are aligned. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0021] <Summary> First, an outline of the embodiment will be described. In this embodiment, the fingertip positions of the human and robot are aligned, while intuitive operation is possible. To achieve this, in this embodiment, in the first process, the size of each operator's fingers is measured and a geometric model is constructed, and the fingertip positions are calculated using forward kinematics (FK) using the sensor values ​​of the data glove. In the first process, the geometric structure of the operator's joint positions and link lengths is modeled, and then a system is constructed to measure the joint positions and link lengths of each operator. Furthermore, in the second process, the FK fingertip positions are used as command values ​​for the robot, and the joint angles of the end effector 1 are derived, for example, using inverse kinematics (IK).

[0022] <First process> First, the first process will be described. FIG. 1 is a diagram for explaining modeling of the geometric structure of the operator's joint positions and link lengths in the first process according to this embodiment. Images g1 and g2 show the movement pattern of the thumb. Image g3 shows the axis configuration. In the model, as shown in image g3, the first joint of each of the index finger, middle finger, ring finger, and little finger is designated as YZ, and the second and third joints are designated as Y. Furthermore, for example, the wrist joint is designated as the reference point X. For the thumb, the CM joint between the first metacarpal bone and the hand, the MP joint between the first metacarpal bone and the proximal phalanx, and the IP joint between the proximal phalanx and the distal phalanx are designated as Y. Furthermore, the thumb has an opposable joint structure.

[0023] FIG. 2 is a diagram showing an example of a system for measuring finger size for each operator according to this embodiment. Image g11 is an example in which numbers and names are assigned to the joints of the hand and each finger. In this example, the reference point is numbered 0. Image g13 is a model of the hand and fingers, and images g12 and g14 are example images taken during actual measurement. In actual measurement, the operator holds their hand toward the camera, for example.

[0024] Since the size of fingers varies from person to person, even if the position of a person's fingertips is calculated using the angles of each finger joint obtained using a data glove worn by the remote operator, the positions will differ from person to person. In contrast, in this embodiment, a system for measuring the size of an operator's finger applies a captured image to a model and measures the size of each part. In this embodiment, for example, measurement is performed using a model that has undergone image processing and machine learning on the captured image. In the measurement, the reference length is, for example, the length of the first link of the index finger, as will be described later. The reference length may also be the first link of another finger. Furthermore, measurements may be performed multiple times to calculate an average, or noise may be removed. FIG. 3 is a diagram showing an example of the size of an operator's finger that has been actually measured.

[0025] <Calculation of fingertip position> Next, the calculation of the fingertip position will be described. 4 is a diagram for explaining the calculation of the fingertip position according to this embodiment. In this embodiment, a model of the operator's hand is constructed (g21), the operator wears a data glove on his / her hand, and the positions of the operator's fingertips are calculated by FK (g23) using sensor values ​​detected by a sensor included in the data glove (g22). Then, in this embodiment, commands for the operator's hand model and fingertip positions are output to the end effector 1 (g24).

[0026] In calculating the fingertip position, the fingertip position commands of the operator's hand model and the robot's hand model are matched by matching common parts between them. For example, the base of the thumb may be matched, as shown in images g27 and g28 in Figure 5. Figure 5 is a diagram for explaining an example of matching between a human hand model, a robot hand model, and their positions.

[0027] Image g25 is an example of a human hand model, and is an example of calculating the fingertip position from the hand coordinate origin. Image g26 is an example of a robot hand model, showing the case where the fingertip position of a human hand model is specified from the origin of the robot hand model. Image g27 is an example of a human hand model, and is an example of calculating the fingertip position from the base of the thumb. Image g28 is an example of a robotic hand model, in which the thumb base positions of the human hand model and the robotic hand model are first aligned, and then the fingertip position of the thumb origin of the human hand model is indicated from the thumb base of the robotic hand model. In FIG. 5, Xh_ft[n]=Xh_ft[n]−Xh_tb+Xr_tb (n=0, 1, 2, 3).

[0028] <Derivation of end effector joint angles> In this embodiment, the FK fingertip position is used as a command value for the robot, and the joint angle of the end effector 1 is derived by IK. That is, as shown in Fig. 6, the FK fingertip position is used as a command value for the fingertip position of the end effector 1. Fig. 6 is a diagram for explaining the derivation of the joint angle of the end effector 1 by IK according to this embodiment. In this embodiment, the pseudo-inverse matrix J is calculated as shown in the following equations (1) to (3). + The joint angle is derived using

[0029]

number

[0030]

number

[0031]

number

[0032] In addition, in formula (1), x · (≧δx) is the change between the current fingertip position and the next fingertip position command, and q · (≧δq) is the change in each joint angle required to move to the next commanded fingertip position, and J is the transformation matrix that converts the change in hand position when each joint is moved slightly. In equation (2), I is the unit matrix, and ξ is the weight (sensitivity). In equation (3), n is the degree of freedom of the joint, and m is the degree of freedom of the task.

[0033] <Use of taxonomy> In this embodiment, the following A and B are performed to enable selection for each finger between an intuitive joint angle command based on the sensor values ​​of the data glove and the model of the operator's hand, and a position command based on the IK of the end effector 1. The following switching is performed by a selection unit 212, which will be described later.

[0034] A. Switching modes depending on taxonomy (see Reference 1) 7 is a diagram showing an example of taxonomy. In this embodiment, a mode IK that calculates and commands joint angles using IK and a mode DEV that commands using sensor information from the device are used, as shown in the following taxonomies I to VII. In I to VII, T represents the thumb, I represents the index finger, M represents the middle finger, and R represents the ring finger. I.Medium Wrap: T[IK], I[DEV], M[DEV], R[DEV] II.Power Disk: T[IK], I[IK], M[IK], R[DEV] III.Prizmatic3:T[DEV], I[IK], M[IK], R[DEV] IV.Prizmatic2:T[DEV], I[IK], M[DEV], R[DEV] V.Palmar Pinch: T[IK], I[IK], M[DEV], R[DEV] VI.Tripod:T[IK], I[IK], M[IK], R[DEV] VII.Ratchet wrench:T[IK], I[DEV], M[DEV], R[DEV]

[0035] Reference 1; Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human Grasp Types” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb. 2016), IEEE, p66-77

[0036] B. Switch modes according to finger speed In this embodiment, IK and DEV are used under the following conditions. If (fingertip speed > threshold): Mode DEV commanded by device sensor information Else: Mode IK where joint angles are calculated and commanded by IK

[0037] <Using the command value of the second joint of the finger of the end effector> In this embodiment, as shown in Fig. 8, in the position command mode by IK of the end effector 1, the command value of the second joint of the finger of the end effector 1 is obtained by "measuring the size of the finger of each operator, constructing a geometric model, and calculating the position of the fingertip by FK using the sensor value of the data glove," thereby enabling intuitive operation (g31 to g34). Fig. 8 is a diagram for explaining the use of the command value of the second joint of the finger of the end effector.

[0038] In this embodiment, in the IK convergence calculation, the value of the second joint that can be obtained from the sensor is given as a constraint condition, improving intuitive operability. Since the answer is given in advance in this way, it is thought that the convergence calculation will also be faster according to this embodiment.

[0039] <Realizing pinch operation> In this embodiment, the following calibration is performed to enable pinching with the thumb, index finger, and middle finger: Fig. 9 is a diagram for explaining how pinching is achieved. A. Calibration of the fingertip position offset between the actual human finger posture and the model "measured the finger size for each operator, constructed a geometric model, and calculated the fingertip position using FK using the sensor values ​​of the data glove" (image g41 in Figure 9). B. Calibration of the constructed geometric model and the fingertip position offset of end effector 1 (g42)

[0040] <Calibration of human pose and human model pose using data gloves> Next, we will explain the calibration of a human posture and a human model posture using a data glove. Figures 10 and 11 are diagrams for explaining the calibration of a human posture and a human model posture using a data glove. Figure 10 shows an example of a pinching posture taken by an actual worker. The circle g51 is the common target position. In the case of a robot, the common target position is the target position after aligning the base of the thumb. Although the target position is common, calibration is performed first for the human hand model, and then for the robot hand model. In other words, calibration is performed separately. This process cancels out each offset.

[0041] Calibration will be further described with reference to Fig. 11. In this embodiment, the posture of the human model using the data glove and the posture of the robot model are calibrated separately. As shown in image g52, the offset removal of the human model involves adjusting the sensor value offset and model adjustment. The process removes the offset (a) between the target position and the thumb, the offset (b) between the target position and the index finger, and the offset (c) between the target position and the middle finger in the human model. As shown in image g53, the offset removal of the robot model removes the offset between the target position and the thumb (d), the offset between the target position and the index finger (e), and the offset between the target position and the middle finger (f).

[0042] Here, we will consider the issues with typical data gloves. Typical data gloves used for remote control, etc., have the following issues. Problem 1. Unable to position the thumb facing the wrist Difficulty in pinching (if you try to pinch something by forcing your fingers, your fingers may get in the way) Posture calibration issues Problem 2. The posture of the third joint is not detected Linked to the second joint (the interface graph provided by the data glove only displays the first and second joints) In this embodiment, the offset calibration mode of the pinch posture is performed locally for Task 1. Also, in this embodiment, for Task 2, the angle of the third joint of the data glove is defined by the linkage algorithm of the third joint of the end effector 1.

[0043] In this embodiment, in order to match the fingertip positions estimated by the data glove with the fingertip positions of the end effector 1, the joint angles of each finger of the end effector 1 are calculated using IK from the positions of each fingertip of the operator. Figure 12 shows an example of the verification results from a simulation. In the simulation, the joint angles of each robot's finger were calculated using IK from the position of each user's fingertip when each joint was moved. In Figure 12, images g81 and g83 show the IK from the task command for end effector 1, the FK from IK, and the FK from the end effector 1 position after the task. Images g82 and g84 show the FK from the operator's joint command, and the FK from FK to the task command for end effector 1. The results of this simulation and verification showed that the tolerance for error was set at 5 mm, and operation was confirmed with a similar degree of error.

[0044] <Configuration example of end effector control device> Next, a description will be given of an example of the configuration of the end effector control device 2. Fig. 13 is a diagram showing an example of the configuration of the end effector control device according to this embodiment. As shown in FIG. 13 , the end effector control device 2 includes an imaging unit 201 (operator finger length measurement unit), a finger size measurement unit 202 (operator finger length measurement unit), a data creation unit 203 (operator finger length measurement unit), a model creation unit 204, an acquisition unit 205 (operator joint angle information acquisition unit), a sensitivity / offset adjustment unit 206, a calibration unit 207 (position correction unit), an FK processing unit 208 (operator fingertip position calculation unit), an IK processing unit 210 (end effector joint angle derivation unit), an instruction generation unit 211, an origin setting unit 212 (coordinate origin setting unit), and a selection unit 213 (mode selection unit).

[0045] The end effector 1 includes multiple fingers, a force sensor, a position sensor, etc. An example of the configuration of the end effector 1 will be described later.

[0046] The data glove 3 is a device worn by an operator on his / her hand when remotely operating the end effector 1. The data glove 3 is equipped with a sensor 31 (operator joint angle information acquisition unit) that detects the position and movement of the fingers. The sensor 31 is, for example, a six-axis sensor.

[0047] The imaging unit 201 is, for example, a CCD (Charge Coupled Device) imaging device, a CMOS (Complementary MOS) imaging device, or the like.

[0048] As described above, the finger size measurement unit 202 measures the size of the worker's finger by using image processing and a trained model on the image captured by the imaging unit 201. The imaging unit 201 and the finger size measurement unit 202 may be external devices. In this case, the finger size measured by the external device may be output to the end effector control device 2.

[0049] The data creation unit 203 creates the finger size measured by the finger size measurement unit 202 as data in the same format as the end effector 1.

[0050] The model creation unit 204 creates a kinematics model using the data created by the data creation unit 203.

[0051] The acquisition unit 205 acquires the sensor value detected by the sensor 31 from the data glove 3 worn by the operator. The acquisition unit 205 acquires information on the joint angles of the operator's fingers during operation based on the sensor value. Note that the finger size measurement unit 202 may acquire information on the joint angles of the operator's fingers during operation.

[0052] The sensitivity / offset adjustment unit 206 adjusts the sensitivity and offset of the sensor value acquired by the acquisition unit 205 .

[0053] The calibration unit 207 acquires the average value of the fingertip positions of each finger of the operator's hand model. The calibration unit 207 acquires the average value of the fingertip positions of each finger of the end effector 1. The calibration unit 207 calibrates the offset of the fingertip positions between the actual human finger posture and the model. The calibration unit 207 calibrates the offset of the fingertip positions between the model and the end effector 1. The calibration unit 207 may measure the difference between the target points of the operator's fingertips and the positions of the fingertips of the operator's hand model, and correct the tip positions of each finger as viewed from the local coordinate origin of the operator model, as an offset in the fingertip position command. The calibration unit 207 may also measure the difference between the target points of the operator's fingertips and the target points of the fingers of the end effector, and correct the tip positions of each finger as viewed from the local coordinate origin of the operator model, as an offset in the fingertip position command. In the case of a pinching motion using the thumb and index finger, the center point of the positions of both fingertips may be set as the target position. Alternatively, the center point of the positions of the tips of both end effectors may be set as the target position.

[0054] The FK processing unit 208 acquires the model created by the model creation unit 204, and calculates the position of the fingertip by FK using the data adjusted by the sensitivity / offset adjustment unit 206 and the value calibrated by the calibration unit 207. The output of the FK processing unit 208 is a command for the fingertip position of the model of the operator's hand.

[0055] The IK processing unit 210 uses the fingertip positions calculated by the FK processing unit 208 as command values ​​for the robot, and derives the joint angles of the end effector 1 by IK using data calibrated by the calibration unit 207. The output of the IK processing unit 210 is angle commands for each finger of the end effector 1.

[0056] The instruction generation unit 211 generates a control instruction for the end effector 1 based on the angle instruction for each finger of the end effector 1.

[0057] The origin setting unit 212 sets the coordinate origins (reference points) of the operator and the end effector 1. The origin setting unit 212 sets the coordinate origins of the operator and the end effector 1, for example, based on the data created by the data creation unit 203 and the known dimensions of the end effector 1.

[0058] As will be described later, the selection unit 213 selects for each finger a mode that derives the joint angle of the end effector 1 (retargeting) and a mode that estimates the posture of the operator's hand using sensor information from the robot equipped with the end effector 1 and moves the end effector.

[0059] The processing procedures of each section will be described later.

[0060] <Example of end effector configuration> Fig. 14 is a diagram showing an example of the configuration of an end effector according to this embodiment. Although the end effector 1 in Fig. 14 is an example having four fingers, the number of fingers may be two or more. The end effector 1 includes a finger portion 101, a finger portion 102, a finger portion 103, a finger portion 104, and a base body 111. The end effector 1 is connected to an arm 121 via a joint.

[0061] Finger portion 101 corresponds to, for example, a human thumb. Finger portion 102 corresponds to, for example, a human index finger. Finger portion 103 corresponds to, for example, a human middle finger or ring finger. Finger portion 104 corresponds to, for example, a human ring finger. Each finger portion has a joint and a knuckle. Base 111 includes positions corresponding to the back and palm of a human hand. Finger portions 101 to 104 are connected to base 111.

[0062] <Finger size measurement, model creation process> Next, an example of the processing procedure performed by the photographing unit 201, finger size measurement unit 202, data creation unit 203, and model creation unit 204 will be described. Note that these processes are performed before remote control. Fig. 15 is a flowchart of the finger size measurement and model creation processing procedure according to this embodiment.

[0063] (Step S101) The operator inputs the actual length of the first link of the index finger of the hand performing the remote operation into the end effector control device 2. The end effector control device 2 is equipped with an input device (not shown) (for example, a keyboard or a touch panel sensor) and acquires input from the operator.

[0064] (Step S102) The end effector control device 2 starts up a motion capture system that uses a trained image processing model of the hand. The image capturing unit 201 captures an image including the worker's hand.

[0065] (Step S103) The finger size measurement unit 202 acquires the average value of the joint position information.

[0066] (Step S104) The finger size measurement unit 202 estimates the length of each link from the joint position information.

[0067] (Step S105) The data creating unit 203 scales all link lengths based on the input first link length of the index finger, the measured value, and the estimated value of each link.

[0068] (Step S106) The data creating unit 203 scales the position of the base of each finger based on the input length of the first link of the index finger, the measured value, and the estimated value of each link.

[0069] (Step S107) The data creation unit 203 creates data in a format that can be read by, for example, a robot definition file. The model creation unit 204 creates a kinematic model based on the measured data.

[0070] <Calibration process> Next, an example of the processing procedure performed by the calibration unit 207 will be described. Fig. 16 is a diagram showing the processing procedure performed by the calibration unit according to this embodiment. Note that the following processing is an example in which an operator performs a pinching operation.

[0071] (Step S201) The sensor 31 of the data glove 3 detects the sensor value when the worker takes a pinching posture.

[0072] (Step S202) The calibration unit 207 acquires the average value X[i] of the fingertip positions of each finger of the model of the operator's hand. Note that the calibration unit 207 performs, for example, the processes of steps S202 to S205 and the process of step S206 in parallel. Also, the calibration unit 207 performs, for example, the processes of steps S202 to S204 and the process of step S210 in parallel.

[0073] (Step S203) The calibration unit 207 derives the center point Cp from the average value of the fingertip positions of the fingers of the model of the operator's hand.

[0074] (Step S204) The calibration unit 207 specifies the pinch position of each finger from the center point Cp (Y[i]=Cp+offset).

[0075] (Step S205) The calibration unit 207 calculates the pinch position qy[i] of each finger from the center point Cp using IK.

[0076] (Step S206) The calibration unit 207 calculates the average value qx[i] of the fingertip position of each finger using IK.

[0077] (Step S207) The calibration unit 207 subtracts the average value qx[i] of the fingertip position of each finger from the pinch position qy[i] of each finger from the center point Cp.

[0078] (Step S208) The calibration unit 207 creates calibration data for the offset of the human model (operator model) using data obtained by subtracting the average value qx[i] of the positions of the fingertips of each finger from the position qy[i] of the pinch of each finger from the center point Cp.

[0079] (Step S209) The calibration unit 207 performs calibration of the offset of the human model using the created data. After the calibration, the calibration unit 207 ends the process.

[0080] (Step S210) The calibration unit 207 obtains the average value Z[i] of the positions of the fingertips of each finger of the end effector 1.

[0081] (Step S211) The calibration unit 207 subtracts the average value Z[i] of the positions of the fingertips of each finger of the end effector 1 from the position of the pinch of each finger from the center point Cp. Note that the calibration data for the offset between the human model and the end effector corresponds to position data.

[0082] (Step S212) The calibration unit 207 creates calibration data for the offset between the human model and the end effector.

[0083] (Step S213) The calibration unit 207 performs calibration of the offset between the human model and the end effector using the created data. After the calibration, the calibration unit 207 ends the process.

[0084] <Processing procedure of the IK processing unit> Next, an example of the processing procedure performed by the IK processing unit 210 will be described using FIGS. 17 to 19. FIG. 17 is a diagram illustrating the operation of pinching a pull tab with an end effector by remote control. Image g101 shows the position g111 of the fingertip of the finger portion 102, the position g112 of the second joint, and the position g113 of the MP joint when the end effector 1 pinches the pull tab. Image g103 shows the position g122 of the second joint of the index finger and the position g123 of the MP joint when a remote control instruction is given using the data glove. Image g102 shows the relationship between the position based on the sensor value of the end effector 1 and the position based on the command value. As described above, the IK processing unit 210 (operator joint angle information acquisition unit) introduces, i.e., acquires, information on the joint angles of the operator's fingers through IK processing.

[0085] The following process is an example in which an operator performs a pinching operation. The following process example is an example in which the end effector 1 has four fingers as shown in Fig. 14. Fig. 18 is a flowchart of the processing procedure performed by the IK processing unit according to the embodiment.

[0086] (Step S241) The IK processing unit 210 acquires the current fingertip position.

[0087] (Steps S242 to S247) The IK processing unit 210 repeats the convergence calculations of steps S203 to S207 until the errors between the target positions and the current positions of all fingers become smaller than the threshold. Note that the IK processing unit 210 ends the convergence calculations when the number of convergence calculations reaches a set value, and starts the convergence calculations for the next command value.

[0088] (Step S243) The IK processing unit 210 performs processing of the IK processing block corresponding to the thumb.

[0089] (Step S244) The IK processing unit 210 performs processing of the IK processing block corresponding to the index finger.

[0090] (Step S245) The IK processing unit 210 performs processing of the IK processing block corresponding to the middle finger.

[0091] (Step S246) The IK processing unit 210 performs processing of the IK processing block corresponding to the ring finger.

[0092] (Step S247) The IK processing unit 210 performs FK processing.

[0093] Next, as an example of processing performed in the IK processing block, processing in the IK processing block corresponding to the thumb (S243 in FIG. 18) will be described. Fig. 19 is a flowchart of the processing procedure performed in the IK processing block according to this embodiment.

[0094] (Step S221) The IK processing unit 210 determines whether the IK mode of the thumb is on and the error X is greater than the threshold value. If the IK mode of the thumb is on and the error X is greater than the threshold value (step S221; YES), the IK processing unit 210 proceeds to the processing of step S222. If the IK mode of the thumb is off or the error X is equal to or less than the threshold value (step S221; NO), the IK processing unit 210 skips the IK processing of the thumb and ends the processing. In this way, in this embodiment, the IK calculation of a finger whose error from the target position is less than the threshold value is skipped.

[0095] (Step S222) The IK processing unit 210 performs IK processing corresponding to the thumb.

[0096] (Step S223) If the threshold value is not greater than the error X and the IK mode of the thumb is not in the OFF state, the IK processing unit 210 turns the IK mode of the thumb on.

[0097] Next, a description will be given of the IK processing for the thumb in S222 of Fig. 19. Fig. 20 is a flowchart of the IK processing procedure for the thumb according to this embodiment.

[0098] (Step S231) The IK processing unit 210 acquires the position of the tip of the thumb.

[0099] (Steps S232 to S234) The IK processing unit 210 repeats the processes of steps S232 to S234.

[0100] (Step S232) The IK processing unit 210 performs pseudo-inverse matrix operations of Expression (2) and Expression (3).

[0101] (Step S233) The IK processing unit 210 overwrites the value of the 2Y joint with the joint angle of the human model using the sensor value of the end effector 1.

[0102] (Step S234) The IK processing unit 210 performs FK processing.

[0103] (Step S235) The IK processing unit 210 generates a joint angle command for the end effector 1.

[0104] <FK processing> Next, a processing example performed by the FK processing unit 208 will be described. FIG. 21 is a diagram showing a coordinate system in FK processing. The image g201 shows the x-axis direction, the z-axis direction, and the angle θ r0z (= variable) of the reference point and the mechanical offset. The image g202 shows the rotation direction of each joint and the angles (θ r1z , θ r1z , θ r1x (= variable), θ r2Y , θ r3Y ) between the links.

[0105] The simultaneous rotation matrix is such that the rotation by an angle θ around the x-axis is given by the following Expression (4), the rotation by an angle θ around the y-axis is given by the following Expression (5), the rotation by an angle θ around the z-axis is given by the following Expression (6), the translation by d around the x-axis is given by the following Expression (7), the translation by d around the y-axis is given by the following Expression (8), and the translation by d around the z-axis is given by the following Expression (9).

[0106]

Equation

[0107]

Equation

[0108]

number

[0109]

number

[0110]

number

[0111]

number

[0112] An example of a transformation matrix used in FK processing will be explained using the thumb as an example. rotation matrix T base is the following equation (10), and the rotation matrix (T base ) 1Z is the following equation (11), and the rotation matrix (T 1Z ) 1Y is the following equation (12), and the matrix (T 1Y ) 1X is the following equation (13), and the rotation matrix (T 1X ) 2Y is the following equation (14), and the rotation matrix (T 2Y ) 3Y is the following equation (15), and the rotation matrix (T 3Y ) Fingertip is expressed as the following equation (16).

[0113]

number

[0114]

number

[0115]

number

[0116]

number

[0117]

number

[0118]

number

[0119]

number

[0120] Using the rotation matrices of the above equations (10) to (16), the equation from the reference point to each fingertip can be expressed as in the following equation (17).

[0121]

number

[0122] <Human model coordinate origin and end effector coordinate origin> Here, as shown in FIGS. 22 to 25, the coordinate origin of the human model and the coordinate origin of the robot do not coincide.

[0123] FIG. 22 is a diagram showing the reference point and the base of the thumb of a human model, and the reference point and the base of the thumb in robot coordinates. Image g300 shows the reference point (origin) g301 of the hand coordinate system of the human model, the base position (Xh_tb) g302 of the thumb base as viewed from the reference point of the hand coordinate system of the human model, and the fingertip positions (Xh_ft) g303 of each finger as viewed from the reference point of the hand coordinate system of the human model. Image g310 shows the reference point (origin) g311 of the robot coordinate system of the end effector 1, the base position (Xr_tb) g312 of the thumb base as viewed from the reference point of the robot coordinate system of the end effector 1, and the fingertip position (Xr_ft) g313 of each finger as viewed from the reference point of the hand coordinate system of the end effector 1.

[0124] When the fingertip positions of the human model viewed from the coordinate origin of the human model are instructed to the end effector 1, the posture of the fingers of the end effector 1 may differ significantly from that of a human. For example, when the operator performs a natural pinching motion, the pinching position of the end effector 1 may be too close to or too far from the palm of the hand, making it impossible to maintain a proper posture. For example, when the same fingertip position is commanded as in image g320 in Fig. 23, the reference points of the coordinate systems do not match. Also, when the same fingertip position is commanded and the reference points of the coordinate systems are matched as in image g330 in Fig. 23, the position of the base of the finger may shift, making it impossible to maintain the correct posture. Fig. 23 is a diagram for explaining the relationship between the reference point of the human model and the base position of the thumb, and between the reference point and the base position of the thumb in the robot coordinate system.

[0125] From the perspective of working with the fingertips, aligning the bases of the fingers of the person and the end effector 1 improves workability and allows for a more natural posture. In other words, if we consider that the movement and posture of the thumb and index finger are dominant when a person performs delicate work, it is more effective to align the base of the thumb or index finger of the person and the robot. In addition, generally, the tip of the index finger can bend significantly toward the base of the thumb, and the thumb cannot move as far forward as the index finger.

[0126] For this reason, in this embodiment, the position of the base of the thumb of the human model is made to match that of the end effector 1. In this embodiment, for example, as shown in image g340 in Fig. 23, by correcting the command value so that the base of the thumb matches, the human model will have a similar posture, enabling intuitive operation.

[0127] The fingertip position (Xr_ft) of each finger as viewed from the reference point of the hand coordinate system of the end effector 1 can be calculated as in the following equation (18).

[0128]

number

[0129] 24 and 25 are diagrams showing an example of the transition of links up to the fingertips when the base positions of the thumbs are aligned. In Fig. 24 and Fig. 25, the chronological order is images g350, g360, g370, and g380. The transition diagrams shown in FIGS. 24 and 25 are merely examples, and the present invention is not limited to these.

[0130] As described above, in this embodiment, the following processing is performed. - Develop a system to measure the size of the operator's fingers and build a finger kinematics model for each operator. - Develop an IK retargeting algorithm and develop an algorithm that converts human fingertip position commands into robot fingertip positions in task space. - Errors in fingertip position caused by differences between the robot and human models are corrected using a calibration algorithm. To enable operators to intuitively operate a multi-fingered robotic hand and reduce the computational load of IK, it is possible to select between IK joint angle command mode and device joint angle command mode for each finger based on taxonomy. To enable intuitive operation of the multi-fingered robot hand by the operator and reduce the IK calculation load, the angle command value of the second joint resulting from the IK convergence calculation for each joint is overwritten with direct angle information detected by the device.

[0131] As a result, according to this embodiment, the following effects can be obtained. By building a system to measure the size of the operator's fingers, it became possible to build a detailed finger kinematics model for each operator. We developed an IK retargeting algorithm that converts the position commands of a human fingertip into the position of a robot fingertip in the task space, making it possible to accurately command the position of each fingertip even if the structure and size differ between the human and robot. Even if there is an error in the fingertip position due to differences between the robot and human models, by performing calibration using the operator's pinching posture, it is now possible to reset the positional deviation when the robot's pinching posture is commanded. -By making it possible to select between IK joint angle command mode and device joint angle command mode for each finger based on taxonomy, it has become possible for operators to intuitively operate a multi-fingered robot hand. -By making it possible to select between IK joint angle command mode and device joint angle command mode for each finger based on taxonomy, it has become possible to reduce the calculation load of IK. By overwriting the angle command value of the second joint in the IK convergence calculation results of each joint with direct angle information detected by the device, the IK convergence calculation was made faster. The movement, speed, and acceleration / deceleration of the second joint were defined as being dominant in the effect of intuitive finger movement, and the angle command value of the second joint, which is the result of the convergence calculation of the IK of each joint, was overwritten with direct angle information detected by the device, making it possible for the operator to intuitively command the joint angles of a multi-fingered hand.

[0132] A program for implementing all or part of the functions of the end effector control device 2 of the present invention may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed to perform all or part of the processing performed by the end effector control device 2. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a web page provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0133] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0134] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0135] 1...end effector, 2...end effector control device, 201...photographing unit, 202...finger size measurement unit, 203...data creation unit, 204...model creation unit, 205...acquisition unit, 206...sensitivity / offset adjustment unit, 207...calibration unit, 208...FK processing unit, 210...IK processing unit, 211...instruction generation unit, 212...origin setting unit, 213...selection unit, 3...data glove, 31...sensor

Claims

1. A control system in which an operator remotely controls an end effector having fingers, a coordinate origin setting unit that sets a coordinate origin of the operator and the end effector; an operator joint angle information acquisition unit that acquires information on the joint angles of the fingers of the operator; an operator's finger length measuring unit that measures the length of each link of the operator's finger; an operator fingertip position calculation unit that calculates the position of a tip of each finger of the operator as viewed from the origin of a local coordinate system of an operator model, based on the information acquired by the operator joint angle information acquisition unit and the operator finger length measurement unit; an end effector joint angle derivation unit that derives a joint angle of the end effector from the position of the operator's fingertip and the origin of a local coordinate system of the end effector by inverse kinematics; Equipped with The operator's finger length measuring unit A geometric structure of the joint positions and link lengths of the operator is modeled, and the joint positions of the operator and the lengths of each link of the fingers are measured based on the length of a first link of one of the operator's five fingers; Remote control system.

2. A control system in which an operator remotely controls an end effector having a finger portion, a coordinate origin setting unit that sets a coordinate origin of the operator and the end effector; an operator joint angle information acquisition unit that acquires information on the joint angles of the fingers of the operator; an operator's finger length measuring unit that measures the length of each link of the operator's finger; an operator fingertip position calculation unit that calculates the position of a tip of each finger of the operator as viewed from the origin of a local coordinate system of an operator model, based on the information acquired by the operator joint angle information acquisition unit and the operator finger length measurement unit; an end effector joint angle derivation unit that derives a joint angle of the end effector from the position of the operator's fingertip and the origin of a local coordinate system of the end effector by inverse kinematics; Equipped with The end effector joint angle derivation unit generating a command value for a second joint of a finger portion of the end effector based on the measured length of each link of the operator and the calculated position of the fingertip of the operator; Remote control system.

3. A control system in which an operator remotely controls an end effector having a finger portion, a coordinate origin setting unit that sets a coordinate origin of the operator and the end effector; an operator joint angle information acquisition unit that acquires information on the joint angles of the fingers of the operator; an operator's finger length measuring unit that measures the length of each link of the operator's finger; an operator fingertip position calculation unit that calculates the position of a tip of each finger of the operator as viewed from the origin of a local coordinate system of an operator model, based on the information acquired by the operator joint angle information acquisition unit and the operator finger length measurement unit; an end effector joint angle derivation unit that derives a joint angle of the end effector from the fingertip position of the operator and the origin of a local coordinate system of the end effector by inverse kinematics; a position correction unit that measures a difference between the center points of the operator's fingertips and the center points of the fingers of the end effector, and corrects the tip positions of the fingers as viewed from the origin of a local coordinate system of the operator model as an offset of a fingertip position command. Remote control system.

4. A control system in which an operator remotely controls an end effector having a finger portion, a coordinate origin setting unit that sets a coordinate origin of the operator and the end effector; an operator joint angle information acquisition unit that acquires information on the joint angles of the fingers of the operator; an operator's finger length measuring unit that measures the length of each link of the operator's finger; an operator fingertip position calculation unit that calculates the position of a tip of each finger of the operator as viewed from the origin of a local coordinate system of an operator model, based on the information acquired by the operator joint angle information acquisition unit and the operator finger length measurement unit; an end effector joint angle derivation unit that derives a joint angle of the end effector from the fingertip position of the operator and the origin of a local coordinate system of the end effector by inverse kinematics; a position correction unit that measures a difference between a target point of the operator's fingertip and a position of the fingertip of the operator's hand model, and corrects the tip position of each finger as an offset of a fingertip position command, as viewed from the origin of a local coordinate system of the operator model; Remote control system.

5. A control system in which an operator remotely controls an end effector having a finger portion, a coordinate origin setting unit that sets a coordinate origin of the operator and the end effector; an operator joint angle information acquisition unit that acquires information on the joint angles of the fingers of the operator; an operator's finger length measuring unit that measures the length of each link of the operator's finger; an operator fingertip position calculation unit that calculates the position of a tip of each finger of the operator as viewed from the origin of a local coordinate system of an operator model, based on the information acquired by the operator joint angle information acquisition unit and the operator finger length measurement unit; an end effector joint angle derivation unit that derives a joint angle of the end effector from the fingertip position of the operator and the origin of a local coordinate system of the end effector by inverse kinematics; a position correction unit that measures a difference between a target point of a fingertip of the operator and a target point of a finger of the end effector, and corrects the position of the tip of each finger as viewed from the origin of a local coordinate system of the operator model as an offset of a fingertip position command. Remote control system.

6. A control system in which an operator remotely controls an end effector having a finger portion, a coordinate origin setting unit that sets a coordinate origin of the operator and the end effector; an operator joint angle information acquisition unit that acquires information on the joint angles of the fingers of the operator; an operator's finger length measuring unit that measures the length of each link of the operator's finger; an operator fingertip position calculation unit that calculates the position of a tip of each finger of the operator as viewed from the origin of a local coordinate system of an operator model, based on the information acquired by the operator joint angle information acquisition unit and the operator finger length measurement unit; an end effector joint angle derivation unit that derives a joint angle of the end effector from the fingertip position of the operator and the origin of a local coordinate system of the end effector by inverse kinematics; a mode for deriving a joint angle of the end effector; a mode selection unit capable of selecting, for each finger, a mode for moving the robot by estimating the posture of the operator's hand based on sensor information of the robot equipped with the end effector, Remote control system.

7. The mode selection unit Switching modes according to taxonomy or according to the speed of the operator's fingertip; The remote control system of claim 6.

Images