Remote control system, remote control device, remote control method, and program

Abstract

The remote control system is a control system in which an operator remotely operates an end effector with jointed fingers. It comprises: a first detection unit that detects the position of the operator's fingertip; a second detection unit that detects the angle of the operator's finger joints; a judgment unit that determines the operator's operation; a priority determination unit that, based on the operation determined by the judgment unit, decides whether to prioritize the calculation of the end effector's finger joint angle using either the operator's fingertip position or the operator's joint angle; a calculation unit that performs optimal calculation of the joint angle, including the priority determined by the priority determination unit; and a control unit that outputs the joint angle calculated by the calculation unit as a control command to the end effector.

Description

Technical Field

[0001] This invention relates to remote control systems, remote control devices, remote control methods, and programs.

[0002] This application claims priority based on Japanese Patent Application No. 2024-044918, filed on March 21, 2024, the contents of which are incorporated herein by reference. Background Technology

[0003] A system is being developed for an operator to remotely control a robot to perform tasks. The robot performing the task may have an arm and a robotic hand. Furthermore, the robotic hand sometimes has more than two fingers. In such a system, it is difficult to manipulate objects with a multi-finger hand in the same way as a human hand. This is because, due to the differences in shape and degrees of freedom between the robotic hand and the human hand, even using human hand movements as simple input to make the robotic hand respond is insufficient for manipulation. Therefore, a method has been proposed to appropriately map the shape of a human hand to the robotic hand (e.g., see Patent Document 1).

[0004] [Existing technical documents]

[0005] [Patent Literature]

[0006] Patent Document 1: Japanese Patent Application Publication No. 2023-131033 Summary of the Invention

[0007] [The problem the invention aims to solve]

[0008] However, when manipulating objects using human hands and robots, there is no mapping method that can be universally adapted across multiple taxonomy systems. Furthermore, the taxonomy may be, for example, a classification of grasping. Therefore, in existing technologies, the manipulability of a robot's multi-fingered hand is sometimes poor.

[0009] The present invention addresses the aforementioned problems and aims to provide a remote control system, remote control device, remote control method, and program that can improve the operation of a robot's multi-fingered hand.

[0010] [Methods used to solve problems]

[0011] To address the aforementioned issues, the present invention employs the following approach.

[0012] (1) One aspect of the present invention relates to a remote control system that allows an operator to remotely operate an end effector having a jointed finger, wherein the remote control system comprises: a first detection unit that detects the position of the operator's fingertip; a second detection unit that detects the joint angle of the operator's finger; a judgment unit that determines the operation content of the operator; a priority determination unit that determines, based on the operation content determined by the judgment unit, whether to set the priority for calculating the joint angle of the end effector to either the operator's fingertip position or the operator's joint angle; a calculation unit that performs an optimization calculation of the joint angle including the priority determined by the priority determination unit; and a control unit that outputs the joint angle calculated by the calculation unit as a control command to the end effector.

[0013] (2) In the above method (1), it is also possible that, The calculation unit calculates a first cost function related to the relationship between the fingertip positions of the end effector and the operator based on the priority, and the control unit performs optimization calculations based on the calculated first cost function to calculate the joint angle of the fingertip of the end effector.

[0014] (3) In the above method (1), the calculation unit may calculate a second cost function related to the relationship between the angles of the joints of each finger of the end effector and the operator based on the priority, and the control unit may perform optimization calculation based on the calculated second cost function to calculate the joint angles of the finger of the end effector.

[0015] (4) In the above method (3), the calculation unit may also calculate the second cost function by multiplying either the joint angle of the finger of the end effector or the joint angle of the operator's finger by a weight related to the bending of the end effector and the operator's finger.

[0016] (5) In any of the above methods (2) to (4), the first detection unit detects the distance between each finger of the operator, the priority determination unit calculates a third cost function related to the distance between each finger of the end effector and the operator, and the control unit also uses the third cost function to perform optimization calculation to calculate the joint angle of the finger of the end effector.

[0017] (6) In any of the above methods (1) to (5), it can also be that... The first detection unit detects the position of the operator's fingertips based on the distance between the fingertips and the center of the operator's palm. The shape of the operator's hand and fingers when the position of the operator's fingertips is detected is a predetermined shape for grasping.

[0018] (7) One aspect of the present invention relates to a remote control device that remotely operates an end effector having a jointed finger when an operator remotely operates the end effector, wherein the remote control device comprises: a first detection unit that detects the position of the operator's fingertip; a second detection unit that detects the joint angle of the operator's finger; a judgment unit that determines the operation content of the operator; a priority determination unit that, based on the operation content determined by the judgment unit, determines whether to set the priority for calculating the joint angle of the end effector to either the operator's fingertip position or the operator's joint angle; a calculation unit that performs an optimization calculation of the joint angle including the priority determined by the priority determination unit; and a control unit that outputs the joint angle calculated by the calculation unit as a control command to the end effector.

[0019] (8) One aspect of the present invention relates to a remote control method for remotely operating an end effector having a jointed finger, wherein a first detection unit detects the fingertip position of the operator, a second detection unit detects the joint angle of the operator's finger, a judgment unit determines the operation content of the operator, a priority determination unit determines, based on the determined operation content, whether to set the priority for calculating the joint angle of the finger of the end effector to the operator's fingertip position or the operator's joint angle, a calculation unit performs an optimization calculation of the joint angle including the determined priority, and a control unit outputs the calculated joint angle as a control command to the end effector.

[0020] (9) In one aspect of the present invention, when an operator remotely operates an end effector having a jointed finger, the computer of the remote control device that remotely operates the end effector performs the following processing: detecting the position of the operator's fingertip; detecting the joint angle of the operator's finger; determining the operation content of the operator; determining, based on the determined operation content, whether to set the priority of calculating the joint angle of the finger of the end effector to the operator's fingertip position or the operator's joint angle; performing an optimization calculation of the joint angle including the determined priority; and outputting the calculated joint angle as a control command to the end effector.

[0021] [Invention Effects]

[0022] According to the method of the present invention, the operation of a robot's multi-fingered hand can be improved. Attached Figure Description

[0023] Figure 1 This is a diagram used to illustrate the overview of remote control and operation of a robot.

[0024] Figure 2 This is a diagram illustrating a classification example of grasping based on the GRASP taxonomy.

[0025] Figure 3 The diagram shows examples of tasks using the fingertips and tasks using the pads of the fingers and the palm.

[0026] Figure 4 This is a diagram illustrating an example of the structure of the remote control system according to this embodiment.

[0027] Figure 5 This is a diagram showing an example of information stored in the storage unit.

[0028] Figure 6 This is a diagram showing the first mapping example of the fingertip position.

[0029] Figure 7 This is a diagram showing a second example of a mapping of fingertip positions.

[0030] Figure 8 This is a diagram showing an example of a joint angle mapping.

[0031] Figure 9 This is a flowchart of the processing steps of a remote control device according to an embodiment. Detailed Implementation

[0032] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Furthermore, in the drawings used in the following description, the scale of each component has been appropriately altered to make each component a recognizable size.

[0033] In addition, in all the drawings used to illustrate the embodiments, the same symbols are used for parts with the same function, and repeated descriptions are omitted.

[0034] Furthermore, the phrase "based on XX" in this application means "at least based on XX," and also includes situations where it is based on other elements besides XX. Additionally, "based on XX" is not limited to the direct use of XX, but also includes situations where XX has been processed or manipulated. "XX" can be any element (e.g., any information).

[0035] [Overview of Remote Control and Operating Entities]

[0036] First, an overview of remote control and operation will be provided.

[0037] Figure 1 This is a diagram used to illustrate the overview of remote control and operation of a robot. For example... Figure 1 As shown, in the remote operating space, the operator Us wears, for example, the image display device 4 on their head and the operation input units 5 (5L, 5R) on their hands. An environmental sensor 3 is provided in the robot's workspace. Furthermore, the environmental sensor 3 can also be mounted on the robot 2. Additionally, the robot 2 includes an arm 21 (21L, 21R) and an end effector 22 (22L, 22R) (robot hand). Moreover, the end effector 22 has two or more fingers.

[0038] exist Figure 1 In the example, the object obj is, for example, a plastic bottle or bottle, which has a body and a cap.

[0039] The operator Us, for example, remotely operates the robot 2 to manipulate the object obj by moving the hand and fingers wearing the operation input unit 5 while observing the image displayed on the HMD4. Figure 1 One example of the operation is installing and closing the cap on the main body, or opening the cap from the main body. In such operations, Robot 2 needs to hold the main body with its hands and pinch the cap with its fingers.

[0040] [Classification System]

[0041] Next, the taxonomy of grasping will be explained.

[0042] In object grasping and manipulation, grasping is classified based on the request task, the distribution of grasping records, and the size (see, for example, Reference 1).

[0043] Figure 2 This is a diagram illustrating a classification example of grasping based on the GRASP taxonomy. (See diagram for example.) Figure 2 As shown, grip classification is categorized by column direction based on the allocation of strength grip, intermediate grip, and fine grip, human-computer interaction, and the allocation of virtual fingers. Additionally, column classification is based on thumb position, specifically whether the thumb rotates outward or inward. Figure 2 The grasping classification system shown is an example, but is not limited to it.

[0044] Reference 1: Thomas Feix, Javier Romero, others, “The GRASP Taxonomy of Human GraspTypes”, IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb. 2016), IEEE, pp. 66-77

[0045] Figure 3 The diagram shows examples of tasks using the fingertips and tasks using the pads of the fingers and the palm.

[0046] The image of symbol g10 is an example of a task performed using the fingertips, such as picking up a screw (obj1) as the first object and moving it to a second object (obj2) as the object on which the screw is installed. This type of object manipulation using the fingertips in a point-contact manner falls under the category of palmar pinching in the classification system. Figure 2 (No. 9).

[0047] Image g20 illustrates an operation using the fingertips and palm, specifically grasping an object (obj) as a sphere with the fingertips and the palm. This type of object manipulation using the fingertips for surface contact falls under the category of tip-pinch in the classification system. Figure 2 (No. 24).

[0048] In both palm-side pinching and fingertip pinching, the angles of the human joints are roughly the same, but the methods of manipulating the object are different.

[0049] Therefore, in this embodiment, the mapping method of switching applications is based on job status, job content, etc.

[0050] [Remote Control System]

[0051] Next, an example of the structure of the remote control system 1 will be described. Figure 4 This is a diagram illustrating an example of the structure of the remote control system according to this embodiment.

[0052] The remote control system 1 includes, for example, a robot 2, an environmental sensor 3, an image display device 4, an operation input unit 5, a remote control device 6, and a measurement sensor 7.

[0053] Robot 2 may include, for example, an arm 21, an end effector 22, a sensor 23, a drive unit 24, and a communication unit 25.

[0054] The image display device 4 includes, for example, a display unit 41 and a gaze detection unit 42.

[0055] The operation input unit 5 may include, for example, an operation detection unit 51.

[0056] The remote control device 6 includes, for example, an acquisition unit 61, a measurement unit 62 (a first detection unit and a second detection unit), a judgment unit 63, a priority determination unit 64, a calculation unit 65, a control unit 66, an image generation unit 67, an output unit 68, and a storage unit 69.

[0057] The environmental sensor 3 is, for example, an RGB (red, green, blue)-D imaging device capable of acquiring depth information. The environmental sensor 3 is installed in the robot's workspace. Alternatively, the environmental sensor 3 can also be installed in a remote operating space. Furthermore, the environmental sensor 3 can be a combination of a distance sensor and an RGB imaging device. The environmental sensor 3 is connected to the remote control device 6 via a wired or wireless network. The environmental sensor 3 has a communication unit (not shown) that outputs the acquired data to the remote control device 6.

[0058] The image display device 4 is, for example, an HMD (head-mounted display). The image display device 4 and the remote control device 6 are connected via a wired or wireless network. The display unit 41 displays images output by the remote control device 6 required for remote operation in the robot's workspace.

[0059] The gaze detection unit 42 detects the operator's gaze. Alternatively, the image display device 4 may not include the gaze detection unit 42.

[0060] The image display device 4 acquires images from the remote control device 6 and outputs the detected gaze information to the remote control device 6.

[0061] The operation input unit 5 and the remote control device 6 are connected via a wired or wireless network.

[0062] The operation detection unit 51, for example, is a data glove, which detects the movement and position of the operator's hand or fingers. Additionally, the operation input unit 5 includes a communication unit (not shown) and outputs detection data to the remote control device 6.

[0063] The measurement sensor 7 is, for example, an RGB (red, green, blue)-D imaging device capable of acquiring depth D information. The measurement sensor 7 can also be a combination of a distance sensor and an RGB imaging device. The measurement sensor 7 is installed, for example, in a remote operating space. The measurement sensor 7 measures data about the operator's hand (finger length, fingertip spacing, etc.). Furthermore, the measurement sensor 7 is connected to the remote control device 6 via a wired or wireless network. The measurement sensor 74 has a communication unit (not shown) and outputs the measured data to the remote control device 6.

[0064] (robot)

[0065] Robot 2 has at least an arm 21 and an end effector 22. Robot 2 can be a two-armed robot, a bipedal robot capable of walking, and may also have a head and body. In addition, robot 2 is connected to a remote control device 6 via a wired or wireless network.

[0066] One end of arm 21 is connected to the shoulder via a joint, for example, and the other end is connected to the end effector 22 via a joint.

[0067] The end effector 22 is a multi-finger hand with two or more fingers 221 (221-1, ..., 221-n, where n is an integer of 2 or more). The following example illustrates a case with five fingers 221, but the number of fingers 221 is not limited to this; two or more fingers are acceptable. Each finger 221 has at least two joints.

[0068] Sensors 23 are mounted on various joints of the arm 21 and the end effector 22. Sensors 23 may be, for example, encoders, accelerometers, force sensors, 6-axis sensors, etc.

[0069] The drive unit 24 drives the arm 21 and the end effector 22 according to the control commands obtained from the remote control device 6. In addition, actuators are installed at each joint of the arm 21 and the end effector 22.

[0070] The communication unit 25 obtains control commands from the remote control device 6 and outputs the detection values ​​of the sensor 23 to the remote control device 6.

[0071] (Remote control device)

[0072] The remote control device 6 uses information obtained from the environmental sensor 3, the operation input unit 5, and the measurement sensor 7 to operate the robot 2.

[0073] The acquisition unit 61 acquires measurement data from the measurement sensor 7. The acquisition unit 61 acquires data detected by the sensor 23 from the robot 2. The acquisition unit 61 acquires data detected by the environmental sensor 3. The acquisition unit 61 acquires information related to the detected operation from the operation input unit 5.

[0074] The measurement unit 62 acquires measurement data, for example, before remote operation begins, and uses this data to measure the length of the operator's fingers, the distance between the fingertips when the hand is open, etc. Furthermore, these measurements can be performed in advance and stored in the storage unit 69. During measurement, the operator, for example, faces the measurement sensor 7 with their fingers extended, and the palm side of their hand faces the sensor to perform the measurement. The length of the fingers being measured will be described later. The measurement unit 62 uses the measurement values ​​from the measurement sensor 7 to measure the distance between the operator's fingers and the position of the fingertips of each finger.

[0075] The judgment unit 63 determines the task the operator intends to perform. The task may be, for example, a fine motor task using the fingertips, or a task using the fingertips and palm. Furthermore, based on information obtained from the environmental sensor 3 and operation instructions obtained from the operation input unit 5, the judgment unit 63 uses methods such as those described in Japanese Patent Application Publication No. 2022-157101 to estimate the target of the operation and the intention of the operation.

[0076] The priority determination unit 64 determines, based on the judgment result of the judgment unit 63, whether to set the priority of calculating the joint angle of the finger of the end effector 22 to the operator's fingertip position or the operator's joint angle.

[0077] For example, in the case of fingertip operation, the calculation unit 65 calculates weights (e.g., cost functions) related to the fingertip position based on the operator's fingertip position. For example, in the case of operation using the fingertips and palm, the calculation unit 65 calculates weights (e.g., cost functions) related to the joint angles based on the operator's finger joint angles. Furthermore, the weights can also be calculated by the priority determination unit 64 based on priority. The calculation unit 65 performs optimization calculations to calculate the joint angles relative to the robot 2.

[0078] The control unit 66 uses the joint angles for the robot 2 calculated by the calculation unit 65 to generate control commands, and outputs the generated control commands to the robot 2 via the output unit 68. In addition, the control unit 66 can also perform control in a manner that supports the operator's operation, based on the estimated operating intention, such as using the method described in Japanese Patent Application No. 2023-045616.

[0079] The image generation unit 67 uses, for example, data obtained from the environmental sensor 3 to generate an image required for remote operation, which is then displayed on the image display device 4. The image required for remote operation is, for example, an image containing a finger and an object. The image generation unit 67 outputs the generated image data to the image display device 4 via the output unit 68.

[0080] The output unit 68 outputs the image data generated by the image generation unit 67 to the image display device 4. The output unit 68 also outputs the control commands generated by the control unit 66 to the robot 2.

[0081] Storage unit 69 stores the programs, thresholds, mathematical formulas, etc., used by each part of remote control device 6. Storage unit 69, for example... Figure 5 That way, the mapping data is associated with the identification information and stored. Figure 5 This is a diagram showing an example of information stored in the storage unit.

[0082] Additionally, using Figure 4 The structure and operation described are just one example, and are not limited to this. For example, each device has a power source.

[0083] [Mapping]

[0084] Next, the mapping will be explained.

[0085] Figure 6 This is a diagram showing the first mapping example of the fingertip position. Figure 6 An example is setting the position of the fingertips as the distance from the origin O of the palm. Furthermore, the origin O of the palm position can be arbitrarily set by the measuring unit 62, or it can be set by determining the center of the palm by the measuring unit 62. Additionally, Figure 6 The shape of the operator's hand during such measurement can also be predetermined, for example, displayed on the image display device 4. Furthermore, the shape of the hand during measurement is, for example, the shape of the hand when grasping an object, with the thumb and forefinger facing each other. Figure 6 In this context, drOI (I = 1, ..., 5) represents the distance from the origin of the robot's palm to each finger. dhOI (I = 1, ..., 5) represents the distance from the origin of the operator's palm to each finger.

[0086] The operator and robot 2 have different hand shapes and sizes. Therefore, the measurement unit 62 uses the measurement values ​​from the measurement sensor 7 to measure the dhOI related to the length of the operator's fingers and the drOI related to the length of the robot 2's fingers. Then, the measurement unit 62 uses the measured drOI and dhOI to calculate the first cost function Cost. do Furthermore, the measurement unit 62 can also calculate the difference between the distance from the origin of the robot 2's palm to each finger and the distance from the origin of the operator's palm to each finger, and sum the differences between each finger to obtain the first cost function. Alternatively, the measurement unit 62 can also calculate the ratio of the distance from the origin of the robot 2's palm to each finger to the distance from the origin of the operator's palm to each finger, and average the ratios between each finger to obtain the first cost function.

[0087] Figure 7 This is a diagram showing a second example of a mapping of fingertip positions. Figure 7 An example is measuring the distance between the fingers. Figure 7 In this context, drTI (I = 1, ..., 5) represents the distance between the robot's fingers, and dhTI (I = 1, ..., 5) represents the distance between the operator's fingers. Furthermore, in... Figure 7The examples show distances from the tip of the thumb to the tip of the index finger, from the tip of the thumb to the tip of the middle finger, from the tip of the thumb to the tip of the ring finger, from the tip of the thumb to the tip of the little finger, from the tip of the middle finger to the tip of the index finger, from the tip of the ring finger to the tip of the middle finger, and from the tip of the little finger to the tip of the ring finger. However, the reference finger used in measuring the distance can also be different. For example, "from the tip of the middle finger to the tip of the index finger" could also be "from the tip of the index finger to the tip of the middle finger," as long as it is the distance between the fingertips.

[0088] The measurement unit 62 uses the measurement values ​​from the measurement sensor 7 to measure the operator's finger-to-finger distance (dhTI) and the robot 2's drTI. Then, the measurement unit 62 uses these measured finger-to-finger distances (drTI and dhTI) to calculate the third cost function, Cost. df Furthermore, the measuring unit 62 can also calculate the difference between the distance between each finger of the robot 2 and the distance between each finger of the operator, and sum the differences between the distances between each finger to obtain the third cost function. Alternatively, the measuring unit 62 can also calculate the ratio between the distance between each finger of the robot 2 and the distance between each finger of the operator, and average the ratios between the distances between each finger to obtain the third cost function.

[0089] Figure 8 This is a diagram showing an example of a joint angle mapping. Figure 8 An example is the shape of the hand and fingers when pinching an object with the thumb and forefinger. Additionally, Figure 8 The shape of the operator's hand during such measurement can also be predetermined, for example, displayed on the image display device 4.

[0090] The measuring unit 62 uses the results measured by the measuring sensor 7 to calculate the angles of each joint of each finger. For example, the joint angle θ hI0 It is the angle of the third joint of the operator's index finger, joint angle θ. hI1 It is the angle of the second joint of the operator's index finger, joint angle θ hI3 It is the angle of the first joint of the operator's index finger. Additionally, the joint angle θ... rI0 It is the angle of the third joint of the robot's index finger, joint angle θ. rI1 It is the angle of the second joint of the robot's index finger, joint angle θ. rI3 It is the angle of the first joint of the robot's index finger.

[0091] The measurement unit 62 uses the measurement values ​​from the measurement sensor 7 to measure the joint angles of the operator and the robot 2. Then, the measurement unit 62 uses the measured joint angles of each finger of the operator and the robot to calculate the second cost function, Cost. fsAlternatively, the measurement unit 62 can calculate the difference between the joint angles of each finger of the robot 2 and the joint angles of each finger of the operator, and sum the differences between the joint angles of each finger to obtain the second cost function. Alternatively, the measurement unit 62 can calculate the ratio of the joint angles of each finger of the robot 2 to the joint angles of each finger of the operator, and average the ratios between the joint angles of each finger to obtain the second cost function. Furthermore, the measurement unit 62 can also multiply the weights representing the different bending ease of the robot 2 and operator's fingers by the joint angles of the robot 2 or the operator to calculate the second cost function.

[0092] [Processing Steps]

[0093] Next, examples of processing steps for remotely operated tasks will be explained. Additionally, examples of tasks involving "finger-tip tasks" and "finger-pad and palm tasks" will be provided below. Figure 9 This is a flowchart of the processing steps of the remote control device in this embodiment.

[0094] (Step S1) The measuring unit 62 uses the measurement values ​​from the measuring sensor 7 to measure the distance between the operator's fingers and the fingertip position of each finger. The measuring unit 62 also uses the measurement values ​​from the measuring sensor 7 to measure the joint angles of each finger. Furthermore, these processes can be performed in advance and stored in the storage unit 69. Additionally, the measuring unit 62 can measure the fingertip position and joint angles of the robot 2 while measuring the operator's fingertip position and joint angles, and these measurements can also be pre-measured and stored in the storage unit 69.

[0095] Next, examples of processing steps for remotely operated tasks will be explained. Additionally, examples of tasks involving "finger-tip tasks" and "finger-pad and palm tasks" will be provided below. Figure 9 This is a flowchart of the processing steps of the remote control device in this embodiment.

[0096] (Step S2) The judgment unit 63 uses data obtained from the environmental sensor 3 to estimate and obtain the work content. Alternatively, the work content can also be selected or input by the operator through operation of the input unit 5, etc. In this case, the acquisition unit 61 can also acquire the selected or input work content. For example, selection can also cause the display unit 41 of the image display device 4 to display work content candidates, allowing selection by line of sight. Alternatively, the operator can also operate a keyboard (not shown) connected to the remote control device 6 to input or select the work content.

[0097] (Step S3) The determination unit 63 determines whether the task content is, for example, "task using the fingertips" or "task using the fingertips and palm". If the task content is "task using the fingertips", the determination unit 63 proceeds to the processing of step S4, and if the task content is "task using the fingertips and palm", it proceeds to the processing of step S5.

[0098] (Step S4) Based on the judgment result of the judgment unit 63, the priority determination unit 64 determines the operator's fingertip position as the priority for calculating the joint angle of the end effector's fingers. Subsequently, the calculation unit 65 calculates a weight related to the fingertip position (e.g., a second cost function) based on the operator's fingertip position. Alternatively, the calculation unit 65 calculates a weight related to the distance between the operator's fingers (e.g., a third cost function). After processing, the calculation unit 65 proceeds to step S6.

[0099] (Step S5) Based on the judgment result of the judgment unit 63, the priority determination unit 64 determines the operator's joint angle as the priority when calculating the joint angle of the finger of the end effector. Next, the calculation unit 65 calculates the weights (e.g., a second cost function) related to the joint angle based on the joint angle of the operator's finger. After processing, the calculation unit 65 proceeds to the processing in step S6.

[0100] (Step S6) The calculation unit 65 uses the weights obtained in step S6 or step S7, for example, to perform optimization calculations to calculate the joint angles of the robot 2.

[0101] (Step S7) The control unit 66 generates control commands for the robot 2 based on the calculated joint angles.

[0102] (Step S8) The control unit 66 outputs the generated control commands to the robot 2 via the output unit 68 to control the robot 2's actions.

[0103] Thus, in this embodiment, the joint angles of robot 2 are determined based on the task content, for example, by calculating the weights of an evaluation function that prioritizes the operator's fingertip position or the joint angles of the operator's fingers. Furthermore, the weights are, for example, weights for the first to third cost functions respectively, such as α, β, and γ in equation (1) below. The calculation unit 65 changes these weights α, β, and γ according to the task content, thereby changing the dominant cost function for control. Then, the calculation unit 65 performs optimization calculations to minimize the sum of the weighted cost functions (cost) and determines the joint angles of each finger of robot 2. For example, in the case of a pinching task using fingertips, the weights β or γ, or weights β and γ, are changed. Additionally, for example, in the case of a task using the palm, the weight α is changed.

[0104] [Formula 1]

[0105] Furthermore, steps S2 to S8 in the above process can be performed at predetermined time intervals to change the weights used when switching job content. Additionally, step S1 can also be performed at predetermined time intervals.

[0106] Furthermore, the examples above illustrate switching weights based on whether the task content is "tasks using fingertips" or "tasks using fingertips and palms," but are not limited to this. Task content is not limited to "tasks using fingertips" and "tasks using fingertips and palms," but can also be... Figure 2 For other types of tasks, the number of tasks to switch between is not limited to two; it can also be three or more.

[0107] As described above, in this embodiment, when calculating the joint angles of robot 2 based on the operation content, the weights of fingertip position and joint angle are changed.

[0108] It should be noted that in existing technologies, because the fingertip position and classification system are roughly the same, it is possible to perform fingertip pinching, but it is difficult to perform grasping using the fingertips and palms. In contrast, according to this embodiment, both fingertip-based and fingertip-based object manipulation are possible.

[0109] Furthermore, according to this embodiment, both object manipulation using the fingertips and object manipulation using the fingertips and palms can be considered, thereby improving the operability of the robot's multi-fingered hand.

[0110] Furthermore, while the example above illustrates a robot 2 with one arm, it could also be a dual-arm robot with two arms. In such a case, the arm and end effector to be used can be selected according to the task and controlled using the method described above. Alternatively, the arm can be controlled one by one using the method described above, allowing the two end effectors to coordinate the task.

[0111] Furthermore, all or part of the program used to implement the functions of the remote control device 6 in this invention can be recorded on a computer-readable recording medium, allowing the computer system to read and execute the program recorded on the recording medium, thereby performing all or part of the processing performed by the remote control device 6. In addition, the term "computer system" here includes hardware such as an operating system and peripheral devices. Furthermore, "computer system" also includes a WWW system with a homepage providing environment (or display environment). Furthermore, "computer-readable recording medium" refers to removable media such as floppy disks, optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. Moreover, "computer-readable recording medium" also includes media that retain programs for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0112] Alternatively, some or all of these components can be implemented using hardware (including the circuitry) based on LSI (Large Scale Integration) such as ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System on Chip), or through the coordinated use of software and hardware.

[0113] Furthermore, the aforementioned program can also be transmitted from a computer system storing the program in a storage device or similar device to other computer systems via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium capable of transmitting information, such as a network (communication network) like the Internet or a communication line (communication line) like a telephone line. Additionally, the aforementioned program can also be used to implement a portion of the aforementioned functions. Furthermore, it can also be a so-called differential file (differential program) capable of implementing the aforementioned functions through combination with a program already recorded in the computer system.

[0114] The above describes the methods for implementing the present invention using embodiments, but the present invention is not limited to such embodiments, and various modifications and substitutions can be applied without departing from the spirit of the present invention.

[0115] Explanation of reference numerals in the attached figures

[0116] 1…Remote control system, 2…Robot, 3…Environmental sensor, 4…Image display device, 5…Operation input unit, 6…Remote control device, 7…Measuring sensor, 21…Arm, 22…End effector, 23…Sensor, 24…Drive unit, 25…Communication unit, 41…Display unit, 42…Line of sight detection unit, 51…Operation detection unit, 61…Acquisition unit, 62…Measurement unit, 63…Judgment unit, 64…Priority determination unit, 65…Calculation unit, 66…Control unit, 67…Image generation unit, 68…Output unit, 69…Storage unit.

Claims

1. A remote control system, wherein an operator remotely operates an end effector having a jointed finger, wherein... The remote control system has the following features: The first detection unit detects the position of the operator's fingertips; The second detection unit detects the joint angle of the operator's fingers; The judgment unit determines the operation content performed by the operator. The priority determination unit determines, based on the operation content determined by the judgment unit, whether to set the priority for calculating the joint angle of the finger of the end effector to the operator's fingertip position or the operator's joint angle. The calculation unit performs optimization calculations of joint angles, including the priorities determined by the priority determination unit. as well as The control unit outputs the joint angle calculated by the calculation unit as a control command to the end effector.

2. The remote control system according to claim 1, wherein, Based on the priority, the computing unit calculates a first cost function relating to the relationship between the fingertip positions of the end effector and the operator. The control unit performs optimization calculations based on the calculated first cost function to calculate the joint angle of the finger of the end effector.

3. The remote control system according to claim 1, wherein, Based on the priority, the computing unit calculates a second cost function relating to the angles of each joint of each finger of the end effector and the operator. The control unit performs optimization calculations based on the calculated second cost function to calculate the joint angle of the finger of the end effector.

4. The remote control system according to claim 3, wherein, The calculation unit calculates the second cost function by multiplying either the joint angle of the finger of the end effector or the joint angle of the operator's finger by a weight related to the bending of the end effector and the operator's finger.

5. The remote control system according to claim 2 or 3, wherein, The first detection unit detects the distance between each of the operator's fingers. The priority determination unit calculates a third cost function related to the distance between each finger of the end effector and the operator. The control unit also uses the third cost function to perform optimization calculations to calculate the joint angle of the finger of the end effector.

6. The remote control system according to claim 1 or 2, wherein, The first detection unit detects the position of the operator's fingertips based on the distance between the fingertips and the center of the operator's palm. The shape of the operator's hand and fingers when detecting the position of the operator's fingertips is a prescribed shape for grasping.

7. A remote control device for remotely operating an end effector having a jointed finger, wherein... The remote control device includes: The first detection unit detects the position of the operator's fingertips; The second detection unit detects the joint angle of the operator's fingers; The judgment unit determines the operation content performed by the operator. The priority determination unit determines, based on the operation content determined by the judgment unit, whether to set the priority for calculating the joint angle of the finger of the end effector to the operator's fingertip position or the operator's joint angle. The calculation unit performs optimization calculations of joint angles, including the priorities determined by the priority determination unit. as well as The control unit outputs the joint angle calculated by the calculation unit as a control command to the end effector.

8. A remote control method, wherein the remote control method is a method of remotely operating an end effector having a jointed finger while an operator remotely operates the end effector, wherein, The first detection unit detects the position of the operator's fingertips. The second detection unit detects the joint angle of the operator's fingers. The judgment unit determines the operation content performed by the operator. Based on the determined operation content, the priority determination unit decides whether to prioritize the calculation of the joint angle of the end effector's fingertip or the operator's joint angle. The calculation unit performs optimization calculations on the joint angles, including the determined priorities. The control unit outputs the calculated joint angle as a control command to the end effector.

9. A program that, when an operator remotely operates an end effector having a jointed finger, causes a computer of a remote control device remotely operating the end effector to perform the following processing: Detect the position of the operator's fingertips; Detect the joint angle of the operator's fingers; Determine the content of the operator's operation; Based on the determined operation content, it is decided whether to prioritize the calculation of the joint angle of the finger of the end effector as either the operator's fingertip position or the operator's joint angle. Perform optimization calculations of the joint angles, including the determined priorities; and The calculated joint angle is output as a control command to the end effector.

Images