Control device, control method, program, and robot control system

Abstract

To provide a control device, a control method, a program and a robot control system which can control operation of a robot arm with high accuracy.SOLUTION: A robot control system comprises a control device, a tracker and a glove. The control device comprises an input device-side interface part, and a control function part. The tracker transmits a signal representing information on a position of the hand of a user and information on a posture of the hand of the user to the input device-side interface part of the control device. The glove transmits a signal representing information of bending angles of joints of the fingers in the hand of the user to the input device-side interface part of the control device. The tracker is mounted on the glove.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control device, a control method, a program, and a robot control system. [Background technology]

[0002] In recent years, robots are expected to be used in a variety of fields. One method of controlling a robot is for a person to operate it remotely. One operation method is to expect the robot arm to move exactly as the person (operator) operates it using their arm or hand from a remote location. This type of remote operation of a robot arm can be used, for example, to have a robot perform work in place of a person in disaster areas or places that are difficult for people to enter (including places that are dangerous for people or that may have adverse effects on the human body). It is also expected to be used in "remote surgery," where surgeons perform surgery from a remote location without visiting the site. It is also expected to be used for experiential operation in "online tours" conducted via communication lines without people having to go to the site in person.

[0003] "Teleoperation," in which a robot arm in a remote location moves in accordance with the operations (movements) of a person (operator), does not require (or requires few) special skills to be acquired for operation, allowing the person (operator) to operate it intuitively.

[0004] Patent Document 1 describes a robot control method. In particular, paragraph 0034 of Patent Document 1 states, "The operation computer 10 generates commands for the robot 3 mainly based on the motion of the operator 40." Furthermore, paragraph 0266 of Patent Document 1 states, "It operates without delay in the virtual space, simulates the movement of the robot 3 with a physical simulator, and uses the simulation results to synchronize and operate the operator 40 and avatar 41 in the virtual space. Then, data indicating the movement of the avatar 41 is stored in memory and sequentially transmitted to the robot 3."

[0005] Patent Document 2 describes a technology for guiding a robot arm. In particular, paragraph 0192 of Patent Document 2 states, "The robot arm 6 is guided by a motion controller GEN_TRAJ that takes into account various constraints for controlling the robot arm. In particular, this controller takes into account the planning of a set trajectory and the definition of a zone to be treated by the user." Furthermore, paragraph 0187 of Patent Document 2 states, "An advantage of the method of the present invention is that it is possible to generate a new modeled surface in less than 30 ms. These response times make it possible, in particular, to control the movement of the robot arm 6 fast enough to anticipate collisions or sudden movements in order to leave the area as quickly as possible."

[0006] Patent Document 3 describes a technology for vision-based teleoperation of a robot system. In particular, paragraph 0015 of Patent Document 3 states, "FIG. 3 illustrates an example of a system for tracking a human hand in real time and controlling a robot to perform corresponding actions, according to at least one embodiment." Furthermore, paragraph 0020 of Patent Document 3 states, "FIG. 4 illustrates an example of a color glove used to acquire hand pose and segmentation. In at least one embodiment, glove 402 includes five fingers 404, 406, 408, 410, and 412, each colored with a different color of fabric. In at least one embodiment, unique colors are printed to minimize annotation generation using OpenCV color thresholding. In at least one embodiment, the color of back of hand 414 uniquely determines hand pose." Furthermore, paragraph 0023 of Patent Document 3 states, "In at least one embodiment, hand pose can be estimated by three unique keypoints represented by three different colored blobs on the back of the glove."

[0007] Patent Document 4 describes a technique for tracking hands and fingers in a surgical system. For example, FIGS. 2A to 2D of Patent Document 4 disclose an example in which sensors are attached to the fingers of a surgeon. Furthermore, paragraph 0052 of Patent Document 4 states, "The master finger tracking grip 270 includes finger-worn sensors 211, 212 (which may also be referred to as finger- and thumb-worn sensors 211, 212) to independently track the location (position and orientation in one embodiment) of the tip of the index finger 292B and the tip of the thumb 292A, i.e., track the locations of the two fingers of the surgeon's hand. Thus, in contrast to tracking the location of a master tool grip in known minimally invasive surgical systems, the location of the hand itself is tracked." Furthermore, paragraph 0056 of Patent Document 4 states, "The locations (position and orientation) of the thumb 292A and index finger 292B in Figures 2A to 2D are mapped to grip closure parameters, e.g., standardized grip closure values, which are used to control the grip of a teleoperated slave surgical instrument coupled to the master finger-tracking grip 270. In particular, the sensed locations of the thumb 292A and index finger 292B are mapped to grip closure parameters by the hand tracking controller 130." Furthermore, paragraph 0057 of Patent Document 4 states, "Accordingly, the location of a portion of the hand of the surgeon 181 is tracked. Based on the tracked location, a system control parameter, i.e., a grip closure parameter, for the minimally invasive surgical system 100 is generated by the hand tracking controller 130 and provided to the system controller 140. The system controller 140 uses the grip closure parameter when generating a system command that is sent to the teleoperated slave surgical instrument. The system command instructs the teleoperated surgical instrument to configure its end effector to have a grip closure that corresponds to the grip closure parameter. Thus, the minimally invasive surgical system 100 uses the grip closure parameter to control the operation of the teleoperated slave surgical instrument of the minimally invasive surgical system 100." [Prior art documents] [Patent documents]

[0008] [Patent Document 1] International Publication No. 2018 / 097223 [Patent Document 2] Special Publication No. 2022-528499 [Patent Document 3] Patent Publication No. 2021-049635 [Patent Document 4] Special Publication No. 2013-510673 Summary of the Invention [Problem to be solved by the invention]

[0009] Remote control, which involves moving a robot arm in a remote location in accordance with the instructions of a human (operator) using background related technology, is used in fields such as surgery and cell culture, but requires dedicated equipment for each field of application, which is expensive.

[0010] There have been attempts to achieve remote control using a dedicated controller and a general-purpose robot arm, but problems exist with the robot arm's low control precision and poor operability. Furthermore, using a dedicated controller requires some getting used to, making it difficult for the person (operator) to perform the operation intuitively.

[0011] In the related background technology, the surgeon attaches sensors to two fingers to independently track the positions of the thumb tip and the index finger tip. However, this related technology cannot operate a robotic arm based on the movement of the entire human (operator) hand.

[0012] The present invention has been made in consideration of the above circumstances, and aims to solve the problem of controlling a robot arm with high precision. Another problem to be solved by the present invention is to control a robot arm in response to the delicate movements of the entire hand (including the finger joints, etc.) of a person (operator).

[0013] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a control device, a control method, a program, and a robot control system that solve the above-mentioned problems. [Means for solving the problem]

[0014] [1] In order to solve the above problem, a control device according to one aspect of the present invention includes an input device-side interface unit that acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on bending angles of the joints of each finger of the user's hand; a control function unit that generates and outputs a control signal for controlling the movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angles of the joints; and an output device-side interface unit that transmits the control signal generated by the control function unit to the robot arm and receives status information on the robot arm at a predetermined timing and passes it to the control function unit, wherein the status information on the robot arm is information representing the position of the hand, the posture of the hand, and the bending angles of the joints of each finger of the robot arm, which correspond to the position of the hand, the posture of the hand, and the bending angles of the joints, and the control function unit generates the control signal at the predetermined timing based on the status information passed from the output device-side interface unit so that the status of the user's hand and the status information on the robot arm are consistent.

[0015] [2] In another aspect of the present invention, in the control device of [1] above, the control function unit includes a noise correction function unit that, when noise is detected based on the temporal progression of at least one of information on the position of the user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger on the user's hand, corrects any of the information on the position of the user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger on the user's hand that caused the detected noise so as to eliminate or mitigate the detected noise.

[0016] [3] Furthermore, one aspect of the present invention is that in the control device of [1] or [2] above, the input device side interface unit acquires signals representing information on the bending angles of all the joints in the user's hand, and the control function unit generates and outputs the control signal based on the information on the bending angles of all the joints.

[0017] [4] Furthermore, one aspect of the present invention is a control device according to any one of [1] to [3] above, further comprising a capturing function unit that stores a time series of the control signal generated by the control function unit, and, when requested, reads out the stored time series of the control signal and controls the output device side interface unit to transmit the time series of the control signal to the robot arm.

[0018] [5] Also, one aspect of the present invention is a control method including: a step in which an input device-side interface unit acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on bending angles of the joints of each finger of the user's hand; a step in which a control function unit generates and outputs a control signal for controlling movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angles of the joints; and a step in which an output device-side interface unit transmits the control signal generated by the control function unit to the robot arm and receives status information of the robot arm at a predetermined timing and passes it to the control function unit, wherein the status information of the robot arm is information representing the position of the hand, the posture of the hand, and the bending angles of the joints of each finger of the robot arm, which correspond to the position of the hand, the posture of the hand, and the bending angles of the joints, and the control function unit generates the control signal at the predetermined timing based on the status information passed from the output device-side interface unit so that the status of the user's hand and the status information of the robot arm are consistent.

[0019] [6] Also, one aspect of the present invention is a program for causing a computer to function as a control device, comprising: an input device-side interface unit that acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on bending angles of the joints of each finger of the user's hand; a control function unit that generates and outputs a control signal for controlling the movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angles of the joints; and an output device-side interface unit that transmits the control signal generated by the control function unit to the robot arm and receives status information of the robot arm at a predetermined timing and passes it to the control function unit, wherein the status information of the robot arm is information representing the position of the hand, the posture of the hand, and the bending angles of the joints of each finger of the robot arm, which correspond to the position of the hand, the posture of the hand, and the bending angles of the joints; and the control function unit generates the control signal at the predetermined timing based on the status information passed from the output device-side interface unit so that the status of the user's hand and the status information of the robot arm are consistent.

[0020] [7] Another aspect of the present invention is a robot control system comprising: a control device according to any one of [1] to [4] above; a tracker that transmits signals representing information on the position of a user's hand and information on the posture of the user's hand to the input device side interface unit of the control device; and a glove that transmits signals representing information on the bending angle of each finger joint in the user's hand to the input device side interface unit of the control device.

[0021] [8] Another aspect of the present invention is a robot control system comprising: a control device according to any one of [1] to [4] above; and a robot arm that operates based on the control signal transmitted from the output device side interface unit of the control device and transmits the status information of the robot arm to the output device side interface unit at the predetermined timing.

[0022] [9] Another aspect of the present invention is a robot control system comprising: a control device according to any one of [1] to [4] above; a tracker that transmits signals representing information on the position of a user's hand and information on the posture of the user's hand to the input device side interface unit of the control device; a glove that transmits signals representing information on the bending angle of each finger joint in the user's hand to the input device side interface unit of the control device; and a robot arm that operates based on the control signal transmitted from the output device side interface unit of the control device and transmits the status information of its own device to the output device side interface unit at the predetermined timing. [Effects of the Invention]

[0023] According to the present invention, the control function unit can generate a control signal based on the status information received from the robot arm so that the status of the user's hand matches the status information of the robot arm, which means that the control device can control the robot arm with high precision. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a robot control system according to a first embodiment of the present invention. [Figure 2] 1 is a schematic diagram showing an example of the appearance of a glove used in a robot control system according to a first embodiment. FIG. [Figure 3] FIG. 2 is a schematic diagram showing an example of the appearance of a tracker used in the robot control system according to the first embodiment. [Figure 4] 1 is a schematic diagram showing an example of the appearance of a robot arm used in a robot control system according to a first embodiment. FIG. [Figure 5] FIG. 3 is a block diagram showing a more detailed functional configuration of a control function unit according to the first embodiment. [Figure 6]2 is a schematic diagram for explaining the principle when the control function unit according to the first embodiment controls the robot arm 7. FIG. [Figure 7] 4 is a graph showing the transition of the speed on the input side before the noise correction function unit according to the first embodiment corrects noise. [Figure 8] 10 is a graph showing the transition of the speed on the input side after the noise correction function unit according to the first embodiment corrects noise. [Figure 9] FIG. 2 is a schematic diagram showing the operation of capturing and reproducing a motion by the motion capture teaching function unit according to the first embodiment. [Figure 10] 4 is a flowchart showing a procedure of processing (control) by the control device according to the first embodiment. [Figure 11] FIG. 10 is a functional block diagram showing an example of a minimum configuration of a control device according to a second embodiment. [Figure 12] FIG. 10 is a functional block diagram showing the internal functional configuration of a control function unit in a second embodiment. [Figure 13] FIG. 2 is a block diagram showing the functional configuration of a computer for realizing the first or second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0025] Next, an embodiment of the present invention will be described with reference to the drawings. In the embodiment described below, a control device controls a robot arm to move in accordance with the hand movement of a human (operator) detected by an input device. Specifically, the control device acquires the state of the input device and generates speed information of each part of the output device (robot arm) based on that information. This speed information is control information. The control device controls the output device by transmitting the generated control information to the output device. In this embodiment, the output device detects its own state and feeds back that state information to the control device. The control device can improve the accuracy of control by generating control information based on the fed-back state information. However, because frequent feedback of state information from the output device to the control device increases the load on the entire control, feedback of state information is performed only at specified times.

[0026] [First embodiment] 1 is a block diagram showing a schematic functional configuration of a robot control system according to this embodiment. As shown in the figure, the robot control system 1 includes a control device 2, a glove 3, a tracker 4, a robot arm 7, and a communication line 9. Of these components, the glove 3 and the tracker 4 are also called "input devices," and are devices that input (supply) signals that serve as the basis for controlling the robot arm 7 to the control device 2. The robot arm 7 is also called "output device," and is a device that is controlled by control signals output by the control device 2.

[0027] At least one of the glove 3 and the tracker 4 may be located outside the robot control system 1. In this case, the glove 3 and the tracker 4 each input (supply) a signal (the content of the signal will be explained later) to the control device 2. The robot arm 7 may also be located outside the robot control system 1. In this case, the movement of the robot arm 7 is controlled by a control signal output from the control device 2. The robot control system 1 may also be configured not to include a communication line 9. In this case, signals are exchanged between the control device 2 and the robot arm 7 without going through the communication line 9.

[0028] The schematic functional configuration of the control device 2 is as follows: As shown in the figure, the control device 2 includes an input device side interface unit 21, a control function unit 22, and an output device side interface unit 23.

[0029] At least some of the functions of the control device 2, glove 3, tracker 4, and robot arm 7 may be implemented using electronic circuits. At least some of the functions of the glove 3, tracker 4, and robot arm 7 may be implemented using a computer and a program. The control device 2, glove 3, tracker 4, and robot arm 7 may include storage means for at least temporarily storing information. The storage means may include non-volatile storage means such as a magnetic hard disk drive (HDD) or a solid state drive (SSD). The storage means may also include volatile memory.

[0030] Each function constituting the robot control system 1 will be explained below.

[0031] The control device 2 controls the movement of the robot arm 7 based on signals sent from the glove 3 and the tracker 4. Furthermore, if the robot arm 7 has a robot hand, the control device 2 also controls the movement of the robot hand held by the robot arm.

[0032] The control device 2 may control one robot arm 7 in response to the movement of only one hand of a human (operator). In this case, the control device 2 receives signals representing the bending angles of the joints of one hand (left hand or right hand) of the human (operator) from one glove 3. The control device 2 also receives signals representing the position and posture (orientation) of one hand (left hand or right hand) of the human (operator) from one tracker 4. The control device 2 controls one robot arm 7 based on these signals.

[0033] Alternatively, the control device 2 may control two robot arms 7 in response to the movements of both hands of a human (operator). In this case, the control device 2 basically controls the movement of one of the two robot arms 7 (the robot arm corresponding to the right hand) based on operation with the human's (operator's) right hand. The control device 2 also controls the movement of the other robot arm 7 (the robot arm corresponding to the left hand) based on operation with the human's (operator's) left hand. The control device 2 receives signals representing the bending angles of the joints of one hand (left hand or right hand) of the human (operator) from each of the two gloves 3. The control device 2 also receives signals representing the position and posture (orientation) of one hand (left hand or right hand) of the human (operator) from each of the two trackers 4. The control device 2 controls the two robot arms 7 based on these signals.

[0034] The glove 3 is a device that can be worn on the hand of a human (operator). The glove 3 is equipped with multiple angle sensors that detect the bending angles of the joints of the hand of the human wearing the glove 3 and output the angle information. The fingers of a human hand are made up of many joints. One glove 3 detects the bending angles of a total of 15 joints for one human hand (right or left hand), for example, the DIP joint (first joint), MP joint (third joint), and CM joint (fourth joint) of the thumb, the DIP joint (first joint), PIP joint (second joint), and MP joint (third joint) of the index finger, the DIP joint (first joint), PIP joint (second joint), and MP joint (third joint) of the middle finger, the DIP joint (first joint), PIP joint (second joint), and MP joint (third joint) of the ring finger, and the DIP joint (first joint), PIP joint (second joint), and MP joint (third joint) of the little finger, and outputs the information. The angles are expressed as a numerical value in units of degrees or radians, for example. When gloves 3 are used for both hands, the pair of gloves 3 detects the bending angles of a total of 30 joints and outputs that information. The gloves 3 pass signals representing information on the angles of multiple joints to the control device 2. In other words, the gloves 3 pass angle information expressed as, for example, a 15-dimensional vector for each hand to the control device 2.

[0035] That is, the glove 3 transmits a signal representing information about the bending angle of each joint of the finger of the user's hand to the input device side interface unit 21 of the control device 2. The glove 3 may detect information about the bending angles of all joints in the user's hand and pass the signal representing the information about the bending angles of all the joints to the input device side interface unit 21.

[0036] As an example, the MANUS Prime II series gloves can be used as the gloves 3. However, other products may also be used as the gloves 3.

[0037] The tracker 4 is a device that detects its own position and orientation and outputs that information. In this embodiment, one tracker 4 is attached to one glove 3. That is, one tracker 4 detects the position and orientation of the hand wearing the glove 3 and outputs that information. When gloves 3 are used on both hands, the trackers 4 output position and orientation information for each hand wearing the glove 3. The position is a position in a coordinate system of X-, Y-, and Z-axes (e.g., a Cartesian coordinate system) and is expressed as a three-dimensional vector. The orientation is, for example, the amount of rotation about each of the X-, Y-, and Z-axes and is expressed as a three-dimensional vector. That is, the tracker 4 passes position and orientation information expressed as a six-dimensional vector for each hand to the control device 2.

[0038] That is, the tracker 4 transmits to the input device side interface unit 21 of the control device 2 a signal representing information on the position of the user's hand and information on the posture of the user's hand.

[0039] As an example, the VIVE Tracker by HTC CORPORATION can be used as the tracker 4. However, other products may also be used as the tracker 4.

[0040] The robot arm 7 is controlled by the control device 2. The robot arm 7 has arms and joints that correspond to a human arm, which allows it to change the position and posture (direction) of the parts that correspond to a human hand. The robot arm 7 also has joints that correspond to the joints of a human hand (finger joints), and the bending angle of each joint can be changed independently. As mentioned above, there may be one or two robot arms 7. When there are two robot arms 7, the two robot arms 7 are controlled by the control device 2 based on the movements of both hands (left and right hands) of the human (operator). The robot arm itself is realized using existing technology.

[0041] The robot arm 7 has a function of feeding back the state of its own device to the control device 2. In other words, the robot arm 7 detects the position and posture (direction) of the parts corresponding to the hand, and the bending angles of the joints of the parts corresponding to each finger. Then, the robot arm 7 outputs this information (an output vector V Ok (information expressed as) is fed back to the control device 2. The robot arm 7 feeds back such information, and the control device 2 performs control based on the feedback information from the robot arm 7, thereby synchronizing the movements between the input device side (gloves 3 and tracker 4) and the output device side (robot arm 7).

[0042] In other words, the robot arm 7 operates based on the control signal transmitted from the output device side interface unit 23 of the control device 2, and transmits the status information of its own device to the output device side interface unit at the predetermined timing.

[0043] As an example, products such as UR3e, UR5e, UR10e, UR16e, and UR20 manufactured by Universal Robots can be used as the robot arm 7. However, other products may also be used as the robot arm 7.

[0044] The communication line 9 enables the exchange of information between the control device 2 and the robot arm 7. The communication line 9 may be, for example, a dedicated line provided for a specific user, or a public line (e.g., the Internet) shared by unspecified users. The existence of the communication line 9 makes it possible for the control device 2 to control the robot arm 7 even when the control device 2 and the robot arm 7 are located remotely from each other.

[0045] The input device side interface unit 21 of the control device 2 receives signals representing the position and orientation of the hand from the tracker 4, and receives signals representing the bending angles of the joints of each finger held by the hand from the glove 3. The input device side interface unit 21 passes the information (numerical information) represented by these received signals to the control function unit 22.

[0046] That is, the input device interface unit 21 acquires signals representing information on the position of the user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand. The input device interface unit 21 may acquire signals representing information on the bending angles of all the joints in the user's hand from the glove 3.

[0047] The control function unit 22 of the control device 2 generates and outputs a signal for controlling the robot arm 7 based on information (numerical information) received from the input device side interface unit 21. The control function unit 22 passes the generated signal for controlling the robot arm 7 to the output device side interface unit 23. The control signal in this embodiment is, for example, a signal that indicates the speed of operation of each part of the robot arm 7.

[0048] That is, the control function unit 22 generates and outputs a control signal for controlling the movement of the robot arm 7 corresponding to the user's hand, based on the position, posture, and bending angles of the joints of the user's hand. Furthermore, the control function unit 22 generates a control signal at a predetermined timing (the timing corresponding to when the output device side interface unit 23 receives state information from the robot arm 7) based on the state information passed from the output device side interface unit 23, so that the state of the user's hand (i.e., the state of the input device) and the state information of the robot arm 7 are consistent. How this consistency (synchronization) is achieved will be described in more detail later. When the input device side interface unit 21 acquires signals representing information on bending angles of all the joints in the user's hand from the glove 3, the control function unit 22 may generate and output a control signal based on the information on bending angles of all of the joints.

[0049] The output device side interface unit 23 of the control device 2 receives the control signal (a signal for controlling the robot arm 7) generated by the control function unit 22 and transmits it to the robot arm 7. The output device side interface unit 23 may transmit the control signal to the robot arm 7 via the communication line 9. That is, the output device side interface unit 23 transmits the control signal generated by the control function unit 22 to the robot arm 7. The output device side interface unit 23 also receives status information on the robot arm 7 at a predetermined timing (for example, every certain number of frames, but is not limited to this timing) and passes it to the control function unit 22. The status information on the robot arm 7 is information that represents the hand position, hand posture, and bending angles of the joints of each finger on the robot arm 7 side, which correspond to the hand position, hand posture, and bending angles of the joints on the input device side.

[0050] FIG. 2 is a schematic diagram showing the appearance of glove 3. As shown in the figure, glove 3 has a shape suitable for wearing on a human hand. While glove 3 shown in FIG. 2 is intended to be worn on a human right hand, a glove 3 having a symmetrical shape and intended to be worn on a human left hand may also be implemented. Glove 3 shown in FIG. 2 is made of, for example, cloth (fiber), synthetic resin, rubber, etc., and its shape can flexibly change in response to the bending of the joints in the human hand. Glove 3 also includes sensors for detecting the bending angle of each joint in the human hand. The tracker 4 can be attached to the back of the glove 3. However, the tracker 4 may be attached to a part of the glove 3 other than the back of the hand.

[0051] 3 is a schematic diagram showing an example of the appearance of tracker 4. Tracker 4 is a small device that can be used by attaching it to, for example, glove 3 as described above. By attaching it to glove 3, tracker 4 can output numerical values ​​that represent the position (coordinate values ​​on three-dimensional coordinate axes) and posture (amount of rotation around three-dimensional axes) of the hand of a person (operator) wearing glove 3.

[0052] FIG. 4 is a schematic diagram showing an example of the appearance of the robot arm 7. As shown in the figure, the base of the robot arm 7 can be fixed to a stand or the like for use. The robot arm 7 also has multiple arms and joints. A robot hand can be attached to the tip of the robot arm 7.

[0053] 5 is a block diagram showing a more detailed functional configuration of the above-described control function unit 22. As shown in the figure, the control function unit 22 includes a control information generation unit 220, a synchronization control function unit 221, a noise correction function unit 222, and a motion capture teaching function unit 225.

[0054] The control information generation unit 220 generates and outputs a control signal for controlling the movement of the robot arm corresponding to the hand based on the position, posture, and joint bending angle of the human (operator, user) hand passed from the input device side interface unit 21.

[0055] The control information generation unit 220 generates a control signal at a predetermined timing based on the status information passed from the output device side interface unit 23 (status information of the robot arm 7 passed as feedback information from the robot arm 7 side) so that the status of the human (operator, user) hand (i.e., the status detected by the input device (glove 3 or tracker 4)) matches with the status information of the robot arm 7 side. The matching between these two is synchronization based on the status information fed back. The synchronization control function unit 221 performs the processing for this synchronization, and the control information generation unit 220 generates control information based on the result of the synchronization processing.

[0056] Based on the above feedback information (status information returned from the robot arm 7), the synchronization control function unit 221 performs processing to align the status of the human (operator, user) hand (i.e., the status detected by the input device (glove 3 or tracker 4)) with the status information on the robot arm 7.

[0057] The noise correction function unit 222 corrects noise contained in information passed from the input device (gloves 3 or tracker 4) side. Specifically, when the velocity information calculated based on the information passed from the input device (gloves 3 or tracker 4) side has a singular value, the noise correction function unit 222 corrects the velocity information.

[0058] Specifically, the noise correction function unit 222 determines whether noise is detected based on the time progression (i.e., speed information) of at least one of the following information: information on the position of the user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger on the user's hand. If noise is detected in any of the information, the noise correction function unit 222 corrects any of the information on the position of the user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger on the user's hand that caused the noise so as to remove or mitigate the detected noise (for example, by taking the average value with the speeds in nearby frames, as described below).

[0059] The noise correction process by the noise correction function unit 222 will be described in detail later with reference to FIGS.

[0060] The motion capture teaching function unit 225 receives and stores the time series of control signals generated by the control function unit 22. When requested, the motion capture teaching function unit 225 reads out the stored time series of control signals and controls the output device side interface unit 23 to transmit the time series of control signals to the robot arm 7.

[0061] The motion capture teaching function unit 225 is also called a "capturing function unit." The motion capture teaching function unit 225 realizes highly accurate teaching. For example, the motion capture teaching function unit 225 enables highly flexible and highly accurate teaching through a few key inputs by the user. The motion capture teaching function unit 225 may also record and list multiple teachings (time series of control information). By listing them, the user (operator) can select and execute a desired motion.

[0062] 6 is a schematic diagram for explaining the principle when the control function unit 22 controls the robot arm 7. Here, a method for the control function unit 22 to control the robot arm 7 will be described with reference to FIG.

[0063] FIG. 6(A) shows a series of input vectors. FIG. 6(B) shows a series of vectors representing the state of the robot arm 7 (output side). V in FIG. 6(A) I0 ,V I1 ,V I2 ,V I3 ,··· (continued below) is a sequence of input vectors. V in Figure 6(B) O0 ,V O1 ,V O2 ,V O3 ,V O4 , (continued below) is a sequence of output vectors (vectors representing the state of the robot arm 7). I0 ,V I1 ,V I2 ,V I3 ,··· and V in Fig. 6(B) O0 ,V O1 ,V O2 ,V O3 ,V O4 ,··· are multidimensional vectors as shown below, but for convenience they are shown projected onto two-dimensional space.

[0064] Here, the input vector sequence V I0 ,V I1 ,V I2 ,V I3 , may be a sequence of numerical values ​​(vectors) output by the tracker 4. Also, the input vector sequence V I0 ,V I1 ,V I2 ,V I3 , ... IkEach of (k=0,1,2,...) is a 6-dimensional vector (position (3D) and orientation (3D) for either the left or right hand alone) or a 12-dimensional vector (position and orientation for both the left and right hands).

[0065] Alternatively, the input vector sequence V I0 ,V I1 ,V I2 ,V I3 , ... I0 ,V I1 ,V I2 ,V I3 , ... Ik Each of (k=0,1,2,...) is a 21-dimensional vector (the position (3 dimensions), posture (3 dimensions), and bending angles of 15 joints for either the left or right hand only), or a 42-dimensional vector (the above 21 dimensions for each of the left and right hands, a total of 42 dimensions).

[0066] However, the vector V Ik The number of dimensions (k=0, 1, 2,...) does not necessarily have to be as described above. For example, information on the bending angles of some joints may be missing, or information other than the information of the number of dimensions described above may be included.

[0067] Also, the output vector sequence V O0 ,V O1 ,V O2 ,V O3 ,V O4 ,..., for example, the input vector sequence V I0 ,V I1 ,V I2 ,V I3 , ... Ik (including the coordinate system of the 3D space output by the tracker 4) and the output vector V OkThe space represented by the coordinate system (including the coordinate system of the three-dimensional space in which the robot arm 7 operates) may not necessarily be the same. In other words, the input vector and the output vector may be represented as values ​​in different coordinate systems. In this case, the control function unit 22 appropriately performs two-way conversion of coordinate values ​​etc. between both coordinate systems.

[0068] The control function unit 22 calculates the input vector V Ik Furthermore, the control function unit 22 can generate an output vector V from the output device (robot arm 7) as needed. Ok However, as will be described later, the control function unit 22 can obtain the output vector V from the output device (robot arm 7). Ok Reducing the frequency of acquiring this information leads to a reduction in the processing load of the control device 2 itself and a reduction in the communication load of the communication line 9.

[0069] The control function unit 22 controls the robot arm 7 at a fixed frame period. This frame period is designated as Δt. Δt may be, for example, Δt≦22.300 ms (milliseconds). As an example, Δt=22.222 ms. When Δt=22.222 ms, the frame rate (1 / Δt) is 45.000 fps (frames per second) with three decimal places. In other words, the frame rate is approximately 45 fps.

[0070] The shorter the frame period Δt for control, the more accurate the control by the control function unit 22. On the other hand, the shorter the frame period Δt, the more computational resources the control device 2 requires for control. Furthermore, if the frame period Δt is set too short, the glove 3, tracker 4, and robot arm 7 will require high-speed responsiveness, resulting in higher costs for the device. Furthermore, to reduce the frame period Δt, the communication line 9 will also require specifications such as high capacity and low latency, which will increase costs. A frame rate of approximately 45 fps (Δt = 22.222 [ms]) or a frame rate in the range of 40 fps to 50 fps can achieve both high control accuracy and low-cost system implementation.

[0071] Specifically, the control function unit 22 generates velocity information (velocity vector) to control the robot arm 7, and transmits the velocity information to the robot arm 7 via the output device side interface unit 23. The robot arm 7 operates in accordance with the velocity information (velocity vector) passed from the control function unit 22.

[0072] The control function unit 22 can control the robot arm 7 by appropriately using the following two types of velocity information (velocity vectors). The first velocity information and second velocity information are control information for controlling the robot arm 7.

[0073] The first velocity information is velocity information based only on information supplied from the input side device (glove 3 or tracker 4). In other words, the first velocity information is information for controlling the robot arm 7 only by feedforward. The first velocity information is s k In addition, the speed information s k is sought in a discrete time series.

[0074]

number

[0075] That is, the kth first speed information in the sequence, s k is the previous input vector V I(k-1) This time's input vector V Ik The displacement of vector s is divided by the frame period Δt. k The number of dimensions of the input vector V Ik The number of dimensions is the same as that of

[0076] The second velocity information is velocity information obtained using information fed back from the output side device (robot arm 7). In other words, the first velocity information is information for controlling the robot arm 7 using a feedback element. The second velocity information is s k is.

[0077]

number

[0078] The function f() is a function that converts the coordinate system of the input vector into the coordinate system of the output vector. -1 () is the inverse function of the above function f(), and is a function that converts the coordinate system of the output vector into the coordinate system of the input vector. In other words, s, which is the kth second speed information in the sequence, k is the previous output vector V O(k-1) From the input vector corresponding to Ik The displacement of vector s is divided by the frame period Δt. k The number of dimensions of the input vector V Ik It is possible to assume that the number of dimensions of the input vector and the number of dimensions of the output vector are the same.

[0079] Second speed information s k is generated by the control information generator 220 based on the results of the synchronization process performed by the synchronization control function unit 221 described above. (control information)

[0080] The control function unit 22 may control the robot arm 7 by appropriately using the first velocity information or the second velocity information.

[0081] As an example, the control function unit 22 controls the robot arm 7 using the second velocity information only once (for example, only when the frame number is divisible by the integer P1) per P1 frame (P1 is a positive integer and a parameter value that can be set in advance; for example, P1=5, etc.). In this case, the control function unit 22 controls the robot arm 7 using the first velocity information at other times (for example, when the frame number is not divisible by the integer P1). In this case, the input vector V Ik The information represented by and the output vector V Ok Synchronization with the information represented by is attempted.

[0082] In addition to the above example, the input vector V Ik The information represented by and the output vector V Ok The information may be synchronized with the information represented by the .

[0083] Specifically, the control function unit 22 applies the function f() to the first velocity information or the second velocity information, thereby transmitting a control value (velocity information) in the coordinate system of the output vector to the robot arm 7. That is, the control function unit 22 transmits velocity information shown in the following equation (3) to the robot arm 7.

[0084]

number

[0085] That is, the control function unit 22 calculates the speed information s using the function f(). k is transformed into the coordinate system of the output vector, and the robot arm 7 is controlled using the transformation result.

[0086] 7 and 8 are graphs for explaining the noise correction by the noise correction function unit 222 in the control function unit 22. The noise correction function unit 222 corrects the input vector V Ik , which corrects velocity noise caused by noise that may be present in the input vector V. FIG. 7 is a line graph showing the transition of velocity before correction by the noise correction function unit 222. FIG. 8 is a line graph showing the transition of velocity after correction by the noise correction function unit 222. The noise correction function unit 222 corrects velocity noise caused by noise that may be present in the input vector V. Ik Instead of correcting the overall velocity, the input vector V Ik Speed ​​correction is performed for each element of (however, there may be elements for which speed correction is not performed).

[0087] Here, the input vector V Ik The m-th element of V Ik (m) Then, for the m-th element, the velocity s calculated from the (k-1)th input and the k-th input is k (m) is expressed by the following equation (4).

[0088]

number

[0089] Figure 7 shows the input vector V Ik For a specific m-th element of k (m) The horizontal axis of Fig. 7 is the frame number (discrete time), and the vertical axis is the velocity s k (m) In Figure 7, the unit of velocity is meters per second (m / sec).

[0090] The method by which the noise correction function unit 222 corrects the noise contained in this speed is as follows: First, the noise correction function unit 222 corrects the speed s k (m)Specifically, the noise correction function unit 222 determines whether the current speed s k (m) Whether or not the absolute value of is P3 times or more the average of the absolute values ​​of the most recent P2 speeds is used to determine whether or not the speed is noise.

[0091] Note that P2 is a configurable parameter value and is a positive integer. For example, P2=3 may be used. P3 is a configurable parameter value and is a real number greater than 1.0. For example, the value of P3 may be set appropriately within the range of 5.0≦P3≦10.0. In other words, for example, when P2=3 and P3=5.0, the noise correction function unit 222 calculates the current speed s k (m) The absolute value of s k-1 (m) The absolute value of and s k-2 (m) The absolute value of and s k-3 (m) If the absolute value of is 5.0 times or more of the average of k (m) is determined to contain noise.

[0092] This time's speed s k (m) If the result of the above determination is that noise is included, the noise correction function unit 222 calculates the average value of the absolute value of the speed of the most recent P2 times and the absolute value of the current speed (a total of (P2+1) absolute values ​​of speeds), and calculates the current speed s k (m) The velocity s is forced to be equal to the absolute value of the calculated average value. k (m) At this time, the noise correction function unit 222 corrects s k (m) The sign (positive or negative) is corrected to be the same as the original sign.

[0093] That is, the noise correction function unit 222 corrects the s k (m) The input vector V Ik Forces the value of the mth element of to be rewritten.

[0094] The noise correction function unit 222 calculates the input vector V Ik The noise correction function unit 222 may perform the noise correction for all elements (all m) of the input vector V Ik The noise correction function unit 222 may perform the noise correction only on the position information (three-dimensional (X-axis, Y-axis, Z-axis) coordinate values, respectively, output by the tracker 4 among the elements of the input vector V. Ik The above noise correction may be performed only on other element sets that .

[0095] Figure 8 shows the input vector V Ik The velocity s after the noise correction function unit 222 performs noise correction on the m-th element of k (m) In Fig. 8, the horizontal axis represents the frame number (discrete time), and the vertical axis represents the velocity s k (m) The unit of this velocity is meters per second (m / sec).

[0096] 7 and 8, noises appearing in the graph of FIG. 7 (for example, noises indicated by symbols N1, N2, and N3 in FIG. 7) are removed in the graph of FIG. In other words, even if the input device (tracker 4 or glove 3) generates peculiar noise, the noise correction function unit 222 can correct the noise, thereby stabilizing the control of the robot arm 7 by the control function unit 22.

[0097] FIG. 9 is a schematic diagram showing the operation of capturing and reproducing a motion by the motion capture teaching function unit 225. FIG. 9(A) shows the operation when capturing a motion. Also, FIG. 9(B) shows the operation when reproducing the captured motion. Whether or not to capture a motion may be settable during operation of the control device 2. Furthermore, when reproducing a captured motion, it may be performed based on a request from an external user, for example. The "control signal" below refers to f(s k ) In other words, the control signal is a signal that represents information about the speed to be instructed to the robot arm 7 in each frame.

[0098] 9(A), during capturing, the control signal generated by the control information generation unit 220 is also passed to the motion capture teaching function unit 225. The motion capture teaching function unit 225 stores the time series of this control signal in a recording medium or the like.

[0099] 9(B), the motion capture teaching function unit 225 reproduces the stored motion when requested. That is, the motion capture teaching function unit 225 reads the stored time series of control signals from a recording medium or the like, and controls the output device side interface unit 23 to transmit the time series of control signals to the robot arm 7. In the example shown in FIG. 9(B), the motion capture teaching function unit 225 passes the read time series of control signals to the output device side interface unit 23 via the control information generation unit 220. Alternatively, the motion capture teaching function unit 225 may pass the read time series of control signals directly to the output device side interface unit 23 without passing through the control information generation unit 220.

[0100] In this way, the motion capture teaching function unit 225 can capture motion and reproduce the captured motion. When reproducing the captured motion, the control function unit 22 does not receive feedback of state information from the robot arm 7.

[0101] 10 is a flowchart showing the procedure of processing by the control device 2. The processing procedure (control method) will be explained below with reference to this flowchart.

[0102] In step S1, the control device 2 performs initialization for controlling the robot arm 7. The initialization processing in this step includes processing for matching (initial synchronization) the position, posture, and bending angles of the joints on the input device (gloves 3, tracker 4) side with the position, posture, and bending angles of the joints on the output device (robot arm 7) side. In addition, in this step, the control device 2 may perform other initialization processing for controlling the robot arm 7.

[0103] The following steps S2 to S9 are processes for controlling each frame. The processing for each frame may be performed at predetermined time intervals (the above-mentioned frame period Δt).

[0104] In step S2, the input device side interface unit 21 of the control device 2 receives the current vector information (the aforementioned input side vector V Ik The input device side interface unit 21 obtains the obtained input side vector V Ik is passed to the control function unit 22. The control function unit 22 then Ik is stored at least temporarily.

[0105] In step S3, the control function unit 22 (noise correction function unit 222) performs a correction process for the velocity noise. The correction process for the velocity noise has already been described. By the correction process for the velocity noise, the input vector VIk In addition, if there is no velocity noise, the correction process in this step does not produce any specific effect, and the input vector V obtained in step S2 is Ik is used as is.

[0106] In step S4, the control function unit 22 (synchronization control function unit 221) determines whether or not to synchronize the position and the like with the robot arm 7 in this control. If synchronization with the robot arm 7 is to be performed (step S4: YES), the process proceeds to step S6. If synchronization with the robot arm 7 is not to be performed (step S4: NO), the process proceeds to step S5.

[0107] As described above, the synchronization control function unit 221 may perform synchronization (step S4: YES) for every certain number of frames (in the above example, every P1 frames). Alternatively, it may determine whether to perform synchronization based on other conditions.

[0108] When the process proceeds to step S5, in this step, the control function unit 22 (control information generation unit 220) calculates the current vector on the input device side (input side vector V Ik ) and the previous vector on the input device side (input side vector V I(k-1) ) and based on the velocity information (velocity vector) s k In this step, the first speed information s is generated using the calculation according to the above-mentioned formula (1). k After the processing of this step, the process proceeds to step S8.

[0109] When the process proceeds to step S6, in this step, the output device side interface unit 23 receives the previous vector information (output side vector V O(k-1) The output device side interface unit 23 obtains the obtained output side vector V O(k-1) is passed to the control function unit 22. The control function unit 22 then O(k-1)After this step, the process proceeds to step S7.

[0110] In step S7, the control function unit 22 (control information generation unit 220) calculates the current vector on the input device side (input side vector V Ik ) and the previous vector on the output device side (output side vector V O(k-1) ) and based on the velocity information (velocity vector) s k In this step, the second speed information s is generated using the calculation according to the above formula (2). k The processing in this step is to synchronize and align the information on the input device side with the information on the output device side (robot arm 7 side). The synchronization control is performed by the synchronization control function unit 221. After the processing in this step, the process proceeds to step S8.

[0111] Next, in step S8, the control function unit 22 (control information generation unit 220) generates the velocity information (velocity vector) s k In other words, the control function unit 22 (control information generation unit 220) controls the robot arm 7 using the velocity information f(s k ) to the robot arm 7, thereby controlling the robot arm 7. The robot arm 7 controls the robot arm 7 by transmitting this velocity information f(s k ) and receives the speed information f(s k ) and operates according to the

[0112] In step S9, the control device 2 determines whether or not to terminate the control. The determination result in this step is that the control will be terminated, for example, when a user (operator, etc.) issues an instruction to terminate the robot arm 7, when the operation period of the robot arm 7 ends according to a predetermined schedule, or when the occurrence of some kind of abnormality makes it impossible or difficult to continue operating the robot arm 7. If the control is to be terminated (step S9: YES), the entire processing of this flowchart is terminated. If the control is not to be terminated (step S9: NO), the process returns to step S2 to execute control of the next frame.

[0113] Next, a number of modified examples of the first embodiment will be described. Note that a number of modified examples may be combined to the extent that they are combinable.

[0114] [Variation 1] In the first embodiment, the control function unit 22 has a motion capture teaching function unit 225 (see FIG. 5). As a modified example, the control function unit 22 does not have to have the motion capture teaching function unit 225. In this case, it is not possible to capture motion or reproduce captured motion, but other functions of the control device 2 are realized.

[0115] [Variation 2] In the first embodiment, it was assumed that one glove 3 detects the bending angles of all the joints in one hand (left hand or right hand) of the user. As a modification, the glove 3 may not detect the bending angles of some joints. In this case, the amount of information that the glove 3 passes to the control device 2 is relatively reduced. Also, the control device 2 cannot perform control according to the bending angles of the joints that were not detected. However, the control device 2 maintains the other functions that it had in the first embodiment.

[0116] [Variation 3] In the first embodiment, the control function unit 22 has the noise correction function unit 222 (see FIG. 5). As a modified example, the control function unit 22 may not have the noise correction function unit 222. In this case, it is not possible to correct noise in the input information, but other functions of the control device 2 are realized.

[0117] [Second embodiment] Next, a second embodiment of the present invention will be described. Note that the following description may omit the matters already described in the previous embodiment. Here, the description will focus on matters unique to this embodiment.

[0118] FIGS. 11 and 12 are functional block diagrams showing the general functional configuration of a control device with a minimum configuration according to the second embodiment. As shown in FIG. 11, the control device 1002 of this embodiment includes an input device interface unit 21, a control function unit 1022, and an output device interface unit 23. The functions of the input device interface unit 21 and the output device interface unit 23 are similar to those described in the first embodiment. The control function unit 1022 does not have some of the functions of the control function unit 22 in the first embodiment. As shown in FIG. 12, the control function unit 1022 of this embodiment includes a control information generation unit 1220 and a synchronization control function unit 1221. The control function unit 1022 of this embodiment does not have the functions of the noise correction function unit 222 and the motion capture teaching function unit 225 that the control function unit 22 in the first embodiment has. In other words, the control function unit 1022 does not have the function of correcting noise contained in signals input from the input device. Furthermore, the control function unit 1022 does not store a time series of control signals representing motion, nor does it perform playback based on the stored time series of control signals.

[0119] In this embodiment, the control function unit 1022 also receives signals from the input device (glove or tracker) and generates and outputs control signals for controlling the movement of the robot arm corresponding to the hand based on the hand position, hand posture, and joint bending angles. Furthermore, the control function unit 1022 generates the control signals at predetermined timing based on robot arm status information passed from the output device interface unit 23 so that the status of the user's hand matches the status information of the robot arm.

[0120] [Realization by computer and program] FIG. 13 is a block diagram showing an example of the internal configuration for realizing the functions of each device in the first and second embodiments. At least some of the functions of each device (control device 2 or 1002, glove 3, tracker 4, and robot arm 7) can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to RAM 902, reads data from RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element in RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices, etc. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.

[0121] At least some of the functions of each device (control device 2 or 1002, glove 3, tracker 4, and robot arm 7) in the embodiments can be implemented by a computer and a program. In this case, the functions can be implemented by recording a program for implementing the functions on a computer-readable recording medium and loading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines when transmitting programs via networks such as the Internet or communication lines such as telephone lines, or media that store programs for a certain period of time, such as volatile memory within the computer systems that serve as the server or client in such cases. Furthermore, the above program may be one that realizes part of the functions described above, or may be one that can realize the functions described above in combination with a program already recorded in the computer system.

[0122] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0123] As described above, according to any of the embodiments, the control function unit (22, 1022) generates a control signal at a predetermined timing based on the status information (i.e., status information indicating the operating status of the robot arm (7)) passed from the output device interface unit (23) so that the status of the user's hand (i.e., the status of the input device (glove (3) or tracker (4))) matches the status information of the robot arm (7). This allows the control function unit (22, 1022) to control the robot arm (7) with high precision. In other words, it is possible to synchronize the status (position, posture, bending angle of joints, etc.) on the input device side and the output device side. This synchronization may be performed, for example, at regular time intervals or at other times.

[0124] In addition, the noise correction function unit (222) can correct noise that may occur on the input device (glove (3) or tracker (4) side). In other words, the control by the control function unit (22) is stable. In other words, noise that occurs due to disturbances at the input stage can be corrected.

[0125] The input device side interface unit (21) acquires signals representing information on the bending angles of all the joints in the user's (operator's) hands. The control function unit (22, 1022) generates and outputs control signals based on the information on the bending angles of all the joints. This allows the control function unit (22) to control the robot arm (7) faithfully in accordance with the bending angles of all the joints in the user's (operator's) hands.

[0126] According to this embodiment, even when a robot control system is realized by combining general-purpose items (such as gloves, trackers, and robot arms), it is possible to control a robot arm with high precision. In other words, an inexpensive and highly precise robot control system can be realized. In other words, it is possible to achieve highly accurate control that matches the intuition of the user (operator).

[0127] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0128] (Appendix 1) an input device interface unit that acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand; a control function unit that generates and outputs a control signal for controlling a movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angle of the joint; an output device side interface unit that transmits the control signal generated by the control function unit to the robot arm, and receives status information of the robot arm at a predetermined timing and passes the status information to the control function unit; Equipped with the state information of the robot arm is information representing a hand position, a hand posture, and bending angles of each finger joint on the robot arm side, which correspond to the hand position, the hand posture, and bending angles of the joints, the control function unit generates the control signal at the predetermined timing based on the state information passed from the output device side interface unit so that the state of the user's hand and the state information of the robot arm are consistent with each other. Control device.

[0129] (Appendix 2) The control function unit a noise correction function unit that, when detecting noise based on a temporal transition of at least any one of information regarding the position of the user's hand, information regarding the posture of the user's hand, and information regarding the bending angles of the joints of the fingers of the user's hand, corrects any one of the information regarding the position of the user's hand, information regarding the posture of the user's hand, and information regarding the bending angles of the joints of the fingers of the user's hand, which caused the detected noise, so as to eliminate or mitigate the detected noise; 2. The control device of claim 1, comprising:

[0130] (Appendix 3) the input device side interface unit acquires signals representing information on bending angles of all joints in the user's hand; The control function unit generates and outputs the control signal based on information on bending angles of all the joints. 10. The control device of claim 1 or 2.

[0131] (Appendix 4) a capturing function unit that stores a time series of the control signal generated by the control function unit, and, when requested, reads out the stored time series of the control signal and controls the output device side interface unit to transmit the time series of the control signal to the robot arm; 4. The control device according to claim 1, further comprising:

[0132] (Appendix 5) an input device interface unit acquiring signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand; a step in which a control function unit generates and outputs a control signal for controlling a movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angle of the joint; an output device side interface unit transmitting the control signal generated by the control function unit to the robot arm, and receiving status information of the robot arm at a predetermined timing and passing the information to the control function unit; Including, the state information of the robot arm is information representing a hand position, a hand posture, and bending angles of each finger joint on the robot arm side, which correspond to the hand position, the hand posture, and bending angles of the joints, the control function unit generates the control signal at the predetermined timing based on the state information passed from the output device side interface unit so that the state of the user's hand and the state information of the robot arm are consistent with each other. Control method.

[0133] (Appendix 6) an input device interface unit that acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand; a control function unit that generates and outputs a control signal for controlling a movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angle of the joint; an output device side interface unit that transmits the control signal generated by the control function unit to the robot arm, and receives status information of the robot arm at a predetermined timing and passes the status information to the control function unit; Equipped with the state information of the robot arm is information representing a hand position, a hand posture, and bending angles of each finger joint on the robot arm side, which correspond to the hand position, the hand posture, and bending angles of the joints, the control function unit generates the control signal at the predetermined timing based on the state information passed from the output device side interface unit so that the state of the user's hand and the state information of the robot arm are consistent with each other. A program that makes a computer function as a control device.

[0134] (Appendix 7) A control device according to any one of Supplementary Note 1 to Supplementary Note 4; a tracker that transmits signals representing information on the position of a user's hand and information on the posture of the user's hand to the input device side interface unit of the control device; a glove that transmits a signal representing information on the bending angle of each finger joint of the user's hand to the input device side interface unit of the control device; A robot control system comprising:

[0135] (Appendix 8) A control device according to any one of Supplementary Note 1 to Supplementary Note 4; a robot arm that operates based on the control signal transmitted from the output device-side interface unit of the control device and transmits the status information of its own device to the output device-side interface unit at the predetermined timing; A robot control system comprising:

[0136] (Appendix 9) A control device according to any one of Supplementary Note 1 to Supplementary Note 4; a tracker that transmits signals representing information on the position of a user's hand and information on the posture of the user's hand to the input device side interface unit of the control device; a glove that transmits a signal representing information on the bending angle of each finger joint of the user's hand to the input device side interface unit of the control device; a robot arm that operates based on the control signal transmitted from the output device-side interface unit of the control device and transmits the status information of its own device to the output device-side interface unit at the predetermined timing; A robot control system comprising:

[0137] (Appendix 10) The information on the position of the user's hand is expressed as three-dimensional coordinate values ​​for one hand, The user's hand posture information is expressed as a three-dimensional rotation angle for one hand. 5. The control device according to any one of claims 1 to 4.

[0138] (Appendix 11) The control signal generated by the control function unit is a signal representing information on the speed of a portion of the robot arm. 11. The control device according to any one of claims 1 to 4 and claim 10.

[0139] (Appendix 12) the control function unit generates and outputs the control signal for each frame at a fixed time interval; The predetermined timing is a timing for every fixed number of frames. 12. The control device according to any one of Supplementary Notes 1 to 4 and Supplementary Notes 10 to 11.

[0140] (Appendix 12) The frame frequency is 45 frames per second (45 fps) or higher. 13. The control device of claim 12. [Industrial Applicability]

[0141] The present invention can be used in industries that use robots, for example. For example, the present invention can be used in controlling robots in the manufacturing industry, remote surgery (or businesses that provide equipment for such surgery), online tours, etc. However, the applicability is not limited to these industries. Furthermore, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0142] 1. Robot Control System 2. Control device 3 Gloves 4. Tracker 7. Robotic Arm 9. Communication Lines 21 Input device side interface section 22 Control Function Unit 23 Output device interface section 220 Control information generation unit 221 Synchronous control function unit 222 Noise correction function section 225 Motion capture teaching function unit (capturing function unit) 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus 1002 control device 1022 Control Function Unit 1220 Control information generation unit 1221 Synchronous control function unit

Claims

[Claim 1] an input device interface unit that acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand; a control function unit that generates and outputs a control signal for controlling a movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angle of the joint; an output device side interface unit that transmits the control signal generated by the control function unit to the robot arm, and receives status information of the robot arm at a predetermined timing and passes the status information to the control function unit; Equipped with the state information of the robot arm is information representing a hand position, a hand posture, and bending angles of each finger joint on the robot arm side, which correspond to the hand position, the hand posture, and bending angles of the joints, the control function unit generates the control signal at the predetermined timing based on the state information passed from the output device side interface unit so that the state of the user's hand and the state information of the robot arm are consistent with each other. a control device; a tracker that transmits signals representing information on the position of a user's hand and information on the posture of the user's hand to the input device side interface unit of the control device; a glove that transmits a signal representing information on the bending angle of each finger joint of the user's hand to the input device side interface unit of the control device; A robot control system comprising: The tracker is attached to the glove. Robot control system.

Images