Robot teaching method and device

Abstract

Provided is a technology which can implement the facilitation or the like of the reduction or introduction of previously adjusted man-hours for teaching by enabling even a person who does not have any knowledge of a robot to intuitively and efficiently teach the robot working motions by demonstrating work that maintains as much as possible a typical motion of the person. This robot teaching method performs teaching for generating a motion of a hand mechanism of a robot corresponding to a working motion on the basis of measuring the working motion including an operation for an object to be operated by the hand of a teacher. The method comprises: as steps executed by a computer system, a step for acquiring a first measurement pose which is a measured pose of the object to be operated and a second measurement pose which is a measured pose of the hand of the teacher at the time of the working motion; a step for detecting an operation of the object to be operated by the teacher; and a step for generating a teaching pose for generating robot motion data on the basis of the first measurement pose, the second measurement pose, and the detected operation.

Description

[Technical Field]

[0001] The present invention relates to a technique for teaching a robot a human work movement. [Background technology]

[0002] An example of human work, or in other words, work motions, is the actions of a worker in a factory, such as grasping a tool with his or her hand, moving the tool, and then releasing it. Work motions consist of one or more operations, such as grasping and releasing. Operations consist of more detailed actions and movements, such as moving the arm, moving the wrist, and moving the fingers. To realize robot movements that correspond to such human work motions, there is a technology that teaches robots human work motions by demonstrating them.

[0003] For the sake of explanation, the main part that performs the target task / operation / motion may be referred to as a human "hand" or a robot "hand." This term refers to a hand in a broad sense, and is used as a general term that includes the shoulder, arm, wrist, palm, fingers, etc., of the human body and robot, including joints. When we say that a hand moves, we actually mean that the arm also moves in conjunction with the hand in the narrow sense. Furthermore, when distinguishing a "robot hand" from a "human hand," it may be referred to as a hand portion, hand mechanism, gripping portion, gripping mechanism, etc.

[0004] Examples of conventional technology include the following. Examples of objects to be manipulated include tools and components that can be grasped. The hand at the end of the robot's arm is equipped with a mechanism that enables manipulation such as grasping. The system detects the human hand grasping the object using a camera or other device, determines the position and posture of the object relative to the position and posture of the hand, and associates these with the position and posture of the robot's hand. As a result, the system generates data on the movements of the robot's hand and other devices that correspond to the human's work movements.

[0005] An example of prior art is Japanese Patent Application Laid-Open No. 2021-167060 (Patent Document 1). Patent Document 1 describes, as "robot teaching by human demonstration," that "a method is provided for teaching a robot to perform an operation based on human demonstration using images from a camera," and that "the method includes a teaching step in which a 2D or 3D camera detects a human hand gripping and moving a workpiece, and analyzes images of the hand and workpiece to determine the posture and position of the robot gripper equivalent to the posture and position of the hand, and the corresponding position and posture of the workpiece, and then generates robot programming commands from the calculated posture and position of the gripper relative to the posture and position of the workpiece." [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-167060 Summary of the Invention [Problem to be solved by the invention]

[0007] When replacing human tasks with robotic movements, it is necessary to teach the robot the movements of the human task, replacing them with movements that can be executed by a robot with a different physique, joint structure, etc. This type of work teaching requires a lot of effort, time, and cost to implement and implement.

[0008] In order to improve the efficiency of work instruction as described above, there is a method, as disclosed in Patent Document 1, in which a human work movement is detected using a camera or the like, and the human work movement is converted into a robot movement to reproduce the work movement.

[0009] However, except in special cases, the joints and movement structures of a human hand and the like often differ from those of a robot's hand. Therefore, due to these differences, when a human demonstrates a task for teaching purposes, the human performer must first consider the structure and limitations of the robot's hand and then perform the task so as to approximate, for example, imitate, the task as if it were a robot's hand. In the example of Patent Document 1, depending on the shape of the robot's gripper and the shape of the object to be grasped, the human must change the hand posture that the human normally uses when performing the task to a posture that is easily reproduced by the robot's gripper. However, in this case, the demonstrator of the task is required to have knowledge of the robot's mechanism, which places a heavy burden on the human performer in demonstrating the task and makes efficient teaching difficult.

[0010] Furthermore, robot teaching methods such as those in the prior art require analyzing camera images to determine the position and orientation of each part, which can require advanced image analysis processing and computational resources, making it difficult to determine the correspondence between the human hand and the robot's hand. In the example of Patent Document 1, to ensure the accuracy of the work, it is necessary to recognize the orientation of the human hand with high precision and associate the recognition result with the robot's gripping part. This prior adjustment for recognition and association requires a lot of man-hours.

[0011] The purpose of the present disclosure is to provide a technology related to robot teaching that enables even a person with no knowledge of robots to intuitively and efficiently teach a robot work movements by demonstrating the work while maintaining normal human movements as much as possible, thereby reducing the labor required for pre-adjustment of the teaching and facilitating implementation. [Means for solving the problem]

[0012] A representative embodiment of the present disclosure has the following configuration: A robot teaching method according to the embodiment is a robot teaching method for teaching, based on measurement of a task motion including manipulation of a manipulated object by a hand of a teacher, to generate robot motion data including a joint displacement sequence as a motion of the robot's hand mechanism corresponding to the task motion, and includes, as steps executed by a computer system, a step of acquiring a first measured pose by measuring a time-series pose consisting of a position and posture of the manipulated object during the task motion, a step of acquiring a second measured pose by measuring a time-series pose consisting of a position and posture of the teacher's hand during the task motion, a step of detecting the manipulation of the manipulated object by the teacher, and a step of generating a teaching pose for generating the robot motion data based on the first measured pose, the second measured pose, and the detected manipulation. [Effects of the Invention]

[0013] According to a representative embodiment of the present disclosure, with regard to technology related to robot teaching, even a person with no knowledge of robots can intuitively and efficiently teach a robot work movements by demonstrating a work that maintains as much as possible the normal human movements, thereby realizing a reduction in the amount of pre-adjustment work required for teaching, facilitating implementation, etc. Problems, configurations, effects, etc. other than those described above are described in the description for carrying out the invention. [Brief explanation of the drawings]

[0014] [Figure 1] 1 shows an outline of the overall configuration of a robot teaching system that is a robot teaching device according to a first embodiment. [Figure 2] 2 shows an example of a functional block configuration of a control device in the robot teaching device according to the first embodiment. [Figure 3] 2 shows an example of the configuration of a computer system that constitutes a control device of the robot teaching device of the first embodiment. [Figure 4] In the first embodiment, an example of the arrangement of cameras and the like relative to a workbench will be shown. [Figure 5]In the first embodiment, a configuration example of a test tube, which is an object to be grasped, and a marker plate to be placed on the test tube will be shown. [Figure 6] In the first embodiment, a configuration example of a pipette, which is an object to be grasped, and a marker plate to be installed on the pipette will be shown. [Figure 7A] In the first embodiment, a configuration example of a marker plate attached to the right hand of an instructor is shown. [Figure 7B] In the first embodiment, an example of the configuration of a marker plate attached to the left hand of an instructor is shown. [Figure 8] FIG. 2 is a perspective view showing a state in which a hand portion of a hand mechanism of a robot grips a test tube in the first embodiment. [Figure 9A] FIG. 2 shows a side view of the first embodiment in which the hand unit is gripping a test tube. [Figure 9B] FIG. 2 shows a top view of the first embodiment in which the hand unit is gripping a test tube. [Figure 10] 1 is an explanatory diagram showing the relationship between the coordinate system of the marker on the test tube and the coordinate system of the hand unit in a top view in which the hand unit is gripping the test tube in the first embodiment. FIG. [Figure 11] FIG. 10 is an explanatory diagram showing the relationship between the coordinate system of the marker in the left hand and the coordinate system of the hand unit in a top view of a state in which the instructor's hand is holding a test tube in the first embodiment. [Figure 12] In the first embodiment, a left-hand gripping operation and a right-hand gripping operation are shown as examples of the instructor's work movements. [Figure 13] In the first embodiment, as an example of the instructor's working motion, a state in which the pipette in the right hand approaches the test tube in the left hand is shown. [Figure 14] 2 shows a processing flow in the robot teaching method and device according to the first embodiment. [Figure 15A] In the first embodiment, graphs of the first measurement pose, the second measurement pose, etc. are shown. [Figure 15B] In the first embodiment, graphs of poses before and after correction related to errors are shown. [Figure 15C] 1 shows a graph of the teaching pose generated in the first embodiment. [Figure 16A] 13 shows a graph of pose correction in Modification 1A of Embodiment 1. [Figure 16B] 13 shows a graph before pose correction in Modification 1B of Embodiment 1. [Figure 16C] 13 shows a graph after pose correction in Modification 1B of Embodiment 1. [Figure 17A] 13 shows a graph of pose correction in Modification 1C of Embodiment 1. [Figure 17B] 13 shows a graph of pose correction in Modification 1D of Embodiment 1. [Figure 18] In the first embodiment, a display example of a screen including a GUI will be shown. [Figure 19] 10 shows an example of a state in which a pipette is placed in a holder on a workbench in a robot teaching method and device according to a second embodiment. [Figure 20] In the second embodiment, an example of the setting contents of the correction area corresponding to the restriction when the hand part of the hand mechanism of the robot grips the pipette in the holder will be shown. [Figure 21A] In the second embodiment, graphs of the first measurement pose, the second measurement pose, etc. are shown. [Figure 21B] In the second embodiment, a graph regarding correction related to operation restrictions is shown. [Figure 21C] In the second embodiment, a graph of the generated teaching pose is shown. [Figure 22A] 13 shows a graph of pose correction in Modification 2A of Embodiment 2. [Figure 22B] 13 shows a graph of pose correction in Modification 2B of Embodiment 2. [Figure 22C] 13 shows a graph of pose correction in Modification 2C of Embodiment 2. [Figure 23] In the second embodiment, a display example of a screen including a GUI will be shown. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, identical parts are generally designated by the same reference numerals, and repeated explanations will be omitted. In the drawings, the representation of components may not represent their actual positions, sizes, shapes, scopes, etc., in order to facilitate understanding. When there are multiple similar components and each component is described separately, an alphabet or the like may be added to the end of the reference numeral indicating a generic concept.

[0016] For the sake of explanation, when describing processing by a program, the program, function, processing unit, etc. may be described as the main body, but the main hardware body for these is a processor, or a controller, device, computer, system, etc. that is configured with the processor, etc. A computer executes processing according to a program read into memory using resources such as memory and communication interfaces as appropriate through the processor. This realizes predetermined functions, processing units, etc. A processor is configured, for example, with semiconductor devices such as a CPU / MPU or GPU. Processing is not limited to software program processing, but can also be implemented using dedicated circuits. Dedicated circuits such as FPGAs, ASICs, and CPLDs can be used.

[0017] The program may be pre-installed as data on the target computer, or may be distributed as data from a program source to the target computer. The program source may be a program distribution server on a communication network, or a non-transitory computer-readable storage medium such as a memory card or disk. The program may be composed of multiple modules. The computer system may be composed of multiple devices. The computer system may be composed of a client-server system, a cloud computing system, an IoT system, etc. Various data and information may be composed of structures such as tables and lists, for example, but are not limited to these. Expressions such as identification information, identifiers, IDs, names, and numbers are interchangeable.

[0018] <Solutions, etc.> A robot teaching method and device according to an embodiment have the following configuration. The robot teaching method according to the embodiment is a method for teaching a robot equipped with a mechanism such as a hand mechanism capable of manipulating a manipulated object, a task movement including manipulation of a manipulated object by a human hand or the like. This method uses a camera, sensor, or measurement device to measure or detect a pose consisting of the time-series position and posture of an object. The object may be a human hand, a tool or component that is a manipulated object, or a marker attached thereto. The position may be, for example, a position coordinate in a three-dimensional coordinate system space, and the posture may be, for example, the orientation of each axis of the three-dimensional coordinate system. The pose is a trajectory connecting the position and posture at each point in time on the time series.

[0019] The robot teaching method of the embodiment includes steps of measuring a first pose, which is the pose of a manipulated object such as a tool, and a second pose, which is the pose of the instructor's hand, as poses to be measured. The first pose may also be referred to as a tool-side pose. The second pose may also be referred to as a hand-side pose. These steps can be more specifically divided into a step of measuring the pose of a first marker attached to the manipulated object to obtain the first measured pose, and a step of measuring the pose of a second marker attached to the hand to obtain the second measured pose.

[0020] The robot teaching method of the embodiment includes a step of generating a teaching pose, in other words, teaching data, which is a pose for teaching a work movement to generate a movement of the robot, based on the first pose and the second pose.

[0021] The robot teaching method of the embodiment includes a step of starting the measurement or generation process based on a teaching instruction such as starting a work demonstration by a manager or instructor, and a step of ending the measurement or generation process based on a teaching instruction such as completing the work demonstration by a manager or instructor.

[0022] The robot teaching method of the embodiment includes a step of detecting an operation performed by an instructor on an object to be manipulated, more specifically, a step of detecting an operation instruction from the instructor. After starting a task demonstration, the instructor may input an operation instruction as appropriate during the task. This operation instruction is input by the instructor to inform the system that "an operation such as grasping or releasing an object to be manipulated is currently being performed or has been performed." The operation instruction may be input using, for example, an instruction input device. Alternatively, the robot teaching device of the embodiment may detect the operation from a camera image or the like.

[0023] The step of generating the teaching data in the robot teaching method of the embodiment is a step of generating a teaching pose based on the first pose and the second pose in response to the detection of the operation.

[0024] The robot teaching method according to the embodiment includes a step of generating a robot motion based on the generated teaching pose, more specifically, generating robot motion data including a sequence of joint displacements for causing the robot to perform the motion.

[0025] Furthermore, the robot teaching method of the embodiment includes a step of determining an error between a first measured pose of a first marker on the tool side and a second measured pose of a second marker on the hand side when the instructor operates the object to be manipulated, in the step of generating the teaching data, and a step of generating the teaching data by correcting the first measured data and the second measured data using the error.

[0026] A robot teaching method according to another embodiment includes a step of generating an operation pose as a pose that enables operation of the object to be manipulated so as to satisfy the constraints, based on the first pose and the second pose, and correction data that takes into account the structure of the robot's hand mechanism, the structure of the object to be manipulated such as a tool, and constraints corresponding thereto. This method also includes a step of generating teaching data by using the operation pose to replace or correct a corresponding part of the teaching pose.

[0027] The structure of the robot's hand mechanism, etc., refers to a mechanism attached to the tip of an arm to enable manipulation of an object to be manipulated, such as a gripping mechanism as an end effector, and varies depending on the implementation. The structure of the shape of the object to be manipulated, such as a tool, refers to a structure of a pipette or test tube, and varies depending on the implementation. The limitations, for example, refer to limitations on the allowable direction, position, distance, speed, force, etc., when the hand mechanism accesses or moves to manipulate a part, such as a pipette, for example.

[0028] <First Embodiment> 1 to 18, a robot teaching method and device according to a first embodiment of the present disclosure will be described. The robot teaching method according to the first embodiment is a method having steps performed by the robot teaching device according to the first embodiment. The robot teaching device according to the first embodiment is a system including at least a computer.

[0029] [Robot teaching system] FIG. 1 shows an overview of the overall configuration of a robot teaching system 1, which is a robot teaching device 1 according to the first embodiment. This robot teaching system 1 is a device or system for teaching a robot 2 the work movements of a human worker U1. The worker U1 may also be referred to as a first user or a teacher. Such teaching may also be referred to as work teaching. The robot teaching system 1 includes a robot teaching control device 100. The robot teaching control device 100 may also be simply referred to as a control device.

[0030] The work motions are motions that constitute a predetermined work to be taught. The predetermined work to be taught is a work that is performed by the worker U1 using at least one hand 5 to manipulate a predetermined tool or component with the hand 5. Examples of work motions and manipulations include the motion of grasping a tool such as a test tube 7 with the left hand 5L or the right hand 5R as one hand 5, and the motion of releasing the tool. Examples of objects to be manipulated, particularly tools to be grasped, are the test tube 7 and the pipette 8. Note that when teaching, a model may be used as the tool such as the test tube 7 instead of the actual object.

[0031] In the method and apparatus of the first embodiment, an example will be described in which an instructor, an operator U1, teaches a robot 2 to operate a tool, which is an object to be operated. As an example, an example will be mainly described in which an operator U1 teaches a task of grasping a test tube 7 with the left hand 5L on a workbench 10. Note that FIG. 1 not only illustrates this example, but also an example in which a pipette 8 is grasped with the right hand 5R. The test tube 7 is a tube that contains a substance. The pipette 8 is a micropipette that can aspirate a substance from the test tube 7 and dispense a substance into the test tube 7.

[0032] As shown in FIG. 4 (described later), a coordinate system Σw, which is a work table coordinate system, is set on the work table 10. The poses of each part, such as the tools and the hand 5, are expressed as positions and orientations in this coordinate system Σw. The surface of the work table 10 also has, for example, a holder 81 for holding a pipette 8, in other words, a pipette stand, and a holder 71 for holding a test tube 7, in other words, a test tube stand. Note that the left hand 5L and the right hand 5R as the hand 5 of the operator U1 are generic terms that include the forearm to the fingertips unless otherwise specified.

[0033] The robot teaching system 1 includes a control device 100, cameras 20 {20a, 20b, 20c, 20d} installed on a workbench 10, markers 3 (3a, 3b) installed on test tubes 7 (7a, 7b), a marker 3 (3p) installed on a pipette 8, and markers 4 (4L, 4R) attached to the hands 5 (5L, 5R) of an instructor U1.

[0034] The control device 100 is externally connected to an input / output device 120. The control device 100 may also have an input / output device 120 built in. The input / output device 120 is a device that inputs settings and instructions related to the arithmetic processing of the control device 100 and outputs data and information such as the status and results related to the arithmetic processing.

[0035] The cameras 20 (20a to 20d) are devices such as cameras that constitute the motion capture system 200 of FIG. 2. Note that the cameras 20 may be replaced with sensors, measuring devices, or the like. The motion capture system 200 includes at least the cameras 20, but the details are not limited, and other components such as a processor and circuits may also be included. For example, the motion capture system 200 may store and output data and information detected as a result of a processor processing images from the cameras 20.

[0036] In this example, the markers 3, which are the first markers on the tool side, are markers 3a, 3b, and 3p. Markers 3a and 3b are marker plates attached to a test tube 7 (7a, 7b), which is the first tool and an object to be manipulated by the instructor U1 and the robot 2. Similarly, marker 3p is a marker plate attached to a pipette 8, which is the second tool and an object to be manipulated by the instructor U1 and the robot 2. The tool-side markers 3 are attached in positions that do not interfere with operations such as grasping by the hand 5 or the hand mechanism 6.

[0037] In this example, the instructor U1 has markers 4L and 4R as the second markers 4 on the hand 5 side. The markers 4 (4L, 4R) are marker plates attached to the left hand 5L and the right hand 5R as the hands 5 of the instructor U1. The marker 4 on the hand 5 side is placed, for example, on the forearm near the wrist so as not to interfere with movements using the wrist. When distinguishing between the tool side and the hand side, the tool side may be described as the first and the hand side as the second.

[0038] The robot 2 is equipped with hand mechanisms 6 (6L, 6R) as a dual-arm mechanism capable of performing operations such as grasping an object and positioning it at any position and posture within a movable range. The robot 2 also has a stereo camera 30 in the area of ​​its head where the eyes would be. The robot 2 has a joint axis in its neck, which allows the stereo camera 30 to be pointed in any direction of the workbench 10. The robot 2 can detect and recognize objects in its field of view based on the photographing and distance measurement performed by the stereo camera 30. The robot 2 is installed, for example, on an unmanned guided vehicle (not shown), and can move around the room using a pre-created map of the room to autonomously move in front of the workbench 10.

[0039] The robot 2 has at least mechanisms associated with parts of a human hand, such as the forearm, elbow, wrist, palm, and fingers, as well as joint mechanisms, and these mechanisms are collectively referred to as hand mechanisms 6. The hand mechanisms 6 (6L, 6R) have end effectors 9 (9L, 9R) at their tips.

[0040] The robot 2 is controlled by a robot teaching control device 100. The robot teaching control device 100 controls the operation of the robot 2 based on the robot operation data generated by the teaching function 100F. However, this is not limiting, and the robot 2 may be provided with a dedicated control device separate from the robot teaching control device 100. In this case, the dedicated control device controls the operation of the robot 2 based on the robot operation data from the robot teaching control device 100.

[0041] In the example of the first embodiment, an example of teaching a single robot 2 equipped with two arms to perform a task using one of a person's hands will be mainly described. However, the teaching in the embodiment is similarly applicable to multiple dual-arm robots or one or multiple single-arm robots. When multiple single-arm or dual-arm robots are used, the robots may be different types of robots, for example, robots with different mechanisms. Furthermore, the teaching in the embodiment is not limited to single-handed task movements using the left hand 5L or the right hand 5R, but is similarly applicable to task movements using both hands. Furthermore, the teaching in the embodiment is not limited to task movements using a test tube 7 or a pipette 8 as an example of a tool and including grasping as an example of an operation, but is similarly applicable to other operations using other tools.

[0042] The robot teaching control device 100 has a teaching function 100F as its main function. The teaching function 100F is realized by processing by a processor, etc. As shown in the figure, the teaching function 100F generally includes functions for measuring poses related to work movements, generating teaching data related to work movements, and generating robot movement data.

[0043] [Robot teaching control device] Fig. 2 shows an example of a functional block configuration of the robot teaching control device 100 in Fig. 1. The control device 100 has, as input / output components, a motion capture system 200, an instruction input device 300, and an input / output device 120. The control device 100 can be implemented using a computer such as a general PC or server, for example, but is not limited to this.

[0044] The control device 100 has, as internal functional blocks, a pose measurement unit 101, a first measurement pose storage unit 102, a second measurement pose storage unit 103, an operation instruction detection unit 104, a teaching instruction detection unit 105, a correction data storage unit 106, a teaching data generation unit 107, a teaching data storage unit 108, a robot motion generation unit 109, a robot motion data storage unit 110, a robot motion execution unit 111, and an operation pose generation unit 112. Note that the operation pose generation unit 112 is not used in the first embodiment, but is used in the second embodiment described below.

[0045] Based on the above-mentioned markers 3 and 4, the pose measurement unit 101 measures the positions and postures in three-dimensional space of the markers on the person's hand 5 and the tool side and their corresponding objects as time-series poses from images captured by the cameras 20 (20a to 20d) of the motion capture system 200. The pose measurement unit 101 measures the poses of the markers 3a, 3b, and 3p of the marker 3, which is the first marker on the tool side, as the first measurement pose. In addition, the pose measurement unit 101 measures the poses of the markers 4L and 4R of the marker 4, which is the second marker on the hand 5 of the worker U1, as the second measurement pose.

[0046] The first measurement pose storage unit 102 stores data of the first measurement pose based on a command from the pose measurement unit 101. The second measurement pose storage unit 103 stores data of the second measurement pose based on a command from the pose measurement unit 101. Note that the various data storage units, in other words, storage units, are not limited to using memories within the control device 100, and may be realized by using external storage devices for the control device 100.

[0047] The operation instruction detection unit 104 detects a predetermined operation by the instructor U1, in other words, an operation instruction, in synchronization with the measurement by the pose measurement unit 101 during the teaching work by the instructor U1. This operation instruction is input proactively and consciously by the instructor U1. This predetermined operation is, for example, an operation to grip or release a test tube 7 or a pipette 8 as a tool to be gripped, and the corresponding operation instruction is a grip operation instruction that conveys the grip operation or a release operation instruction that conveys the release operation.

[0048] The operation instruction detection unit 104 detects the input of an operation instruction by the instructor U1 using, for example, an instruction input device 300. Although not shown in FIG. 1 , the instruction input device 300 may be any of various input devices, such as a pressure-sensitive switch attached to the fingers of the instructor U1, a foot switch operated by the instructor U1 at his / her feet, a microphone that inputs the instructor U1's voice, or a voice recognition device. The instruction input device 300 may be any other input means that does not interfere with the work operation of the instructor U1.

[0049] Furthermore, the operation instruction detection unit 104 is not limited to input / detection using the instruction input device 300, and may automatically determine / detect a predetermined operation based on an input image or measurement results by the pose measurement unit 101. For example, the operation instruction detection unit 104 may detect a grasping operation by determining a change in the first measurement pose of the grasped object measured by the pose measurement unit 101 or a change in the second measurement pose of the hand 5, or a state in which the first measurement pose or the second measurement pose does not change for a certain period of time or more.

[0050] In the example of embodiment 1, the operations that are the subject of instruction and operation instruction detection are mainly described as gripping operations, but this is not limited to this and can be similarly applied to other operations, such as pushing operations, pulling operations, and turning operations.

[0051] During the teaching work by the instructor U1, the teaching instruction detection unit 105 detects various teaching instructions, such as an instruction to start or complete the teaching work, an instruction to register the contents of the teaching work, and an instruction to check the work status, in synchronization with the measurement by the pose measurement unit 101. Teaching instructions are processing instructions for the control device 100, such as a start instruction, a completion instruction, a pause instruction, a setting instruction, and a display instruction. These teaching instructions may be input by the instructor, the worker U1, or by a separate administrator U2. The worker U1 may input the teaching instructions using the instruction input device 300. The worker U1 or the administrator U2 may input the teaching instructions using the input / output device 120. In one example of the teaching work, the administrator U2 inputs a start instruction, and after the teaching work starts, the worker U1 performs the teaching work while inputting operation instructions as appropriate, and finally the administrator U2 inputs a completion instruction to end the teaching work.

[0052] Furthermore, the teaching instruction detection unit 105 is not limited to detecting the input of teaching instructions by the worker U1 or the manager U2, but may automatically determine and detect teaching instructions based on the input image and measurement results in the pose measurement unit 101.

[0053] The teaching data generating unit 107 generates a teaching pose, which is a pose for generating a robot movement, based on both the first measurement pose and the second measurement pose. The teaching data generating unit 107 generates a teaching pose based on the first measurement pose, the second measurement pose, the operation instruction, the teaching instruction, and the correction data, and stores the teaching pose in the teaching pose storage unit 108.

[0054] The correction data storage unit 106 stores correction data, which is data for correcting and calculating the position and posture of a measurement pose or a teaching pose. The control device 100 sets the correction data in advance and stores it in the correction data storage unit 106. In other words, the correction data is correction setting information. Examples of the correction data in the first embodiment include data for coordinate transformation, which will be described later, and data for error correction.

[0055] The teaching data generating unit 107 starts the teaching process including the generation of the teaching pose in accordance with the start instruction as the teaching instruction. The teaching data generating unit 107 ends the teaching process including the generation of the teaching pose in accordance with the completion instruction as the teaching instruction.

[0056] The teaching data generation unit 107 acquires the first measurement pose stored in the first measurement pose storage unit 102 and the second measurement pose stored in the second measurement pose storage unit 103, and generates a teaching pose by selecting one or both of the first measurement pose and the second measurement pose, or by processing the data, depending on the timing and content of the operation instruction detected by the operation instruction detection unit 104, for example, a gripping operation instruction. Furthermore, at that time, the teaching data generation unit 107 acquires correction data stored in the correction data storage unit 106, and generates a corrected teaching pose by performing a correction calculation on the position and posture of the generated teaching pose based on the correction data.

[0057] The teaching data generation unit 107 generates teaching data in a format in which, for example, information on operation instructions detected by the operation instruction detection unit 104 and information on teaching instructions detected by the teaching instruction detection unit 105 are associated in a time series so that the timing is synchronized, for example, for teaching pose data based on the measurement pose.

[0058] The teaching data storage unit 108 stores the generated teaching data based on a command from the teaching data generation unit 107. The teaching data storage unit 108 stores, as teaching data, data including the teaching pose generated by the teaching data generation unit 107 and operation instructions and teaching instructions synchronized with the teaching pose.

[0059] The robot motion generation unit 109 generates robot motion data for operating the hand mechanism 6 and other components of the robot 2 in accordance with the content of the operation, in synchronization with the timing of the operation instruction detected by the operation instruction detection unit 104, based on the teaching data stored in the teaching data storage unit 108. This robot motion data is data including a joint displacement sequence of the hand mechanism 6 and other components of the robot 2, and motion commands for operating the hand mechanism 6 and other components synchronized with the joint displacement sequence. Examples of the motion of the hand mechanism 6 include opening and closing the end effector 9 for grasping. The robot motion data is configured, for example, as motion commands in a predetermined format given to the robot 2, in other words, data such as operation commands and commands.

[0060] In this example, the robot movement data can be generated by conversion from the teaching data, but this is not limited to this. For example, the teaching data generation unit 107 and the robot movement generation unit 109 may be combined into one unit, and robot movement data may be generated directly as teaching data from measurement poses, etc.

[0061] The robot operation data storage unit 110 stores the robot operation data generated by the robot operation generation unit 109. For example, in response to an instruction from an administrator U2, the robot operation execution unit 111 drives the robot 2 in accordance with the contents of the robot operation data stored in the robot operation data storage unit 110, thereby causing the robot 2 to execute an operation corresponding to the taught work operation.

[0062] [Computer System] FIG. 3 shows an example of the configuration of a computer system as an implementation example of the robot teaching control device 100 of FIG. 2. The computer system of FIG. 3 constituting the control device 100 has at least a computer 1000. The computer 1000 includes a processor 1001, a memory 1002, a communication interface device 1003, an input / output interface device 1004, and the like. These components are connected to a bus and can communicate with each other. An input device 1005 and an output device 1006 corresponding to the above-mentioned input / output device 120 are externally connected to the computer 1000 via the input / output interface device 1004. Note that the input device 1005 and the output device 1006 may be built into the computer 1000.

[0063] The processor 1001 is configured with a semiconductor device such as a CPU, MPU, or GPU. The processor 1001 includes a ROM, a RAM, and various peripheral functions. The processor 1001 executes processing in accordance with a control program 1011 stored in a memory 1002. This realizes functions such as a teaching function 100F. The teaching function 100F, as outlined in FIG. 1, is a function of teaching the robot 2 work operations based on the detection and measurement of the work demonstration by the instructor U1, in other words, a function of generating robot operation data corresponding to the work instruction, and a function of managing and controlling the same.

[0064] The memory 1002 stores a control program 1011, setting information 1012, image data 1013, processing data 1014, and various types of data described below. The control program 1011 is a program for causing the processor 1001 to execute processing to realize each function. The setting information 1012 is system setting information and user setting information for each function. The image data 1013 is data such as an image acquired from the camera 20 in FIG. 1. The processing data 1014 is data generated during the processing of each function.

[0065] The communication interface device 1003 or the input / output interface device 1004 is a device that implements communication interfaces with various parts, including the motion capture system 200 including the camera 20 in Fig. 2, the instruction input device 300, and, if connected to a communication network such as a LAN, the communication network. An input device 1005 and an output device 1006 corresponding to the input / output device 120 are externally connected to the input / output interface device 1004. Examples of the input device 1005 include a keyboard and a mouse. Examples of the output device 1006 include a display and a printer.

[0066] Furthermore, an external storage device, such as a memory card or a disk, may be connected to the computer 1000 via a communication interface, or a server device or the like may be connected as an external device via a communication network such as a LAN. The computer 1000 may read and write data and information from the external storage device, the server device, or the like as appropriate.

[0067] A user uses the control device 100 through input operations on the input device 1005, a screen display on the output device 1006, or the like. The user may be the same person as the instructor U1 in FIG. 1, or may be a different person from the instructor U1, such as an administrator U2 of the robot teaching system 1. A user such as the administrator U2 may access the control device 1000 via communication from another client terminal and use the control device 1000. That is, the functions of the control device 1000 may be used in a client-server manner. Furthermore, processing related to the functions of the control device 1000 may be performed online in near real time, or offline, in which necessary data and information are first acquired and stored and then performed later.

[0068] When using a client-server system, it can be realized, for example, as follows: A user accesses the server function of the control device 1000 from a client terminal. The server function of the control device 1000 sends data such as a web page including a graphical user interface (GUI) to the client terminal. The client terminal displays the web page on a display based on the received data. The user views the web page, checks information related to the function, and inputs settings and instructions as necessary. The client terminal sends the information entered by the user to the control device 1000. The control device 1000 executes processing related to the function based on the information entered by the user and saves the results. The control device 1000 sends data such as a web page including processing results to the client terminal. The client terminal displays a web page including processing results on a display. The user views and confirms the processing results.

[0069] [Workbench and Camera] FIG. 4 shows the relative positions of the workbench 10 and the cameras 20, and is a schematic diagram of the horizontal plane, which is the top surface of the workbench 10, viewed vertically from above. Four cameras 20a, 20b, 20c, and 20d are installed as shown in the figure, facing upward at different positions on the workbench 10. Fields of view 21a to 21d indicate the field of view of each of the cameras 20a to 20d. The workbench coordinate system Σw shown in the figure is a three-dimensional spatial coordinate system based on the surface of the workbench 10, with the two orthogonal axes constituting the horizontal plane being the X-axis and the Y-axis, and the vertical direction perpendicular to these axes being the Z-axis. The origin of the coordinate system Σw is set at a predetermined position on the top surface of the workbench 10 as shown in FIG. 4, but is not limited thereto.

[0070] Cameras 20 (20a to 20d) capture images of an area including markers 3 (3a, 3b, 3p), which are first markers, placed on test tubes 7 or pipettes 8 as tools placed on the top surface of workbench 10 or tools held in the hands of operator U1, and also capture images of an area including markers 4 (4L, 4R), which are second markers, placed on the hand 5 side of operator U1. Based on the images captured by cameras 20, pose measurement unit 101 measures the poses of the first markers and the second markers.

[0071] In order to measure these markers 3 and 4, the cameras 20 (20a to 20d) are installed on the opposite side of the work table 10 from the instructor U1, with their optical axes facing toward the instructor U1 and the tools. The cameras 20 (20a to 20d) are also installed at offset positions so that their respective fields of view 21a to 21d overlap and cover almost the entire top surface of the work table 10.

[0072] After cameras 20 (20a to 20d) are installed on workbench 10, the cameras 20 capture images of reflective markers (not shown) whose positions are known multiple times, thereby calibrating motion capture system 200. Based on the calibration, a coordinate system Σw is set as the coordinate system for measurements by motion capture system 200. Poses measured by motion capture system 200 are expressed based on coordinate system Σw, which is the workbench coordinate system. The pose measurement unit 101 of control device 100 and the like perform processing based on this coordinate system Σw.

[0073] Additionally, the upper surface of the workbench 10 is provided with markers 31 for the stereo camera 30 of the robot 2. Three markers 31 are shown in FIG. 4, and their positions are determined in advance. The positions of the markers 31 for the stereo camera 30 are registered in advance in the control device 100. When the robot 2 operates in front of the workbench 10, the robot 2 can grasp the positional relationship between the robot 2 and the coordinate system Σw by measuring the markers 31 with the stereo camera 30.

[0074] [Tools and First Marker] Fig. 5 shows a perspective view of a test tube 7 as an example of a first tool and a marker 3a, which is a first marker, placed on the test tube 7. Fig. 6 shows a perspective view of a pipette 8 as an example of a second tool and a marker 3p, which is a first marker, placed on the pipette 8.

[0075] The marker 3 is a marker plate for detection by the camera 20 of the motion capture system 200. The marker 3 is not limited to a specific type, and various types can be applied, and one example is shown in the first embodiment. In this example, the marker 3 has marker points, for example, four reflective markers, formed in a unique pattern on the surface of a rectangular flat plate so that a unique ID or the like can be detected for each marker 3. The marker points are formed in a different pattern for each marker 3. The marker points are also referred to as first reflective markers. In this example, marker points are formed at four positions out of 5 × 5 candidate locations on the surface of the rectangular flat plate. Four marker points P31 to P34 are formed on the marker 3a in FIG. 5. Four marker points P35 to P38 are formed on the marker 3p in FIG. 6.

[0076] 5, a predetermined attachment 72a is attached to the top of the test tube 7a, and a marker 3a is attached to the attachment 72a. This attachment 72a is a component that is configured with consideration given to the fact that the hand 5 of the operator U1 or the hand mechanism 6 of the arm of the robot 2 will grip and manipulate the test tube 7a, a tool with a specific shape, and that the marker 3a will be appropriately attached to the tool.

[0077] 6 has a predetermined attachment 82 attached to its top, and a marker 3p attached to the attachment 82. This attachment 82 is a component that is configured with consideration given to the fact that the hand 5 of the operator U1 or the hand mechanism 6 of the arm of the robot 2 will grip and otherwise manipulate the pipette 8, a tool with a specific shape, and that the marker 3p will be appropriately attached to the tool.

[0078] Furthermore, the attachment of the marker 3 to the tool via the attachments 72, 82 is designed taking into consideration that the hand 5 of the worker U1 and the hand mechanism 6 of the robot 2 will operate the same part of the tool. In this example, the attachments 72, 82 have a roughly ring structure that supports the tool by sandwiching its diameter in accordance with the diameter of the tool, and the flat plate of the marker 3 is fixed to one point on the ring.

[0079] The attachments 72, 82 and marker 3 in Figures 5 and 6 are configured so that the flat plate of the marker 3 is fixed to one point on the side of a test tube 7 or a pipette 8, which are roughly rod-shaped tools, and the Z axis shown as the axis perpendicular to the surface of the flat plate of the marker 3 is perpendicular to the one point on the side of the tool.

[0080] The control device 100 can identify the unique ID of the marker 3a and its position and orientation in three-dimensional space by detecting, for example, marker points P31 to P34, which are four reflective markers of the marker 3a, from the image from the camera 20. By identifying the ID, position, and orientation of the marker 3a, it can also identify the ID, position, and orientation of the test tube 7a, which is the tool associated with the marker 3a. This same effect applies to the marker points P35 to P38 of the marker 3p of the pipette 8. This same effect applies to the marker 4 attached to the hand 5 of the operator U1, which will be described later.

[0081] In this example, the four marker points on each marker 3 are arranged asymmetrically in the vertical and horizontal directions. In other words, the four marker points on each marker 3 are arranged in the coordinate system Σ TLM are arranged asymmetrically with respect to the X, Y, and Z axes. This arrangement allows the four marker points, in other words, the image formed by the marker points, to be uniquely detected from the image taken by the camera 20.

[0082] Each of the first markers, marker 3 (3a, 3b), is assigned a coordinate system Σ as a marker plate coordinate system. TLM is set. Coordinate system Σ TLM The three axes of X, Y, and Z in the coordinate system Σ of the first marker are the X-axis and Y-axis, which are two orthogonal axes that form the surface of the rectangular flat plate, and the Z-axis, which is perpendicular to the XY plane. TLM is also referred to as the first marker plate coordinate system.

[0083] The control device 100 calculates the arrangement pattern of the four reflective markers P31 to P34 of the marker 3a in FIG. 5 and the coordinate system Σ TLM Similarly, the control device 100 registers in advance the positional relationship between the four reflective markers P35 to P38 of the marker 3p in FIG. 6 and the coordinate system Σ TLM Based on the registered positional relationships, the pose measurement unit 101 in Fig. 2 calculates the coordinate system Σw of each marker 3 based on the coordinate system Σw, which is the work table coordinate system in Fig. 4. TLM The position and posture of the marker 3 are measured as a pose of the coordinate system Σ TLM Origin O at TLM The posture of the marker 3 can be expressed by the directions of the X, Y, and Z axes.

[0084] The test tube 7a and the marker 3a in Fig. 5 are fixed via an attachment 72a, and a predetermined positional and pose relationship is maintained between the test tube 7a and the marker 3a. The pipette 8 and the marker 3p in Fig. 6 are fixed via an attachment 82, and a predetermined positional and pose relationship is maintained between the pipette 8 and the marker 3p. This prevents the positional and pose relationship between the tool, which is the object to be operated, and the marker 3 from changing during the teaching operation.

[0085] The attachments 72, 82 fix the reflective marker of the marker 3 to a tool such as a test tube 7 or a pipette 8 at a position and angle that allows easy measurement by the camera 20 of the motion capture system 200. The attachments 72, 82 are configured to allow the marker 3 to be attached and detached to the tool. The attachments 72, 82 are configured in a shape that does not interfere with the instructor U1 or the hand mechanism's gripping or other manipulation of the tool.

[0086] [Second marker on hand side] 7A and 7B are perspective views of marker 4, a second marker worn on the hand 5 of worker U1; FIG. 7A shows marker 4R on the right hand 5R, and FIG. 7B shows marker 4L on the left hand 5L. In the example of the first embodiment, marker 4, a second marker, is configured to be worn on a person's forearm near the wrist, and specifically has a wristband-like attachment 92. Marker 4R in FIG. 7A has attachment 92R. Marker 4L in FIG. 7B has attachment 92L.

[0087] In FIG. 7A, the marker 4R worn on the right hand 5R is attached to one location on the side of the attachment 92R. The marker 4 has a rectangular flat plate, similar to the marker 3 on the tool side. Four reflective markers, marker points P41 to P44, are formed on the surface of the flat plate of the marker 4R in FIG. 7A. Four reflective markers, marker points P45 to P48, are formed on the surface of the flat plate of the marker 4L in FIG. 7B. The four reflective markers of each marker 4 are arranged asymmetrically in the vertical and horizontal directions and have a unique pattern so that the ID, position, and orientation of each marker 4 can be detected.

[0088] In the example of the first embodiment, the marker 4 is attached and fixed to a location on the forearm near the wrist using an attachment 92 such as a wristband. This location is not the wrist itself. For the sake of explanation, this location may be referred to as the wrist portion. The configuration of this marker 4 and attachment 92 is designed with consideration given to the fact that the marker 4 is fixed so that the relationship of the position and posture of the marker 4 with respect to the hand 5 of the worker U1 does not change during the teaching work, that the marker point of the marker 4 is easily detected by the camera 20, and that it does not interfere with gripping and other operations during the work motion.

[0089] Each marker 4 has a coordinate system Σ as the second marker plate coordinate system. PRM and coordinate system Σ PLM The marker 4R in Fig. 7A is set in the coordinate system Σ PRM The marker 4L in Figure 7B is set in the coordinate system Σ PLM is set. Coordinate system Σ PRM ,Σ PLM The three axes of X, Y, and Z in the figure include the X-axis and Y-axis, which are two orthogonal axes that form the surface of a rectangular flat plate, and the Z-axis, which is perpendicular to the XY plane.

[0090] The control device 100 determines the arrangement pattern of the four reflective markers P41 to P44 of the marker 4R and the coordinate system Σ PRM The control device 100 registers in advance the position and orientation relationship between the marker 4L and the coordinate system P45 to P48. Σ PLM The position-posture relationship between the markers 4 and the work table coordinate system Σw is registered in advance. Based on the registered relationship, the pose measurement unit 101 measures the pose of each marker 4 based on the coordinate system Σw, which is the work table coordinate system.

[0091] The right wrist of the right hand 5R of the instructor U1 in FIG. 1 is fixed to the marker 4R via an attachment 92R like a wristband in FIG. 7A. The left wrist of the left hand 5L of the instructor U1 is fixed to the marker 4L via an attachment 92L in FIG. 7B. The attachment 92R maintains the position-posture relationship between the right wrist and the marker 4R. The attachment 92L maintains the position-posture relationship between the left wrist and the marker 4L.

[0092] The attachment 92 (92R, 92L) fixes the marker 4 (4R, 4L) to the wrist of the worker U1 in a position and posture that allows for easy measurement by the camera 20 of the motion capture system 200. The attachment 92 also allows the marker 4 to be attached and detached to the hand 5 of the worker U1, and is configured so that the attachment position and posture are adjustable.

[0093] In the first embodiment, the multiple markers shown in Fig. 1 and Figs. 5 to 7B are markers 3 (3a, 3b, 3p) which are first markers on the tool side, and markers 4 (4L, 4R) which are second markers on the hand 5 side. The reflective markers provided on each of these marker plates have a unique pattern for each marker plate, as described above. This allows the control device 100 to identify the individual IDs of five markers, for example, markers 3a, 3b, 3p and markers 4L, 4R, and to simultaneously measure the pose of each marker.

[0094] Instead of using different patterns of reflective markers for each marker plate, the number, size, shape, color, etc. of reflective markers on each marker plate may be different, or identification information may be written on each marker plate, thereby enabling identification of each marker and pose measurement. Furthermore, the number of reflective markers on each marker plate may be three or more. The number of reflective markers may differ for each marker. Markers of different types or structures may be used as the first marker and the second marker.

[0095] In the example of the first embodiment, a plurality of reflective markers are arranged two-dimensionally on the surface of a rectangular flat plate, but they may be arranged three-dimensionally. The reflective markers may be made of, for example, a light-reflective material so as to be easily detected by the camera 20, but this is not limiting. The example shows the camera 20 optically detecting light from the marker plate, but this is not limiting. For example, instead of the camera 20, a motion capture system using a sensor such as a Hall element or a measuring device for magnetic detection may be employed, and the pose of the object to be operated and the arm of the instructor U1 may be measured from the movement of the marker plate.

[0096] Alternatively, a configuration may be adopted in which, without using a marker plate, the movement of the object to be operated and the hand 5 of the instructor U1 are photographed by the camera 20, and the pose is measured by analyzing the image from the camera 20. However, in this case, the above-mentioned problems arise. For this reason, as in the first embodiment, a configuration is adopted in which markers are used on the tool side and the hand 5 side. This makes it easier to determine the correspondence between the hand 5 of the operator U1 and the hand mechanism 6 of the robot 2 without requiring advanced image analysis processing or computational resources, enabling efficient measurement and teaching.

[0097] [Robot hand mechanism] Returning to FIG. 1, the robot 2 is equipped with hand mechanisms 6 configured to resemble a human hand, including a hand mechanism 6R including an arm on the right arm side and a hand mechanism 6L including an arm on the left arm side. In particular, the hand mechanisms 6 are equipped with end effectors 9 at the portions corresponding to the fingertips. The hand mechanism 6R is equipped with an end effector 9R, and the hand mechanism 6L is equipped with an end effector 9L. In the example of the first embodiment, the end effector 9 is a gripping mechanism capable of gripping and releasing a tool, which is an object to be operated.

[0098] 8 to 10 are explanatory diagrams of the hand mechanism 6 of the robot 2, using an example of gripping a test tube 7. FIGS. 11 and 12 are explanatory diagrams of the relationship between coordinate systems, etc. Using FIGS. 8 to 11, the relationship between the poses and coordinate systems among the hand mechanism 6L on the left side of the robot 2, the left hand 5L of the operator U1, and the test tube 7a, which is the object to be gripped, will be explained as an example.

[0099] FIG. 8 is a perspective view of the left hand mechanism 6L. FIG. 8 shows a hand unit 520, which is a part of the hand mechanism 6L. In other words, the hand unit 520 is a link mechanism that resembles a human wrist or the like and is capable of rotating around the axis of a flange 522. The hand unit 520 is equipped with a gripping mechanism serving as an end effector 9L at its tip. In this example, the gripping mechanism has two finger mechanisms, 521a and 521b. A test tube 7 is gripped by the finger mechanisms 521a and 521b of the end effector 9L. The finger mechanisms 521a and 521b are configured to be movable left and right in the Y-axis direction in FIG. 8 by a slide drive mechanism. The test tube 7 can be gripped by closing the finger mechanisms 521a and 521b, and the test tube 7 can be released by opening the finger mechanisms 521a and 521b. The test tube 7 is gripped by being pinched with an appropriate pressure by the finger mechanisms 521a and 521b of the hand part 520.

[0100] Fig. 9A is a side view (in the direction of the Y axis in the drawing) of the hand mechanism 6L in Fig. 8, and Fig. 9B is a top view (in the direction of the X axis in the drawing) of the hand mechanism 6L in Fig. 8. Figs. 8 to 10 show a state in which, for example, when the long axis of a test tube 7 is positioned along the vertical direction, the hand portion 520 of the left hand mechanism 6L of the robot 2 grasps one portion of the upper side surface of the test tube 7 by sandwiching it from the left and right with the finger mechanisms 521a, 521b of the end effector 9L.

[0101] 9A and 9B, the gripping center of the finger mechanisms 521a and 521b of the end effector 9L in the hand portion 520 of the hand mechanism 6L is defined by a coordinate system Σ LTThe gripping center also corresponds to the position of the test tube 7. Coordinate system Σ LT Of the three axes, X, Y, and Z, the up-down direction along the long axis of the test tube 7 is the X axis, the front-to-back direction relative to the gripping mechanism of the hand unit 520 is the Z axis, and the left-to-right direction relative to the gripping mechanism is the Y axis.

[0102] The flange 522 is a mechanism corresponding to the left wrist (in other words, the left link) of the arm of the robot 2 to which the hand portion 520 of the hand mechanism 6L is attached. At the center of the flange 522, a coordinate system Σ LE is set. Coordinate system Σ LE The three axes X, Y, and Z of the coordinate system Σ LT Similarly, the vertical direction with respect to the hand unit 520 is the X axis, the front-rear direction is the Z axis, and the left-right direction is the Y axis. LT and the coordinate system Σ LE The coordinate system Σ in FIGS. 9A and 9B shows the relationship between the position and the orientation of the object. LT Origin of O LT and the coordinate system Σ LE Origin of O LE has a distance Lt in the Z-axis direction.

[0103] Although not detailed here, the right hand mechanism 6R of the robot 2 can have a structure similar to that of the left hand mechanism 6L. For example, the right hand mechanism 6R can grip the pipette 8 in a similar manner. Of course, the left and right hand mechanisms can also have different structures.

[0104] [Regarding the relationship between the tool, hand mechanism, hand, and each marker's coordinate system] FIG. 10 shows the coordinate system (Σ LT ,Σ LE ) and the coordinate system Σ set at marker 3a of test tube 7a TLM 10 is a schematic explanatory diagram of the relationship between the coordinate system Σ, which is the left gripping unit coordinate system set in the hand unit 520 of the hand mechanism 6L that grips the test tube 7. LT , and the coordinate system Σ, which is the coordinate system of the tip of the left link LE(FIGS. 9A and 9B), and a coordinate system which is a first marker plate coordinate system set at marker 3a which is the first marker on the test tube 7 side. Σ TLM 10 shows the relationship with (FIG. 5). In order to facilitate understanding, in FIG. 10, the state in which the marker 3a is attached to the test tube 7a is shown by a virtual dotted line.

[0105] FIG. 11 shows the coordinate system Σ of the test tube 7a and the hand mechanism 6L associated with it, which is the same as FIG. 10. LT The coordinate system Σ is set to the marker 4L on the left hand 5L of the worker U1. PLM 11 is a schematic explanatory diagram of the relationship between the test tube 7a held by the left hand 5L of the operator U1 and the coordinate system Σ of the hand mechanism 6L (not shown), in correspondence with FIG. LT and the coordinate system Σ LE and the coordinate system Σ set at marker 4L on the left hand side 5L PLM 11 shows the relationship between the test tube 7a and the left hand 5L (FIG. 7B). In FIG. 11, when the long axis of the test tube 7a is positioned vertically as in FIG. 10, the test tube 7a is grasped by fingers 1101 of the left hand 5L, such as the thumb and other fingers, at one point on the side of the upper part of the test tube 7a, sandwiching the test tube 7a from both sides. Also, FIG. 11 shows the state in which the flat plate of the marker 4L is placed via an attachment 92L on one side of the forearm of the left hand 5L near the wrist, in a direction corresponding to the back of the palm 1102.

[0106] 10 and 11, the coordinate system (Σ LT ,Σ LE ) and the coordinate system Σ set on the marker 3a on the side of the tool, the test tube 7a TLM and the coordinate system Σ set at marker 4L on the left hand side 5L PLM By considering Figure 10 and Figure 11 together, it is possible to understand the relationship between the coordinate systems.

[0107] As shown in Figure 10, the coordinate system Σ of the marker 3a TLMis the coordinate system Σ, which is the left grip coordinate system. LT is translated in the Z-axis direction (to the right in FIG. 10) by a distance Lm. LT is the coordinate system of the tip of the left link, Σ LE is translated in the Z-axis direction by a distance Lt.

[0108] Therefore, if the pose of the marker 3a is measured, the origin O, which is the grip center of the hand unit 520 at that time, can be calculated from the relationship between these coordinate systems. LT Furthermore, the origin O, which is the center of the left wrist flange 522, can be calculated. LE That is, the pose of the marker 3a can be associated with the pose of the operation of the hand portion 520 of the hand mechanism 6L based on the coordinate transformation.

[0109] In FIG. 10, as shown by the dashed arrow, the coordinate transformation is performed by using a coordinate system Σ that corresponds to the coordinate system of the object to be grasped, with the test tube 7a being the object to be grasped as the reference. TLM and a coordinate system Σ, which is a coordinate system of the left gripping part at the tip, based on the hand part 520 of the hand mechanism 6L. LT Coordinate transformation T1 and coordinate system Σ LT and the coordinate system Σ LE Furthermore, in FIG. 11, for a part of the coordinate transformation, the transformation T LT The transformation T LT is the coordinate system Σ of the marker 4L on the left hand 5L PLM From the pose at the position, the coordinate system Σ corresponding to the gripping center of the test tube 7 at the tip of the hand part 520 is LT This is the conversion to a pose.

[0110] By performing such coordinate transformation based on the measurement of the pose of the marker 3a during the work demonstration, the teaching data generation unit 107 in Figure 2 can calculate the pose of the hand portion 520 of the hand mechanism 6L that corresponds to the pose of the test tube 7a as a teaching pose.

[0111] In this example, instructor U1 holds test tube 7a with fingers 1101 of left hand 5L in the same pose as in normal work, as shown in Fig. 11. Hand unit 520 of hand mechanism 6L in Fig. 10 is instructed to hold test tube 7 with finger mechanisms 521a and 521b of hand unit 520 in a pose similar to that shown in Fig. 11.

[0112] At this time, the coordinate system Σ, which is the left gripping coordinate system of the hand unit 520 that grips the test tube 7, LT is defined as the same coordinate system as in Figure 10, and at the same time, the coordinate system Σ PLM is used as the base, and the coordinate system Σ PLM From the coordinate system Σ LT Transformation T to derive LT The control device 100 determines the transformation T LT The conversion coefficients for the coordinate conversion in the above equation are set in advance. This setting may be performed by the administrator U2, or may be calculated by the control device 100.

[0113] In addition, the coordinate system Σ corresponding to the position of the test tube 7a and the gripping center LT and the coordinate system Σ, which is the coordinate system of the tip of the left link. LE The transformation between these coordinate systems is obtained by the coordinate transformation T2. ​​Based on the relationship between these coordinate systems, the coordinate system Σ of the marker 4L on the left hand 5L is PLM and the coordinate system Σ of the hand part 520 LE By measuring the pose of the marker 4L of the left hand 5L, the control device 100 can convert the origin O, which is the grip center of the hand unit 520 of the hand mechanism 6L at that time, into the pose T3. LT The pose of the flange 522 can be calculated. LE The pose can be calculated.

[0114] That is, the teaching data generating unit 107 in FIG. 2 uses the coordinate system Σ PLM and a coordinate system Σ based on the hand part 520 of the hand mechanism 6L. LT and the coordinate system Σ LEThe pose of the hand unit 520 is generated as a teaching pose by conversion based on the relationship between the above.

[0115] Furthermore, based on the above-described coordinate system relationships and taught poses, the robot motion generation unit 109 calculates the joint displacements of the mechanism of the robot 2 as solutions of inverse kinematics calculations from the time-series data of the pose of the center of the flange 522 of the hand mechanism 6 of the robot 2, in other words, the tip of the link. Then, the robot motion generation unit 109 generates a joint displacement sequence of the mechanism of the robot 2 from this calculation, and generates robot motion data according to this joint displacement sequence.

[0116] Each coordinate system (Σ TLM ,Σ TRM ,Σ PRM ,Σ PLM ) and the coordinate system (Σ LE ,Σ LT ) can be converted into a pose in the coordinate system Σw, which is the workbench coordinate system, based on the correspondence between the coordinate systems.

[0117] The relationship between the above-mentioned coordinate systems is based on the relationship of parallel translation in the Z-axis direction in FIG. 10 and the transformation T LT The relationship between the coordinate systems is not limited to a relationship including rotation of the coordinate systems as long as the relationship between the coordinate systems is known. Furthermore, the relationship between the coordinate systems does not need to be fixed, and may vary as long as the relationship is known.

[0118] In the above example, a test tube 7a is grasped using the left hand 5L of the instructor U1 and the left hand mechanism 6L of the robot 2, but this is not limited to this and can also be applied to cases where a pipette 8 or other test tube 7 is grasped or otherwise operated, or when the right hand 5R of the instructor U1 and the right hand mechanism 6R of the robot 2 are used.

[0119] Next, using Figures 12 and beyond, we will explain examples of work movements by instructor U1, work teaching using control device 100, calculation and correction of teaching poses for conversion into movements of robot 2, etc. in the robot teaching method and device of embodiment 1.

[0120] [Work demonstration] 12 and 13 show examples of work movements in a work demonstration when operator U1, who is an instructor, performs a teaching task using the robot teaching device 1. FIG. 12 shows a state in which, as part of a work movement, operator U1 uses both hands, holding a test tube 7 with his left hand 5L and a pipette 8 with his right hand 5R. Initially, test tubes 7 (7a, 7b) are held in holder 71, and pipette 8 is held in holder 81. FIG. 13 shows a state in which, after the work movement of FIG. 12, operator U1 dispenses a substance in the pipette 8 in his right hand 5R into the test tube 7 in his left hand 5L, or aspirates a substance in the test tube 7 in his left hand 5L into the pipette 8 in his right hand 5R, as another part of the work movement.

[0121] As shown in the figure, the instructor U1 wears a second marker, 4R, near his right wrist and a marker, 4L, near his left wrist. In FIG. 12, the instructor U1 demonstrates operation M1 by performing operation M1 of grasping a test tube 7a equipped with a first marker, 3a, using his left hand 5L. In other words, operation M1 is a left-hand grasping action, a left-hand grasping step, and a left-hand grasping process. FIG. 12 also illustrates operation M2 using the right hand 5R. In the work demonstration, the instructor U1 demonstrates operation M2 by performing operation M2 of grasping a pipette 8 equipped with a first marker, 3p, using his right hand 5R. In other words, operation M2 is a right-hand grasping action, a right-hand grasping step, and a right-hand grasping process.

[0122] In the following explanation, the features of the first embodiment will be explained using the example of the grasping operation M1 with the left hand 5L targeting the test tube 7a, but the same can be applied to the case of the operation M2 with the right hand 5R. In the following explanation, each pose measured and calculated by the control device 100 is based on the coordinate system Σw, which is the workbench coordinate system.

[0123] [Processing flow] Fig. 14 shows a processing flow in the robot teaching device 1 and method according to the first embodiment. The processing is mainly performed by the control device 100. The flow in Fig. 14 has steps S101 to S110. It is assumed that necessary correction data and the like have been set in advance.

[0124] In step S101, the instructor U1 or the manager U2 inputs a start instruction as a teaching instruction. The instructor U1 starts a work motion as a work demonstration. In response to the start instruction, the system, which is the robot teaching device 1, starts measurements, etc., and the camera 20 of the motion capture system 200 starts taking pictures. In the work demonstration, the instructor U1 first starts the left-hand grip operation M1 in FIG. 13. At this time, the control device 100 detects the start instruction, in other words the start of the work demonstration, based on a voice signal such as "start" input via the instruction input device 300, for example, a microphone, through processing by the teaching instruction detection unit 105. Based on this, the control device 100 starts measurement processing in the pose measurement unit 101, while sending data of a teaching command ("start") indicating the start of teaching data to the teaching data generation unit 107. Teaching pose Generated to associate Let do.

[0125] In step S102, the control device 100 inputs and acquires images from the camera 20 of the motion capture system 200 through processing by the pose measurement unit 101, and acquires a first measurement pose obtained by measuring the marker 3a on the tool side and a second measurement pose obtained by measuring the marker 4L on the hand 5L in time series in accordance with the demonstration by the instructor U1. Then, the pose measurement unit 101 stores the first measurement pose in the first measurement pose storage unit 102, and stores the second measurement pose in the second measurement pose storage unit 103.

[0126] Next, in step S103, when performing operation M1, the instructor U1 inputs an operation instruction corresponding to operation M1, for example, a gripping operation instruction. The control device 100 detects the gripping operation instruction through processing by the operation instruction detection unit 104. The operation instruction detection unit 104 detects the gripping operation instruction input at the timing when the instructor U1 grips the test tube 7a with the left hand 5L. As a result, the control device 100 detects the gripping operation M1 of the test tube 7a, which is the object to be operated.

[0127] At this time, the instructor U1 uses the instruction input device 300 to input the gripping operation instruction at the same timing as the operation M1. For example, the instructor U1 inputs the gripping operation instruction when gripping the test tube 7a with the left hand 5L, for example, immediately after pinching the test tube 7 with the fingers 1101 as shown in FIG. 11 . When giving the gripping operation instruction, the instructor U1 inputs a signal by pressing, for example, a pressure-sensitive switch attached to the fingers as the instruction input device 300. Alternatively, the instructor U1 inputs a signal by pressing a foot switch at the feet. Alternatively, the instructor U1 inputs a predetermined voice representing the gripping operation M1, such as "close," as a voice input through a microphone.

[0128] The input voice for the operation instruction can be determined in advance and can be "grasp," "hold," "get," etc. Other operations, such as a release operation, can be "open," "release," etc. Furthermore, when a physical switch button is used, it can be determined in advance what operation instruction the signal of that switch button represents. Furthermore, when the order of various operations is determined in a time-series scenario, which will be described later, it is possible to determine what operation instruction the input signal represents by determining the number of times and order of signal inputs corresponding to that.

[0129] In step S104, the control device 100 checks whether an operation instruction for a predetermined operation was detected in step S103, and if an operation instruction for, for example, operation M1 was detected (Yes), the control device 100 proceeds to step S105, and if it was not detected (No), the control device 100 proceeds to step S107.

[0130] In step S105, the control device 100 generates a teaching pose in time series using the first measurement pose and the second measurement pose stored from the start of the task demonstration to the present time through processing by the teaching data generation unit 107, and generates an operation movement command synchronized with the teaching pose at the latest time. In other words, the teaching data generation unit 107 generates a teaching pose in time series associated with an operation movement command at a timing matching the operation command, and stores the teaching pose in the teaching data storage unit 108. The operation movement command is, for example, a command representing the execution of a grasping movement by the hand mechanism 6L of the robot 2 synchronized with the operation command for the left-hand grasping operation M1, and is a teaching command for generating robot movement data, which will be described later.

[0131] In step S106, the control device 100 corrects the teaching pose up to the latest time point by processing the teaching data generation unit 107 based on reference to the correction data in the correction data storage unit 106, generates corrected teaching data, and stores it in the teaching data storage unit 108. The correction will be described later.

[0132] In step S107, the control device 100 checks whether a completion instruction has been detected as a teaching instruction through processing by the teaching instruction detection unit 105, and if it has been detected (Yes), proceeds to step S108, and if it has not been detected (No), returns to step S102 and repeats the same process. At this time, the completion instruction can be, for example, input by the worker U1 using the instruction input device 300 via, for example, a microphone, a predetermined voice, such as "end," and the teaching instruction detection unit 105 detects it as a completion instruction based on the voice signal.

[0133] In step S108, the control device 100 generates corresponding teaching data up to the latest time point based on the first measurement pose and the second measurement pose as measurement poses up to the latest time point at which the completion instruction was received, and stores the generated teaching data in the teaching data storage unit 108. At this time, the control device 100 additionally generates teaching poses in a time series up to the latest time point at which the completion instruction was received, which have not been generated at the time of step S108, and generates the teaching poses so as to associate data of a teaching command ("end") that means the end of the teaching data.

[0134] In step S109, the control device 100 corrects the teaching data as necessary, and ends the processing related to the teaching data.

[0135] In step S110, the control device 100 generates robot motion data by processing the robot motion generation unit 109 based on the teaching data in the teaching data storage unit 108, and stores the generated robot motion data in the robot motion data storage unit 110. The process of generating the robot motion data from the teaching data may be executed automatically when the teaching data is generated, or may be executed at a later timing in response to an instruction input by the administrator U2, etc.

[0136] In the above flow, for ease of understanding, a processing example focusing on the teaching of one operation M1 has been described. However, this is not limiting, and teaching multiple operations in a work demonstration can also be realized by performing similar processing for each operation. For example, if the above-mentioned gripping operation M1 is followed by another gripping operation M2, similar processing can be applied to that operation M2. Alternatively, if the above-mentioned gripping operation M1 is followed by a release operation, such as releasing the test tube 7a from the left hand 5L at a predetermined position, similar processing can be applied to that release operation. Different correction data can be applied depending on the object to be operated and the operation, such as gripping or releasing.

[0137] [Teaching data calculation part 1] Calculation and correction of the teaching pose generated in the processing of steps S105 and S106 will be described using FIG. 15A and other figures. FIGS. 15A, 15B, and 15C are explanatory diagrams of calculation and correction of the teaching pose in embodiment 1. FIG. 15A and other figures show graphs representing, for example, the results of acquiring the first and second measurement poses by the pose measurement unit 101 and the results of generating teaching data by the teaching data generation unit 107 when teaching the left-hand grip operation M1 in FIG. 12. The horizontal axis of the graph represents time (t), and the vertical axis represents the position coordinate in the X-axis direction, for example, as one direction in the coordinate system Σw of the workbench 10. For ease of understanding, FIG. 15A and other figures show time-series data of only the position in the X-axis direction in the coordinate system Σw. However, similar position information is also included in the Y-axis and Z-axis, and orientation information is also included in each axis representing the posture, although not shown. The time sampling period is predetermined.

[0138] FIG. 15A shows data D1, D2, D11, and D12. Data D1 is the displacement data on the X axis as time series data of the first measurement pose on the tool side, i.e., the measurement pose of the marker 3a of the test tube 7a. Data D2 is the displacement data on the X axis as time series data of the second measurement pose on the hand 5 side, i.e., the measurement pose of the marker 4L of the left hand 5L. Data D11 is the displacement data on the X axis as time series data of the measurement pose of the marker 4L of the left hand 5L in the coordinate system Σ of FIG. TLMThe coordinate system Σ in Fig. 10 is calculated from the data D1 of the first measurement pose corresponding to the pose at LT The data D12 is the time series data of the pose at the time of the second measurement pose. LT The coordinate system Σ in Figure 10 is calculated using LT In other words, data D11 is the time series data of the pose at the grip center position (origin O) from data D1. LT The data D12 is the data converted into a pose at the grip center position (origin O) from the data D2. LT ) is the data converted into a pose.

[0139] 15B shows data D20 and D21. FIG. 15B also shows examples of timing of the teaching instructions and operation instructions described above, and shows data D30, D31, and D32 generated corresponding to them. Data D20 is calculated based on the coordinate system Σ, which is the two left grip coordinate systems shown in FIG. 15A. LT The data D20 is time-series data of the taught pose calculated from the pose data D11 and D12 at time t, and is shown by a solid line. Specifically, the data D20 is generated by selecting the data D12. The data D21 is time-series data after correcting the taught pose data D20, and is shown by a dashed line. In this example, the data D20 from time t=0 to m has been corrected to data D21.

[0140] Data D30 is data of a start instruction ("start") as a teaching instruction. Data D32 is data of a completion instruction ("end") as a teaching instruction. Data D31 is data of a grip operation instruction ("close") as an operation instruction. Time t=0 is the time when the start instruction ("start") is input and detected. Time t=m is the time when the grip operation instruction ("close") of operation M1 is input and detected. Time t=n is the time when the completion instruction ("end") is input and detected. The left hand grip operation M1 corresponds particularly to the movement around time t=m.

[0141] FIG. 15C shows data D40. Data D40 is time-series data based on the corrected data D21 of FIG. 15B, and is teaching data consisting of data D41 and data D42. Data D41 is teaching pose data from time t=0 to m, and data D42 is teaching pose data from time t=m to n. Data D41 corresponds to the movement leading up to operation M1, and data D4 2 corresponds to the action after operation M1.

[0142] First, calculation of a basic teaching pose will be described. The time when the teaching pose generation process in step S105 in Fig. 14 is executed is set to time t=m, which corresponds to the timing of the gripping operation instruction ("close"). At that time, as shown in Fig. 15A, the control device 100 has already measured chronologically data D1 of the pose of the tool-side marker 3a and data D2 of the pose of the hand-side marker 4L from time 0 to time m. In step S105, the control device 100 calculates each of the data D1 and data D2 for the time up to time m in the coordinate system Σ, which is the left gripping part coordinate system, based on the relationship in Figs. 10 and 11, as shown in Fig. 15A. LT The poses are converted into data D11 and D12.

[0143] Also, in step S105, the control device 100 recognizes that the operation instruction is for an ungrasped state up to time m, based on the fact that a gripping operation instruction ("close") is input at time m or that data D1 (e.g., X-axis position coordinate) has not changed from time 0 to time m.

[0144] Therefore, the control device 100 selects data D12, which is the measurement result of the pose of the hand side up to the grasping at time m, from data D11 and D12, as the teaching pose and generates it as data D20 shown in Fig. 15B. The portion of data D20 in Fig. 15B from time 0 to m is the same as the portion of data D12 in Fig. 15A from time 0 to m.

[0145] [Teaching Pose Correction] Next, correction of the teaching pose will be explained. The tool-side data D11 shown in FIG. 15A is a direct measurement of the pose of the test tube 7a itself, and therefore represents a pose that the hand mechanism 6L of the robot 2 can grasp with high precision at time m. On the other hand, the hand-side data D12 is generated by converting the pose of the marker 4L near the left wrist of the instructor U1. Therefore, this data D12 represents the grasping operation at time m, particularly the trajectory of the left hand 5L up to the grasping operation instruction ("close"), but the pose of grasping the test tube 7a with the left hand 5L at time m contains a large amount of error. The teaching pose data D20 generated by selecting the data D12 also has similar characteristics.

[0146] 15A and other figures, the error caused by the hand movement of worker U1 is shown as a difference ΔX in the X-axis position. The robot teaching device 1 and method of embodiment 1 correct the teaching data by taking such an error into account, thereby generating suitable teaching data. Therefore, in step S106, the control device 100 calculates a correction coefficient for the correction so as to match the pose of the hand-side data D12 at time m with the pose of the tool-side data D11.

[0147] The control device 100 performs a correction calculation on the data D20 corresponding to the data D12 shown in Fig. 15B using a correction coefficient, going back in time from the time m of gripping, to obtain corrected taught pose data D21. In this example, in this correction calculation, the difference ΔX in the positions of the data D12 and the data D11 in the X-axis direction is used as the correction coefficient. The taught data generation unit 107 generates the position of the corrected taught pose data D21 by subtracting the difference ΔX from the position of the data D20 going back in time from the time m, going back to t = 0 in this example.

[0148] Then, the teaching data generating unit 107 generates teaching data by combining the corrected teaching pose data D21 with the teaching instruction ("start") data D30 and the operation instruction ("close") data D31 that are stored in synchronization with the time. The generated teaching data corresponds to data D41 as data D40 from time 0 to time m shown in FIG. 15C.

[0149] Thereafter, the teaching data generating unit 107 stores the teaching data D41 from time 0 to m, together with various related data used to generate the teaching data D41, in the teaching data storage unit 108 as teaching data D40.

[0150] In addition, when calculating the corrected teaching pose data D21, a limit is set in advance for the range of going back in time based on physical quantities such as the moving distance and the moving time, and the limit is set and stored as part of the correction data in the correction data storage unit 106. The control device 100 may perform the above correction process using the correction data.

[0151] In this case, the teaching data generation unit 107 performs a correction operation on the data D20 corresponding to the data D12 going back in time until it reaches a certain time i corresponding to one of the restrictions in the correction data, i.e., the boundary time of the range, to generate data D21 from time i to time m, which is used as part of the corrected data D41. For the part before time i, the teaching data generation unit 107 selects the data D20 corresponding to the data D12 without correction, and uses this as part of the corrected data D41.

[0152] Furthermore, in order to suppress the amount of change in the positions of data D20 and data D21 before and after the time i at which the limit is reached, the teaching data generation unit 107 may apply a correction coefficient to data D20 within the range of the limit to perform a correction calculation, and may also perform additional correction near the time i at which the limit is reached so that the pose matches that of data D20 or so that the change in the pose is as small as possible. Examples of processing using the above-mentioned limits will be described later.

[0153] [Teaching data calculation part 2] 15A and other figures, the details of generating teaching data after time m in steps S108 and S109 are as follows. As shown in FIG. 15A, the execution time of step S108 is set to n. The control device 100 has measured data D1 and data D2 in chronological order from time m to time n. At this time, the control device 100 can recognize that, from time m to n, the left hand 5L moved the test tube 7a while still gripping it, as a work motion, based on the fact that no release operation instruction corresponding to the operation of releasing the test tube 7a has been input or detected between the time when the instructor U1 performed the gripping operation M1 at time m and the latest time n, or that data D11, which is the measurement result of the pose of the test tube 7a itself, has changed.

[0154] Therefore, from time m to time n, the control device 100 can accurately trace the pose of the test tube 7a handled by the instructor U1 in the demonstration by matching the gripping pose of the hand mechanism 6L of the robot 2 to data D11 after conversion of the tool-side pose. Therefore, in step S108, the teaching data generation unit 107 selects data D11 from time m to time n to generate it as teaching pose data D20, which is data D42 in FIG. 15C.

[0155] Furthermore, in this example, in step S109, the teaching pose itself generated in step S108 is not corrected, but teaching data is generated by combining teaching pose data D20 with teaching instruction ("end") data D32 stored in synchronization with the time. The generated teaching data corresponds to data D42 as data D40 from time m to n in FIG. 15C. Thereafter, the teaching data generating unit 107 stores the teaching data D42 from time m to n, together with various related data used to generate the teaching data D42, as teaching data D40 in the teaching data storage unit 108.

[0156] 15A and other examples have mainly described correction of the pose position, but correction can also be applied to the posture. For example, the control device 100 calculates the difference between the posture of the first measurement pose on the tool side and the posture of the second measurement pose on the hand side, and generates teaching data using this posture difference.

[0157] 8 to 11, the case has been described in which the locations where the hand 5 of the instructor U1 and the end effector 9 of the hand mechanism 6 each grip or manipulate the same tool (for example, the upper side of a test tube 7) are predetermined. Even if the location where the hand 5 of the instructor U1 manipulates the tool deviates from the predetermined location during instruction, the deviation can be tolerated by correcting for the error.

[0158] [Variations] FIG. 16A and the like are explanatory diagrams of modified examples related to correction that take into account the error in FIG. 15A and the like, and show graphs similar to those in FIG. 15A and the like. The measurement pose obtained as a premise is the same as that in FIG. 15A. FIG. 16A shows Modification 1A as a modified example. In Modification 1A, the control device 100 traces back from time m, when the operation M1 is detected, to time i within a time range 1601 determined according to a restriction, and corrects data D20 corresponding to data D12 within that range 1601 to data D21. This correction is, for example, subtracting a difference ΔX, as described above. After correction, the data D21 is as shown by the dashed line. Data D20 before time i and data D20 after time m are connected to data D21 in the range 1601 from time i to time m to form teaching data. In this example, the range 1601 corresponding to the restriction is the time from time m to the time i three times before. The range 1601 corresponding to the restriction may be determined in advance as a predetermined time, or distances on each axis such as the X axis may be determined in advance as thresholds, and the range 1601 may be determined as the time when a displacement occurs corresponding to the distance threshold. It is more preferable to determine this restriction range 1601 according to the structure of the manipulated object and the hand mechanism 6, etc.

[0159] 16B and 16C show a further modified example, Modification 1B. Modification 1B performs additional correction on the corrected data so as to connect uncorrected data before and after it in the time series as smoothly as possible, in other words, to minimize the amount of change in pose between the previous and next data in the time series. In this example, data D20 different from the example in FIG. 16A is used, and the fluctuation in the X-axis position is greater up to time m. First, as in Modification 1A in FIG. 16A, data D20 is corrected to data D21 from time i to m within the range 1601 according to the constraint. This correction is, for example, subtraction of a difference ΔX. The portion after this first correction is shown as data D21a. Next, a second correction is performed on data D21a. The corrected portion is shown as data D21b. In this second stage of correction, correction is performed so that the connection between data D21a and data D20 becomes smoother near time i, which is the boundary of range 1601, i.e., so that the amount of change in the pose position and posture is as small as possible.

[0160] In a pause obtained by connecting data D20 before and after data D21a, for example, the amount of change in position in the X-axis direction between time i and the immediately preceding time j is large. The control device 100, for example, calculates the amount of change between times, and if the amount of change is greater than or equal to a threshold, performs a second-stage correction. For example, the amount of change between time j and time i is set to A1, and the amount of change A1 is set to greater than or equal to a threshold. The control device 100 determines a range 1602 for the second-stage correction, for example, based on time i, which is the boundary of range 1601. The range 1602 is determined so as to include at least the time from time j to time i corresponding to the amount of change A1. In the illustrated example, the range 1602 is the time from time j to time i. The control device 100 corrects the pause at each time within this range 1602. An example of this correction is statistical processing such as averaging.

[0161] FIG. 16C shows an example of data D21b after the second-stage correction. The open circles indicate the positions after correction. In this example, the average values ​​of the positions at time j and time i before correction are calculated, and this average value is used as the positions at time j and time i after correction. This correction is not limited to this, and range 1602 may be broadened, or other statistical processing may be employed. Furthermore, while the time-series data representing the pose trajectory has been described as connecting the positions and postures at each time with straight lines, it may also be time-series data generated by connecting the positions and postures at each time with curved lines.

[0162] FIG. 17A shows another modified example, Modification 1C. In Modification 1C, after a first-stage correction is performed, similar to FIG. 16B, for example, suppose that the amount of change A1 between time j and time i is greater than or equal to a threshold. In this case, in Modification 1C, the data at time j is considered noise, and a second-stage correction is performed to align it with the position of time i, for example. The corrected data is shown as data D21c.

[0163] FIG. 17B shows another modified example, Modification 1D. In this Modification 1D, after a first stage of correction is performed, similar to FIG. 16B, for example, suppose that the amount of change A1 between time j and time i is greater than or equal to a threshold. In this case, in Modification 1D, the positions of each time are corrected so that a straight line connects the position of time j to the position of time m. The corrected data is shown as data D21d by a dashed line.

[0164] As described above, in the methods of each modification, the step of generating teaching data includes a step of additionally correcting a portion of the data that is to satisfy a predetermined value by statistical processing, noise removal processing, or the like, when the amount of change between the data immediately before and after the data in the measurement data in the vicinity of where the operation is detected and the data after correction to reduce errors does not satisfy a predetermined specified value. Alternatively, data may be corrected by a method using known statistical processing, or the like, within a range in which the pose of the hand-side data D12 can be matched to the pose of the tool-side data D11 at the same time as the data D31 of the operation instruction ("close") (time m of the operation instruction).

[0165] [GUI screen] FIG. 18 shows an example of a screen display including a GUI that the robot teaching device 1 of the first embodiment provides to the user, a worker U1, or a manager U2. The screen of FIG. 18 may be provided in the form of a web page, for example. The screen of FIG. 18 is displayed on the display screen of the display device of the input / output device 120 (the output device 1006 in FIG. 3) based on processing by the teaching instruction detection unit 105 of the control device 100 in FIG. 2, for example. The screen of FIG. 18 shows an example configured as a work scenario display screen. This screen allows the setting and editing of a work operation scenario. This screen includes a scenario editing screen section 1801 and a teaching data confirmation screen section 1802.

[0166] On this screen, the user, worker U1 or administrator U2, can set the target work operation, necessary correction data, setting information, etc., and can check the data and information, as well as the teaching data generated according to the work demonstration, etc. In addition, the aforementioned teaching instructions, etc. may be input on this screen, or a separate GUI screen may be provided for tasks such as teaching instructions.

[0167] The scenario editing screen unit 1801 displays a work scenario and has a GUI that allows the user to edit it. A work scenario is a scenario of the work actions to be demonstrated. It is divided into task units and can be associated with the teaching data required for each task unit. In the illustrated example, the work scenario of the target work actions is configured as a sequential function chart having task units or steps such as "Initial," "Get test tube," and "Move test tube." "Get test tube" shown in step S1810 is an "action / step of grasping a test tube" and corresponds to the aforementioned operation M1 of grasping the test tube 7. "Move test tube" shown in step S1811 is an "action / step of moving a test tube" and corresponds to the operation of moving the grasped test tube 7 after the grasping operation M1.

[0168] The user can set what work actions to teach by editing the work scenario on the scenario editing screen section 1801. On this screen, the user can create a new work scenario, save it with a name, and open and close a work scenario that has already been set. In accordance with the set work scenario, the control device 100 creates data and information such as the measurement data and teaching data (e.g., data D41, D42) shown in FIG. 15A etc. so as to associate each action / step of the work scenario.

[0169] In this example, the work scenario in the scenario editing screen portion 1801 is written as a sequential function chart, but it is not limited to this and any other format may be used as long as the work scenario and transitions can be expressed.

[0170] The teaching data confirmation screen section 1802 has a GUI that displays and allows confirmation of the teaching data of the work operation generated in association with the work scenario, as well as the contents of various data used to generate the teaching data. When a user selects a work scenario or an operation / step (e.g., step S1810) on the scenario editing screen section 1801, the GUI of the screen shown in FIG. 18 has a function of displaying data corresponding to the selected step, such as the teaching data D40 shown in FIG. 15C and various data used to generate the teaching data. This screen may also allow the user to select the measurement data, teaching data, and other items to be displayed. In this example, the teaching data confirmation screen section 1802 displays various measurement data, teaching data, and the like in a graph format, similar to FIG. 15A . In this example, the value of the difference ΔX corresponding to the error described above is also displayed, allowing confirmation of the teaching data before and after correction.

[0171] Although not shown, the data of the teaching instructions and operation instructions described above can also be displayed on the teaching data confirmation screen section 1802. Furthermore, when a judgment threshold or the like is used for the correction that takes the error into consideration or the additional correction described above, setting information such as the threshold can also be displayed on the screen, allowing the user to check and set the setting information. Furthermore, multiple correction methods may be implemented as shown in each of the above-mentioned modified examples, and in that case, the user may be able to select and set the correction method to be applied from the multiple correction methods on the screen.

[0172] In this example, the teaching data confirmation screen section 1802 displays a graph of each piece of data such as teaching data, but is not limited to this and may be anything that allows the content of each piece of data to be confirmed. For example, it may display a database data table, a command, or a three-dimensional display of a pose as in the example described below.

[0173] [Effects of the First Embodiment] According to the robot teaching method of the first embodiment, even a person without knowledge of robots can intuitively and efficiently teach a robot a work motion by demonstrating the work while maintaining normal human movements as much as possible, thereby reducing the amount of pre-adjustment work required for teaching and facilitating implementation. According to the first embodiment, the work motions normally performed by an actual worker can be taught while maintaining the poses of the normal work motions without requiring or minimizing changes for teaching. Therefore, even a person without knowledge of robots can easily and intuitively realize efficient teaching through work demonstrations. Furthermore, according to the first embodiment, high-precision recognition and image analysis processing of the instructor's hand pose and tool pose are not required, thereby saving computational resources, reducing the amount of pre-adjustment work, and facilitating the introduction and implementation of a robot teaching system.

[0174] According to embodiment 1, the teaching data is generated using a first marker on the tool side and a second marker on the hand side, including corrections for errors, so that it is possible to teach work movements that include situations where the instructor U1's hand is away from the tool.

[0175] Various modifications of the first embodiment are possible. For example, the motion capture system 200 uses the camera 20, but this is not a limitation and other measuring devices or sensors may be used to detect the position and posture of the target object. For example, a gyro sensor, an acceleration sensor, or an optical sensor compatible with laser light or infrared light may be used. While a tool has been described as an example of the manipulated object, this is not a limitation and the object may be a component or product in a manufacturing process. Although the hand-side marker 3 has been described as being placed on the forearm near the wrist, this is not a limitation and the marker may be placed on a part of the hand of the worker U1 that does not interfere with the work motion.

[0176] In the first embodiment, the hand pose is the pose of the marker 4 attached to the forearm near the wrist. As mentioned above, the position and detailed method of attaching the marker are not limited to this. In another example, if a marker is formed on the palm beyond the wrist, the hand pose is obtained as the pose of the marker that reflects the movement of the palm.

[0177] As a modified example, for correction using an error such as that shown in FIG. 15B described above, the control device 100 may, for example, compare the calculated difference ΔX with a set threshold value, and if the difference ΔX is equal to or greater than the threshold value, perform the correction described above, and if the difference ΔX is less than the threshold value, not perform the correction.

[0178] <Embodiment 2> 19 and subsequent figures will be used to explain the robot teaching method and device of embodiment 2. The basic configuration of embodiment 2 etc. is the same as embodiment 1, and the following mainly explains the components of embodiment 2 etc. that are different from embodiment 1.

[0179] The second embodiment differs from the first embodiment in that the teaching data is corrected taking into account limitations imposed by the hand mechanism of the robot when accessing a tool during an operation such as grasping. These limitations are determined based on the relationship between the structure of the hand mechanism and the structure of the tool, such as its shape. More specifically, these limitations include a limit range related to the allowable direction, position, distance, speed, force, etc., within which movement or operation is permitted when the hand mechanism accesses the tool during an operation such as grasping. The limit range may be defined by a reference value, upper or lower limit, etc. Examples of these limitations include grasping only a specific part of the tool, approaching or moving away from the tool from a specific direction within a certain distance range, accessing the tool at a certain speed or less within a certain range, etc.

[0180] The correction in the generation of teaching data in the second embodiment is to generate a teaching pose that enables efficient operation of the robot mechanism so as to satisfy such restrictions, in other words, so as to prioritize the restrictions. The correction in the second embodiment is to set the content of the restrictions as correction data, and when generating teaching data, to generate teaching data by correcting part of the measurement pose by replacement or the like using the correction data.

[0181] Note that the restriction in the second embodiment is a different concept from the range of retroactive restrictions in the first embodiment, and may be referred to as an operation restriction, a motion restriction, an access restriction, etc. for the sake of distinction. Furthermore, the correction taking into account the operation restriction in the second embodiment is a different concept from the correction taking into account errors due to hand movements in the first embodiment. The second embodiment will be described as an example in which, in addition to the function of correcting errors in the first embodiment, a function of correcting the operation restriction is also provided. However, the second embodiment can also be realized as an example in which only the function of correcting errors in the first embodiment is provided, without the function of correcting errors in the first embodiment.

[0182] An example of the configuration of the control device 100 of the robot teaching device 1 of the second embodiment is the same as that shown in FIG. 2, and the main difference is that it includes an operation pose generation unit 112. The operation pose generation unit 112 is, in other words, a teaching pose correction unit. Furthermore, for correction related to the operation restrictions in the second embodiment, correction data, in other words, correction setting information, is set and stored in the correction data storage unit 106 of FIG. 2. This correction data may be set in advance by an administrator U2, or the control device 100 may calculate the correction data. The correction data in the second embodiment includes correction data related to the operation restrictions in addition to the correction data in the first embodiment.

[0183] The operation pose generation unit 112 uses the correction data in the correction data storage unit 106 to generate an operation pose that takes into account the operation restrictions. The operation pose generation unit 112 or the teaching data generation unit 107 corrects a part of the teaching data by replacing or correcting it using the operation pose. The corrected teaching data is stored in the teaching data storage unit 108. Note that the operation pose generation unit 112 in FIG. 2 may be considered as part of the teaching data generation unit 107.

[0184] [Work demonstration] An example of a work demonstration in the second embodiment will be described using the aforementioned FIG. 12. In this example, as shown in FIG. 12, an instructor U1 performs an operation M2 in which a pipette 8 equipped with a marker 3p is grasped with a right hand 5R equipped with a marker 4R. Initially, the pipette 8 is placed in a holder 81 on a workbench 10. The instructor U1 moves the right hand 5R equipped with the marker 4R to perform an operation M2 in which the instructor grasps the pipette 8 on the holder 81, and then moves the right hand 5R while still grasping the pipette 8. The following description focuses on teaching such an operation M2.

[0185] [Pipette grip] 19 is a perspective view showing an example of the configuration of a holder 81 on a workbench 10, the state of a pipette 8 placed in the holder 81, and the state of a marker 3p attached to the pipette 8 via an attachment 82. A pipette 8 and a marker 3p similar to those shown in FIG. 6 are placed in the holder 81.

[0186] In this example, the holder 81 is structured with a flat plate portion 81a placed on the XY plane, which is the upper surface of the workbench 10 and a horizontal plane, a support column portion 81b standing on the flat plate portion 81a in the Z-axis direction, which is the vertical direction, and a support portion 81c at the upper end of the support column portion 81b, which is parallel to the XY plane. The support portion 81c has a shape with one of its four sides cut out in the negative X-axis direction as shown in the figure, so that it can support predetermined locations of the pipette 8 and attachment 82. The long axis of the pipette 8 is positioned within the area of ​​the cutout of the support portion 81c.

[0187] The attachment 82 attached to the top of the pipette 8 has a shape that allows it to be placed on the support portion 81c and to be grasped by the finger mechanism 521 (finger mechanisms 521a and 521b in FIG. 8) of the hand mechanism 6. When the pipette 8 is installed in the holder 81, the attachment 82 of the marker 3p is placed on the support portion 81c of the support portion 81b, so that the weight of the pipette 8, attachment 82, and marker 3p is supported by the holder 81. The marker 3p connected to the attachment 82 is positioned at approximately the same height as the attachment 82, with its rectangular flat plate extending outward from the support portion 81c, for example, on the XZ plane. The marker 3p is also positioned so as not to interfere with the grasping operation.

[0188] When the pipette 8 is set or held in the holder 81, it is placed on the support part 81c by moving it in a direction corresponding to one of the four sides of the XY plane of the support part 81c, for example, from negative to positive on the X axis 1901. Conversely, when the pipette 8 is removed from the holder 81, it is removed from the support part 81c by moving it in a direction corresponding to one of the four sides of the XY plane of the support part 81c, for example, from positive to negative on the X axis 1902.

[0189] In order to grasp and move a pipette 8 mounted on such a holder 81 using the finger mechanism 521 at the tip of the hand mechanism 6R of the robot 2, it is essential that the mechanism be able to grasp the upper part of the test tube 8 on the holder 81, and that the mechanism be operated while maintaining a position and posture that allows access without colliding with the holder 81. These are the operational limitations mentioned above.

[0190] In this example, the attachment 82 and other components are designed in advance so that the left and right portions of the side of the attachment 82 can be grasped by the finger mechanism 521 at the top of the pipette 8. When the tool or mechanism is different, the operation portion and the configuration of the attachment will also differ accordingly, and the operation restrictions will also differ.

[0191] An example of an operation by the hand unit 520 of the hand mechanism 6 of the robot 2 and the associated operation restrictions are as follows: When gripping the pipette 8, the hand mechanism 6 assumes a posture that keeps the finger mechanism 521 horizontal, and as a positional trajectory, it accesses by moving parallel to the direction of the notch in the support portion 81c, in this example, in a direction 1901 from negative to positive on the X-axis, until it reaches a portion on the side of the attachment 82 above the pipette 8 and grips that portion from the left and right. Then, with the pipette 8 gripped by the finger mechanism 521, the hand mechanism 6 pulls out the pipette 8 by moving the finger mechanism 521 parallel to the direction 1902 from positive to negative on the X-axis while maintaining the horizontal position.

[0192] [Operation restrictions and operation pauses] FIG. 20 is an explanatory diagram of a pose in which the finger mechanism 521 (finger mechanisms 521a and 521b in FIG. 8), which is the fingertip of the hand mechanism 6R on the right side of the robot 2, can access the pipette 8 in the holder 81 in FIG. 19, and shows a side view of the XZ plane in the coordinate system Σw. Note that in FIG. 20, for ease of understanding, only the finger mechanism 521 of the hand mechanism 6R is virtually shown by a dashed line to show the pose of the mechanism of the robot 2 that can access the pipette 8. Also, the attachment 82 and the marker 3p are not shown. Furthermore, in the following description, the coordinate system for expressing poses will be the coordinate system Σw, which is the workbench coordinate system, as in the first embodiment, unless otherwise specified.

[0193] 20, there is an operation limit range R20. The operation limit range R20 is a range in three-dimensional space where operation restrictions are required. For example, the operation limit range R20 is a range from position X1 to position X2 in the X-axis direction, a range including both left and right sides of the position where the long axis of the pipette 8 is located (not shown) in the Y-axis direction, and a range from the top surface of the workbench 10 at 0 to a predetermined height Z2 above height Z1 in the Z-axis direction.

[0194] 20, the trajectory of the pose, in other words, the path, of the access operation in direction 1901 in FIG. 19 when the finger mechanism 521 grips the pipette 8 is shown as trajectory 2000. The trajectory of the pull-out operation in direction 1902 is the opposite trajectory to the trajectory 2000 of the operation in direction 1901. The trajectory 2000 is formed by connecting a trajectory 2001 within the operation limit range R20 and a trajectory 2002 outside the operation limit range R20.

[0195] In this example, the hand mechanism 6R of the robot 2 performs the following operation taking into consideration operational limitations in order to access and grasp the pipette 8 in the holder 81 without interfering with the holder 81, i.e., without unnecessary contact. First, outside the operational limit range R20, the hand mechanism 6R moves the finger mechanism 521 to a position p21 corresponding to the boundary of the operational limit range R20 on a basically free trajectory 2002 that allows for changes in posture. The starting point of the trajectory 2002 is not particularly limited. Positions p21 and p22 are shown as the gripping center positions of the finger mechanism 521. Position p21 corresponds to position X2 in the X-axis direction, the position where the long axis of the pipette 8 is located in the Y-axis direction, and a position at a height Z1 from the top surface of the work table 10 in the Z-axis direction.

[0196] Next, within the operation limit range R20, the hand mechanism 6R moves the finger mechanism 521 in parallel along the X-axis by at least a distance Lf in the negative to positive direction while maintaining a horizontal orientation at position p22, which corresponds to the installation and gripping position of the pipette 8 on the holder 81 and is at height Z1 on the Z-axis. This movement is represented by a trajectory 2001 from position X2 to position X1. This prevents interference between the finger mechanism 521 and the holder 81. By moving the finger mechanism 521 in the Y-axis direction at position p22, the hand mechanism 6R grips a predetermined portion of the upper side of the pipette 8 above the support portion 81c.

[0197] Furthermore, in order for the hand mechanism 6R to remove the pipette 8 from the holder 81 while holding it without interfering with the holder 81, i.e., without making unnecessary contact, the hand mechanism 6R performs the following operation with operational limitations taken into consideration. First, within the operational limitation range R20, the hand mechanism 6R maintains a horizontal orientation of the finger mechanism 521 holding the pipette 8 at position p22, which corresponds to the installation and holding position of the pipette 8 on the holder 81, and at height Z1 on the Z axis, while translating the finger mechanism 521 in the X-axis direction from positive to negative by at least a distance Lf. This operation is a trajectory from position X1 to position X2, in the opposite direction to trajectory 2001. This prevents interference between the finger mechanism 521 and the holder 81. After that, the hand mechanism 6R allows the finger mechanism 521 holding the pipette 8 to move essentially freely from position p21, while allowing for changes in orientation. This operation is, for example, a trajectory in the opposite direction to the trajectory 2002, that is, a trajectory in the negative direction on the X axis from the position X2, and the end point of the trajectory is not particularly limited.

[0198] In order to teach the robot 2 to operate the hand mechanism 6 while taking into consideration such operational limitations, in the second embodiment, a pose such as the operational limitation range R20 or a corresponding trajectory 2001 is set in advance as correction data related to the operational limitations. The control device 100 uses such correction data to generate an operational pose corresponding to the operational limitations for the measured pose of the right hand 5R side of the instructor U1 during the work demonstration.

[0199] In this correction data, the pose corresponding to the trajectory 2001 within the operation limit range R20 is a pose that is uniquely determined depending on the grasping operation, the object to be grasped, the grasping mechanism, etc. During correction, such a pose is generated as an operation pose. This operation pose has an approach pose when the mechanism approaches the tool, as in the trajectory 2001, and a departure pose when the mechanism moves away from the tool as a reverse trajectory.

[0200] A pose such as a trajectory 2001 in which the mechanism and tool do not interfere with each other is defined in correspondence with the operation limit range R20. The operation limit range R20 includes, for example, the X-axis direction as the restricted displacement direction, the distance Lf as the restricted distance, and the horizontal posture as the restricted posture. The operation limit range R20 includes an approach pose range, which is the range of the distance Lf in the positive direction of the X-axis in which the mechanism should operate while taking an approach pose, and a departure pose range, which is the range of the distance Lf in the negative direction of the X-axis in which the mechanism should operate while taking a departure pose.

[0201] The control device 100 sets at least one of the above-described operation limit range R20 or the trajectory pose as correction data in the correction data storage unit 106. The control device 100 may calculate the trajectory pose based on the operation limit range R20 set by the user on the screen and set it as correction data. The control device 100 may also set the trajectory pose set by the user on the screen as correction data.

[0202] In this example, during a grasping operation, the trajectory 2001 when approaching and the trajectory when releasing are trajectories with the same posture but displacement in opposite directions, but this is not limited to this, and depending on the object, they may be different trajectories, and different operation restrictions may be set.

[0203] [Correction taking into account operational limitations] Next, using FIG. 21A and other figures, generation of teaching data including corrections that take into account operation limitations in the robot teaching method and other aspects of the second embodiment will be described. Similar to FIG. 15A and other figures, FIGS. 21A, 21B, and 21C show graphs of a detailed example of teaching data generation in the second embodiment. FIG. 21A and other figures show various data during right-hand grasping operation M2, in which instructor U1 grasps the pipette 8 in holder 81 with his right hand 5R, as shown in FIG. 12. For ease of understanding, FIG. 21A and other figures show time-series data of only the position in the X-axis direction of coordinate system Σw, as described above. The period from time 0 to time m corresponds to the movement of right hand 5R to move to and grasp pipette 8, and at time m, a grasping operation instruction is input and detected in response to operation M2.

[0204] Figure 21A shows data D1' of the first measurement pose of the marker 3p on the pipette 8 side, data D2' of the second measurement pose of the marker 4R on the right hand 5R side of the operator U1, data D11' converted from data D1' to a pose at the grip center position, and data D12' converted from data D2' to a coordinate-converted pose at the grip center position.

[0205] FIG. 21B shows data D20', data D21', and data D200. Data D20' is a teaching pose generated by selecting data D12' based on data D11' and data D12', similar to data D20 in FIG. 15. Data D21' is a teaching pose generated by correction from data D20'. The correction from data D20' to data D21' is a correction calculation using subtraction of difference ΔX from time 0 to m, taking into account errors in hand movement, similar to the first embodiment. FIG. 21B also includes data D30' of a teaching instruction ("start") at time 0, data D31' of an operation instruction ("close") at time m, and data D32' of a teaching instruction ("end") at time n.

[0206] 21B, the range of positions corresponding to the distance Lf in the X-axis direction for the approach pose range of the operation limited range R20 in FIG. 20 is indicated by range 2101. Position Xr1 corresponds to position X2 in FIG. 20, and position Xr2 corresponds to position X1. Times Tra to Trb indicate the times when the positions in the X-axis direction for data D21' and data D20' fall within range 2101, which corresponds to the approach pose range of the distance Lf for the operation limited range R20 in FIG. 20. Time Tra corresponds to the time when the position for data D21' exceeds position Xr1, and time Trb corresponds to the time when the position for data D20' falls below position Xr1. The position for data D21' reaches vertex position Xr2 at time m, and is connected to the position of uncorrected data D20' from time m onwards.

[0207] After calculating a teaching pose such as data D21', the teaching data generation unit 107 generates an operation pose corresponding to the operation limit range R20 by referring to the correction data stored in the correction data storage unit 106 through processing by the operation pose generation unit 112 in Fig. 2. The operation pose generation unit 112 generates an approach pose such as a trajectory 2001 corresponding to the operation limit range R20. Then, the teaching data generation unit 107 uses the operation pose to make corrections related to the operation limits.

[0208] In Fig. 21B, the approach pose is represented by data Da of a straight line from time ta to time m. The operation pose generation unit 112 calculates the approach pose data Da so that at time m of the grip operation M2, the position Xr2 and posture of the pose match those of data D11' and data D21'. Accordingly, the start time ta of the approach pose data Da is also calculated. Then, the teaching data generation unit 107 replaces a part of data D21' corresponding to times ta to m with the approach pose.

[0209] The data Db of the departure pose after time m of the gripping operation M2 can be calculated in a similar manner. The operation pose generation unit 112 generates a departure pose that corresponds to the operation limit range R20 in FIG. 20, such as a reverse trajectory of the trajectory 2001. In FIG. 21, the departure pose is represented by data Db of a straight line from time m to time tb. The operation pose generation unit 112 calculates the data Db of the departure pose at time m of the gripping operation M2 so that the data Db matches the position Xr2 and posture of the pose that are the same as those of data D11' and data D21'. Accordingly, the end time tb of the data Da of the departure pose is also calculated. Then, the teaching data generation unit 107 replaces a portion of data D20' corresponding to times m to tb with the departure pose.

[0210] The data D200 (Da, Db) corrected by replacement using the approach pose and the departure pose, which are the operation poses, becomes part of the data D40' in FIG. 21C. The data D40' is divided into data D41' including data Da of the approach pose up to time m and data D41' including data Db of the departure pose after time m.2 and the data (D30', D31', D32') of the teaching instructions and operation instructions are also associated with it. In addition, in the data D41' up to time m, the data from time Tra to time ta before correction is corrected to data Dc which remains unchanged at position Xr1. In the data D42' after time m, the data from time tb to time Trb before correction is corrected to data Dd which remains unchanged at position Xr1.

[0211] As described above, in the second embodiment, first, the data D21' and D20 of the teaching pose are generated as in the first embodiment, and then the data Da and Db of the operation pose are generated taking into consideration the operation limit range R20. As part of the data D21' and D20 of the teaching pose, the range from time Tra to time Trb around time m of the grasping operation is corrected so as to be replaced with the operation pose. This results in the teaching data D40' including the corrected data D200.

[0212] [Variations] As a modification of the second embodiment, a more detailed processing example regarding correction taking the above-mentioned operation restrictions into consideration will be described. In the correction processing example in FIG. 21A etc., for the operation pose data D200 (Da, Db), both the approach pose and the departure pose are generated as straight lines with a constant slope, in other words, poses that displace at a constant speed, and a case is shown in which these poses are replaced as they are. In this case, the start time ta and the end time tb of the operation pose are determined according to the slope of the straight line, and the times before and after these are corrected as straight lines that do not displace, as in the case of data Dc and Dd. The slope and speed of the above-mentioned straight lines are also determined according to the operation restrictions. The correction processing example is not limited to this, and various processing examples are possible.

[0213] FIG. 22A shows a modified example, Modification Example 2A. The underlying measurement data and the like are the same as those in FIG. 21A. In Modification Example 2A, the operation pose generation unit 112 generates approach pose data Da' as a straight line connecting time Tra and time m based on the time Tra and time Trb corresponding to the distance Lf of the operation limit range R20, with the position of time m of the gripping operation as the apex, and generates departure pose data Db' as a straight line connecting time m and time Trb. The data Da' and data Db' have a gentler slope than the aforementioned data Da and data Db. Then, by replacing the taught pose data D21' and data D20' with these operation poses D200a (Da', Db'), corrected taught data as shown in the figure is generated.

[0214] FIG. 22B shows another modified example, Modification 2B. The underlying measurement data, etc., are assumed to fluctuate and shake more than in the case of FIG. 21A. In addition, in this example, the position of the apex during the grasping operation continues from time m to time m'. In other words, this shows a case where the instructor U1 maintains the state of holding the pipette 8 in his right hand 5R during operation M2 for a certain period of time. In addition, in this example, because the displacement during release was large, time tb is later than time Trb.

[0215] The data D21' and data D20' are teaching poses before correction. The operation pose generation unit 112 generates an operation pose D200b (Da, Db) for the data D21' and data D20' in the same manner as the processing example of FIG. 21A etc. of the second embodiment. The operation pose D200b has data Da of an approach pose from time ta to time m and data Db of a departure pose from time m' to time tb. In this modification 2B, the poses before and after the operation pose data D200b are left as the original data D21' and data D20' as much as possible and are not corrected. In this example, the period from time Tra to time ta includes, for example, data De connecting position Xr4 at time tc and position Xr1 at time ta as the original data D21'. In addition, the period around time Trb is corrected so that data Db takes priority until time tb, and is connected to the position of data D20' after time tb.

[0216] Correction as in Modification 2B may also be performed. However, if a portion where the amount of pose variation is large, such as a portion including position Xr4 at time tc, is of concern, further correction as follows may also be performed.

[0217] Fig. 22C shows another modified example, modified example 2C. In this modified example 2C, after performing a correction using an operation pause that takes into account operation restrictions, similar to Fig. 22B, an additional correction is performed so that the connection between the replaced portion using the operation pause and the portion before and after it becomes smoother; in other words, so that the amount of change in the pause is made as small as possible. The concept of this correction is the same as the correction in the modified example of embodiment 1, and various processing examples can be applied in the same way.

[0218] In the example of FIG. 22C , for example, assume that the displacement A4 in the X-axis direction between time tc and time ta is greater than or equal to a threshold. In this case, the taught data generation unit 107 corrects the taught pose using methods such as statistical processing and noise removal to reduce this displacement A4. In one example of using statistical processing, the average value of position Xr4 at time tc and position Xr1 at time ta is calculated, and the position at time tc with the largest change is corrected, e.g., replaced, by this average value. In this case, the corrected taught pose becomes data Df. In another example, position Xr1 at time ta may also be replaced by this average value, and the slope of the straight line of data Da may be made gentler. In one example of using noise removal, the position at time tc with the largest change is considered noise and corrected to a straight line without displacement so as to be the same as position Xr1 at times Tra and ta. In this case, the corrected taught pose becomes data Dg.

[0219] The various correction processing examples described above can also be applied to the departure operation after time m.

[0220] As another modification, the time during which the vertex position is maintained without displacement, such as times m to m', may be corrected to be shortened to 0 or within a predetermined time. The time of the vertex may be determined as part of the correction data, and correction may be performed to ensure that the time is that time.

[0221] [GUI screen] FIG. 23 shows an example of a screen display including a GUI that the robot teaching device 1 provides to users, such as a worker U1 or a manager U2, in the second embodiment. The screen of FIG. 23 is displayed on the display screen of the display device of the input / output device 120 (the output device 1006 in FIG. 3) based on processing by the teaching instruction detection unit 105 of the control device 100 in FIG. 2, for example. The screen of FIG. 23 shows an example configured as a workspace editing screen. This screen enables the setting and editing of the workspace. This screen includes a workbench setting screen section 2301, a correction data setting screen section 2302, and a correction data editing screen section 2303.

[0222] The user can set correction data in advance in the control device 100 using the workspace editing screen of Fig. 23, thereby executing the processing by the robot teaching method of the second embodiment. The user, a worker U1 or a manager U2, can set the workspace, necessary correction data, etc. on the screen of Fig. 23, and can check the data and information. The workspace corresponds to the space on the workbench 10 as shown in Fig. 1, and a coordinate system Σw can also be set. The user can set each workspace on the screen of Fig. 23, save it with a name, and can read out and check the saved settings, etc.

[0223] The workbench setting screen section 2301 has a GUI for determining the type and arrangement of objects to be placed on the workbench 10 as shown in FIG. 1. The objects are tools to be operated and related objects such as holders for the tools. In this example, the workbench setting screen section 2301 displays a holder 71 for a test tube 7, a holder 81 for a pipette 8, and the like in a perspective view of the workbench 10. The user can set the position and orientation of the objects by, for example, manipulating the cursor of a mouse, while changing the view of the workbench 10. The workbench setting screen section 2301 may also be configured to allow the position and orientation of the camera 20 as shown in FIG. 4 to be set.

[0224] The correction data setting screen section 2302 has a GUI for setting correction data such as the trajectory of an operation pose, such as an approach pose or a departure pose, as shown in FIG. 20, or the operation limit range R20, for an object (e.g., FIG. 19) including an object to be operated that can be set in the workbench setting screen section 1301. In this example, a side view of the XZ plane of the coordinate system Σw is displayed in the area 2302b at the bottom of the correction data setting screen section 2302, as in FIG. 20. The correction data setting screen section 2302 allows the setting and confirmation of the name and view of the object, the limit distance and force on each axis of the coordinate system Σw for the operation limit range R20, the file of the operation pose trajectory, and the like. The trajectory of the operation pose may be set by the user operating the mouse cursor, for example, in the area 2302b or the correction data editing screen section 2303. The set trajectory of the operation pose may be displayed in the area 2302b or the correction data editing screen section 2303 for confirmation. The correction data setting screen section 2302 may be configured to allow the instructor U1 to set the initial positions of the hand 5 and hand mechanism 6, in other words, the initial positions of the pose trajectory.

[0225] The correction data editing screen section 2303 has a GUI that enables the user to confirm and edit the details of the correction data that can be set in the correction data setting screen section 2302 as a motion, such as a pose, of the tip of the hand mechanism 6 of the target robot 2, for example, the finger mechanism 521 of the hand unit 520 in FIG. 8. In this example, the correction data editing screen section 2303 displays, as an animation or the like, the motion of grasping or releasing the pipette 8 by the finger mechanism 521 of the hand unit 520 of the right hand mechanism 6R, in accordance with the range R20 and the trajectory of the operation pose set in the correction data setting screen section 2302. The motion can also be displayed as a still image by specifying a time in the time series.

[0226] Although not shown, if there are multiple candidate hand mechanisms of the robot 2, the target mechanism may be set on the screen. Furthermore, although not shown, the contents of the teaching data generated by the system of the second embodiment can be displayed and confirmed on a screen similar to that of Fig. 18.

[0227] Although not shown, when a judgment threshold or the like is used for correction taking into account the above-mentioned operation limitations or additional correction, setting information for the threshold or the like can also be displayed on the screen, allowing the user to check and set the setting information. Furthermore, multiple correction methods, such as those in the above-mentioned modified examples, may be implemented, and in that case, the user may be able to select and set the correction method to be applied from the multiple correction methods on the screen.

[0228] [Effects of the second embodiment] As described above, in addition to the effects of embodiment 1, embodiment 2 provides the following: In embodiment 2, when different hand poses can be assumed during work operations by the human instructor U1 and the robot 2 depending on the shape of the gripping mechanism of the robot 2 and the shape of the object to be grasped, an appropriate teaching pose can be more easily generated from a demonstration of normal work operations by making corrections that take into account operational limitations.

[0229] Although the embodiments of the present disclosure have been specifically described above, they are not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the present disclosure. Except for essential components, components can be added, deleted, or replaced in each embodiment. Unless otherwise specified, each component can be singular or plural. A combination of each embodiment and its variations is also possible.

[0230] [Note] A robot teaching method according to an embodiment may be as follows: The robot teaching method according to an embodiment includes a step of calculating a teaching pose expressed in the work table coordinate system by coordinate transformation based on the relationship between the work table coordinate system, the coordinate system of the pose of a first marker attached to the object to be operated, the coordinate system of the pose of a second marker attached to the instructor's hand, and the coordinate system of the robot's hand mechanism. [Explanation of symbols]

[0231] 1...robot teaching device (robot teaching system), 2...robot, 3 (3a, 3b, 3p)...marker (marker plate, hand-side marker, first marker), 4 (4L, 4R)...marker (marker plate, tool-side marker, second marker), 5 (5L, 5R)...hand, 6 (6L, 6R)...hand mechanism, 7 (7a, 7b)...test tube (first tool), 8...pipette (second tool, micropipette), U1...instructor (worker, first user), U2...administrator (second user), 9...end effector, 10...workbench, 20 (20a, 20b, 20c, 20d)...camera, 100...control device (robot teaching control device), D1, D2, D11, D12, D20, D21, D30, D31, D32, D40, D41, D42...data.

Claims

1. A robot teaching method for teaching a robot to generate robot motion data including a joint displacement sequence as a motion of a hand mechanism of the robot corresponding to a work motion, based on measurement of the work motion including manipulation of an object to be manipulated by a hand of a teacher, comprising: The steps executed by the computer system include: acquiring a first measured pose by measuring a time-series pose consisting of the position and posture of the object to be operated during the work operation; acquiring a second measured pose by measuring a time-series pose consisting of the hand position and posture of the instructor during the task movement; detecting the operation of the object to be operated by the instructor; generating a teaching pose for generating the robot motion data based on the first measurement pose, the second measurement pose, and the detected operation; A robot teaching method comprising:

2. The robot teaching method according to claim 1, The step of generating a teaching pose includes: acquiring difference data as an error between the first measured pose of the object and the second measured pose of the hand; generating the teaching pose by correcting measurement data in the vicinity of the operation in the first measurement pose and the second measurement pose using the difference data so as to reduce the error; A robot teaching method comprising:

3. 3. The robot teaching method according to claim 1, The step of generating a teaching pose includes: acquiring, as the set correction data, correction data representing an operation restriction regarding an operation pose when the hand mechanism of the robot performs the operation on the operated object; generating the teaching pose by correcting measurement data in the vicinity of the operation in the first measurement pose and the second measurement pose using the correction data so as to satisfy the operation restriction; A robot teaching method comprising:

4. The robot teaching method according to claim 1, the step of acquiring the first measured pose is a step of measuring a time-series pose consisting of a position and an orientation of a first marker installed on the object to be operated, the step of acquiring the second measured pose is a step of measuring a time-series pose consisting of a position and a posture of a second marker placed on the hand of the instructor, the first marker and the second marker are marker plates each having a unique arrangement pattern of a plurality of reflective markers, The first marker and the second marker are measured by a camera. Robot teaching method.

5. The robot teaching method according to claim 1, the step of detecting the operation of the operated object by the instructor is a step of detecting an operation instruction representing the operation input by the instructor using an instruction input device, or a step of detecting the operation by automatically determining the operation based on the first measurement pose and the second measurement pose. Robot teaching method.

6. 3. The robot teaching method according to claim 2, The step of generating a teaching pose includes: acquiring the difference data at a time corresponding to the timing of the detected operation; a step of going back from a time corresponding to the timing of the detected operation to a past time within a time range determined in accordance with a set physical quantity, and performing a correction to reduce the error within the time range; A robot teaching method comprising:

7. 3. The robot teaching method according to claim 2, The step of generating the teaching pose includes a step of additionally correcting a portion of the data that is to satisfy a predetermined value by statistical processing or noise removal processing when a change amount between the data obtained by performing correction to reduce the error and the data obtained before and after the correction does not satisfy a predetermined value. Robot teaching method.

8. 7. The robot teaching method according to claim 6, the step of performing correction to reduce the error within the time range is a step of correcting the uncorrected measurement data within the time range by statistical processing or noise removal processing. Robot teaching method.

9. 4. The robot teaching method according to claim 3, the operation restriction is determined in accordance with a structure of the operated object and a structure of the hand mechanism of the robot for operating the operated object, and includes a restriction range regarding a direction and a distance of movement of the hand mechanism on each axis of a spatial coordinate system in the vicinity of the operated object; Robot teaching method.

10. 4. The robot teaching method according to claim 3, the operation restriction includes a restriction range for at least one of a pose when the hand mechanism of the robot approaches the object to be operated on the work table and performs the operation, and a pose when the hand mechanism of the robot leaves the object to be operated on the work table after performing the operation, Robot teaching method.

11. 4. The robot teaching method according to claim 3, the step of correcting the measurement data so as to satisfy the operation restriction includes a step of replacing a part of the measurement data in the vicinity of the operation in the first measurement pose and the second measurement pose with the operation pose. Robot teaching method.

12. 4. The robot teaching method according to claim 3, The step of correcting to satisfy the operation restriction includes: acquiring a time at which the first measurement pose and the second measurement pose reach a boundary of the range of the operation limit; a step of correcting the operation pose so as to connect measurement data at times corresponding to the timings at which the operation was detected in the first measurement pose and the second measurement pose with measurement data at times on the boundary of the operation limit range; A robot teaching method comprising:

13. 4. The robot teaching method according to claim 3, The step of correcting to satisfy the operation restriction includes: acquiring a time at which the first measurement pose and the second measurement pose reach a boundary of the range of the operation limit; a step of additionally correcting portions of measurement data before and after the portions corrected by the operation pose in the first measurement pose and the second measurement pose by statistical processing or noise removal processing; A robot teaching method comprising:

14. 5. The robot teaching method according to claim 4, the first marker includes a first attachment for attaching the first marker to the object to be operated and maintaining a pose relationship with the object to be operated; the second marker includes a second attachment for attaching the second marker to the instructor's hand and maintaining a pose relationship with the hand; The operation of the operated object by the hand mechanism of the robot is an operation on a location of the first attachment. Robot teaching method.

15. A robot teaching device that teaches a robot to generate robot motion data including a joint displacement sequence as a motion of a hand mechanism of a robot corresponding to a work motion, based on measurement of the work motion including manipulation of an object to be manipulated by a hand of a teacher, the device comprising: A computer system is provided. The computer system includes: acquiring a first measured pose by measuring a time-series pose consisting of the position and posture of the object to be operated during the work operation; acquiring a second measured pose by measuring a time-series pose consisting of the hand position and posture of the instructor during the task movement; detecting the operation of the object to be operated by the instructor; generating a teaching pose for generating the robot motion data based on the first measurement pose, the second measurement pose, and the detected operation; Robot teaching device.

Images