Robot teaching system and robot teaching method

Abstract

To support teaching of a teaching point at a position where direct teaching is difficult by using a finger of a worker or a marker pen.SOLUTION: A robot teaching system stores teaching point data corresponding to a teaching point used in display of teaching data of a robot, acquires a relative positional relation between actual environment and a display device for displaying an image so that the image is superposed on an image of the actual environment or the actual environment itself, generates a display image displaying the teaching point so that the teaching point has a predetermined positional relation relative to the display device, outputs the display image to the display device, detects aerial operation of a worker relative to workpiece, and, when aerial operation indicating a predetermined direction with respect to the actual environment is executed, generates a display image displaying the teaching point at an intersection between the predetermined direction and the surface of the workpiece.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a robot teaching system and a robot teaching method. [Background technology]

[0002] Patent Document 1 discloses a method for programming a robot to perform an operation based on human demonstration. This method involves demonstrating an operation on a workpiece using a human hand, analyzing a camera image of the hand demonstrating the operation on the workpiece using a computer to create demonstration data, analyzing a camera image of a new workpiece to determine an initial position and orientation of the new workpiece, generating a robot motion command based on the demonstration data and the initial position and orientation of the new workpiece to cause the robot to perform the operation on the new workpiece, and having the robot perform the operation on the new workpiece. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-167060 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure aims to provide a robot teaching system and a robot teaching method that use an operator's fingers, a marker pen, or the like to assist in teaching a teaching point at a position that is difficult to teach directly. [Means for solving the problem]

[0005] The present disclosure provides a robot teaching system comprising: a teaching data memory unit that stores teaching data for a robot present in a real environment; a teaching point memory unit that stores teaching point data corresponding to teaching points used to display the teaching data; a display device that is configured to be wearable by a worker and that displays an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit that acquires a relative positional relationship between the real environment and the display device; an image generation unit that generates a display image for displaying the teaching point so that it is in a predetermined positional relationship with respect to the display device based on the relative positional relationship and the teaching point data; an output unit that outputs the display image to the display device; and a detection unit that detects an aerial operation, which is an operation performed by a worker in the air away from the display device on a workpiece present in the real environment, when an operation indicating a predetermined direction with respect to the real environment is performed as the aerial operation, the image generation unit generates the display image for displaying the teaching point at the position of an intersection of a virtual axis along the predetermined direction and the surface of the workpiece.

[0006] the display unit configured to acquire a relative positional relationship between the real environment and the display device; an image generation unit configured to generate a display image for displaying the 3D model and the teaching point so as to have a predetermined positional relationship with respect to the display device based on the relative positional relationship, the 3D model, and the teaching point data; an output unit configured to output the display image to the display device; and a detection unit configured to detect an aerial operation, which is an operation performed by a worker in the air away from the display device on the 3D model displayed on the display device, when an operation indicating a predetermined direction with respect to the real environment is performed as the aerial operation, the image generation unit generates the display image for displaying the teaching point at a position of an intersection of a virtual axis along the predetermined direction and the surface of the workpiece.

[0007] The present disclosure also provides a robot teaching method performed by a system including at least one computer, which stores teaching data of a robot present in a real environment and teaching point data corresponding to teaching points used to display the teaching data, acquires a relative positional relationship between the real environment and a display device configured to be wearable by a worker and configured to display an image of the real environment or the real environment itself by superimposing it on the image, generates a display image for displaying the teaching point so that it is in a predetermined positional relationship with the display device based on the relative positional relationship and the teaching point data, and outputs the display image to the display device, detects an aerial operation, which is an operation performed by a worker in the air away from the display device on a workpiece present in the real environment, and when an operation indicating a predetermined direction with respect to the real environment is performed as the aerial operation, generates the display image for displaying the teaching point at a position where a virtual axis along the predetermined direction intersects with the surface of the workpiece, and outputs the display image to the display device.

[0008] The present disclosure also provides a robot teaching method performed by a system including at least one computer, the system storing a three-dimensional model of a workpiece existing in a real environment, teaching data for a robot existing in the real environment, and teaching point data corresponding to teaching points used to display the teaching data, acquiring a relative positional relationship between the real environment and a display device configured to be wearable by a worker and configured to display an image of the real environment or the real environment itself by superimposing it on the image, generating a display image for displaying the three-dimensional model and the teaching point so that the three-dimensional model and the teaching point are in a predetermined positional relationship with the display device based on the relative positional relationship, the three-dimensional model, and the teaching point data, and outputting the display image to the display device, detecting an aerial operation, which is an operation performed by a worker in the air away from the display device on the three-dimensional model existing in the real environment, and when an operation indicating a predetermined direction with respect to the real environment is performed as the aerial operation, generating the display image for displaying the teaching point at a position of an intersection of a virtual axis along the predetermined direction and the surface of the workpiece, and outputting the display image to the display device. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to use an operator's finger, a marker pen, or the like to assist in teaching a teaching point at a position where direct teaching is difficult. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of a welding teaching system according to an embodiment. [Figure 2] FIG. 1 shows an example of the internal configuration of an MR device and a processing device. [Figure 3] A diagram illustrating an example of a fingertip teaching method. [Figure 4] 10 is a flowchart showing an example of a procedure for teaching a teaching position using a fingertip teaching method for an MR device according to an embodiment. [Figure 5] FIG. 1 is a diagram illustrating an example of a remote point teaching method. [Figure 6] 10A and 10B are diagrams illustrating other teaching methods and other correction methods for teaching positions; [Figure 7] A diagram explaining teaching example 1 of teaching posture [Figure 8] A diagram explaining teaching example 2 of teaching posture [Figure 9] A diagram explaining teaching example 3 of teaching posture [Figure 10] 10 is a flowchart showing an example of an operation procedure of an MR device according to an embodiment. [Figure 11] 1 is a flowchart showing an example of a teaching procedure using a fingertip teaching method for an MR device according to an embodiment. [Figure 12] 1 is a flowchart showing an example of a teaching procedure using a remote point display method for an MR device according to an embodiment. [Figure 13] 10 is a flowchart showing an example of a procedure for teaching a posture of an MR device according to an embodiment. [Figure 14] 10 is a flowchart showing an example of a procedure for calculating the orientation of an MR device according to an embodiment. [Figure 15] FIG. 10 is a diagram showing an example of a mixed reality space visually recognized by a worker. DETAILED DESCRIPTION OF THE INVENTION

[0011] (Background to this disclosure) Conventionally, in a teaching operation for teaching a welding robot welding operations during welding as in Patent Document 1, there has been a method of teaching a welding position using a worker's finger or a marker pen, etc., rather than a teaching tool (hereinafter referred to as a "teaching tool") that resembles a welding torch equipped on the welding robot. However, this teaching method has had a problem in that, particularly when it is difficult to point to the teaching position from nearby due to the height of the workpiece to be welded, the shape of the workpiece, the teaching environment, etc., the distance between the worker's finger or marker pen and the teaching position becomes large, making it impossible to teach the teaching point.

[0012] Therefore, in the following embodiments, a robot teaching system and a robot teaching method will be described that use an operator's finger or a marker pen to assist in teaching a teaching point at a position where direct teaching is difficult.

[0013] Hereinafter, with reference to the drawings as appropriate, detailed descriptions of embodiments specifically disclosing a robot teaching system and a robot teaching method according to the present disclosure will be provided. However, unnecessary detailed descriptions may be omitted. For example, detailed descriptions of well-known matters or redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure and are not intended to limit the subject matter recited in the claims.

[0014] <Welding teaching system overview> First, a welding teaching system 100 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of welding teaching system 100 according to an embodiment. Note that welding teaching system 100 shown in Fig. 1 is an example and is not limited to this.

[0015] Welding teaching system 100 receives teaching of the position and posture of a teaching point for teaching welding robot RB a welding operation to be performed by the worker's hand HND or the like. Welding teaching system 100 executes teaching of the welding operation by transmitting information on the position and posture of the taught teaching point to a robot controller that controls welding robot RB.

[0016] In the present disclosure, the workpiece Wk used to teach the teaching points may be an actual workpiece or a virtual workpiece constructed based on 3D model data, etc. Furthermore, the welding operation referred to here may include not only the welding operation for welding the workpiece Wk, but also an approach operation in which the welding robot RB (welding torch TC) approaches the workpiece Wk, an avoidance operation in which the welding robot RB (welding torch TC) avoids an obstacle, a free-running operation in which the welding torch TC free-runs, or a leaving operation in which the welding robot RB (welding torch TC) moves away from the workpiece Wk.

[0017] The welding teaching system 100 includes a workpiece Wk, an MR device DV, and a processing device P1. If the MR device DV can realize the function of the processing device P1, the processing device P1 may be omitted.

[0018] In the following description of the present disclosure, an example will be described in which teaching points are taught using the operator's hand HND, but a tool such as a marker pen may also be used. Also, an example will be described in which the workpiece Wk in the description of the present disclosure is not a virtual workpiece but an actual workpiece.

[0019] The MR device DV is a so-called head-mounted display, and is connected to the processing device P1 so as to be able to communicate data with it. The MR device DV is worn on the head of the worker, and forms a virtual space by superimposing images showing the results of the worker's operations (e.g., virtual teaching points) and images of virtual production equipment (e.g., virtual workpieces, virtual welding robots VRB, virtual welding torches VTC, or virtual jigs) on a captured image of a real space corresponding to the worker's field of vision, and displays the virtual space on the display unit 13, thereby visualizing the virtual space for the worker.

[0020] The MR device DV detects the worker's hands HND or production equipment (for example, workpiece Wk, welding robot RB, or jig, etc.) from the captured image captured by the camera 15. The MR device DV acquires information about the welding robot RB transmitted from the processing device P1. Note that the information about the welding robot RB here includes the world coordinate system or the robot coordinate system of the welding robot RB relative to the workpiece Wk, etc., the coordinate system of the welding torch TC, a 3D model of the welding robot RB, etc.

[0021] The MR device DV receives, based on the operator's operation, a registration operation for the position (three-dimensional) of a teaching point and the posture (three-dimensional) of a welding torch TC that welds the teaching point. Based on the registered position and posture of the teaching point, the MR device DV superimposes the teaching point and virtual production equipment (for example, a virtual welding robot VRB) on an image of the real world captured by the camera 15, thereby generating and displaying a simulation image SC31 (see FIG. 15) that teaches welding operations for a workpiece Wk in the real world.

[0022] The processing device P1 is connected to the MR device DV and the robot controller so that data can be communicated between them. The processing device P1 executes the teaching process for each teaching point by transmitting information on the position (three-dimensional) and posture (three-dimensional) of the teaching point transmitted from the MR device DV to the robot controller that controls and drives the welding robot in the real world.

[0023] Next, an example of the internal configuration of the MR device DV and the processing device P1 will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the internal configuration of the MR device DV and the processing device P1.

[0024] The MR device DV includes a communication unit 10, a processor 11, a memory 12, a display unit 13, a depth sensor 14, and a camera 15.

[0025] The communication unit 10 is connected to the processing device P1 so as to be able to communicate wirelessly or via wires, and transmits and receives data. The communication unit 10 outputs various data transmitted from the processing device P1 to the processor 11. The communication unit 10 transmits various data output from the processor 11 to the processing device P1. The wireless communication here refers to communication via a wireless local area network (LAN) such as Wi-Fi (registered trademark). When the processing device P1 is omitted from the welding teaching system 100, the communication unit 10 is connected to a robot controller so as to be able to communicate data.

[0026] Processor 11 is configured using, for example, a central processing unit (hereinafter referred to as "CPU") or a field programmable gate array (hereinafter referred to as "FPGA"), and performs various processes and controls in cooperation with memory 12. Specifically, processor 11 references programs and data stored in memory 12 and executes the programs to realize various functions, such as a function to accept teaching of teaching points, a function to generate teaching information to be taught to welding robot RB, and a function to generate simulation image SC31 (see FIG. 15). When processing device P1 is omitted from welding teaching system 100, processor 11 is configured to be able to realize functions similar to those of processor 21 of processing device P1.

[0027] The memory 12 includes, for example, a random access memory (hereinafter referred to as "RAM") as a work memory used when executing each process of the processor 11, and a read only memory (hereinafter referred to as "ROM") that stores programs and data that define the operations of the processor 11. The RAM temporarily stores data or information generated or acquired by the processor 11. The ROM stores programs that define the operations of the processor 11.

[0028] The display unit 13 is configured using, for example, a Liquid Crystal Display (LCD) or an organic electroluminescence (EL). The display unit 13 displays an image of the real world itself or a virtual space in which virtual production equipment is superimposed on the real world. The display unit 13 realizes mixed reality by displaying, for example, an image of the virtual space in which virtual production equipment generated by the processor 11 is superimposed on an image of the real world captured by the camera 15, an image of the taught teaching point, or a virtual operation menu VBT (see FIG. 15) including virtual operation buttons that can accept operator operations.

[0029] The depth sensor 14 is a sensor that measures the distance between the MR device DV and an object in the real world and recognizes the three-dimensional shape of the object in the real world (for example, the worker's hand HND or fingers FNG, the workpiece Wk, or a jig, etc.). The depth sensor 14 outputs the recognition result to the processor 11. Based on the recognition result output from the depth sensor 14, the processor 11 recognizes the position, shape, or posture of the worker's hand HND or fingers FNG in the air, or recognizes the movement of the worker's hand HND or fingers FNG in the air, and accepts the worker's operation in the air (hereinafter referred to as "air operation").

[0030] The camera 15 captures an image of an area (real world) corresponding to the field of view of the worker wearing the MR device DV. The camera 15 outputs the captured image to the processor 11.

[0031] The processing device P1 includes a communication unit 20, a processor 21, and a memory 22.

[0032] The communication unit 20 is connected to the MR device DV and the robot controller so as to be able to communicate wirelessly or via wires, and transmits and receives data. The communication unit 20 outputs various data transmitted from the MR device DV to the processor 21. The communication unit 20 transmits various data output from the processor 21 to the MR device DV or the robot controller. The wireless communication here refers to communication via a wireless LAN such as Wi-Fi (registered trademark).

[0033] Processor 21 is configured using, for example, a CPU or FPGA, and performs various processes and controls in cooperation with memory 22. Specifically, processor 21 references the programs and data stored in memory 22 and executes the programs to realize various functions for generating a welding teaching program.

[0034] The memory 22 includes, for example, a RAM as a work memory used when executing each process of the processor 21, and a ROM for storing programs and data that define each operation of the processor 21. The RAM temporarily stores data or information generated or acquired by the processor 21. The ROM has written therein programs that define the operation of the processor 21. The memory 22 includes a teaching information recording unit 221 and a work information recording unit 222. The teaching information recording unit 221 and the work information recording unit 222 may be recorded in the memory 12 of the MR device DV. The memory 22 records a 3D model of the welding robot RB, information about the robot coordinate system of the welding robot RB, etc.

[0035] The teaching information recording unit 221 records information on the positions and orientations of a plurality of teaching points transmitted from the MR device DV for each workpiece Wk.

[0036] The workpiece information recording unit 222 records a 3D model of the workpiece Wk. The 3D model of the workpiece Wk may be generated based on the external shape of the workpiece Wk detected by the depth sensor 14 of the MR device DV.

[0037] <Method of teaching positions using fingertip teaching> A method for teaching a teaching position using the fingertip teaching method will be described. The fingertip teaching method here refers to a method for teaching a teaching position of a teaching point based on the direction pointed by the fingertip of the worker's finger FNG. The fingertip teaching method is performed when the distance between the fingertip of the worker's finger FNG and the workpiece Wk1 is short (for example, 2 cm or 5 cm) and it is assumed that the welding quality will not be degraded due to the accuracy of the teaching position taught by the worker's finger FNG. Note that the above-mentioned distance is a distance at which it is assumed that the welding quality will not be degraded due to the accuracy of the teaching position taught by the worker's finger FNG, and any distance may be set based on the workpiece Wk1, the required welding quality, etc.

[0038] Next, an example of teaching a teaching position using the fingertip teaching method will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining an example of the fingertip teaching method.

[0039] 3, the worker points the index finger FNG of his hand HND toward the workpiece Wk1 to teach a teaching point. The MR device DV accepts the worker's teaching operation of the teaching point based on the captured image captured by the camera 15 and the object recognized by the depth sensor 14 (here, the workpiece Wk1 and the worker's hand HND and finger FNG).

[0040] Specifically, when the MR device DV starts accepting processing for a teaching operation of a teaching point by the worker, it detects the fingertip position Pt0 (three-dimensional position) of the recognized worker's finger FNG (index finger) and the direction the finger FNG points (i.e., the extension direction of the finger FNG). The MR device DV calculates the intersection position where the direction the finger FNG points intersects with the mesh data (here, the workpiece Wk1), and registers (records) this as the teaching position Pt11 of the teaching point. The method for calculating the teaching position Pt11 will be described in detail with reference to FIG. 4.

[0041] Furthermore, the MR device DV generates a teaching image SC11 in which an image of a point (circle indicating the teaching position Pt11) indicating the position pointed to by the worker's fingertip position Pt0 is superimposed on the intersection position on the mesh data, and displays the generated teaching image SC11 on the display unit 13. In this way, the MR device DV supports the teaching work of the teaching position by visualizing to the worker the teaching position being taught based on the worker's operation.

[0042] Next, an example of a teaching process of a teaching position by the fingertip teaching method will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of a teaching procedure of a teaching position by the fingertip teaching method of the MR device DV in the embodiment.

[0043] The MR device DV determines whether or not there is an input operation to register the teaching position of the teaching point based on the captured image captured by the camera 15 and the recognition result of the worker's hand HND recognized by the depth sensor 14 (St11).

[0044] If the MR device DV determines in step St11 that an input operation to register the teaching position of the teaching point has been performed (St11, YES), it measures the distance between the workpiece Wk1 and the worker's finger FNG (fingertip position Pt0) based on the recognition result obtained by the depth sensor 14. The MR device DV determines whether the distance between the workpiece Wk1 and the worker's finger FNG (fingertip position Pt0 shown in FIG. 3) is less than a threshold value (e.g., 2 cm or 5 cm) (St12). The threshold value is a distance at which welding quality is not expected to deteriorate due to the accuracy of the teaching position taught by the worker's finger FNG, and may be set to any distance based on the workpiece Wk1, the required welding quality, etc.

[0045] On the other hand, when it is determined in step St11 that there is no input operation to register the teaching position of the teaching point (St11, NO), the MR device DV ends the teaching process of the teaching position shown in FIG.

[0046] If the MR device DV determines in step St12 that the distance between the workpiece Wk1 and the worker's finger FNG is less than the threshold value (St12, YES), it calculates the intersection position on the workpiece Wk1 where the direction pointed by the finger FNG (i.e., the extension direction of the finger FNG) intersects with the mesh data (here, the workpiece Wk1) (St13).

[0047] The MR device DV corrects the teaching position to the intersection position closest to the fingertip of the finger FNG among the calculated intersection positions, and additionally registers it as the teaching position of the teaching point corresponding to the workpiece Wk1 (St14).

[0048] The correction of the teaching position is not limited to the correction process described above, and other correction processes may be executed.

[0049] For example, in step St13, if there are sides (sides LN1, LN2, LN3, and LN4 in the example shown in FIG. 3) corresponding to the outer periphery of the workpiece Wk1 around the corrected teaching position (teaching position Pt11 in the example shown in FIG. 3), the MR device DV selects the side (side LN1 in the example shown in FIG. 3) that is closest to the teaching position among these sides. The MR device DV may correct the position on this side that has the shortest Euclidean distance between the side and the teaching position (teaching position Pt12 in the example shown in FIG. 3) to the teaching position, and may additionally register this position as the teaching position of the teaching point of the workpiece Wk1 (St14).

[0050] In addition, in step St13, the MR device DV may output two positions, the calculated intersection position (i.e., the taught position Pt11 shown in FIG. 3) and the position where the Euclidean distance between the side and the taught position is shortest (i.e., the taught position Pt12 shown in FIG. 3), as candidates for the taught position of the corrected taught point. Furthermore, in step St13, if a vertex exists around the position (point) where the Euclidean distance between the side and the taught position is shortest, the MR device DV may output three positions, the calculated intersection position (i.e., the taught position Pt11 shown in FIG. 3), the position on the side where the Euclidean distance between the side and the taught position is shortest (i.e., the taught position Pt12 shown in FIG. 3), and the position of the vertex existing around the position where the Euclidean distance between the side and the taught position is shortest (i.e., the taught position Pt12 shown in FIG. 3), as candidates for the taught position of the corrected taught point. The MR device DV generates virtual operation buttons that can be selected (operated) by the operator through mid-air operation or candidates for teaching positions of teaching points as virtual teaching points, and displays them on the display unit 13.

[0051] When outputting multiple corrected positions as candidates for the teaching position of the teaching point, the MR device DV accepts a selection operation of a virtual operation button or a virtual teaching point displayed on the display unit 13 by an operator's mid-air operation recognized by the depth sensor 14. The MR device DV may additionally register a teaching position based on the virtual operation button or virtual teaching point selected by mid-air operation as a teaching position of the teaching point (St14).

[0052] As described above, the MR device DV in the embodiment can more effectively suppress a decrease in the positional accuracy of the teaching position by accepting the teaching position of the teaching point from the operator's finger FNG only when it determines that the distance between the fingertip position Pt0 and the mesh data that intersects with the direction pointed by the finger FNG is less than a threshold value.

[0053] Furthermore, the MR device DV in the embodiment corrects the position of mesh data that intersects with the direction pointed by the finger FNG and outputs candidates for the teaching position. Thus, even when the worker uses the finger FNG to teach a position that is difficult to point to, such as a point on an edge of the workpiece Wk1 or a vertex, the MR device DV outputs candidates for the teaching position, thereby supporting the worker in teaching the teaching position and enabling a more accurate teaching position to be obtained. The worker can more easily teach the teaching position by selecting the teaching position desired by the worker from the candidates for the teaching position (e.g., teaching positions Pt11 to Pt13) displayed on the display unit 13. As described above, the MR device DV can more effectively suppress a decrease in the positional accuracy of the teaching position, even when the teaching position is taught by the worker's hand HND (finger FNG).

[0054] <Method of teaching a teaching position using the remote point teaching method> A method for teaching a teaching position using the remote point teaching method will be described. The remote point teaching method here is a method for teaching a teaching position of a teaching point located far away from the worker's finger FNG based on the direction pointed by the fingertip of the worker's finger FNG. The remote point teaching method is executed when there is another mesh closer to the fingertip than the mesh intended by the worker, and the distance between the fingertip of the worker's finger FNG and the teaching point on the workpiece Wk1 intended by the worker is far (for example, 10 cm or 30 cm or more). Note that the above-mentioned distance is a distance assumed to be applicable to a workpiece having a simple shape, and any distance may be set based on the shape of the workpiece, the worker's range of movement, etc.

[0055] Next, an example of teaching a teaching position using the remote point teaching method will be described with reference to Fig. 5. Fig. 5 is a diagram for explaining an example of the remote point teaching method.

[0056] 5, the worker points the index finger FNG of his hand HND toward the workpiece Wk1 to teach the teaching position Pt21. The MR device DV accepts the worker's teaching operation of the teaching point based on the captured image captured by the camera 15 and the object recognized by the depth sensor 14 (here, the workpiece Wk1 and the worker's hand HND and finger FNG).

[0057] Specifically, when the MR device DV starts the process of accepting the operator's teaching operation of the teaching point, it detects the fingertip position Pt0 (three-dimensional position) of the operator's recognized finger FNG (index finger) and the direction the finger FNG points (i.e., the extension direction of the finger FNG). The MR device DV calculates the intersection position where the direction the finger FNG points intersects with the mesh data (here, the workpiece Wk1), and registers (records) it as the teaching position Pt21 of the teaching point.

[0058] Furthermore, the MR device DV generates a teaching image SC21 in which an image of a point indicating the position indicated by the worker's fingertip position Pt0 (a circle indicating the teaching position Pt21) is superimposed on the intersection position on the mesh data, and displays the generated teaching image SC21 on the display unit 13. In this way, the MR device DV supports the teaching work of the teaching position by visualizing to the worker the teaching position being taught based on the worker's operation.

[0059] Next, another teaching method of a teaching position using the operator's hand HND will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining another teaching method of a teaching position and another correction method.

[0060] 6, the worker teaches the teaching position Pt21A based on the direction in which the palm of the hand HND faces the workpiece Wk2 and the center position Pt0A of the palm. The MR device DV accepts the worker's teaching operation of the teaching point based on the image captured by the camera 15 and the object (here, the workpiece Wk2 and the worker's palm) recognized by the depth sensor 14.

[0061] Specifically, when the MR device DV starts accepting processing for the operator's teaching operation of the teaching point, it detects the recognized center position Pt0A (three-dimensional position) of the operator's palm and the direction in which the palm is facing (i.e., the direction from the back of the hand toward the palm). The MR device DV calculates the intersection position where a line extended from the palm center position Pt0A in the direction in which the palm is facing intersects with the mesh data (here, workpiece Wk2). The MR device DV corrects the teaching position to the intersection position closest to the palm center position Pt0A among the calculated intersection positions (i.e., teaching position Pt21A), and additionally registers it as the teaching position Pt21A of the workpiece Wk2.

[0062] The MR device DV may register (record) a teaching position Pt22A obtained by correcting the teaching position Pt21A. For example, the MR device DV may output two positions, the teaching position Pt21A whose intersection position has been corrected and the position where the Euclidean distance between the teaching position Pt21A and the boundary LN21 where the base materials constituting the workpiece Wk2 overlap (i.e., the teaching position Pt22A shown in FIG. 6), as candidates for the teaching position of the corrected teaching point.

[0063] Furthermore, the MR device DV generates a teaching image SC22 by superimposing an image of a point indicating the calculated intersection position (a circle indicating the teaching position Pt21A) or an image of points indicating the positions of the candidate teaching positions (circles indicating the teaching position Pt21A and the teaching position Pt22A) on the intersection position on the mesh data, and displays the generated teaching image SC22 on the display unit 13. In this way, the MR device DV visualizes the teaching position taught based on the operator's operation, thereby supporting the teaching work of the teaching position. When displaying points indicating the candidate teaching positions, the MR device DV accepts an operator's operation to select one of the points indicating the candidate teaching positions as the teaching position.

[0064] The teaching process for the teaching position in this embodiment has been described above. In the following description, the teaching process for the teaching attitude in this embodiment will be described.

[0065] <How to teach posture> The MR device DV detects the direction (posture) of the worker's fingers based on the captured image captured by the camera 15 and the recognition result recognized by the depth sensor 14. The MR device DV accepts a teaching operation to teach the posture of the welding torch TC at the teaching position based on the detected direction (posture) of the worker's fingers.

[0066] <Teaching posture teaching method 1> First, a teaching example 1 of the teaching posture in this embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram for explaining teaching example 1 of the teaching posture.

[0067] In teaching example 1 of the teaching posture, the MR device DV receives the teaching of the teaching posture based on the direction of the worker's two fingers (thumb and index finger). The worker forms an L-shaped posture with his thumb and index finger, and teaches the hand HND in this posture by imagining it as the posture of the welding torch TC at the teaching point.

[0068] The MR device DV detects the directions of the worker's thumb and index finger based on the captured image captured by the camera 15 and the recognition results obtained by the depth sensor 14. The MR device DV acquires the detected direction of the index finger as a first direction Hx1 and acquires the direction of the thumb perpendicular to the first direction Hx1 as a second direction Hz1. The MR device DV acquires the direction perpendicular to the acquired first direction Hx1 and second direction Hz1 as a third direction Hy1. The MR device DV calculates a taught coordinate system Oh, which uses the intersection Pt0B of the detected first direction Hx1, second direction Hz1, and third direction Hy1 as the origin position and the acquired first direction Hx1, second direction Hz1, and third direction Hy1 as a three-dimensional coordinate system. In other words, the taught coordinate system Oh corresponds to the coordinate system of the welding torch TC.

[0069] The MR device DV acquires the taught posture of the welding torch TC at the taught position Pt22A based on the taught coordinate system Oh relative to the coordinate system of the real world (i.e., the world coordinate system) captured in the image captured by the camera 15. Here, the welding torch TC is provided at the tip of the welding robot RB, which is an articulated robot. Therefore, the MR device DV calculates transformation parameters for converting the acquired taught posture of the welding torch TC into a posture of the welding robot RB for performing a welding operation in the robot coordinate system of the welding robot RB, and registers the calculated transformation parameters as the taught posture at the taught position.

[0070] 7, the MR device DV may generate a virtual welding torch VTC corresponding to the taught posture based on the taught posture taught by the worker's hand HND. The MR device DV may generate a teaching image SC23 in which the tip position of the generated virtual welding torch VTC is aligned and superimposed on the teaching position Pt22A of the workpiece Wk2 shown in the captured image captured by the camera 15, and display the generated teaching image SC23 on the display unit 13. This allows the MR device DV to visualize the teaching posture being taught to the worker.

[0071] <Teaching posture teaching method 2> First, a teaching example 2 of the teaching posture in this embodiment will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining teaching example 2 of the teaching posture.

[0072] In teaching example 2 of the teaching posture, the MR device DV accepts the teaching of the teaching posture based on the orientation of the worker's three fingers (thumb, index finger, and middle finger). The worker creates a posture used in so-called Fleming's rule, in which the orientations of the thumb, index finger, and middle finger are perpendicular to each other, and teaches the hand HND in this posture as the posture of the welding torch TC at the teaching point.

[0073] The MR device DV detects the directions in which the thumb, index finger, and middle finger of the worker are pointing based on the captured image captured by the camera 15 and the recognition results obtained by the depth sensor 14. The MR device DV acquires the detected direction of the index finger as a first direction Hx2, the detected direction of the thumb as a second direction Hz1, and the detected direction of the middle finger as a third direction Hy2. The MR device DV calculates a teaching coordinate system Oh using the acquired first direction Hx2, second direction Hz2, and third direction Hy2 as a three-dimensional coordinate system, with the intersection Pt0B of the detected first direction Hx2, second direction Hz2, and third direction Hy2 as the origin position.

[0074] <Teaching posture teaching method 3> First, a teaching example 3 of the teaching posture in this embodiment will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining teaching example 3 of the teaching posture.

[0075] In teaching example 3 of the teaching posture, the MR device DV receives the teaching of the teaching posture based on the direction of one finger of the worker (the index finger in the example shown in FIG. 9) and the back surface SF or palm surface of the hand HND. The worker creates a posture with one finger used for posture teaching extended, and teaches the hand HND in this posture by imitating the posture of the welding torch TC on the teaching point. Note that the example shown in FIG. 9 describes an example of receiving the teaching of the teaching posture based on the back of the hand HND.

[0076] The MR device DV detects the direction in which the index finger of the worker is pointing and the back surface SF of the hand HND based on the captured image captured by the camera 15 and the recognition result recognized by the depth sensor 14. The MR device DV acquires the detected direction of the index finger as a first direction Hx3. The MR device DV also calculates a normal direction to the detected back surface SF of the hand and acquires this normal direction and a direction perpendicular to the first direction Hx3 as a second direction Hz3. The MR device DV acquires a direction perpendicular to the acquired first direction Hx3 and second direction Hz3 as a third direction Hy3. The MR device DV calculates a teaching coordinate system Oh using the intersection point Pt0B of the detected first direction Hx3, second direction Hz3, and third direction Hy3 as the origin position, and the acquired first direction Hx2, second direction Hz2, and third direction Hy2 as a three-dimensional coordinate system.

[0077] <Welding teaching system operation procedure> Next, an operation procedure of the MR device DV in the embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of an operation procedure of the MR device DV in the embodiment.

[0078] The MR device DV requests the processing device P1 for information on the robot coordinate system of the welding robot RB to be taught, and acquires the information on the robot coordinate system (St21).

[0079] The MR device DV starts accepting the teaching operation of the teaching point by the operator's hand HND. The MR device DV acquires information indicating whether the teaching of the teaching point is performed by the fingertip teaching method or the remote point teaching method (St22). Note that the MR device DV may acquire the information on the teaching method based on a selection operation by the operator, or may acquire the information on the teaching method set in advance.

[0080] The MR device DV determines whether or not there is a change in the teaching method for the teaching position and teaching attitude of the teaching point, based on the acquired teaching method information and the currently set teaching method information (St23).

[0081] If the MR device DV determines in step St23 that there is a change in the teaching method for the teaching position and teaching posture of the teaching point (St23, YES), it changes the current teaching method to the other teaching method (St24). On the other hand, if the MR device DV determines in step St23 that there is no change in the teaching method for the teaching position and teaching posture of the teaching point (St23, NO), it omits changing the current teaching method.

[0082] The MR device DV determines whether the current teaching method is the fingertip teaching method or the remote point teaching method (St25).

[0083] When the MR device DV determines in step St25 that the current teaching method is the fingertip teaching method (St25, fingertip teaching method), it executes teaching processing of the teaching position and teaching posture of the teaching point by the fingertip teaching method (St26).

[0084] On the other hand, when the MR device DV determines in step St25 that the current teaching method is the remote point teaching method (St25, remote point teaching method), it executes teaching processing of the teaching position and teaching posture of the teaching point by the remote point teaching method (St27).

[0085] After the MR device DV has finished teaching the teaching point by the fingertip teaching method or the remote point teaching method, it ends the operation procedure shown in Fig. 10. Note that the operation procedure shown in Fig. 10 is an example of an operation procedure for teaching one teaching point, but multiple teaching points may be taught by repeatedly executing the process of step St26 or step St27 a number of times corresponding to the number of teaching points.

[0086] <Procedure for teaching points using fingertip teaching method> Next, a teaching procedure (step St26) of a teaching point by the fingertip teaching method shown in Fig. 10 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of a teaching procedure by the fingertip teaching method of the MR device DV in the embodiment. In the description of Fig. 11, an example will be described in which an input operation of a teaching posture is accepted by the method shown in teaching example 1 of teaching posture.

[0087] When the MR device DV starts the teaching process using the fingertip teaching method, it detects an input operation by the operator to register the teaching position based on the captured image captured by the camera 15 and the recognition result recognized by the depth sensor 14 (St260). The input operation by the operator to register the teaching position may be any input operation, such as selecting (pressing) a physical button in the real world or a virtual operation button displayed on the display unit 13, a voice input by the operator's voice, an input operation based on the movement of the operator's eyes or eyelids, or another input operation specified in advance.

[0088] The MR device DV recognizes the worker's finger FNG based on the captured image captured by the camera 15 and the recognition results recognized by the depth sensor 14, and calculates the intersection position between the extension line extending from the fingertip position Pt0 of the finger FNG toward the finger FNG and the mesh data based on the recognition results recognized by the depth sensor 14 (St261).

[0089] The MR device DV registers the intersection position as a teaching position (St262). Note that the MR device DV may register a corrected position (e.g., teaching position Pt12 shown in FIG. 3) obtained by correcting the intersection position (e.g., teaching position Pt11 shown in FIG. 3) as the teaching position, or may output (display) the intersection position and each of the corrected positions (e.g., teaching positions Pt11 to Pt13 shown in FIG. 3) as candidates for the teaching position, and register any one of the positions selected by an operator operation as the teaching position.

[0090] The MR device DV detects the operator's input operation of the teaching posture based on the captured image captured by the camera 15 and the recognition result obtained by the depth sensor 14 (St263).

[0091] The MR device DV detects the direction vectors of the thumb and index finger of the worker based on the captured image captured by the camera 15 and the recognition result obtained by the depth sensor 14 (St264).

[0092] The MR device DV acquires the detected direction of the index finger as a first direction Hx1 (Step St265), acquires the direction of the thumb perpendicular to the first direction Hx1 as a second direction Hz1 (Step St266), and acquires the direction perpendicular to the acquired first direction Hx1 and second direction Hz1 as a third direction Hy1 (Step St267).

[0093] The MR device DV calculates a teaching coordinate system Oh, which has the intersection Pt0B of the detected first direction Hx1, second direction Hz1, and third direction Hy1 as the origin position and the acquired first direction Hx1, second direction Hz1, and third direction Hy1 as a three-dimensional coordinate system. Based on the calculated teaching coordinate system Oh and the robot coordinate system of the welding robot RB, the MR device DV calculates transformation parameters for transforming the posture of the welding robot RB into a posture of the welding robot RB for performing a welding operation in the acquired teaching posture of the welding torch TC on the robot coordinate system of the welding robot RB (St268).

[0094] The MR device DV records the calculated transformation parameters as the teaching posture at the teaching point in association with the teaching position of the teaching point (Step St269). After acquiring the teaching position and teaching posture of the teaching point, the MR device DV ends the teaching process using the fingertip teaching method.

[0095] In addition, in the processing of step St261, the MR device DV may execute a determination as to whether or not the distance between the fingertip position Pt0 and the intersection position of the workpiece Wk1 (mesh data) is less than a threshold value. If the MR device DV determines that the distance between the fingertip position Pt0 and the intersection position of the workpiece Wk1 (mesh data) is less than the threshold value, the MR device DV may generate and output a notification that the distance between the fingertip position Pt0 and the intersection position of the workpiece Wk1 (mesh data) is less than the threshold value, or may generate and output a notification recommending switching to the remote point teaching method.

[0096] As described above, the MR device DV in the embodiment can accept registration of the teaching position and teaching posture of the teaching point using the worker's hand HND and finger FNG.

[0097] <Procedure for teaching points using the remote point teaching method> Next, the teaching procedure (step St27) of the teaching point by the remote point teaching method shown in Fig. 10 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the teaching procedure by the remote point teaching method of the MR device DV in the embodiment.

[0098] When the MR device DV starts the teaching process using the remote point teaching method, it recognizes the worker's finger FNG based on the image captured by the camera 15 and the recognition results recognized by the depth sensor 14, and calculates the intersection position between the extension line from the fingertip position Pt0 of the finger FNG toward the finger FNG and the mesh data based on the recognition results recognized by the depth sensor 14 (St271).

[0099] The MR device DV detects whether or not the operator has performed an input operation to register the teaching position based on the captured image captured by the camera 15 and the recognition result recognized by the depth sensor 14. The operator's input operation to register the teaching position may be any input operation, such as selecting (pressing) a physical button in the real world or a virtual operation button displayed on the display unit 13, voice input by the operator's voice, an input operation based on the operator's eye or eyelid movement, or another input operation specified in advance. The MR device DV determines whether or not the operator has performed an input operation to register the teaching position (St272).

[0100] When it is determined in step St272 that the operator has performed an input operation for registering a teaching position (St272, YES), the MR device DV executes a posture teaching process for accepting a teaching posture at the teaching point (St273).

[0101] On the other hand, if the MR device DV determines in step St272 that the operator has not performed an input operation to register the teaching position (St272, NO), it determines whether the operator has performed a cancellation operation (input) to cancel the registration of the teaching position of the teaching point (St274).

[0102] When it is determined in step St274 that a cancel operation (input) has been made (St274, YES), the MR device DV ends the teaching process of the teaching point by the remote point teaching method (that is, the process of step St27).

[0103] On the other hand, when it is determined in step St274 that there is no cancel operation (input) (St274, NO), the MR device DV ends the standby loop process that waits for registration of the posture teaching of the teaching point by the remote point teaching method.

[0104] The MR device DV determines, based on the operator's operation, whether or not there is a registration operation for the teaching posture of the teaching point acquired by the posture teaching process (St275).

[0105] When it is determined in step St275 that a registration operation for the teaching posture of the teaching point has been performed (St275, YES), the MR device DV registers the registered teaching position and teaching posture as the teaching point (St276).

[0106] On the other hand, when it is determined in step St275 that there is no registration operation of the teaching posture of the teaching point (St275, NO), the MR device DV ends the teaching process of the teaching point by the remote point teaching method (that is, the process of step St27).

[0107] Next, with reference to Fig. 13 and Fig. 14, the posture teaching procedure (step St273) and posture calculation procedure (step St273C) of the teaching point by the remote point teaching method shown in Fig. 10 will be described. Fig. 13 is a flowchart showing an example of a posture teaching process procedure of the MR device DV in the embodiment. Fig. 14 is a flowchart showing an example of a posture calculation process procedure of the MR device DV in the embodiment. In the description of Fig. 13 and Fig. 14, as an example, an example will be described in which an input operation of the teaching posture is accepted by the method shown in teaching posture teaching example 1.

[0108] The MR device DV detects the direction vectors of the thumb and index finger of the worker based on the captured image captured by the camera 15 and the recognition result obtained by the depth sensor 14 (St273A).

[0109] The MR device DV determines whether or not the operator has performed an input operation to register the teaching posture (St273B).

[0110] If the MR device DV determines in step St273B that an input operation to register a teaching posture has been performed (St273B, YES), it performs a posture calculation process based on the direction vectors indicated by the detected thumb and index finger of the operator (St273C).

[0111] On the other hand, if the MR device DV determines in step St273B that there is no input operation to register the teaching posture (St273B, NO), it determines whether there is a cancellation operation (input) by the operator to cancel the registration of the teaching position of the teaching point (St273D).

[0112] When it is determined in step St273D that a cancel operation (input) has been made (St273D, YES), the MR device DV ends the teaching process of the teaching point by the remote point teaching method (that is, the process of step St273D).

[0113] On the other hand, when it is determined in step St273D that there is no cancel operation (input) (St273D, NO), the MR device DV ends the standby loop process that waits for registration of the posture teaching of the teaching point by the remote point teaching method.

[0114] In step St273C, when the MR device DV starts the posture calculation process, it acquires the detected direction of the index finger as a first direction Hx1 (St273C1), acquires the direction of the thumb perpendicular to the first direction Hx1 as a second direction Hz1 (St273C2), and acquires the direction perpendicular to the acquired first direction Hx1 and second direction Hz1 as a third direction Hy1 (St273C3).

[0115] The MR device DV calculates a teaching coordinate system Oh, which has the intersection Pt0B of the detected first direction Hx1, second direction Hz1, and third direction Hy1 as the origin position and the acquired first direction Hx1, second direction Hz1, and third direction Hy1 as a three-dimensional coordinate system. Based on the calculated teaching coordinate system Oh and the robot coordinate system of the welding robot RB, the MR device DV calculates transformation parameters for transforming the posture of the welding robot RB into a posture for performing a welding operation in the robot coordinate system of the welding robot RB with the acquired teaching posture of the welding torch TC (St273C4).

[0116] The MR device DV records the calculated transformation parameters as the teaching attitude at the teaching point in association with the teaching position of the teaching point (St273C5). After acquiring the teaching position and teaching attitude of the teaching point, the MR device DV ends the teaching process by the remote point teaching method.

[0117] As described above, the MR device DV in the embodiment can accept registration of the teaching position and teaching posture of the teaching point by the worker's hand HND and finger FNG, even when the distance between the workpiece Wk or the teaching position on the workpiece Wk and the worker's hand HND and finger FNG is far enough that it is difficult to point to the teaching point directly.

[0118] Next, an example of a mixed reality space in which a teaching result is displayed will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of a mixed reality space visually recognized by an operator.

[0119] Based on each of the taught teaching points, the MR device DV generates images of virtual teaching points Pt31, Pt32, Pt33, Pt34, and Pt35 for visualizing the teaching positions, a virtual welding motion trajectory RT of the welding robot RB that welds the workpiece (not shown), a virtual welding robot VRB equipped with a virtual welding torch VTC, and an operation menu VBT that can accept operations for the taught teaching points (i.e., virtual teaching points Pt31 to Pt35).

[0120] The operation menu VBT includes at least one operation button that can accept operations such as editing or deleting teaching points. The MR device DV accepts selection of the operation button based on the worker's aerial operation. The simulation image SC31 may also include operation buttons that can accept operations such as simulating the respective movements of the virtual welding robot VRB and the virtual welding torch VTC that perform welding operations based on the teaching points and welding operation trajectories, and replaying the respective movements of the virtual welding robot VRB and the virtual welding torch VTC that are the simulation results.

[0121] The MR device DV generates and displays a simulation image SC31 (see FIG. 15) which is a mixed reality space in which each of the virtual teaching points Pt31 to Pt35, the virtual welding operation trajectory RT, and the virtual welding robot VRB are superimposed on an image of the real world captured by the camera 15. The simulation image SC31 may include a real-world or virtual workpiece, or other real-world or virtual production equipment, etc.

[0122] As described above, the MR device DV in the embodiment can assist the worker in confirming whether the teaching results of the teaching points by the worker's hands HND and fingers FNG are the teaching contents desired by the worker, and in correcting the teaching points. This allows the worker to correct the teaching points based on the positions of the virtual teaching points Pt31 to Pt35 displayed in the simulation image SC31, the virtual welding motion trajectory RT, and the welding motions of the virtual welding robot VRB and the virtual welding torch VTC.

[0123] (Addendum) The above description of each embodiment discloses the following techniques.

[0124] (Technology 1-1) a teaching data storage unit (memory 12) that stores teaching data (i.e., teaching information data) of a robot (welding robot RB) existing in a real environment (i.e., the real world); A teaching point storage unit (memory 12) that stores teaching point data (image data indicating a teaching position, for example, data of "◯" indicating the teaching position Pt11 shown in FIG. 3) corresponding to the teaching point used to display the teaching data; a display device (display unit 13) configured to be wearable by a worker and configured to display an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit (processor 11) that acquires a relative positional relationship between the real environment and the display device (display unit 13); an image generating unit (processor 11) that generates a display image (e.g., a teaching image SC11 shown in FIG. 3) for displaying the teaching point so as to have a predetermined positional relationship with respect to the display device (display unit 13) based on the relative positional relationship and the teaching point data; an output unit (processor 11) that outputs the display image to the display device (display unit 13); A detection unit (depth sensor 14 or camera 15) that detects an aerial operation that is an operation performed by a worker in the air away from the display device (display unit 13) on a workpiece Wk present in the real environment; A feature point extraction unit (depth sensor 14 or camera 15) that recognizes feature points (edges or vertices of the workpiece Wk), The image generation unit (processor 11) generates the display image for displaying the teaching point when a designated position (for example, a teaching position Pt11 shown in FIG. 3) designated by the aerial operation is designated near the feature point. Robot teaching system (MR Device DV). With this configuration, the MR device DV can visualize the teaching position that is being taught based on the operator's operation, thereby supporting the operator in visually confirming the teaching position (designated position). As a result, the MR device DV improves the teaching accuracy of the teaching position even when the teaching position of the teaching point is taught using the operator's hand HND and finger FNG.

[0125] (Technology 1-2) the image generation unit (processor 11) generates a candidate display image in which candidates for the teaching point are displayed at the designated position and the position of the feature point, when the designated position is designated near the feature point by the mid-air operation; The output unit (processor 11) displays the candidate display image on the display device (display unit 13), The detection unit (depth sensor 14 or camera 15) detects the aerial operation performed by the worker on the candidate display image, the image generation unit (processor 11) receives an operation of selecting one of the designated position and the position of the feature point displayed in the candidate display image by the aerial operation, and generates the display image for displaying the teaching point at the selected designated position or the position of the feature point. A robot teaching system (MR device DV) described in (Technology 1-1). With this configuration, the MR device DV outputs, as candidates, positions obtained by correcting the taught position taught based on the operator's operation, and allows the operator to select which position is the correct taught position. This makes it possible to easily correct the taught position even if the taught position (designated position) pointed to by the operator's hand HND and finger FNG is deviated from the taught position desired by the operator. As a result, the MR device DV can improve the positional accuracy of the taught position even when the taught position of the taught point is taught using the operator's hand HND and finger FNG.

[0126] (Technology 1-3) The detection unit (depth sensor 14 or camera 15) detects the posture of the worker's fingers (e.g., thumb, index finger, middle finger, palm, back of hand, etc.) in the air relative to the workpiece Wk, The image generation unit (processor 11) generates a posture image (teaching image SC23) for displaying the posture of the robot at the specified position specified by the posture of the fingers (i.e., the posture of the welding robot RB, which is the posture of a virtual welding torch VTC indicating the posture of a welding torch TC provided in the welding robot RB); The output unit (processor 11) outputs to the display device (display unit 13). A robot teaching system (MR device DV) according to (Technology 1-1) or (Technology 1-2). With this configuration, even if the position pointed by the finger FNG is unclear, the MR device DV can visualize the teaching position pointed by the worker's finger FNG by displaying the teaching image SC21 on which the image "◯" indicating the teaching position Pt21 is superimposed. As a result, the MR device DV can improve the positional accuracy of the teaching position even when the teaching position of the teaching point is taught using the worker's hand HND and finger FNG.

[0127] (Technology 1-4) a model storage unit (memory 12) that stores a three-dimensional model (3D model) of a workpiece Wk that exists in a real environment (i.e., the real world); a teaching data storage unit (memory 12) that stores teaching data (i.e., teaching information data) of a robot (welding robot RB) existing in the real environment; A teaching point storage unit (memory 12) that stores teaching point data (image data indicating a teaching position, for example, data of "◯" indicating the teaching position Pt11 shown in FIG. 3) corresponding to the teaching point used to display the teaching data; a display device (display unit 13) configured to be wearable by a worker and configured to display an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit (processor 11) that acquires a relative positional relationship between the real environment and the display device (display unit 13); an image generation unit (processor 11) that generates a display image (e.g., a teaching image SC11 shown in FIG. 3 ) for displaying the three-dimensional model and the teaching point so as to have a predetermined positional relationship with respect to the display device (display unit 13) based on the relative positional relationship, the three-dimensional model, and the teaching point data; an output unit (processor 11) that outputs the display image to the display device (display unit 13); a detection unit (depth sensor 14 or camera 15) that detects an aerial operation, which is an operation performed by a worker in the air away from the display device (display unit 13) on the three-dimensional model displayed on the display device (display unit 13); A feature point extraction unit (depth sensor 14 or camera 15) that recognizes feature points (edges or vertices of the workpiece Wk), The image generation unit (processor 11) generates the display image for displaying the teaching point when a designated position (for example, a teaching position Pt11 shown in FIG. 3) designated by the aerial operation is designated near the feature point. Robot teaching system (MR Device DV). With this configuration, the MR device DV can visualize the teaching position that is being taught based on the operator's operation, thereby supporting the operator in visually confirming the teaching position (designated position). As a result, the MR device DV improves the teaching accuracy of the teaching position even when the teaching position of the teaching point is taught using the operator's hand HND and finger FNG.

[0128] (Technology 1-5) the image generation unit (processor 11) generates a candidate display image that displays the designated position and the position of the feature point when the designated position is designated near the feature point by the mid-air operation, The output unit (processor 11) displays the candidate display image on the display device (display unit 13), The detection unit (depth sensor 14 or camera 15) detects the aerial operation performed by the worker on the candidate display image, the image generation unit (processor 11) receives an operation of selecting one of the designated position and the position of the feature point displayed in the candidate display image by the aerial operation, and generates the display image for displaying the teaching point at the selected designated position or the position of the feature point. A robot teaching system (MR device DV) described in (Technology 4). With this configuration, the MR device DV outputs, as candidates, positions obtained by correcting the taught position taught based on the operator's operation, and allows the operator to select which position is the correct taught position. This makes it possible to easily correct the taught position even if the taught position (designated position) pointed to by the operator's hand HND and finger FNG is deviated from the taught position desired by the operator. As a result, the MR device DV can improve the positional accuracy of the taught position even when the taught position of the taught point is taught using the operator's hand HND and finger FNG.

[0129] (Technology 1-6) The detection unit (depth sensor 14 or camera 15) detects the posture of the worker's fingers (e.g., thumb, index finger, middle finger, palm, back of hand, etc.) in the air relative to the three-dimensional model, The image generation unit (processor 11) generates a posture image (teaching image SC23) for displaying the posture of the robot (welding robot RB) at the specified position specified by the posture of the fingers (i.e., the posture of the welding robot RB, which is the posture of the virtual welding torch VTC indicating the posture of the welding torch TC provided in the welding robot RB); The output unit (processor 11) outputs to the display device (display unit 13). A robot teaching system (MR device DV) according to (Technology 4) or (Technology 5). With this configuration, even if the position pointed by the finger FNG is unclear, the MR device DV can visualize the teaching position pointed by the worker's finger FNG by displaying the teaching image SC21 on which the image "◯" indicating the teaching position Pt21 is superimposed. As a result, the MR device DV can improve the positional accuracy of the teaching position even when the teaching position of the teaching point is taught using the worker's hand HND and finger FNG.

[0130] (Technology 1-7) A robot teaching method performed by a system (MR device DV) including at least one computer (processor 11), Store teaching data (i.e., data of teaching information) of a robot (welding robot RB) existing in the real environment and teaching point data (image data indicating a teaching position, for example, data of a "circle" indicating the teaching position Pt11 shown in FIG. 3) corresponding to the teaching point used to display the teaching data, acquiring a relative positional relationship between the real environment and a display device (display unit 13) configured to be wearable by a worker and to display an image of the real environment or the real environment itself by superimposing the image; Based on the relative positional relationship and the teaching point data, a display image (for example, teaching image SC11 shown in FIG. 3) for displaying the teaching point so as to have a predetermined positional relationship with respect to the display device (display unit 13) is generated, and output to the display device (display unit 13); Detecting an aerial operation, which is an operation performed by a worker in the air away from the display device (display unit 13) on a workpiece Wk present in the real environment, A feature point of the workpiece is recognized, and when the designated position designated by the aerial operation is designated near the feature point, the display image for displaying the teaching point is generated and output to the display device (display unit 13). Robot teaching method. With this configuration, even if the position pointed by the finger FNG is unclear, the MR device DV can visualize the teaching position pointed by the worker's finger FNG by displaying the teaching image SC21 on which the image "◯" indicating the teaching position Pt21 is superimposed. As a result, the MR device DV can improve the positional accuracy of the teaching position even when the teaching position of the teaching point is taught using the worker's hand HND and finger FNG.

[0131] (Technology 1-8) A robot teaching method performed by a system (MR device DV) including at least one computer (processor 11), A three-dimensional model (3D model) of a workpiece Wk existing in a real environment, teaching data (i.e., teaching information data) of a robot (welding robot RB) existing in the real environment, and teaching point data (image data indicating a teaching position, for example, data of a "circle" indicating the teaching position Pt11 shown in FIG. 3) corresponding to a teaching point used to display the teaching data are stored; acquiring a relative positional relationship between the real environment and a display device (display unit 13) configured to be wearable by a worker and to display an image of the real environment or the real environment itself by superimposing the image; Based on the relative positional relationship, the three-dimensional model, and the teaching point data, a display image (e.g., teaching image SC11 shown in FIG. 3) for displaying the three-dimensional model and the teaching point so as to have a predetermined positional relationship with respect to the display device (display unit 13) is generated, and output to the display device (display unit 13); Detecting an aerial operation that is an operation performed by a worker in the air away from the display device (display unit 13) on the three-dimensional model present in the real environment; The feature point of the workpiece is recognized, and when a designated position (for example, a teaching position Pt11 shown in FIG. 3) designated by the aerial operation is designated near the feature point, the display image for displaying the teaching point is generated and output to the display device (display unit 13). Robot teaching method. With this configuration, even if the position pointed by the finger FNG is unclear, the MR device DV can visualize the teaching position pointed by the worker's finger FNG by displaying the teaching image SC21 on which the image "◯" indicating the teaching position Pt21 is superimposed. As a result, the MR device DV can improve the positional accuracy of the teaching position even when the teaching position of the teaching point is taught using the worker's hand HND and finger FNG.

[0132] (Technology 2-1) a teaching data storage unit (memory 12) that stores teaching data (i.e., teaching information data) of a robot (welding robot RB) existing in a real environment (i.e., the real world); A teaching point storage unit (memory 12) that stores teaching point data (image data indicating a teaching position, for example, data of "◯" indicating the teaching position Pt11 shown in FIG. 3) corresponding to the teaching point used to display the teaching data; a display device (display unit 13) configured to be wearable by a worker and configured to display an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit (processor 11) that acquires a relative positional relationship between the real environment and the display device (display unit 13); an image generating unit (processor 11) that generates a display image (e.g., a teaching image SC11 shown in FIG. 3) for displaying the teaching point so as to have a predetermined positional relationship with respect to the display device (display unit 13) based on the relative positional relationship and the teaching point data; an output unit (processor 11) that outputs the display image to the display device (display unit 13); A detection unit (depth sensor 14 or camera 15) that detects an aerial operation that is an operation performed by a worker in the air away from the display device (display unit 13) on a workpiece Wk present in the real environment; Equipped with When an operation indicating a predetermined direction with respect to the real environment is executed as the aerial operation, the image generation unit (processor 11) generates the display image (e.g., teaching image SC21 shown in FIG. 5 ) for displaying the teaching point at a position of intersection between a virtual axis along the predetermined direction and the surface of the workpiece (e.g., teaching position Pt21 shown in FIG. 5 ). Robot teaching system (MR Device DV). With this configuration, the MR device DV can visualize the teaching position that is being taught based on the operator's operation, thereby supporting the operator in visually confirming the teaching position (designated position). As a result, the MR device DV improves the teaching accuracy of the teaching position even when the teaching position of the teaching point is taught using the operator's hand HND and finger FNG.

[0133] (Technology 2-2) The detection unit (depth sensor 14 or camera 15) detects the posture of the worker's fingers (e.g., thumb, index finger, middle finger, palm, back of hand, etc.) in the air relative to the workpiece Wk, The image generation unit (processor 11) generates a posture image (teaching image SC23) for displaying the posture of the robot at the specified position specified by the posture of the fingers (i.e., the posture of the welding robot RB, which is the posture of a virtual welding torch VTC indicating the posture of a welding torch TC provided in the welding robot RB); The output unit (processor 11) outputs to the display device (display unit 13). A robot teaching system (MR device DV) described in (Technology 2-1). With this configuration, even if the distance between the worker's finger FNG and the taught position Pt21 taught based on the worker's operation is long and the position pointed to by the finger FNG is unclear, the MR device DV can visualize the taught position pointed to by the worker's finger FNG by displaying a taught image SC21 on which an image "◯" indicating the taught position Pt21 is superimposed. As a result, the MR device DV can improve the positional accuracy of the taught position even when the taught position of the taught point is taught using the worker's hand HND and finger FNG.

[0134] (Technology 2-3) a model storage unit (memory 12) that stores a three-dimensional model (3D model) of a workpiece Wk that exists in a real environment (i.e., the real world); a teaching data storage unit (memory 12) that stores teaching data (i.e., teaching information data) of a robot (welding robot RB) existing in the real environment; A teaching point storage unit (memory 12) that stores teaching point data (image data indicating a teaching position, for example, data of "◯" indicating the teaching position Pt11 shown in FIG. 3) corresponding to the teaching point used to display the teaching data; a display device (display unit 13) configured to be wearable by a worker and configured to display an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit (processor 11) that acquires a relative positional relationship between the real environment and the display device (display unit 13); an image generation unit (processor 11) that generates a display image (e.g., a teaching image SC11 shown in FIG. 3 ) for displaying the three-dimensional model and the teaching point so as to have a predetermined positional relationship with respect to the display device (display unit 13) based on the relative positional relationship, the three-dimensional model, and the teaching point data; an output unit (processor 11) that outputs the display image to the display device (display unit 13); a detection unit (depth sensor 14 or camera 15) that detects an aerial operation that is an operation performed by a worker in the air away from the display device (display unit 13) on the three-dimensional model displayed on the display device (display unit 13), When an operation indicating a predetermined direction with respect to the real environment is executed as the aerial operation, the image generation unit (processor 11) generates the display image (e.g., teaching image SC21 shown in FIG. 5 ) for displaying the teaching point at a position of intersection between a virtual axis along the predetermined direction and the surface of the workpiece (e.g., teaching position Pt21 shown in FIG. 5 ). Robot teaching system (MR Device DV). With this configuration, the MR device DV can visualize the teaching position that is being taught based on the operator's operation, thereby supporting the operator in visually confirming the teaching position (designated position). As a result, the MR device DV improves the teaching accuracy of the teaching position even when the teaching position of the teaching point is taught using the operator's hand HND and finger FNG.

[0135] (Technology 2-4) The detection unit (depth sensor 14 or camera 15) detects the posture of the worker's fingers (e.g., thumb, index finger, middle finger, palm, back of hand, etc.) in the air relative to the three-dimensional model, the image generation unit (processor 11) generates a posture image (teaching image SC23) for displaying the posture of the robot (welding robot RB) at the position of the intersection between a virtual axis along the predetermined direction based on the posture of the fingers and the surface of the workpiece (i.e., the posture of the welding robot RB, which is the posture of the virtual welding torch VTC indicating the posture of the welding torch TC provided in the welding robot RB); The output unit (processor 11) outputs to the display device (display unit 13). A robot teaching system (MR device DV) described in (Technology 2-3). With this configuration, even if the distance between the worker's finger FNG and the taught position Pt21 taught based on the worker's operation is long and the position pointed to by the finger FNG is unclear, the MR device DV can visualize the taught position pointed to by the worker's finger FNG by displaying a taught image SC21 on which an image "◯" indicating the taught position Pt21 is superimposed. As a result, the MR device DV can improve the positional accuracy of the taught position even when the taught position of the taught point is taught using the worker's hand HND and finger FNG.

[0136] (Technology 2-5) A robot teaching method performed by a system (MR device DV) including at least one computer (processor 11), Store teaching data (i.e., data of teaching information) of a robot (welding robot RB) existing in the real environment and teaching point data (image data indicating a teaching position, for example, data of a "circle" indicating the teaching position Pt11 shown in FIG. 3) corresponding to the teaching point used to display the teaching data, acquiring a relative positional relationship between the real environment and a display device (display unit 13) configured to be wearable by a worker and to display an image of the real environment or the real environment itself by superimposing the image; Based on the relative positional relationship and the teaching point data, a display image (for example, teaching image SC11 shown in FIG. 3) for displaying the teaching point so as to have a predetermined positional relationship with respect to the display device (display unit 13) is generated, and output to the display device (display unit 13); Detecting an aerial operation, which is an operation performed by a worker in the air away from the display device (display unit 13) on a workpiece Wk present in the real environment, When an operation indicating a predetermined direction with respect to the real environment is executed as the aerial operation, the display image (e.g., teaching image SC21 shown in FIG. 5 ) for displaying the teaching point at a position of intersection between a virtual axis along the predetermined direction and the surface of the workpiece (e.g., teaching position Pt21 shown in FIG. 5 ) is generated and output to the display device (display unit 13). Robot teaching method. With this configuration, even if the distance between the worker's finger FNG and the taught position Pt21 taught based on the worker's operation is long and the position pointed to by the finger FNG is unclear, the MR device DV can visualize the taught position pointed to by the worker's finger FNG by displaying a taught image SC21 on which an image "◯" indicating the taught position Pt21 is superimposed. As a result, the MR device DV can improve the positional accuracy of the taught position even when the taught position of the taught point is taught using the worker's hand HND and finger FNG.

[0137] (Technology 2-6) A robot teaching method performed by a system (MR device DV) including at least one computer (processor 11), A three-dimensional model (3D model) of a workpiece Wk existing in a real environment, teaching data (i.e., teaching information data) of a robot (welding robot RB) existing in the real environment, and teaching point data (image data indicating a teaching position, for example, data of a "circle" indicating the teaching position Pt11 shown in FIG. 3) corresponding to a teaching point used to display the teaching data are stored; acquiring a relative positional relationship between the real environment and a display device (display unit 13) configured to be wearable by a worker and to display an image of the real environment or the real environment itself by superimposing the image; Based on the relative positional relationship, the three-dimensional model, and the teaching point data, a display image (e.g., teaching image SC11 shown in FIG. 3) for displaying the three-dimensional model and the teaching point so as to have a predetermined positional relationship with respect to the display device (display unit 13) is generated, and output to the display device (display unit 13); Detecting an aerial operation that is an operation performed by a worker in the air away from the display device (display unit 13) on the three-dimensional model present in the real environment; When an operation indicating a predetermined direction with respect to the real environment is executed as the aerial operation, the display image (e.g., teaching image SC21 shown in FIG. 5 ) for displaying the teaching point at a position of intersection between a virtual axis along the predetermined direction and the surface of the workpiece (e.g., teaching position Pt21 shown in FIG. 5 ) is generated and output to the display device (display unit 13). Robot teaching method. With this configuration, even if the distance between the worker's finger FNG and the taught position Pt21 taught based on the worker's operation is long and the position pointed to by the finger FNG is unclear, the MR device DV can visualize the taught position pointed to by the worker's finger FNG by displaying a taught image SC21 on which an image "◯" indicating the taught position Pt21 is superimposed. As a result, the MR device DV can improve the positional accuracy of the taught position even when the taught position of the taught point is taught using the worker's hand HND and finger FNG.

[0138] Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components of the various embodiments described above may be combined in any manner without departing from the spirit of the invention. [Industrial Applicability]

[0139] The present disclosure is useful as a robot teaching system and a robot teaching method that use an operator's fingers, a marker pen, or the like to assist in teaching a teaching point at a position that is difficult to teach directly. [Explanation of symbols]

[0140] 10,20 Communications Department 11,21 processor 12,22 memory 13 Display section 14 Depth Sensor 15 Camera 100 Welding Teaching System 221 Instruction Information Recording Unit 222 Work information recording unit FNG finger HND Hand Hx1,Hx2,Hx3 First direction Hy1, Hy2, Hy3 Third direction Hz1,Hz2,Hz3 Second direction LN1, LN2, LN3, LN4 sides LN21 boundary Oh teaching coordinate system P1 Processing Unit Pt0 fingertip position Pt0A center position Pt11, Pt12, Pt13, Pt21, Pt21A, Pt22A Teaching position Pt31, Pt32, Pt33, Pt34, Pt35 Virtual teaching points RB welding robot RT welding motion trajectory SC11, SC21, SC22, SC23 Teaching images SC31 simulation image TC welding torch VBT Operation Menu VRB Virtual Welding Robot VTC Virtual Welding Torch Wk, Wk1, Wk2 Work

Claims

1. a teaching data storage unit that stores teaching data of a robot existing in a real environment; a teaching point storage unit that stores teaching point data corresponding to teaching points used to display the teaching data; a display device configured to be wearable by a worker and configured to display an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit that acquires a relative positional relationship between the real environment and the display device; an image generating unit that generates a display image for displaying the teaching point so as to have a predetermined positional relationship with respect to the display device, based on the relative positional relationship and the teaching point data; an output unit that outputs the display image to the display device; a detection unit that detects an aerial operation that is an operation performed by a worker in the air away from the display device on a workpiece present in the real environment; Equipped with the image generation unit generates the display image for displaying the teaching point at a position of an intersection between a virtual axis along the predetermined direction and a surface of the workpiece when an operation indicating a predetermined direction with respect to the real environment is executed as the aerial operation. Robot teaching system.

2. the detection unit detects the posture of the worker's fingers in the air relative to the workpiece, the image generation unit generates a posture image for displaying the posture of the robot at the position of intersection between a virtual axis along the predetermined direction based on the posture of the fingers and a surface of the workpiece; The output unit outputs to the display device. The robot teaching system according to claim 1 .

3. a model storage unit that stores a three-dimensional model of a workpiece existing in a real environment; a teaching data storage unit that stores teaching data of the robot existing in the real environment; a teaching point storage unit that stores teaching point data corresponding to teaching points used to display the teaching data; a display device configured to be wearable by a worker and configured to display an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit that acquires a relative positional relationship between the real environment and the display device; an image generation unit that generates a display image for displaying the three-dimensional model and the teaching point so as to have a predetermined positional relationship with respect to the display device, based on the relative positional relationship, the three-dimensional model, and the teaching point data; an output unit that outputs the display image to the display device; a detection unit that detects an aerial operation that is an operation performed by a worker in the air away from the display device on the three-dimensional model displayed on the display device, the image generation unit generates the display image for displaying the teaching point at a position of an intersection between a virtual axis along the predetermined direction and a surface of the workpiece when an operation indicating a predetermined direction with respect to the real environment is executed as the aerial operation. Robot teaching system.

4. the detection unit detects a posture of the worker's hand or fingers in the air relative to the three-dimensional model; the image generation unit generates a posture image for displaying the posture of the robot at the position of intersection between a virtual axis along the predetermined direction based on the posture of the fingers and a surface of the workpiece; The output unit outputs to the display device. The robot teaching system according to claim 3 .

5. A robot teaching method performed by a system including at least one computer, comprising: storing teaching data of a robot existing in a real environment and teaching point data corresponding to teaching points used to display the teaching data; acquiring a relative positional relationship between the real environment and a display device configured to be wearable by a worker and displaying an image of the real environment or the real environment itself by superimposing the image; generating a display image for displaying the teaching point so as to have a predetermined positional relationship with respect to the display device based on the relative positional relationship and the teaching point data, and outputting the generated image to the display device; detecting an aerial operation that is an operation performed by a worker in the air away from the display device on a workpiece present in the real environment; When an operation indicating a predetermined direction with respect to the real environment is executed as the aerial operation, the display image for displaying the teaching point at a position of an intersection between a virtual axis along the predetermined direction and a surface of the workpiece is generated and output to the display device. Robot teaching method.

6. A robot teaching method performed by a system including at least one computer, comprising: storing a three-dimensional model of a workpiece existing in a real environment, teaching data of a robot existing in the real environment, and teaching point data corresponding to teaching points used to display the teaching data; acquiring a relative positional relationship between the real environment and a display device configured to be wearable by a worker and displaying an image of the real environment or the real environment itself by superimposing the image; generating a display image for displaying the three-dimensional model and the teaching point so as to have a predetermined positional relationship with respect to the display device based on the relative positional relationship, the three-dimensional model, and the teaching point data, and outputting the generated image to the display device; detecting an aerial operation that is an operation performed by a worker in the air away from the display device on the three-dimensional model that exists in the real environment; When an operation indicating a predetermined direction with respect to the real environment is executed as the aerial operation, the display image is generated to display the teaching point at a position of an intersection between a virtual axis along the predetermined direction and a surface of the workpiece. Robot teaching method.

Images