Robot teaching system and robot teaching method

Abstract

To support teaching work of a welding torch posture at a teaching point in teaching a welding robot operation.SOLUTION: A robot teaching system stores a three-dimensional model corresponding to at least a part of a robot or a welding torch, or a teaching member, outputs a display image for displaying the three-dimensional model and an operation screen on a display device configured to be mountable to a worker on the basis of the three-dimensional model and operation screen data and displaying an image to be superimposed on an image of an actual environment or the actual environment itself, and generates a display image for displaying the post-change three-dimensional model after changing a posture of the three-dimensional model on the basis of an aerial operation of the worker in the air separated from the display device.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a robot teaching system and a robot teaching method. [Background technology]

[0002] Patent Document 1 discloses a welding system having a welding robot equipped with a torch and a welding robot control program creation device. The welding system acquires position information of the welding start point and welding end point of welding on a workpiece, and posture information that can identify the posture of the torch relative to the welding line at a welding teaching point on the welding line connecting the welding start point and welding end point, creates a welding robot control program for welding from the welding start point to the welding end point based on the position information and posture information, and welds the workpiece based on the welding robot control program. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 251087 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide a robot teaching system and a robot teaching method that assist in teaching the posture of a welding torch at a teaching point in teaching a welding robot operation. [Means for solving the problem]

[0005] The present disclosure provides a robot teaching system comprising: a model memory unit that stores a three-dimensional model corresponding to at least a part of a robot existing in a real environment or a welding torch used for welding, or a teaching component used to teach the robot; an operation screen memory unit that stores operation screen data corresponding to an operation screen used to display and operate the three-dimensional model; a display device that is configured to be wearable by a worker and that displays an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit that acquires a relative positional relationship between the real environment, the teaching component, and the display device; an image generation unit that generates a display image for displaying the three-dimensional model and the operation screen so that they are in a predetermined positional relationship with respect to the display device based on the relative positional relationship, the three-dimensional model, and the operation screen data; an output unit that outputs the display image to the display device; and a detection unit that detects an aerial operation, which is an operation performed by the worker in the air away from the display device on the operation screen displayed on the display device, wherein the image generation unit generates the display image for displaying a modified three-dimensional model in which the posture of the three-dimensional model is modified based on the aerial operation.

[0006] The present disclosure also provides a robot teaching system comprising: a model memory unit that stores a three-dimensional model corresponding to at least a part of a robot existing in a real environment or a welding torch used for welding, or a teaching member used to teach the robot; a display device that is configured to be wearable by a worker and that displays an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit that acquires the relative positional relationship between the real environment, the teaching member, and the display device; an image generation unit that generates a display image for displaying the three-dimensional model so that it is in a predetermined positional relationship with respect to the display device based on the relative positional relationship and the three-dimensional model; an output unit that outputs the display image to the display device; and a detection unit that detects an aerial operation, which is an operation performed by a worker in the air away from the display device on the three-dimensional model displayed on the display device, wherein the image generation unit generates the display image for displaying a modified three-dimensional model in which the posture of the three-dimensional model is modified based on the aerial operation.

[0007] The present disclosure also provides a robot teaching method performed by a system including at least one computer, the system storing a three-dimensional model corresponding to at least a part of a robot or a welding torch used for welding that exists in a real environment, or a teaching component used to teach the robot, and operation screen data corresponding to an operation screen used to display and operate the three-dimensional model; acquiring a relative positional relationship between the real environment, the teaching component, and a display device that is configured to be wearable by a worker and that displays an image by superimposing it on an image of the real environment or the real environment itself; generating a display image for displaying the three-dimensional model and the operation screen so that they are in a predetermined positional relationship with the display device based on the relative positional relationship, the three-dimensional model, and the operation screen data, and displaying the display image on the display device; detecting an aerial operation, which is an operation performed by the worker in the air away from the display device with respect to the operation screen displayed on the display device, and generating the display image for displaying a modified three-dimensional model in which the posture of the three-dimensional model has been modified based on the aerial operation.

[0008] The present disclosure also provides a robot teaching method performed by a system including at least one computer, which stores a three-dimensional model corresponding to a robot existing in a real environment or at least a part of a welding torch used for welding, or a teaching component used to teach the robot, obtains a relative positional relationship between the real environment, the teaching component, and a display device that is configured to be attachable to a worker and displays an image superimposed on an image of the real environment or the real environment itself, generates a display image for displaying the three-dimensional model so that it is in a predetermined positional relationship with the display device based on the relative positional relationship and the three-dimensional model, displays the display image on the display device, detects an aerial operation that is an operation performed by the worker in the air away from the display device on the three-dimensional model displayed on the display device, and generates the display image for displaying a modified three-dimensional model in which the posture of the three-dimensional model is modified based on the aerial operation. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to assist in teaching the posture of a welding torch at a teaching point when teaching the operation of a welding robot. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of a welding teaching system according to an embodiment. [Figure 2] FIG. 1 shows an example of the internal configuration of an MR device and a processing device. [Figure 3] A diagram explaining the difference between teaching a teaching point and correcting a teaching point [Figure 4] A diagram comparing the posture of the virtual teaching tool before and after the posture change. [Figure 5] A diagram explaining example 1 of changing the attitude of teaching points [Figure 6] A diagram explaining example 2 of changing the posture of the teaching point [Figure 7] A diagram explaining example 3 of changing the attitude of the teaching point [Figure 8] A diagram explaining example 4 of changing the attitude of the teaching point [Figure 9] A diagram explaining example 5 of changing the attitude of the teaching point [Figure 10] 1 is a flowchart showing an example of an overall operation procedure of an MR device according to an embodiment. [Figure 11] 10 is a flowchart showing an example of a procedure for newly registering teaching information for an MR device according to an embodiment. [Figure 12] 10 is a flowchart showing an example of a procedure for changing the posture of an MR device in accordance with teaching information of the embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0011] (Background to this disclosure) Recently, a teaching method has been developed that uses Augmented Reality (AR) equipment to read and teach the position of a teaching point taught by an operator and the posture of a teaching tool at the teaching point, as in the welding system described in Patent Document 1. This teaching method allows the operator to directly teach the teaching point, thereby shortening the time required to teach the teaching point compared to using a general offline teaching system such as a teach pendant. However, because the operator wears a head-mounted display to perform the teaching work, it can be difficult to teach the teaching point in the desired posture based on the position or arrangement of the operator wearing the head-mounted display, the workpiece, and the welding robot relative to the workpiece, or the positional relationship of the teaching point relative to the workpiece.

[0012] Therefore, in the following embodiments, a robot teaching system and a robot teaching method will be described that assist in teaching the posture of a welding torch at a teaching point in teaching a welding robot operation.

[0013] Hereinafter, with reference to the drawings as appropriate, detailed descriptions of embodiments specifically disclosing a robot teaching system and a robot teaching method according to the present disclosure will be provided. However, unnecessary detailed descriptions may be omitted. For example, detailed descriptions of well-known matters or redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure and are not intended to limit the subject matter recited in the claims.

[0014] <Welding system overview> First, a welding teaching system 100 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of welding teaching system 100 according to an embodiment. Note that welding teaching system 100 shown in Fig. 1 is an example and is not limited to this.

[0015] When a worker teaches a teaching point, welding teaching system 100 generates a virtual teaching tool VTL that has the posture of the welding torch TC taught on the real world or virtual workpiece Wk and resembles the welding torch TC. Note that the virtual teaching tool VTL in the following description may be replaced with a virtual welding torch.

[0016] The welding teaching system 100 generates a mixed reality image by superimposing an image of the generated virtual teaching tool VTL on a captured image of the real world, visualizes the image for the worker, and accepts an operation to change the posture of the virtual teaching tool VTL shown in the image. The welding teaching system 100 records information on the posture changed by the change operation for each teaching point, and generates a mixed reality image by superimposing an image of the virtual teaching tool VTL after the posture change, visualizes the image for the worker.

[0017] In the following description, an example of the posture will be described in which the angle (i.e., twist angle) is set around the TX axis, which is the direction in which the welding wire WW is fed from the welding torch TC toward the welding point (i.e., the teaching point) on the workpiece Wk. However, the changeable posture (angle) is not limited to the angle (twist angle) around the TX axis. The changeable posture (angle) may be, for example, an angle (i.e., tilt angle) around a direction that follows the movement trajectory of the tip of the welding torch TC as the rotation axis, or an angle (i.e., forward / rearward advance angle) around a direction that is perpendicular to the movement trajectory of the tip of the welding torch TC and that follows the surface of the workpiece Wk as the rotation axis.

[0018] In addition, in the following explanation, we will explain an example in which the work displayed in the mixed reality space is an actual work Wk that exists in the real world, but it may also be a virtual work constructed based on 3D model data, etc.

[0019] Furthermore, the teaching points taught in the present disclosure may include not only welding points where the workpiece Wk is welded, but also approach points where the welding robot RB (welding torch TC) approaches the workpiece Wk, avoidance points where the welding robot RB (welding torch TC) avoids obstacles, free-running points where the welding torch TC is made to free-run, or departure points where the welding robot RB (welding torch TC) moves away from the workpiece Wk.

[0020] The welding teaching system 100 includes at least an MR device DV. The welding teaching system 100 shown in FIG. 1 includes a workpiece Wk, a teaching tool TL, an MR device DV, and a processing device P1. Note that the workpiece Wk shown in FIG. 1 is a real-world (actual) workpiece Wk, but the workpiece Wk may be a virtual workpiece constructed based on a 3D model of the workpiece Wk. Furthermore, if the MR device DV can realize the functions of the processing device P1, the processing device P1 may be omitted.

[0021] The MR device DV is a so-called head-mounted display, and is connected to the processing device P1 so as to be able to communicate data with it. The MR device DV is worn on the head of the worker, and forms a virtual space in which an image of virtual production equipment (for example, a virtual workpiece, a virtual welding robot VRB, or a virtual jig) is superimposed on a captured image of a real space corresponding to the worker's field of vision, and displays the virtual space on the display unit 13, thereby visualizing the virtual space for the worker.

[0022] The welding robot RB in this disclosure has a welding torch TC and a wire feeder WW1, and is a robot that performs welding by feeding the welding wire WW from the welding torch TC to a welding point on a workpiece Wk using the wire feeder WW1. The welding robot RB is controlled by a robot controller (not shown) that is connected to a processing device P1 (described later) so as to be capable of data communication.

[0023] The MR device DV generates a virtual teaching tool VTL (i.e., a virtual welding torch) having a taught posture at the position of the taught teaching point based on information including the position and posture of the taught teaching point (hereinafter referred to as "teaching information"). The MR device DV generates a teaching image in which the generated virtual teaching tool VTL is superimposed at the position of the teaching point on the workpiece Wk shown in the captured image captured by the camera 15, and displays the teaching image on the display unit 13, thereby visualizing the posture of the teaching tool TL (i.e., the welding torch) at the taught teaching point.

[0024] The MR device DV also receives an operation to change the posture of the teaching tool (welding torch) at the teaching point that has been taught using each method shown in each posture change example described later, i.e., the posture included in the teaching information. Upon receiving the posture change operation, the MR device DV generates and displays a teaching image in which a virtual teaching tool VTL (i.e., a virtual welding torch) corresponding to the changed posture is superimposed on the captured image. The MR device DV also generates teaching information including information on the changed posture and transmits it to the processing device P1.

[0025] The processing device P1 is connected between the MR device and the robot controller so that data communication can be performed between them. The processing device P1 records each piece of taught teaching information (i.e., information on the position (three-dimensional) and posture (three-dimensional) of the teaching point). The processing device P1 transmits the recorded teaching information to the MR device DV, which executes a posture change process, acquires the changed teaching information transmitted from the MR device DV, and updates (records) the recorded teaching information before the change to the changed teaching information. The processing device P1 also executes teaching processing for each teaching point by transmitting the position and posture information of the teaching point to the robot controller, which controls and drives the welding robot RB in the real world.

[0026] Next, an example of the internal configuration of the MR device DV and the processing device P1 will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the internal configuration of the MR device DV and the processing device P1.

[0027] The MR device DV includes a communication unit 10, a processor 11, a memory 12, a display unit 13, a depth sensor 14, and a camera 15.

[0028] The communication unit 10 is connected to the teaching tool TL and the processing device P1 so as to be able to communicate wirelessly or via wires, and transmits and receives data. The communication unit 10 outputs various data transmitted from the teaching tool TL and the processing device P1 to the processor 11. The communication unit 10 transmits various data output from the processor 11 to the processing device P1. The wireless communication here refers to communication via a wireless local area network (LAN) such as Wi-Fi (registered trademark). When the processing device P1 is omitted from the welding teaching system 100, the communication unit 10 is connected to the robot controller so as to be able to communicate data.

[0029] Processor 11 is configured using, for example, a Central Processing Unit (hereinafter referred to as "CPU") or a Field Programmable Gate Array (hereinafter referred to as "FPGA"), and performs various processes and controls in cooperation with memory 12. Specifically, processor 11 references programs and data stored in memory 12 and executes the programs to realize functions such as accepting new teaching information, changing already-taught teaching information, and generating a teaching image by generating a virtual teaching tool corresponding to the teaching information. When processing device P1 is omitted from welding teaching system 100, processor 11 is configured to be able to realize functions similar to those of processor 21 of processing device P1.

[0030] Processor 11 calculates the relative positional relationship in three-dimensional space for each of the recognized or detected objects and production equipment based on the objects detected by depth sensor 14, the captured images captured by camera 15, and the 3D model data of various production equipment stored in memory 12. Specifically, processor 11 calculates the relative positional relationship in three-dimensional space for each of the detected or recognized worker's fingers, teaching tool TL (virtual teaching tool VTL), workpiece Wk (virtual workpiece), welding robot RB (virtual welding robot VRB), marker Mk (virtual marker VMk), etc. As a result, processor 11 can display on display unit 13 an image of a virtual space in which the virtual production equipment is superimposed on a captured image of the real world, and can accept operator operation (air operation) with respect to the displayed image. As a result, processor 11 can generate teaching information to be taught to real-world welding robot RB that welds workpiece Wk.

[0031] The memory 12 includes, for example, a random access memory (hereinafter referred to as "RAM") as a work memory used when executing each process of the processor 11, and a read only memory (hereinafter referred to as "ROM") that stores programs and data that define the operations of the processor 11. The RAM temporarily stores data or information generated or acquired by the processor 11. The ROM stores programs that define the operations of the processor 11.

[0032] The memory 12 stores a three-dimensional model of at least a part of the welding robot RB or welding torch TC that welds the workpiece Wk, a three-dimensional model of a teaching tool TL used to teach the welding robot RB, or a three-dimensional model of a marker Mk used to teach the posture of the welding robot RB. The memory 12 stores various data that are generated by the processor 11 and displayed on the display unit 13.

[0033] The memory 12 also stores, for each teaching point, teaching information that has been taught and transmitted from the processing device P1, or information on the attitude that has been received by any of the attitude change operations described below.

[0034] The display unit 13 is configured using, for example, a Liquid Crystal Display (LCD) or an organic electroluminescence (EL). The display unit 13 displays an image of the real world itself or a virtual space in which virtual production equipment is superimposed on the real world. The display unit 13 realizes mixed reality by, for example, displaying an image of the virtual space (e.g., a teaching image) in which virtual production equipment generated by the processor 11 is superimposed on an image of the real world captured by the camera 15.

[0035] The depth sensor 14 is a sensor that measures the distance between the MR device DV and an object in the real world and recognizes the three-dimensional shape of the object in the real world (for example, a workpiece Wk, a welding robot RB, a jig, etc.). The depth sensor 14 outputs the recognition result to the processor 11.

[0036] The camera 15 captures an image of an area (real world) corresponding to the field of view of the worker wearing the MR device DV. The camera 15 outputs the captured image to the processor 11.

[0037] The processing device P1 includes a communication unit 20, a processor 21, and a memory 22.

[0038] The communication unit 20 is connected to the MR device DV and the robot controller so as to be able to communicate wirelessly or via wires, and transmits and receives data. The communication unit 20 outputs various data transmitted from the MR device DV to the processor 21. The communication unit 20 transmits various data output from the processor 21 to the MR device DV or the robot controller. The wireless communication here refers to communication via a wireless LAN such as Wi-Fi (registered trademark).

[0039] Processor 21 is configured using, for example, a CPU or FPGA, and performs various processes and controls in cooperation with memory 22. Specifically, processor 21 references the programs and data stored in memory 22 and executes the programs to realize various functions for generating a welding teaching program.

[0040] The memory 22 includes, for example, a RAM as a work memory used when executing each process of the processor 21, and a ROM for storing programs and data that define each operation of the processor 21. The RAM temporarily stores data or information generated or acquired by the processor 21. The ROM has written therein programs that define the operation of the processor 21. The memory 22 includes a teaching information recording unit 221 and a work information recording unit 222. The teaching information recording unit 221 and the work information recording unit 222 may be recorded in the memory 12 of the MR device DV. The memory 22 records a 3D model of the welding robot RB, information about the coordinate system of the welding robot, etc.

[0041] The teaching information recording unit 221 records each of the plurality of pieces of teaching information for each workpiece Wk. The workpiece information recording unit 222 records a 3D model of the workpiece Wk.

[0042] Next, referring to Fig. 3, the teaching images MXR1 and MXR2 displayed on the MR device DV when teaching is performed using a real workpiece Wk and a real teaching tool TL will be described. Fig. 3 is a diagram illustrating the difference between the captured image WLD and the teaching images MXR1 and MXR2. Note that the captured image WLD and teaching images MXR1 and MXR2 shown in Fig. 3 are merely examples and are not limiting. For example, the workpiece Wk may be a virtual workpiece. Furthermore, the teaching tool TL may be the operator's finger, etc.

[0043] The captured image WLD is the real world captured by the camera 15. The captured image WLD is an image that shows the actual workpiece Wk and the actual teaching tool TL. In the captured image WLD, the teaching tool TL shows how teaching information is being taught.

[0044] The teaching image MXR1 is an image in which the teaching tool TL shown in the real-world image WLD captured by the camera 15 is replaced with a virtual teaching tool VTL, and shows how the teaching tool TL is teaching teaching information.

[0045] The MR device DV acquires the position and orientation of the teaching tool TL at the time when the button on the teaching tool TL is operated by the operator, and generates a virtual teaching tool VTL having the acquired orientation of the teaching tool TL. The MR device DV deletes the teaching tool TL appearing in the captured image WLD, and generates and displays a teaching image MXR1 in place of the teaching tool TL, in which the virtual teaching tool VTL generated at the position of the teaching tool TL on the captured image WLD is superimposed.

[0046] Here, the MR device DV receives an operation to change the attitude of the teaching tool TL, which is included in the teaching information, based on an operator's operation. The MR device DV acquires information about the attitude of the teaching tool TL at the time when the operator operates a button on the teaching tool TL as the changed attitude. The MR device DV generates a virtual teaching tool VTL whose attitude has been changed based on the teaching information after the attitude change. The MR device DV deletes the virtual teaching tool VTL before the attitude change, and generates and displays a teaching image MXR2 in which the generated virtual teaching tool VTL after the attitude change is superimposed at the position of the teaching point based on the teaching information.

[0047] The teaching image MXR2 is an image generated after the posture of the teaching tool TL taught in the captured image WLD is changed. The teaching image MXR2 is an image on which the virtual teaching tool VTL after the posture change is superimposed.

[0048] As described above, the MR device DV in the present disclosure generates and displays a teaching image MXR1 in which a real-world teaching tool TL is replaced with a virtual teaching tool VTL in a mixed reality space. Furthermore, when the posture of the teaching information is changed by an operator's operation, the MR device DV generates and displays a teaching image MXR2 including the virtual teaching tool VTL after the posture change, thereby visualizing the posture of the teaching tool after the posture change to the operator.

[0049] Next, the attitude changed by the operator's operation will be described with reference to Fig. 4. Fig. 4 is a diagram comparing the attitude of the virtual teaching tool VTL11 before the attitude change and the attitude of the virtual teaching tool VTL12 after the attitude change.

[0050] The virtual teaching tool VTL11 corresponds to the attitude of the teaching tool TL before the attitude change when the teaching information before the attitude change was taught. The MR device DV calculates the TX axis, which is the angle of the virtual teaching tool VTL11, based on the attitude of the teaching information before the attitude change, and sets the calculated TX axis as the reference angle (= 0 (zero) °).

[0051] The MR device DV receives an operation by the operator to change the attitude of the virtual teaching tool VTL around the TX axis. In the example shown in Fig. 4, the MR device DV receives an attitude change operation to rotate the virtual teaching tool VTL11 by 30 degrees around the TX axis. After the attitude change, the MR device DV generates a virtual teaching tool VTL12 that is rotated 30 degrees around the TX axis from the attitude of the virtual teaching tool VTL11 (i.e., the reference angle).

[0052] <Posture change operation example 1> Next, a first example of an attitude change operation for the teaching point Pt1 will be described with reference to Fig. 5. Fig. 5 is a diagram illustrating the first example of an attitude change operation for the teaching point Pt1. Note that the example shown in Fig. 5 shows an example in which there is one teaching point, but is not limited to this.

[0053] 5, a teaching point Pt1 is taught to the workpiece Wk. The MR device DV generates a teaching point selection image MXR11 in which the teaching point Pt1 is superimposed on the workpiece Wk based on at least one piece of teaching information corresponding to the workpiece Wk (i.e., information on the teaching point Pt1), and displays the image on the display unit 13. The worker selects (presses) the teaching point Pt1 on the teaching point selection image MXR11.

[0054] The MR device DV recognizes the position and movement of the worker's fingers in the air (hereinafter referred to as "air operation") based on the captured image captured by the camera 15, and when it determines that the teaching point Pt1 on the teaching point selection image MXR11 has been selected, it generates at least one virtual button indicating the operation content for the teaching point Pt1. The MR device DV generates an operation content selection image MXR12 in which virtual buttons are further superimposed on the teaching point selection image MXR11, and displays it on the display unit 13. The virtual buttons here are an adjustment button BT11, a details button BT12, and a delete button BT13.

[0055] The adjustment button BT11 is a button that accepts adjustment (change) of the posture of the selected teaching point Pt1. When the adjustment button BT11 is selected (pressed) based on an aerial operation by the operator, the MR device DV starts accepting an operation to change the posture.

[0056] The detail button BT12 is a button that displays teaching information of the selected teaching point Pt1. When the detail button BT12 is selected (pressed) by the operator's mid-air operation, the MR device DV generates and displays an image on which the teaching information is further superimposed.

[0057] The delete button BT13 is a button for deleting the teaching information of the selected teaching point Pt1. When the delete button BT13 is selected (pressed) by an operator's mid-air operation, the MR device DV deletes the teaching point Pt1 from the image displayed on the display unit 13, and also generates a control command requesting deletion of the teaching information, transmits the control command to the processing device P1, and deletes the teaching information of the teaching point Pt1.

[0058] When the adjustment button BT11 is selected (pressed) by the worker through mid-air operation, the MR device DV generates a virtual teaching tool VTL11 corresponding to the teaching information of the selected teaching point Pt1 and operation buttons BT21, BT22, BT31, and BT32 that accept operations related to changing the attitude (angle) of this virtual teaching tool VTL11. The MR device DV generates a change operation image MXR13 in which the generated virtual teaching tool VTL11 and each of the operation buttons BT21, BT22, BT31, and BT32 are superimposed on a captured image of the real world, and displays the change operation image MXR13 on the display unit 13.

[0059] The operation button BT21 accepts an operation to rotate the virtual teaching tool VTL11 around the TX axis by +1° increments, and the operation button BT32 accepts an operation to rotate the virtual teaching tool VTL11 around the TX axis by -1° increments.

[0060] The operation button BT31 accepts an operation to rotate the virtual teaching tool VTL11 by 90° around the TX axis. The operation button BT32 accepts an operation to rotate the virtual teaching tool VTL11 by 180° around the TX axis. Note that the angle size changed by the operation buttons BT31 and BT32 (here, 90° and 180°) is just an example and is not limited to this.

[0061] When the MR device DV recognizes that any of the operation buttons BT21, BT22, BT31, and BT32 has been selected by the worker through mid-air operation, it generates a virtual teaching tool VTL12A by rotating the virtual teaching tool VTL11 by the angle corresponding to the selected operation button. The MR device DV displays a change operation image MXR13 including the virtual teaching tool VTL12A on the display unit 13.

[0062] 5 shows an example in which a virtual teaching tool VTL12A is displayed after the virtual teaching tool VTL11 has been rotated 175° around the TX axis by selecting the operation buttons BT21, BT22, BT31, and BT32. Furthermore, the change operation image MXR13 shown in Fig. 5 shows the virtual teaching tool VTL11 before the attitude change for ease of explanation, but the display of the virtual teaching tool VTL11 is not essential and may be deleted or may be displayed with a higher transparency than the virtual teaching tool VTL12A.

[0063] As described above, in posture change operation example 1, the MR device DV displays buttons (i.e., the adjust button BT11, the detail button BT12, the delete button BT13, and the operation buttons BT21, BT22, BT31, and BT32) that can accept mid-air operations by the worker on each of the operation content selection image MXR12 and the change operation image MXR13. The MR device DV changes the posture (angle) included in the teaching information by recognizing and accepting the worker's operations (mid-air operations) on the buttons displayed on the operation content selection image MXR12 and the change operation image MXR13 using the camera 15. The worker can easily change the posture of the teaching point Pt1 by operating the operation buttons displayed on the display unit 13 using mixed reality. This allows the MR device DV to support the worker in changing the posture of the teaching information.

[0064] <Posture change operation example 2> Next, a second example of the posture change operation of the teaching point Pt1 will be described with reference to Fig. 6. Fig. 6 is a diagram illustrating the second example of the posture change operation of the teaching point Pt1. Note that the example shown in Fig. 6 shows an example in which there is one teaching point, but is not limited to this. In addition, the marker Mk shown in Fig. 6 is, for example, a polyhedron having a cubic shape.

[0065] 6, a teaching point Pt1 is taught to the workpiece Wk. The MR device DV generates a teaching point selection image MXR11 in which the teaching point Pt1 is superimposed on the workpiece Wk based on at least one piece of teaching information corresponding to the workpiece Wk (i.e., information on the teaching point Pt1), and displays the image on the display unit 13. The worker selects (presses) the teaching point Pt1 on the teaching point selection image MXR11.

[0066] The MR device DV recognizes the aerial operation performed by the worker based on the captured image captured by the camera 15, and when it determines that the teaching point Pt1 on the teaching point selection image MXR11 has been selected, it generates a change operation image MXR22 and displays it on the display unit 13. The change operation image MXR22 is an image that can accept a posture change operation using a real-world teaching tool TL or marker Mk held by the worker, or a posture change operation by generating a virtual teaching tool VTL or virtual marker VMk and operating the virtual teaching tool VTL or virtual marker VMk.

[0067] The operator may set whether to use the real-world teaching tool TL, the real-world marker Mk, the virtual teaching tool VTL, or the virtual marker VMk to perform the change operation. Furthermore, a combination of the real-world teaching tool TL, the real-world marker Mk, the virtual teaching tool VTL, and the virtual marker VMk, which are tools for accepting the change operation, may be used. Based on this setting, the MR device DV generates a change operation image MXR22 that can accept a posture change operation using any of the real-world teaching tool TL, the real-world marker Mk, the virtual teaching tool VTL, or the virtual marker VMk, and displays the change operation image MXR22 on the display unit 13.

[0068] For example, when the MR device DV receives a posture change operation using a real-world teaching tool TL or a marker Mk held by the worker, the MR device DV detects the real-world teaching tool TL or the marker Mk from the captured image captured by the camera 15. Here, when the MR device DV receives a posture change operation using a real-world marker Mk or a virtual marker VMk, the MR device DV may read a two-dimensional code (e.g., a barcode or a QR code (registered trademark)) attached to each face of the cubic marker Mk or the virtual marker VMk to acquire the posture of the marker Mk or the virtual marker VMk. The MR device DV generates a change operation image MXR22 in which a virtual teaching tool VTL or a virtual marker VMk corresponding to the posture of the teaching information is superimposed on the real-world captured image in which the teaching tool TL or the marker Mk is captured.

[0069] Furthermore, for example, when the MR device DV accepts a posture change operation using a virtual teaching tool VTL or a virtual marker VMk, it generates a change operation image MXR22 in which the virtual teaching tool VTL or the virtual marker VMk corresponding to the posture of the teaching information is superimposed on a captured image of the real world. The MR device DV accepts a posture change operation corresponding to the rotation direction and rotation amount by calculating the rotation direction and rotation amount of the virtual teaching tool VTL or the virtual marker VMk based on the worker's aerial operation.

[0070] The MR device DV recognizes the posture of the real-world teaching tool TL, the real-world marker Mk, the virtual teaching tool VTL, or the virtual marker VMk changed by the worker based on the captured image captured by the camera 15. The MR device DV generates a virtual teaching tool VTL12B by rotating the virtual teaching tool VTL11 corresponding to the posture before the posture change based on the recognized posture after the change. The MR device DV displays a change operation image MXR23 including the virtual teaching tool VTL12B on the display unit 13.

[0071] 6 shows an example in which a virtual teaching tool VTL12B is displayed after the virtual teaching tool VTL11 has been rotated 180° around the TX axis based on the attitude change operation received by the change operation image MXR22. The change operation image MXR23 shown in Fig. 6 shows the virtual teaching tool VTL11 before the attitude change for ease of understanding, but the display of the virtual teaching tool VTL11 is not essential and may be deleted or may be displayed with a higher transparency than the virtual teaching tool VTL12B.

[0072] As described above, in posture change operation example 2, the MR device DV changes the posture (angle) included in the teaching information by accepting an operator's operation using one or more tools from the real-world teaching tool TL, the real-world marker Mk, the virtual teaching tool VTL, or the virtual marker VMk in the change operation image MXR22. The operator can easily change the posture of the teaching point Pt1 by changing the posture of any tool in the real world or mixed reality. This allows the MR device DV to support the operator in changing the posture of the teaching information.

[0073] <Posture change operation example 3> Next, a third example of the attitude change operation for the teaching point Pt1 will be described with reference to Fig. 7. Fig. 7 is a diagram illustrating a third example of the attitude change operation for the teaching point Pt1. Note that the example shown in Fig. 7 shows an example in which there is one teaching point, but is not limited to this.

[0074] 7, a teaching point Pt1 is taught to the workpiece Wk. The MR device DV generates a teaching point selection image MXR11 in which the teaching point Pt1 is superimposed on the workpiece Wk based on at least one piece of teaching information corresponding to the workpiece Wk (i.e., information on the teaching point Pt1), and displays it on the display unit 13. The worker performs an aerial operation to select (press) the teaching point Pt1 on the teaching point selection image MXR11.

[0075] When the MR device DV determines, based on the captured image captured by the camera 15, that the teaching point Pt1 on the teaching point selection image MXR11 has been selected by the operator through aerial manipulation, it generates a change operation image MXR32 and displays it on the display unit 13. The change operation image MXR32 superimposes the virtual teaching tool VTL11, whose orientation corresponds to the position and orientation of the teaching point Pt1, on the captured image of the real world. The change operation image MXR32 is an image that can accept rotation operation of the virtual teaching tool VTL11 around the TX axis through aerial manipulation by the operator. Note that the virtual teaching tool VTL11 displayed on the change operation image MXR32 is superimposed with its tip fixed at the position of the teaching point Pt1 and is generated so as to accept only rotation operation in the direction around the TX axis. This prevents the MR device DV from unintentionally changing the position of the teaching point Pt1 during an orientation change operation.

[0076] The worker grasps the virtual teaching tool VTL11 displayed on the change operation image MXR32 in the air and rotates it to a desired posture. The MR device DV rotates the virtual teaching tool VTL11 displayed on the change operation image MXR32 around the TX axis based on the worker's aerial operation shown in the captured image captured by the camera 15.

[0077] Note that the change operation image MXR33 shown in Figure 7 shows an example in which a virtual teaching tool VTL12B is displayed after rotating the virtual teaching tool VTL11 by 180 degrees around the TX axis based on the posture change operation received by the change operation image MXR32.

[0078] As described above, in posture change operation example 3, the MR device DV changes the posture (angle) included in the teaching information by accepting a posture change operation using the virtual teaching tool VTL11 in the change operation image MXR32. The worker can easily change the posture of the teaching point Pt1 by performing an aerial operation in mixed reality to grasp and rotate the virtual teaching tool VTL11. This allows the MR device DV to support the worker in changing the posture of the teaching information.

[0079] <Posture change operation example 4> Next, a fourth example of a posture change operation for a plurality of teaching points Pt1 to Pt3 will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining the fourth example of a posture change operation for the teaching points Pt1 to Pt3. Note that, in the example shown in Fig. 8, an example in which there are three teaching points is shown, but the present invention is not limited to this.

[0080] The workpiece Wk shown in Fig. 8 has three teaching points Pt1, Pt2, and Pt3 taught to it. The MR device DV generates a teaching point selection image MXR41 in which each of the teaching points Pt1 to Pt3 is superimposed on the workpiece Wk based on the teaching information corresponding to the workpiece Wk (i.e., information on the teaching points Pt1 to Pt3), and displays the image on the display unit 13. The worker selects (presses) at least one teaching point from the teaching points Pt1 to Pt3 on the teaching point selection image MXR41, the posture of which he or she desires to change. Fig. 8 illustrates an example in which three teaching points Pt1 to Pt3 are selected.

[0081] The MR device DV recognizes the positions and movements of the worker's fingers based on the captured image captured by the camera 15, and when it determines that teaching points Pt1 to Pt3 on the teaching point selection image MXR11 have been selected, it generates a change operation image MXR42 including a change button BT41 that makes it possible to collectively change the postures corresponding to each of the selected teaching points Pt1 to Pt3, and displays this on the display unit 13. When the change button BT41 is selected (pressed) by the worker, the MR device DV generates a change operation image MXR42 in which virtual teaching tools VTL11, VTL21, and VTL31 in postures corresponding to the positions and postures of the selected teaching points Pt1 to Pt3 are superimposed on the captured image of the real world, and displays this on the display unit 13.

[0082] The operator performs one of the above-described posture change operation examples 1 to 3, and rotates one of the virtual teaching tools displayed in the change operation image MXR42 to a desired posture. Note that Fig. 8 omits illustration and description of the selection process for which posture change operation to use to perform the posture change operation.

[0083] Based on the operator's operation, the MR device DV simultaneously rotates all of the virtual teaching tools VTL11, VTL21, and VTL31 displayed on the change operation image MXR42 around the TX axis so that they assume the same posture.

[0084] Note that the change operation image MXR43 shown in Figure 8 shows an example in which virtual teaching tools VTL12D, VTL22, and VTL32 are displayed after rotating the virtual teaching tools VTL11, VTL21, and VTL31T by 180 degrees around the X axis based on the posture change operation received by the change operation image MXR42.

[0085] As described above, in posture change operation example 4, the MR device DV changes the postures (angles) included in the plurality of pieces of teaching information by accepting the selection operation of the plurality of teaching points. Since the worker can change the postures of the plurality of pieces of teaching information at once, the time required for changing the postures of the plurality of teaching points can be further reduced. This allows the MR device DV to support the worker in changing the postures of the plurality of pieces of teaching information.

[0086] <Posture change operation example 5> Next, a fifth example of a posture change operation for teaching points Pt1 to Pt3 will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining a fifth example of a posture change operation for teaching points Pt1 to Pt3. Note that, in the example shown in Fig. 9, an example in which there are three teaching points is shown, but the present invention is not limited to this.

[0087] 9 shows a state in which the teaching point (teaching information) to be the target of the posture change is not registered, that is, not taught. The worker teaches the posture with respect to the virtual teaching point Pt0 by performing one of the posture change operations of the posture change operation examples 1 to 3 described above.

[0088] The MR device DV generates a virtual teaching tool VTL0 corresponding to the taught posture. The MR device DV generates a posture teaching image MXR51 by superimposing the virtual teaching tool VTL51 on a captured image of the workpiece Wk in the real world, and displays it on the display unit 13. The MR device DV records information on the taught posture.

[0089] After completing the teaching of the posture, the MR device DV accepts a teaching operation by the operator for the position of at least one of the teaching points Pt1 to Pt3. The teaching of the teaching points may be performed by any method. The MR device DV generates each of the taught teaching points Pt1 to Pt3 and virtual teaching tools VTL11, VTL21, and VTL31 corresponding to the posture of the teaching tool at the time when each of the teaching points Pt1 to Pt3 was taught. The MR device DV generates a teaching image MXR52 by superimposing the teaching points Pt1 to Pt3 and the virtual teaching tools VTL11, VTL21, and VTL31 on an image captured by the camera 15, and displays the generated image on the display unit 13.

[0090] Based on the teaching points taught by the operator, the MR device DV generates virtual teaching tools VTL12E, VTL22E, and VTL32E by changing the postures of the virtual teaching tools VTL11, VTL21, and VTL31 corresponding to each of the teaching points Pt1 to Pt3 to the posture defined by the virtual teaching tool VTL0.

[0091] Here, the MR device DV may change the attitude of the virtual teaching tool to the attitude defined by the virtual teaching tool VTL0 each time a teaching point is taught, or may change the attitude of the virtual teaching tool corresponding to all taught teaching points to the attitude defined by the virtual teaching tool VTL0 when it determines that teaching of all teaching points has been completed.

[0092] The MR device DV generates a change operation image MXR53 by superimposing the teaching points Pt1 to Pt3 and the virtual teaching tools VTL12E, VTL22E, and VTL32E on the captured image captured by the camera 15, and displays it on the display unit 13.

[0093] As described above, in posture change operation example 5, the MR device DV can change (set) the posture of at least one teaching point taught to the workpiece Wk to the same posture by accepting a teaching operation that teaches only the posture in advance. Since the worker can change (set) the postures of the teaching information to be taught thereafter all at once by teaching the posture once, the time required for changing the postures of multiple teaching points can be particularly reduced. This allows the MR device DV to support the worker in changing the posture of the teaching information.

[0094] <Welding teaching system operation procedure> Next, an example of an operation procedure of the MR device DV in the embodiment will be described with reference to Fig. 10 to Fig. 12. Fig. 10 is a flowchart showing an example of an overall operation procedure of the MR device DV in the embodiment. Fig. 11 is a flowchart showing an example of a procedure for newly registering teaching information of the MR device DV in the embodiment. Fig. 12 is a flowchart showing an example of a procedure for changing the posture in teaching information of the MR device DV in the embodiment.

[0095] The MR device DV receives a selection operation of the welding robot RB to be taught (St11). Based on the operator's operation, the MR device DV determines whether the teaching work for the selected welding robot RB is new creation of teaching information (St12). Note that the operator's operation may be received by the processing device P1 or by an image (not shown) displayed on the display unit 13 of the MR device DV.

[0096] When the MR device DV determines in step St12 that the teaching work for the selected welding robot RB is new creation of teaching information (St12, YES), it executes a process of generating teaching information (St13).

[0097] <Generation of teaching information> An example of the operation procedure in the generation process of teaching information will now be described. The worker wears the MR device DV, holds the virtual teaching tool VTL, and starts teaching work on the workpiece Wk.

[0098] The MR device DV is in a standby state until an operator selects (presses) a button (not shown) provided on the teaching tool TL or a virtual button (not shown) superimposed on an image captured by the camera 15 and displayed on the display unit 13, that is, until teaching information is acquired (St131).

[0099] When an operator selects (presses) a button (not shown) or a virtual button (not shown) provided on the teaching tool TL, the MR device DV acquires teaching information (i.e., information on the position and posture of the teaching point) at the time of the operator operation and adds (records) it as new teaching information corresponding to the workpiece Wk (St132).

[0100] The MR device DV generates a virtual teaching tool VTL with a posture corresponding to the teaching information, generates a teaching image MXR1 (mixed reality space, see Figure 3) superimposed on the captured image captured by the camera 15, and displays it on the display unit 13 (St133).

[0101] If the MR device DV determines in step St12 that the teaching work for the selected welding robot RB is not new creation of teaching information (St12, NO), it determines based on the operator's operation whether the teaching work for the selected welding robot RB is adjustment of teaching information (St14).

[0102] When the MR device DV determines in step St14 that the teaching work for the selected welding robot RB is adjustment of teaching information (St14, YES), it executes the adjustment processing of teaching information (St15).

[0103] <Adjustment of teaching information> An example of an operation procedure in the adjustment process of the teaching information will now be described. The worker wears the MR device DV, holds the virtual teaching tool VTL, and starts the adjustment work (posture change work) on the workpiece Wk.

[0104] The MR device DV accepts an operator's selection operation of at least one teaching point whose teaching information is to be changed (adjusted). Based on the teaching information of the selected teaching point, the MR device DV sets the current teaching posture corresponding to the teaching point to the reference angle (=0 (zero)°) around the TX axis (St151).

[0105] The MR device DV accepts a posture change operation according to any one of posture change operation examples 1 to 4, and acquires information about the posture after the change (St152).

[0106] The MR device DV recalculates the posture of the teaching information based on the changed posture, and changes and records the posture information included in the teaching information (St153).

[0107] The MR device DV generates a virtual teaching tool VTL with a posture corresponding to the changed teaching information, generates a teaching image MXR2 (mixed reality space, see Figure 3) superimposed on the image captured by the camera 15, and displays it on the display unit 13 (St154).

[0108] When the MR device DV determines in step St14 that the teaching work for the selected welding robot RB is not adjustment of teaching information (St14, NO), it ends the flow shown in Fig. 10. The MR device DV records the added teaching information or the teaching information whose posture has been changed, or transmits it to the processing device P1 to be recorded.

[0109] Here, when performing a posture change operation according to posture change operation example 5, the MR device DV may accept a posture teaching operation using the method shown in Fig. 9 before performing the teaching information generation process in step St13. In this case, after the teaching information generation process in step St13, the MR device DV automatically performs a posture adjustment process, and changes (adjusts) the taught posture (for example, the posture of the virtual teaching tool VTL11, see Fig. 9) to a posture taught in advance (for example, the posture of the virtual teaching tool VTL0, see Fig. 9), and records the changed posture.

[0110] As described above, welding teaching system 100 according to the embodiment can change (adjust) the posture of at least one teaching point to be taught to workpiece Wk by the methods shown in posture change operation examples 1 to 5. Even if it is difficult for the worker to teach the teaching point in the posture to be taught based on the welding environment, i.e., the position or arrangement of workpiece Wk and welding robot RB relative to workpiece Wk, or the positional relationship of the teaching point relative to workpiece Wk, the worker can more easily change (adjust) the posture of a taught or untaught teaching point. This allows welding teaching system 100 according to the embodiment to support the worker's teaching of teaching information and posture change work.

[0111] (Addendum) The above description of each embodiment discloses the following techniques.

[0112] (Technology 1) a model storage unit (memory 12) that stores a three-dimensional model corresponding to at least a part of a robot (welding robot RB) existing in a real environment (i.e., the real world) or a welding torch TC used for welding, or a teaching member (teaching tool TL or marker Mk) used for teaching the robot (welding robot RB); an operation screen storage unit (memory 12) for storing operation screen data (for example, operation buttons BT21, BT22, BT31, BT32, change button BT41, or virtual marker VMk) corresponding to an operation screen used for displaying and operating the three-dimensional model; a display device (display unit 13) configured to be wearable by a worker and configured to display an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit (depth sensor 14 or camera 15) that acquires a relative positional relationship between the real environment, the teaching member (teaching tool TL or marker Mk), and the display device (display unit 13); an image generation unit (processor 11) that generates display images (change operation images MXR13, MXR22) for displaying the three-dimensional model and the operation screen so as to have a predetermined positional relationship with respect to the display device (display unit 13) based on the relative positional relationship, the three-dimensional model, and the operation screen data; an output unit (processor 11) that outputs the display image to the display device (display unit 13); a detection unit (camera 15) that detects an aerial operation that is an operation performed by the worker in the air away from the display device (display unit 13) on the operation screen displayed on the display device (display unit 13), the image generation unit (processor 11) generates the display image for displaying a modified three-dimensional model in which the posture of the three-dimensional model has been modified based on the aerial operation; Robot teaching system (MR Device DV). With this configuration, the MR device DV displays change operation images MXR13, MXR22 on the display unit 13 using mixed reality, and can easily change (teach) the posture of the teaching tool TL at the teaching point, i.e., the welding torch TC when welding the teaching point, based on the worker's aerial operation detected by the camera 15 and the image displayed on the display unit 13. In addition, the MR device DV displays the virtual teaching tool VTL after the posture change, making it easier to visually confirm whether the posture after the posture change is the posture desired by the worker, thereby supporting the worker in changing the posture of the teaching information.

[0113] (Technology 2) The operation screen includes virtual buttons (operation buttons BT21, BT22, BT31, and BT32) that can be operated by the mid-air operation, When the virtual buttons (operation buttons BT21, BT22, BT31, BT32) are operated by the mid-air operation, the image generation unit (processor 11) generates the display image for displaying the changed 3D model in which the attitude of the 3D model is changed according to the operation amount of the virtual buttons. A robot teaching system (MR device DV) described in (Technology 1). With this configuration, the MR device DV displays change operation images MXR13 and MXR22 on the display unit 13 using mixed reality, and can easily change (teach) the posture of the teaching tool TL at the teaching point, i.e., the welding torch TC when welding the teaching point, based on the worker's aerial operation detected by the camera 15 and the images of the operation buttons BT21, BT22, BT31, and BT32 displayed on the display unit 13. In addition, the MR device DV displays a virtual teaching tool VTL after the posture change, making it easier to visually confirm whether the posture after the posture change is the posture desired by the worker, thereby supporting the worker in changing the posture of the teaching information.

[0114] (Technology 3) the operation screen is a virtual polyhedron (virtual marker VMk) that can be operated by the mid-air operation, when the virtual polyhedron (virtual marker VMk) is operated by the mid-air operation, the image generation unit (processor 11) generates the display image for displaying the changed 3D model in which the posture of the 3D model is changed according to the amount of operation of the virtual polyhedron (virtual marker VMk). A robot teaching system (MR device DV) according to (Technology 1) or (Technology 2). With this configuration, the MR device DV receives a worker's operation (air operation) on the virtual marker VMk displayed on the display unit 13 by mixed reality, and can easily change (teach) the posture of the teaching tool TL at the teaching point, i.e., the welding torch TC when welding the teaching point, based on the operation amount (i.e., rotation amount) of the virtual marker VMk. Furthermore, the MR device DV displays the virtual teaching tool VTL after the posture change, making it easier to visually confirm whether the posture after the posture change is the posture desired by the worker, thereby supporting the worker in changing the posture of the teaching information.

[0115] (Technology 4) The robot further includes a start detection unit (camera 15) that detects a first start operation (for example, an operation of selecting a teaching point Pt1 that is a posture change target) in which a predetermined aerial operation performed by the worker is detected by the detection unit (camera 15), or a second start operation in which the teaching member (for example, a real-world or virtual teaching tool, or a real-world or virtual marker, etc.) is operated by the worker, the image generation unit (processor 11) generates the display image for displaying the operation screen after the first start action or the second start action is detected by the start detection unit (camera 15); A robot teaching system (MR device DV) according to any one of (Technology 1) to (Technology 3). With this configuration, when the MR device DV detects an operator's operation (air operation) on the screen displayed on the display unit 13 by mixed reality, it can generate and output change operation images MXR13 and MXR22 that can accept a posture change operation. This allows the MR device DV to support the operator's posture change operation.

[0116] (Technology 5) The positional relationship acquisition unit (depth sensor 14 or camera 15) acquires the position of the teaching member (teaching tool TL or marker Mk) as a reference position (i.e., reference angle) when the first start operation or the second start operation is detected by the start detection unit (camera 15). A robot teaching system (MR device DV) described in (Technology 4). With this configuration, the MR device DV accepts an operation to change the twist angle of the virtual teaching tool VTL with the posture of the virtual teaching tool VTL before the posture change as the reference angle (=0 (zero)°) and the posture of the virtual teaching tool VTL before the posture change as the center of rotation. This allows the MR device DV to support the posture change operation performed by the worker.

[0117] (Technology 6) The robot (welding robot RB) is a welding robot (welding robot RB) having a wire feeder (WW1) that feeds a welding wire WW. A robot teaching system (MR device DV) according to any one of (Technology 1) to (Technology 5). With this configuration, the MR device DV can assist the welding robot that welds the workpiece Wk using the welding wire WW in teaching the welding operation to the worker.

[0118] (Technology 7) The changed three-dimensional model has a shape corresponding to the three-dimensional model of the robot (welding robot RB) or the teaching member (teaching tool TL or marker Mk) rotated by a predetermined angle around an axis (TX axis) along the feed direction of the welding wire WW. A robot teaching system (MR device DV) described in (Technology 6). With this configuration, the MR device DV can receive instruction on the twist angle of the welding torch TC during welding, which is an angle with the TX axis as the center of rotation in the posture of the teaching point, and after the twist angle is changed, it can assist the worker in changing the posture by displaying a virtual teaching tool VTL corresponding to the changed twist angle.

[0119] (Technology 8) the predetermined angle in the modified three-dimensional model is determined in accordance with the aerial operation performed by the worker. A robot teaching system (MR device DV) described in (Technology 7). With this configuration, the MR device DV can accept a posture change operation by a worker without an actual tool (a real-world teaching tool TL, a welding robot RB, or a welding torch TC).

[0120] (Technology 9) a teaching information storage unit (memory 12) that stores teaching information including a teaching position taught by the operator and orientation information of the three-dimensional model at the teaching position, The image generation unit (processor 11) generates the display image that displays the 3D model corresponding to the teaching information and the operation screen, generates changed teaching information corresponding to the changed 3D model, and stores the changed teaching information in the teaching information storage unit (memory 12). A robot teaching system (MR device DV) according to any one of (Technology 1) to (Technology 8). With this configuration, the MR device DV can change the posture of the taught teaching point and store the changed posture, thereby assisting the operator in changing the posture.

[0121] (Technology 10) a model storage unit (memory 12) that stores a three-dimensional model corresponding to at least a part of a robot (welding robot RB) existing in a real environment (i.e., the real world) or a welding torch TC used for welding, or a teaching member (teaching tool TL or marker Mk) used for teaching the robot (welding robot RB); a display device (display unit 13) configured to be wearable by a worker and configured to display an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit (depth sensor 14 or camera 15) that acquires a relative positional relationship between the real environment, the teaching member (teaching tool TL or marker Mk), and the display device (display unit 13); an image generation unit (processor 11) that generates a display image (change operation images MXR32, MXR42) for displaying the three-dimensional model so as to have a predetermined positional relationship with respect to the display device (display unit 13) based on the relative positional relationship and the three-dimensional model; an output unit that outputs the display image to the display device (display unit 13); a detection unit (camera 15) that detects an aerial operation that is an operation performed by a worker in the air away from the display device (display unit 13) on the three-dimensional model displayed on the display device (display unit 13), the image generation unit (processor 11) generates the display image for displaying a modified three-dimensional model in which the posture of the three-dimensional model has been modified based on the aerial operation; Robot teaching system (MR Device DV). With this configuration, the MR device DV displays change operation images MXR32, MXR42 on the display unit 13 using mixed reality, and can easily change (teach) the posture of the teaching tool TL at the teaching point, i.e., the welding torch TC when welding the teaching point, based on the worker's aerial operation detected by the camera 15 and the image of the virtual teaching tool VTL displayed on the display unit 13. In addition, the MR device DV displays the virtual teaching tool VTL after the posture change, making it easier to visually confirm whether the posture after the posture change is the posture desired by the worker, thereby supporting the worker in changing the posture of the teaching information.

[0122] (Technology 11) The robot (welding robot RB) is a welding robot (welding robot RB) having a wire feeder WW1 that feeds a welding wire WW. A robot teaching system (MR device DV) described in (Technology 10).

[0123] (Technology 12) The changed three-dimensional model has a shape corresponding to the three-dimensional model of the robot (welding robot RB) or the teaching member (teaching tool TL or marker Mk) rotated by a predetermined angle around an axis along the feed direction of the welding wire WW. A robot teaching system (MR device DV) according to (Technology 10) or (Technology 11).

[0124] (Technology 13) a teaching information storage unit (memory 12) that stores teaching information including a teaching position taught by the operator and orientation information of the three-dimensional model at the teaching position, The image generation unit (processor 11) generates the display image that displays the 3D model corresponding to the teaching information, generates changed teaching information corresponding to the changed 3D model, and stores the changed teaching information in the teaching information storage unit (memory 12). A robot teaching system (MR device DV) according to any one of (Technique 10) to (Technique 12). With this configuration, the MR device DV can change the posture of the taught teaching point and store the changed posture, thereby assisting the operator in changing the posture.

[0125] (Technology 14) A robot teaching method performed by a system (MR device DV) including at least one computer (processor 11), storing a three-dimensional model corresponding to at least a part of a robot (welding robot RB) or a welding torch used for welding that exists in a real environment (i.e., the real world), or a teaching member (teaching tool TL or marker Mk) used to teach the robot (welding robot RB), and operation screen data (e.g., operation buttons BT21, BT22, BT31, BT32, change button BT41, or virtual marker VMk, etc.) corresponding to an operation screen used to display and operate the three-dimensional model; acquire a relative positional relationship between the real environment, the teaching member (teaching tool TL or marker Mk), and a display device (display unit 13) configured to be wearable by a worker and displaying an image of the real environment or the real environment itself superimposed thereon; based on the relative positional relationship, the three-dimensional model, and the operation screen data, generate display images (change operation images MXR13, MXR22) for displaying the three-dimensional model and the operation screen so as to have a predetermined positional relationship with respect to the display device (display unit 13), and display the display images on the display device (display unit 13); Detecting an aerial operation, which is an operation performed by the worker in the air away from the display device (display unit 13) on the operation screen displayed on the display device (display unit 13), generating the display image for displaying a modified three-dimensional model in which the posture of the three-dimensional model has been modified based on the aerial manipulation; Robot teaching method. With this configuration, the MR device DV that executes the robot teaching method displays change operation images MXR13, MXR22 on the display unit 13 using mixed reality, and can easily change (teach) the posture of the teaching tool TL at the teaching point, i.e., the welding torch TC when welding the teaching point, based on the worker's aerial operation detected by the camera 15 and the image displayed on the display unit 13. In addition, the MR device DV displays the virtual teaching tool VTL after the posture change, making it easier to visually confirm whether the posture after the posture change is the posture desired by the worker, thereby supporting the worker in changing the posture of the teaching information.

[0126] (Technology 15) A robot teaching method performed by a system (MR device DV) including at least one computer (processor 11), storing a three-dimensional model corresponding to at least a part of a robot (welding robot RB) or a welding torch TC used for welding that exists in a real environment (i.e., the real world), or a teaching member (teaching tool TL or marker Mk) used to teach the robot (welding robot RB); acquire a relative positional relationship between the real environment, the teaching member (teaching tool TL or marker Mk), and a display device (display unit 13) configured to be wearable by a worker and displaying an image of the real environment or the real environment itself superimposed thereon; based on the relative positional relationship and the three-dimensional model, generate display images (change operation images MXR32, MXR42) for displaying the three-dimensional model so as to have a predetermined positional relationship with respect to the display device (display unit 13), and display the display images on the display device (display unit 13); Detecting an aerial operation that is an operation performed by the worker in the air away from the display device (display unit 13) on the three-dimensional model displayed on the display device (display unit 13), generating the display image for displaying a modified three-dimensional model in which the posture of the three-dimensional model has been modified based on the aerial manipulation; Robot teaching method. With this configuration, the MR device DV that executes the robot teaching method displays change operation images MXR32, MXR42 on the display unit 13 using mixed reality, and can easily change (teach) the posture of the teaching tool TL at the teaching point, i.e., the welding torch TC when welding the teaching point, based on the worker's aerial operation detected by the camera 15 and the image of the virtual teaching tool VTL displayed on the display unit 13. In addition, the MR device DV displays the virtual teaching tool VTL after the posture change, making it easier to visually confirm whether the posture after the posture change is the posture desired by the worker, thereby supporting the worker in changing the posture of the teaching information.

[0127] Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components of the various embodiments described above may be combined in any manner without departing from the spirit of the invention. [Industrial Applicability]

[0128] The present disclosure is useful as a robot teaching system and a robot teaching method that assist in teaching the posture of a welding torch at a teaching point in teaching a welding robot operation. [Explanation of symbols]

[0129] 10,20 Communications Department 11,21 processor 12,22 memory 13 Display section 14 Depth Sensor 15 Camera 100 Welding Teaching System BT11 adjustment button BT21, BT22, BT31, BT32 operation buttons BT41 Change Button DV MR device Mk Marker MXR1, MXR2, MXR52 teaching image MXR11,MXR41 Teaching point selection image MXR12 operation content selection image MXR13,MXR22,MXR23,MXR32,MXR33,MXR42,MXR43,MXR53 change operation image MXR51 posture teaching image P1 Processing Unit Pt0 Virtual teaching point Pt1, Pt2, Pt3 teaching points RB welding robot TL Teaching Tool VMk Virtual Marker VRB Virtual Welding Robot VTL, VTL0, VTL11, VTL12 Virtual teaching tools Wk Work WLD image capture

Claims

1. a model storage unit that stores a three-dimensional model corresponding to at least a part of a robot or a welding torch used in welding that exists in a real environment, or a teaching member used to teach the robot; an operation screen storage unit that stores operation screen data corresponding to an operation screen used to display and operate the three-dimensional model; a display device configured to be wearable by a worker and displaying an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit that acquires a relative positional relationship between the real environment, the teaching member, and the display device; an image generation unit that generates a display image for displaying the three-dimensional model and the operation screen so as to have a predetermined positional relationship with respect to the display device, based on the relative positional relationship, the three-dimensional model, and the operation screen data; an output unit that outputs the display image to the display device; a detection unit that detects an aerial operation that is an operation performed by the worker in the air away from the display device on the operation screen displayed on the display device, the image generation unit generates the display image for displaying a modified three-dimensional model obtained by modifying the posture of the three-dimensional model based on the aerial operation. Robot teaching system.

2. the operation screen includes virtual buttons operable by the mid-air operation, when the virtual button is operated by the mid-air operation, the image generation unit generates the display image for displaying the changed three-dimensional model in which the attitude of the three-dimensional model is changed in accordance with the operation amount of the virtual button. The robot teaching system according to claim 1 .

3. the operation screen is a virtual polyhedron operable by the mid-air operation, when the virtual polyhedron is operated by the aerial operation, the image generation unit generates the display image for displaying the changed three-dimensional model in which the posture of the three-dimensional model is changed according to the amount of operation of the virtual polyhedron. The robot teaching system according to claim 1 .

4. a start detection unit that detects a first start operation in which a predetermined aerial operation performed by the operator is detected by the detection unit, or a second start operation in which the teaching member is operated by the operator; the image generation unit generates the display image for displaying the operation screen after the start detection unit detects the first start action or the second start action. The robot teaching system according to claim 1 .

5. the positional relationship acquisition unit acquires, as a reference position, a position of the teaching member when the first start motion or the second start motion is detected by the start detection unit; The robot teaching system according to claim 4 .

6. The robot is a welding robot having a wire feeding unit that feeds a welding wire. The robot teaching system according to any one of claims 1 to 5.

7. the changed three-dimensional model has a shape corresponding to the three-dimensional model of the robot or the teaching member rotated by a predetermined angle around an axis along the feeding direction of the welding wire. The robot teaching system according to claim 6 .

8. the predetermined angle in the modified three-dimensional model is determined in accordance with the aerial operation performed by the worker. The robot teaching system according to claim 7 .

9. a teaching information storage unit that stores teaching information including information on the teaching position of the welding and information on the posture of the three-dimensional model at the teaching position, the image generation unit generates the display image that displays the three-dimensional model corresponding to the teaching information and the operation screen, generates changed teaching information corresponding to the changed three-dimensional model, and stores the changed teaching information in the teaching information storage unit; The robot teaching system according to claim 1 .

10. a model storage unit that stores a three-dimensional model corresponding to at least a part of a robot or a welding torch used in welding that exists in a real environment, or a teaching member used to teach the robot; a display device configured to be wearable by a worker and displaying an image superimposed on an image of the real environment or the real environment itself; a positional relationship acquisition unit that acquires a relative positional relationship between the real environment, the teaching member, and the display device; an image generation unit that generates a display image for displaying the three-dimensional model in a predetermined positional relationship with respect to the display device, based on the relative positional relationship and the three-dimensional model; an output unit that outputs the display image to the display device; a detection unit that detects an aerial operation that is an operation performed by a worker in the air away from the display device on the three-dimensional model displayed on the display device, the image generation unit generates the display image for displaying a modified three-dimensional model obtained by modifying the posture of the three-dimensional model based on the aerial operation. Robot teaching system.

11. The robot is a welding robot having a wire feeding unit that feeds a welding wire. The robot teaching system according to claim 10.

12. the changed three-dimensional model has a shape corresponding to the three-dimensional model of the robot or the teaching member rotated by a predetermined angle around an axis along the feeding direction of the welding wire. The robot teaching system according to claim 11.

13. a teaching information storage unit that stores teaching information including a teaching position taught by the operator and orientation information of the three-dimensional model at the teaching position, the image generation unit generates the display image displaying a three-dimensional model corresponding to the teaching information, generates changed teaching information corresponding to the changed three-dimensional model, and stores the changed teaching information in the teaching information storage unit; The robot teaching system according to claim 10.

14. A robot teaching method performed by a system including at least one computer, comprising: storing a three-dimensional model corresponding to at least a part of a robot or a welding torch used for welding that exists in a real environment, or a teaching member used to teach the robot, and operation screen data corresponding to an operation screen used to display and operate the three-dimensional model; acquire a relative positional relationship between the real environment, the teaching member, and a display device configured to be wearable by a worker and configured to display an image superimposed on an image of the real environment or the real environment itself; generating a display image for displaying the three-dimensional model and the operation screen so as to have a predetermined positional relationship with respect to the display device based on the relative positional relationship, the three-dimensional model, and the operation screen data, and displaying the generated display image on the display device; detecting an aerial operation that is an operation performed by the worker in the air away from the display device on the operation screen displayed on the display device; generating the display image for displaying a modified three-dimensional model obtained by changing the posture of the three-dimensional model based on the aerial manipulation; Robot teaching method.

15. A robot teaching method performed by a system including at least one computer, comprising: storing a three-dimensional model corresponding to at least a part of a robot or a welding torch used in welding that exists in a real environment, or a teaching member used to teach the robot; acquire a relative positional relationship between the real environment, the teaching member, and a display device configured to be wearable by a worker and configured to display an image superimposed on an image of the real environment or the real environment itself; generating a display image for displaying the three-dimensional model in a predetermined positional relationship with respect to the display device based on the relative positional relationship and the three-dimensional model, and displaying the display image on the display device; detecting an aerial operation that is an operation performed by the worker in the air away from the display device on the three-dimensional model displayed on the display device; generating the display image for displaying a modified three-dimensional model obtained by changing the posture of the three-dimensional model based on the aerial manipulation; Robot teaching method.

Images