Robot teaching system and robot teaching method

Abstract

The invention provides a robot teaching system and a robot teaching method. A robot teaching system stores a three-dimensional model of a robot, teaching data of the robot, and operation image data used in a display operation of the three-dimensional model, and acquires a relative positional relationship between an actual environment and a display device that displays an image of the actual environment or the actual environment itself in an overlapping manner. A display image is generated for displaying the three-dimensional model and the operation image so that the three-dimensional model and the operation image have a predetermined positional relationship with respect to the display device, and is output to the display device. The display image is used for displaying the three-dimensional model in a posture corresponding to the posture of the robot specified by the aerial operation performed by the operator in the motion trajectory of the robot between the first teaching point and the second teaching point.

Description

Technical Field

[0001] The present disclosure relates to a robot teaching system and a robot teaching method. Background Art

[0002] Patent Document 1 discloses an offline teaching device comprising: a display unit for displaying a teaching program and a model diagram; a storage unit for storing the instructions constituting the teaching program and the model data of the model diagram; and a control unit for controlling the display unit and the storage unit. In the offline teaching device, the teaching program includes a position detection program consisting of a plurality of position detection instructions and a welding program consisting of a plurality of welding instructions. The model diagram displays the motion trajectory of a robot when executing the teaching program, and the model diagram also displays a portion of the plurality of position detection instructions and a portion of the plurality of welding instructions.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2016 / 021130 Summary of the Invention

[0006] An object of the present disclosure is to provide a robot teaching system and a robot teaching method that can more easily simulate the motion of a welding robot being taught.

[0007] The present disclosure provides a robot teaching system comprising: a model storage unit for storing a three-dimensional model corresponding to a robot existing in an actual environment; a teaching data storage unit for storing teaching data of the robot; an operation image storage unit for storing operation image data corresponding to the operation image used in the display operation of the three-dimensional model; a display device configured to be wearable by an operator and to display an image superimposed with an image of the actual environment or the actual environment itself; a positional relationship acquisition unit for acquiring the relative positional relationship between the actual environment and the display device; and an image generation unit for generating a display image based on the relative positional relationship, the three-dimensional model, and the operation image data. The display image is used to display the three-dimensional model and the operation image so that they are in a given positional relationship relative to the display device. The output unit outputs the display image to the display device; and the detection unit detects the operation performed by the operator on the operation image displayed on the display device in the air away from the display device, i.e., the aerial operation. The image generation unit generates the display image, and the display image is used to display the three-dimensional model in a posture corresponding to the posture of the robot specified by the aerial operation, among the posture changes of the robot in the movement trajectory of the robot between the first teaching point and the second teaching point included in the teaching data.

[0008] The present disclosure also provides a robot teaching method, performed by a system including at least one computer. In the robot teaching method, a three-dimensional model corresponding to a robot existing in a real environment and teaching data for the robot are stored, and operation image data corresponding to an operation image used in displaying the three-dimensional model is stored. A relative positional relationship between the real environment and a display device is acquired, the display device being wearable by an operator and displaying an image superimposed with an image of the real environment or the real environment itself. A display image is generated based on the relative positional relationship, the three-dimensional model, and the operation image data. The display image is used to display the three-dimensional model and the operation image so as to have a predetermined positional relationship relative to the display device. The display image is output to the display device. An operation performed by the operator in mid-air, away from the display device, i.e., an aerial operation, on the operation image displayed on the display device is detected. The display image is generated to display the three-dimensional model in a posture corresponding to the posture of the robot specified by the aerial operation, among posture changes of the robot in a motion trajectory of the robot between a first teaching point and a second teaching point included in the teaching data.

[0009] According to the present disclosure, the motion of a taught welding robot can be simulated more easily. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a diagram illustrating an example of a welding teaching system according to an embodiment.

[0011] Figure 2 It is a diagram showing an example of the internal structure of an MR device and a processing device.

[0012] Figure 3 This figure explains a teaching example of welding operation.

[0013] Figure 4 is a diagram showing an example of a simulation image.

[0014] Figure 5 This is a diagram showing an example of switching of the playback menu.

[0015] Figure 6 This is a flowchart showing an example of the operation sequence of the MR device in the embodiment.

[0016] Figure 7 It shows Figure 4 The flowchart of an example of the operation sequence in each event of the MR device is shown.

[0017] Figure 8 FIG. 1 is a diagram showing an example of a disturbance notification image.

[0018] Explanation of symbols

[0019] 10, 20 Ministry of Communications

[0020] 11, 21 processors

[0021] 12, 22 Memory

[0022] 13 Display

[0023] 14 Depth Sensor

[0024] 15 cameras

[0025] 100 Welding Action Simulation System

[0026] 221 Teaching information recording unit

[0027] 222 Workpiece information recording unit

[0028] AP1, AP2, AP3, WP1, WP2, WP3, WP4 teaching points

[0029] BT1 Playback Button

[0030] BT2 stop button

[0031] DV MR equipment

[0032] IF interference area

[0033] Mk1 reference position acquisition mark

[0034] P1 Processing Unit

[0035] PS Reproduction Position

[0036] RM, RM1, RM2 playback menu

[0037] SB virtual drag bar

[0038] SC11 simulated image

[0039] SC12 Interference Notification Image

[0040] SD Virtual Slider

[0041] T0 playback start position

[0042] T1 reproduction end position

[0043] VRB welding robot

[0044] Wk, Wk0 workpiece. DETAILED DESCRIPTION

[0045] (The process of realizing this disclosure)

[0046] Traditionally, teaching a welding robot welding motions requires not only checking whether the taught position or angle is correct, but also verifying whether the welding robot is interfering with surrounding equipment (e.g., production equipment such as jigs) or the workpiece being welded. However, when teaching while moving the actual welding robot from a first teaching point to a second teaching point, there is a risk that the welding robot will interfere with surrounding equipment or the workpiece, necessitating both simulation-based verification and interference verification. Furthermore, when teaching while moving the actual welding robot, there is the issue of being unable to verify interference before the welding robot is positioned at the production site where welding is to take place.

[0047] However, conventional simulation-based verification processes involve using a 3D model of the welding robot to perform welding operations (simulation) on a pre-created 3D model of the workpiece to verify interference between the welding robot and surrounding equipment or the workpiece. Consequently, the operator performing the teaching process must create a 3D model of the welding robot and its surroundings (equipment), or a 3D model of the workpiece, which is quite cumbersome.

[0048] Therefore, in the following embodiments, a robot teaching system and a robot teaching method are described that make it easier to simulate the motion of a taught welding robot.

[0049] Hereinafter, with appropriate reference to the accompanying drawings, the embodiments of the robot teaching system and the robot teaching method involved in the present disclosure will be described in detail. However, unnecessary detailed descriptions are sometimes omitted. For example, detailed descriptions of known matters and repeated descriptions of substantially the same structures are sometimes omitted. This is to avoid the following description from becoming unnecessarily lengthy and to make it easier for those skilled in the art to understand. In addition, the drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure and are not intended to limit the subject matter described in the scope of the patent claim.

[0050] First, refer to Figure 1 Next, a welding operation simulation system 100 according to the embodiment will be described. Figure 1 1 is a diagram showing an example of a welding action simulation system 100 according to an embodiment. Figure 1 The welding operation simulation system 100 shown is an example and is not limited thereto.

[0051] After the operator completes the teaching, welding motion simulation system 100 generates a simulated image based on each of the multiple teaching points, visualizing the motion of the welding robot performing welding on a real-world or virtual workpiece. Welding motion simulation system 100 outputs the generated simulated image to MR device DV, visualizing it for the operator.

[0052] In the present disclosure, the workpiece used to generate the simulated image is a real workpiece or a virtual workpiece constructed based on 3D model data. Furthermore, the welding action referred to here not only includes welding the workpiece, but also includes approaches by a welding robot (welding torch) to the workpiece, avoidance by the welding robot (welding torch) to avoid obstacles, free-running by the welding torch, and separation by the welding robot (welding torch) from the workpiece.

[0053] The welding operation simulation system 100 includes a reference position acquisition marker Mk1, an MR device DV, and a processing device P1. If the MR device DV can realize the function of the processing device P1, the processing device P1 may be omitted.

[0054] The reference position acquisition marker Mk1 is implemented, for example, as a barcode or a two-dimensional barcode such as a QR code (registered trademark). The reference position acquisition marker Mk1 is a square-shaped marker placed in the real world and contains information indicating the reference position for displaying a simulated image of the welding robot. Furthermore, the reference position acquisition marker Mk1 enables the relative angle (relative posture) between the reference position acquisition marker Mk1 and the MR device DV to be determined based on its shape reflected in the image captured by the camera 15 of the MR device DV.

[0055] The MR device DV is a so-called head-mounted display and is connected to the processing device P1 for data communication. The MR device DV is worn on the operator's head and creates a virtual space by superimposing an image of virtual production equipment (e.g., a virtual workpiece Wk, a virtual welding robot VRB, or a virtual fixture) on an image of the real space corresponding to the operator's field of view. This image is then displayed on the display unit 13, thereby visualizing the virtual space for the operator.

[0056] The MR device DV detects a real-world reference position acquisition marker Mk1 based on the image captured by the camera 15 and reads the reference position information from the reference position acquisition marker Mk1. Based on the position and posture information of each of the multiple teaching points sent from the processing device P1, the MR device DV generates a motion path for the welding robot taught using these teaching points. The MR device DV generates a simulation result of a virtual welding robot moving along this motion path and performing welding. The MR device DV overlays the generated simulation result of the welding robot based on the position of the reference position on the real-world image captured by the camera 15, thereby generating and displaying a simulated image that visualizes the motion of the welding robot performing welding on a real-world or virtual workpiece Wk.

[0057] The processing device P1 is connected to the MR device DV and the robot controller for data communication. The processing device P1 transmits information about the position (3D) and posture (3D) of the teaching points to the MR device DV. Furthermore, the processing device P1 transmits this information to the robot controller, which controls and drives the welding robot in the real world, thereby executing the teaching process for each teaching point.

[0058] In the following description of this disclosure, an example is described in which the teaching points for welding the workpiece Wk0 are previously taught using a teaching tool TL, which serves as a welding torch of a welding robot that performs welding on the workpiece Wk0. Furthermore, although not described in this disclosure, corrections to the teaching points using the teaching tool TL are also possible based on simulation results.

[0059] The teaching tool TL, as discussed herein, is held by the operator and receives instruction on the teaching point on the workpiece Wk, specifically the taught position on the workpiece Wk and the three-dimensional posture of the welding torch at the taught point. The teaching tool TL includes a marker Mk2 for detecting the position and posture of the teaching tool TL, and a tip portion TC, which serves as the tip of the welding torch. The teaching tool TL is connected to the MR device DV via wireless or wired communication, transmitting information on the position and posture of the taught point taught by the operator.

[0060] The marker Mk2 is a hexahedron in the shape of a cube, with a two-dimensional code (e.g., a barcode, QR code (registered trademark), etc.) readable by the MR device DV on each surface. The marker Mk2 enables the posture of the teaching tool TL to be read using the two-dimensional code on each surface. The posture that can be read using the marker Mk2 includes information on the tilt angle, forward / backward angle, and twist angle of the teaching tool TL relative to the coordinate system set for the teaching tool TL.

[0061] Furthermore, teaching of the teaching point is not limited to teaching using the teaching tool TL; for example, it can also be performed by the operator's hand. In such a case, the position of the teaching point is taught based on the position pointed by the operator's finger, and the posture of the welding torch at the teaching point is taught based on the posture of the operator's finger.

[0062] Next, refer to Figure 2 An example of the internal configuration of the MR device DV and the processing unit P1 will be described. Figure 2 1 is a diagram showing an example of the internal configuration of the MR device DV and the processing apparatus P1.

[0063] The MR device DV includes a communication unit 10 , a processor 11 , a memory 12 , a display unit 13 , a depth sensor 14 , and a camera 15 .

[0064] The communication unit 10 is connected to the teaching tool TL and the processing device P1 for wireless or wired communication, and performs data transmission and reception. The communication unit 10 outputs various data sent from the teaching tool TL and the processing device P1 to the processor 11. The communication unit 10 transmits various data output from the processor 11 to the processing device P1. The wireless communication referred to here refers to communication via a wireless local area network (LAN) such as Wi-Fi (registered trademark). If the processing device P1 is omitted from the welding simulation system 100, the communication unit 10 is connected to the robot controller for data communication.

[0065] Processor 11 is constructed using, for example, a central processing unit (CPU) or a field programmable gate array (FPGA), and collaborates with memory 12 to perform various processing and control functions. Specifically, processor 11 references programs and data stored in memory 12 and executes these programs, thereby implementing various functions such as accepting instruction points and generating and displaying simulated images of the welding robot. Furthermore, if processing device P1 is omitted from welding simulation system 100, processor 11 is configured to implement the same functions as processor 21 of processing device P1.

[0066] The memory 12 includes, for example, a random access memory (RAM) serving as working memory for executing various processes of the processor 11, and a read-only memory (ROM) storing programs and data that define the various operations of the processor 11. The RAM temporarily stores data or information generated or acquired by the processor 11. The ROM stores programs that define the operations of the processor 11.

[0067] The display unit 13 is constructed using, for example, a liquid crystal display (LCD) or an organic electroluminescence (EL) display. It displays an image of the real world itself or an image of a virtual space in which virtual production equipment is superimposed on the real world. For example, the display unit 13 displays an image of a virtual space in which virtual production equipment, generated by the processor 11, is superimposed on an image of the real world captured by the camera 15, thereby achieving mixed reality.

[0068] The depth sensor 14 measures the distance between the MR device DV and an object in the real world and recognizes the three-dimensional shape of the object in the real world (eg, a workpiece Wk or a jig). The depth sensor 14 outputs the recognition result to the processor 11 .

[0069] The camera 15 captures an image of an area (real world) corresponding to the visual field of the operator wearing the MR device DV, and outputs the captured image to the processor 11 .

[0070] The processing device P1 includes a communication unit 20 , a processor 21 , and a memory 22 .

[0071] The communication unit 20 is connected to the MR device DV and the robot controller via wireless or wired communication, transmitting and receiving data. The communication unit 20 outputs various data sent from the MR device DV to the processor 21. The communication unit 20 transmits various data output from the processor 21 to the MR device DV or the robot controller. The wireless communication mentioned here refers to communication via a wireless LAN such as Wi-Fi (registered trademark).

[0072] Processor 21 is configured using, for example, a CPU or FPGA, and performs various processes and controls in cooperation with memory 22. Specifically, processor 21 refers to programs and data stored in memory 22 and executes the programs, thereby realizing various functions for generating a welding teaching program.

[0073] The memory 22 includes, for example, a RAM serving as a working memory used when executing various processes of the processor 21, and a ROM storing programs and data that define the various actions of the processor 21. The RAM temporarily stores data or information generated or acquired by the processor 21. The ROM stores programs that define the actions of the processor 21. The memory 22 includes a teaching information recording unit 221 and a workpiece information recording unit 222. Furthermore, the teaching information recording unit 221 and the workpiece information recording unit 222 may also be recorded in the memory 12 of the MR device DV. The memory 22 stores information such as a 3D model of the welding robot and information related to the welding robot's coordinate system.

[0074] The teaching information recording unit 221 records information on the positions and postures of a plurality of teaching points for each workpiece Wk.

[0075] The workpiece information recording unit 222 records a 3D model of the workpiece Wk. The 3D model of the workpiece Wk recorded in the workpiece information recording unit 222 is used in interference determination processing to determine whether interference exists between the welding robot and the workpiece Wk or production equipment such as a jig during simulation of the welding robot based on the teaching results. Furthermore, the 3D model of the workpiece Wk can be generated based on the apparent shape of the workpiece Wk detected by the depth sensor 14 of the MR device DV.

[0076] Next, refer to Figure 3 The following describes a teaching example of welding action. Figure 3 This figure explains the teaching example of welding action. Figure 3 The teaching lines shown are, as an example, a welding line, an avoidance line, and an approach line. However, a free travel line for free travel of the welding torch, a separation line for separating the welding torch from the workpiece Wk, etc. may be taught.

[0077] exist Figure 3In the example shown, weld line WL1 is taught using teaching point WP1, which is the welding start point, and teaching point WP2, which is the welding end point, and is the welding section in which welding is performed. Avoidance line AL1 is taught using teaching point WP2, which is the avoidance start point, and teaching point AP1, which is the avoidance end point, and is the avoidance section in which the welding torch avoids collision or interference. Avoidance line AL2 is taught using teaching point AP1, which is the avoidance start point, and teaching point AP2, which is the avoidance end point. Avoidance line AL3 is taught using teaching point AP2, which is the avoidance start point, and teaching point AP3, which is the avoidance end point. Avoidance line AL4 is taught using teaching point AP3, which is the avoidance start point, and teaching point WP3, which is the avoidance section in which the welding torch approaches the workpiece Wk. Weld line WL2 is a welding section taught by a teaching point WP3 for teaching a welding start point and a teaching point WP4 for teaching a welding end point.

[0078] When such welding operation is taught, the welding robot moves the welding torch from teaching point WP1 to teaching point WP4 along the teaching lines in the order of welding line WL1, avoidance line AL1, avoidance line AL2, avoidance line AL3, avoidance line AL4, and welding line WL2.

[0079] Next, refer to Figure 4 as well as Figure 5 Next, the simulation image SC11 and the reproduction menu RM will be described. Figure 4 1 is a diagram showing an example of a simulation image SC11. Figure 5 : is a diagram showing an example of switching of the reproduction menu RM. Figure 4 The teaching points and teaching lines shown are Figure 3 The teaching points (teaching points WP1 to WP4, AP1 to AP3) and the lines (welding lines WL1, WL2, avoidance lines AL1 to AL4) shown in FIG are the same. Figure 4 In order to facilitate understanding of the drawings, the assignment of symbols and the drawing of symbol lead lines are partially omitted.

[0080] MR device DV is based on Figure 3 The teaching results (each teaching line) described above simulate the movement of a welding robot performing welding along the teaching lines. MR device DV constructs a virtual welding robot VRB based on the simulation results and a 3D model of the welding robot welding workpiece Wk.

[0081] The MR device DV generates a simulated image in which the taught teaching points and teaching lines, as well as the constructed virtual welding robot VRB, are superimposed on the real world imaged by the camera 15, and outputs the image to the display unit 13 for display. The workpiece Wk projected in the simulated image may be either a real-world workpiece Wk or a virtual workpiece Wk.

[0082] Thus, the operator can see through this image the welding robot welding the workpiece Wk in the real world. In addition, by visualizing each teaching point and each teaching line superimposed on the workpiece Wk, the operator can confirm the positions of the teaching points and teaching lines as a result of the teaching for the workpiece Wk.

[0083] The MR device DV accepts an operator's operation to select any of the teaching lines displayed in the simulated image. The MR device DV generates multiple simulated images, each of which is capable of replaying, frame by frame at predetermined intervals, the motion (posture) of a virtual welding robot VRB performing welding within a teaching interval from the start point (e.g., welding start point or avoidance start point) of the selected teaching line to the end point (e.g., welding end point or avoidance end point) of the teaching line. The MR device DV generates a playback menu RM that accepts operations related to playback of the simulated image of the selected teaching line and displays it on the display unit 13.

[0084] The selected teaching line becomes the trajectory of the welding robot's welding torch. Furthermore, the multiple simulated images simulate the welding robot's posture at various positions as it moves along the teaching line at predetermined intervals (e.g., 1 cm or 2 cm increments). Each of the multiple simulated images is played frame by frame along the teaching line, representing the posture changes of the real-world welding robot as it moves along the teaching line.

[0085] The simulation image SC11 is an image showing the simulation result of the virtual welding robot VRB performing welding operation on the selected teaching line. Figure 4 In the example shown, simulation image SC11 shows the results of a simulation of a welding operation (avoidance) performed on a teaching line (welding line WL1) selected by the operator. Simulation image SC11 includes at least a real-world or virtual workpiece Wk, various teaching points and teaching lines, a virtual welding robot VRB performing welding on the selected teaching line (welding line WL1), and a playback menu RM for displaying the simulation results.

[0086] The playback menu RM includes virtual buttons (play button BT1 or stop button BT2) and a virtual slider SB, and accepts user operations. The virtual buttons accept playback operations to play the simulation image frame by frame at a given interval or stop operations to stop the frame-by-frame playback of the simulation image.

[0087] One end of the virtual drag bar SB is the starting point of the teaching line ( Figure 4 In the example shown, the welding start point (teach point WP1) is the reproduction start position T0, and the other end is the end point of the teaching line (at Figure 4 In the example shown, this is the reproduction end position T1 of the welding end point (teach point WP2). The virtual slider SD indicates the position (reproduction position) of the simulated image currently displayed on the display unit 13, among the multiple simulated images of the virtual welding robot VRB performing welding operations on the selected teach line.

[0088] The virtual slider SD moves from the left end to the right end of the virtual drag bar SB while the simulated image is being played back, and stops moving during a stop operation. The virtual slider SD can be moved to any playback position between the left and right ends of the virtual drag bar SB by the user. When the virtual slider SD is moved by the user, a simulated image corresponding to the moved position is displayed.

[0089] The virtual buttons are displayed by switching between the play button BT1 and the stop button BT2 based on a selection (pressing) operation by the operator. Figure 5 The reproduction menu RM1 shown includes a reproduction button BT1 for starting simulated reproduction, and the reproduction menu RM2 includes a stop button BT2 for stopping simulated reproduction.

[0090] When the operator selects the playback button BT1, the MR device DV starts a simulated playback operation of playing the simulated image frame by frame. When the operator selects the stop button BT2, the MR device DV stops playing the simulated image frame by frame and stops the simulated playback operation.

[0091] When the user changes the position of the virtual slider SD on the virtual scrub bar SB, the MR device DV changes the simulation image playback start position (time) to the playback time corresponding to the changed position of the virtual slider SD. If the user further selects the playback button BT1, the MR device DV starts playing (i.e., reproducing) the simulation results frame by frame from the changed simulation playback start position (time).

[0092] Next, refer to Figure 6 as well as Figure 7 An example of the operation sequence of the MR device DV will be described. Figure 6This is a flowchart showing an example of the operation sequence of the MR device DV in the embodiment. Figure 7 It shows Figure 6 The flowchart of an example of the operation sequence of the MR device DV in each event is shown.

[0093] The MR device DV acquires each of the plurality of teaching points transmitted from the processing apparatus P1. The MR device DV captures the real world corresponding to the operator's field of view using the camera 15, and detects the reference position acquisition marker Mk1 from the captured image.

[0094] The MR device DV calculates its relative position with respect to the real-world coordinate system (i.e., the world coordinate system) based on the position and shape of the reference position acquisition marker Mk1 in the captured image. Based on this calculated relative position and a pre-registered robot coordinate system containing a 3D model of the welding robot, the MR device DV calculates the relative position and posture of the welding robot with respect to the real-world or virtual workpiece Wk.

[0095] Based on the calculated relative position of the MR device DV with respect to the coordinate system of the real world and the relative position and relative posture of the welding robot, the MR device DV generates a simulated selection image in which each of multiple teaching points for teaching the workpiece Wk, at least one teaching line connecting two teaching points, and a virtual welding robot VRB welding the workpiece Wk are superimposed on the captured image of the real world, and is displayed on the display unit 13.

[0096] The simulation selection image mentioned here is the simulation image SC11 that does not include the reproduction menu RM. In addition, the simulation selection image and the simulation image may also reflect other real-world or virtual production equipment (eg, jigs, actual welding site equipment environment, etc.).

[0097] The MR device DV receives a plurality of teaching lines (such as welding lines, avoidance lines, approach lines, idle lines, or separation lines) that are visualized on the simulated selection image displayed on the display unit 13. Figure 4 In the example shown, the operation ( St11 ) is to select a movement line for performing a simulation of a welding operation using a virtual welding robot VRB among the welding lines WL1 and WL2 and the avoidance lines AL1 to AL4 .

[0098] Specifically, the MR device DV displays a simulated selection image of virtual production equipment and multiple operator-operable teaching lines in the real world, and then accepts a selection operation to select at least one teaching line on the simulated selection image. The MR device DV detects the movement of the operator's finger in mid-air from the image captured by the camera 15 and accepts the teaching line selection operation based on the position and movement of the operator's finger relative to the simulated selection image (i.e., the selection operation). Alternatively, the selected teaching line may be a continuous series of teaching lines, such as the weld line WL1 and the avoidance line AL1.

[0099] The MR device DV generates a playback menu RM (St12) that allows for playback of the results of the simulation of the welding robot VRB performing welding movements along the selected travel line. The MR device DV generates a simulated image SC11 that includes the generated playback menu RM, the real-world or virtual workpiece Wk, the taught teaching lines (teaching sections), and the welding robot VRB, and displays it on the display unit 13 (St12). The MR device DV receives user-operated playback menu RM operations by displaying the simulated image SC11 of the virtual production equipment and the playback menu RM in the real world. Subsequent user operations on the playback menu RM are also recognized using the same method.

[0100] The MR device DV determines whether an operator operation has been accepted in the reproduction menu RM of the simulated image SC11 ( St13 ).

[0101] In step St13, it is determined that the operator has selected (pressed) the reproduction button BT1 (see Figure 5 ) (St13, select the playback button), the MR device DV sets the simulation start position (start time) of the welding robot VRB moving on the moving line to the time (initial time) corresponding to the playback start position T0 (St14).

[0102] The MR device DV executes a display position update event (step St15). After executing the display position update event (step St38), the MR device DV executes a playback event (step St16). After executing the playback event (steps St31 to St36), the MR device DV returns to the process of step St13.

[0103] In addition, it is determined in step St13 that the stop button BT2 (see Figure 5) (St13, selection of the stop button), the MR device DV executes a stop event (St17). After executing the stop event (step St36), the MR device DV returns to the process of step St13.

[0104] Furthermore, if it is determined in step St13 that the operator has selected a playback position (playback start time) in the playback menu RM (St13, playback position selection), the MR device DV executes a stop event (St18). The MR device DV sets the simulation start position (start time) to the selected playback position (playback time) (St19) and executes a display position update event (St20).

[0105] Here, the processing executed in each event will be described. First, the playback event will be described, in which a simulated image of a virtual welding robot VRB moving along a selected movement line is played frame by frame at predetermined intervals.

[0106] When a playback event is started, the MR device DV generates a stop button BT2 and displays a simulation image SC11 having a playback menu RM2 including the generated stop button BT2 (St31). The MR device DV updates playback time information based on the playback time (playback position) corresponding to the simulation image currently displayed on the display unit 13, generates a simulation image representing the simulation result of the welding robot VRB corresponding to the updated playback time (playback position), and displays it (St32).

[0107] The updated playback time (playback position) here refers to the time (position) after a predetermined time has passed since the previous playback time (playback position) when one frame has been played back frame by frame. For example, after updating the playback time information, the MR device DV generates and displays a simulated image of a virtual welding robot VRB moving a predetermined distance (e.g., 1 cm) along a teaching line.

[0108] The MR device DV determines whether the simulated image of the welding robot VRB corresponding to the updated playback time is within an interference zone where the workpiece Wk or a fixture, etc., interferes with the welding robot VRB (St33). The interference determination of whether the workpiece Wk or a fixture, etc., interferes with the welding robot VRB can be performed when generating multiple simulated images as simulation results. The MR device DV sets the zone within the teaching line where the workpiece Wk or a fixture, etc., interferes with the welding robot VRB as an interference zone.

[0109] If it is determined in step St33 that the simulated image of the welding robot VRB corresponding to the updated reproduction time is in the interference section (St33, Yes), the MR device DV generates an interference notification image SC12 (see FIG. 1 ) notifying that the workpiece Wk or the fixture or other equipment is interfering with the welding robot VRB. Figure 8 ) is displayed on the display unit 13 (St34). While the disturbance notification image SC12 is displayed, the MR device DV generates a playback button BT1. The MR device DV generates a disturbance notification image SC12 including a playback menu RM including the generated playback button BT1 and displays it on the display unit 13 (St36). The MR device DV ends processing the playback event.

[0110] On the other hand, when it is determined in step St33 that the simulation image of the welding robot VRB corresponding to the updated reproduction time is not an interference interval (St33, No), the MR device DV determines whether the updated reproduction time is the end time of the simulation result, that is, whether the simulation image of the updated reproduction position is the simulation image of the reproduction end position T1 (St35).

[0111] If, in step St35, it is determined that the updated playback time is the end time of the simulation results (St35, Yes), the MR device DV generates a playback button BT1. The MR device DV then generates a playback menu RM that includes the generated playback button BT1 and displays the simulated image at the end time (i.e., playback end position T1) of the playback menu RM1 on the display unit 13 (St36). The MR device DV then terminates processing of the playback event.

[0112] On the other hand, when it is determined in step St35 that the updated reproduction time is not the end time of the simulation result ( St35 , No), the MR device DV returns to the process of step St32 and continues the reproduction (frame-by-frame display) of the simulation image.

[0113] Next, a stop event for stopping the reproduction of the simulation result of the welding robot VRB moving on the selected movement line will be described.

[0114] When a stop event is initiated, the MR device DV generates a playback button BT1. The MR device DV then generates a playback menu RM1 that includes the generated playback button BT1. The MR device DV displays the simulated image corresponding to the position (time) of the virtual slider SD when the stop button BT2 was selected, including the generated playback menu RM1, on the display unit 13 (St36). The MR device DV then terminates processing of the stop event.

[0115] Next, a display position update event for displaying (reproducing) a simulation result from a reproduction time corresponding to the current reproduction position will be described.

[0116] When the display position update event starts, the MR device DV generates a simulation image corresponding to the simulation start time (start position) and displays it on the display unit 13 ( St38 ).

[0117] As described above, the welding action simulation system 100 in the embodiment uses a real-world workpiece Wk when a 3D model of the workpiece Wk is not available. Alternatively, if the physical workpiece Wk is not available, a 3D model or a virtual workpiece Wk generated by pre-scanning the real-world workpiece Wk is used. This allows for easy simulation of the welding action of a welding robot based on the teaching results of the workpiece Wk. The welding action simulation system 100 displays a mixed reality simulation result, in which the virtual welding robot VRB is superimposed on the real world (i.e., the location where welding of the workpiece Wk is actually performed), thereby allowing the operator to visualize the simulated welding action based on the teaching results.

[0118] The welding motion simulation system 100 also generates multiple simulated images that simulate the robot's motion posture at various locations along a teaching line, as seen in the real world. This allows visualization of the robot's posture at each location along the teaching line. Furthermore, the welding motion simulation system 100 plays each of the multiple simulated images frame by frame in sequence along the teaching line, allowing the robot's posture changes along the teaching line to be observed as a dynamic image.

[0119] Furthermore, the welding action simulation system 100 receives a playback menu RM on the simulation image, which allows the operator to display the simulation result at a desired position within the movement line (teaching section) by accepting a playback operation or a stop operation for the playback (display) of the simulation result, or by selecting a playback position (playback time) for the playback (display) of the simulation result.

[0120] Furthermore, the welding action simulation system 100 stops reproducing the simulation results if it determines that the workpiece Wk or a fixture, etc., is interfering with the welding robot VRB. This allows the welding action simulation system 100 to visualize for the operator the moments (i.e., the interference locations on the teaching line) when the workpiece Wk or a fixture, etc., interferes with the welding robot VRB during the teaching results.

[0121] Next, refer to Figure 8 The disturbance notification image SC12 will be described. Figure 8 1 is a diagram showing an example of a disturbance notification image SC12. Figure 8 In order to facilitate understanding of the figure, the teaching points and teaching lines are omitted.

[0122] From the operator's viewpoint, the interference notification image SC12 highlights the interference area IF in the simulation results, where the welding robot VRB interferes with other production equipment (e.g., a workpiece Wk or a fixture), using a given color (e.g., red). To make the location of the interference area IF easier for the operator to understand in mixed reality, the interference notification image SC12 includes an arrow ICN indicating the location of the interference area IF.

[0123] (Note)

[0124] The following technologies are disclosed through the description of the above embodiments.

[0125] (Technique 1)

[0126] The robot teaching system (welding action simulation system 100) includes: a model storage unit (memory 22) for storing a three-dimensional model corresponding to a robot (welding robot) existing in an actual environment; a teaching data storage unit (teaching information recording unit 221) for storing teaching data of the robot (welding robot); an operation image (reproduction menu RM) storage unit (memory 22) for storing operation image (reproduction menu RM) data corresponding to the operation image (reproduction menu RM) used in display operation of the three-dimensional model; a display device (display unit 13) configured to be wearable by an operator and to display an image superimposed on an image of the actual environment or the actual environment itself; a positional relationship acquisition unit (camera 15) for acquiring a relative positional relationship between the actual environment and the display device (display unit 13); and an image generation unit (processor 11) for generating a display image (simulation image SC1) based on the relative positional relationship, the three-dimensional model, and the operation image (reproduction menu RM) data. 1), the display image is used to display the three-dimensional model and the operation image (reproduction menu RM) so as to be in a given positional relationship relative to the display device (display unit 13); an output unit (processor 11) outputs the display image (simulation image SC11) to the display device (display unit 13); and a detection unit (depth sensor 14) detects an operation performed by an operator on the operation image (reproduction menu RM) displayed on the display device (display unit 13) in the air away from the display device (display unit 13), i.e., an aerial operation, the image generation unit (processor 11) generates the display image (simulation image SC11), the display image is used to display the three-dimensional model in a posture corresponding to the posture of the robot (welding robot) specified by the aerial operation, among the posture changes of the robot (welding robot) in the motion trajectory of the robot (welding robot) between the first teaching point and the second teaching point included in the teaching data.

[0127] The welding motion simulation system 100 thus displays a mixed reality display in which the virtual welding robot VRB is superimposed on the real world (i.e., the actual location where welding of the workpiece Wk is performed). This allows the operator to visualize the results of the welding motion simulation based on the teaching results. Furthermore, the welding motion simulation system 100 generates multiple simulation images that simulate the welding robot's motion posture at various positions as the real-world welding robot moves along the teaching line at predetermined intervals (e.g., 1 cm or 2 cm increments). The welding motion simulation system 100 plays these multiple simulation images frame by frame based on the operator's aerial operation, allowing the operator to observe the changes in the welding robot's posture as it moves along the teaching line as a dynamic image.

[0128] (Technique 2)

[0129] In the robot teaching system (welding action simulation system 100) described in (Technology 1), the operation image (reproduction menu RM) is a virtual drag bar SB that corresponds to the action trajectory and can adjust the position of the virtual slider SD through the aerial operation. The image generation unit (processor 11) generates the image, which is used to display the three-dimensional model in a posture corresponding to the posture of the robot (welding robot) at a time point corresponding to the position of the virtual slider SD adjusted through the aerial operation.

[0130] Thus, the welding action simulation system 100 can display the posture of the welding robot corresponding to the position desired by the operator among multiple simulation images that simulate the action posture of the welding robot at each moving position when the welding robot in the real world moves along the teaching line at given intervals (for example, 1 cm or 2 cm scales, etc.).

[0131] (Technique 3)

[0132] In the robot teaching system (welding action simulation system 100) described in (Technology 1) or (Technology 2), the operation image (reproduction menu RM) is a virtual button (reproduction button BT1 or stop button BT2) that can be operated through the air operation. When the virtual button is operated through the air operation, the image generation unit (processor 11) sequentially generates the display image (simulation image SC11) for displaying the posture change of the three-dimensional model corresponding to the posture change of the robot (welding robot) in the action trajectory.

[0133] Thus, the welding action simulation system 100 can display the posture of the welding robot corresponding to the position desired by the operator among multiple simulation images that simulate the action posture of the welding robot at each moving position when the welding robot in the real world moves along the teaching line at given intervals (for example, 1 cm or 2 cm scales, etc.).

[0134] (Technique 4)

[0135] In the robot teaching system (welding motion simulation system 100) described in any one of (Technology 1) to (Technology 3), the position relationship acquisition unit (camera 15) is capable of acquiring the relative position relationship between the object (workpiece Wk) existing in the actual environment and the motion trajectory, and the robot teaching system further includes: an interference detection unit (processor 11 or processor 21), which detects whether the object (workpiece Wk) and the robot (welding robot) interfere with each other during the posture change of the robot (welding robot) in the motion trajectory of the robot (welding robot) based on the relative position relationship; and a notification unit (display unit 13), which notifies the operator when interference is detected by the interference detection unit (processor 11 or processor 21).

[0136] Thus, the welding operation simulation system 100 can notify the operator of the time when the workpiece Wk and the welding robot VRB interfere with each other (that is, the interference position on the teaching line) in the teaching results.

[0137] (Technique 5)

[0138] In the robot teaching system (welding action simulation system 100) described in (Technique 4), when interference between the object (workpiece Wk) and the robot (welding robot) is detected, the image generation unit (processor 11) generates an interference image (interference notification image SC12), and the interference image emphasizes the interference area IF where the object (workpiece Wk) and the robot (welding robot) interfere on the display image (simulation image SC11) at the moment when the object (workpiece Wk) and the robot (welding robot) interfere, and the notification unit (display unit 13) notifies the interference image (interference notification image SC12).

[0139] Thus, the welding operation simulation system 100 can visualize the interference area IF between the workpiece Wk and the welding robot VRB in the teaching result to the operator.

[0140] (Technique 6)

[0141] A robot teaching method is a robot teaching method performed by a system (welding action simulation system 100) having at least one computer (MR device DV or processing device P1). In the robot teaching method, a three-dimensional model corresponding to a robot (welding robot) existing in an actual environment and teaching data for the robot (welding robot) are stored, and operation image (reproduction menu RM) data corresponding to an operation image (reproduction menu RM) used in a display operation of the three-dimensional model are stored. The relative positional relationship between the actual environment and a display device (display unit 13) is obtained, the display device being configured to be wearable by an operator and displaying an image superimposed with an image of the actual environment or the actual environment itself, and a display image (simulation image SC11) is generated based on the relative positional relationship, the three-dimensional model, and the operation image (reproduction menu RM) data. ), the display image is used to display the three-dimensional model and the operation image (reproduction menu RM) so as to be in a given positional relationship relative to the display device (display unit 13), the display image (simulation image SC11) is output to the display device (display unit 13), and the operation performed by the operator on the operation image (reproduction menu RM) displayed on the display device (display unit 13) in the air away from the display device (display unit 13), that is, the air operation, is detected to generate the display image (simulation image SC11), the display image is used to display the three-dimensional model in a posture corresponding to the posture of the robot (welding robot) specified by the air operation, among the posture changes of the robot (welding robot) in the motion trajectory of the robot (welding robot) between the first teaching point and the second teaching point included in the teaching data.

[0142] The welding motion simulation system 100 thus displays a mixed reality display in which the virtual welding robot VRB is superimposed on the real world (i.e., the actual location where welding of the workpiece Wk is performed). This allows the operator to visualize the results of the welding motion simulation based on the teaching results. Furthermore, the welding motion simulation system 100 generates multiple simulation images that simulate the welding robot's motion posture at various positions as the real-world welding robot moves along the teaching line at predetermined intervals (e.g., 1 cm or 2 cm increments). The welding motion simulation system 100 plays these multiple simulation images frame by frame based on the operator's aerial operation, allowing the operator to observe the changes in the welding robot's posture as it moves along the teaching line as a dynamic image.

[0143] While various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to the examples described. Those skilled in the art will readily be able to devise various variations, modifications, substitutions, additions, deletions, and equivalents within the scope of the claims, and it should be understood that these variations also fall within the technical scope of the present disclosure. Furthermore, the various structural elements of the various embodiments described above may be arbitrarily combined without departing from the spirit of the invention.

[0144] Industrial applicability

[0145] The present disclosure is useful as a robot teaching system and a robot teaching method that more easily simulate the motion of a taught welding robot.

Claims

1. A robot teaching system comprising: A model storage unit for storing a three-dimensional model corresponding to a robot existing in an actual environment; a teaching data storage unit for storing teaching data of the robot; an operation image storage unit that stores operation image data corresponding to an operation image used in a display operation of the three-dimensional model; a display device configured to be wearable by an operator and to display the image superimposed on an image of the actual environment or the actual environment itself; a position relationship acquiring unit, configured to acquire a relative position relationship between the actual environment and the display device; an image generating unit for generating a display image based on the relative positional relationship, the three-dimensional model, and the operation image data, wherein the display image is used to display the three-dimensional model and the operation image so as to have a predetermined positional relationship with respect to the display device; an output unit, outputting the display image to the display device; and a detection unit for detecting an operation performed by an operator on the operation image displayed on the display device in the air away from the display device, i.e., an aerial operation; The image generating unit generates the display image, and the display image is used to display the three-dimensional model in a posture corresponding to the posture of the robot specified by the aerial operation, among the posture changes of the robot in the movement trajectory of the robot between the first teaching point and the second teaching point included in the teaching data.

2. The robot teaching system according to claim 1, wherein: The operation image is a virtual drag bar that corresponds to the motion trajectory and can adjust the position of the virtual slider through the mid-air operation. The image generation unit generates the image for displaying the three-dimensional model in a posture corresponding to the posture of the robot at a point in time corresponding to the position of the virtual slider adjusted by the mid-air operation.

3. The robot teaching system according to claim 1, wherein: The operation image is a virtual button that can be operated by the mid-air operation. When the virtual button is operated by the mid-air operation, the image generation unit sequentially generates the display images for displaying posture changes of the three-dimensional model corresponding to posture changes of the robot in the motion trajectory.

4. The robot teaching system according to any one of claims 1 to 3, wherein: The positional relationship acquisition unit is capable of acquiring the relative positional relationship between the object existing in the actual environment and the motion trajectory. The robot teaching system also has: an interference detection unit that detects whether the object interferes with the robot during a change in the posture of the robot in a motion trajectory of the robot based on the relative positional relationship; and The notification unit notifies an operator when interference is detected by the interference detection unit.

5. The robot teaching system according to claim 4, wherein: When interference between the object and the robot is detected, the image generating unit generates an interference image that emphasizes an interference area where the object and the robot interfere on the display image at the time when the object and the robot interfere. The notification unit notifies the interference image.

6. A robot teaching method, which is performed by a system having at least one computer, wherein: storing a three-dimensional model corresponding to a robot existing in the actual environment and teaching data of the robot, storing operation image data corresponding to an operation image used in a display operation of the three-dimensional model, Acquiring a relative positional relationship between the actual environment and a display device, wherein the display device is configured to be wearable by an operator and to display an image superimposed on an image of the actual environment or the actual environment itself, generating a display image based on the relative positional relationship, the three-dimensional model, and the operation image data, the display image being used to display the three-dimensional model and the operation image so as to have a predetermined positional relationship with respect to the display device; outputting the display image to the display device, detecting an operation performed by an operator on the operation image displayed on the display device in the air away from the display device, i.e., an aerial operation; Generate the display image, which is used to display the three-dimensional model in a posture corresponding to the posture of the robot specified by the aerial operation, among the posture changes of the robot in the movement trajectory of the robot between the first teaching point and the second teaching point included in the teaching data.

Images