Teaching system, robot system, robot teaching method, and robot teaching program

Abstract

A teaching system 100 comprises: a teaching point generator 2 that generates teaching points for a robot 1; an operation unit 3 operated by a user; an image generator 4 that generates a VR image 8 in which a virtual robot 81 corresponding to the robot 1 is placed in a VR space 80; and a display unit 5 that displays the VR image 8. The image generator 4 generates a VR image 8 in which the virtual robot 81 operates in accordance with the operations performed on the operation unit 3. The teaching point generator 2 generates teaching points corresponding to the positions of the virtual robot 81 in the VR space 80, generated by the image generator 4.

Description

[Technical Field]

[0001] The technology disclosed herein relates to a teaching system, a robot system, a robot teaching method, and a robot teaching program. [Background technology]

[0002] Patent Document 1 discloses a teaching device that extracts a movement trajectory of a workpiece from an image captured by a camera and converts the extracted movement trajectory into a movement trajectory of a robot. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-188477 Summary of the Invention

[0004] In the above-mentioned teaching device, since the user is located far away from the robot, it is difficult for the user to check the robot's operation, and therefore it may be difficult to properly generate teaching points for the robot.

[0005] The technology disclosed herein has been made in consideration of the above points, and its purpose is to appropriately generate teaching points for a robot.

[0006] The teaching system disclosed herein comprises a teaching point generation device that generates teaching points for a robot, a controller operated by a user, an image generation device that generates a virtual image in which a virtual robot corresponding to the robot is placed in a virtual space, and a display that displays the virtual image, wherein the image generation device generates the virtual image in which the virtual robot moves in response to operations on the controller, and the teaching point generation device generates the teaching points that correspond to the positions in the virtual space of the virtual robot generated by the image generation device.

[0007] The teaching system disclosed herein comprises a teaching point generation device that generates teaching points for a robot, an image generation device that generates a virtual image in which a virtual robot corresponding to the robot is placed in a virtual space, and a display that displays the virtual image, and the image generation device generates the virtual image in which the virtual robot operates in accordance with the teaching points generated by the teaching point generation device.

[0008] The robot system disclosed herein comprises the teaching system and a robot that operates in accordance with the teaching points generated by the teaching point generating device.

[0009] The robot teaching method disclosed herein includes generating a virtual image in which a virtual robot corresponding to the robot is placed in a virtual space, displaying the virtual image, operating the virtual robot in the virtual image in response to a user's operation on a controller for operating the virtual robot, and generating a teaching point for the robot corresponding to the position of the virtual robot in the virtual space.

[0010] The robot teaching program disclosed herein enables a computer to realize the following functions: generating a virtual image in which a virtual robot corresponding to the robot is placed in a virtual space; operating the virtual robot in the virtual image in response to a user's operation on a controller for operating the virtual robot; and generating a teaching point for the robot corresponding to the position of the virtual robot in the virtual space.

[0011] According to the teaching system, teaching points for the robot can be generated appropriately.

[0012] According to the robot system, teaching points for the robot can be generated appropriately.

[0013] According to the robot teaching method, teaching points for the robot can be generated appropriately.

[0014] According to the robot teaching program, teaching points for the robot can be generated appropriately. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 shows the schematic configuration of the robot system. [Figure 2] FIG. 2 is a diagram illustrating an example of the VR space. [Figure 3] Figure 3 is an example of a VR image. [Figure 4] FIG. 4 is a block diagram showing a schematic hardware configuration of the teaching system. [Figure 5] FIG. 5 is a functional block diagram of the image generating device. [Figure 6] FIG. 6 is a functional block diagram of the teaching point generating device. [Figure 7] FIG. 7 is a flowchart of the first teaching mode. [Figure 8] FIG. 8 is an example of a VR image at the start of painting in the first teaching mode. [Figure 9] FIG. 9 is an example of a VR image during painting in the first teaching mode. [Figure 10] FIG. 10 is another example of a VR image in the first teaching mode. [Figure 11] FIG. 11 is a flowchart of the second teaching mode. [Figure 12] FIG. 12 is an example of the VR space in the second teaching mode. [Figure 13] FIG. 13 is an example of a VR image in the second teaching mode. [Figure 14] FIG. 14 is a flowchart of the playback mode. DETAILED DESCRIPTION OF THE INVENTION

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An exemplary embodiment will now be described in detail with reference to the accompanying drawings.

[0017] The robot system 1000 includes a robot 1 and a teaching system 100. The teaching system 100 creates teaching data for the robot 1. The robot 1 and teaching system 100 are connected so as to be able to send and receive signals.

[0018] The robot 1 is an industrial robot. The robot 1 displaces or deforms with a degree of freedom. The robot 1 performs a process on a workpiece W. The workpiece W is positioned relative to the robot 1. For example, the process performed by the robot 1 is a processing process for machining the workpiece W. Specifically, the processing process is a process for injecting a predetermined injection target toward the workpiece W. In this example, the processing process is painting, in which paint is injected toward the workpiece W. The paint is an example of an injection target.

[0019] The robot 1 is equipped with a manipulator. More specifically, the robot 1 is equipped with a multi-joint robot arm 11 and an end effector 12 connected to the robot arm 11. The end effector 12 is an example of a tool. The robot arm 11 changes the position and posture of the end effector 12 by displacing and deforming. A predetermined robot coordinate system is set for the robot 1. By controlling the positions and postures of the robot arm 11 and the end effector 12 in the robot coordinate system, the robot arm 11 is displaced and deformed, and accordingly the position and posture of the end effector 12 change.

[0020] The robot arm 11 is, for example, a vertical multi-joint arm. More specifically, the robot arm 11 includes a plurality of links 13, a plurality of joints 14, and a plurality of motors 15 (see FIG. 4). Each joint 14 of the robot arm 11 rotatably connects two adjacent links 13. Each motor 15 rotationally drives the corresponding joint 14. The motor 15 is, for example, a servo motor. The robot arm 11 operates in accordance with the driving of the motor 15, and is displaced and deformed. The operation of the robot arm 11 changes the position of the end effector 12, etc.

[0021] The end effector 12 performs processing on the workpiece W. In this example, the end effector 12 injects an injection target toward the workpiece W. Specifically, the end effector 12 is a coating device that injects paint onto the workpiece W, which is an object to be coated. The end effector 12 has an injection port 16 that injects the paint toward a predetermined injection axis.

[0022] The teaching system 100 includes a teaching point generation device 2 that generates teaching points for the robot 1, a controller 3 operated by a user who will be the instructor, an image generation device 4 that generates a virtual image 8 in which a virtual robot 81 corresponding to the robot 1 is placed in a virtual space 80, and a display device 5 that displays the virtual image 8. The image generation device 4 generates the virtual image 8 in which the virtual robot 81 moves in response to the user's operation of the controller 3. The teaching point generation device 2 generates teaching points that correspond to positions in the virtual space 80 of the virtual robot 81 generated by the image generation device 4. Furthermore, the teaching point generation device 2 creates teaching data based on the generated teaching points.

[0023] That is, the teaching system 100 uses virtual technology, i.e., XR (Cross Reality) technology, to operate a virtual robot 81 in a virtual space 80 to generate virtual teaching points in the virtual space 80, and to generate teaching points in real space corresponding to the virtual teaching points. The teaching system 100 creates teaching data for the robot 1 based on the generated teaching points. XR includes VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and SR (Substitutional Reality). In this example, the teaching system uses VR technology. Hereinafter, the virtual space will also be referred to as a "VR space." Furthermore, the virtual image will also be referred to as a "VR image."

[0024] In this example, the teaching system 100 creates teaching points and therefore teaching data for the robot 1 to paint. The robot 1 paints the workpiece W by injecting paint from the end effector 12 while changing the position and posture of the end effector 12 by operating the robot arm 11 in accordance with the teaching data.

[0025] In the teaching system 100, the teaching point generating device 2 and the image generating device 4 are connected so as to be able to send and receive signals. The operating device 3 and the display device 5 are connected to the image generating device 4 so as to be able to send and receive signals.

[0026] The controller 3 is used to operate the virtual robot 81 in the virtual space 80. The controller 3 is a portable device that is held by the user, for example, in the user's hand. The controller 3 has a controller main body 30 and an operation unit 31 that accepts input from the user.

[0027] The controller main body 30 has a shape that can be held by a user. The controller main body 30 is provided with an operation unit 31. The controller main body 30 may have a shape that simulates a spray gun that can be held by a user. When the user holds the controller main body 30 during the preparation stage, the user can imagine that they are holding a spray gun. Therefore, when generating teaching points, the user operates the controller 3 while looking at the VR image 8 displayed on the display 5, and can operate the controller 3 with the feeling of holding a spray gun. When operated by the user, the operation unit 31 outputs a corresponding operation signal. The controller 3 outputs the operation signal from the operation unit 31 to the image generation device 4. The operation signal is used by the image generation device 4 to generate the VR image 8.

[0028] The display device 5 displays the VR image 8 generated by the image generation device 4. The VR image 8 generated by the image generation device 4 is input to the display device 5. The display device 5 is worn on the user's head. For example, the display device 5 is an HMD (Head Mounted Display). The HMD may be a goggle-shaped device dedicated to VR with a display, or may be configured by attaching a mobile terminal such as a smartphone to a holder that can be worn on the head.

[0029] The teaching system 100 further includes a tracking system 6 that detects the position and orientation of the controller 3 in real space and the position and orientation of the display device 5 in real space. The tracking system 6 includes multiple (e.g., two) light emitters 61, a first sensor 62 provided on the controller 3, a second sensor 63 provided on the display device 5, and an image generation device 4 as a calculation unit that calculates the respective positions of the controller 3 and the display device 5 from the detection results of the first sensor 62 and the second sensor 63. The tracking system 6 is an outside-in tracking system. The light emitter 61, the first sensor 62, and the second sensor 63 are connected to the image generation device 4 so as to be able to send and receive signals. The tracking system 6 detects the position and orientation of the first sensor 62 and the second sensor 63 within a motion area in real space where the user can move. A predetermined motion coordinate system is set in the motion area. The tracking system 6 detects the position and orientation of the first sensor 62 and the second sensor 63 in the motion coordinate system. The tracking system 6 is an example of a controller sensor that detects the position and orientation of the controller in real space.

[0030] The operation coordinate system of the tracking system 6 corresponds to the VR coordinate system set in the VR space 80. The operation coordinate system of the tracking system 6 also corresponds to the robot coordinate system of the robot 1. Furthermore, the VR coordinate system of the VR space 80 also corresponds to the robot coordinate system of the robot 1. In other words, the robot coordinate system, operation coordinate system, and VR coordinate system correspond to each other. For example, once the position of the operation coordinate system is determined, the position of the robot coordinate system and the position of the VR coordinate system are uniquely determined.

[0031] The light emitter 61 emits light into real space. For example, the light emitter 61 is an infrared laser that scans real space with infrared laser light. The light emitter 61 scans the laser light in vertical and horizontal directions while intermittently emitting the laser light. Irradiation information of the laser light from the light emitter 61 is input to the image generation device 4.

[0032] The first sensor 62 and the second sensor 63 are sensors that detect light, such as infrared sensors. The detection results of the first sensor 62 and the second sensor 63 are input to the image generation device 4.

[0033] When the first sensor 62 detects the laser light, the image generation device 4 determines the position and orientation of the controller 3 in real space based on the arrival time of the laser light from the light emitter 61 to the first sensor 62 and irradiation information of the laser light from the light emitter 61 (for example, the direction in real space in which the laser light is being irradiated). When the second sensor 63 detects the laser light, the image generation device 4 similarly determines the position and orientation of the display device 5 in real space. The image generation device 4 regards the position and orientation of the display device 5 in real space as the position and direction of the user's line of sight in real space, respectively.

[0034] In this example, the image generating device 4 functions as a calculation unit of the tracking system 6. However, the calculation unit may be provided separately from the image generating device 4.

[0035] The image generation device 4 determines the position and direction of the user's gaze and the position and orientation of the controller 3 in the VR space 80 from the position and direction of the user's gaze and the position and orientation of the controller 3 in the real space. In other words, the image generation device 4 converts the position and direction of the user's gaze and the position and orientation of the controller 3 in the operation coordinate system into the position and direction of the user's gaze and the position and orientation of the controller 3 in the VR coordinate system.

[0036] The image generation device 4 generates a VR image 8 in which a virtual robot 81 corresponding to the robot 1 and a first virtual workpiece 84a corresponding to the workpiece W are arranged in a VR space 80. FIG. 2 is a diagram showing an example of the VR space 80. FIG. 3 is an example of the VR image 8. The VR image 8 in FIG. 3 is a first-person perspective image expressed from the user's viewpoint. The image generation device 4 generates a VR image 8 in which the virtual robot 81 operates to perform processing on the first virtual workpiece 84a in the VR space 80. The image generation device 4 generates a VR image 8 according to the position and direction of the user's line of sight in the VR space 80, and transmits the generated VR image 8 to the display device 5.

[0037] 2, a virtual support device 87, a virtual booth 88, a virtual fence 89, and the like are arranged in the VR space 80. The virtual support device 87, the virtual booth 88, and the virtual fence 89 are examples of virtual facilities that correspond to facilities that exist around the robot 1 in the real space. Therefore, depending on the user's line of sight, the virtual support device 87, the virtual booth 88, the virtual fence 89, and the like may be included in the VR image 8.

[0038] The virtual support device 87 is an object in the VR space 80 that corresponds to the support device 19 (see FIG. 1) that supports the workpiece W in the real space. The virtual booth 88 is an object in the VR space 80 that corresponds to the painting booth in the real space where the workpiece W is placed and painting work is performed. The virtual fence 89 is an object in the VR space 80 that corresponds to the safety fence that surrounds the robot 1 in the real space. The relative positional relationship between the virtual robot 81, the virtual support device 87, the virtual booth 88, and the virtual fence 89 in the VR space 80 matches the relative positional relationship between the robot 1, the support device 19, the painting booth, and the safety fence in the real space.

[0039] A first area 80a, which is an area inside the virtual fence 89, includes at least the range of movement of the virtual robot 81. A second area 80b, which is an area outside the virtual fence 89, is a safety area in which the virtual robot 81 does not move.

[0040] A virtual robot 81 and a first virtual workpiece 84a are placed in a first area 80a. Specifically, the virtual robot 81 is placed inside a virtual fence 89. The virtual robot 81 has a virtual arm 82 corresponding to the robot arm 11 and a virtual end effector 83 corresponding to the end effector 12. The virtual end effector 83 is an example of a virtual tool. The first virtual workpiece 84a is placed in a virtual booth 88. The relative positional relationship between the first virtual workpiece 84a and the virtual end effector 83 in the VR space 80 matches the relative positional relationship between the workpiece W and the end effector 12 in the real space.

[0041] The image generation device 4 extracts a portion of the VR space 80 according to the user's line of sight to generate the VR image 8. In this example, the image generation device 4 generates the VR image 8 according to the position and orientation of the display device 5. At this time, the image generation device 4 displays various images according to the settings. For example, FIG. 3 shows the VR image 8 in a state where the virtual end effector 83 and the first virtual workpiece 84a are in close proximity. In this example, a virtual manipulator 85 corresponding to the manipulator 3 and a second virtual workpiece 84b for simulating the processing of the robot 1 by the virtual manipulator 85 are further disposed in the VR space 80. The image generation device 4 may also display various additional images. For example, the additional image may be an image for supporting a teaching operation or a virtual image different from the real space. For example, the image generation device 4 may display an additional image indicating that the virtual arm 82 has reached a shape corresponding to a singular point or is approaching that shape.

[0042] The virtual end effector 83 has a first virtual injection port 83a corresponding to the injection port 16. A first virtual injection axis P1 corresponding to the injection axis of the paint from the injection port 16 is set in the virtual end effector 83. The first virtual injection axis P1 extends from the first virtual injection port 83a. The first virtual injection axis P1 indicates the attitude of the virtual end effector 83.

[0043] Furthermore, the first virtual injection axis P1 also represents the distance from the virtual end effector 83. Specifically, the type of line representing the first virtual injection axis P1 changes depending on the distance from the first virtual injection outlet 83a. In the example of FIG. 3, the portion of the first virtual injection axis P1 whose distance from the first virtual injection outlet 83a corresponds to the appropriate range for painting is represented by a dashed line, and the other portion is represented by a solid line. Depending on which portion of the first virtual injection axis P1 intersects with the surface of the first virtual workpiece 84a, the user can determine whether the distance between the virtual end effector 83 and the first virtual workpiece 84a is appropriate. The first virtual injection axis P1 is an example of an image indicating the degree of distance from the virtual tool.

[0044] The virtual manipulator 85 is placed at a position in the VR space 80 that corresponds to the position of the first sensor 62 in the real space. The virtual manipulator 85 has a second virtual ejection outlet 85a. A second virtual ejection axis P2 is set in the virtual manipulator 85. The second virtual ejection axis P2 extends from the second virtual ejection outlet 85a. The second virtual ejection axis P2 indicates the attitude of the virtual manipulator 85.

[0045] The second virtual injection axis P2 represents the distance from the virtual operator 85. Specifically, the type of line representing the second virtual injection axis P2 changes depending on the distance from the second virtual injection outlet 85a. In the example of FIG. 3, the portion of the second virtual injection axis P2 whose distance from the second virtual injection outlet 85a corresponds to the appropriate range for painting is represented by a dashed line, and the other portion is represented by a solid line. Depending on which portion of the second virtual injection axis P2 intersects with the surface of the second virtual workpiece 84b, the user can determine whether the distance between the virtual operator 85 and the second virtual workpiece 84b is appropriate.

[0046] The second virtual workpiece 84b has the same size and shape as the first virtual workpiece 84a.

[0047] The virtual end effector 83 is disposed at an offset position relative to the virtual manipulator 85. The first virtual workpiece 84a is disposed at an offset position relative to the second virtual workpiece 84b, similar to the virtual end effector 83. As a result, the relative positional relationship between the second virtual workpiece 84b and the virtual manipulator 85 matches the relative positional relationship between the first virtual workpiece 84a and the virtual end effector 83. Specifically, the angle of the first virtual injection axis P1 relative to the first virtual workpiece 84a matches the angle of the second virtual injection axis P2 relative to the second virtual workpiece 84b. The position of the intersection of the first virtual injection axis P1 with the first virtual workpiece 84a matches the position of the intersection of the second virtual injection axis P2 with the second virtual workpiece 84b. The distance from the first virtual injection port 83a to the first virtual workpiece 84a matches the distance from the second virtual injection port 85a to the second virtual workpiece 84b.

[0048] The image generation device 4 causes the virtual robot 81 to move in response to the user's operation of the controller 3. For example, the image generation device 4 causes the virtual robot 81 in the VR space 80 to perform a movement linked to the movement of the controller 3 in the real space, based on the detection result of the first sensor 62. Specifically, the image generation device 4 places the virtual robot 81 in the VR space 80 so as to correspond to the position and orientation of the controller 3 in the real space, which are determined based on the detection result of the first sensor 62. As the image generation device 4 continues this process, the virtual robot 81 in the VR space 80 moves in conjunction with the movement of the controller 3 in the real space.

[0049] The teaching point generation device 2 generates, as teaching points, the position and posture of the robot 1 in the real space that correspond to the position and posture of the virtual robot 81 in the virtual space 80. More specifically, the teaching point generation device 2 generates, as teaching points, the position and posture of a representative part of the robot 1. In this example, the representative part of the robot 1 is the end effector 12. That is, the teaching point generation device 2 generates, as teaching points, positions that include the posture of the end effector 12. More specifically, the teaching point is the position and posture (i.e., orientation) of the injection port 16 of the painting device on the end effector 12.

[0050] Specifically, while operating the virtual robot 81 in the VR space 80, the user generates a virtual teaching point, which is a teaching point in the VR space 80, via the operation unit 31 of the controller 3. At this time, the user also inputs whether or not to eject the ejection target to the image generation device 4 via the controller 3. The image generation device 4 generates a virtual teaching point based on a generation signal from the controller 3. The image generation device 4 outputs the position and orientation of the controller 3 in the operation coordinate system, which correspond to the virtual teaching point, to the teaching point generation device 2. The teaching point generation device 2 determines the position and orientation of the end effector 12 in the robot coordinate system from the position and orientation of the controller 3 in the operation coordinate system, and sets this as a teaching point. As a result, the teaching point generation device 2 generates a teaching point corresponding to the position of the virtual robot 81 in the virtual space 80. By repeating this process while operating the virtual robot 81 in the VR space 80, the teaching point generation device 2 generates a plurality of teaching points arranged in time series and also sets whether or not to eject the associated teaching points.

[0051] The teaching point generation device 2 creates teaching data based on the teaching points. In this example, the teaching data is a group of data for operating the robot 1 so as to trace a trajectory that passes through a plurality of teaching points in chronological order. In this example, the teaching data also includes information regarding the ejection of the ejection target (for example, whether or not to eject).

[0052] FIG. 4 is a block diagram showing a schematic hardware configuration of the teaching system 100. As shown in FIG.

[0053] The teaching point generating device 2 includes a storage unit 21 and a processing unit 22 .

[0054] The storage unit 21 is a computer-readable storage medium that stores various programs and various data. The storage unit 21 is formed of a magnetic disk such as a hard disk, an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a semiconductor memory. Note that the storage units 42 and 52, which will be described later, are also configured in the same manner as the storage unit 21. The storage unit 21 stores teaching points and teaching data created based on the teaching points.

[0055] The processing unit 22 realizes various functions of the teaching point generation device 2 by reading and executing various programs stored in the storage unit 21. The processing unit 22 has various processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) and / or a DSP (Digital Signal Processor), and various semiconductor memories such as a VRAM (Video Random Access Memory), a RAM (Random Access Memory) and / or a ROM (Read Only Memory). Note that processing units 43 and 53, which will be described later, have the same configuration as the processing unit 22.

[0056] The operation unit 31 of the operation device 3 has a generation switch 32, an injection switch 33, a stop switch 34, and an end switch 35. The generation switch 32, the injection switch 33, the stop switch 34, and the end switch 35 each output an operation signal.

[0057] The generating switch 32 is a switch for generating a teaching point. The teaching point generating device 2 generates, as a teaching point, the position and posture in the robot coordinate system of the robot 1 corresponding to the virtual robot 81 when the generating switch 32 is operated. At this time, the image generating device 4 generates, as a virtual teaching point, the position and posture of the virtual robot 81 when the generating switch 32 is operated. Therefore, the generating switch 32 can also be considered as a switch for generating a virtual teaching point.

[0058] The injection switch 33 is a switch for setting the execution of paint injection. When the user operates the injection switch 33 when the virtual robot 81 is positioned in the VR space 80 to start paint injection, the image generation device 4 sets the execution of paint injection at the position of the virtual robot 81 at that time.

[0059] Stop switch 34 is a switch for setting a stop of paint injection. When virtual robot 81 is positioned in VR space 80 at a position where paint injection should be stopped, the user operates stop switch 34, and image generation device 4 sets paint injection to be stopped at the position of virtual robot 81 at that time.

[0060] The end switch 35 is a switch for ending the teaching work. When the virtual robot 81 reaches the position where the teaching work ends in the VR space 80, the user operates the end switch 35, and the image generation device 4 sets the end of the teaching work at the position of the virtual robot 81 at that time.

[0061] The display device 5 includes a display unit 51 , a storage unit 52 , and a processing unit 53 .

[0062] The display unit 51 is, for example, a liquid crystal display or an organic electroluminescence (EL) display. The display unit 51 can display an image for the right eye and an image for the left eye. In other words, the display unit 51 displays a three-dimensional image including an image for the right eye and an image for the left eye.

[0063] The storage unit 52 is a computer-readable storage medium that stores various programs and various data. The storage unit 52 is formed of a semiconductor memory or the like. The storage unit 52 stores programs and various data that cause the computer, i.e., the processing unit 53, to realize various functions for displaying the VR image 8 on the display unit 51.

[0064] The processing unit 53 reads and executes various programs stored in the memory unit 52, thereby controlling each part of the display device 5 in an integrated manner and realizing various functions for displaying the VR image 8 on the display unit 51.

[0065] The display 5 may have headphones, through which the user can hear audio information necessary for the teaching work.

[0066] The image generating device 4 includes an input unit 41 , a storage unit 42 , and a processing unit 43 .

[0067] The input unit 41 accepts operation input from a user. The input unit 41 outputs an input signal corresponding to the operation input to the processing unit 43. For example, the input unit 41 is a keyboard or a mouse. The input unit 41 is used to input settings for displaying or hiding various objects, which will be described later, and various settings for the display content.

[0068] The storage unit 42 is a computer-readable storage medium that stores various programs and various data, including a teaching program 42a, field definition data 42b, and object definition data 42c.

[0069] The field definition data 42b defines a field in which the virtual robot 81 and the like are placed. For example, the field definition data 42b defines the floor and walls of a factory. The object definition data 42c defines objects necessary for realizing the processing of the virtual robot 81 in the VR space 80. For example, object definition data 42c is prepared for the virtual robot 81, a first virtual workpiece 84a, a second virtual workpiece 84b, a virtual operator 85, and virtual equipment. For example, the field definition data 42b and the object definition data 42c are created based on actual design data of the robot 1 and the like. In this case, there is no need to obtain an actual workpiece W for teaching, and teaching work can be performed before obtaining the workpiece W. Note that the field definition data 42b and the object definition data 42c may be created based on actual measurement values ​​obtained by actually measuring the robot 1 and the like.

[0070] The processing unit 43 reads out and executes programs such as the teaching program 42a stored in the storage unit 42, thereby causing the image generating device 4 as a computer to realize various functions for generating the VR image 8.

[0071] 5 is a functional block diagram of the image generating device 4. FIG. 6 is a functional block diagram of the teaching point generating device 2.

[0072] The image generation device 4 has functional blocks including a tracking control unit 44, an image generation unit 45, a motion generation unit 46, and a generation processing unit 47. The image generation unit 45 has a function of generating a virtual image in which a virtual robot corresponding to the robot is placed in a virtual space. The motion generation unit 46 has a function of moving the virtual robot in the virtual image in response to a user's operation on a controller for moving the virtual robot. The generation processing unit 47 has a function of generating teaching points for the robot based on the position of the virtual robot in the virtual space.

[0073] The tracking control unit 44 calculates the positions and orientations of the controller 3 and display unit 5 in the VR space 80 based on the detection results of the tracking system 6. The tracking control unit 44 executes various calculation processes related to tracking based on the illumination information of the light emitter 61, the detection results of the first sensor 62, and the detection results of the second sensor 63. Specifically, the tracking control unit 44 determines the position and orientation of the controller 3 in the operation coordinate system of real space based on the detection results of the first sensor 62 and the illumination information of the light emitter 61. The tracking control unit 44 determines the position and orientation of the display unit 5 in the operation coordinate system of real space based on the detection results of the second sensor 63 and the illumination information of the light emitter 61.

[0074] The image generation unit 45 generates a VR image 8. The image generation unit 45 reads the field definition data 42b and the object definition data 42c from the storage unit 42, and generates a VR space 80 and various objects. A VR coordinate system corresponding to the robot coordinate system in the real space is set in the VR space 80. The image generation unit 45 places a virtual robot 81, a first virtual workpiece 84a, virtual equipment, etc. in the VR space 80 based on the VR coordinate system. The image generation unit 45 matches the relative positional relationship between the virtual robot 81, the first virtual workpiece 84a, and the virtual equipment in the VR coordinate system with the relative positional relationship between the robot 81, the workpiece W, and the equipment in the robot coordinate system. Furthermore, the image generation unit 45 generates the virtual robot 81 according to the movement calculated by the movement generation unit 46.

[0075] Then, the image generation unit 45 sets a line of sight in the VR space 80 and generates a VR image 8 according to the line of sight. Specifically, the image generation unit 45 determines the position and orientation of the display device 5 in the VR space 80 based on a predetermined coordinate correspondence relationship from the position and orientation of the display device 5 in the operation coordinate system determined by the tracking control unit 44. The coordinate correspondence relationship here is the correspondence relationship between the operation coordinate system of the tracking system 6 and the VR coordinate system of the VR space 80. The image generation unit 45 sets the position of the display device 5 in the VR coordinate system to the position of the user's line of sight (i.e., the position of the eyes) in the VR space 80, and sets the orientation of the display device 5 in the VR coordinate system to the direction of the user's line of sight in the VR space 80. The image generation unit 45 generates the VR image 8 according to the set position and direction of the line of sight. The image generation unit 45 outputs the generated VR image 8 to the display device 5.

[0076] The motion generation unit 46 generates motion of the virtual robot 81 in the VR space 80 according to the position and orientation of the controller 3 in the real space calculated by the tracking control unit 44. The motion generation unit 46 calculates the position and orientation of the virtual controller 85 in the VR coordinate system from the position and orientation of the controller 3 in the manipulation coordinate system based on the coordinate correspondence between the manipulation coordinate system and the VR coordinate system. The motion generation unit 46 sets the position of the virtual end effector 83 (more specifically, the position of the first virtual injection outlet 83a) at a position shifted by a predetermined offset amount in a predetermined offset direction from the position of the virtual controller 85 (more specifically, the position of the second virtual injection outlet 85a). The motion generation unit 46 matches the orientation of the virtual end effector 83 with the orientation of the virtual controller 85. More specifically, the motion generation unit 46 makes the first virtual injection axis P1 parallel to the second virtual injection axis P2. In this way, the position and orientation of the virtual end effector 83 are determined.

[0077] In addition, the motion generation unit 46 determines the position and orientation of the end effector 12 in the real space from the position and orientation of the virtual end effector 83 in the VR space 80 based on the coordinate correspondence between the VR coordinate system and the robot coordinate system. The motion generation unit 46 outputs the position and orientation of the end effector 12 to the teaching point generation device 2. As will be described in detail later, the teaching point generation device 2 calculates the rotation angle of each joint 14 of the robot arm 11 (i.e., the rotation angle of each motor 15) in order to realize the position and orientation of the end effector 12. The teaching point generation device 2 outputs the calculated rotation angle of each joint 14 to the motion generation unit 46. The motion generation unit 46 generates a virtual arm 82 so as to realize the rotation angle of each joint 14 input from the teaching point generation device 2.

[0078] By performing such processing, the action generation unit 46 can generate a virtual robot 81 having a virtual end effector 83 placed at a position and orientation corresponding to the position and orientation of the controller 3 in real space. The action generation unit 46 can cause the virtual robot 81 to operate in conjunction with the controller 3 by generating, i.e., updating, the virtual robot 81 according to the position and orientation of the controller 3 in real space that are determined as needed.

[0079] Furthermore, the motion generation unit 46 sets motion limits on the motion range of the virtual robot 81. When the virtual robot 81 reaches the motion limits, the motion generation unit 46 notifies the user that the virtual robot 81 has reached the motion limits. For example, the motion generation unit 46 outputs a warning signal to the image generation unit 45 when the virtual robot 81 reaches the motion limits. Upon receiving the warning signal, the image generation unit 45 displays a message indicating that the motion limits have been reached. This notifies the user that the virtual robot 81 has reached the motion limits. The motion limits include, for example, motion limits of the virtual arm 82 itself and motion limits caused by the surrounding environment of the virtual arm 82. The motion limits of the virtual arm 82 itself include, for example, limits on the rotation angles of the joints of the virtual arm 82, such as singular points. The limits on the rotation angles of the joints of the virtual arm 82 are the same as the limits on the rotation angles of the joints 14 of the robot arm 11 in real space. The motion limits caused by the surrounding environment of the virtual arm 82 include, for example, limits due to interference between the virtual arm 82 and virtual equipment. The virtual equipment, which will be described in detail later, includes a virtual support device 87, a virtual booth 88, and a virtual fence 89. The movement restrictions of the virtual robot 81 are set with margins for the limits of the rotation angles of the joints 14 and for interference with the equipment.

[0080] The generation processing unit 47 outputs a generation command to generate a teaching point to the teaching point generation device 2 based on an operation signal from the operator 3. In detail, when the generation processing unit 47 receives an operation signal from the generation switch 32, it outputs a generation command to generate a teaching point together with the position and orientation of the operator 3 in the operation coordinate system at that time to the teaching point generation device 2. In addition, the generation processing unit 47 sets the position and orientation of the virtual end effector 83 at the time of receiving the operation signal from the generation switch 32 as a virtual teaching point.

[0081] In addition, the generation processing unit 47 sets whether or not to inject paint based on an operation signal from the operating device 3. More specifically, when the generation processing unit 47 receives an operation signal from the injection switch 33, it sets the paint injection to be executed. On the other hand, when the generation processing unit 47 receives an operation signal from the stop switch 34, it sets the paint injection to be stopped. When outputting a generation command to the teaching point generation device 2, the generation processing unit 47 also outputs paint injection information (i.e., whether to execute injection or stop injection).

[0082] Furthermore, the generation processing unit 47 outputs paint ejection information to the image generation unit 45. The image generation unit 45 switches between displaying and not displaying the virtual paint ejected from the virtual end effector 83 in accordance with the paint ejection information.

[0083] Furthermore, when the generation processing unit 47 receives an operation signal from the end switch 35, it outputs an end command to the teaching point generation device 2 to execute end processing for ending the teaching work.

[0084] The teaching point generating device 2 has, as functional blocks, a rotation angle generating unit 23, a teaching point generating unit 24, a teaching data generating unit 25, and a robot control unit 26.

[0085] The rotation angle generation unit 23 calculates the rotation angle of each joint 14 of the robot arm 11 (i.e., the rotation angle of each motor 15) for realizing the position and posture of the end effector 12 in real space input from the motion generation unit 46. Because the length of each link 13 of the robot arm 11 is known, once the angle of each joint 14 is determined, the position and posture of the robot 1, i.e., the position and posture of the end effector 12, are uniquely determined. Therefore, when the position and posture of the end effector 12 are given, the rotation angle generation unit 23 can determine the rotation angle of each joint 14 based on the length of each link 13. The rotation angle generation unit 23 stores the determined rotation angle of each joint 14 in the memory unit 21 and outputs it to the motion generation unit 46.

[0086] The teaching point generation unit 24 generates a teaching point. The teaching point generation unit 24 determines the position and orientation of the end effector 12 in the robot coordinate system as a teaching point based on the position and orientation of the manipulator 3 in the operation coordinate system that are input together with a teaching point generation command. In this example, since the virtual end effector 83 is positioned at an offset position relative to the virtual manipulator 85 in the VR image 8, the teaching point generation unit 24 determines the position and orientation of the end effector 12 in the robot coordinate system as the position and orientation obtained by similarly offsetting the position and orientation of the manipulator 3 in the operation coordinate system from the position and orientation obtained by converting it into the robot coordinate system. The teaching point generation unit 24 stores the generated teaching point in the memory unit 21. At this time, the teaching point generation unit 24 also stores paint ejection information together with the teaching point in the memory unit 21. The teaching point generation unit 24 stores the teaching point and paint ejection information in the memory unit 21 every time a generation command is input. The teaching point generation unit 24 accumulates the teaching point and paint ejection information in the memory unit 21.

[0087] The teaching data generating unit 25 generates teaching data based on the teaching points generated by the teaching point generating unit 24. When receiving an end command from the generation processing unit 47, the teaching data generating unit 25 generates teaching data as one of the end processes for ending the teaching work.

[0088] The teaching data generation unit 25 reads out the generated teaching points from the storage unit 21. Furthermore, the teaching data generation unit 25 reads out the rotation angles of the joints 14 of the robot arm 11 corresponding to each of the teaching points, as well as paint ejection information, from the storage unit 21. In this way, the teaching data generation unit 25 generates teaching data. The teaching data includes multiple sets of rotation angles of the joints 14, time-series information on the multiple sets, and paint spraying information (spray execution or spray stop) for each set.

[0089] The robot control unit 26 controls the robot 1 based on the teaching data to operate the robot 1 in real space. The robot control unit 26 performs processing after the teaching data is generated and does not directly contribute to the generation of the teaching data. Specifically, the robot control unit 26 executes automatic operation of the robot 1. For example, a user operates the teaching point generation device 2 to input a command for automatic operation of the robot 1 to the robot control unit 26. The robot control unit 26 controls the robot 1 based on the teaching data. Specifically, the robot control unit 26 calculates the rotation angles of the corresponding motors 15 to realize the rotation angles of the multiple joints 14 and supplies currents corresponding to the calculated rotation angles to each motor 15. The robot control unit 26 also controls the end effector 12 in accordance with paint ejection information included in the teaching data. Specifically, when paint spraying is set to be executed, the robot control unit 26 executes paint spraying from the end effector 12. On the other hand, when paint spraying is set to be stopped, the robot control unit 26 stops paint spraying from the end effector 12.

[0090] In this way, the robot 1 moves along a trajectory that passes through the generated teaching points, and sprays paint at specific sections along the trajectory, thereby painting the workpiece W.

[0091] A teaching task using the teaching system 100 is performed by a user wearing a display device 5 on their head and operating the operating device 3 while viewing a VR image 8 displayed on the display device 5. In this example, the teaching system 100 is configured to be switchable between a first teaching mode and a second teaching mode. The first teaching mode is a mode in which a teaching task is performed while displaying a VR image 8 seen from a user who is within the movable range of the virtual robot 81. The second teaching mode is a mode in which a teaching task is performed while displaying a VR image 8 seen from a user who is outside the movable range of the virtual robot 81. The user switches between the first teaching mode and the second teaching mode via the input unit 41, for example.

[0092] The correspondence relationship between the operation coordinate system of the tracking system 6 and the VR coordinate system of the VR space 80 differs between the first teaching mode and the second teaching mode. That is, the correspondence relationship between the operation coordinate system of the tracking system 6 and the VR coordinate system of the VR space 80 includes a first correspondence relationship for the first teaching mode and a second correspondence relationship for the second teaching mode. According to the first correspondence relationship, the operation area of ​​the tracking system 6 is associated with a first region 80a inside the virtual fence 89. According to the second correspondence relationship, the operation area of ​​the tracking system 6 is associated with a second region 80b outside the virtual fence 89. That is, in the first teaching mode, a user operating in the operation area of ​​the tracking system 6 in the real space operates inside the virtual fence 89, which is the first region 80a in the VR space 80. On the other hand, in the second teaching mode, a user operating in the operation area of ​​the tracking system 6 in the real space operates outside the virtual fence 89, which is the second region 80b in the VR space 80. The image generating device 4 switches between the VR image 8 for the first teaching mode and the VR image 8 for the second teaching mode by switching between the first correspondence relationship and the second correspondence relationship.

[0093] First, the first teaching mode will be described. Fig. 7 is a flowchart of the first teaching mode. Fig. 8 is an example of a VR image 8 at the start of painting in the first teaching mode. Fig. 9 is an example of a VR image 8 during painting in the first teaching mode.

[0094] When the teaching operation in the first teaching mode is started, in step S101, the tracking system 6 starts tracking. That is, the tracking system 6 starts detecting and tracking the positions and orientations of the controller 3 and the display device 5 in real space.

[0095] After tracking by the tracking system 6 starts, in step S102, the image generation device 4 generates a VR image 8. The display device 5 displays the generated VR image 8. In the first teaching mode, the image generation device 4 generates the VR image 8 as seen by a user present in the first area 80a. For example, the VR image 8 in FIG. 3 is the VR image 8 in the first teaching mode.

[0096] Specifically, the image generation unit 45 reads the field definition data 42b and the object definition data 42c from the storage unit 42, and generates the VR space 80 and the virtual robot 81. In the first teaching mode, the second virtual workpiece 84b is located in the first area 80a and near the first virtual workpiece 84a.

[0097] The action generation unit 46 generates an action of the virtual robot 81 according to the position and posture of the controller 3. At this time, the image generation unit 45 and the action generation unit 46 convert the positions and postures of the controller 3 and the display device 5 in the operation coordinate system of the tracking system 6 into positions and postures in the VR coordinate system of the VR space 80, using the first correspondence relationship for the first teaching mode. The image generation unit 45 makes the virtual robot 81 perform the action generated by the action generation unit 46, and generates a VR image 8 of the line of sight according to the position and posture of the display device 5.

[0098] The user can operate the virtual robot 81 in the VR image 8 by operating the operating device 3 while viewing the VR image 8 displayed on the display device 5. When the user attempts to paint a second virtual workpiece 84b in the VR space 80 using the virtual operating device 85, the user will be located in the first area 80a. In the first teaching mode, the user is located in the first area 80a, and can therefore perform teaching work while observing the state of the virtual end effector 83 and the first virtual workpiece 84a from a position close to the virtual robot 81. The first teaching mode can also be called an approach teaching mode.

[0099] Step S102 corresponds to generating a virtual image in which a virtual robot corresponding to the robot is placed in a virtual space, displaying the virtual image, and operating the virtual robot in the virtual image in response to a user's operation on a controller for operating the virtual robot.

[0100] In step S103, the movement generation unit 46 determines whether or not the virtual robot 81 has reached its movement limit. If the virtual robot 81 has reached its movement limit, a warning is issued in step S104. For example, the image generation unit 45 displays a warning in the VR image 8 that the virtual robot 81 has reached its movement limit. Steps S103 and S104 are repeated until the movement of the virtual robot 81 is within the movement limit.

[0101] In other words, when the user operates the controller 3 to operate the virtual robot 81, the motion range of the virtual robot 81 is restricted in the same way as the real robot 1. In the real space, the robot 1 is restricted in motion by limitations on the rotation angle of the joints 14, and interference between the robot arm 11 and a safety fence or a paint booth, etc. This prevents the virtual robot 81 from performing motions that cannot be realized in the real space.

[0102] If the virtual robot 81 has not reached the movement limit, in step S105, the image generation device 4 determines whether or not it has received an instruction to generate a teaching point. Specifically, the generation processing unit 47 determines whether or not it has received an operation signal for the generation switch 32 from the operation device 3. If the generation processing unit 47 has not received an operation signal for the generation switch 32, the process returns to step S103. In other words, monitoring of the movement limit and determining whether or not it has received an instruction to generate a teaching point are repeated.

[0103] For example, when the virtual robot 81 is positioned at a desired teaching point, the user operates the generation switch 32. When the generation processing unit 47 receives the operation signal of the generation switch 32, the teaching point generation device 2 generates the teaching point in step S106. Specifically, when the generation processing unit 47 receives the operation signal of the generation switch 32, it outputs a generation command to the teaching point generation device 2. Meanwhile, the teaching point generation device 2 constantly receives from the image generation device 4 the position and orientation of the end effector 12 in real space corresponding to the position and orientation of the virtual end effector 83, and calculates the rotation angles of the multiple joints 14. The teaching point generation unit 24 generates the position and orientation of the end effector 12 closest to the time of receiving the generation command as a teaching point and stores them in the storage unit 21. The teaching point generation unit 24 also stores in the storage unit 21 the rotation angles of the multiple joints 14 corresponding to the position and orientation of the end effector 12 generated as a teaching point. At this time, the generation processing unit 47 sets the position and orientation of the virtual end effector 83 in the VR coordinate system when the operation signal of the generation switch 32 is received as a virtual teaching point.

[0104] When the teaching points are generated, the image generation device 4 displays the virtual teaching points 92 in the VR image 8, as shown in Fig. 3. Specifically, the image generation unit 45 generates the VR image 8 in which the virtual teaching points 92 generated by the generation processing unit 47 are arranged in the VR space 80. Furthermore, the image generation unit 45 displays a trajectory 93 that connects the multiple virtual teaching points 92 in time series. In other words, the image generation unit 45 arranges a predetermined number of the most recent virtual teaching points in the VR space 80 from among the multiple virtual teaching points that are sequentially generated over time.

[0105] When a teaching point is generated, the generation processing unit 47 determines whether painting is to be performed or stopped in step S107. Specifically, the generation processing unit 47 determines whether an operation signal for the injection switch 33 and an operation signal for the stop switch 34 are present. If the generation processing unit 47 receives an operation signal for the injection switch 33, it sets painting to be performed. On the other hand, if the generation processing unit 47 receives an operation signal for the stop switch 34, it sets painting to be stopped. The generation processing unit 47 outputs the painting settings to the teaching point generation device 2 and the image generation unit 45. The teaching point generation unit 24 of the teaching point generation device 2 associates the painting settings with the teaching point and stores them in the memory unit 21.

[0106] The image generation unit 45 switches between displaying and not displaying the virtual end effector 83 and the virtual paint from the virtual manipulator 85, depending on the painting setting. Specifically, when painting is set to be performed, the image generation unit 45 displays the first virtual paint 86a sprayed from the virtual end effector 83 and the second virtual paint 86b sprayed from the virtual manipulator 85 in step S108. On the other hand, when painting is set to be stopped, the image generation unit 45 hides the first virtual paint 86a sprayed from the virtual end effector 83 and the second virtual paint 86b sprayed from the virtual manipulator 85 in step S109.

[0107] 8, when the virtual end effector 83 moves to a paint injection start position near the first virtual workpiece 84a, the user operates the generation switch 32 and the injection switch 33. Then, the image generation unit 45 displays the virtual teaching point 92, as well as the first virtual paint 86a and the second virtual paint 86b, in the VR image 8.

[0108] The first virtual paint 86a corresponds to the paint ejected from the end effector 12. The first virtual paint 86a is ejected from the first virtual ejection port 83a. The first virtual paint 86a has a conical shape with the first virtual ejection axis P1 as its central axis. The shape of the first virtual paint 86a represents a virtual ejection range that corresponds to the paint ejection range in real space (for example, the range in which the paint can be effectively ejected). The first virtual paint 86a is an example of a virtual ejection target.

[0109] The second virtual paint 86b is injected from the second virtual injection port 85a. The second virtual paint 86b has a conical shape with the second virtual injection axis P2 as its central axis. The shape of the second virtual paint 86b is the same as the shape of the first virtual paint 86a. In other words, the shape of the second virtual paint 86b represents a virtual injection range that corresponds to the paint injection range in real space.

[0110] Furthermore, when painting is performed, the image generation unit 45 displays a first coating film 91a on the painted portion of the first virtual workpiece 84a. The first coating film 91a is displayed in a color different from the surface of the first virtual workpiece 84a. Similarly, the image generation unit 45 displays a second coating film 91b on the painted portion of the second virtual workpiece 84b. The second coating film 91b is displayed in a color different from the surface of the second virtual workpiece 84b. The first coating film 91a and the second coating film 91b are examples of displays indicating that the work has been treated, i.e., painted.

[0111] Returning to the explanation of the flow of the first teaching mode, in step S110, the image generating device 4 determines whether or not an instruction to end the teaching work has been given. Specifically, the image generating device 4 determines whether or not an operation signal of the end switch 35 has been received. If the image generating device 4 has not received an operation signal of the end switch 35, the process returns to step S103. In other words, the process from monitoring the operational restrictions is repeated, and the generation of teaching points continues.

[0112] As shown in FIG. 9, the user continues to paint a first virtual workpiece 84a with the virtual end effector 83 and a second virtual workpiece 84b with the virtual manipulator 85 while generating teaching points.

[0113] At this time, the image generation unit 45 displays the overlapping portion of the first paint film 91a in a distinguishable manner. In this example, the overlapping portion 91c where the first paint film 91a overlaps is displayed in a different color from the other portions. Similarly, the overlapping portion 91d where the second paint film 91b overlaps is displayed in a different color from the other portions. By displaying the first paint film 91a and the overlapping portion 91c, etc., the user can generate teaching points while determining whether the painting is appropriate.

[0114] When the generation of the teaching points is completed, the user operates the end switch 35. When the image generation device 4 receives an operation signal of the end switch 35, the generation processing unit 47 outputs an end command to the teaching point generation device 2. In step S111, the teaching point generation device 2 generates teaching data. Specifically, the teaching data generation unit 25 creates teaching data based on the multiple teaching points and the like stored in the memory unit 21, and saves the created teaching data in the memory unit 21. This completes the teaching operation in the first teaching mode.

[0115] Note that a process of resetting the generated teaching points may be executed between the start and end of the generation of the teaching points. When this process is executed, the generation of the teaching points is restarted from the beginning. That is, the teaching points stored in the storage unit 21 are reset, and the process is restarted from step S103.

[0116] In this way, the user operates the controller 3 while viewing the VR image 8 to operate the virtual robot 81. In the first teaching mode, the user's line of sight is positioned in the first area 80a, so a VR image 8 approaching the virtual end effector 83 can be generated, as shown in FIG. 3 and other figures. This allows the user to perform teaching while checking the virtual end effector 83 up close. The VR image 8 allows the user to approach the virtual end effector 83 without worrying about interference between the user and the virtual robot 81. For example, the user can generate a teaching point while viewing the virtual end effector 83 from a position where it will interfere with the virtual arm 82.

[0117] Here, the image generation unit 45 switches between displaying and hiding various objects and adjusts the display contents according to settings made by the user. Specifically, the image generation unit 45 is configured to be able to individually switch between displaying and hiding the virtual arm 82, the second virtual workpiece 84b, the virtual manipulator 85, the first virtual paint 86a, the second virtual paint 86b, the first virtual injection axis P1, the second virtual injection axis P2, the virtual support device 87, the virtual booth 88, the virtual fence 89, the virtual teaching point 92, and the trajectory 93. The user can set the display / hide switching and the display contents via the input unit 41.

[0118] FIG. 10 is another example of the VR image 8 in the first teaching mode. In the VR image 8 in FIG. 10, the virtual arm 82 is not displayed. In this way, the image generation device 4 can switch between displaying and hiding the virtual arm 82. By displaying the virtual arm 82, it is possible to generate teaching points while checking the status of not only the virtual end effector 83 but also the virtual arm 82. On the other hand, by hiding the virtual arm 82, it becomes easier to check the status of the virtual end effector 83. This makes it possible to check the part that was hidden by the virtual arm 82.

[0119] 10, the virtual manipulator 85 is displayed superimposed on the virtual end effector 83, and the second virtual workpiece 84b, the second virtual paint 86b, and the second virtual injection axis P2 are hidden. At this time, the offset amount of the virtual end effector 83 with respect to the virtual manipulator 85 is set to 0. The image generation unit 45 displays the virtual end effector 83 at the position of the virtual manipulator 85 in the VR space 80. Note that the image generation unit 45 may completely hide the virtual manipulator 85 and display only the virtual end effector 83. When the offset amount is 0, the VR image 8 appears as if the user is holding and moving the virtual end effector 83. In the example of FIG. 10, the virtual arm 82 is hidden, but the virtual end effector 83 may be superimposed on the virtual manipulator 85 while displaying the virtual arm 82.

[0120] The offset amount of the virtual end effector 83 relative to the virtual manipulator 85 can be set to any value, not just 0. The offset direction can also be set to any direction. The offset direction and offset amount are set by the user via the input unit 41. The image generation unit 45 positions the virtual end effector 83 with respect to the virtual manipulator 85, based on the set offset direction and offset amount.

[0121] Regarding the display content, the image generation unit 45 may change, for example, the outer shape (e.g., the spread angle, etc.) of the first virtual paint 86a. In the example of Fig. 10, the spread angle of the first virtual paint 86a is narrower and the dimension of the first virtual paint 86a in the direction of the first virtual injection axis P1 is longer than in the example of Fig. 8, etc. When the second virtual paint 86b is displayed, the second virtual paint 86b may also be changed in the same way as the first virtual paint 86a.

[0122] Furthermore, the image generating unit 45 can change the display mode of the virtual teaching points 92. For example, the image generating unit 45 is configured to be switchable between a mode in which all virtual teaching points 92 are displayed and a mode in which a finite number of nearest virtual teaching points 92 are displayed. When a finite number of virtual teaching points 92 are displayed, the number of virtual teaching points 92 to be displayed can be changed. In the example of FIG. 10, the number of virtual teaching points 92 to be displayed is the two nearest ones. Note that the image generating unit 45 can also hide the trajectory 93.

[0123] Next, the second teaching mode will be described. Fig. 11 is a flowchart of the second teaching mode. Fig. 12 is an example of the VR space 80 in the second teaching mode. Fig. 13 is an example of the VR image 8 in the second teaching mode.

[0124] When the teaching operation in the second teaching mode is started, in step S201, the tracking system 6 starts tracking. That is, the tracking system 6 starts detecting and tracking the positions and orientations of the controller 3 and the display device 5 in real space.

[0125] After tracking by the tracking system 6 starts, in step S202, the image generation device 4 generates a VR image 8. The display device 5 displays the generated VR image 8. In the second teaching mode, the image generation device 4 generates the VR image 8 as seen by a user present in the second area 80b.

[0126] Specifically, the image generation unit 45 reads the field definition data 42b and the object definition data 42c from the storage unit 42, and generates the VR space 80 and the virtual robot 81. In the second teaching mode, as shown in FIG. 12, the second virtual workpiece 84b is placed in the second area 80b.

[0127] The action generation unit 46 generates an action of the virtual robot 81 according to the position and posture of the controller 3. At this time, the image generation unit 45 and the action generation unit 46 convert the positions and postures of the controller 3 and the display device 5 in the operation coordinate system of the tracking system 6 into positions and postures in the VR coordinate system of the VR space 80, using the second correspondence relationship for the second teaching mode. The image generation unit 45 makes the virtual robot 81 perform the action generated by the action generation unit 46, and generates a VR image 8 of the line of sight according to the position and posture of the display device 5.

[0128] The user can operate the virtual robot 81 in the VR image 8 by operating the operating device 3 while viewing the VR image 8 displayed on the display device 5. In the second teaching mode, as shown in FIG. 13 , the user generates teaching points while painting a second virtual workpiece 84b in the second area 80b using the virtual operating device 85. Because the user is located in the second area 80b, the user can perform teaching work while observing the overall state of the virtual robot 81 from a position distant from the virtual robot 81. The second teaching mode can also be called a remote teaching mode.

[0129] In the second teaching mode, the user is away from the virtual end effector 83 and the first virtual workpiece 84a. However, in the second teaching mode, as in the first teaching mode, the relative positional relationship between the second virtual workpiece 84b and the virtual manipulator 85 coincides with the relative positional relationship between the first virtual workpiece 84a and the virtual end effector 83. Therefore, by checking the relative positional relationship between the second virtual workpiece 84b and the virtual manipulator 85, the user can substantially check the relative positional relationship between the first virtual workpiece 84a and the virtual end effector 83. In other words, in the second teaching mode, it is possible to generate teaching points while checking the overall operation of the virtual robot 81 and substantially checking the operation of the virtual end effector 83.

[0130] Step S202 corresponds to generating a virtual image in which a virtual robot corresponding to the robot is placed in a virtual space, displaying the virtual image, and operating the virtual robot in the virtual image in response to a user's operation on a controller for operating the virtual robot.

[0131] The processing from step S203 onwards in the second teaching mode is basically the same as the processing from step S103 onwards in the first teaching mode. However, since the user is away from the virtual end effector 83, the display of the first virtual injection axis P1, the first virtual paint 86a, the first paint film 91a, the overlapping portion 91c, the virtual teaching point 92, and the trajectory 93 may be omitted.

[0132] Furthermore, the image generating device 4 of the teaching system 100 has a playback mode that displays a VR image 8 in which a virtual robot 81 moves according to teaching points. Switching to the playback mode is performed, for example, by the user via the input unit 41. The playback mode is executed, for example, when checking teaching data. In the playback mode, generation of teaching points and generation of teaching data are not performed. The playback mode is executed, for example, with the display device 5 attached to the user's head. Figure 14 is a flowchart of the playback mode.

[0133] When the teaching operation in the playback mode is started, in step S301, the tracking system 6 starts tracking. At this time, the tracking system 6 starts detecting and tracking the position and orientation of the display device 5 in real space. Unlike the first teaching mode and the second teaching mode, the tracking system 6 does not detect and track the position and orientation of the controller 3.

[0134] The image generating device 4 acquires teaching data from the storage unit 21 of the teaching point generating device 2 in step S302.

[0135] Furthermore, in step S303, the image generation device 4 generates a VR image 8. The display device 5 displays the generated VR image 8. Specifically, the image generation unit 45 reads the field definition data and the object definition data from the storage unit 42 and generates the VR space 80, the virtual robot 81, and the like. In the playback mode, the second virtual workpiece 84b and the virtual manipulator 85 are not generated. The action generation unit 46 operates the virtual robot 81 according to the teaching data. The image generation unit 45 basically generates the VR image 8 as seen by the user present in the second area 80b. That is, the image generation unit 45 converts the position and orientation of the display device 5 in the operation coordinate system of the tracking system 6 into the position and orientation in the VR coordinate system of the VR space 80 using the second correspondence relationship for the second teaching mode. The image generation unit 45 generates the VR image 8 of the line of sight according to the position and orientation of the display device 5.

[0136] In the playback mode, a VR image 8 is easily formed in which the entire virtual robot 81 is included in the angle of view. The user can check the teaching data by checking the movement of the virtual robot 81 displayed on the display device 5. Even in the playback mode, the display device 5 is tracked, and a VR image 8 is generated according to the position and posture of the display device 5, so the user can check the movement of the virtual robot 81 from a desired position and angle.

[0137] In the playback mode, virtual equipment and the like can be hidden to make it easier to check the operation of the virtual robot 81. For example, the virtual booth 88 and the virtual fence 89 can be hidden. The user can set whether to display or hide various objects and the like via the input unit 41.

[0138] Furthermore, the user's position in the playback mode can be changed. For example, the image generation device 4 may generate a VR image 8 from the same user position as in the first teaching mode. That is, the image generation unit 45 converts the position and orientation of the display device 5 in the operation coordinate system of the tracking system 6 into the position and orientation of the VR coordinate system of the VR space 80 using a first correspondence relationship for the first teaching mode. Alternatively, the image generation unit 45 may convert the position and orientation of the display device 5 in the operation coordinate system of the tracking system 6 into the position and orientation of the VR coordinate system of the VR space 80 using a correspondence relationship dedicated to the playback mode, which is different from the first teaching mode and the second teaching mode.

[0139] Furthermore, the movement speed of the virtual robot 81 in the playback mode may be changeable. The movement speed of the virtual robot 81 in the playback mode may be changeable between a normal speed, which is the speed of the robot 1 actually moving according to the teaching data, a speed faster than the normal speed (i.e., high speed), and a speed slower than the normal speed (i.e., low speed).

[0140] In the teaching system 100 configured as described above, the first virtual workpiece 84a to be processed by the virtual end effector 83 is displayed in the VR image 8, allowing the user to generate teaching points while checking the relative positional relationship between the virtual end effector 83 and the first virtual workpiece 84a. Unlike generating teaching points while checking the operation of the robot 1 in real space, the user generates teaching points while checking the VR image 8 on the display 5, eliminating the need to worry about interference between the actual robot 1 and the user. Therefore, the user can easily determine whether the position and movement of the robot 1 are appropriate when generating teaching points. For example, when the robot 1 is performing a process of injecting an injection target into a workpiece W, the distance from the injection port 16 to the workpiece W and the orientation of the injection port 16 relative to the workpiece W (i.e., the injection direction) may affect the quality of the processing. When the surface of the workpiece W is not flat, the influence of the distance from the injection port 16 and the orientation of the injection port 16 is particularly significant. According to the teaching system 100, the distance from the first virtual injection port 83a of the virtual end effector 83 to the workpiece W and the direction of the first virtual injection port 83a can be easily confirmed in the VR image 8.

[0141] Furthermore, in the VR image 8, the virtual end effector 83 is performing processing, i.e., painting, on the first virtual workpiece 84a. This allows the user to generate teaching points while checking the status of processing by the virtual end effector 83.

[0142] Specifically, the first paint film 91a is applied to the painted portion of the first virtual workpiece 84a. This allows the user to generate teaching points so that no untreated portions are generated. Furthermore, overlapping portions 91c where the first paint film 91a overlaps are displayed so that they can be distinguished from other portions. This allows the user to generate teaching points so as to reduce unevenness in the processing.

[0143] In addition, the VR image 8 displays a first virtual paint 86a as a target to be injected from the virtual end effector 83. By displaying the first virtual paint 86a, the user can easily determine whether the relative positional relationship between the virtual end effector 83 and the first virtual workpiece 84a is appropriate. This is particularly effective for processing such as painting, in which the virtual end effector 83 and the first virtual workpiece 84a are performed without contact.

[0144] Here, the second virtual paint 86b is also injected from the virtual manipulator 85, and painting is performed on the second virtual workpiece 84b, so the user can understand the state of the virtual end effector 83 and the first virtual workpiece 84a without looking at them. When the user is away from the first virtual workpiece 84a and the virtual end effector 83, as in the second teaching mode, painting the second virtual workpiece 84b with the virtual manipulator 85 is particularly effective. Even in the first teaching mode, when the second virtual workpiece 84b is away from the first virtual workpiece 84a, painting the second virtual workpiece 84b with the virtual manipulator 85 is effective.

[0145] The VR image 8 also displays virtual equipment corresponding to the equipment present around the robot 1. Specifically, the VR image 8 displays a virtual support device 87 and the like. The user can also avoid interference between the robot 1 in the real space and the surrounding equipment by generating teaching points so that the virtual robot 81 does not interfere with the virtual equipment.

[0146] Furthermore, the image generation device 4 sets movement limits on the movement range of the virtual robot 81, and notifies, i.e., issues a warning, when the virtual robot 81 reaches the movement limits. In the VR image 8, unlike in the real space, the virtual robot 81 can move freely. The virtual robot 81 can even perform movements that the robot 1 cannot perform in the real space. Therefore, by setting movement limits on the movement range of the virtual robot 81, it is possible to prevent the virtual robot 81 from performing movements that the robot 1 cannot perform in the real space.

[0147] The virtual robot 81 moves in conjunction with the movement of the controller 3 in real space. That is, the movement of the controller 3 in real space is tracked by the tracking system 6. The virtual end effector 83 is moved in conjunction with the movement of the controller 3 based on the tracking results of the controller 3. Therefore, the user can move the virtual end effector 83 with the same sensation as moving his or her own hand. This allows the user to generate teaching points while moving the virtual end effector 83 precisely and as intended. For example, in the case of teaching using a teach pendant, proficiency is required to operate the teach pendant. In contrast, by linking the virtual robot 81 to the movement of the controller 3 in real space, teaching points can be specified intuitively and do not require high proficiency. As a result, teaching points can be easily generated.

[0148] In this example, a virtual controller 85 is further displayed in the VR image 8. Since the virtual controller 85 corresponding to the controller 3 operated by the user is displayed in the VR image 8, the sensation of the hand matches the visual sense, making it easier to operate the controller 3, and therefore the virtual end effector 83.

[0149] In addition, in the VR image 8 of the first teaching mode, the virtual manipulator 85 is placed at a position offset by a predetermined amount with respect to the virtual robot 81. Specifically, since the virtual end effector 83 is placed at a position offset from the virtual manipulator 85, the user can easily check the state of the virtual end effector 83.

[0150] Furthermore, a second virtual workpiece 84b corresponding to the virtual manipulator 85 is displayed in the VR image 8, and the relative positional relationship between the virtual manipulator 85 and the second virtual workpiece 84b matches the relative positional relationship between the virtual end effector 83 and the first virtual workpiece 84a. Therefore, when the virtual manipulator 85 paints the second virtual workpiece 84b, the virtual end effector 83 paints the same position of the first virtual paint 86a as the painting position of the second virtual workpiece 84b. In other words, when the user paints the second virtual workpiece 84b while looking at the virtual manipulator 85 and the second virtual workpiece 84b in the VR image 8, the virtual end effector 83 paints the first virtual workpiece 84a in the same way. The user can generate teaching points while looking at the virtual manipulator 85 and the second virtual workpiece 84b, without looking at the virtual end effector 83. On the other hand, the user can also check the status of the virtual end effector 83 by looking toward the virtual end effector 83 while painting the second virtual workpiece 84b with the virtual manipulator 85. This allows the user to check the status of the virtual end effector 83 from an angle different from that of the virtual manipulator 85.

[0151] Furthermore, the VR image 8 displays virtual teaching points 92 corresponding to the generated teaching points. This allows the user to generate teaching points while checking the generated teaching points in the VR image 8. Furthermore, the VR image 8 displays a plurality of virtual teaching points 92 arranged in chronological order. This allows the user to imagine the trajectory of the virtual end effector 83 from the plurality of virtual teaching points 92. In other words, the user can generate teaching points while imagining the trajectory of the virtual end effector 83. More specifically, the VR image 8 displays a finite number of the most recent virtual teaching points 92. In other words, old teaching points that are not very useful in forming the future trajectory of the virtual end effector 83 are not displayed. This allows only a useful number of virtual teaching points 92 to be displayed in the VR image 8, thereby simplifying the display of the VR image 8.

[0152] Other Embodiments As described above, the above embodiment has been described as an example of the technology disclosed in this application. However, the technology of the present disclosure is not limited to this and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, the components described in the above embodiment can be combined to create new embodiments. Furthermore, the components described in the accompanying drawings and detailed description may include not only components essential for solving the problem, but also components that are not essential for solving the problem in order to exemplify the technology. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not be interpreted as immediately determining that these non-essential components are essential.

[0153] The robot 1 is not limited to an industrial robot, but may also be a medical robot. The processing performed by the robot 1 is not limited to painting, but may also be, for example, welding, cleaning, or shot blasting. The processing performed by the robot 1 may also be cutting or polishing, a transport processing for transporting a workpiece, or an inspection processing for inspecting a workpiece. The target ejected by the robot 1 is not limited to paint, but may also be ink, cleaning liquid, water, filler material, abrasive (e.g., shot material), sealant, laser, flame, ultrasound, electromagnetism, etc. The inspection processing may also be a processing for inspecting the appearance of a workpiece using a camera used in the end effector 12.

[0154] Furthermore, the robot arm 11 may have redundancy. In other words, there may be multiple shapes (i.e., joint angles) of the robot arm 11 for placing the end effector 12 at a predetermined position and posture, and a unique shape may not be determined. In this case, the user may select the joint angle of the robot arm 11 corresponding to the generated teaching point from multiple joint angles, or the teaching point generation device 2 may select the joint angle based on various conditions. For example, as a configuration in which the user selects the shape of the robot arm 11, the image generation device 4 may present multiple shapes of the robot arm 11 to the user via the display 5, and the user may select one shape of the robot arm 11.

[0155] The mechanism for changing the position and posture of the end effector 12 is not limited to the robot arm 11. The robot 1 may be provided with another mechanism for changing the position and posture of the end effector 12 instead of the robot arm 11. The robot arm 11 is not limited to a vertical articulated type, and may be a horizontal articulated type, a parallel link type, a rectangular coordinate type, a polar coordinate type, or the like.

[0156] The teaching point generation device 2 may be formed from one device or from multiple devices. The teaching point generation device 2 does not have to have the robot control unit 26. In other words, the teaching point generation device 2 does not have to have the function of controlling the robot 1 based on the teaching data.

[0157] The teaching point generation device 2 receives the position and orientation of the controller 3 in the operation coordinate system corresponding to the virtual teaching point from the image generation device 4, and generates the teaching point of the robot 1 from the received position and orientation of the controller 3, but is not limited to this. The teaching point generation device 2 can employ any method as long as it generates a teaching point corresponding to the position of the virtual robot in the virtual space.

[0158] For example, the teaching point generation device 2 may determine the position and orientation of the end effector 12 in the robot coordinate system from the detection results of the tracking system 6, and further receive a command to generate a teaching point from the manipulator 3 or via the image generation device 4 to generate a teaching point from the position and orientation of the end effector 12. In other words, the teaching point generation device 2 may have the function of the tracking control unit 44. The detection results of the first sensor 62 of the tracking system 6 and illumination information of the light emitter 61 are input to the teaching point generation device 2, and the teaching point generation device 2 determines the position and orientation of the manipulator 3 in the manipulation coordinate system based on the detection results of the first sensor 62 and the illumination information of the light emitter 61. The teaching point generation device 2 converts the position and orientation of the manipulator 3 in the manipulation coordinate system into the position and orientation of the end effector 12 in the robot coordinate system. The teaching point generation device 2 sets the position and orientation of the end effector 12 at the time of receiving the command to generate a teaching point as the teaching point.

[0159] Alternatively, the teaching point generation device 2 may generate a teaching point by converting the position and orientation in the VR coordinate system of the virtual teaching point 92 generated by the image generation device 4 into the position and orientation of the teaching point in the robot coordinate system.

[0160] Alternatively, the teaching point generation device 2 may determine the position and orientation of the end effector 12 in the robot coordinate system from the detection result of the tracking system 6, and the image generation device 4 may convert the position and orientation of the end effector 12 determined by the teaching point generation device 2 into the position and orientation of a virtual end effector 83 in the VR coordinate system to generate the VR image 8, and generate a virtual teaching point in response to a command from the user.The teaching point generation device 2 may then receive a command to generate a teaching point via the controller 3 or the image generation device 4, and use the position and orientation of the end effector 12 at the time of receiving the command to generate the teaching point as the teaching point.

[0161] That is, since the operation coordinate system, the robot coordinate system, and the VR coordinate system correspond to one another, the teaching point generation device 2 may generate teaching points in the robot coordinate system from the position and posture of the manipulator 3 in the operation coordinate system, or may generate teaching points in the robot coordinate system from the positions and postures of the virtual teaching point 92, the virtual end effector 83, the virtual manipulator 85, etc. in the VR coordinate system. In either case, the generated teaching points in the robot coordinate system correspond to positions in the virtual space of the virtual robot.

[0162] The robot coordinate system, operation coordinate system, and VR coordinate system only need to correspond to each other, and they do not need to have the same scale. In other words, the image generation device 4 may generate a VR image 8 based on a VR coordinate system that is twice the scale of the robot coordinate system. Furthermore, the robot coordinate system, operation coordinate system, and VR coordinate system only need to correspond to each other, and they may be flipped left and right or up and down.

[0163] The teaching point generating device 2 and the image generating device 4 are configured as separate hardware devices, but may also be configured as a single device.

[0164] The image generation device 4 may be formed from one device or multiple devices. The virtual image generated by the image generation device 4 is not limited to a VR image, but may be an AR (Augmented Reality) image or an MR (Mixed Reality) image. Some or all of the virtual arm 82, first virtual workpiece 84a, second virtual workpiece 84b, virtual manipulator 85, first virtual paint 86a, second virtual paint 86b, virtual equipment, first paint film 91a, second paint film 91b, virtual teaching point 92, and trajectory 93 included in the virtual image do not have to be configured to be switchable between display and non-display. The VR image 8 generated by the image generating device 4 only needs to have at least a virtual robot 81, and some or all of the first virtual workpiece 84a, the second virtual workpiece 84b, the virtual manipulator 85, the first virtual paint 86a, the second virtual paint 86b, the virtual equipment, the first paint film 91a, the second paint film 91b, the virtual teaching point 92, and the trajectory 93 can be omitted.

[0165] Image generating device 4 may have only one or two of the first teaching mode, the second teaching mode, and the playback mode, or one or two of the first teaching mode, the second teaching mode, and the playback mode may be omitted. For example, of the first teaching mode, the second teaching mode, and the playback mode, image generating device 4 may have only the first teaching mode, only the second teaching mode, or only the playback mode.

[0166] The robot coordinate system, operation coordinate system, and VR coordinate system only need to correspond to each other, and they do not need to have the same scale. In other words, the image generation device 4 may generate a VR image 8 based on a VR coordinate system that is twice the scale of the robot coordinate system. Furthermore, the robot coordinate system, operation coordinate system, and VR coordinate system only need to correspond to each other, and they may be flipped left and right or up and down.

[0167] The operation device 3 may be, for example, a teaching pendant. The operation unit 31 may be provided in a device separate from the operation device 3. For example, the operation unit 31 may be provided in the display device 5 or the image generating device 4.

[0168] The controller sensor that detects the position and orientation of the controller 3 in real space is not limited to the tracking system 6. For example, the controller sensor may be a tracking system different from the tracking system 6. For example, the controller sensor may be a tracking system having a light-emitting tracking marker provided on the controller 3 and a camera that takes stereo images of the real space in which the user exists. For example, the tracking marker emits infrared light. The camera is an infrared camera. Image data captured by the camera is transmitted to the image generation device 4. The image generation device 4 processes the image data from the camera to determine the position and orientation of the controller 3 in real space. The tracking system may be either an outside-in system or an inside-out system.

[0169] The method of notifying the user that the virtual robot 81 has reached its movement limit is not limited to the notification information included in the VR image 8. For example, the notification to the user that the virtual robot 81 has reached its movement limit may be a sound or vibration emitted by a device such as the display device 5 or the operation device 3.

[0170] In this example, a teaching point is generated each time the generation switch 32 of the controller 3 is operated, but the generation of teaching points is not limited to being performed intermittently in this manner. For example, teaching points may be generated continuously over a period from the start time of generation to the end time of generation determined in accordance with the operation of the controller 3. In this case, for example, multiple points on the movement trajectory of the virtual controller 85 moving in the virtual space 80 are generated as teaching points.

[0171] The command for generating the teaching data is not limited to an end command. A command for generating teaching data may be input to the teaching data generation unit 25 by a user operating the input unit 41 of the controller 3, the display 5, the image generation device 4, or the input unit of the teaching point generation device 2. The teaching data may include a set of multiple joints 14 for complementing the movement of the end effector 12 between one teaching point and the next teaching point. Furthermore, the rotation angle of the joint 14 may not be the rotation angle itself, but may be a value related to the rotation angle. For example, the rotation angle of the joint 14 may be the rotation angle of the motor 15.

[0172] The display of the posture of the virtual end effector 83 and the display of the distance from the virtual end effector 83 described above are merely examples. For example, with regard to the posture of the virtual end effector 83, the angle of the first virtual injection axis P1 in the virtual coordinate system or the angle of the first virtual injection axis P1 with respect to the normal direction of the first virtual workpiece 84a may be displayed as a numerical value in the VR image 8. With regard to the posture of the virtual end effector 83, the first virtual paint 86a may be color-coded according to the distance from the virtual end effector 83. That is, the color of the first virtual paint 86a may represent the distance from the virtual end effector 83. Alternatively, the distance from the virtual end effector 83 may be displayed as a numerical value in the VR image 8. Furthermore, the distance from the virtual end effector 83 to the workpiece W may be displayed.

[0173] The display device 5 is not limited to an HMD, and may be, for example, a general display.

[0174] The function of operating the robot 1 of the teaching system 100 is not essential. That is, the teaching point generating device 2 does not need to have the robot control unit 26.

[0175] The teaching system 100 may not have the function of generating teaching points, but may instead generate a virtual image in which a virtual robot operates in accordance with the teaching points and display the virtual image. That is, the teaching system may include a teaching point generation device that generates teaching points for a robot, an image generation device that generates a virtual image in which a virtual robot corresponding to the robot is placed in a virtual space, and a display that displays the virtual image, and the image generation device may generate the virtual image in which the virtual robot operates in accordance with the teaching points generated by the teaching point generation device. In this case, the teaching point generation device simply generates input teaching points, and the image generation device generates a virtual image in which the virtual robot operates in accordance with the teaching points generated by the teaching point generation device. Such a teaching system allows a user to determine the appropriateness of the teaching points by observing the operation of the virtual robot.

[0176] The field definition data 42b, the object definition data 42c, etc. are not limited to those created based on actual design data of the robot 1, etc. or actual measurements of the robot 1, etc. For example, the field definition data 42b, the object definition data 42c, etc. may be created using simulation software such as an offline programming tool or offline teaching software. In a computer loaded with the simulation software, an instructor can create data related to the field and objects in the VR space (e.g., the field definition data 42b, the object definition data 42c, etc.) by inputting numerical information, etc. via a user interface. The simulation software may cause the computer to implement a function for creating data related to the field and objects by scanning a site, such as a factory, where the robot 1 is actually located.

[0177] The teaching system 100 may output the generated teaching data to an external device. For example, the teaching system 100 may output the teaching data to a computer into which simulation software has been loaded. The computer can display the teaching data using the functions of the simulation software.

[0178] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, application-specific integrated circuits (ASICs), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuitry. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or processor.

[0179] The technology of the present disclosure can be summarized as follows.

[0180] [1] The teaching system 100 comprises a teaching point generation device 2 that generates teaching points for the robot 1, an operator 3 operated by a user, an image generation device 4 that generates a VR image 8 (virtual image) in which a virtual robot 81 corresponding to the robot 1 is placed in a VR space 80 (virtual space), and a display 5 that displays the VR image 8, wherein the image generation device 4 generates the VR image 8 in which the virtual robot 81 moves in response to an operation on the operator 3, and the teaching point generation device 2 generates teaching points corresponding to the position in the VR space 80 of the virtual robot 81 generated by the image generation device 4.

[0181] According to this configuration, the user can operate the virtual robot 81 in the VR space 80 by operating the controller 3 while viewing the VR image 8 displayed on the display device 5. The user can then generate teaching points while operating the virtual robot 81 in the VR space 80. In this case, because it is a VR image 8, the user can check the operation of the virtual robot 81 from various viewpoints, such as a viewpoint close to the virtual robot 81 and a viewpoint far from the virtual robot 81. For example, the user can check the operation of the virtual robot 81 from a position close to the virtual robot 81, in other words, from within the operating range of the robot 81, which cannot be realized in real space. This allows the user to appropriately generate teaching points.

[0182] [2] In the teaching system 100 described in [1], the image generating device 4 places in the VR space 80 a first virtual workpiece 84a (virtual workpiece) corresponding to the workpiece W to be processed by the robot 1.

[0183] According to this configuration, the first virtual workpiece 84a is displayed in addition to the virtual robot 81 in the VR image 8, so the user can generate teaching points while checking the relative positional relationship between the virtual robot 81 and the first virtual workpiece 84a.

[0184] [3] In the teaching system 100 described in [1] or [2], the image generating device 4 causes the virtual robot 81 to execute processing on the first virtual workpiece 84a in response to an operation on the operating device 3.

[0185] According to this configuration, the user can generate teaching points while checking whether the virtual robot 81 is properly processing the first virtual workpiece 84a. As a result, teaching points can be generated properly.

[0186] [4] In the teaching system 100 described in any one of [1] to [3], the robot 1 includes a multi-joint robot arm 11 and an end effector 12 connected to the robot arm 11, the image generation device 4 places a virtual arm 82 corresponding to the robot arm 11 and a virtual end effector 83 corresponding to the end effector 12 in the VR space 80, and the teaching point generation device 2 generates, as a teaching point, the position and posture of the end effector 12 in the real space corresponding to the position and posture of the virtual end effector 83 in the VR space 80.

[0187] According to this configuration, the robot 1 includes a robot arm 11 and an end effector 12. Then, based on the virtual end effector 83, the position and posture of the end effector 12 are generated as teaching points. The end effector 12 is a part of the robot 1 that applies an action to a workpiece. By generating the position and posture of the end effector 12 as teaching points, the operation of the robot 1 can be appropriately defined by the teaching points.

[0188] [5] In the teaching system 100 according to any one of [1] to [4], the image generating device 4 is configured to be able to switch between displaying and hiding the virtual arm 82 in the VR image 8.

[0189] According to this configuration, the user can easily check the operation of the virtual end effector 83 by, for example, hiding the virtual arm 82. On the other hand, the user can check the operation of the virtual arm 82 in addition to the virtual end effector 83 by displaying the virtual arm 82.

[0190] [6] In the teaching system 100 described in any one of [1] to [5], the image generation device 4 places a virtual support device 87, a virtual booth 88 or a virtual fence 89 (virtual equipment) corresponding to the equipment present around the robot 1 in the VR space 80.

[0191] According to this configuration, the user can generate teaching points while avoiding interference between the virtual robot 81 and the virtual equipment.

[0192] [7] In the teaching system 100 according to any one of [1] to [6], the image generating device 4 generates a VR image 8 seen by a user who is present within the movable range of the virtual robot 81.

[0193] According to this configuration, it is possible to reproduce a situation viewed from a position where the user and the robot 1 would interfere with each other in the real space using the VR image 8. Therefore, the user can generate teaching points while observing the virtual end effector 83 and the like from an angle that cannot be observed in the real space.

[0194] [8] In the teaching system 100 described in any one of [1] to [7], the teaching system 100 further includes a tracking system 6 (operator sensor) that detects the position and posture of the controller 3 in the real space, and the image generating device 4 causes the virtual robot 81 to perform an action in the VR space 80 that is linked to the action of the controller 3 in the real space based on the detection results of the tracking system 6.

[0195] According to this configuration, the virtual robot 81 is linked to the movement of the controller 3 in real space, so the user can operate the virtual robot 81 with the same feeling as operating the controller 3. For example, when teaching using a teach pendant, proficiency is required to operate the teach pendant. In contrast, by linking the virtual robot 81 to the movement of the controller 3 in real space, teaching points can be specified intuitively, and high proficiency is not required. As a result, teaching points can be generated easily.

[0196] [9] In the teaching system 100 described in any one of [1] to [8], the image generation device 4 places a virtual manipulator 85 corresponding to the manipulator 3 in the VR space 80 based on the detection result of the tracking system 6.

[0197] According to this configuration, a virtual controller 85 corresponding to the controller 3 being operated by the user in the real space is displayed in the VR image 8, so that the user's sense of operation matches their vision, allowing them to operate the controller 3 without feeling uncomfortable.

[0198]

[10] In the teaching system 100 according to any one of [1] to [9], the image generating device 4 places the virtual manipulator 85 in the VR space 80 at a position offset by a predetermined amount with respect to the virtual robot 81.

[0199] According to this configuration, the virtual robot 81 is placed at a position a certain distance away from the virtual manipulator 85 operated by the user in the VR image 8. It is easier to confirm the operation of the virtual robot 81 if the virtual robot 81 is somewhat farther away from the user than if the virtual robot 81 is too close to the user.

[0200]

[11] In the teaching system 100 according to any one of [1] to

[10] , the image generating device 4 places a virtual teaching point 92 corresponding to the teaching point in the VR space 80.

[0201] According to this configuration, the user can check the generated teaching points in the VR image 8.

[0202]

[12] In the teaching system 100 according to any one of [1] to

[11] , the image generating device 4 places a predetermined number of nearest virtual teaching points 92 in the VR space 80.

[0203] This configuration makes it easier for the user to imagine the trajectory of the virtual robot 81 from the predetermined number of nearest virtual teaching points 92. Furthermore, by limiting the number of virtual teaching points 92 to be displayed to the predetermined number of nearest points, it is possible to prevent the VR image 8 from becoming cluttered.

[0204]

[13] In the teaching system 100 described in any one of [1] to

[12] , the image generating device 4 sets a movement limit on the movement range of the virtual robot 81, and when the virtual robot 81 reaches the movement limit, it notifies the user that the virtual robot 81 has reached the movement limit.

[0205] According to this configuration, the user can set restrictions on the movement of the virtual robot 81 even in the VR space 80, thereby preventing teaching points from being generated in a range in which the robot 1 cannot move in the real space.

[0206]

[14] In the teaching system 100 described in any one of [1] to

[13] , the image generation device 4 has a playback mode that displays a VR image 8 in which a virtual robot 81 moves according to teaching points generated by the teaching point generation device 2.

[0207] According to this configuration, the user can check the behavior of the virtual robot 81 according to the generated teaching points, and can easily determine whether the generated teaching points are appropriate.

[0208]

[15] The teaching system 100 includes a teaching point generation device 2 that generates teaching points for the robot 1, an image generation device 4 that generates a VR image 8 in which a virtual robot 81 corresponding to the robot 1 is placed in a VR space 80, and a display 5 that displays the VR image 8, and the image generation device 4 generates the VR image 8 in which the virtual robot 81 operates according to the teaching points generated by the teaching point generation device 2.

[0209] According to this configuration, the user can check the behavior of the robot 1 operating in accordance with the teaching points by viewing the VR image 8 in which the virtual robot 81 operates. In this case, the user can check the behavior of the virtual robot 81 from viewpoints located at various locations, such as positions close to the virtual robot 81.

[0210]

[16] The robot system 1000 includes the teaching system 100 according to any one of [1] to

[15] and a robot 1 that operates according to teaching points generated by a teaching point generating device 2.

[0211] According to this configuration, the user can operate the virtual robot 81 in the VR space 80 by operating the controller 3 while viewing the VR image 8 displayed on the display device 5. The user can then generate teaching points while operating the virtual robot 81 in the VR space 80. In this case, because it is a VR image 8, the user can check the operation of the virtual robot 81 from various viewpoints, such as a viewpoint close to the virtual robot 81 and a viewpoint far from the virtual robot 81. For example, the user can check the operation of the virtual robot 81 from a position close to the virtual robot 81, in other words, from within the operating range of the robot 81, which cannot be realized in real space. This allows the user to appropriately generate teaching points.

[0212]

[17] The teaching method for the robot 1 includes generating a VR image 8 in which a virtual robot 81 corresponding to the robot 1 is placed in a VR space 80, displaying the VR image 8, operating the virtual robot 81 in the VR image 8 in response to a user's operation on an operating device 3 for operating the virtual robot 81, and generating a teaching point for the robot 1 corresponding to the position of the virtual robot 81 in the VR space 80.

[0213] According to this configuration, the user can operate the virtual robot 81 in the VR space 80 by operating the controller 3 while viewing the displayed VR image 8. The user can then generate teaching points while operating the virtual robot 81 in the VR space 80. In this case, because it is a VR image 8, the user can check the operation of the virtual robot 81 from various viewpoints, such as a viewpoint close to the virtual robot 81 and a viewpoint far from the virtual robot 81. For example, the user can check the operation of the virtual robot 81 from a position close to the virtual robot 81, in other words, from within the operating range of the robot 81, which cannot be realized in the real space. This allows the user to appropriately generate teaching points.

[0214]

[18] The teaching program 42a for the robot 1 enables the computer to realize the following functions: generating a VR image 8 in which a virtual robot 81 corresponding to the robot 1 is placed in a VR space 80; operating the virtual robot 81 in the VR image 8 in response to a user's operation on the controller 3 for operating the virtual robot 81; and generating a teaching point for the robot 1 corresponding to the position of the virtual robot 81 in the VR space 80.

[0215] According to this configuration, the user can operate the virtual robot 81 in the VR space 80 by operating the controller 3 while viewing the VR image 8. The user can then generate teaching points while operating the virtual robot 81 in the VR space 80. In this case, because it is the VR image 8, the user can check the operation of the virtual robot 81 from various viewpoints, such as a viewpoint close to the virtual robot 81 and a viewpoint far from the virtual robot 81. For example, the user can check the operation of the virtual robot 81 from a position close to the virtual robot 81, in other words, from within the operating range of the robot 81, which cannot be realized in the real space. This allows the user to appropriately generate teaching points. [Explanation of symbols]

[0216] 1000 Robot System 100 Teaching System 1. Robot 11 Robotic Arm 12 End Effector 14 Joints 2 Teaching point generator 3 Controller 4. Image generation device 42a Teaching Program 5 Display 6 Tracking system (operator sensor) 8 VR images (virtual images) 80 VR space (virtual space) 81 Virtual Robot 82 Virtual Arm 83 Virtual End Effector 84a First Virtual Work (Virtual Work) 85 Virtual Manipulator 87 Virtual Support Device (Virtual Equipment) 88 Virtual Booths (Virtual Facilities) 89 Virtual Fence (Virtual Facility)

Claims

1. a teaching point generating device that generates teaching points for a robot in real space; a controller operated by a user in real space; an image generation device that generates a virtual image in which a virtual robot corresponding to the robot is placed in a virtual space; a display that displays the virtual image; a tracking system that detects the position and orientation of the controller in real space and the position and direction of the user's line of sight; the image generation device generates the virtual image in which the virtual robot performs an action linked to the position and orientation of the controller in real space detected by the tracking system, and the virtual image corresponds to the position and orientation of the user's line of sight in the virtual space detected by the tracking system; The teaching point generation device is a teaching system that generates the teaching point corresponding to the position in the virtual space of the virtual robot generated by the image generation device.

2. 2. The teaching system according to claim 1, The image generation device is a teaching system that places a virtual workpiece corresponding to the workpiece to be processed by the robot in the virtual space.

3. 3. The teaching system according to claim 2, The image generation device is a teaching system that causes the virtual robot to perform processing on the virtual workpiece in response to an operation on the controller.

4. 2. The teaching system according to claim 1, the robot includes an articulated robot arm and an end effector coupled to the robot arm; the image generation device places a virtual arm corresponding to the robot arm and a virtual end effector corresponding to the end effector in the virtual space; The teaching point generating device is a teaching system that generates, as the teaching point, the position and orientation of the end effector in real space that correspond to the position and orientation of the virtual end effector in the virtual space.

5. 5. The teaching system according to claim 4, The image generating device is a teaching system configured to be able to switch between displaying and hiding the virtual arm in the virtual image.

6. 2. The teaching system according to claim 1, The image generation device is a teaching system that places virtual equipment corresponding to equipment present around the robot in the virtual space.

7. 2. The teaching system according to claim 1, The image generation device is a teaching system that generates the virtual image as seen by a user who is within the movable range of the virtual robot.

8. a teaching point generating device that generates teaching points for the robot; an operating device operated by a user; an image generation device that generates a virtual image in which a virtual robot corresponding to the robot is placed in a virtual space; a display that displays the virtual image; a controller sensor that detects the position and orientation of the controller in real space; The image generating device generating the virtual image in the virtual space that causes the virtual robot to perform an action linked to the action of the controller in the real space based on the detection result of the controller sensor; placing a virtual controller corresponding to the controller in the virtual space based on the detection result of the controller sensor; The teaching point generation device is a teaching system that generates the teaching point corresponding to the position in the virtual space of the virtual robot generated by the image generation device.

9. 9. The teaching system according to claim 8, The image generation device is a teaching system that positions the virtual manipulator in the virtual space at a position offset by a predetermined amount with respect to the virtual robot.

10. 2. The teaching system according to claim 1, The image generating device is a teaching system that places a virtual teaching point corresponding to the teaching point in the virtual space.

11. 11. The teaching system according to claim 10, The image generating device is a teaching system that places a predetermined number of the nearest virtual teaching points in the virtual space.

12. 2. The teaching system according to claim 1, The image generation device sets movement limits on the movement range of the virtual robot, and when the virtual robot reaches the movement limits, the teaching system notifies the user that the virtual robot has reached the movement limits.

13. 2. The teaching system according to claim 1, The image generation device has a playback mode for displaying the virtual image in which the virtual robot operates according to the teaching points generated by the teaching point generation device.

14. A teaching system according to any one of claims 1 to 13; a robot that operates in accordance with the teaching points generated by the teaching point generating device.

15. A method for teaching a robot, comprising: generating a virtual image in which a virtual robot corresponding to the robot in the real space is placed in a virtual space, the virtual image corresponding to the position and direction of the user's line of sight in the virtual space; displaying the virtual image; In response to a user's operation on a controller in real space for operating the virtual robot, the virtual robot is caused to perform an operation in the virtual image that is linked to the position and orientation of the controller in real space; generating a teaching point for the robot in real space corresponding to a position of the virtual robot in the virtual space.

16. A robot teaching program, a function of generating a virtual image in which a virtual robot corresponding to the robot in the real space is placed in a virtual space, the virtual image corresponding to the position and direction of the user's line of sight in the virtual space; and a function of causing the virtual robot to perform an action in the virtual image that is linked to the position and orientation of a controller in real space in response to a user's operation on the controller in real space for operating the virtual robot; and generating a teaching point for the robot in real space that corresponds to a position of the virtual robot in the virtual space.

Images