Device for teaching, control method of device for teaching, and program of device for teaching

Abstract

To provide a technique for using cross reality (XR) to teach a robot while verifying the effects of a light source.SOLUTION: A device 100 comprises: an XR screen generation unit for generating an XR screen 12 for blending a real space with a virtual space; an XR event unit for detecting events related to the XR screen; a placement unit for updating a position of a light source 146 provided at a tip of a robot 150 in the virtual space; and a teaching point storage unit for storing teaching points of the robot 150. The XR screen generating unit displays, on the XR screen 12, a work-piece and the light source 146 in the virtual space. The placement unit moves the light source 146, on the XR screen 12, on the basis of detection of a movement event for the light source 146. The teaching point storage unit stores the teaching points for the robot 150 as moving in association with the light source 146.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a teaching technique for an industrial robot, and more particularly to a teaching technique using XR technology. [Background technology]

[0002] Today, articulated industrial robots (hereinafter, simply referred to as "robots") are widely used in factory production lines (hereinafter, the term "production lines" and "inspection lines" will be used collectively to refer to lines). These robots can be used to assemble, transport, or inspect workpieces. Furthermore, these robots require a process known as "teaching." Traditionally, workers have taught robots by operating the actual robots on-site.

[0003] In recent years, a technology called XR (Cross Reality) has been developing. XR encompasses technologies related to virtual spaces and technologies that combine virtual and real spaces. XR is also beginning to be used in the industrial field.

[0004] Regarding technology related to robot teaching using XR, for example, Japanese Patent Application Laid-Open No. 2023-038781 (Patent Document 1) discloses a robot teaching system. This robot teaching system "has a display unit that displays a virtual robot in a virtual space, a recognition unit that recognizes instructions given by a teacher to the virtual robot, and a notification unit that notifies whether or not a real robot can be driven based on the instructions. The notification unit also notifies that the real robot can be driven based on the instructions, or that the real robot cannot be driven based on the instructions" (see [Abstract]). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-038781 Summary of the Invention [Problem to be solved by the invention]

[0006] When using a robot in an inspection process, the light source is an important factor. The appropriate robot teaching point may also change depending on the light source. According to the technology disclosed in Patent Document 1, the person in charge cannot perform the robot teaching work while checking the influence of the light source. Therefore, there is a need for a technology that allows the robot teaching work to be performed while checking the influence of the light source.

[0007] The present disclosure has been made in consideration of the above-described background, and an object of one aspect is to provide a technology for using XR to perform robot teaching work while checking the influence of a light source. [Means for solving the problem]

[0008] According to one embodiment, an apparatus for teaching a robot is provided. The apparatus includes an XR screen generation unit for generating an XR screen that blends real space and virtual space, an XR event unit for detecting events related to the XR screen, a placement unit for updating the position of a light source attached to the tip of the robot in the virtual space, and a teaching point storage unit for storing teaching points for the robot. The XR screen generation unit displays a workpiece and a light source in the virtual space on the XR screen. The placement unit moves the light source on the XR screen based on the detection of a light source movement event. The teaching point storage unit stores teaching points for the robot that move in conjunction with the light source.

[0009] According to the above disclosure, by using the device, a user can intuitively teach a robot while checking the positions of the workpiece and light source on the XR screen.

[0010] In one aspect, the XR screen generation unit reads ambient light data relating to ambient light in the real world, and reproduces the reflected light from the surface of the workpiece on the XR screen based on the light from the ambient light in the real world and the light from the light source.

[0011] According to the disclosure above, the workpiece on the XR screen can reproduce the reflected light of the workpiece in the real world. Therefore, by using the device, a user can teach a robot even when they are not at the planned construction site of the line, as if they were at the planned construction site.

[0012] In one aspect, the placement unit fixes the light source on the XR screen based on the detection of a light source release event, and storing teaching points of the robot includes storing teaching points of the robot corresponding to the position and orientation of the light source after fixation.

[0013] According to the above disclosure, the device can store teaching points for the robot that correspond to the positions of fixed light sources on the XR screen.

[0014] In one aspect, the device further includes a photographing unit that photographs the workpiece using a camera virtually positioned near the light source based on the detection of a photographing event.

[0015] According to the above disclosure, the device can capture an image of the workpiece at any teaching point. For example, a user can obtain an image of the workpiece captured by the inspection robot at a tentative teaching point.

[0016] In one aspect, the XR screen generation unit displays an image of the photographed workpiece on the XR screen.

[0017] According to the above disclosure, the user can easily check the captured image on the XR screen.

[0018] In one aspect, the device further includes a UI processing unit that displays an image of the photographed workpiece on a display of the device.

[0019] According to the above disclosure, the user can easily check the captured image on the display of the device.

[0020] In one aspect, the apparatus further includes an evaluation unit that performs image analysis on the captured image of the workpiece. Displaying the image of the workpiece includes displaying the image analysis results.

[0021] According to the disclosure above, the apparatus can analyze images captured at any teaching point. For example, a user can obtain the results of an appearance inspection of a workpiece by an inspection robot at a tentative teaching point.

[0022] In one aspect, the device further includes a light source adjustment unit for changing at least one of the on / off, model number, brightness, or light emission color of the light source based on the detection of a light source setting change event, and a light source setting memory unit for storing the changed light source settings.

[0023] According to the above disclosure, the device can change at least one of the on / off, model number, brightness, or light color of the light source, and further store the changed settings.

[0024] In one aspect, the device further includes an anomaly detection unit that detects an anomaly in the virtual space, and a warning unit that outputs a warning based on the detected anomaly. The anomaly includes a light source being outside a range of motion of the robot.

[0025] According to the above disclosure, the device may output a warning based on detecting an abnormality in the virtual space, which may include an abnormality related to the range of motion of the robot.

[0026] In one aspect, the abnormality includes at least one of the amount of light received by the workpiece when the light source is turned on being equal to or greater than a predetermined first threshold value or being less than a predetermined second threshold value.

[0027] According to the above disclosure, the device can output a warning when the amount of light received by the workpiece is equal to or less than a certain level. For example, by referring to the warning, the user can recognize that the light source is too close to the workpiece, or that the light from the light source is not shining on the workpiece.

[0028] In one aspect, the warning output by the warning unit includes at least one of displaying warning information on an XR screen, displaying warning information on an output device connected to the device, or vibrating an XR input device to move a light source.

[0029] According to the above disclosure, the device may make it easy for the user to recognize the warning.

[0030] In one aspect, the device further includes a setting output unit that outputs teaching points for the robot.

[0031] According to the above disclosure, the device may output teaching points for the robot. For example, the device may directly transmit the teaching points for the robot to the actual robot.

[0032] According to one embodiment, there is provided a robot teaching method executed by an apparatus, the method including: generating an XR screen that blends real space and virtual space; displaying, on the XR screen, a workpiece in the virtual space and a light source attached to the robot; moving the light source on the XR screen based on detection of a light source movement event; and storing teaching points for the robot that moves in conjunction with the light source.

[0033] According to the above disclosure, by using a device that executes the method, a user can intuitively teach a robot while checking the positions of a workpiece and a light source on an XR screen.

[0034] According to one embodiment, there is provided a program for causing an apparatus to perform robot teaching. The program causes the apparatus to generate an XR screen that blends real space and virtual space, display on the XR screen a workpiece in the virtual space and a light source attached to the robot, move the light source on the XR screen based on detection of a light source movement event, and store teaching points for the robot that moves in conjunction with the light source.

[0035] According to the above disclosure, by using a device that executes the program, a user can intuitively teach a robot while checking the positions of the workpiece and light source on the XR screen. [Effects of the Invention]

[0036] According to one embodiment, XR can be used to perform robot teaching tasks while checking the effects of a light source.

[0037] The above and other objects, features, aspects and advantages of the present disclosure will become apparent from the following detailed description of the disclosure taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0038] [Figure 1] 1A to 1C are diagrams showing an example of teaching a robot using the system 10 according to the present embodiment. [Figure 2] 1 is a diagram illustrating an example of the configuration of an apparatus 100. FIG. [Figure 3] FIG. 10 is a diagram showing an example of an XR screen 300 in an initial state. [Figure 4] FIG. 10 is a diagram showing an example of an XR screen 400 after a movement event occurs in the XR input device 130. [Figure 5] This is a diagram showing an example of the XR screen 500 after the occurrence of the hold event of the light source 146. [Figure 6] This is a diagram showing an example of the XR screen 600 after the occurrence of the movement event of the light source 146. [Figure 7] This is a diagram showing an example of the XR screen 700 after the occurrence of the release event of the light source 146. [Figure 8] This is a diagram showing an example of a function to reproduce the influence of the light source 146 on the workpiece 144 on the XR screen 12. [Figure 9] This is a diagram showing an example of the imaging function of the workpiece 144. [Figure 10] This is a diagram showing an example of the brightness adjustment function of the light source 146. [Figure 11] This is a diagram showing an example of the emission color adjustment function of the light source 146. [Figure 12] This is a diagram showing an example of the processing procedure of the system 10.

Mode for Carrying Out the Invention

[0039] Hereinafter, embodiments of the technical idea according to the present disclosure will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. Also, each embodiment, each modification example, each software configuration, each hardware configuration, each function, and each process, etc. may be selectively combined as appropriate.

[0040] <A. Application Example>

[0041] FIG. 1 is a diagram showing an example of teaching a robot using the system 10 according to the present embodiment. The system 10 is a system for teaching a robot while confirming the influence of a light source by using XR. The user can intuitively perform teaching of the robot by using the system 10.

[0042] (a. Terms)

[0043] First, some terms necessary for understanding the technology of the present disclosure will be explained. More specifically, the terms line, work, robot, teaching, XR, light source, and system will be explained.

[0044] In this specification, the term "line" refers to equipment composed of a conveyor belt, various sensors, robots and their control devices, etc. The term "line" includes a manufacturing line for manufacturing products or parts thereof. The term "line" also includes a conveyance line used in a logistics warehouse. The term "line" also includes an inspection line for products or parts thereof. The term "line" may also be used for any other purpose. Furthermore, the term "line" may be a combination of lines for various purposes.

[0045] In this specification, a "workpiece" refers to an object that is processed, assembled, transported, or inspected on a line. A workpiece may be a finished product. A workpiece may also be a part used to assemble a product. In this specification, a workpiece may refer to a workpiece that is a virtual object reproduced in a virtual space. In this specification, a workpiece may also refer to a workpiece that exists in real space.

[0046] In this specification, the term "robot" refers to an industrial robot used in conjunction with a production line. The robot includes a vertical articulated robot, a horizontal articulated robot, a parallel link robot, an orthogonal robot, and a robot that combines these mechanisms.

[0047] In this specification, "teaching" refers to the task of setting a robot or its control device how the robot should operate. Teaching can include setting the robot's destination position or path. Teaching is an extremely important task when using a robot and places a significant burden on the operator. As an example, consider a robot used for product assembly or inspection. In this case, the robot must move extremely accurately to the product assembly or inspection position. Therefore, the operator repeatedly fine-tunes the robot's destination position while checking the robot's operation on-site. Hereinafter, the robot's destination position may also be referred to as a "teaching point." A teaching point can include the coordinates and orientation of the robot's tip or the tool attached to the tip. The orientation can be expressed as a combination of angles around each coordinate axis (X, Y, Z).

[0048] In this specification, "XR" is an abbreviation for Cross Reality. XR is a general term for technologies that combine the real world and the virtual world, and encompasses virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0049] VR is a technology that allows users to experience a virtual world. Users use a VR headset to experience VR. For example, by using VR technology, users can experience being immersed in the world of a game.

[0050] AR is a technology that overlays virtual objects onto real-world scenery. Users experience AR using a smartphone, tablet, or AR glasses. For example, AR technology allows users to view content virtually placed in the city.

[0051] MR is a technology that generates mixed reality by overlaying the real world and the virtual world. When experiencing MR, users use a headset designed for MR. MR is a technology that further develops AR. AR simply displays virtual objects on AR glasses, overlaying the virtual objects on the real-world scenery. In contrast, MR uses an MR device to capture real-world images. Then, virtual objects are combined with the real-world images to generate mixed reality. As a result, for example, MR can generate images that take into account the context between real-world objects and virtual objects.

[0052] In this specification, the term "light source" refers to any object that emits light. As an example, lighting is a light source. Sources of natural light or ambient light are also light sources. In this specification, the term "light source" may also refer to a light source reproduced in a virtual world. In VR and AR, a light source may illuminate virtual objects. In MR, a light source may illuminate both real and virtual objects. In other words, in XR, which includes VR, AR, and MR, a light source may illuminate at least virtual objects.

[0053] As used herein, a "system" refers to a single device or a combination of multiple devices. A device may include a personal computer, a workstation, a server, a tablet, or a smartphone. A device may also include a system-on-a-chip (SoC) or a system-on-module (SoM). A device may also include any peripheral devices, such as a switch, a router, a display, a keyboard, and a mouse. A device may also include virtual machines and instances built in a cloud environment. In some aspects, a system may be connected to input / output devices, such as a display and a keyboard, and used as a standalone device. In other aspects, a system may provide various functions as a service or web application over a network. In this case, a user may use the system's functions via a browser or client software installed on their device.

[0054] (b. System Overview)

[0055] Next, the configuration of the system 10 will be described. The system 10 includes a device 100, an input device 110, an output device 120, an XR input device 130, and an XR output device 140. In some aspects, the system 10 may not include the input device 110, the output device 120, or both. In these cases, the XR input device 130 may also function as the input device 110. Furthermore, the XR output device 140 may also function as the output device 120.

[0056] The device 100 executes a teaching process based on input from a user. More specifically, the device 100 generates and outputs an XR image, stores teaching points, determines the type and position of a light source, etc. The device 100 can also recognize operations based on user gestures by analyzing the XR screen 12 captured by the XR output device 140.

[0057] The input device 110 inputs various settings to the device 100. The input device 110 may be configured to be able to input the same information to the device 100 as information input to the device 100 from the XR input device 130. The output device 120 displays a teaching setting screen. The output device 120 may display various settings, the XR screen 12, and the virtual space 15. The output device 120 may also display an image captured by a camera 152 within the virtual space 15.

[0058] The XR input device 130 is an interface for manipulating objects within the XR screen 12. In the system 10, the XR input device 130 can change the position of a light source 146 of the robot 150. The light source 146 corresponds to a light attached to the tip of the robot 150. Detailed operation will be described later. The XR input device 130 may include an acceleration sensor, a gyro sensor, and any other sensors. In one aspect, the XR input device 130 may calculate the position of the XR input device 130 based on signals from each sensor. In this case, the XR input device 130 transmits information regarding the position of the XR input device 130 to the device 100. In another aspect, the XR input device 130 may calculate the direction and amount of movement of the XR input device 130 based on signals from each sensor. In this case, the XR input device 130 transmits information regarding the direction and amount of movement of the XR input device 130 to the device 100. Furthermore, in another aspect, the XR input device 130 may transmit signals from each sensor to the device 100. In this case, the device 100 calculates the movement direction and amount of movement of the XR input device 130 based on the signals from each sensor. In either case, the device 100 may detect the current position, movement direction, movement amount, etc. of the XR input device 130 in real time.

[0059] The XR input device 130 may also accept various operational inputs from the user. In one aspect, the XR input device 130 may include any operation unit such as a button, a dial, a touch sensor, or a wheel. The XR input device 130 may accept various operational inputs from the user via these operation units. In another aspect, the XR input device 130 may read a user's gestures using an acceleration sensor, a gyro sensor, or the like. In this case, the XR input device 130 accepts the gestures as various operational inputs from the user. Alternatively, the XR output device 140 may capture images of the user's gestures using a built-in camera (not shown). In this case, the device 100 may receive video from the XR output device 140 and analyze the video to detect the gestures. In either case, information regarding the various operational inputs from the user is transmitted to the device 100.

[0060] The XR output device 140 outputs the XR screen 12. The XR output device 140 may be realized as a headset, a glasses-type device, or the like. The user can view the XR screen 12 by wearing the XR output device 140. The XR output device 140 may have a function for capturing the XR screen 12. The XR output device 140 transfers the captured XR screen 12 to the device 100. In one aspect, the XR output device 140 may record only images of virtual objects composited into the real world. In another aspect, the device 100 may have a capture function. In this case, the device 100 records images of virtual objects in the virtual space 15.

[0061] (c.XR screen)

[0062] Next, the XR screen 12 displayed on the XR output device 140 will be described. In the example of FIG. 1, a user uses the system 10 to teach a robot 150 reproduced in a virtual space 15 for visual inspection of a workpiece 144. A light source 146 and a camera 152 are attached to the tip of the robot 150. Visual inspection is one example, and the system 10 can also be used to teach any operation of the workpiece 144, such as processing, assembling, or picking up.

[0063] The XR screen 12 is a screen displayed on the XR output device 140. In one aspect, the XR screen 12 may be a screen displayed using AR technology. In this case, the XR screen 12 is a screen on which virtual objects are superimposed on the real world. In another aspect, the XR screen 12 may be a screen displayed using MR technology. In this case, the XR screen 12 is a mixed reality screen on which virtual objects are combined with an image of the real world.

[0064] In the example of FIG. 1, the XR screen 12 displays real-space objects and virtual objects. The input device 110, the output device 120, and the marker 142 are real-space objects. The real-space objects may include any object within the user's field of view in addition to the objects shown in FIG. 1. The workpiece 144, the light source 146, and the laser 148 are virtual objects. In some aspects, the XR screen 12 may include a display of the robot 150, the camera 152, or both. In this case, the robot 150, the camera 152, or both may be displayed semi-transparently, or only their outlines may be displayed.

[0065] The marker 142 is a reference for determining the display coordinates of a virtual object. The workpiece 144 and the light source 146 may be displayed at a predetermined location as viewed from the marker 142. In one aspect, the system 10 may recognize the marker 142 as the origin of three-dimensional space. The system 10 may then place the workpiece 144 and the light source 146 at a predetermined relative position (which may be expressed in XYZ coordinates) as viewed from the origin.

[0066] Workpiece 144 corresponds to workpiece 144 in virtual space 15. Light source 146 corresponds to lighting attached to the tip of robot 150 in virtual space 15. Light source 146 may be represented by lighting CAD (Computer Aided Design) data, or may be represented by any shape such as a cylinder.

[0067] The laser 148 indicates the direction in which the tip of the XR input device 130 is pointing. To the user, the laser 148 appears as a visible laser emitted from the tip of the XR input device 130. The laser 148 may be represented as a dotted line, a straight line, or any other shape. The laser 148 may also be represented as any color or brightness.

[0068] The system 10 reproduces the effects of various light sources on the workpiece 144 within the XR screen 12. The first effect is the effect of the light source 146, which is a light attached to the robot 150. When the light source 146 is placed near the workpiece 144, the user sees the workpiece 144 as if it were illuminated by the light source 146. The second effect is the effect of ambient light. The device 100 is configured to allow pre-registration of ambient light information, such as the planned construction site of the production line. As an example, the ambient light information may be an HDR (High Dynamic Range) image or video. By reading the ambient light information, the device 100 can reproduce the ambient light, such as the planned construction site of the production line, within the virtual space 15 and the XR screen 12. This makes the workpiece 144 appear to the user as if it were illuminated by ambient light. However, the device 100 does not necessarily need to place an ambient light source within the XR screen 12. The device 100 may reproduce only the reflection of light on the surface of the workpiece 144 due to ambient light within the XR screen 12.

[0069] (d. Teaching Procedure)

[0070] Next, a teaching procedure performed by a user using the system 10 will be described. First, the user points the laser 148 at the light source 146. Then, the user performs a first operation on the XR input device 130, thereby drawing the light source 146 to the tip of the XR input device 130 or to its vicinity. In one aspect, the first operation may be pressing any button provided on the XR input device 130. In another aspect, the first operation may be a gesture using the XR input device 130 or a part of the user's body.

[0071] Next, the user moves the XR input device 130 to move the light source 146, which has been attracted to the XR input device 130, to a desired position. The XR input device 130 is held by the user. That is, the user can intuitively move the light source 146 as if moving a flashlight held in their hand. When the user moves the light source 146, the light source 146 in the virtual space 15 also moves in conjunction with it. As described above, in the virtual space 15, the light source 146 is attached to the tip of the robot 150. Therefore, when the light source 146 moves, the robot 150 also moves in conjunction with it. That is, it can be said that the user moves the tip of the robot 150 together with the light source 146. Hereinafter, a state in which the light source 146 is held at or near the tip of the XR input device 130 will be referred to as a "hold state." Furthermore, putting the light source 146 in a hold state will be referred to as "holding the light source 146." More specifically, in the hold state, the device 100 acquires position information of the destination of the light source 146 from the XR input device 130 as needed. Alternatively, in the hold state, the device 100 may calculate position information of the destination of the light source 146 as needed, based on the positional relationship between the XR input device 130 and the light source 146. The device 100 moves the light source 146 in the virtual space 15 based on the position information of the destination of the light source 146. Furthermore, the device 100 moves the robot 150 based on the movement of the light source 146 in the virtual space 15. Moving the robot 150 includes changing the angle of each joint of the robot 150.

[0072] Next, the user can fix the light source 146 within the XR screen 12 by performing a second operation. In a certain aspect, the second operation may be pressing any button provided on the XR input device 130. In other aspects, the second operation may be a gesture using the XR input device 130 or a part of the user's body. Hereinafter, the state in which the light source 146 is fixed within the XR screen 12 is referred to as the "release state". Also, making the light source 146 enter the release state is referred to as "releasing the light source 146".

[0073] By using the function of holding and releasing the light source 146, the user can intuitively determine the position of the light source 146 with respect to the workpiece 144. Once the position of the light source 146 is determined, the teaching point of the robot corresponding to the position of the light source 146 is also automatically determined. That is, the user can intuitively determine the teaching point of the robot equipped with the light source 146.

[0074] In the XR screen 12, the pre-registered ambient light is reproduced. That is, in the XR screen 12, the ambient light and the reflected light of the surface of the workpiece 144 by the light source 146 are reproduced. As an example, the pre-registered ambient light may be the ambient light at the planned construction site of the line. In this case, the user can observe the workpiece 144 through the XR screen 12 as if they were at the planned construction site of the line. Details of the various functions provided by the system 10 will be described later with reference to FIGS. 3 to 12.

[0075] <B. Device Configuration>

[0076] FIG. 2 is a diagram showing an example of the configuration of the device 100. The device 100 includes a processor 201, a RAM (Random Access Memory) 202, a storage unit 203, an input unit 204, an output unit 205, an external device connection unit 206, and a communication unit 207. Also, the storage unit 203 stores a program 210.

[0077] The processor 201 may execute programs for implementing various functions of the device 100. The processor 201 may be configured, for example, with at least one integrated circuit. According to an embodiment, the integrated circuit may include at least one central processing unit (CPU), at least one graphics processing unit (GPU), at least one field programmable gate array (FPGA), at least one application specific integrated circuit (ASIC), at least one artificial intelligence (AI) chip, or a combination thereof.

[0078] The RAM 202 functions as a workspace for the processor 201. The RAM 202 stores programs executed by the processor 201 and data referenced by the processor 201. In one aspect, the RAM 202 can be realized by a dynamic random access memory (DRAM), a static random access memory (SRAM), or the like.

[0079] Storage unit 203 is a non-volatile memory that stores programs executed by processor 201 and data referenced by processor 201. Processor 201 executes programs read from storage unit 203 to RAM 202 and references data read from storage unit 203 to RAM 202. In one aspect, storage unit 203 can be realized by a hard disk drive (HDD), a solid state drive (SSD), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash memory, or the like.

[0080] The input unit 204 may be connected to any input device 110, such as a keyboard, a mouse, a touchpad, or a gamepad. In one aspect, the input unit 204 may be realized by a USB terminal, a PS / 2 terminal, a Bluetooth (registered trademark) module, or the like.

[0081] Output unit 205 can be connected to any output device 120, such as a cathode ray tube display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. In one aspect, output unit 205 can be realized by a USB terminal, a D-sub terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, a DisplayPort terminal, or the like.

[0082] The external device connection unit 206 can be connected to any external device, such as the XR input device 130, the XR output device 140, a printer, a scanner, or an external HDD. In one aspect, the external device connection unit 206 can be implemented by a USB (Universal Serial Bus) terminal or the like. In another aspect, the XR input device 130 may be connected to the input unit 204. Furthermore, the XR output device 140 may be connected to the output unit 205.

[0083] Communication unit 207 is connected to other devices via a wired network or a wireless network. In one aspect, communication unit 207 may be implemented by a wired local area network (LAN) port, a Wi-Fi (registered trademark) (Wireless Fidelity) module, or the like. In another aspect, communication unit 207 may transmit and receive data using a communication protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol) or UDP (User Datagram Protocol).

[0084] The program 210 includes various functional units for realizing the robot's teaching function. The program 210 may also reference various data for realizing the robot's teaching function. More specifically, the program 210 includes, as functional units, a virtual space generation unit 212, an XR screen generation unit 214, an XR event unit 216, a placement unit 218, a capture unit 220, a light source adjustment unit 222, a teaching point storage unit 224, a light source setting storage unit 226, an abnormality detection unit 228, a warning unit 230, an evaluation unit 232, and a UI (User Interface) processing unit 234. In some aspects, some of these functional units may be realized as hardware. The program 210 also references object data 236 and ambient light data 238.

[0085] The virtual space generation unit 212 generates the virtual space 15. The virtual space generation unit 212 places at least the robot 150 to be taught, the light source 146, and the workpiece 144 within the virtual space 15. In some aspects, the virtual space generation unit 212 may place various tools, such as a camera 152 attached to the tip of the robot 150, in addition to the light source 146, within the virtual space 15. In other aspects, the virtual space generation unit 212 may place some or all of the lines within the virtual space 15. The virtual space generation unit 212 may also read ambient light data 238 and reproduce the ambient light of the real world within the virtual space 15. Furthermore, the virtual space generation unit 212 also updates the virtual space 15 based on an update of the XR screen 12. This may link the positions of each object, the light source settings, and the like between the virtual space 15 and the XR screen 12. The virtual space generation unit 212 and the XR screen generation unit 214 may be configured to constantly exchange the latest statuses with each other.

[0086] The XR screen generation unit 214 generates the XR screen 12. In one aspect, the XR screen generation unit 214 may overlay a virtual object on the real world using AR technology. In another aspect, the XR screen generation unit 214 may generate a mixed reality screen by combining the real world and the virtual world using MR technology. The XR screen generation unit 214 may determine the display position of each object using the markers 142.

[0087] The XR event unit 216 detects various events related to the XR screen 12. The XR event unit 216 notifies other functional units of the detected events as necessary. The various events may include various user operation inputs. The events include a hold event corresponding to holding the light source 146 and a release event corresponding to releasing the light source 146. The events also include a movement event. The movement event relates to movement of the XR input device 130 in a released state and movement of the XR input device 130 in a held state. The events also include a photographing event corresponding to photographing the workpiece 144 by the camera 152. The events also include a lighting switching event corresponding to switching the light source 146 on and off. The events also include a light source setting change event corresponding to a change in at least one of the model number, brightness, or light emission color of the light source 146.

[0088] The placement unit 218 updates the placement of objects on the XR screen 12 based on the events acquired from the XR event unit 216. Typically, the placement unit 218 updates the placement of the light source 146 by detecting a hold event and a movement event. In some aspects, the XR screen generation unit 214 may include the functionality of the placement unit 218. In this case, the XR event unit 216 notifies the XR screen generation unit 214 of the detected event.

[0089] The photographing unit 220 photographs the virtual space 15 using the camera 152 based on the photographing event acquired from the XR event unit 216. As an example, the camera 152 photographs the workpiece 144 in the virtual space 15. The ambient light and light source 146 of the real space are reproduced in the virtual space 15. Therefore, the photographing unit 220 can photograph an image similar to the image of the workpiece 144 photographed in the real world. The photographing unit 220 outputs the image obtained by photographing to the evaluation unit 232.

[0090] The light source adjustment unit 222 switches the light source 146 between on and off based on the lighting switching event acquired from the XR event unit 216. Furthermore, the light source adjustment unit 222 changes at least one of the model number, brightness, and emitted color of the light source 146 based on the light source setting change event acquired from the XR event unit 216.

[0091] The teaching point storage unit 224 updates and stores the teaching points based on a release event acquired from the XR event unit 216. In some aspects, the teaching point storage unit 224 may update and store the teaching points based on the occurrence of a teaching point storage event. In this case, the XR input device 130 is configured to be able to accept an operation input for storing the teaching points.

[0092] The light source setting storage unit 226 updates and stores the settings of the light source 146. In one aspect, the light source setting storage unit 226 may store the current settings of the light source 146 in the virtual space 15 based on a lighting switch event and / or a light source setting change event. In another aspect, the light source setting storage unit 226 may store the current settings of the light source 146 in the virtual space 15 based on the occurrence of a storage event for the settings of the light source 146. In this case, the XR input device 130 is configured to be able to accept an operation input for changing the settings of the light source 146.

[0093] The anomaly detection unit 228 detects one or more anomalies in the virtual space 15. Furthermore, information about the detected anomaly is output to the warning unit 230. The one or more anomalies include an anomaly related to the teaching point and an anomaly related to the light source. As an example, suppose that the user moves the XR input device 130 significantly, causing the light source 146 to move outside the movable range of the robot 150. In this case, the anomaly detection unit 228 detects an anomaly related to the teaching point. As another example, suppose that the light irradiated on the workpiece 144 is brighter than a first threshold value or darker than a second threshold value. In this case, the anomaly detection unit 228 detects an anomaly related to the light source. That is, the anomaly detection unit 228 may detect an anomaly such as insufficient light irradiating the workpiece 144 or too much light irradiating the workpiece 144.

[0094] The warning unit 230 displays information about the one or more detected abnormalities on the XR screen 12, the output device 120, or both. By referring to the displayed information about the one or more abnormalities, the user can recognize that there is a problem with their own operation input.

[0095] The evaluation unit 232 analyzes the image acquired from the imaging unit 220 and displays the analysis results on the XR screen 12, the output device 120, or both. As an example, by referring to the analysis results, the user can recognize whether or not the appearance inspection of the workpiece 144 has been completed. By using the functions of the evaluation unit 232, the user can easily check the results of the appearance inspection of the workpiece 144 for each teaching point.

[0096] The UI processing unit 234 displays a UI related to the system 10 on the output device 120. The UI processing unit 234 may also receive any operation input from the user via the UI displayed on the output device 120. As an example, the UI processing unit 234 may display at least one of the XR screen 12, a screen of the virtual space 15, a warning screen, and a screen showing the results of image analysis on the output device 120. All screens displayed on the output device 120 may also be displayed via the UI processing unit 234.

[0097] The object data 236 is data of an object to be placed in the virtual space 15. As an example, the object data 236 may be CAD data. A virtual object displayed on the XR screen 12 may also be generated from the object data 236. The device 100 may register necessary object data 236 in advance via the UI processing unit 234 or the like.

[0098] The ambient light data 238 is data for reproducing ambient light. As an example, the ambient light data 238 may be an HDR image or video. The required ambient light data 238 may be registered in advance in the device 100 via the UI processing unit 234 or the like. Furthermore, the device 100 may be configured to be able to switch between multiple sets of ambient light data 238. This allows the user to reproduce the environments of multiple factories in the virtual space 15 and easily teach robots installed in each factory.

[0099] As described with reference to FIG. 2 , the device 100 includes an XR screen generation unit 214 for generating an XR screen 12 that blends the real space and the virtual space 15. The device 100 also includes an XR event unit 216 for detecting an event related to the XR screen 12. The device 100 also includes a placement unit 218 for updating the position of a light source attached to the tip of the robot 150 in the virtual space 15. The device 100 also includes a teaching point storage unit 224 for storing teaching points for the robot 150. The XR screen generation unit 214 displays the workpiece 144 and the light source 146 in the virtual space 15 on the XR screen 12. The placement unit 218 moves the light source 146 on the XR screen 12 based on the detection of a movement event of the light source 146. The teaching point storage unit 224 stores teaching points for the robot 150 that move in conjunction with the light source 146.

[0100] In a certain situation, the XR screen generation unit 214 reads environmental light data 238 regarding the environmental light in the real world. Then, based on the light from the environmental light in the real world and the light from the light source 146, the XR screen generation unit 214 reproduces the reflected light on the surface of the workpiece 144 on the XR screen 12.

[0101] In a certain situation, the apparatus 100 may further include a setting output unit (not shown) that outputs the teaching points of the robot 150. The setting output unit may also output the teaching points and the light source settings. The setting output unit can output various settings to any device such as the actual robot 150 or its control device.

[0102] <C. Teaching Procedure>

[0103] Referring to FIGS. 3 to 11, the teaching procedure of the robot 150 using the system 10 will be described. In addition, the functions regarding the light source 146 and the test function of the appearance inspection will also be described. The screens shown in FIGS. 3 to 11 are an example of the display of the XR screen 12.

[0104] (a. Teaching Point Storage Procedure)

[0105] FIGS. 3 to 7 show a typical example of a series of procedures until the teaching points of the robot 150 are stored. Referring to FIGS. 3 to 7, the procedure from the display of the initial screen until the user stores the teaching points of the robot will be described. The various functions shown in FIGS. 8 to 11 can be performed even during the operations shown in FIGS. 3 to 7. Each XR screen shown in each of FIGS. 3 to 11 is an example of the display of the XR screen 12 in various scenes.

[0106] FIG. 3 is a diagram illustrating an example of an XR screen 300 in an initial state. The XR screen 300 is a screen before a user performs a teaching task. The device 100 places the workpiece 144 and the light source 146 on the XR screen 300 based on the position of the marker 142. The device 100 also displays the laser 148 on the XR screen 300 based on CAD data stored in the storage unit 203. The device 100 also determines the brightness and emitted color of the light source 146 based on the light source data stored in the storage unit 203. The device 100 also reproduces ambient light on the XR screen 300 based on ambient light data 238 stored in the storage unit 203. As an example, the device 100 reproduces the light reflected from the surface of the workpiece 144 due to the ambient light.

[0107] In one aspect, the system 10 may display the camera 152 and / or the robot 150 on the XR screen 300. In this case, the system 10 may display a semi-transparent version of the camera 152 and / or the robot 150 on the XR screen 300. Alternatively, the system 10 may display only the outline of the camera 152 and / or the robot 150 on the XR screen 300. The same applies to the XR screens of FIGS. 4 to 11.

[0108] 4 is a diagram showing an example of an XR screen 400 after a movement event occurs for the XR input device 130. The XR screen 400 is a screen of a scene after the XR screen 300. When the user moves the XR input device 130, the laser 148 also moves in conjunction with the XR input device 130. In the example of FIG. 4, the laser 148 is directed toward the light source 146.

[0109] The operation of FIG. 4 will be described in more detail. When the user moves the XR input device 130, information regarding the direction and amount of movement of the XR input device 130 is transmitted from the XR input device 130 to the apparatus 100. In connection with this, the XR event unit 216 detects the occurrence of a movement event of the XR input device 130. The XR event unit 216 outputs the detected event to the placement unit 218. The placement unit 218 moves the position of the laser 148 in conjunction with the XR input device 130. At this time, the placement unit 218 may refer to information regarding the direction and amount of movement of the XR input device 130. The XR screen generation unit 214 updates the XR screen 300 to the XR screen 400 based on the movement of the object.

[0110] The same processing as the above is also executed for the operations in FIGS. 5 to 7 . That is, the XR event unit 216 detects the occurrence of some event and outputs the detected event to the arrangement unit 218. Then, the arrangement unit 218 moves the object based on the notification of the event. At this time, the arrangement unit 218 may refer to various data in the storage unit 203 as necessary. Furthermore, the XR screen generation unit 214 updates the XR screen 12 based on the movement of the object. In one aspect, the XR screen generation unit 214 may update the XR screen 12 based on the notification of the event from the XR event unit 216. In another aspect, the XR screen generation unit 214 may update the XR screen 12 based on the notification of the movement of the object from the arrangement unit 218.

[0111] FIG. 5 is a diagram showing an example of an XR screen 500 after a hold event occurs for the light source 146. The XR screen 500 is a screen of a scene after the XR screen 400. Suppose that the user inputs a hold operation into the XR input device 130 while the laser 148 is shining on the light source 146. In this case, the light source 146 is attracted to the tip of the XR input device 130 or its vicinity. In other words, the light source 146 is held by the XR input device 130.

[0112] 6 is a diagram showing an example of an XR screen 600 after a movement event occurs for the light source 146. Assume that the user moves the XR input device 130 while the light source 146 is being held. In this case, the light source 146 also moves while being held by the XR input device 130. The movement here includes translation or rotation of the light source 146, or a combination thereof.

[0113] FIG. 7 is a diagram illustrating an example of an XR screen 700 after a release event of the light source 146 occurs. Assume that, while the light source 146 is held, the user inputs a release operation to the XR input device 130. In this case, the light source 146 is fixed at the position where the release operation was input. When the light source 146 is released, a new teaching point is stored in the teaching point storage unit 224. The new teaching point is the position where the light source 146 is fixed. The position of the light source 146 includes the coordinates and orientation of the light source 146 in the virtual space 15. In this way, the placement unit 218 fixes the light source on the XR screen 12 based on the detection of a light source release event. Storing the teaching point of the robot 150 includes storing a teaching point of the robot 150 corresponding to the position and orientation of the light source 146 after fixation.

[0114] 3 to 7, the user can intuitively determine the relative position of the light source 146 with respect to the workpiece 144. In other words, the user can intuitively determine the relative position of the tool or end effector at the tip of the robot 150 with respect to the workpiece 144.

[0115] (b. Function to reproduce the effects of light sources)

[0116] FIG. 8 is a diagram showing an example of a function that reproduces the effect of a light source 146 on a workpiece 144 on an XR screen 12. In an XR screen 800A, the light source 146 is located beside the workpiece 144. In this case, light from the light source 146 illuminates the side surface of the workpiece 144. As a result, the side surface of the workpiece 144 appears to be strongly illuminated to the user. In an XR screen 800B, the light source 146 is located above the workpiece 144. In this case, light from the light source 146 illuminates the top surface of the workpiece 144. As a result, the top surface of the workpiece 144 appears to be strongly illuminated to the user.

[0117] As described above, the device 100 reproduces the reflected light on the surface of the workpiece 144 in accordance with the position of the light source 146. More specifically, the device 100 reproduces the reflected light on the surface of the workpiece 144 in the virtual space 15 based on the light source 146 and the ambient light. Then, the device 100 displays the workpiece 144 with the reflected light reproduced on the XR screen 12.

[0118] The user can switch the lighting of the light source 146 on and off by inputting an operation to switch the lighting of the light source 146 to the XR input device 130. The user can switch the lighting of the light source 146 on and off by using a gesture or by pressing a button on the XR input device 130. When an operation to switch the lighting of the light source 146 is input to the XR input device 130, the XR event unit 216 detects an event to switch the lighting of the light source 146.

[0119] As described above, the user can determine the placement of the light source 146 while checking the brightness of the surface of the workpiece 144 via the XR screen 12. In other words, the user can determine the teaching points for the robot 150 while checking the brightness of the surface of the workpiece 144 via the XR screen 12.

[0120] (c. Visual inspection test function)

[0121] FIG. 9 is a diagram showing an example of a photographing function for a workpiece 144. On an XR screen 900, a user photographs a workpiece using a camera 152 attached to the tip of a robot 150. The position of the camera 152 is substantially the same as the light source 146. Therefore, the user can photograph the workpiece 144 while intuitively adjusting the position of the camera 152 by moving the XR input device 130. The device 100 can photograph the workpiece 144 based on receiving an input operation for photographing. The device 100 can receive an input operation for photographing via the XR input device 130 or the input device 110.

[0122] The camera 152 does not capture a portion of the XR screen 900, but captures only the objects in the virtual space 15. That is, the captured image 905 shows only the objects present in the virtual space 15. Using FIG. 9 as an example, the marker 142 and the workpiece 144 are present in the image 905. Assume that the user performs a capture with the camera 152 capturing the marker 142 and the workpiece 144 within the capture range. In this case, only the workpiece 144 appears in the image 905 obtained by capturing. Furthermore, the reflected light from the object is reproduced within the virtual space 15. Therefore, the image 905 also includes the reflected light from the surface of the object, the workpiece 144.

[0123] The apparatus 100 can capture an image of the workpiece 144 regardless of the state of the light source 146. That is, the apparatus 100 can capture an image of the workpiece 144 while the light source 146 is held. The apparatus 100 can also capture an image of the workpiece 144 while the light source 146 is released.

[0124] Image 905 is displayed on XR screen 900, output device 120, or both. In one aspect, displayed image 905 may include image analysis result 910. As an example, assume that a user is teaching robot 150 for visual inspection of workpiece 144. In this case, when the user photographs workpiece 144, device 100 executes an image recognition program for the actual visual inspection on the photographed image. Then, device 100 outputs image 905 including image analysis result 910. In one aspect, device 100 may be configured to be able to accept settings for the image recognition program to be used via UI processing unit 234, etc.

[0125] The user can photograph an object in the virtual space 15 with the camera 152 by inputting a photographing operation into the XR input device 130. The user can photograph the workpiece 144 by using a gesture or pressing a button on the XR input device 130. When a photographing operation is input into the XR input device 130, the XR event unit 216 detects a photographing event.

[0126] As described above, the user can easily check whether the appearance inspection is performed appropriately at the current position of the camera 152 while intuitively adjusting the position of the camera 152 via the XR screen 12. This allows the user to find a position where the appearance inspection is performed appropriately and register the position in the device 100 as a teaching point.

[0127] 9, the device 100 includes the photographing unit 220. The photographing unit 220 photographs the workpiece 144 using a camera 152 virtually placed near the light source 146 based on the detection of a photographing event.

[0128] In one aspect, the XR screen generation unit 214 may display an image 905 of the photographed workpiece 144 on the XR screen 12. In another aspect, the UI processing unit 234 may display the image 905 of the photographed workpiece 144 on the output device 120, which is the display of the apparatus 100. The image 905 may also be displayed on both the XR screen 12 and the output device 120.

[0129] The apparatus 100 also includes an evaluation unit 232. The evaluation unit 232 performs image analysis on the photographed image 905 of the workpiece 144. Displaying the image 905 of the workpiece includes displaying the image analysis result 910.

[0130] (d. Lighting setting change function)

[0131] FIG. 10 is a diagram illustrating an example of a function for adjusting the brightness of the light source 146. The user can adjust the brightness of the light source 146 while referring to the XR screen 12. XR screens 1000A, 1000B, and 1000C show how the brightness of the light source 146 changes. Referring to the XR screens 1000A, 1000B, and 1000C, it can be seen that the brightness of the light source 146 gradually increases. The device 100 can adjust the brightness of the light source 146 gradually higher and gradually lower.

[0132] More specifically, the device 100 adjusts the brightness of the light source 146 in the virtual space 15. Then, the result of adjusting the brightness of the light source 146 is reflected on the XR screen 12. The device 100 can adjust the brightness of the light source 146 regardless of the state of the light source 146. That is, the device 100 can adjust the brightness of the light source 146 when the light source 146 is held. Furthermore, the device 100 can adjust the brightness of the light source 146 when the light source 146 is released.

[0133] The user can adjust the brightness of the light source 146 by inputting an operation to adjust the brightness of the light source 146 to the XR input device 130. The user can adjust the brightness of the light source 146 by using a gesture or by pressing a button on the XR input device 130. When an operation to adjust the brightness of the light source 146 is input to the XR input device 130, the XR event unit 216 detects an adjustment event of the brightness of the light source 146.

[0134] As described above, the user can easily adjust the brightness of the light source 146 via the XR screen 12. That is, the user can easily adjust the brightness of the lighting attached to the tip of the robot 150 via the XR screen 12. As an example, the user can combine the photography function and the function for adjusting the brightness of the light source 146 to find the brightness of the light source 146 that is suitable for visual inspection of the workpiece 144.

[0135] FIG. 11 is a diagram illustrating an example of a function for adjusting the emission color of light source 146. The user can adjust the emission color of light source 146 while referring to XR screen 12. XR screens 1100A and 1100B illustrate changes in the emission color of light source 146. In one aspect, device 100 may be configured to individually adjust the brightness of each RGB color of light source 146. In another aspect, device 100 may be configured to enable selection of the emission color of light source 146 from among multiple candidates. In another aspect, device 100 may be configured to provide a function for selecting a color from a gradation, color palette, or the like displayed on XR screen 12. Furthermore, device 100 may be configured to enable switching of the operation procedure or UI for adjusting the emission color based on the model number of the lighting device selected as light source 146.

[0136] Similar to adjusting the brightness of the light source 146, the device 100 adjusts the emission color of the light source 146 in the virtual space 15. The result of adjusting the emission color of the light source 146 is then reflected on the XR screen 12. The device 100 can also adjust the emission color of the light source 146 regardless of the hold / release state of the light source 146. The user can adjust the emission color of the light source 146 by inputting an operation to adjust the emission color of the light source 146 to the XR input device 130. When an operation to adjust the emission color of the light source 146 is input to the XR input device 130, the XR event unit 216 detects an adjustment event of the emission color of the light source 146.

[0137] As described above, the user can easily adjust the emission color of the light source 146 via the XR screen 12. That is, the user can easily adjust the emission color of the lighting attached to the tip of the robot 150 via the XR screen 12. As an example, by combining the photography function and the function of adjusting the emission color of the light source 146, the user can find the emission color of the light source 146 that is suitable for visual inspection of the workpiece 144.

[0138] In one aspect, the device 100 may be configured to be able to change the model number of the lighting corresponding to the light source 146. In this case, the device 100 changes the model number of the lighting corresponding to the light source 146 in the virtual space 15 and reflects the result on the XR screen 12. Furthermore, the XR event unit 216 detects a model number change event based on the XR input device 130 receiving an operation input to change the model number of the lighting.

[0139] 10 and 11 , the device 100 includes a light source adjustment unit 222. The light source adjustment unit 222 changes at least one of the on / off, model number, brightness, or emitted color of the light source 146 based on the detection of a setting change event of the light source 146. The setting change event of the light source 146 includes an event of switching the light source 146 on and off, an event of adjusting the brightness of the light source 146, an event of adjusting the emitted color of the light source 146, and an event of changing the model number of the lighting corresponding to the light source 146. The light source setting storage unit 226 stores the changed setting of the light source 146.

[0140] <D. Processing Procedure of the Device>

[0141] FIG. 12 is a diagram showing an example of the processing procedure of system 10. In a certain aspect, the processor 201 may execute program 210 to perform the processing of FIG. 12. At that time, program 210 is read from the storage unit 203 into the RAM 202. In other aspects, part or all of the processing may also be realized as a combination of circuit elements configured to execute the processing. Also, each step shown in FIG. 12 may be executed in a swapped order.

[0142] In step S1205, system 10 reads object data 236 and ambient light data 238 from the storage unit 203. In step S1210, system 10 generates virtual space 15. More specifically, system 10 arranges various objects and various light sources in virtual space 15. In step S1215, system 10 generates XR screen 12. More specifically, system 10 generates XR screen 12 by superimposing or synthesizing the real space and virtual space 15. In step S1220, system 10 displays XR screen 12. System 10 displays XR screen 12 at least on XR output device 140. Also, system 10 may further display XR screen 12 on output device 120.

[0143] In step S1225, system 10 determines whether it has detected the movement of XR input device 130. That is, system 10 determines whether it has detected a movement event of XR input device 130. If system 10 determines that it has detected the movement of XR input device 130 (YES in step S1225), the control proceeds to step S1230. Otherwise (NO in step S1225), system 10 moves the control to step S1235.

[0144] In step S1230, the system 10 updates the position information of the XR input device 130. This position information may be a position in a three-dimensional coordinate space with the marker 142 as the origin. The system 10 also updates the position information of the laser 148. Furthermore, if the XR input device 130 is holding the light source 146, the system 10 also updates the position information of the light source 146. Furthermore, the system 10 updates the XR screen 12. When the XR screen 12 is updated, the laser 148 and the light source 146 each move on the XR screen 12 if their position information has been updated.

[0145] In step S1235, the system 10 determines whether or not a hold operation of the light source 146 has been detected. That is, the system 10 determines whether or not a hold event of the light source 146 has been detected. If the system 10 determines that a hold operation of the light source 146 has been detected (YES in step S1235), the system 10 transfers control to step S1240. Otherwise (NO in step S1235), the system 10 transfers control to step S1245. In step S1240, the system 10 causes the XR input device 130 to hold the light source 146.

[0146] In step S1245, the system 10 determines whether or not a photographing operation has been detected. That is, the system 10 determines whether or not a photographing operation event has been detected. If the system 10 determines that a photographing operation has been detected (YES in step S1245), it transfers control to step S1250. If not (NO in step S1245), the system 10 transfers control to step S1255. In step S1250, the system 10 photographs the workpiece 144 and evaluates the obtained image. More specifically, the system 10 analyzes the image using an appearance inspection program or the like. The system 10 displays the image and the image analysis results on the XR screen 12, the output device 120, or both.

[0147] In step S1255, the system 10 determines whether or not a release operation of the light source 146 has been detected. That is, the system 10 determines whether or not a release event of the light source 146 has been detected. If the system 10 determines that a release operation of the light source 146 has been detected (YES in step S1255), the system 10 transfers control to step S1260. Otherwise (NO in step S1255), the system 10 transfers control to step S1270. In step S1260, the system 10 fixes the position of the light source 146 in the virtual space 15. As a result, the position of the light source 146 is also fixed on the XR screen 12. In step S1265, the system 10 stores, in the storage unit 203, a teaching point of the robot 150 corresponding to the fixed position of the light source 146.

[0148] In step S1270, the system 10 determines whether an adjustment operation of the light source 146 has been detected. That is, the system 10 determines whether an adjustment event of the light source 146 has been detected. The adjustment operation of the light source 146 here is any one of an on / off operation, an adjustment operation of brightness, an adjustment operation of emitted color, and an operation to change the model number. If the system 10 determines that an adjustment operation of the light source 146 has been detected (YES in step S1270), the system 10 transfers control to step S1275. Otherwise (NO in step S1270), the system 10 transfers control to step S1285. In step S1275, the system 10 changes the setting of the light source 146. In step S1280, the system 10 stores the setting of the light source 146 in the storage unit 203.

[0149] In step S1285, the system 10 determines whether an abnormality has been detected. The abnormality here may include at least one of an abnormality related to the teaching point and an abnormality related to the light source. An abnormality related to the teaching point indicates that the light source 146 has moved outside the movable range of the robot 150. An abnormality related to the light source indicates that insufficient light is shining on the workpiece 144, or that the light irradiating the workpiece 144 is too intense. If the system 10 determines that an abnormality has been detected (YES in step S1285), the system 10 transfers control to step S1290. If not (NO in step S1285), the system 10 transfers control to step S1225.

[0150] In step S1290, the system 10 outputs a warning. In one aspect, the system 10 may display the warning on the XR output device 140, the output device 120, or both. In another aspect, the system 10 may vibrate the XR input device 130. Furthermore, in another aspect, the system 10 may output a warning in both of the above aspects.

[0151] Upon receiving an operation input for ending the teaching work, the system 10 may end the processing shown in Fig. 12. The system 10 may be configured to be able to receive an operation input for ending the teaching work from the input device 110, the XR input device 130, or both.

[0152] 12, the device 100 can perform processing related to teaching by executing the program 210. The processing includes generating an XR screen 12 that combines the real space and the virtual space 15, displaying on the XR screen 12 a workpiece in the virtual space 15 and a light source attached to the robot 150, moving the light source on the XR screen 12 based on the detection of a light source movement event, and storing teaching points for the robot 150 that moves in conjunction with the light source 146.

[0153] The device 100 also includes an abnormality detection unit 228 that detects abnormalities within the virtual space 15. The device 100 also includes a warning unit 230 that outputs a warning based on the detection of an abnormality. The abnormality may include the light source 146 being outside the movable range of the robot 150. The abnormality may also include at least one of the amount of light received by the workpiece 144 when the light source 146 is turned on being equal to or greater than a predetermined first threshold or being less than a predetermined second threshold. In other words, the abnormality detection unit 228 may detect abnormalities such as insufficient light shining on the workpiece 144 or the light irradiated on the workpiece 144 being too strong.

[0154] Furthermore, the warning output by the warning unit 230 includes at least one of displaying warning information on the XR screen 12, displaying warning information on the output device 120 connected to the device 100, or vibrating the XR input device 130 to move the light source 146.

[0155] In some aspects, the device 100 may be provided as a web application or a cloud-based service or system. In this case, the input device 110, the output device 120, the XR input device 130, and the XR output device 140 are connected to a user's terminal. The device 100 is configured to be able to communicate with the user's terminal via a communication unit 207. The device 100 receives various information output by the input device 110, the XR input device 130, and the XR output device 140 from the user's terminal. The various information received by the device 100 includes information related to the user's operation input, information related to the position or movement of the XR input device 130, and captured video. The device 100 may update the virtual space 15 and the XR screen 12 based on this information. The device 100 may also store teaching points and settings of the light source 146 based on this information. The device 100 may also transmit the generated XR screen 12 to the user's terminal.

[0156] <E.まとめ>

[0157] As described above, the system 10 according to the present embodiment has a function for intuitively teaching the robot 150. More specifically, the system 10 generates an XR screen 12 in which the real space and the virtual space 15 are superimposed or synthesized. The XR screen 12 displays the workpiece 144 and the light source 146 corresponding to the illumination attached to the tip of the robot 150. Further, in the XR screen 12, the influence of the ambient light on the workpiece 144 is reproduced. That is, by using the system 10, the user can teach the robot 150 under conditions similar to the real world in terms of brightness. Thereby, as an example, the user can teach the robot 150 using the system 10 without going to the planned construction site of the line and adjusting the actual robot 150.

[0158] In addition, the system 10 has a function of photographing the workpiece 144 and an image analysis function of the image obtained by the photographing. By using these functions, the user can confirm whether the appearance inspection of the workpiece 144 can be appropriately performed at the current teaching point.

[0159] Further, the system 10 can change one or more of the on / off, model number, brightness, and emission color of the light source 146. By using this function, the user can perform teaching under various lighting conditions.

[0160] Furthermore, by applying XR technology to the teaching of the robot 150, the system 10 enables intuitive teaching work even in a place where there is no actual robot 150. By using the system 10, the user can teach the robot 150 under conditions where the real ambient light is reproduced regardless of the location. Also, the intuitive user interface of the system 10 can reduce the teaching man-hours.

[0161] <F. Supplementary Note>

[0162] As described above, the present embodiment includes the following disclosures. [Configuration 1] an XR (Cross Reality) screen generation unit (214) for generating an XR (Cross Reality) screen (12) that combines real space and virtual space (15); an XR event unit (216) for detecting an event related to the XR screen (12); a placement unit (218) for updating the position of a light source (146) attached to the tip of a robot (150) in a virtual space (15); a teaching point storage unit (224) for storing teaching points for the robot (150); The XR screen generation unit (214) displays a workpiece (144) and a light source (146) in the virtual space (15) on the XR screen (12), The placement unit (218) moves the light source (146) on the XR screen (12) based on the detection of a movement event of the light source (146); The teaching point storage unit (224) of the device (100) stores teaching points for the robot (150) that moves in conjunction with the light source (146). [Configuration 2] The XR screen generation unit (214) Loads ambient light data (238) about real-world ambient light, The device (100) according to configuration 1 reproduces the reflected light of the surface of a workpiece (144) on an XR screen (12) based on light from ambient light in the real world and light from a light source (146). [Configuration 3] The placement unit (218) fixes the light source (146) on the XR screen (12) based on the detection of a release event of the light source (146); The apparatus (100) of configuration 1 or 2, wherein storing teaching points for the robot (150) includes storing teaching points for the robot (150) corresponding to the position and posture of the light source (146) after fixation. [Configuration 4] The device (100) according to configuration 1 or 2, further comprising an imaging unit (220) for photographing the workpiece (144) using a camera virtually positioned near the light source (146) based on the detection of a photography event. [Configuration 5] The device (100) according to configuration 4, wherein the XR screen generation unit (214) displays an image of the photographed workpiece (144) on the XR screen (12). [Configuration 6] The apparatus (100) according to configuration 4, further comprising a UI processing unit (234) that displays an image of the photographed workpiece (144) on a display of the apparatus (100). [Configuration 7] Further provided is an evaluation unit (232) that performs image analysis on the photographed image of the workpiece (144), 5. The apparatus (100) of configuration 4, wherein displaying the image of the workpiece (144) includes displaying the image analysis results. [Configuration 8] a light source adjusting unit (222) for changing at least one of the on / off, model number, brightness, and luminous color of the light source (146) based on the detection of a setting change event of the light source (146); The device (100) according to configuration 1 or 2, further comprising a light source setting storage unit (226) that stores the changed settings of the light source (146). [Configuration 9] an abnormality detection unit (228) that detects an abnormality in the virtual space (15); a warning unit (230) that outputs a warning based on the detection of an abnormality; 3. The apparatus (100) of configuration 1 or 2, wherein the abnormality includes the light source (146) being outside the range of motion of the robot (150). [Configuration 10] The apparatus (100) of configuration 9, wherein the abnormality includes at least one of the amount of light received by the workpiece (144) being equal to or greater than a predetermined first threshold or being less than a predetermined second threshold when the light source (146) is turned on. [Configuration 11] The device (100) according to configuration 9, wherein the warning output by the warning unit includes at least one of displaying warning information on an XR screen (12), displaying warning information on an output device (100) connected to the device (100), or vibrating an XR input device (100) to move the light source (146). [Configuration 12] The device (100) according to configuration 1 or 2, further comprising a setting output unit that outputs teaching points for the robot (150). [Configuration 13] Generating an XR screen (12) that blends real space and virtual space (15); Displaying a workpiece (144) in a virtual space (15) and a light source (146) attached to a robot (150) on an XR screen (12); Moving the light source (146) on the XR screen (12) based on the detection of a movement event of the light source (146); and storing teaching points for a robot (150) that moves in conjunction with the light source (146). [Configuration 14] Generating an XR screen (12) that blends real space and virtual space (15); Displaying a workpiece (144) in a virtual space (15) and a light source (146) attached to a robot (150) on an XR screen (12); Moving the light source (146) on the XR screen (12) based on the detection of a movement event of the light source (146); and storing teaching points for a robot (150) that moves in conjunction with the light source (146).

[0163] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope equivalent to the claims. Furthermore, the disclosures described in the embodiments and each modification are intended to be implemented, as far as possible, either alone or in combination. [Explanation of symbols]

[0164] 10 System, 12, 300, 400, 500, 600, 700, 800A, 800B, 900, 1000A, 1000B, 1000C, 1100A, 1100B XR screen, 15 Virtual space, 100 Device, 110 Input device, 120 Output device, 130 XR input device, 140 XR output device, 142 Marker, 144 Workpiece, 146 Light source, 148 Laser, 150 Robot, 152 Camera, 201 Processor, 202 RAM, 203 Memory unit, 204 Input unit, 205 Output unit, 206 External device connection unit, 207 Communication unit, 210 Program, 212 Virtual space generation unit, 214 XR screen generation unit, 216 XR event unit, 218 Placement unit, 220 Photography unit, 222 light source adjustment unit, 224 teaching point memory unit, 226 light source setting memory unit, 228 abnormality detection unit, 230 warning unit, 232 evaluation unit, 234 UI processing unit, 236 object data, 238 ambient light data, 905 image, 910 image analysis result.

Claims

1. an XR (Cross Reality) screen generation unit for generating an XR screen that combines real space and virtual space; an XR event unit for detecting an event related to the XR screen; a placement unit for updating the position of a light source attached to the tip of the robot in the virtual space; a teaching point storage unit for storing teaching points of the robot; the XR screen generation unit displays the workpiece and the light source in the virtual space on the XR screen; the placement unit moves the light source on the XR screen based on the detection of a movement event of the light source; The teaching point storage unit stores the teaching points of the robot that moves in conjunction with the light source.

2. The XR screen generation unit Loads ambient light data about real-world ambient light, The device according to claim 1 , wherein the device reproduces reflected light from the surface of the workpiece on the XR screen based on the light from the ambient light in the real world and the light from the light source.

3. the placement unit fixes the light source on the XR screen based on the detection of a release event of the light source; The apparatus according to claim 1 or 2, wherein storing the teaching points of the robot includes storing the teaching points of the robot corresponding to the position and orientation of the light source after fixation.

4. The device according to claim 1 or 2, further comprising a photographing unit that photographs the workpiece using a camera virtually positioned near the light source based on the detection of a photographing event.

5. The device according to claim 4 , wherein the XR screen generation unit displays the photographed image of the workpiece on the XR screen.

6. The apparatus according to claim 4 , further comprising a UI processing unit that displays the photographed image of the workpiece on a display of the apparatus.

7. Further provided is an evaluation unit that performs image analysis on the photographed image of the workpiece, The apparatus of claim 4 , wherein displaying the image of the workpiece includes displaying an image analysis result.

8. a light source adjustment unit for changing at least one of on / off, model number, brightness, and luminous color of the light source based on the detection of a setting change event of the light source; The device according to claim 1 , further comprising a light source setting storage unit that stores the changed light source setting.

9. an abnormality detection unit that detects an abnormality in the virtual space; a warning unit that outputs a warning based on the detection of the abnormality, The apparatus of claim 1 or 2, wherein the abnormality includes the light source being outside a movable range of the robot.

10. The device according to claim 9 , wherein the abnormality includes at least one of the amount of light received by the workpiece when the light source is turned on being equal to or greater than a predetermined first threshold value or being less than a predetermined second threshold value.

11. 10. The device according to claim 9, wherein the warning output by the warning unit includes at least one of displaying warning information on the XR screen, displaying the warning information on an output device connected to the device, or vibrating an XR input device to move the light source.

12. The device according to claim 1 or 2, further comprising a setting output unit that outputs the teaching points of the robot.

13. Generating an XR screen that blends real space and virtual space; Displaying a workpiece in the virtual space and a light source attached to a robot on the XR screen; moving the light source on the XR screen based on the detection of a movement event of the light source; and storing teaching points for the robot that moves in conjunction with the light source.

14. Generating an XR screen that blends real space and virtual space; Displaying a workpiece in the virtual space and a light source attached to a robot on the XR screen; moving the light source on the XR screen based on the detection of a movement event of the light source; and storing teaching points for the robot that moves in conjunction with the light source.

Images