Teaching device, control method for teaching device, and program for teaching device

Abstract

Provided is a technology that is for using cross reality (XR) to teach a robot while verifying the effects of a light source. A device (100) comprises an XR screen generation unit that is for generating an XR screen that blends a real space and a virtual space, an XR event unit that is for detecting events related to the XR screen (12), a placement unit that is for updating the position of a light source (146) that is provided to the tip of a robot (150) that is in the virtual space, and a teaching point storage unit that is for storing teaching points for the robot (150). The XR screen generation unit displays a workpiece and the light source (146) that are in the virtual space on the XR screen (12). The placement unit moves the light source (146) on the XR screen (12) on the basis of detection of a movement event for the light source (146). The teaching point storage unit stores teaching points for the robot (150) as moving in association with the light source (146).

Description

Teaching device, teaching device control method, and teaching device program

[0001] The present disclosure relates to a teaching technique for an industrial robot, and more particularly to a teaching technique using XR technology.

[0002] Today, articulated industrial robots (hereinafter simply referred to as "robots") are widely used in factory production lines (hereinafter, the term "production lines" and "inspection lines" will be used collectively to refer to "lines"). These robots can be used for assembling, transporting, or inspecting workpieces. Furthermore, these robots require a process known as "teaching." Traditionally, workers have taught robots by operating the actual robots on-site.

[0003] Furthermore, in recent years, a technology called XR (Cross Reality) has been developing. XR encompasses technologies related to virtual spaces and technologies that combine virtual spaces with real spaces. XR is also beginning to be used in industrial fields.

[0004] Regarding technology related to robot teaching using XR, for example, Japanese Patent Laid-Open Publication No. 2023-038781 (Patent Document 1) discloses a robot teaching system. This robot teaching system includes "a display unit that displays a virtual robot in a virtual space, a recognition unit that recognizes instructions given by a teacher to the virtual robot, and a notification unit that notifies whether or not a real robot can be driven based on the instructions. The notification unit notifies that the real robot can be driven based on the instructions, or that the real robot cannot be driven based on the instructions" (see [Abstract]).

[0005] JP 2023-038781 A

[0006] When using a robot in an inspection process, the light source is an important factor. The appropriate robot teaching point may also change depending on the light source. According to the technology disclosed in Patent Document 1, the person in charge cannot perform the robot teaching work while checking the influence of the light source. Therefore, there is a need for a technology that allows the robot teaching work to be performed while checking the influence of the light source.

[0007] The present disclosure has been made in consideration of the above-described background, and an object of one aspect is to provide a technology for using XR to perform robot teaching work while checking the influence of a light source.

[0008] According to one embodiment, an apparatus for teaching a robot is provided. The apparatus includes an XR screen generation unit for generating an XR screen that blends real space and virtual space, an XR event unit for detecting events related to the XR screen, a placement unit for updating the position of a light source attached to the tip of the robot in the virtual space, and a teaching point storage unit for storing teaching points for the robot. The XR screen generation unit displays a workpiece and a light source in the virtual space on the XR screen. The placement unit moves the light source on the XR screen based on the detection of a light source movement event. The teaching point storage unit stores teaching points for the robot that move in conjunction with the light source.

[0009] According to the above disclosure, by using the device, a user can intuitively teach a robot while checking the positions of the workpiece and light source on the XR screen.

[0010] In one aspect, the XR screen generation unit reads ambient light data relating to ambient light in the real world, and reproduces the reflected light from the surface of the workpiece on the XR screen based on the light from the ambient light in the real world and the light from the light source.

[0011] According to the disclosure above, the workpiece on the XR screen can reproduce the reflected light of the workpiece in the real world, so that by using the device, a user can teach a robot even when they are not at the planned construction site of the line, as if they were actually at the planned construction site.

[0012] In one aspect, the arrangement unit fixes the light source on the XR screen based on the detection of a light source release event, and storing teaching points for the robot includes storing teaching points for the robot corresponding to the position and orientation of the light source after fixation.

[0013] According to the above disclosure, the device can store teaching points for the robot that correspond to the positions of fixed light sources on the XR screen.

[0014] In one aspect, the device further includes a photographing unit that photographs the workpiece using a camera virtually positioned near the light source based on the detection of a photographing event.

[0015] According to the above disclosure, the device can capture an image of the workpiece at any teaching point. For example, a user can obtain an image of the workpiece captured by the inspection robot at a tentative teaching point.

[0016] In one aspect, the XR screen generation unit displays an image of the photographed workpiece on the XR screen.

[0017] According to the above disclosure, the user can easily check the captured image on the XR screen.

[0018] In one aspect, the device further includes a UI processing unit that displays an image of the photographed workpiece on a display of the device.

[0019] According to the above disclosure, the user can easily check the captured image on the display of the device.

[0020] In one aspect, the apparatus further includes an evaluation unit that performs image analysis on the captured image of the workpiece. Displaying the image of the workpiece includes displaying the image analysis results.

[0021] According to the disclosure above, the apparatus can analyze images captured at any teaching point. For example, a user can obtain the results of an appearance inspection of a workpiece by an inspection robot at a tentative teaching point.

[0022] In one aspect, the device further includes a light source adjustment unit for changing at least one of the on / off, model number, brightness, or light emission color of the light source based on the detection of a light source setting change event, and a light source setting memory unit for storing the changed light source settings.

[0023] According to the above disclosure, the device can change at least one of the on / off, model number, brightness, or light emission color of the light source, and further store the changed settings.

[0024] In one aspect, the device further includes an anomaly detection unit that detects an anomaly in the virtual space, and a warning unit that outputs a warning based on the detected anomaly. The anomaly includes a light source being outside a movable range of the robot.

[0025] According to the above disclosure, the device may output a warning based on detecting an abnormality in the virtual space, which may include an abnormality related to the range of motion of the robot.

[0026] In one aspect, the abnormality includes at least one of the amount of light received by the workpiece when the light source is turned on being equal to or greater than a predetermined first threshold value or being less than a predetermined second threshold value.

[0027] According to the above disclosure, the device can output a warning when the amount of light received by the workpiece is equal to or less than a certain level. For example, by referring to the warning, the user can recognize that the light source is too close to the workpiece, or that the light from the light source is not shining on the workpiece.

[0028] In one aspect, the warning output by the warning unit includes at least one of displaying warning information on an XR screen, displaying warning information on an output device connected to the device, or vibrating an XR input device to move a light source.

[0029] According to the above disclosure, the device may make it easy for the user to recognize the warning.

[0030] In one aspect, the device further includes a setting output unit that outputs teaching points for the robot.

[0031] According to the above disclosure, the device may output teaching points for the robot. For example, the device may directly transmit the teaching points for the robot to the actual robot.

[0032] According to one embodiment, there is provided a robot teaching method executed by an apparatus, the method including: generating an XR screen that blends a real space and a virtual space; displaying, on the XR screen, a workpiece in the virtual space and a light source attached to the robot; moving the light source on the XR screen based on detection of a light source movement event; and storing teaching points for the robot that moves in conjunction with the light source.

[0033] According to the above disclosure, by using a device that executes the method, a user can intuitively teach a robot while checking the positions of a workpiece and a light source on an XR screen.

[0034] According to one embodiment, there is provided a program for causing an apparatus to perform robot teaching, the program causing the apparatus to generate an XR screen that blends real space and virtual space, display on the XR screen a workpiece in the virtual space and a light source attached to the robot, move the light source on the XR screen based on detection of a light source movement event, and store teaching points for the robot that moves in conjunction with the light source.

[0035] According to the above disclosure, by using a device that executes the program, a user can intuitively teach a robot while checking the positions of the workpiece and light source on the XR screen.

[0036] According to one embodiment, XR can be used to perform robot teaching tasks while checking the effects of a light source.

[0037] The above and other objects, features, aspects and advantages of the present disclosure will become apparent from the following detailed description of the disclosure taken in conjunction with the accompanying drawings.

[0038] 1 is a diagram showing an example of robot teaching using the system 10 according to the present embodiment. FIG. 1 is a diagram showing an example of the configuration of the device 100. FIG. 2 is a diagram showing an example of an XR screen 300 in an initial state. FIG. 3 is a diagram showing an example of an XR screen 400 after a movement event of the XR input device 130 has occurred. FIG. 4 is a diagram showing an example of an XR screen 500 after a hold event of the light source 146 has occurred. FIG. 5 is a diagram showing an example of an XR screen 600 after a movement event of the light source 146 has occurred. FIG. 6 is a diagram showing an example of an XR screen 700 after a release event of the light source 146 has occurred. FIG. 7 is a diagram showing an example of a function for reproducing the effect of the light source 146 on the workpiece 144 on the XR screen 12. FIG. 8 is a diagram showing an example of a function for photographing the workpiece 144. FIG. 9 is a diagram showing an example of a function for adjusting the brightness of the light source 146. FIG. 10 is a diagram showing an example of a function for adjusting the emitted color of the light source 146. FIG. 11 is a diagram showing an example of a processing procedure of the system 10.

[0039] Hereinafter, embodiments of the technical concept of the present disclosure will be described with reference to the drawings. In the following description, identical components are assigned the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. Furthermore, each embodiment, each modification, each software configuration, each hardware configuration, each function, each process, etc. may be selectively combined as appropriate.

[0040] <A. Application example>

[0041] 1 is a diagram showing an example of robot teaching using a system 10 according to the present embodiment. System 10 is a system for teaching a robot while checking the effect of a light source using XR. By using system 10, a user can intuitively teach a robot.

[0042] (a. Terminology)

[0043] First, some terms necessary for understanding the technology of this disclosure will be explained, more specifically, line, work, robot, teaching, XR, light source, and system.

[0044] In this specification, the term "line" refers to equipment composed of a conveyor belt, various sensors, robots and their control devices, etc. The term "line" includes a manufacturing line for manufacturing products or parts thereof. The term "line" also includes a conveyance line used in a logistics warehouse. The term "line" also includes an inspection line for products or parts thereof. The term "line" may also be used for any other purpose. Furthermore, the term "line" may be a combination of lines for various purposes.

[0045] In this specification, a "workpiece" refers to an object that is processed, assembled, transported, or inspected on a line. A workpiece may be a finished product. A workpiece may also be a part used to assemble a product. In this specification, a workpiece may refer to a workpiece that is a virtual object reproduced in a virtual space. In this specification, a workpiece may also refer to a workpiece that exists in real space.

[0046] In this specification, the term "robot" refers to an industrial robot used in conjunction with a production line. The robot includes a vertical articulated robot, a horizontal articulated robot, a parallel link robot, an orthogonal robot, and a robot that combines these mechanisms.

[0047] As used herein, "teaching" refers to the task of configuring a robot or its control device to perform certain operations. Teaching can include setting the robot's destination position or path. Teaching is an extremely important task when using a robot and places a significant burden on the operator. As an example, consider a robot used for product assembly or inspection. In this case, the robot must move extremely accurately to the product assembly or inspection position. Therefore, the operator repeatedly fine-tunes the robot's destination position while checking the robot's operation on-site. Hereinafter, the robot's destination position may be referred to as a "teaching point." A teaching point can include the coordinates and orientation of the robot's tip or a tool attached to the tip. The orientation can be expressed as a combination of angles around each coordinate axis (X, Y, Z).

[0048] In this specification, "XR" is an abbreviation for Cross Reality. XR is a general term for technologies that combine the real world and the virtual world, and encompasses virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0049] VR is a technology that allows users to experience a virtual world. Users use a VR headset or the like to experience VR. For example, by using VR technology, users can experience being immersed in a game world.

[0050] AR is a technology that overlays virtual objects onto real-world scenery. When experiencing AR, a user uses a smartphone, tablet, AR glasses, or the like. For example, by using AR technology, a user can view content that is virtually placed in a city.

[0051] MR is a technology that generates mixed reality by overlaying the real world and the virtual world. When experiencing MR, a user uses an MR headset or the like. MR is a technology that further develops AR. AR simply overlays virtual objects on real-world scenery by displaying the virtual objects on AR glasses or the like. In contrast, MR uses an MR device to capture real-world images. Then, virtual objects are combined with the real-world images to generate mixed reality. As a result, for example, MR can generate images that take into account the anteroposterior relationship between real-world objects and virtual objects.

[0052] In this specification, the term "light source" refers to any object that emits light. As an example, lighting is a light source. Sources of natural light or ambient light are also light sources. In this specification, the term "light source" may also refer to a light source reproduced in a virtual world. A light source in VR and AR may illuminate a virtual object. A light source in MR may illuminate both real and virtual objects. In other words, a light source in XR, which includes VR, AR, and MR, may illuminate at least a virtual object.

[0053] As used herein, a "system" refers to a single device or a combination of multiple devices. A device may include a personal computer, a workstation, a server, a tablet, or a smartphone. A device may also include a system-on-a-chip (SoC) or a system-on-module (SoM). A device may also include any peripheral devices, such as a switch, a router, a display, a keyboard, and a mouse. A device may also include virtual machines and instances built in a cloud environment. In some aspects, a system may be connected to input / output devices, such as a display and a keyboard, and used as a standalone device. In other aspects, a system may provide various functions as a service or web application over a network. In this case, a user may use the system's functions through a browser or client software installed on their device.

[0054] (b. System Overview)

[0055] Next, the configuration of the system 10 will be described. The system 10 includes a device 100, an input device 110, an output device 120, an XR input device 130, and an XR output device 140. In some aspects, the system 10 may not include the input device 110, the output device 120, or both. In these cases, the XR input device 130 may also function as the input device 110. Furthermore, the XR output device 140 may also function as the output device 120.

[0056] The device 100 executes a teaching process based on input from the user. More specifically, the device 100 generates and outputs XR images, stores teaching points, determines the type and position of a light source, etc. The device 100 can also recognize user gesture operations by analyzing the XR screen 12 captured by the XR output device 140.

[0057] The input device 110 inputs various settings to the device 100. The input device 110 may be configured to be able to input the same information to the device 100 as information input to the device 100 from the XR input device 130. The output device 120 displays a teaching setting screen. The output device 120 may display various settings, the XR screen 12, and the virtual space 15. The output device 120 may also display an image captured by a camera 152 in the virtual space 15.

[0058] The XR input device 130 is an interface for manipulating objects within the XR screen 12. In the system 10, the XR input device 130 can change the position of the light source 146 of the robot 150. The light source 146 corresponds to a light attached to the tip of the robot 150. Detailed operation will be described later. The XR input device 130 may include an acceleration sensor, a gyro sensor, and any other sensors. In one aspect, the XR input device 130 may calculate the position of the XR input device 130 based on signals from each sensor. In this case, the XR input device 130 transmits information regarding the position of the XR input device 130 to the device 100. In another aspect, the XR input device 130 may calculate the direction and amount of movement of the XR input device 130 based on signals from each sensor. In this case, the XR input device 130 transmits information regarding the direction and amount of movement of the XR input device 130 to the device 100. Furthermore, in another aspect, the XR input device 130 may transmit signals from each sensor to the device 100. In this case, the device 100 calculates the direction and amount of movement of the XR input device 130 based on the signals from each sensor. In either case, the device 100 can detect the current position, direction and amount of movement of the XR input device 130 in real time.

[0059] The XR input device 130 may also accept various operational inputs from the user. In one aspect, the XR input device 130 may include any operation unit, such as a button, a dial, a touch sensor, or a wheel. The XR input device 130 may accept various operational inputs from the user via these operation units. In another aspect, the XR input device 130 may read a user's gestures using an acceleration sensor, a gyro sensor, or the like. In this case, the XR input device 130 accepts the gestures as various operational inputs from the user. Alternatively, the XR output device 140 may capture images of the user's gestures using a built-in camera (not shown). In this case, the device 100 may receive video from the XR output device 140 and analyze the video to detect the gestures. In either case, information regarding the various operational inputs from the user is transmitted to the device 100.

[0060] The XR output device 140 outputs the XR screen 12. The XR output device 140 may be realized as a headset, a glasses-type device, or the like. The user can view the XR screen 12 by wearing the XR output device 140. The XR output device 140 may have a function for capturing the XR screen 12. The XR output device 140 transfers the captured XR screen 12 to the device 100. In one aspect, the XR output device 140 may record only images of virtual objects superimposed on the real world. In another aspect, the device 100 may have a capture function. In this case, the device 100 records images of virtual objects in the virtual space 15.

[0061] (c.XR screen)

[0062] Next, the XR screen 12 displayed on the XR output device 140 will be described. In the example of FIG. 1 , a user uses the system 10 to teach a robot 150 reproduced in a virtual space 15 for visual inspection of a workpiece 144. A light source 146 and a camera 152 are attached to the tip of the robot 150. Visual inspection is one example, and the system 10 can also be used to teach any operation of the workpiece 144, such as processing, assembling, or picking up.

[0063] The XR screen 12 is a screen displayed on the XR output device 140. In one aspect, the XR screen 12 may be a screen displayed using AR technology. In this case, the XR screen 12 is a screen on which a virtual object is superimposed on the real world. In another aspect, the XR screen 12 may be a screen displayed using MR technology. In this case, the XR screen 12 is a mixed reality screen on which a virtual object is combined with an image of the real world.

[0064] In the example of FIG. 1 , the XR screen 12 displays real-space objects and virtual objects. The input device 110, the output device 120, and the marker 142 are real-space objects. The real-space objects may include any object within the user's field of view in addition to the objects shown in FIG. 1 . The workpiece 144, the light source 146, and the laser 148 are virtual objects. In some aspects, the XR screen 12 may include a display of the robot 150, the camera 152, or both. In this case, the robot 150, the camera 152, or both may be displayed semi-transparently, or only their outlines may be displayed.

[0065] The marker 142 is a reference for determining the display coordinates of a virtual object. The workpiece 144 and the light source 146 may be displayed at a predetermined location as viewed from the marker 142. In one aspect, the system 10 may recognize the marker 142 as the origin of three-dimensional space. The system 10 may then place the workpiece 144 and the light source 146 at a predetermined relative position (which may be expressed in XYZ coordinates) as viewed from the origin.

[0066] The workpiece 144 corresponds to the workpiece 144 in the virtual space 15. The light source 146 corresponds to a light attached to the tip of the robot 150 in the virtual space 15. The light source 146 may be represented by CAD (Computer Aided Design) data of the light, or may be represented by any shape such as a cylinder.

[0067] The laser 148 indicates the direction in which the tip of the XR input device 130 is pointing. To the user, the laser 148 appears as a visible laser emitted from the tip of the XR input device 130. The laser 148 may be represented as a dotted line, a straight line, or any other shape. The laser 148 may also be represented as any color or brightness.

[0068] The system 10 reproduces the effects of various light sources on the workpiece 144 within the XR screen 12. The first effect is the effect of the light source 146, which is a light source attached to the robot 150. When the light source 146 is placed near the workpiece 144, the user sees the workpiece 144 as being illuminated by the light source 146. The second effect is the effect of ambient light. The device 100 is configured to allow pre-registration of ambient light information, such as the planned construction site of the production line. As an example, the ambient light information may be a high dynamic range (HDR) image or video. By reading the ambient light information, the device 100 can reproduce the ambient light, such as the planned construction site of the production line, within the virtual space 15 and the XR screen 12. This makes the workpiece 144 appear to the user as if it were illuminated by ambient light. However, the device 100 does not necessarily need to place an ambient light source within the XR screen 12. The device 100 may reproduce only the reflection of light on the surface of the workpiece 144 due to ambient light within the XR screen 12.

[0069] (d. Teaching Procedure)

[0070] Next, a teaching procedure performed by a user using the system 10 will be described. First, the user aims the laser 148 at the light source 146. Then, the user performs a first operation on the XR input device 130, thereby drawing the light source 146 to the tip of the XR input device 130 or to its vicinity. In one aspect, the first operation may be pressing any button provided on the XR input device 130. In another aspect, the first operation may be a gesture using the XR input device 130 or a part of the user's body.

[0071] Next, the user moves the XR input device 130 to move the light source 146 that has been drawn to the XR input device 130 to a desired position. The XR input device 130 is held by the user. That is, the user can intuitively move the light source 146 as if moving a flashlight held in their hand. When the user moves the light source 146, the light source 146 in the virtual space 15 also moves in conjunction with it. As described above, in the virtual space 15, the light source 146 is attached to the tip of the robot 150. Therefore, when the light source 146 moves, the robot 150 also moves in conjunction with it. That is, it can be said that the user moves the tip of the robot 150 together with the light source 146. Hereinafter, a state in which the light source 146 is held at or near the tip of the XR input device 130 will be referred to as a "hold state." Furthermore, putting the light source 146 in a hold state will be referred to as "holding the light source 146." More specifically, in the hold state, the device 100 acquires position information of the destination of the light source 146 from the XR input device 130 as needed. Alternatively, in the hold state, the device 100 may calculate position information of the destination of the light source 146 as needed, based on the positional relationship between the XR input device 130 and the light source 146. The device 100 moves the light source 146 in the virtual space 15 based on the position information of the destination of the light source 146. Furthermore, the device 100 moves the robot 150 based on the movement of the light source 146 in the virtual space 15. Moving the robot 150 includes changing the angles of each joint of the robot 150.

[0072] Next, the user can fix the light source 146 within the XR screen 12 by performing a second operation. In one aspect, the second operation may be pressing any button provided on the XR input device 130. In another aspect, the second operation may be a gesture using the XR input device 130 or a part of the user's body. Hereinafter, the state in which the light source 146 is fixed within the XR screen 12 will be referred to as a "released state." Furthermore, putting the light source 146 into the released state will be referred to as "releasing the light source 146."

[0073] By using the function of holding and releasing the light source 146, the user can intuitively determine the position of the light source 146 relative to the workpiece 144. By determining the position of the light source 146, the teaching point of the robot corresponding to the position of the light source 146 is also automatically determined. In other words, the user can intuitively determine the teaching point of the robot equipped with the light source 146.

[0074] Pre-registered ambient light is reproduced on the XR screen 12. That is, the ambient light and the light reflected on the surface of the workpiece 144 by the light source 146 are reproduced on the XR screen 12. As an example, the pre-registered ambient light may be the ambient light at the planned construction site of the production line. In this case, the user can observe the workpiece 144 via the XR screen 12 as if they were at the planned construction site of the production line. Details of the various functions of the system 10 will be described later with reference to FIGS. 3 to 12.

[0075] <B. Device configuration>

[0076] 2 is a diagram showing an example of the configuration of the device 100. The device 100 includes a processor 201, a RAM (Random Access Memory) 202, a storage unit 203, an input unit 204, an output unit 205, an external device connection unit 206, and a communication unit 207. The storage unit 203 also stores a program 210.

[0077] The processor 201 may execute programs for implementing various functions of the device 100. The processor 201 may be configured, for example, with at least one integrated circuit. According to an embodiment, the integrated circuit may include at least one central processing unit (CPU), at least one graphics processing unit (GPU), at least one field programmable gate array (FPGA), at least one application specific integrated circuit (ASIC), at least one artificial intelligence (AI) chip, or a combination thereof.

[0078] RAM 202 functions as a workspace for processor 201. RAM 202 stores programs executed by processor 201 and data referenced by processor 201. In one aspect, RAM 202 can be realized by a dynamic random access memory (DRAM), a static random access memory (SRAM), or the like.

[0079] Storage unit 203 is a non-volatile memory that stores programs executed by processor 201 and data referenced by processor 201. Processor 201 executes programs read from storage unit 203 to RAM 202 and references data read from storage unit 203 to RAM 202. In one aspect, storage unit 203 can be realized by a hard disk drive (HDD), a solid state drive (SSD), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash memory, or the like.

[0080] The input unit 204 may be connected to any input device 110, such as a keyboard, a mouse, a touchpad, or a gamepad. In one aspect, the input unit 204 may be realized by a USB terminal, a PS / 2 terminal, a Bluetooth (registered trademark) module, or the like.

[0081] Output unit 205 can be connected to any output device 120, such as a cathode ray tube display, a liquid crystal display, or an organic electroluminescence (EL) display. In one aspect, output unit 205 can be realized by a USB terminal, a D-sub terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, a DisplayPort terminal, or the like.

[0082] The external device connection unit 206 can be connected to any external device, such as the XR input device 130, the XR output device 140, a printer, a scanner, or an external HDD. In one aspect, the external device connection unit 206 can be implemented by a USB (Universal Serial Bus) terminal or the like. In another aspect, the XR input device 130 may be connected to the input unit 204. Furthermore, the XR output device 140 may be connected to the output unit 205.

[0083] Communication unit 207 is connected to other devices via a wired network or a wireless network. In one aspect, communication unit 207 may be implemented by a wired local area network (LAN) port, a Wi-Fi (registered trademark) (Wireless Fidelity) module, or the like. In another aspect, communication unit 207 may transmit and receive data using a communication protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol) or UDP (User Datagram Protocol).

[0084] The program 210 includes various functional units for implementing the robot's teaching function. The program 210 may also reference various data for implementing the robot's teaching function. More specifically, the program 210 includes, as functional units, a virtual space generation unit 212, an XR screen generation unit 214, an XR event unit 216, a placement unit 218, a capture unit 220, a light source adjustment unit 222, a teaching point storage unit 224, a light source setting storage unit 226, an abnormality detection unit 228, a warning unit 230, an evaluation unit 232, and a UI (User Interface) processing unit 234. In some aspects, some of these functional units may be implemented as hardware. The program 210 also references object data 236 and ambient light data 238.

[0085] The virtual space generation unit 212 generates the virtual space 15. The virtual space generation unit 212 places at least the robot 150 to be taught, the light source 146, and the workpiece 144 within the virtual space 15. In some aspects, the virtual space generation unit 212 may place various tools, such as a camera 152 attached to the tip of the robot 150, in addition to the light source 146, within the virtual space 15. In other aspects, the virtual space generation unit 212 may place some or all of a line within the virtual space 15. The virtual space generation unit 212 may also read ambient light data 238 to reproduce ambient light in the real world within the virtual space 15. Furthermore, the virtual space generation unit 212 also updates the virtual space 15 based on an update of the XR screen 12. This may link the positions of each object, light source settings, and the like between the virtual space 15 and the XR screen 12. The virtual space generation unit 212 and the XR screen generation unit 214 may be configured to constantly exchange the latest statuses with each other.

[0086] The XR screen generation unit 214 generates the XR screen 12. In one aspect, the XR screen generation unit 214 may overlay a virtual object on the real world using AR technology. In another aspect, the XR screen generation unit 214 may generate a mixed reality screen by combining the real world and the virtual world using MR technology. The XR screen generation unit 214 may determine the display position of each object using the markers 142.

[0087] The XR event unit 216 detects various events related to the XR screen 12. The XR event unit 216 notifies other functional units of the detected events as necessary. The various events may include various user operation inputs. The events include a hold event corresponding to holding the light source 146 and a release event corresponding to releasing the light source 146. The events also include a movement event. The movement event relates to the movement of the XR input device 130 in a released state and the movement of the XR input device 130 in a held state. The events also include a photographing event corresponding to photographing the workpiece 144 by the camera 152. The events also include a lighting switching event corresponding to switching the light source 146 on and off. The events also include a light source setting change event corresponding to changing at least one of the model number, brightness, and light emission color of the light source 146.

[0088] The placement unit 218 updates the placement of objects on the XR screen 12 based on the event acquired from the XR event unit 216. Typically, the placement unit 218 updates the placement of the light source 146 by detecting a hold event or a movement event. In one aspect, the XR screen generation unit 214 may include the function of the placement unit 218. In this case, the XR event unit 216 notifies the XR screen generation unit 214 of the detected event.

[0089] The photographing unit 220 photographs the virtual space 15 using the camera 152 based on the photographing event acquired from the XR event unit 216. As an example, the camera 152 photographs the workpiece 144 in the virtual space 15. The ambient light and light source 146 of the real space are reproduced in the virtual space 15. Therefore, the photographing unit 220 can photograph an image similar to the image of the workpiece 144 photographed in the real world. The photographing unit 220 outputs the image obtained by photographing to the evaluation unit 232.

[0090] The light source adjustment unit 222 switches the light source 146 between on and off based on the lighting switching event acquired from the XR event unit 216. Furthermore, the light source adjustment unit 222 changes at least one of the model number, brightness, and emitted color of the light source 146 based on the light source setting change event acquired from the XR event unit 216.

[0091] The teaching point storage unit 224 updates and stores the teaching points based on a release event acquired from the XR event unit 216. In some aspects, the teaching point storage unit 224 may update and store the teaching points based on the occurrence of a teaching point storage event. In this case, the XR input device 130 is configured to be able to accept an operation input for storing the teaching points.

[0092] The light source setting storage unit 226 updates and stores the settings of the light source 146. In one aspect, the light source setting storage unit 226 may store the current settings of the light source 146 in the virtual space 15 based on a lighting switch event and / or a light source setting change event. In another aspect, the light source setting storage unit 226 may store the current settings of the light source 146 in the virtual space 15 based on the occurrence of a storage event for the settings of the light source 146. In this case, the XR input device 130 is configured to be able to accept an operation input for changing the settings of the light source 146.

[0093] The anomaly detection unit 228 detects one or more anomalies in the virtual space 15. Furthermore, information about the detected anomaly is output to the warning unit 230. The one or more anomalies include an anomaly related to the teaching point and an anomaly related to the light source. As an example, suppose that the user moves the XR input device 130 significantly, causing the light source 146 to move outside the movable range of the robot 150. In this case, the anomaly detection unit 228 detects an anomaly related to the teaching point. As another example, suppose that the light irradiated on the workpiece 144 is brighter than a first threshold value or darker than a second threshold value. In this case, the anomaly detection unit 228 detects an anomaly related to the light source. That is, the anomaly detection unit 228 may detect an anomaly such as insufficient light irradiating the workpiece 144 or excessively strong light irradiating the workpiece 144.

[0094] The warning unit 230 displays information about the one or more detected abnormalities on the XR screen 12, the output device 120, or both. By referring to the displayed information about the one or more abnormalities, the user can recognize that there is a problem with their own operation input.

[0095] The evaluation unit 232 analyzes the image acquired from the imaging unit 220 and displays the analysis results on the XR screen 12, the output device 120, or both. As an example, by referring to the analysis results, the user can recognize whether or not the appearance inspection of the workpiece 144 has been completed. By using the functions of the evaluation unit 232, the user can easily check the results of the appearance inspection of the workpiece 144 for each teaching point.

[0096] The UI processing unit 234 displays a UI related to the system 10 on the output device 120. The UI processing unit 234 may also receive any operation input from the user via the UI displayed on the output device 120. As an example, the UI processing unit 234 may display at least one of the XR screen 12, a screen of the virtual space 15, a warning screen, and a screen showing the results of image analysis on the output device 120. All screens displayed on the output device 120 may also be displayed via the UI processing unit 234.

[0097] The object data 236 is data on an object to be placed in the virtual space 15. As an example, the object data 236 may be CAD data. A virtual object displayed on the XR screen 12 may also be generated from the object data 236. The device 100 may register necessary object data 236 in advance via the UI processing unit 234 or the like.

[0098] The ambient light data 238 is data for reproducing ambient light. As an example, the ambient light data 238 may be an HDR image or video. The necessary ambient light data 238 may be registered in advance in the device 100 via the UI processing unit 234 or the like. Furthermore, the device 100 may be configured to be able to switch between multiple sets of ambient light data 238. This allows the user to reproduce the environments of multiple factories in the virtual space 15 and easily teach robots installed in each factory.

[0099] As described with reference to FIG. 2 , the device 100 includes an XR screen generation unit 214 for generating an XR screen 12 that blends the real space and the virtual space 15. The device 100 also includes an XR event unit 216 for detecting an event related to the XR screen 12. The device 100 also includes a placement unit 218 for updating the position of a light source attached to the tip of the robot 150 in the virtual space 15. The device 100 also includes a teaching point storage unit 224 for storing teaching points for the robot 150. The XR screen generation unit 214 displays the workpiece 144 and the light source 146 in the virtual space 15 on the XR screen 12. The placement unit 218 moves the light source 146 on the XR screen 12 based on the detection of a movement event of the light source 146. The teaching point storage unit 224 stores teaching points for the robot 150 that move in conjunction with the light source 146.

[0100] In one aspect, the XR screen generation unit 214 reads ambient light data 238 relating to ambient light in the real world. Then, the XR screen generation unit 214 reproduces the reflected light from the surface of the workpiece 144 on the XR screen 12 based on the light from the ambient light in the real world and the light from the light source 146.

[0101] In one aspect, the device 100 may further include a setting output unit (not shown) that outputs teaching points for the robot 150. The setting output unit may also output the teaching points and light source settings. The setting output unit may output various settings to any device, such as the actual robot 150 or its control device.

[0102] C. Teaching Procedure

[0103] 3 to 11, the procedure for teaching the robot 150 using the system 10 will be described. The functions related to the light source 146 and the visual inspection test function will also be described. The screens shown in FIGS. 3 to 11 are examples of the display on the XR screen 12.

[0104] (a. Teaching point memorization procedure)

[0105] 3 to 7 show typical examples of a series of steps until the robot 150's teaching points are memorized. The steps from the display of the initial screen until the user memorizes the robot's teaching points will be described with reference to FIGS. 3 to 7. The various functions shown in FIGS. 8 to 11 can also be performed during the operations shown in FIGS. 3 to 7. Each of the XR screens shown in FIGS. 3 to 11 is an example of the XR screen 12 displayed in various scenes.

[0106] FIG. 3 is a diagram illustrating an example of an XR screen 300 in an initial state. The XR screen 300 is a screen before a user performs a teaching task. The device 100 places the workpiece 144 and the light source 146 on the XR screen 300 based on the position of the marker 142. The device 100 also displays the laser 148 on the XR screen 300 based on CAD data stored in the memory unit 203. The device 100 also determines the brightness and emitted color of the light source 146 based on the light source data stored in the memory unit 203. The device 100 also reproduces ambient light on the XR screen 300 based on ambient light data 238 stored in the memory unit 203. As an example, the device 100 reproduces the light reflected from the surface of the workpiece 144 by the ambient light.

[0107] In one aspect, the system 10 may display the camera 152 and / or the robot 150 on the XR screen 300. In this case, the system 10 may display a semi-transparent version of the camera 152 and / or the robot 150 on the XR screen 300. Alternatively, the system 10 may display only the outlines of the camera 152 and / or the robot 150 on the XR screen 300. The same applies to the XR screens in FIGS. 4 to 11.

[0108] 4 is a diagram showing an example of an XR screen 400 after a movement event occurs for the XR input device 130. The XR screen 400 is a screen of a scene after the XR screen 300. When the user moves the XR input device 130, the laser 148 also moves in conjunction with the XR input device 130. In the example of FIG. 4, the laser 148 is directed toward the light source 146.

[0109] The operation of FIG. 4 will be described in more detail. When the user moves the XR input device 130, information regarding the direction and amount of movement of the XR input device 130 is transmitted from the XR input device 130 to the apparatus 100. In connection with this, the XR event unit 216 detects the occurrence of a movement event of the XR input device 130. The XR event unit 216 outputs the detected event to the placement unit 218. The placement unit 218 moves the position of the laser 148 in conjunction with the XR input device 130. At this time, the placement unit 218 may refer to information regarding the direction and amount of movement of the XR input device 130. The XR screen generation unit 214 updates the XR screen 300 to the XR screen 400 based on the movement of the object.

[0110] The same processing as described above is also performed for the operations shown in FIGS. 5 to 7 . That is, the XR event unit 216 detects the occurrence of some event and outputs the detected event to the arrangement unit 218. Then, the arrangement unit 218 moves the object based on the notification of the event. At this time, the arrangement unit 218 may refer to various data in the storage unit 203 as necessary. Furthermore, the XR screen generation unit 214 updates the XR screen 12 based on the movement of the object. In one aspect, the XR screen generation unit 214 may update the XR screen 12 based on the notification of the event from the XR event unit 216. In another aspect, the XR screen generation unit 214 may update the XR screen 12 based on the notification of the movement of the object from the arrangement unit 218.

[0111] 5 is a diagram showing an example of an XR screen 500 after a hold event occurs for the light source 146. The XR screen 500 is a screen of a scene after the XR screen 400. Suppose that the user inputs a hold operation into the XR input device 130 while the laser 148 is shining on the light source 146. In this case, the light source 146 is attracted to the tip of the XR input device 130 or its vicinity. In other words, the light source 146 is held by the XR input device 130.

[0112] 6 is a diagram showing an example of an XR screen 600 after a movement event occurs for the light source 146. Assume that the user moves the XR input device 130 while holding the light source 146. In this case, the light source 146 also moves while being held by the XR input device 130. The movement here includes translation, rotation, or a combination of these of the light source 146.

[0113] FIG. 7 is a diagram illustrating an example of an XR screen 700 after a release event for the light source 146 occurs. Assume that while the light source 146 is held, the user inputs a release operation into the XR input device 130. In this case, the light source 146 is fixed at the position where the release operation was input. When the light source 146 is released, a new teaching point is stored in the teaching point storage unit 224. The new teaching point is the position where the light source 146 is fixed. The position of the light source 146 includes the coordinates and orientation of the light source 146 in the virtual space 15. In this way, the placement unit 218 fixes the light source on the XR screen 12 based on the detection of a light source release event. Storing the teaching point for the robot 150 includes storing a teaching point for the robot 150 corresponding to the position and orientation of the light source 146 after fixation.

[0114] 3 to 7, the user can intuitively determine the relative position of the light source 146 with respect to the workpiece 144. In other words, the user can intuitively determine the relative position of the tool or end effector at the tip of the robot 150 with respect to the workpiece 144.

[0115] (b. Function to reproduce the effects of light sources)

[0116] FIG. 8 is a diagram showing an example of a function that reproduces the effect of a light source 146 on a workpiece 144 on an XR screen 12. In an XR screen 800A, the light source 146 is located beside the workpiece 144. In this case, light from the light source 146 illuminates the side surface of the workpiece 144. As a result, the side surface of the workpiece 144 appears to be strongly illuminated to the user. In an XR screen 800B, the light source 146 is located above the workpiece 144. In this case, light from the light source 146 illuminates the top surface of the workpiece 144. As a result, the top surface of the workpiece 144 appears to be strongly illuminated to the user.

[0117] As described above, the device 100 reproduces the reflected light from the surface of the workpiece 144 in accordance with the position of the light source 146. More specifically, the device 100 reproduces the reflected light from the surface of the workpiece 144 in the virtual space 15 based on the light source 146 and ambient light. The device 100 then displays the workpiece 144 with the reflected light reproduced on the XR screen 12.

[0118] The user can switch the illumination of the light source 146 on and off by inputting an operation to switch the illumination of the light source 146 to the XR input device 130. The user can switch the illumination of the light source 146 on and off by using a gesture or by pressing a button on the XR input device 130. When an operation to switch the illumination of the light source 146 is input to the XR input device 130, the XR event unit 216 detects an illumination switching event of the light source 146.

[0119] As described above, the user can determine the placement of the light source 146 while checking the brightness of the surface of the workpiece 144 via the XR screen 12. In other words, the user can determine the teaching points for the robot 150 while checking the brightness of the surface of the workpiece 144 via the XR screen 12.

[0120] (c. Visual inspection test function)

[0121] FIG. 9 is a diagram showing an example of a photographing function for a workpiece 144. On an XR screen 900, a user photographs a workpiece using a camera 152 attached to the tip of a robot 150. The position of the camera 152 is substantially the same as the light source 146. Therefore, the user can photograph the workpiece 144 while intuitively adjusting the position of the camera 152 by moving the XR input device 130. The device 100 can photograph the workpiece 144 based on receiving an input operation for photographing. The device 100 can receive an input operation for photographing via the XR input device 130 or the input device 110.

[0122] The camera 152 does not capture a portion of the XR screen 900, but captures only the objects in the virtual space 15. That is, the captured image 905 shows only the objects present in the virtual space 15. Using FIG. 9 as an example, the marker 142 and the workpiece 144 are present in the image 905. Assume that the user captures a photo with the camera 152 capturing the marker 142 and the workpiece 144 within its capture range. In this case, only the workpiece 144 appears in the image 905 obtained by capturing the photo. Furthermore, the reflected light from the object is reproduced within the virtual space 15. Therefore, the image 905 also includes the reflected light from the surface of the object, the workpiece 144.

[0123] The device 100 can capture an image of the workpiece 144 regardless of the state of the light source 146. That is, the device 100 can capture an image of the workpiece 144 while the light source 146 is held. The device 100 can also capture an image of the workpiece 144 while the light source 146 is released.

[0124] Image 905 is displayed on XR screen 900, output device 120, or both. In one aspect, displayed image 905 may include image analysis result 910. As an example, assume that a user is teaching robot 150 for visual inspection of workpiece 144. In this case, when the user photographs workpiece 144, device 100 executes an image recognition program for the actual visual inspection on the photographed image. Then, device 100 outputs image 905 including image analysis result 910. In one aspect, device 100 may be configured to be able to accept settings for the image recognition program to be used via UI processing unit 234 or the like.

[0125] The user can photograph an object in the virtual space 15 with the camera 152 by inputting a photographing operation to the XR input device 130. The user can photograph the workpiece 144 by using a gesture or pressing a button on the XR input device 130. When a photographing operation is input to the XR input device 130, the XR event unit 216 detects a photographing event.

[0126] As described above, the user can easily check whether the appearance inspection can be properly performed at the current position of the camera 152 while intuitively adjusting the position of the camera 152 via the XR screen 12. This allows the user to find a position where the appearance inspection can be properly performed and register the position in the device 100 as a teaching point.

[0127] 9, the device 100 includes the photographing unit 220. The photographing unit 220 photographs the workpiece 144 using a camera 152 virtually placed near the light source 146 based on the detection of a photographing event.

[0128] In one aspect, the XR screen generation unit 214 may display an image 905 of the photographed workpiece 144 on the XR screen 12. In another aspect, the UI processing unit 234 may display the image 905 of the photographed workpiece 144 on the output device 120, which is the display of the apparatus 100. The image 905 may also be displayed on both the XR screen 12 and the output device 120.

[0129] The apparatus 100 also includes an evaluation unit 232. The evaluation unit 232 performs image analysis on the photographed image 905 of the workpiece 144. Displaying the image 905 of the workpiece includes displaying the image analysis result 910.

[0130] (d. Lighting setting change function)

[0131] 10 is a diagram illustrating an example of a function for adjusting the brightness of the light source 146. The user can adjust the brightness of the light source 146 while referring to the XR screen 12. XR screens 1000A, 1000B, and 1000C show how the brightness of the light source 146 changes. Referring to the XR screens 1000A, 1000B, and 1000C, it can be seen that the brightness of the light source 146 gradually increases. The device 100 can adjust the brightness of the light source 146 gradually higher and gradually lower.

[0132] More specifically, the device 100 adjusts the brightness of the light source 146 in the virtual space 15. Then, the result of adjusting the brightness of the light source 146 is reflected on the XR screen 12. The device 100 can adjust the brightness of the light source 146 regardless of the state of the light source 146. That is, the device 100 can adjust the brightness of the light source 146 when the light source 146 is held. Furthermore, the device 100 can adjust the brightness of the light source 146 when the light source 146 is released.

[0133] The user can adjust the brightness of the light source 146 by inputting an operation to adjust the brightness of the light source 146 to the XR input device 130. The user can adjust the brightness of the light source 146 by using a gesture or by pressing a button on the XR input device 130. When an operation to adjust the brightness of the light source 146 is input to the XR input device 130, the XR event unit 216 detects an adjustment event of the brightness of the light source 146.

[0134] As described above, the user can easily adjust the brightness of the light source 146 via the XR screen 12. That is, the user can easily adjust the brightness of the lighting attached to the tip of the robot 150 via the XR screen 12. As an example, the user can combine the photographing function and the function of adjusting the brightness of the light source 146 to find the brightness of the light source 146 that is suitable for visual inspection of the workpiece 144.

[0135] FIG. 11 is a diagram illustrating an example of a function for adjusting the emitted color of the light source 146. The user may adjust the emitted color of the light source 146 while referring to the XR screen 12. XR screens 1100A and 1100B illustrate changes in the emitted color of the light source 146. In one aspect, the device 100 may be configured to individually adjust the brightness of each of the RGB colors of the light source 146. In another aspect, the device 100 may be configured to allow the user to select the emitted color of the light source 146 from a plurality of candidates. In another aspect, the device 100 may be configured to provide a function for selecting a color from a gradation, a color palette, or the like displayed on the XR screen 12. Furthermore, the device 100 may be configured to switch the operation procedure or UI for adjusting the emitted color based on the model number of the lighting device selected as the light source 146.

[0136] Similar to adjusting the brightness of the light source 146, the device 100 adjusts the emission color of the light source 146 in the virtual space 15. The result of adjusting the emission color of the light source 146 is then reflected on the XR screen 12. The device 100 can also adjust the emission color of the light source 146 regardless of the hold / release state of the light source 146. The user can adjust the emission color of the light source 146 by inputting an operation to adjust the emission color of the light source 146 to the XR input device 130. When an operation to adjust the emission color of the light source 146 is input to the XR input device 130, the XR event unit 216 detects an adjustment event of the emission color of the light source 146.

[0137] As described above, the user can easily adjust the emission color of the light source 146 via the XR screen 12. That is, the user can easily adjust the emission color of the light attached to the tip of the robot 150 via the XR screen 12. As an example, the user can combine the photographing function and the function for adjusting the emission color of the light source 146 to find the emission color of the light source 146 that is suitable for visual inspection of the workpiece 144.

[0138] In one aspect, the device 100 may be configured to be able to change the model number of the lighting corresponding to the light source 146. In this case, the device 100 changes the model number of the lighting corresponding to the light source 146 in the virtual space 15 and reflects the result on the XR screen 12. Furthermore, the XR event unit 216 detects a model number change event based on the XR input device 130 receiving an operation input to change the model number of the lighting.

[0139] 10 and 11 , the device 100 includes a light source adjustment unit 222. The light source adjustment unit 222 changes at least one of the on / off, model number, brightness, or emitted color of the light source 146 based on the detection of a setting change event of the light source 146. The setting change event of the light source 146 includes an event of switching the light source 146 on and off, an event of adjusting the brightness of the light source 146, an event of adjusting the emitted color of the light source 146, and an event of changing the model number of the lighting corresponding to the light source 146. The light source setting storage unit 226 stores the changed setting of the light source 146.

[0140] D. Device Processing Procedures

[0141] 12 is a diagram showing an example of a processing procedure of system 10. In one aspect, processor 201 may execute program 210 to perform the processing of FIG. 12. At that time, program 210 is read from storage unit 203 into RAM 202. In another aspect, part or all of the processing may be realized as a combination of circuit elements configured to perform the processing. Furthermore, the steps shown in FIG. 12 may be executed in a reverse order.

[0142] In step S1205, the system 10 reads the object data 236 and the ambient light data 238 from the storage unit 203. In step S1210, the system 10 generates the virtual space 15. More specifically, the system 10 places various objects and various light sources in the virtual space 15. In step S1215, the system 10 generates the XR screen 12. More specifically, the system 10 generates the XR screen 12 by superimposing or compositing the real space and the virtual space 15. In step S1220, the system 10 displays the XR screen 12. The system 10 displays the XR screen 12 on at least the XR output device 140. Furthermore, the system 10 may further display the XR screen 12 on the output device 120.

[0143] In step S1225, the system 10 determines whether or not movement of the XR input device 130 has been detected. That is, the system 10 determines whether or not a movement event of the XR input device 130 has been detected. If the system 10 determines that movement of the XR input device 130 has been detected (YES in step S1225), the system 10 transfers control to step S1230. If not (NO in step S1225), the system 10 transfers control to step S1235.

[0144] In step S1230, the system 10 updates the position information of the XR input device 130. The position information may be a position in a three-dimensional coordinate space with the marker 142 as the origin. The system 10 also updates the position information of the laser 148. Furthermore, if the XR input device 130 is holding the light source 146, the system 10 also updates the position information of the light source 146. Furthermore, the system 10 updates the XR screen 12. When the XR screen 12 is updated, the laser 148 and the light source 146 each move on the XR screen 12 if their position information has been updated.

[0145] In step S1235, the system 10 determines whether or not a hold operation of the light source 146 has been detected. That is, the system 10 determines whether or not a hold event of the light source 146 has been detected. If the system 10 determines that a hold operation of the light source 146 has been detected (YES in step S1235), the system 10 transfers control to step S1240. Otherwise (NO in step S1235), the system 10 transfers control to step S1245. In step S1240, the system 10 causes the XR input device 130 to hold the light source 146.

[0146] In step S1245, the system 10 determines whether or not a photographing operation has been detected. That is, the system 10 determines whether or not a photographing operation event has been detected. If the system 10 determines that a photographing operation has been detected (YES in step S1245), the system 10 transfers control to step S1250. If not (NO in step S1245), the system 10 transfers control to step S1255. In step S1250, the system 10 photographs the workpiece 144 and evaluates the obtained image. More specifically, the system 10 analyzes the image using an appearance inspection program or the like. The system 10 displays the image and the image analysis results on the XR screen 12, the output device 120, or both.

[0147] In step S1255, the system 10 determines whether a release operation of the light source 146 has been detected. That is, the system 10 determines whether a release event of the light source 146 has been detected. If the system 10 determines that a release operation of the light source 146 has been detected (YES in step S1255), the system 10 transfers control to step S1260. Otherwise (NO in step S1255), the system 10 transfers control to step S1270. In step S1260, the system 10 fixes the position of the light source 146 in the virtual space 15. As a result, the position of the light source 146 is also fixed on the XR screen 12. In step S1265, the system 10 stores, in the storage unit 203, a teaching point of the robot 150 corresponding to the fixed position of the light source 146.

[0148] In step S1270, the system 10 determines whether an adjustment operation of the light source 146 has been detected. That is, the system 10 determines whether an adjustment event of the light source 146 has been detected. The adjustment operation of the light source 146 here is any one of an on / off operation, an adjustment operation of brightness, an adjustment operation of emitted color, and an operation to change the model number. If the system 10 determines that an adjustment operation of the light source 146 has been detected (YES in step S1270), the system 10 transfers control to step S1275. Otherwise (NO in step S1270), the system 10 transfers control to step S1285. In step S1275, the system 10 changes the setting of the light source 146. In step S1280, the system 10 stores the setting of the light source 146 in the storage unit 203.

[0149] In step S1285, the system 10 determines whether an abnormality has been detected. The abnormality here may include at least one of an abnormality related to the teaching point and an abnormality related to the light source. An abnormality related to the teaching point indicates that the light source 146 has moved outside the movable range of the robot 150. An abnormality related to the light source indicates that the workpiece 144 is not receiving enough light or that the light irradiating the workpiece 144 is too intense. If the system 10 determines that an abnormality has been detected (YES in step S1285), the system 10 transfers control to step S1290. If not (NO in step S1285), the system 10 transfers control to step S1225.

[0150] In step S1290, the system 10 outputs a warning. In one aspect, the system 10 may display the warning on the XR output device 140, the output device 120, or both. In another aspect, the system 10 may vibrate the XR input device 130. Furthermore, in another aspect, the system 10 may output the warning in both of the above manners.

[0151] Upon receiving an operation input for ending the teaching work, the system 10 may end the processing shown in Fig. 12. The system 10 may be configured to be able to receive an operation input for ending the teaching work from the input device 110, the XR input device 130, or both.

[0152] 12 , the device 100 can perform processing related to teaching by executing the program 210. The processing includes generating an XR screen 12 that combines the real space and the virtual space 15, displaying a workpiece in the virtual space 15 and a light source attached to the robot 150 on the XR screen 12, moving the light source on the XR screen 12 based on the detection of a light source movement event, and storing teaching points for the robot 150 that moves in conjunction with the light source 146.

[0153] The device 100 also includes an abnormality detection unit 228 that detects abnormalities within the virtual space 15. The device 100 also includes a warning unit 230 that outputs a warning based on the detection of an abnormality. The abnormality may include the light source 146 being outside the movable range of the robot 150. The abnormality may also include at least one of the following: when the light source 146 is turned on, the amount of light received by the workpiece 144 is equal to or greater than a predetermined first threshold value, or less than a predetermined second threshold value. In other words, the abnormality detection unit 228 may detect abnormalities such as insufficient light shining on the workpiece 144, or the light irradiated on the workpiece 144 being too strong.

[0154] Furthermore, the warning output by the warning unit 230 includes at least one of displaying warning information on the XR screen 12, displaying warning information on the output device 120 connected to the device 100, or vibrating the XR input device 130 to move the light source 146.

[0155] In some aspects, the device 100 may be provided as a web application, a cloud service, or a system. In this case, the input device 110, the output device 120, the XR input device 130, and the XR output device 140 are connected to a user's terminal. The device 100 is configured to be able to communicate with the user's terminal via a communication unit 207. The device 100 receives various information output by the input device 110, the XR input device 130, and the XR output device 140 from the user's terminal. The various information received by the device 100 includes information related to the user's operation input, information related to the position or movement of the XR input device 130, and captured images. Based on this information, the device 100 may update the virtual space 15 and the XR screen 12. Based on this information, the device 100 may also store teaching points and settings of the light source 146. The device 100 may also transmit the generated XR screen 12 to the user's terminal.

[0156] <E. Summary>

[0157] As described above, the system 10 according to the present embodiment has a function for intuitively teaching the robot 150. More specifically, the system 10 generates an XR screen 12 in which a real space and a virtual space 15 are superimposed or combined. The XR screen 12 displays a workpiece 144 and a light source 146 corresponding to a light attached to the tip of the robot 150. The XR screen 12 also reproduces the effect of ambient light on the workpiece 144. In other words, by using the system 10, the user can teach the robot 150 under conditions similar to the real world in terms of brightness. As a result, for example, the user can use the system 10 to teach the robot 150 without having to travel to a planned construction site for a production line to adjust the actual robot 150.

[0158] The system 10 also has a function for photographing the workpiece 144 and an image analysis function for the photographed image. By using these functions, the user can confirm whether the visual inspection of the workpiece 144 can be performed appropriately at the current teaching point.

[0159] The system 10 may also change one or more of the on / off, model number, brightness, and color of the light source 146. A user may use this feature to teach under various lighting conditions.

[0160] Furthermore, by applying XR technology to teaching the robot 150, the system 10 enables intuitive teaching work even in locations where the actual robot 150 is not available. By using the system 10, a user can teach the robot 150 under conditions that reproduce real ambient light, regardless of location. Furthermore, the intuitive user interface of the system 10 can reduce the man-hours required for teaching.

[0161] <F. Notes>

[0162] As described above, this embodiment includes the following disclosure: [Configuration 1] An XR (Cross Reality) screen generating unit (214) for generating an XR (Cross Reality) screen (12) that combines real space and virtual space (15), an XR event unit (216) for detecting an event related to the XR screen (12), a placement unit (218) for updating the position of a light source (146) attached to the tip of a robot (150) in the virtual space (15), and a teaching point storage unit (224) for storing teaching points of the robot (150), wherein the XR screen generating unit (214) displays a workpiece (144) and a light source (146) in the virtual space (15) on the XR screen (12), and the placement unit (218) moves the light source (146) on the XR screen (12) based on the detection of a movement event of the light source (146), The teaching point storage unit (224) stores teaching points for a robot (150) that moves in conjunction with a light source (146). [Configuration 2] The device (100) according to Configuration 1, wherein the XR screen generation unit (214) reads ambient light data (238) related to ambient light in the real world, and reproduces reflected light from the surface of the workpiece (144) on the XR screen (12) based on light from the ambient light in the real world and light from the light source (146). [Configuration 3] The device (100) according to Configuration 1 or 2, wherein the placement unit (218) fixes the light source (146) on the XR screen (12) based on detection of a release event for the light source (146), and storing the teaching points for the robot (150) includes storing teaching points for the robot (150) corresponding to the position and posture of the light source (146) after fixation. [Configuration 4] The device (100) according to Configuration 1 or 2, further comprising: a photographing unit (220) for photographing the workpiece (144) using a camera virtually located near the light source (146) based on the detection of a photographing event. [Configuration 5] The device (100) according to Configuration 4, wherein the XR screen generating unit (214) displays an image of the photographed workpiece (144) on the XR screen (12).[Configuration 6] The device (100) according to Configuration 4, further comprising a UI processing unit (234) that displays an image of the photographed workpiece (144) on a display of the device (100). [Configuration 7] The device (100) according to Configuration 4, further comprising an evaluation unit (232) that performs image analysis on the photographed image of the workpiece (144), and displaying the image of the workpiece (144) includes displaying the image analysis results. [Configuration 8] The device (100) according to Configuration 1 or 2, further comprising a light source adjustment unit (222) that changes at least one of the on / off, model number, brightness, or emitted color of the light source (146) based on detection of a setting change event of the light source (146), and a light source setting storage unit (226) that stores the changed settings of the light source (146). [Configuration 9] The device (100) according to Configuration 1 or 2, further comprising: an abnormality detection unit (228) that detects an abnormality in the virtual space (15); and a warning unit (230) that outputs a warning based on the detected abnormality, wherein the abnormality includes the light source (146) being outside the movable range of the robot (150). [Configuration 10] The device (100) according to Configuration 9, wherein the abnormality includes at least one of the amount of light received by the workpiece (144) when the light source (146) is turned on being equal to or greater than a predetermined first threshold or being less than a predetermined second threshold. [Configuration 11] The device (100) according to Configuration 9, wherein the warning output by the warning unit includes at least one of displaying warning information on the XR screen (12), displaying warning information on an output device (100) connected to the device (100), or vibrating the XR input device (100) to move the light source (146). [Configuration 12] The apparatus (100) according to configuration 1 or 2, further comprising a setting output unit that outputs teaching points of the robot (150).[Configuration 13] A control method for an apparatus (100), comprising: generating an XR screen (12) that merges a real space and a virtual space (15); displaying, on the XR screen (12), a workpiece (144) in the virtual space (15) and a light source (146) attached to a robot (150); moving the light source (146) on the XR screen (12) based on detection of a movement event of the light source (146); and storing teaching points for the robot (150) that moves in conjunction with the light source (146). [Configuration 14] A program (210) for causing an apparatus (100) to execute the following operations: generating an XR screen (12) that merges a real space and a virtual space (15); displaying, on the XR screen (12), a workpiece (144) in the virtual space (15) and a light source (146) attached to a robot (150); moving the light source (146) on the XR screen (12) based on detection of a movement event of the light source (146); and storing teaching points for the robot (150) that moves in conjunction with the light source (146).

[0163] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope equivalent to the claims. Furthermore, the disclosures described in the embodiments and each modification are intended to be implemented, as far as possible, either alone or in combination.

[0164] 10 System, 12, 300, 400, 500, 600, 700, 800A, 800B, 900, 1000A, 1000B, 1000C, 1100A, 1100B XR screen, 15 Virtual space, 100 Device, 110 Input device, 120 Output device, 130 XR input device, 140 XR output device, 142 Marker, 144 Work, 146 Light source, 148 Laser, 150 Robot, 152 Camera, 201 Processor, 202 RAM, 203 Memory unit, 204 Input unit, 205 Output unit, 206 External device connection unit, 207 Communication unit, 210 Program, 212 Virtual space generation unit, 214 XR screen generation unit, 216 XR event unit, 218 Placement unit, 220 Photography unit, 222 light source adjustment unit, 224 teaching point storage unit, 226 light source setting storage unit, 228 abnormality detection unit, 230 warning unit, 232 evaluation unit, 234 UI processing unit, 236 object data, 238 ambient light data, 905 image, 910 image analysis result.

Claims

1. An apparatus comprising: an XR screen generation unit for generating an XR (Cross Reality) screen that merges real space and virtual space; an XR event unit for detecting events related to the XR screen; a placement unit for updating the position of a light source attached to the tip of a robot in the virtual space; and a teaching point memory unit for storing teaching points of the robot, wherein the XR screen generation unit displays a workpiece and the light source in the virtual space on the XR screen; the placement unit moves the light source on the XR screen based on the detection of a movement event of the light source; and the teaching point memory unit stores the teaching points of the robot that moves in conjunction with the light source.

2. The device described in claim 1, wherein the XR screen generation unit reads ambient light data relating to ambient light in the real world, and reproduces the reflected light from the surface of the workpiece on the XR screen based on the light from the ambient light in the real world and the light from the light source.

3. The device according to claim 1 or 2, wherein the placement unit fixes the light source on the XR screen based on the detection of a release event of the light source, and storing the teaching points of the robot includes storing the teaching points of the robot corresponding to the position and posture of the light source after fixing.

4. The device according to claim 1 or 2, further comprising a photographing unit for photographing the workpiece using a camera virtually positioned near the light source based on the detection of a photographing event.

5. The device according to claim 4, wherein the XR screen generation unit displays an image of the photographed workpiece on the XR screen.

6. The device according to claim 4, further comprising a UI processing unit that displays the photographed image of the workpiece on a display of the device.

7. The device according to claim 4, further comprising an evaluation unit that performs image analysis on the captured image of the workpiece, and displaying the image of the workpiece includes displaying the image analysis results.

8. The device according to claim 1 or 2, further comprising: a light source adjustment unit for changing at least one of the on / off, model number, brightness, or light emission color of the light source based on the detection of a setting change event of the light source; and a light source setting storage unit for storing the changed light source settings.

9. The device according to claim 1 or 2, further comprising: an abnormality detection unit that detects abnormalities in the virtual space; and a warning unit that outputs a warning based on the detection of the abnormality, wherein the abnormality includes the light source being outside the movable range of the robot.

10. The device of claim 9, wherein the abnormality includes at least one of the amount of light received by the workpiece when the light source is turned on being equal to or greater than a predetermined first threshold value or being less than a predetermined second threshold value.

11. The device according to claim 9, wherein the warning output by the warning unit includes at least one of displaying warning information on the XR screen, displaying the warning information on an output device connected to the device, or vibrating an XR input device to move the light source.

12. The device according to claim 1 or 2, further comprising a setting output unit that outputs the teaching points of the robot.

13. A method for controlling an apparatus, comprising: generating an XR screen that merges real space and virtual space; displaying on said XR screen a workpiece in said virtual space and a light source attached to a robot; moving said light source on said XR screen based on detection of a movement event of said light source; and storing teaching points for said robot that move in conjunction with said light source.

14. A program for causing an apparatus to perform the following operations: generating an XR screen that merges real space and virtual space; displaying on said XR screen a workpiece in said virtual space and a light source attached to a robot; moving said light source on said XR screen based on detection of a movement event of said light source; and storing teaching points for said robot that moves in conjunction with said light source.

Images