Augmented Reality Window

Abstract

and a controller configured to control the display device to display an image on the display screen at screen coordinates corresponding to the visible target. Apparatus 10 comprises a display device 110 including an optically transparent display screen 111 for displaying an image to a user U, a sensor 120 configured to track a gaze direction VD and / or a gaze position VP of user U, a detector 130 configured to determine a location L of a target T viewed by user U through the display screen 111, and a controller 140 configured to control display device 110 to display an image 150 on the display screen 111 at screen coordinates corresponding to the visible target T using the tracked gaze direction VD and / or tracked gaze position VP of user U and the determined location L of target T.

Description

Technical Field

[0001] The present invention relates to augmented reality.

Background Art

[0002] Generally, augmented reality (AR) relates to an interactive experience of a real-world environment, where objects within the real-world environment are augmented in real time by perceptible information, typically visual information, generated by a computer. AR is also known as mixed reality or computer-mediated reality. Thus, AR combines computer-generated perceptible information with a real-world environment.

[0003] Generally, conventional AR devices include a processor and memory, a display, a sensor for determining the orientation of the display, and optionally, a camera. For example, smartphones and tablet computers may be held (i.e., touched) by a user and used as AR devices, and live video images acquired by a camera and displayed on a display are overlaid with computer-generated graphics. The same is true for head-mounted displays (HMDs) worn (i.e., touched) by a user. AR glasses may also be worn by a user and, similarly, display computer-generated graphics on glasses displays or by projection through or reflection from the surface of glasses lenses.

[0004] These conventional AR devices are individual (i.e., for a single user). Thus, a personal AR device is required for each user to experience AR simultaneously. Alternatively, a shared AR device is required for each user to experience AR sequentially.

[0005] Thus, there is a need to improve AR devices.

Summary of the Invention

[0006] One object of the present invention is to provide, inter alia, an apparatus and a method which at least partially eliminate or mitigate at least some of the drawbacks of the prior art, whether specified herein or otherwise. For example, an object of embodiments of the present invention is to provide an apparatus and a method which provide an AR experience to a user jointly and / or contactlessly without the need for individual handheld or wearable AR devices, and which overlay a computer image for a particular tracked user on a large scale by utilizing a transparent "window" that is transparent as an overlay. This operates on the same principle as a head-mounted AR device operates, but does not require the wearing of any device or physical contact with the user in order to function. For example, an object of embodiments of the present invention is to show the state of an industrial robot cell through the projection of the operating state of the individual limbs onto a transparent safety screen surrounding it, and to provide a method of tracking the location of the observer's eyes and the location of each robot in order to achieve this method.

[0007] The first aspect provides an apparatus, the apparatus comprising a display device including an optically transparent display screen for displaying an image to a user; a sensor configured to track the user's line-of-sight direction and / or line-of-sight position; a detector configured to determine the location of a target visible to the user through the display screen; and a controller configured to control the display device to display an image on the display screen at screen coordinates corresponding to the visible target using the tracked line-of-sight direction and / or tracked line-of-sight position of the user and the determined location of the target.

[0008] The second aspect provides a method of displaying an image to a user, the method comprising tracking the user's line-of-sight direction and / or line-of-sight position; Determining the location of a target visible to a user through an optically transparent display screen, using the user's tracked line-of-sight direction and / or tracked line-of-sight position and the determined location of the target to display an image on the display screen at screen coordinates corresponding to the visible target.

Best Mode for Carrying Out the Invention

[0009] According to the present invention, an apparatus as described in the appended claims is provided. A method is also provided. Other features of the present invention will become apparent from the dependent claims and the following description.

[0010] Apparatus A first aspect provides an apparatus that includes a display device including an optically transparent display screen for displaying an image to a user, a sensor configured to track the user's line-of-sight direction and / or line-of-sight position, a detector configured to determine the location of a target visible to the user through the display screen, and a controller configured to control the display device to display an image on the display screen at screen coordinates corresponding to the visible target using the user's tracked line-of-sight direction and / or tracked line-of-sight position and the determined location of the target.

[0011] In this way, the user can optically view both the image and the target displayed on the optically transparent display screen through the optically transparent display screen, thereby providing the user with an AR experience. Since the display screen is optically transparent, the user can directly view the target through it without, for example, using a camera to image the target, thereby improving the line of sight of the target and / or the user's field of view. The user's line of sight direction and / or line of sight position is tracked, and based on the user's tracked line of sight direction and / or tracked line of sight position, the image is displayed at screen coordinates, so the screen coordinates of the image are updated or tracked accordingly when the user moves such that the user's tracked line of sight direction and / or tracked line of sight position changes. That is, the image is displayed at screen coordinates considering the relative movement between the user and the display device. Therefore, since the user's line of sight direction and / or line of sight position is tracked, the user does not need to orient the display device to experience AR. Similarly, the image is displayed at screen coordinates based on the determined location of the target, which may also change. That is, the image displayed at screen coordinates also considers the relative movement between the display device and the target. Therefore, the image is effectively superimposed and displayed in the user's field of view. For example, the display device may be a static display device installed in a manufacturing facility, a laboratory, or a hospital and controlled to display an image to a user who views the target through the optically transparent display screen. In particular, the display device is neither wearable nor handheld and does not require wearing, holding, or contact by the user. Therefore, a personal AR device does not require each user to experience AR simultaneously, and sharing of the shared device does not require each user to experience AR continuously. That is, the number of AR devices is reduced, the need to wear or hold an AR device is eliminated, while sharing of a single AR device, the need for contact, and potential cross-contamination are avoided.

[0012] Smartphones and tablets do not have an optically transparent display screen as will be described below, as the target cannot be seen through them by the user. Instead, live video images acquired by a camera and displayed on the display of the smartphone and tablet are overlaid with computer-generated graphics. In one example, the display device is not a smartphone or a tablet.

[0013] It should be understood that HMDs and AR glasses may have an optically transparent display screen. However, such wearable AR devices typically determine the line-of-sight direction and / or line-of-sight position of the AR device, rather than the user, as will be described below.

[0014] The apparatus comprises a display device, a sensor, a detector, and a controller. It should be understood that the controller is communicatively coupled to the display device, the sensor, and the detector, as needed, in one or two directions, wired and / or wirelessly. In one example, the display device, the sensor, the detector, and / or the controller are physically separate, for example, in separate housings and may be spaced apart. In one example, the display device, the sensor, the detector, and / or the controller are integrated, for example, within the same housing.

[0015] Display device The apparatus comprises a display device including an optically transparent display screen for displaying an image to the user.

[0016] In one example, the display screen comprises and / or is a see-through display. A see-through display is a term in the art and is an electronic display that enables a user to see what is shown on an optically transparent display screen while still being able to see through it. See-through displays are known and include LCD and LED (including both inorganic and organic LEDs such as transparent organic light-emitting devices (TOLED)) display screens. Passive transmissive displays are also known. Other see-through displays are known. Such see-through displays can have a transmittance of about 70%.

[0017] In one example, the display screen comprises and / or is a projection screen (e.g., a front projection screen and / or a rear projection screen), and the display device includes a projector configured to project an image onto the display screen.

[0018] In one example, the projection screen comprises and / or is a semi-transparent projection screen having a rear or front projection film or coating, or a light diffusing screen. Such projection screens are typically used for holographic or "pepper's ghost" projections. Such projection screens can have a transmittance of 80% or more. Suitable projectors are known.

[0019] In one example, the projection screen comprises and / or is a beam splitter. In this way, a head-up display (HUD) can be provided. A typical HUD includes three main components, namely, a projector unit, a combiner, and a video generation computer. The projection unit in a typical HUD is an optical collimator setup, i.e., a convex lens or concave mirror having a cathode ray tube, a light-emitting diode display, or a liquid crystal display at its focus, thereby displaying an image where the light is collimated, i.e., the focus is perceived to be at infinity. The combiner is typically an angled flat piece of glass (beam splitter) positioned in front of the user, changing the orientation of the projected image from the projector so that the user can see the field of view and the projected infinity image simultaneously. The combiner can have a coating that reflects monochromatic light projected onto the combiner from the projector unit and passes light of all other wavelengths. In some optical layouts, the combiner may also have a curved surface to refocus the image from the projector.

[0020] In one example, the display screen has dimensions, e.g., diagonal dimensions, in the range from 0.5 m to 25 m, preferably in the range from 1 m to 10 m, more preferably in the range from 2 m to 5 m. That is, the display screen is relatively large compared to the display screen of a handheld or wearable AR device.

[0021] In one example, the display screen is adapted to provide a safety screen disposed between the user and the target. In this way, the display screen isolates the user from the target by providing a barrier therebetween, preventing the user from contacting the target and / or vice versa.

[0022] In one example, the display screen at least partially surrounds the target. For example, the display screen may surround the target on one or more sides, such as surrounding the target.

[0023] In one example, the display device comprises a stereoscopic display device (also known as a 3D display device), and / or is a stereoscopic display device, such as a stereoscopic display device that requires glasses or an autostereoscopic display that does not require glasses, as understood by those skilled in the art. In this way, the image may be displayed at a predetermined depth and / or a depth corresponding to the target in view, such as a similar depth. In this way, since the user can focus on the target and the image, for example, simultaneously, it becomes easier for the user to view the display.

[0024] Sensor The device comprises a sensor configured to track the user's line of sight direction and / or line of sight position. It should be understood that the sensor is configured to track the user's line of sight direction and / or line of sight position, for example, the user's fixation or line of sight. In contrast, handheld and wearable AR devices are configured to track the line of sight direction and / or line of sight position of these AR devices. That is, a user of a handheld AR device may be pointing the handheld AR device in a particular direction but looking in a different direction. Similarly, a user of a wearable AR device may be orienting the wearable AR device in a particular direction but gazing in a different direction. Accordingly, the device according to the first aspect responds to the user's current line of sight direction and / or current line of sight position. In this way, the image displayed on the display screen is generated in response to and / or in accordance with where the user is looking, for example, in response to screen coordinates. In one example, the sensor is configured to track the user's fixation, direct vision, central vision, eccentric fixation, macular vision, and / or line of sight. In one example, the sensor is configured to track the user's indirect vision (also known as peripheral vision), peripheral vision, mid-peripheral vision, and / or far-peripheral vision.

[0025] It should be understood that the sensor is configured to track the user's line of sight direction and / or line of sight position, for example, currently intermittently or continuously. In this way, the screen coordinates of the image and / or the image may be updated according to the tracked line of sight direction and / or the tracked line of sight position so as to follow where the user is looking and / or the user's movement. In one example, the sensor is configured to track the user's line of sight direction and / or line of sight position by sensing the position of the user's eyes and / or pupils, and / or the respective positions of the user's eyes and / or pupils. For example, it uses face capture, gaze direction detection, and / or eye location software applied to the user's image, such as a camera or video image. Such software is known. An eye location that requires a relatively simple camera and has relatively low computer resource requirements suitable for real-time processing while allowing a relatively high degree of freedom of movement of the user is preferred. In contrast, face capture and / or gaze direction detection may require a relatively complex camera, appropriate lighting, and relatively high computer resource requirements.

[0026] It should be understood that the sensor is configured to track the user's line of sight direction and / or line of sight position with a resolution, accuracy, and / or precision sufficient to calculate the screen coordinates of the image, as described below.

[0027] In one example, the sensor comprises, and / or is, a camera, an accelerometer, a gyroscope, a solid state compass, a radio frequency identification (RFID) detector, and / or a position sensor, such as a global positioning system (GPS). Preferably, the sensor comprises, and / or is, a camera and appropriate software. Suitable sensors include Microsoft® Azure® Kinect®, VicoVR® available from https: / / vicovr.com / , Orbbec Astra® available from https: / / orbbec3d.com / , Stereolabs ZED® available from https: / / www.stereolabs.com / , and OpenPose® available from https: / / github.com / CMU-Perceptual-Computing-Lab / openpose. Other sensors are also known.

[0028] In one example, the sensor is configured to track the respective line-of-sight directions and / or the respective line-of-sight positions of a plurality of users.

[0029] Detector The apparatus comprises a detector configured to determine the location of a target visible to a user through a display screen.

[0030] It should be understood that the target is visible to the user through the display screen since the display screen is optically transparent. That is, the target is not the image displayed on the display screen, but rather an object on the opposite side of the display screen with respect to the user. The location of the target may include a two - dimensional or three - dimensional location that is absolute or relative with respect to the display screen and / or the user, such as a geographical location (optionally including altitude), or a location with respect to a coordinate system, and / or it should be understood that it is a two - dimensional or three - dimensional location. The detector is configured to determine the location of the target with a resolution, accuracy, and / or precision sufficient to calculate the screen coordinates of the image, as described below.

[0031] The detector may be as described with respect to the sensor.

[0032] In one example, the detector is configured to track the location of the target (and / or more of its parts) with the necessary modifications as described with respect to tracking the user's line - of - sight direction and / or line - of - sight position by a sensor. Thus, the device according to the first aspect can respond to the tracked location of the target. In this way, the image displayed on the display screen may respond, for example, in screen coordinates, and / or may be generated according to the current location of the target.

[0033] In one example, the detector comprises and / or is a camera, an accelerometer, a gyroscope, a solid - state compass, a radio - frequency identification (RFID) detector, an ultra - wideband (UWB) sensor, and / or a position sensor, such as a global positioning system (GPS).

[0034] In one example, the detector can be communicatively coupled to the target, for example, in one or two directions, wired or wirelessly, and the detector is configured to determine the location of the target (and / or one or more parts of the target) based on the position data received from the target. That is, the detector may comprise a software sensor (also known as a soft sensor or full virtual sensor) configured to read, infer, and / or calculate the location of the target (and / or one or more parts thereof) using the position data received therefrom. For example, if the target is an industrial robot or a part thereof, the industrial robot can output digital readout (DRO) data including the location of the axes.

[0035] In one example, the detector is configured to determine the respective location of a plurality of targets (and / or one or more parts thereof) visible to the user through a display screen. Preferably, the location is communicated from the respective controller of the target (or targets) to the detector. In other words, a real-time direct data feed from the controller of the target regarding the location of the target (and / or one or more parts thereof) is communicated to the detector. In this way, the present invention advantageously continuously monitors the complex positioning and movement of the target (and / or one or more parts thereof) behind the display device. Preferably, the real-time streaming data directly from the controller of the target is utilized to confirm and continuously monitor such location. This functionality is particularly advantageous for complex targets having multiple components and / or multiple targets.

[0036] Target In one example, the target includes an industrial machine, an agricultural machine, or a medical device, and / or one or more parts thereof, and / or is one of them. Examples of industrial machines include manufacturing, handling, and assembly lines for engineering and construction, as well as chemical process plants. Examples of agricultural machines include agricultural equipment and processing. Examples of medical devices include diagnostic, surgical, and therapeutic devices. In one example, the target is an industrial robot and / or one or more of its components, such as its axis (or axes). In one example, the target includes, and / or is, a landing craft, a ship, and / or an aircraft for road or air traffic management at, for example, a bus stop, a railway station, or an airport. For example, a passenger (i.e., a user) can view a bus (i.e., the target) through a display device at a bus stop, which provides a barrier between them, and the display device can display the relevant bus timetable (i.e., an image) thereon. In one example, the target includes, and / or is, an animal, including a human, for, for example, security surveillance, sports, or hospitality. For example, a server (i.e., a user) can view a table and diners (i.e., the target) through a display device in a restaurant, which provides a barrier between them, and the display device can display the relevant order (i.e., an image) on the display device. Thereby, the user can view the target T (T1 to T5) together with the displayed image through the display screen.

[0037] In one example, the target includes an industrial robot or a part thereof, and / or is an industrial robot or a part thereof. In a preferred example, the target is a robot.

[0038] In one example, the device includes the target.

[0039] Preferably, the target includes one or more of its parts, and more preferably, the target includes a plurality of its parts. Preferably, the target is a multi-axis robot.

[0040] Controller The apparatus comprises a controller configured to control a display device to display an image on a display screen at screen coordinates corresponding to a visible target using the tracked line-of-sight direction and / or tracked line-of-sight position of the user and the determined location of the target. It should be understood that the controller is configured to calculate screen coordinates using the tracked line-of-sight direction and / or tracked line-of-sight position of the user and the location of the target determined, for example, by a geometric configuration. Such geometric structures are known. Other methods of calculating screen coordinates are known. In one example, the controller is configured to control the display device to repeatedly, for example, intermittently, periodically, or continuously calculate screen coordinates. In this way, the screen coordinates of the image may be updated according to the relative movement of the user and / or the target with respect to the display screen such that the image is displayed on the display screen in a substantially fixed arrangement with respect to the target and the user. That is, as the user and / or the target move, the image is displayed at updated screen coordinates that account for this movement. The image displayed at the screen coordinates corresponding to the visible target is understood to be, for example, overlaid on the visible target, adjacent to the visible target, or displayed at predetermined screen coordinates and viewable, for example, by a user having the tracked line-of-sight direction and / or tracked line-of-sight position.

[0041] In one example, the controller is configured to control a display device to display a plurality of images on a display screen at respective screen coordinates corresponding to a plurality of visible targets and / or one or more portions thereof, using the tracked line-of-sight direction and / or tracked line-of-sight position of the user, and the determined location of each of the plurality of targets (and / or one or more portions thereof).

[0042] In one example, the controller comprises a processor and a memory, such as a computer.

[0043] Image In general, it should be understood that an image is associated with a target, such as computer-generated visual information about the target. In one example, an image includes graphics such as text and / or icons, shapes, pictures, models, etc., and / or is text and / or graphics. In one example, the image is a static graphic. In one example, the image is a dynamic graphic.

[0044] Identifier In one example, the device comprises an identifier configured to establish the identity of the user, and the controller is configured to control the display device to display an image based on the established identity. In this way, the display of the image may be conditional (i.e., displayed or not displayed) according to the established identity, and / or the computer-generated visual information about the target may be specific to the user or a group of that user. Therefore, it should be understood that the identifier is a sensing element configured to identify the user. For example, the identifier can establish the identification of the user using face recognition from an image of the user acquired by a camera. For example, the identifier can establish the identification of the user by reading an RFID tag of the user. In general, the identifier may be as described for the sensor with necessary modifications.

[0045] User input device In one example, the apparatus includes a user input device configured to receive input from a user. In this way, the user can interact with the apparatus.

[0046] In one example, the user input device includes a gesture sensor, and / or the display screen includes a touch screen. In this way, the user can interact with the apparatus using gestures (i.e., non-contact) and / or via touch (i.e., contact or near contact).

[0047] In one example, the controller is communicably coupleable to a target, and the controller is configured to control the target according to input received from the user. In this way, the user can remotely control the target via the apparatus, for example, by gestures and / or touch. For example, the user can view the target through the display screen along with an image displayed thereon, and while controlling the target, the user can also view the target through the display screen along with the image displayed thereon.

[0048] Method A second aspect provides a method of displaying an image to a user, the method including tracking the user's line of sight direction and / or line of sight position, determining the location of a target visible to the user through an optically transparent display screen, and using the tracked line of sight direction and / or tracked line of sight position of the user and the determined location of the target to display an image on the display screen at screen coordinates corresponding to the visible target.

[0049] Tracking, line-of-sight direction, line-of-sight position, user, determination, location, target, optically transparent display screen, image and / or screen coordinates may be as described with respect to the first aspect.

[0050] Definition Throughout this specification, the terms "comprising" or "comprises" mean including the specified component but not excluding the presence of other components. The terms "consisting essentially of" or "consists essentially of" mean including the specified component but excluding other components except for materials present as impurities, inevitable materials resulting from the processes used to provide the components, and components added for purposes other than achieving the technical effects of the present invention such as colorants, etc.

[0051] The terms "consisting of" or "consists of" mean including the specified component but excluding other components.

[0052] Whenever appropriate and depending on the context, the use of the terms "comprises" or "comprising" may be interpreted as including the meaning of "consists essentially of" or "consisting essentially of", and may also be interpreted as including the meaning of "consists of" or "consisting of".

[0053] Any optional features described in this specification may, where appropriate, be used either individually or in combination with each other, in any of the combinations, particularly as described in the appended claims. Optional features for each aspect or exemplary embodiment of the invention as described herein are, where appropriate, applicable to all other aspects or exemplary embodiments of the invention. In other words, those skilled in the art reading this specification should consider the optional features for each aspect or exemplary embodiment of the invention as interchangeable and combinable between different aspects and exemplary embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] For a better understanding of the present invention and to show how exemplary embodiments of the present invention are put into practice, reference is made, by way of example, to the accompanying drawings.

Figure 1

Figure 2

Figure 3A

Figure 3B

[0055] FIG. 1 schematically depicts an apparatus 10 according to an exemplary embodiment.

[0056] The apparatus 10 comprises a display device 110 including an optically transparent display screen 111 for displaying an image to a user U, a sensor 120 configured to track the line-of-sight direction VD and / or the line-of-sight position VP of the user U, a detector 130 configured to determine the location L of a target T that can be seen by the user U through the display screen 111, A controller 140 configured to control a display device 110 to display an image 150 on a display screen 111 at screen coordinates corresponding to a visible target T using the tracked line-of-sight direction VD and / or tracked line-of-sight position VP of a user U and the determined location L of the target T.

[0057] FIG. 2 schematically depicts a method according to an exemplary embodiment.

[0058] This method is for displaying an image to a user.

[0059] In S201, the method includes tracking the line-of-sight direction and / or line-of-sight position of the user.

[0060] In S202, the method includes determining the location of a target visible to the user through an optically transparent display screen.

[0061] In S203, the method includes using the tracked line-of-sight direction and / or tracked line-of-sight position of the user and the determined location of the target to display an image on the display screen at screen coordinates corresponding to the visible target.

[0062] FIG. 3A schematically depicts a plan perspective view of an apparatus 20 according to an exemplary embodiment, and FIG. 3B schematically depicts a front perspective view of the apparatus 20.

[0063] The apparatus 20 includes a display device 210 including an optically transparent display screen 211 for displaying an image to a user U, a sensor 220 configured to track the line-of-sight direction VD and / or line-of-sight position VP of the user U, A detector 230 (not shown) configured to determine the locations L (L1 to L5) of targets T (T1 to T5) viewable by a user U through a display screen 211. A controller 240 (not shown) configured to control a display device 210 to display images 250 (250A to 250H) on the display screen 211 at screen coordinates corresponding to the visible targets T (T1 to T5) using the tracked line-of-sight direction VD and / or the tracked line-of-sight position VP of the user U and the determined locations L (L1 to L5) of the targets T (T1 to T5).

[0064] Briefly described, FIGS. 3A and 3B show in large scale an overlaid computer image of a particular tracked user by utilizing a transparent "window" for the overlay. This operates on the same AR principle as a head-mounted AR device but does not require any device wearing or physical contact by the user to function. In this example, the image shows the state of an industrial robot cell by tracking the location of the user's eyes and the location of each robot while projecting the operating state of the individual limbs (axes) onto a transparent safety screen surrounding it.

[0065] In this example, the display device 210, the sensor 220, the detector 240, and / or the controller 250 are physically separate and, for example, are in separate housings and are spaced apart.

[0066] In this example, the display screen 211 is a projection screen 211, particularly a rear projection screen 211, and the display device 210 includes a projector 212 configured to project images 250 (from 250A to 250H) onto the display screen 211. In this example, the projection screen 211 includes and / or is a translucent projection screen 211 having a rear projection film. In this example, the display screen 211 has a diagonal dimension of 1.5 m. In this example, the display screen 211 is adapted to provide a safety screen 211 disposed between the user U and the targets T (from T1 to T5).

[0067] In this example, the display screen 211 at least partially surrounds the targets T (from T1 to T5).

[0068] In this example, the sensor 220 is configured to track the line-of-sight direction VD and / or the line-of-sight position VP of the user U by sensing the position of the eyes and / or pupils of the user U and / or the respective positions of the eyes and / or pupils of the user U using eye location L (from L1 to L5) software applied to an image of the user U, such as a camera or video image. In this example, the sensor 220 includes a camera and appropriate software. In this example, the sensor 220 is configured to track the respective line-of-sight directions VDs and / or the respective line-of-sight positions VPs of a plurality of users.

[0069] In this example, the detector 240 is communicatively coupled bidirectionally and wired to the targets T (T1 to T5), and the detector 240 is configured to determine the locations L (L1 to L5) of the targets T (T1 to T5) based on the position data received from the targets T (T1 to T5). In this example, the targets T (T1 to T5) are an industrial robot or a part thereof, and the industrial robot can output digital readout (DRO) data including the locations L (L1 to L5) of the axes. In this example, the detector 240 is configured to determine the respective locations of the plurality of targets T1 to T5 that can be viewed by the user U through the display screen 211. In this example, the targets T (T1 to T5) include and / or are an industrial robot or a part thereof.

[0070] In this example, the controller 250 is configured to control the display device 210 to repeatedly calculate the screen coordinates, for example, intermittently, periodically, or continuously. In this example, the controller 250 uses the tracked line-of-sight direction VD and / or the tracked line-of-sight position VP of the user U and the respective determined locations L (L1 to L5) of the plurality of targets T (T1 to T5) to control the display device 210 to display a plurality of images 250 (250A to 250H) at the respective screen 211 coordinates corresponding to the plurality of visible targets T (T1 to T5). In this example, the controller 250 comprises a processor and a memory, for example a computer.

[0071] In general, it should be understood that the images 250 (250A to 250H) are associated with computer-generated visual information regarding the targets T (T1 to T5), for example the targets T (T1 to T5). In this example, the images 250 (250A to 250H) include and / or are text and / or graphics.

[0072] Preferred embodiments have been shown and described, but it will be recognized by those skilled in the art that various changes and modifications may be made without departing from the scope of the invention as defined in the appended claims.

[0073] Attention is directed to all documents and writings that are filed simultaneously with or before this specification in connection with this application and are open to public inspection with this specification, and all contents of such documents and writings are incorporated herein by reference.

[0074] All features disclosed in this specification (including any appended claims and drawings), and / or all steps of any method or process thus disclosed, may be combined in any combination, except combinations in which at most some of such features and / or steps are mutually exclusive.

[0075] Each feature disclosed in this specification (including any appended claims and drawings) may, unless otherwise specified, be replaced by an alternative feature serving the same, equivalent, or similar purpose. From this, each disclosed feature is, unless otherwise specified, merely an example of a general series of equivalent or similar features.

[0076] The invention is not limited to the details of the foregoing embodiments. The invention extends to any novel one or any novel combination of features disclosed in this specification (including any appended claims and drawings), or to any novel one or any novel combination of steps of any method or process thus disclosed. The invention described in the claims of the present application at the time of filing is appended below. [C1] An apparatus comprising: A display device including an optically transparent display screen for displaying an image to a user; A sensor configured to track the line-of-sight direction and / or line-of-sight position of the user, and a detector configured to determine the location of a target visible to the user through the display screen; A controller configured to control the display device to display an image on the display screen at screen coordinates corresponding to the visible target using the tracked line-of-sight direction and / or tracked line-of-sight position of the user and the determined location of the target. [C2] The apparatus according to C1, wherein the controller is configured to control the display device to display a plurality of images on the display screen at respective screen coordinates corresponding to a plurality of visible targets using the tracked line-of-sight direction and / or tracked line-of-sight position of the user and the determined location of each of the plurality of targets. [C3] The apparatus according to C1 or 2, wherein the display screen includes and / or is a projection screen, and the display device includes a projector configured to project the image onto the display screen. [C4] The apparatus according to any one of C1 to 3, wherein the display screen is adapted to provide a safety screen disposed between the user and the target. [C5] The apparatus according to C4, wherein the display screen at least partially surrounds the target. [C6] The apparatus according to any one of C1 to 5, wherein the sensor includes and / or is a camera, an accelerometer, a gyroscope, a solid-state compass, a radio frequency identification detector, and / or a position sensor. [C7] The apparatus according to any one of C1 to 6, wherein the detector includes and / or is a camera, an accelerometer, a gyroscope, a solid-state compass, a radio frequency identification detector, an ultra-wideband (UWB) sensor, and / or a position sensor. [C8] The detector is communicably couplable to the target, and the detector is configured to determine the location of the target based on position data received from the target, the apparatus according to any one of C1 to 7. [C9] The apparatus according to any one of C1 to 8, comprising an identifier configured to establish the identity of the user, and the controller is configured to control the display device to display the image based on the established identity. [C10] The apparatus according to any one of C1 to 9, comprising a user input device configured to receive input from the user. [C11] The apparatus according to C10, wherein the user input device comprises a gesture sensor and / or the display screen comprises a touch screen. [C12] The apparatus according to C10 or 11, wherein the controller is communicably couplable to the target, and the controller is configured to control the target according to the input received from the user. [C13] The apparatus according to any one of C1 to 12, wherein the apparatus includes the target. [C14] The apparatus according to C13, wherein the target includes and / or is an industrial robot. [C15] A method of displaying an image to a user, the method comprising: tracking the line-of-sight direction and / or line-of-sight position of the user; determining the location of a target visible to the user through an optically transparent display screen; and using the tracked line-of-sight direction and / or the tracked line-of-sight position of the user and the determined location of the target to display an image on the display screen at screen coordinates corresponding to the visible target.

Claims

1. An apparatus comprising: a display device including an optically transparent display screen for simultaneously displaying an image to a plurality of users; a sensor configured to track a line-of-sight direction and / or a line-of-sight position of at least one user of the plurality of users; a detector configured to determine a location of a target visible to at least one user of the plurality of users through the display screen; a controller configured to control the display device to display an image on the display screen at screen coordinates corresponding to the visible target using the tracked line-of-sight direction and / or the tracked line-of-sight position of at least one user of the plurality of users and the determined location of the target; an identifier configured to establish an identity of at least one user of the plurality of users; wherein the controller is configured to control the display device to display the image specific to at least one user of the plurality of users based on the established identity, the target is an industrial robot, and the image displays an operating state of an individual limb of the industrial robot. The apparatus.

2. The apparatus according to claim 1, wherein the controller is configured to control the display device to display a plurality of images on the display screen at respective screen coordinates corresponding to a plurality of visible targets using the tracked line-of-sight direction and / or the tracked line-of-sight position of at least one user of the plurality of users and the determined location of each of the plurality of targets.

3. The apparatus according to claim 1 or 2, wherein the display screen includes and / or is a projection screen, and the display device includes a projector configured to project the image onto the display screen.

4. The apparatus according to any one of claims 1 to 3, wherein the display screen is adapted to provide a safety screen disposed between at least one user of the plurality of users and the target.

5. The display screen surrounds the target at least partially, the apparatus according to claim 4.

6. The sensor includes a camera, an accelerometer, a gyroscope, a solid state compass, a radio frequency identification detector, and / or a position sensor, and / or is these, the apparatus according to any one of claims 1 to 5.

7. The detector includes a camera, an accelerometer, a gyroscope, a solid state compass, a radio frequency identification detector, an ultra-wideband (UWB) sensor, and / or a position sensor, and / or is these, the apparatus according to any one of claims 1 to 6.

8. The detector is communicably coupled to the target, and the detector is configured to determine the location of the target based on position data received from the target, the apparatus according to any one of claims 1 to 7.

9. The apparatus according to any one of claims 1 to 8, comprising a user input device configured to receive an input from at least one of the plurality of users.

10. The user input device includes a gesture sensor, and / or the display screen includes a touch screen, the apparatus according to claim 9.

11. The controller is communicably coupled to the target, and the controller is configured to control the target according to the input received from at least one of the plurality of users, the apparatus according to claim 9 or 10.

12. The apparatus includes the target, the apparatus according to any one of claims 1 to 11.

13. A method of simultaneously displaying an image to a plurality of users, the method comprising: tracking the line-of-sight direction and / or line-of-sight position of at least one of the plurality of users; determining the location of a target visible to at least one of the plurality of users through an optically transparent display screen; using the tracked line-of-sight direction and / or the tracked line-of-sight position of at least one of the plurality of users and the determined location of the target to display an image on the display screen at screen coordinates corresponding to the visible target; Establishing the identity of at least one of the plurality of users, wherein the displaying is configured to control the display screen to display the image specific to at least one of the plurality of users based on the established identity, the target is an industrial robot, and the image displays the operating state of the individual limbs of the industrial robot, the method comprising.

Images