Optical see-through display and contact lens

Abstract

An Optical See-Through Display (OSTD) (110) is provided. The OSTD (110) is to be worn by a user and comprises a display element (111) configured to emit Infrared (IR) light (141) representing virtual content towards an eye (150) of the user. The emitted IR light (141) has at least a first IR wavelength and is convertible to visible light (142) having a first visible wavelength. The OSTD (110) further comprises a metalens (113) arranged in an optical path between the display element (111) and the eye (150) of the user. The metalens (113) is configured to refract or reflect the emitted IR light (141) towards the eye (150) of the user. A contact lens (120) is provided which is configured to convert IR light (141) having at least a first IR wavelength into visible light (142) having a first visible wavelength.

Description

[0001] OPTICAL SEE-THROUGH DISPLAY AND CONTACT LENS

[0002] Technical field

[0003] The invention relates to an optical see-through display, a contact lens, and an augmented-reality display system.

[0004] Augmented reality (AR) is a technology which overlays computer- generated virtual content onto a user's view of the real world. AR displays are Near-Eye Displays (NED) in the form of spectacles or glasses, Head- Mounted Displays (HMD), or the like.

[0005] In one type of AR displays, known as Optical See-Through Displays (OSTD), the user can view the real world “directly” through one or more optical elements, such as a transparent display which is also used for displaying the virtual content to the user. Other solutions for OSTDs rely on combiners such as waveguides, prisms, or half-mirrors, through which the user can view the real world, and which are arranged for projecting virtual content which is displayed by a micro display or projector towards the eyes of the user. An overview of optical designs for OSTDs can be found in “Towards Indistinguishable Augmented Reality: A Survey on Optical See-through Head-mounted Displays”, Y. Itoh, T. Langlotz, J. Sutton, and A. Plopski, ACM Computing Surveys, vol. 54, pages 1-36, ACM, 2021 , doi: 10.1145 / 3453157.

[0006] Retinal Projection Displays (RPD) use optics which are designed to project light from a display directly onto the retina of the user, thereby eliminating the need for combiners such as waveguides or half-mirrors (see, e.g., “Retinal projection head-mounted display”, J. Lin, D. Cheng, C. Yao, and Y. Wang, Front. Optoelectron., vol. 10, pages 1-8, Springer, 2017, doi: 10.1007 / s12200-016-0662-8). RPDs have the advantage of being less bulky.

[0007] A disadvantage of OSTDs, including RPDs, is that the light which is emitted by displays or projectors to display virtual content to the user of an OSTD may be visible to bystanders. This applies in particular to OSTDs comprised in AR headsets which resemble the design of glasses or spectacles, which are not tightly fitted to the user’s head. Although it may not be possible to discern the virtual content which is being displayed, the visual light which is emitted by OSTDs may be perceived as disturbing by bystanders. The mere fact that virtual content is being displayed to a user may also be a privacy concern, depending on the environment in which OSTDs are used.

[0008] Although smart contact lenses with built-in displays, such as disclosed in US 2016 / 0091737 A1 , at least reduce the issue of a visible glow when virtual content is being displayed, they suffer from being relatively complex and costly, as they require the integration of electronics including display technology and power source into the contact lens, in addition to the dissipation of heat which may be harmful to the eye.

[0009] An RPD NED which is based on a combination of a light-emitting diode panel and a twisted nematic liquid-crystal panel, which is used with a contact lens, has been proposed in “Design of retinal-projection-based near-eye display with contact lens”, Y. Wu, C. P. Chen, L. Mi, W. Zhang, J. Zhao, Y. Lu, W. Guo, B. Yu, Y. Li, and N. Maitlo, Optics Express, vol. 26, pages 11553-11567, 2018, doi: 10.1364 / OE.26.011553.

[0010] Summary

[0011] It is an object of the invention to provide an improved alternative to the above techniques and prior art. More specifically, it is an object of the invention to provide improved AR NED solutions for displaying virtual content to users. In particular, it is an object of the invention to provide AR NED solutions for displaying virtual content to users which is less noticeable to bystanders.

[0012] These and other objects of the invention are achieved by means of different aspects of the invention, as defined by the independent claims. Embodiments of the invention are characterized by the dependent claims.

[0013] According to a first aspect of the invention, an OSTD to be worn by a user is provided. The OSTD comprises a display element which is configured to emit Infrared (IR) light towards an eye of the user. The emitted IR light represents virtual content. The emitted IR light has at least a first IR wavelength and is convertible to visible light having a first visible wavelength. The OSTD further comprises a metalens which is arranged in an optical path between the display element and the eye of the user. The metalens is configured to refract or reflect the emitted IR light towards the eye of the user.

[0014] According to a second aspect of the invention, a contact lens is provided. The contact lens is configured to convert IR light having at least a first IR wavelength into visible light having a first visible wavelength.

[0015] According to a third aspect of the invention, an AR display system is provided. The AR display system comprises an embodiment of the OSTD according to the first aspect of the invention, and an embodiment of the contact lens according to the second aspect of the invention.

[0016] Even though advantages of the invention have in some cases been described with reference to embodiments of the first and the second aspects of the invention, corresponding reasoning applies to embodiments of the third aspect of the invention.

[0017] Further objectives of, features of, and advantages with, the invention will become apparent when studying the following detailed disclosure, the drawings, and the appended claims. Those skilled in the art realize that different features of the invention can be combined to create embodiments other than those described in the following.

[0018] Brief description of the drawings

[0019] The above, as well as additional objects, features and advantages of the invention, will be better understood through the following illustrative and non-limiting detailed description of embodiments of the invention, with reference to the appended drawings, in which:

[0020] Fig. 1 schematically illustrates an OSTD comprising a transparent display element, and a contact lens, in accordance with embodiments of the invention.

[0021] Fig. 2 schematically illustrates an OSTD comprising a waveguide combiner, and a contact lens, in accordance with embodiments of the invention.

[0022] Fig. 3 schematically illustrates an OSTD comprising a free-space combiner, and a contact lens, in accordance with embodiments of the invention.

[0023] Fig. 4 schematically illustrates an OSTD comprising a reflective metalens, and a contact lens, in accordance with embodiments of the invention.

[0024] Fig. 5 schematically illustrates a contact lens in accordance with embodiments of the invention.

[0025] All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the invention, wherein other parts may be omitted or merely suggested. Detailed description

[0026] The invention will now be described more fully herein after with reference to the accompanying drawings, in which certain embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0027] Embodiments of the AR display system disclosed herein rely on a combination of a NED, in the form of an OSTD 110 to be worn by a user, and a contact lens 120 to be worn by the user. The OSTD 110 uses IR light 141 to display virtual content to the user, which IR light 141 is converted by the contact lens 120 into visible light 142. Thereby, virtual content can be overlaid onto a view 143 of the real world, e.g., a real-world scene 130, which the user can see directly through the OSTD 110, without the virtual content being discernable, or even noticeable, to bystanders. This is the case since the conversion from IR light 141 into visible light 142 only happens when the IR light 141 reaches the contact lens 120 which is placed onto the user’s eye or eyes 150.

[0028] In the following, embodiments of the OSTD 110, to be worn by a user, are described with reference to Figs. 1 to 4, which schematically illustrate different optical designs. Advantageously, the use of IR light 141 to display virtual context to the user, in combination with the use of a metalens 113 as optical element which is used to project the IR light 141 representing the virtual content towards the eye or eyes 150 of the user, enables less bulky designs. This is the case since metalenses, which are described in further detail below, can be tailored to exhibit relatively different optical properties, in terms of refraction and reflection, in non-overlapping but not too distant wavelengths ranges. More specifically, a metalens can be designed to refract or reflect IR light while at the same time let visible light pass unchanged, or at least substantially unchanged.

[0029] The OSTD 110 may, e.g., be comprised in AR spectacles or glasses, or in an AR headset such as a Head-Mounted Displays (HMD).

[0030] The OSTD 110 comprises a display element 111 which is configured to emit IR light 141 towards an eye or eyes 150 of the user. IR light is not visible to the human eye, with wavelengths typically above 700 nm. The emitted IR light 141 represents virtual content, e.g., for displaying information, such as one or more images or user-interface elements, as an overlay over the view 143 of the real world 130 as seen by the user through the OSTD 110. The emitted IR light 141 has at least a first IR wavelength and is convertible to visible light 142 having a first visible wavelength. Light which is visible to the human eyes is typically in the wavelength range of 380 to 700 nm. If conversion of the emitted IR light 141 into visible light 142 by the contact lens is based on Second Harmonics Generation (SHG), the wavelength of the emitted IR light 141 ,IR, is twice the wavelength of the converted visible light 142,vis, i.e., AIR= 2vis.

[0031] If a single IR wavelength is used, the virtual content is displayed in one color (i.e., monochrome). That is, the display element 111 is a monochrome display or projector. Optionally, the display element 111 may be configured to emit IR light 141 having three IR wavelengths (IR 1,IR2, andIR 3). Preferably, the three IR wavelengths are selected to approximately correspond, after conversion by the contact lens 120, to the visible wavelengths used for RGB, e.g., 460 nm (blue), 525 nm (green), and 625 nm (red). If conversion by the contact lens 120 is based on SHG, the three IR wavelengths may accordingly be selected as 920 nm, 1050 nm, and 1250 nm. Thereby, virtual content may be displayed to the user in color.

[0032] The display element 111 may be based on Light Emitting Diodes (LEDs), Organic LEDs (OLEDs), Quantum-dot LEDs (QLEDs), or other types of displays which can emit IR light. The OSTD 110 further comprises a metalens 113 which is arranged in an optical path between the display element 111 and the eye(s) 150 of the user, in particular the pupil(s) of the eye(s) 150 of the user. The metalens 113 is configured to refract or reflect the emitted IR light 141 towards the eye(s) 150 of the user. Thereby, the emitted IR light 141 is projected towards the eye(s) 150, and more specifically, the pupil(s), of the user. The IR light 141 is converted into visible light 142 when passing the contact lens 120, as is described further below, before hitting the retina(s) of the eye(s) 150.

[0033] The metalens 113 may be configured to refract or reflect light 141 at IR wavelengths which is emitted by the display element 111 towards the eye(s) 150 of the user, while at the same time letting visible light 143 pass substantially unchanged from a real-world scene 130, which is viewable by the user through the OSTD 110, towards the eye(s) 150 of the user.

[0034] The metalens 113 may be a layer, a surface, or a slab, of a metamaterial. The metalens 113 may be provided as an element which is separate from the display element 111 or the optical combiner 112. Alternatively, the metalens 113 may be provided as a coating on an outer surface of the display element 111 or the optical combiner 112, respectively.

[0035] The metalens 113 is based on a type of material of subwavelength dimensions which modulates electromagnetic waves (i.e., light) mainly through photonic resonances. One example is a metalens comprising nonlocal dielectric meta structures (see, e.g., “The advantages of metalenses over diffractive lenses”, by J. Engelberg and II. Levy, Nature Communications, vol. 11 , article num. 1991 , 2020, doi: 10.1038 / s41467-020- 15972-9, and “Multifunctional resonant wavefront-shaping meta-optics based on multilayer and multi-perturbation nonlocal metasurfaces”, by S. C. Malek, A. C. Overvig, A. Alu, and N. Yu, Light: Science & Applications, vol. 11 , article num. 246, 2022, doi: 10.1038 / s41377-022-00905-6). The specific composition and design of these metastructures enable refraction or reflection of light of certain (IR) wavelengths, in a limited wavelength range, while not (at least not substantially) affecting light at other (visible) wavelengths. If the display element 111 is configured to emit IR light at more than one wavelength, e.g., three IR wavelengths which correspond to the RGB wavelengths after conversion, the metalens 113 needs to be designed to refract or reflect the three IR wavelengths accordingly. In practice, this may be achieved by stacking three metalenses, or layers of meta structures, one for each wavelength. As an alternative, three different meta structures, one for each wavelength, may be arranged in a spatially separated manner, e.g., similar to color filters in a conventional color Liquid Crystal Display (LCD).

[0036] As an alternative to letting visible light 143 pass substantially unchanged from the real-world scene 130 towards the eye(s) 150 of the user, the metalens 113 may be further configured to refract visible light 143 passing from the real-world scene 130 towards the eye(s) 150 of the user to compensate for a vision condition of the user. More specifically, the metalens 113 can be designed to correct vision or eye-refractive errors like myopia (nearsightedness), hyperopia (farsightedness), astigmatism, and presbyopia. For example, to correct hyperopia, the metalens 113 needs to be operative to perform similar to as a convex (positive diopters, magnifying) refractive lens which converges light. For myopia, on the other hand, the metalens 113 needs to be operative to perform similar to a concave (negative diopters) refractive lens which diverges light.

[0037] In the following, different optical configurations of the OSTD 110 are described.

[0038] With reference to Fig. 1 , the display element 111 may be arranged in an optical path between the real-world scene 130, which is viewable by the user through the OSTD 110, and the eye(s) 150 of the user. In this case, the metalens 113 is configured to refract the emitted IR light 141 towards the eye(s) 150 of the user. In other words, the user views the real-world scene 130 through the display element 111 , which is transparent at least in a substantial part or parts of the visible range. This optical configuration is also referred to as retinal-projection-based NED. It is advantageous in that the display element is placed in the optical path between the eye(s) 150 of the user and the real world 130, different from the optical configurations described with reference to Figs. 2 and 3 below and may result in a less bulky design.

[0039] As an alternative, and with reference to Figs. 2 and 3, the OSTD 110 may further comprise an optical combiner 112 which is arranged in an optical path between the real-world scene 130, which viewable by the user through the OSTD 110, and the eye(s) 150 of the user. The optical combiner 112, which may be a waveguide combiner 112 (illustrated in Fig. 2) or a free- space combiner 112 (illustrated in Fig. 3, e.g., a half-mirror), is configured to let visible light 143 pass from the real-world scene 130 towards the eye(s) 150 of the user, and to project the emitted IR light 141 , by refraction and / or reflection, towards the eye(s) 150 of the user. The metalens 113 is configured to refract the emitted IR light 141 towards the eye(s) 150 of the user. In this case, the display element 111 is located outside the optical path between the eye(s) 150 of the user and the real-world scene 130 and projects the emitted IR light 141 towards the optical combiner 112.

[0040] As yet a further alternative, and with reference to Fig. 4, the metalens 113 may be configured to reflect the emitted IR light 141 towards the eye(s) 150 of the user. Similar to the optical configuration described with reference to Figs. 2 and 3, the display element 111 is located outside the optical path between the eye(s) 150 of the user and the real-world scene 130 and projects the emitted IR light 141 towards the metalens 113. Advantageously, this optical configuration does not require a separate optical combiner, such as the optical combiner 112 in Figs. 2 and 3, and may result in a less bulky design.

[0041] The OSTD 110 may further comprise an IR filter 114 which is arranged in an optical path between the display element 111 and the real-world scene 130 which is viewable by the user through the OSTD 110. The IR filter 114 blocks IR light which is emitted by the display element 111 towards the real-world scene 130. Thereby, IR light which is emitted by the display element 111 and leaks to the exterior, and in particular towards the real-world scene 130, i.e. , the space in front of the OSTD 110, is reduced. As a result, the issue of bystanders seeing the emitted IR light, even if they use upconversion glasses or contact lenses similar to the contact lens 120 disclosed herein, is mitigated or at least reduced.

[0042] The OSTD 110, the AR glasses, or the AR headset, may further comprise processing circuitry (not illustrated in the figures) which is operative to control the display element 111 to display the virtual content. In practice, the display element 111 comprises light-emitting elements, aka pixels, such as LEDs, OLEDs, or QLEDs, which are arranged in rows and columns and which can be individually controlled to emit IR light 141 of a certain intensity, and optionally of a certain color. The processing circuitry may, e.g., be operative to control the pixels of the display element 111 to display virtual content which is represented by an image or images such as a bitmap or any other type of image format. The images may represent the virtual content using various color spaces such as binary, grayscale, or color (e.g., RGB). The processing circuitry may comprise one or more processors, such as Central Processing Units (CPUs), microprocessors, application processors, application-specific processors, Graphics Processing Units (GPUs), and Digital Signal Processors (DSPs) including image processors, or a combination thereof, and a memory comprising a computer program, i.e., software, comprising instructions. When executed by the processor(s), the instructions cause the display element 111 to display the virtual content. The processing circuitry may alternatively or additionally comprise one or more Application-Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or the like, which are operative to cause the display element 111 to display the virtual content. The image or images which the processing circuitry uses to control the display element 111 to display virtual content may be generated by the processing circuitry. For instance, this may be the case if the virtual content represents a User Interface (III) or III elements which the user can use to control a configuration, function, or operation, of the OSTD 111 , the AR glasses, or the AR headset. As a further example, the processing circuitry may be operative to generate virtual content representing information such as a current environmental temperature, a current speed and / or current heading of the user, a current pulse or other biometric information of the user, and the like. Such information may be obtained from sensors such as a positioning sensor (e.g., a GPS sensor) or an Inertial Measurement Unit (IMU) comprised in the OSTD 111 , the AR glasses, or the AR headset, or from external sensors such as a fitness tracker. Alternatively, the virtual content may be received from an external source, e.g., from an AR software application or an AR service which is executed on a computing device such as a server on the Internet or in an edge cloud, or a mobile phone, a tablet, a laptop, or the like, which is connected with the OSTD 111 , the AR glasses, or the AR headset, over a wired or wireless connection.

[0043] The OSTD 110, the AR glasses, or the AR headset, may further comprise communications interface circuitry (not illustrated in the figures) which is operative to communicate, i.e. , exchange signaling and data, with other computing devices, such as access points of a cellular communications network or a Wi-Fi / WLAN network, sensor or wearables, smartphones, tablets, or the like, and / or the Internet. The communications interface circuitry may comprise one or more of: a cellular modem (e.g., GSM, UMTS, LTE, 5G, or higher generation, a WLAN / Wi-Fi modem, a Bluetooth modem, an Ethernet interface, an optical interface, or the like, for exchanging signaling and data in accordance with one or more communications protocols.

[0044] In the following, embodiments of the contact lens 120 are described with reference to Fig. 5. The contact lens 120 is configured to convert IR light 141 having at least a first IR wavelength AIRinto visible light 142 having a first visible wavelengthvis. In particular, this may be IR light 141 which is emitted by an OSTD 110 worn by the user, as described hereinbefore.

[0045] The contact lens 120 may comprise metasurfaces 121 which are configured to convert IR light 141 having at least a first IR wavelength AIRinto visible light 142 having a first visible wavelengthvis. The metasurfaces are arranged in a region of the contact lens 120 covering a pupil of an eye 150 of a user, when the contact lens 120 is worn by the user.

[0046] The metasurfaces 121 may be layered metastructures, i.e. , materials or structures with features at a subwavelength scale, or features which are smaller than the wavelength of light that interacts with them. For example, these may be nanoparticles or nanostructures like nanoantennas, such as GaAs crystalline nanoantennas (see, e.g., “Infrared upconversion imaging in nonlinear metasurfaces”, by R. Camacho-Morales et al., Advanced Photonics, vol. 3, article num. 036002, 2021 , doi:10.1117 / 1 .AP.3.3.036002).

[0047] The conversion of IR light 141 , emitted by the display element 111 of the OSTD 110, into visible light 142 is due to upconversion of photons by the contact lens 120. Photon upconversion, also referred to as Second-Harmonic Generation (SHG), is a process in which the sequential absorption of two or more photons leads to the emission of light at shorter wavelength than the excitation wavelength. An example is the conversion of IR light of wavelength AIRto visible light of wavelengthvis, with 7tIR=2vis. Upconversion can take place through different mechanisms which are known in the art.

[0048] As an alternative to upconversion or SHG, embodiments of the invention may rely on Sum-Frequency Generation (SFG), which is the conversion of light (i.e., photons) of wavelengths and A2into light of wavelength with 1 / A = 1 / ^ + 1 / 2. In this case, the OSTD 110 further comprises a light source emitting IR light of wavelength A2which is combinable with IR light 141 of wavelength emitted by the display element 111 , to visible light 142 of wavelength . In general, SFG is a less efficient process than SHG. In embodiments of the OSTD which comprise a display element 111 configured to emit IR light 141 having three IR wavelengths, a single light source emitting IR light of wavelength A2is sufficient for obtaining three visible wavelengths by means of SFG. Advantageously, the three IR wavelengths emitted by the display element 111 and the IR light of wavelengthIR 2may be chosen to result in three visible wavelengths which, after SFG conversion, approximately correspond to the RGB wavelengths. The contact lens 120 may further be configured to correct vision or eye- refractive errors like myopia (nearsightedness), hyperopia (farsightedness), astigmatism, and presbyopia.

[0049] The person skilled in the art realizes that the invention by no means is limited to the embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims.

Claims

CLAIMS1 . An optical see-through display, OSTD, (110) to be worn by a user, comprising: a display element (111 ) configured to emit Infrared, IR, light (141 ) representing virtual content towards an eye (150) of the user, the emitted IR light (141 ) having at least a first IR wavelength and being convertible to visible light (142) having a first visible wavelength, and a metalens (113) arranged in an optical path between the display element (111 ) and the eye (150) of the user, the metalens (113) being configured to refract or reflect the emitted IR light (141 ) towards the eye (150) of the user.

2. The OSTD according to claim 1 , further comprising an IR filter (114) arranged in an optical path between the display element (111 ) and a real- world scene (130) viewable by the user through the OSTD.

3. The OSTD according to claim 1 or 2, the metalens (113) being further configured to let visible light (143) pass substantially unchanged from a real- world scene (130), viewable by the user through the OSTD, towards the eye (150) of the user.

4. The OSTD according to claim 1 or 2, the metalens (113) being further configured to refract visible light (143) passing from a real-world scene (130), viewable by the user through the OSTD (110), towards the eye (150) of the user to compensate for a vision condition of the user.

5. The OSTD according to any one of claims 1 to 4, the display element (111 ) being arranged in an optical path between a real-world scene (130), viewable by the user through the OSTD, and the eye (150) ofthe user, the metalens (113) being configured to refract the emitted IR light (141 ) towards the eye (150) of the user.

6. The OSTD according to any one of claims 1 to 4, further comprising an optical combiner (112) arranged in an optical path between a real-world scene (130), viewable by the user through the OSTD, and the eye (150) of the user, the optical combiner (112) being configured to let visible light (143) pass from the real-world scene (130) towards the eye (150) of the user, and to project the emitted IR light (141 ) towards the eye (150) of the user, the metalens (113) being configured to refract the emitted IR light (141 ) towards the eye (150) of the user.

7. The OSTD according to any one of claims 1 to 4, the metalens (113) being configured to reflect the emitted IR light (141 ) towards the eye (150) of the user.

8. The OSTD according to any one of claims 1 to 7, the display element (111 ) being configured to emit IR light (141 ) having three IR wavelengths.

9. The OSTD according to any one of claims 1 to 8, wherein the wavelength(s) of the emitted IR light (141 ) is / are twice the wavelength(s) of the converted visible light (142).

10. The OSTD according to any one of claims 1 to 9, comprised in Augmented-Reality, AR, glasses or an AR headset.11 . A contact lens (120), configured to convert Infrared, IR, light (141 ) having at least a first IR wavelength into visible light (142) having a first visible wavelength.

12. The contact lens according to claim 11 , comprising metasurfaces (121 ) configured to convert IR light (141 ) having at least a first IR wavelength into visible light (142) having a first visible wavelength, the metasurfaces being arranged in a region of the contact lens (120) covering a pupil of an eye (150) of a user, when the contact lens (120) is worn by the user.

13. The contact lens according to claim 11 or 12, wherein the IR light (141 ) is emitted by an optical see-through display (110) worn by the user.

14. An Augmented-reality, AR, display system comprising the optical see-through display (110) according to any one of claims 1 to 10 and the contact lens (120) according to any one of claims 11 to 13.

Images