Near-eye lightfield projection system with active foveal adaptation

Abstract

To enable foveal projection in a near-eye light field projection system. The near-eye light field projection system includes a light source (10) that generates an incident light beam (100) and a spatial light modulator (20) that projects a plurality of modulated light beams (110). A first optical element (70) forms a plurality of light source images (28) at a light source image plane (30). A second optical element (32) forms a modulator image (114) at a modulator image plane (115). A third optical element (40) provides a first field of view (FOV) (80) and projects a plurality of projection viewpoints (112) onto an exit plane (125). An active deflection element (60) deflects the modulated light beam (110) based on the orientation of the pupil of a user's eye to provide a plurality of deflected and modulated light beams (110a, 110b). The modulator image (114) is spatially shifted in the modulator image plane (115) to form a second FOV (81a, 81b) larger than the first FOV (80) in the direction of the pupil orientation, and the projection viewpoint (112) forms second exit pupils (121a, 121b) in the exit plane (125).

Description

[Technical Field]

[0001] The present disclosure relates to a near-eye lightfield projection system that sequentially projects near-eye projection images onto a user's eyes. More specifically, the present disclosure relates to a near-eye lightfield projection system adapted for virtual, augmented, or mixed reality glasses applications. Furthermore, the present disclosure relates to a wearable device, such as augmented or mixed reality, smart glasses, that includes a near-eye lightfield projection system. [Background technology]

[0002] The human eye has a very wide field of view (FOV). Individually, the human eye has a horizontal FOV of approximately 135° and a vertical FOV of just over 180°. The FOV allows for coverage rather than a single focal point. In virtual reality (VR) and / or mixed reality devices, a wide FOV is essential for an immersive, lifelike experience. A wider FOV also allows for better sensor coverage and accessibility to many other optical devices.

[0003] In practice, a virtual or mixed reality device needs to be able to encompass the FOV of the human eye, with approximately 400,000,000 pixels, with pixels regularly distributed to meet the eye's maximum resolution.

[0004] However, the eye's resolution is not evenly distributed. The eye's resolution is high only in about a 20° field of view around the fovea. A full HD display (1920 x 1080), which encompasses a 20° field of view, already reaches the retinal resolution at the fovea. The eye's resolution gradually decreases the further away from the fovea. The entire field of view (outside the fovea) can be encompassed with roughly the same amount of information within the fovea, requiring a total of about 4 million pixels.

[0005] To exploit exactly this feature of human vision, so-called foveated rendering and foveated projection have been introduced in virtual reality, mixed reality, and augmented reality headsets. However, this is still done using flat images. Light field devices do not yet have a solution for foveated projection or active foveated projection.

[0006] The applicant's patent application JP2007-102093A discloses a light field projector that projects a virtual image with optimized monocular depth cues into a user's eyes, and an augmented reality device including the light field projector. The near-eye light field projector disclosed in this document generates a light field with realistic monocular depth perception, providing the viewer with a realistic finite depth of field and correct accommodation perception in an artificially generated three-dimensional (3D) scene.

[0007] Patent Document 2, filed by the same applicant, discloses a near-eye light-field virtual reality and mixed reality system with central vision projection. This system provides virtual reality and mixed reality experiences for the eyes of a human, animal, or camera, allowing users to enjoy a realistic mix of real 3D scenes and virtual 3D scenes. This system can provide 3D virtual reality and augmented reality information with the comfort of proper eye accommodation.

[0008] FIG. 1a illustrates a light field projection system described in U.S. Patent Application Publication No. 2009 / 0229903. The light field projection system includes a light source array 10 generating multiple incident light beams 100 that illuminate a spatial light modulator (SLM) 20. The SLM 20 is configured to modulate the incident light field 100 and project multiple modulated light field components 110 along a projection axis 170. The light field projection system further includes a first optical element 70 configured to interact with the modulated light beams 110 to form multiple light source images 28 at a light source image plane 30. A second optical element 32 is configured to interact with the modulated light beams 110 to form a modulator image 114 at a modulator image plane 115. A third optical element 40 is configured to interact with the modulated light beams 110 to project multiple projection viewpoints 112 that form an exit pupil 120 at an exit plane 125. An optical device 60 disposed at the light source image plane 30 is configured to deflect one or some of the modulated light field components 110. This causes the modulator image 114 formed by the deflected component or portion of the modulated light field component 110 to be spatially displaced (shifted) at the modulator image plane 115. The deflected component or portion of the modulated light field component 110 can increase the FOV at the exit pupil 120. Figures 1b and 1c show examples of image tiling due to the increased field of view seen by a viewer when their eyes are focused at infinity (Figure 1b) and closer than infinity (Figure 1c) using the light field projection system disclosed in U.S. Patent No. 6,229,499.

[0009] There are several disadvantages to deflecting only one or a portion of the modulated light field component 110. For example, deflecting only one or a portion of the modulated light field component 110 reduces or eliminates the light field effect. The apparent brightness of the expanded FOV is reduced compared to the remainder of the original FOV (corresponding to the undeflected modulated light field component 110) because fewer viewpoints contribute to the total brightness. If the expanded FOV is large, the exit pupil 120 may be too small, so that the user's eyes may be outside the exit pupil 120 when they rotate to view the expanded FOV. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] European Patent No. 3542206 [Patent Document 2] International Publication No. 2021 / 090107 Summary of the Invention

[0011] The present disclosure relates to a near-eye light field projection system comprising a light source array generating a plurality of incident light beams that illuminate an SLM, and an SLM configured to modulate the incident light beams and project a plurality of modulated light beams. A first optical element interacts with the modulated light beams to form a plurality of light source images at a light source image plane. A second optical element interacts with the modulated light beams to form a modulator image at a modulator image plane. A third optical element interacts with the modulated light beams to project a plurality of projected image components at an exit plane that provide a first FOV and define a first exit pupil. The projection system further comprises at least one active deflection element between the first and third optical elements. The active deflection element deflects the modulated light beams based on an orientation of the pupil of a user's eye to provide a plurality of deflected modulated light beams that spatially shift the modulator images at the modulator image plane. The deflected modulated light beam forms a second field of view larger than the first field of view in the pupil orientation, and the projected image components form a second exit pupil at the exit plane. [Effects of the Invention]

[0012] With respect to the prior art, the disclosed near-eye light field projection system deflects all modulated light rays projected by the SLM in such a way that the apparent brightness of the expanded second FOV is not reduced compared to the brightness of the first FOV.

[0013] The steerable, high-resolution FOV reduces the computational power and size of the projection system, while allowing users to perceive only the high-resolution light field image, providing a seamless experience.

[0014] Several exemplary embodiments of the invention are described in the following description and illustrated in the drawings. [Brief explanation of the drawings]

[0015] [Figure 1a]FIG. 1a shows a light field projection system comprising a light modulator for projecting a modulated light field component and an optical device adapted to deflect one or some of the modulated light field components. [Figure 1b] FIG. 1b shows the projected image of the light modulator as seen by a viewer when their eyes are focused at infinity. [Figure 1c] FIG. 1c shows the projected image of the light modulator as seen by a viewer when focused closer than infinity. [Figure 2] FIG. 2 illustrates a near-eye lightfield projection system according to one embodiment. [Figure 3] FIG. 3 illustrates an eye-tracking device for a projection system according to one embodiment. [Figure 4] FIG. 4 shows a projection system according to another embodiment. [Figure 5] FIG. 5 shows a projection system according to yet another embodiment. [Figure 6] FIG. 6 shows an alternative configuration of the projection system of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0016] 2 illustrates a near-eye light field projection system according to one embodiment. The projection system includes a light source array (not shown) that generates multiple incident light beams 100 that sequentially illuminate an SLM 20 from different angles. The SLM 20 modulates the incident light beams 100 and projects multiple modulated light beams 110. The projection system further includes a first optical element 70 that interacts with the modulated light beams 110 to form multiple light source images 28 at a light source image plane 30.

[0017] The projection system further comprises a second optical element 32 adapted to interact with the modulated light beam 110 to form a modulator image 114 at a modulator image plane 115 .

[0018] The projection system further comprises a third optical element 40 adapted to interact with the modulated light beam 110 to provide a first FOV 80 and emit a plurality of projection viewpoints 112 forming a first exit pupil 120 to an exit plane 125.

[0019] The projection system further includes a deflection element 60 configured to deflect the modulated light beam 110 to provide deflected modulated light beams 110a, 110b that spatially displace the modulator image 114 at a modulator image plane 115. The deflected modulated light beams 110a, 110b form second FOVs 81a, 81b, and the projected viewpoint 112 forms exit pupils 121a, 121b at an exit plane 125. The second FOVs 81a, 81b are formed depending on the orientation of the user's pupils 130. For example, when the user looks in a first direction (upward in the example of FIG. 2), the user sees the modulated light beam 110 deflected by the deflection element 60 and the expanded second FOV 81a in this first direction. When the user looks in a second direction (downward in the example of FIG. 2), the user sees the modulated light beam 110 deflected by the deflection element 60 and the expanded second FOV 81b in this second direction. The second FOVs 81 a, 81 b are therefore larger than the first FOV 80 in the direction of the orientation of the user's pupil 130. The content of the modulator image 114 in the expanded second FOVs 81 a, 81 b is adapted depending on the orientation of the user's pupil 130 and the expanded second FOVs 81 a, 81 b. This is achieved by controlling the SLM 20 based on the orientation of the user's pupil 130.

[0020] In one respect, the deflecting element 60 is "passive" and is arranged to deflect all modulated light beams 110 projected by the SLM, in which case the apparent brightness of the expanded second FOV 81 a, 81 b is not reduced compared to the brightness of the first FOV 80.

[0021] Viewed another way, the deflecting element 60 is "active" and deflects the modulated light beam 110 based on the orientation of the user's pupil 130. In this configuration, the first and second exit pupils 120, 121a, 121b are formed sequentially. In other words, when the user looks in a first direction, the deflecting element 60 deflects the modulated light beam 110 to form a first FOV 81a, expanding the FOV in the first direction. When the user looks in a second direction, the deflecting element 60 deflects the modulated light beam 110 to form a second FOV 81b, expanding the FOV in the second direction.

[0022] In one respect, the deflecting element 60 is "passive" and is arranged to deflect all modulated light beams 110 projected by the SLM, in which case the apparent brightness of the expanded second FOV 81 a, 81 b is not reduced compared to the brightness of the first FOV 80.

[0023] Viewed another way, the deflecting element 60 is "active" and deflects the modulated light beam 110 based on the orientation of the user's pupil 130. In this configuration, the first and second exit pupils 120, 121a, 121b are formed sequentially. In other words, when the user looks in a first direction, the deflecting element 60 deflects the modulated light beam 110 to form a first FOV 81a, expanding the FOV in the first direction. When the user looks in a second direction, the deflecting element 60 deflects the modulated light beam 110 to form a second FOV 81b, expanding the FOV in the second direction.

[0024] The active deflection elements 60 may be arranged in any type of pattern, such as tiles, for the second FOVs 81a, 81b. For example, the active deflection elements 60 may be arranged in a square, rectangular, or hexagonal pattern for the second FOVs 81a, 81b. The second FOVs 81a, 81b may or may not overlap the first FOV 80.

[0025] The first optical element 70 and the second optical element 32 may comprise imaging lenses.

[0026] The third optical element 40 may comprise an eyepiece or a combiner. The combiner 40 may be configured to transmit natural light from the real world to the first and second exit pupils 120, 121 a, 121 b, such that the modulated light beam 110, the deflected modulated light beams 110 a, 110 b, and the natural light are projected through the combiner into the first and second exit pupils 120, 121 a, 121 b.

[0027] In one embodiment, the deflecting element 60 is disposed between the first optical element 70 and the second optical element 32. More specifically, the deflecting element 60 may be disposed at a distance of less than 20 mm from the light source image plane 30. For example, the deflecting element 60 may be disposed at the light source image plane 30. In a configuration in which the deflecting element 60 is disposed at a distance of less than 20 mm from the light source image plane 30, the second exit pupils 121 a, 121 b are not significantly spatially offset at the exit plane 125 relative to the first exit pupil 120. When the deflecting element 60 is at the light source image plane 30, the second exit pupils 121 a, 121 b spatially coincide with the exit pupil 120 at the exit plane 125.

[0028] As shown in Figure 3, the projection system may include an eye-tracking device 210 adapted to measure the orientation of a user's pupil 130. The eye-tracking device 210 may be configured to control the SLM 20 to adapt the content of the modulator image 114 in the expanded second FOV 81a, 81b depending on the orientation of the user's pupil 130. In other words, the eye-tracking device 210 may be configured to cause the SLM 20 to control the content of the modulator image 114 (which comprises the image information generated by the SLM) that the user sees when looking in the direction of the orientation of the user's pupil 130. The eye-tracking device 210 may control the SLM 20 directly, as shown in Figure 3, or via a control or actuation unit 220.

[0029] When the deflecting element 60 is adapted to deflect the modulated light beam 110 based on the orientation of the user's pupil 130, the deflecting element 60 can be controlled by the eye tracking device 210 directly or via a control or actuation unit 220, as shown in FIG.

[0030] In one embodiment, the third optical element 40 may be configured such that the second exit pupils 121 a , 121 b are spatially offset in the exit plane 125 relative to the first exit pupil 120 .

[0031] 4 illustrates a projection system in which the third optical element 40 includes multiple sub-elements 40a, 40b, and 40c. Each sub-element 40a, 40b, and 40c interacts with the deflected modulated light beam 110a, 110b to shift a second exit pupil 121a, 121b at the exit plane 125 relative to the first exit pupil 120. More specifically, each sub-element 40a, 40b, and 40c may be configured to interact with the modulated light beam 110 deflected in a predetermined direction by the active deflection element 60 to shift the second exit pupil 121a, 121b. The shift of the second exit pupil 121a, 121b depends on the configuration of the sub-elements 40a, 40b, and 40c. In this configuration, the positions of the second exit pupils 121a, 121b and the second FOVs 81a, 81b in the exit plane 125 can be adjusted simultaneously. In one aspect, each sub-element 40a, 40b, 40c may have additional optical functions to obtain better optical quality at the point of view 112 of the image second exit pupil 121a, 121b (image of the second exit pupil). For example, each sub-element 40a, 40b, 40c may be configured to adapt its refractive power or to correct optical aberrations.

[0032] The virtual content at the first exit pupil 120 and the second exit pupils 121a, 121b entering from the first FOV 80 and the second FOV 81a, 81b respectively may be identical.

[0033] If the deflection element 60 is "passive," the size of the first and second exit pupils 120, 121a, 121b must not be smaller than the size of the user's pupil. Preferably, the size of the first and second exit pupils 120, 121a, 121b should be equal to or greater than the size of the user's pupil to avoid replicating content in an incorrect FOV. Typically, the average size of a human eye pupil is 4 mm. If the deflection element 60 is "active," the deflection element 60 may be smaller than the user's pupil 130.

[0034] 4 shows two second exit pupils 121 a, 121 b in addition to the first exit pupil 120. However, the projection system may be configured to form any number of second exit pupils 121 a, 121 b. Furthermore, the second exit pupils 121 a, 121 b may be formed in one or two dimensions.

[0035] 5 shows another configuration of the projection system in which the active deflection element 60 is positioned between the second optical element 32 and the modulator image plane 115. More generally, the active deflection element 60 can be positioned at a distance of at least 20 mm or more from the light source image plane 30, toward the modulator image plane 115. In this configuration, the deflected modulated light beams 110a, 110b form second FOVs 81a, 81b that are larger than the first FOV 80 and second exit pupils 121a, 121b that are larger than the first exit pupil 120. In this configuration, the active deflection element 60 simultaneously shifts the second exit pupils 121a, 121b and the second FOVs 81a, 81b at the exit plane 125 relative to the first exit pupil 120 and the first FOV 80 based on the pupil orientation. Here, the projection system is able to move the second exit pupils 121a, 121b and the second FOVs 81a, 81b simultaneously without using the third optical element 40 comprising the sub-elements 40a, 40b, 40c.

[0036] Figure 6 shows an alternative configuration of the projection system of Figure 5, in which the third optical element 40 comprises multiple sub-elements 40a, 40b, 40c, each of which interacts with a deflected modulated light beam 110, 110b. The sub-elements 40a, 40b, 40c allow for more precise control of the position of the second exit pupils 121a, 121b than would be possible using a third optical element 40 comprising a single component.

[0037] In one aspect, the sub-elements 40a, 40b, 40c may be further configured to provide one or both of optical power adaptation and optical aberration correction.

[0038] 4 and 6, the third optical element 40 includes three sub-elements 40a, 40b, and 40c. However, the third optical element 40 may include more than three sub-elements 40a, 40b, and 40c.

[0039] The third optical element 40 and the sub-elements 40a, 40b, and 40c have the following characteristics: Holographic Optical Elements (HOE), Active Liquid Crystal Polarization Gratings (LCPG), Active Metasurfaces, Diffractive Optical Elements (DOE), Flat or Curved Mirrors (Spherical, Parabolic, Aspherical, Freeform, Ellipsoidal), or any combination of these The offset of the second exit pupils 121a, 121b depends on the properties of the sub-elements 40a, 40b, 40c. The sub-elements 40a, 40b, 40c allow the spatial positions of the second exit pupils 121a, 121b to be spatially offset with higher precision.

[0040] The active deflection element 60 is Active liquid crystal polarization gratings (LCPGs), Pancharatnam-Berry gratings (PGs), microelectromechanical systems (MEMS) mirrors, active phase modulators, liquid crystal on silicon (LCOS, FLCOS), active metasurfaces, active refractive optical elements such as rotating wedges, sliding wedges, tilting wedges, and any combination thereof. It may have either one.

[0041] In the projection system disclosed in the present invention, the second FOV 81 a, 81 b can be redirected toward the user's pupil, and the first and second FOVs 80, 81 a, 81 b can be smaller (e.g., 30°) than the FOVs (e.g., 60°) typically used in known near-eye light field projection systems, while still providing high resolution, allowing the projection system to be more compact than known projection systems.

[0042] The projection system can be advantageously used in near-eye light-field virtual reality and mixed reality systems, especially near-eye light-field virtual reality and mixed reality systems with foveal adaptive projection.

[0043] The present disclosure further relates to a wearable device comprising a near-eye light field projection system, which may comprise an augmented reality device, a wearable mixed reality device, or smart glasses. [Explanation of symbols]

[0044] 10 Light source array 20 Spatial Light Modulator (SLM) 28 Light source image 30 Light source image plane 32 Second optical element 40 Third optical element 40a, 40b, 40c Combiner sub-elements 60 Active deflection element 70 first optical element 80 1st field of view (FOV) 81a, 81b 2nd FOV 90 eyes 100 incident rays 110 Modulated Light 110a, 110b Deflected modulated light beam 112 Projected Viewpoint 114 Modulator Image 115 Modulator Image Plane 120 1st exit pupil 121a, 121b 2nd exit pupil 125 Output plane 130 Pupil 140 Image ray 170 Projection axis 210 Eye Tracking Device 220 Operating unit

Claims

1. a light source array (10) for generating a plurality of incident light beams (100) that illuminate a spatial light modulator (20) adapted to modulate the incident light beams (100) and project a plurality of modulated light beams (110); a first optical element (70) adapted to interact with the plurality of modulated light beams (110) to form a plurality of light source images (28) at a light source image plane (30); a second optical element (32) adapted to interact with the plurality of modulated light beams (110) to form a plurality of modulator images (114) at a modulator image plane (115); a third optical element (40) adapted to interact with the plurality of modulated light beams (110) to project a plurality of projection viewpoints (112) that provide a first field of view (FOV) (80) and form a first exit pupil (120) at an exit plane (125); 1. A near-eye light field projection system for projecting an image onto a user's eye, comprising: The system comprises: an eye tracking device (210) adapted to determine a pupil orientation of the user's eye (90) and adapt the content of the plurality of modulator images (114) within the expanded second FOV (81 a, 81 b) according to the orientation of the user's pupil (130); At least one deflection element (60) between the first optical element (70) and the third optical element (40), the deflection element (60) is adapted to deflect the modulated light beam (110) to provide a plurality of deflected modulated light beams (110a, 110b) based on an orientation of a pupil of a user's eye; the plurality of deflected modulated light beams spatially shifting the modulator image (114) in a modulator image plane (115); the deflected modulated light beams (110a, 110b) form a second FOV (81a, 81b) in the direction of the pupil orientation that is larger than the first FOV (80); the deflection element (60), wherein the plurality of projection viewpoints (112) form second exit pupils (121a, 121b) in the exit plane (125); 1. A near-eye lightfield projection system for projecting an image onto a user's eye, further comprising:

2. 2. The projection system of claim 1, wherein the active deflection element (60) is disposed between the first optical element and the second optical element (70, 32).

3. 3. The projection system of claim 2, wherein the active deflection element (60) is at a distance of less than 20 mm from the light source image plane (30).

4. 4. The projection system of claim 3, wherein the active deflection element (60) is in the light source image plane (30) and the second exit pupil (121a, 121b) is spatially coincident with the exit pupil (120) in the exit plane (125).

5. 4. The projection system of claim 2 or 3, wherein the third optical element (40) is configured such that the second exit pupil (121a, 121b) is spatially shifted in the exit plane (125) relative to the first exit pupil (120).

6. 6. The projection system of claim 5, wherein the active deflection element (60) is adapted to sequentially shift second exit pupils (121a, 121b).

7. the third optical element (40) comprises a plurality of sub-elements (40a, 40b, 40c), each sub-element (40a, 40b, 40c) adapted to interact with the deflected modulated light beam (110a, 110b); 7. The projection system of claim 1, wherein the third optical element (40) shifts the second exit pupil (121a, 121b) in the exit plane (125) relative to the first exit pupil (120) based on the orientation of the pupil.

8. the active deflection element (60) is spaced from the light source image plane (30) towards the modulator image plane (115) by a distance of at least more than 20 mm; 8. The projection system of claim 1, wherein the second exit pupil (121a, 121b) has a larger size than the first exit pupil (120).

9. 9. The projection system of claim 8, wherein the active deflection element (60) is configured to simultaneously shift the second exit pupil (121 a, 121 b) and the second FOV (81 a, 81 b) relative to the first exit pupil (120) and the first FOV (80), respectively, at the exit plane (125) based on the pupil orientation.

10. 10. The projection system of claim 8 or 9, wherein the third optical element (40) comprises a plurality of sub-elements (40a, 40b, 40c), each sub-element (40a, 40b, 40c) adapted to interact with the deflected modulated light beam (110, 110b).

11. The third optical element (40) and / or the sub-elements (40a, 40b, 40c) are Holographic optical elements (HOEs), active or passive liquid crystal polarization gratings (LCPGs), active or passive metasurfaces, diffractive optical elements (DOEs), flat or curved mirrors, and any combination thereof.

11. The projection system according to claim 7 or 10, comprising:

12. 12. The image projection system according to claim 7, 10 or 11, wherein the sub-elements (40a, 40b, 40c) are adapted to adapt refractive power and / or correct optical aberrations.

13. The active deflection element (60) Any of the following: active liquid crystal polarization gratings (LCPGs), Pancharatnam-Berry gratings (PGs), microelectromechanical system (MEMS) mirrors, active phase modulators, liquid crystal on silicon (LCOS, FLCOS), active metasurfaces, active refractive optical elements such as rotating wedges, sliding wedges, tilting wedges, and any combination thereof.

13. The projection system of claim 1, comprising:

14. 14. The projection system of claim 1, wherein the eye-tracking device (210) is adapted to control the active deflection element (60) to deflect the modulated light beam (110).

15. A wearable device comprising the near-eye light field projection system of any one of claims 1 to 14.

16. 16. The wearable device of claim 15, comprising an augmented reality device, a mixed reality wearable device, or smart glasses.

Images