Method and system for rendering image using pupil enhancement adjustment of eye

Abstract

The present disclosure relates to a method for displaying an image having a specific depth of field. The method comprises the steps of: obtaining information data relating to a focal length adapted to a user looking at the display; determining the pupil size of the user; estimating a depth of field of both eyes of the user based on the focal length and the pupil size; and rendering an image to be displayed on the display based on the depth of field. Further, the present disclosure relates to a system, a head mounted display, and a non-transitory computer readable medium.

Description

[0001] This application is a divisional application of the invention patent application filed on March 26, 2020, with national application number 202010222523.6 and invention title "Method and System for Rendering Images Using Pupil Enhancement Adjustment of the Eye". Technical Field

[0002] This disclosure relates to a method for displaying an image with a specific depth of field. Furthermore, this disclosure relates to a system, a head-mounted display, and a non-transitory computer-readable medium. Background Technology

[0003] As is well known, objects rendered as virtual reality (VR) and augmented reality (AR) images in a display unit are projected at a specific depth from the user. This specific and defined depth creates a convergence-accommodation conflict by forcing the user's brain to unnaturally adapt to the conflicting cue while reducing accuracy. Retinal blur (a visual cue in the human visual system) drives an oculomotor response to accommodation, or adjustment of the eye's lens, to focus on the desired depth and thus minimize blur. Furthermore, retinal parallax is a visual cue that drives convergence, which is the simultaneous movement of both eyes in opposite directions to achieve or maintain binocular single vision.

[0004] One consequence of the parallax between convergence and accommodation is that users may experience visual fatigue (especially during prolonged use of the display unit). In certain fields, such as 3D television or movie viewing, the content and display unit need to meet certain constraints. One of these constraints could be, for example, the retinal parallax that must fall within the 1° safety zone of the focus cues. However, this constraint requires the ability to adjust stereoscopic parameters for each frame before viewing.

[0005] In the context of VR and AR applications (where the content is dynamic and interactive and nearby objects must be shown in different proximity tasks), this type of safe zone constraint is insufficient and therefore violated.

[0006] In addition to security concerns, users also need a more natural experience, such as when images or scenes in VR and AR applications have a field of vision consistent with what the eyes expect.

[0007] In other words, a method and system are needed to solve the above problems. Summary of the Invention

[0008] Accordingly, a system that more closely resembles the real world is provided. This is achieved by taking into account factors that affect eye accommodation, such as the shape of the lens that controls the point of convergence and the pupil size that controls the depth of field.

[0009] Therefore, according to this solution, both the convergence point of the eyes and the user's pupil diameter are captured. By using both the captured convergence point of the eyes and the user's pupil diameter, a hologram can be rendered on the display unit at the correct distance. Objects in the hologram will appear with the correct depth of field, meaning the depth of field of the rendered hologram will correspond perfectly to the real world.

[0010] Therefore, this disclosure relates to a method for displaying an image with a specific depth of field. The method includes the steps of: obtaining focal length-related information data adapted to a user viewing the display; determining the user's pupil size; estimating the depth of field for the user's eyes based on the focal length and the pupil size; and rendering an image to be displayed on the display based on the depth of field. The advantage is that rendering the image based on the depth of field, taking into account the user's pupil size, achieves a more natural user experience.

[0011] This disclosure also relates to a system for displaying an image with a specific depth of field. The system includes a display and at least one processor. The at least one processor is configured to: acquire focal length-related information data adapted to a user viewing the display; determine the user's pupil size; estimate the depth of field of the user's eyes based on the focal length and the pupil size; and render an image to be displayed on the display based on the depth of field. Further, the display is configured to display the rendered image to the user. An advantage is that rendering the image based on the depth of field, taking into account the user's pupil size, achieves a more natural user experience.

[0012] In one example, the system further includes an eye-tracking device configured to determine focal length-related information data and send that information to the processor.

[0013] The display may include an array of optical micro-elements. These optical micro-elements may be active or passive. The array of optical micro-elements may be selected from microlens arrays, microaperture arrays, liquid crystal arrays (such as LCD or LCoS), grating arrays, and phase mask arrays. The digital display elements may be selected from digital displays, such as LED displays, OLED displays, LCoS displays, LCD displays, and SLM displays.

[0014] Additionally, a head-mounted display is provided. The head-mounted display includes a frame suitable for wear by a user, a display, and at least one processor. The at least one processor is configured to: acquire focal length-related information data adapted to a user viewing the display; determine the user's pupil size; estimate the depth of field of the user's eyes based on the focal length and the pupil size; and render an image to be displayed on the display based on the depth of field. Further, the display is configured to display the rendered image to the user. The advantage is that rendering the image based on the depth of field, taking into account the user's pupil size, achieves a more natural user experience.

[0015] In one example, the head-mounted display further includes an eye-tracking device configured to determine focal length-related information data and send that information to the processor.

[0016] In yet another example, the head-mounted display is suitable for virtual reality experiences, augmented reality experiences, mixed reality experiences, or other broad range of reality experiences.

[0017] In addition, a non-transitory computer-readable medium is provided, on which instructions are stored, which can be executed by a computer to cause the computer to perform the following steps: obtaining information data related to a focal length adapted to a user viewing the display; determining the pupil size of the user; estimating the depth of field of the user's eyes based on the focal length and the pupil size; and rendering an image to be displayed on the display based on the depth of field.

[0018] Furthermore, the step of determining pupil size in the above method can be performed by obtaining and selecting a measurement of the user's pupil size using image data from an imaging device configured to measure pupil size. This imaging device can be part of an eye-tracking device. However, if such an imaging device is unavailable, pupil size can be estimated by considering the brightness of the display. Since the light from the display affects pupil size, the pupil size can be calculated based on the brightness of the display.

[0019] Furthermore, the information data related to this focal length may include a measurement of the distance at which the user's left and right eye gaze vectors converge. In this case, the measured focal length is the convergence distance, that is, the location where the user's gaze vectors converge at the convergence point.

[0020] Information related to the focal length can also be based on the user's gaze point toward the object rendered on the display or display unit.

[0021] Furthermore, the above method can further include a step of estimating the accommodative amplitude of the user's eyes, which varies depending on age, overall brightness, visual correction, etc. This estimation is then used to estimate the depth of field of the image to be rendered. The advantage is that the depth of field of the user's eyes can be estimated based on the accommodative amplitude of their eyes, resulting in a more natural experience.

[0022] As explained, this method can obtain eye orientation measurements to determine the convergence point, and can also determine the pupil diameter. These parameters can be used to generate and estimate the depth of field of a real-world eye. If a user-calibrated eye model is used as an additional step in this method, the rendered image will have a more realistic FOV that adapts to the user's vision.

[0023] In addition, the step of estimating the depth of field (DOF) may include applying a user-calibrated eye model, such as the PCCR pupil-center corneal reflectance model. Attached Figure Description

[0024] The present disclosure will be described in more detail below with reference to exemplary methods and systems shown in the accompanying drawings, in which: Figure 1 This illustrates a general representation of eye accommodation.

[0025] Figure 2 Examples of different depths of field due to convergence distance and pupil size are shown.

[0026] Figure 3 A flowchart illustrating an exemplary method according to this disclosure is provided.

[0027] Figure 4 A flowchart illustrating another exemplary embodiment of a method according to this disclosure is provided.

[0028] Figure 5 A system according to any exemplary embodiment of this disclosure is shown. Detailed Implementation

[0029] Apparatus, methods, and systems for displaying images with a specific depth of field will now be described more fully below with reference to the accompanying drawings, in which various examples are shown. For clarity, these drawings are schematic and simplified, and only the details necessary for understanding the invention are shown, while other details are omitted. The appended claims may be embodied in different forms not shown in the drawings and should not be construed as limiting to the examples set forth herein. Rather, these examples are provided so that this disclosure will be exhaustive and complete, and will fully convey the scope of the appended claims to those skilled in the art.

[0030] exist Figure 1The diagram illustrates a general representation of accommodation for the same eye. As can be seen, the eye focuses on the image through an accommodation process, where the ciliary muscle can change the curvature of the lens 101. This still forces both eyes to focus and adjust the lens 101 to the distance (focal length) that should correspond to the user's point of convergence. This is analogous to a pinhole camera, which is a camera with a very small aperture, where the depth of field is very large, and objects close to and far from the camera are rendered sharply on the image plane. On the other hand, a camera with a large aperture has a narrower and shallower depth of field, and nearby objects are rendered sharply, while more distant objects are rendered blurry and appear out of focus.

[0031] For the human eye, the pupil diameter corresponds to the aperture in a camera system. In well-lit environments, the pupil diameter can be 2 mm, which gives a depth of field of approximately 0.6 diopters, while in dark environments, the pupil diameter can be 8 mm, giving a depth of field of approximately 0.2 diopters. This information is typically used when rendering, for example, holograms that need to be convincingly integrated into the environment.

[0032] exist Figure 1 In diagram a, the far focus is shown. The eye is looking at a distant object, at which point light rays from the distant object pass through the cornea and diverge at a smaller angle, that is, closer to parallel, so little accommodation is needed. Then, the ciliary muscle relaxes, and the lens 101 has the longest focal length.

[0033] On the contrary, in Figure 1 In image b, the eye is looking at a closer object, and the light rays from this nearby object have a larger angle, thus requiring more refraction. The focal length of the lens needs to decrease to produce an image on the retina, so the ciliary muscle contracts, causing the lens to bend and protrude. In other words, the lens of the human eye typically changes its refractive index, or curvature, to adapt to a given focal length. This allows a person to see objects clearly and sharply at different distances.

[0034] like Figure 2As seen above, three images are rendered. For all three images, the focus (FP) is the position where the object is projected. The images above illustrate a dark scene (such as a scene rendered under low-light conditions) where the user sees an image of a cat displayed at a specific distance at night. Such scenes can be virtual 3D scenes, such as those in computer games or created in an HMD for virtual reality or augmented reality. The scene contains 3D objects that move within the scene over time. In fact, some objects can move out of the scene, while others can move into it. When these 3D scenes are viewed on a monitor, display, or computer screen, the monitor or screen presents a 2D image through a framed area on the display used to view information (i.e., the viewport), which represents the view of the 3D scene from a specific location in space. The viewport defines the image that will be seen by a camera or eye located at an imagined position about the 3D scene. The combination of camera position and viewport sets the field of view (FOV). On the other hand, when viewing a 3D scene through an HMD, each of the user's eyes is presented on the scene with slightly different viewports, and the user's brain then perceives individual 3D images from them.

[0035] As previously explained, the eye adapts to darkness by dilating the pupil. However, due to the darkness, the depth of field (DOF) is very narrow, and only a portion of the cat is perceived as sharp. The intermediate image shows a bright scene where the user's pupils are therefore smaller to adapt to the amount of light received. In this case, the DOF is larger, and most of the landscape is correctly perceived. However, because objects rendered in VR or AR (i.e., the cat) are projected at a specific depth from the user, the focal point (FP) does not necessarily coincide with the convergence point (i.e., the point where the eyes converge and therefore the object is perceived as sharp). This results in objects in the DOF not being perfectly sharp. Therefore, when a user is observing an object projected in VR or AR (e.g., in a hologram), a higher level of sharpness is needed for the image perceived by the user. This is achieved by the method according to this disclosure, and Figure 2 This effect is shown in the last image. The scene is bright, and therefore the depth of field (DOF) is the same as in the intermediate image. The difference from the previous images is that all objects in the DOF are rendered correctly based on the pupil size of the user looking at the monitor.

[0036] Figure 3This disclosure illustrates a method for displaying an image with a specific depth of field according to exemplary embodiments of the present disclosure. The method includes a series of steps. In step S1, information data is acquired by a device (such as an eye-tracking device or similar device) capable of measuring and adapting information related to the focal length or convergence distance to a user gazing at the display. The display may be monofocal, i.e., having a single lens, or it may be zoomable, i.e., having adjustable optical lenses capable of changing the depth of field across the entire view. Ideally, the display used in this method is monofocal or zoomable and provides the user with a more natural experience in terms of depth of field for HMD images, holograms, and VR experiences.

[0037] Focal length-related information may include a measurement of the distance 1A where the user's left and right eye gaze vectors converge. The measured distance is the distance between the user's position when gazing at an object rendered on the display and the convergence point of the gaze vectors from both eyes; that is, the convergence distance. At this convergence point, the user obtains a single binocular visual field, and therefore the object is focused and sharp. Alternatively, the focal length-related information may be based on the user's gaze point 1B oriented towards the object rendered on the display.

[0038] Information related to the focal length can also be based on the user's gaze point as they face the object being rendered on the display or display unit.

[0039] The tracking device sends the information data to at least one processor, which receives the focal length-related information data and stores it in a storage device. In step S1, it is assumed that the convergence distance equals the focal length. If this is not the case, the difference will be perceived by the user as an image with incorrect depth of field (i.e., out of focus).

[0040] Therefore, in order to estimate the correct depth of field for the user's eyes, the user's pupil size is determined in step S2. The method of this disclosure determines the pupil size either by determining the brightness 2A of the display or by obtaining and selecting measurements 2B of the user's pupil size (these measurements represent image data acquired from, for example, an eye-tracking device or a similar device).

[0041] In step S3, the depth of field for the user's eyes is estimated based on the obtained focal length and the determined pupil size. Then, in step S4, an image is rendered on the display based on the estimated depth of field.

[0042] Figure 4 Additional embodiments using the method according to this disclosure are shown in the figure. In this figure, another step is added to... Figure 3In the method mentioned above, in this case, in step S3, information related to the accommodation amplitude is also considered in order to estimate the depth of field adapted to the user. The accommodation amplitude is the maximum potential increase in optical power that the eye can achieve when adjusting the eye's focal point. It refers to a range of object distances within which the retinal image will be focused as clearly as possible. Due to various factors, particularly age, mental state (affected by medication, fatigue, etc.), ambient light intensity / color, overall light intensity / color, and other factors that reduce the flexibility of the lens in the eye, each user's accommodation amplitude varies. The accommodation amplitude can be estimated based on age and / or through a user-calibrated eye model.

[0043] The method for creating a user-calibrated eye model based on the central pupillary corneal reflection (PCCR) involves performing individual user calibration at different depths / distances and with different pupil sizes / illumination levels. Under different lighting conditions, holograms at different depths are shown to the user individually, and the user is asked to provide feedback (whether the rendered hologram blends into the environment). The results are then collected into the user-calibrated eye model, revealing the user's accommodative amplitude. Therefore, in step S4, depth of field is estimated based on the accommodative amplitude S3, information data related to focal length S1, and the user's pupil size S2. In step S5, an image adapted to the user is rendered on the display.

[0044] exist Figure 5 The image shows a system 500 for displaying an image with a specific depth of field according to any exemplary embodiment. The system 500 includes at least one processor (such as a central processing unit CPU 502 and a graphics processing unit GPU 503), the at least one processor being configured to perform... Figure 3 and Figure 4The system 500 includes all the method steps shown; and a display 506 for displaying the rendered image to a user. The at least one processor may be part of a computer device 504 along with a memory unit or storage device 505. By way of example, the storage device 505 may be a disk drive, an optical storage device, a solid-state device (such as programmable, flash-updatable random access memory (“RAM”) and / or read-only memory (“ROM”)) and / or the like. Further, the system 500 may include an eye-tracking device 501 configured to determine a focal length appropriate for the user. Further still, the system 500 may include an imaging device (not shown). The imaging device may be any imaging device or camera facing at least one of the user's eyes. Further still, the imaging device may be part of the eye-tracking device 501. Further still, the imaging device or camera may be independent of the final eye-tracking device 501. In another example, the system 500 may include a light sensor (not shown) configured to measure the level of ambient light to estimate pupil size.

[0045] Figure 5 The computer 506 shown is configured to include a non-transitory computer-readable medium on which instructions are stored, and which instructions are executable by the computer 506 to cause the computer to perform... Figure 3 and Figure 4 Any of the method steps described herein.

[0046] Another exemplary embodiment of this disclosure (not shown) is a head-mounted display (such as a pair of glasses, a virtual reality headset, an augmented reality headset, a helmet, etc.) for displaying and visualizing computer-generated images. The head-mounted display includes a rack suitable for being worn by a user, and at least one processor (such as a CPU or GPU) configured to perform operations in... Figure 3 and Figure 4 All method steps described herein. The head-mounted display also includes a display for displaying the rendered image to a user. Further, the head-mounted display may include an eye-tracking device configured to determine a focal length appropriate for the user.

Claims

1. A method for displaying an image having a specific depth of field, the method comprising the steps of: - Obtain focal length-related information data adapted to the user's gaze at the display, determine a measured distance at which the user's left and right eye gaze vectors converge based on the focal length-related information data, and set the measured distance as equal to the focal length. - Determine the user's pupil size. - The depth of field of the user's eyes is estimated by applying a user-calibrated eye model based on the focal length and the pupil size, and - Render the image to be displayed on the display based on the depth of field.

2. The method according to claim 1, wherein, The step of determining the pupil size is performed by first determining the brightness of the display and then estimating the pupil size based on the brightness.

3. The method according to claim 1, wherein, The step of determining the user's pupil size is performed by obtaining and selecting a measurement of the user's pupil size using image data from an imaging device.

4. The method according to any one of the preceding claims, wherein, The information data related to the focal length includes a measurement of the distance at which the user's left and right eye gaze vectors converge.

5. The method according to any one of claims 1 to 3, wherein, The information data related to the focal length is based on the user's gaze point, which is oriented towards the object being rendered on the display.

6. The method according to any one of the preceding claims, wherein, The method further includes the step of estimating the accommodative amplitude of the user's eyes, and wherein the step of estimating the depth of field is further based on the accommodative amplitude, wherein the accommodative amplitude is estimated based on the user-calibrated eye model, and the step of estimating the depth of field includes estimating the depth of field of the user's eyes based on the focal length, the pupil size, and the accommodative amplitude.

7. The method according to any one of the preceding claims, wherein, The user-calibrated eye model is based on the pupil-center corneal reflex (PCCR) model.

8. A system for displaying an image having a specific depth of field, the system comprising: - Monitor, and - At least one processor, said at least one processor being configured to: o Obtains focal length-related information data adapted to a user gazing at the display, determines a measured distance at which the user's left and right eye gaze vectors converge based on the focal length-related information data, and sets the measured distance as equal to the focal length. o Determine the user's pupil size o Estimates the depth of field of the user's eyes based on the focal length and pupil size by applying a user-calibrated eye model; and o Render the image to be displayed on the display based on the depth of field. The display is configured to show the rendered image to the user.

9. The system according to claim 8, further comprising: An eye-tracking device configured to determine focal length-related information data and send the information to the processor.

10. The system according to claim 8 or claim 9, wherein, The processor is further configured to first determine the brightness of the display and then estimate the pupil size based on the brightness.

11. The system according to claim 8 or claim 9, wherein, The processor is further configured to obtain and select a measurement of the user's pupil size using image data from an imaging device.

12. The system according to any one of claims 8 to 11, wherein, Information data related to the focal length includes measurements of the distance at which the user's left and right eye gaze vectors converge.

13. The system according to any one of claims 8 to 11, wherein, The information data related to the focal length is based on the user's gaze point, which is oriented toward the object being rendered on the display.

14. The system according to any one of claims 8 to 13, wherein, The processor is further configured to estimate the accommodative amplitude of the user's eyes, and wherein the estimation of the depth of field is further based on the accommodative amplitude, wherein the accommodative amplitude is estimated based on the user-calibrated eye model, and the step of estimating the depth of field includes estimating the depth of field of the user's eyes based on the focal length, the pupil size, and the accommodative amplitude.

15. The system according to any one of claims 8 to 14, wherein, The user-calibrated eye model is based on the pupil-center corneal reflex (PCCR) model.

16. A head-mounted display, the head-mounted display comprising: - A rack, the rack being adapted for use by a user. - Monitor, and - At least one processor, said at least one processor being configured to: o Obtains focal length-related information data adapted to a user gazing at the display, determines a measured distance at which the user's left and right eye gaze vectors converge based on the focal length-related information data, and sets the measured distance as equal to the focal length. o Determine the user's pupil size o Estimates the depth of field of the user's eyes based on the focal length and pupil size by applying a user-calibrated eye model; and o Render the image to be displayed on the display based on the depth of field. The display is configured to show the rendered image to the user.

17. The head-mounted display of claim 16, further comprising: An eye-tracking device configured to determine the information data related to focal length and send the information to the processor.

18. The head-mounted display according to claim 16 or claim 17, wherein, The head-mounted display is suitable for virtual reality experiences, augmented reality experiences, mixed reality experiences, or other broad range of reality experiences.

19. A non-transitory computer-readable medium having instructions stored thereon, the instructions being executable by a computer to cause the computer to perform the following steps: - Obtain focal length-related information data adapted to the user's gaze at the display, determine a measured distance at which the user's left and right eye gaze vectors converge based on the focal length-related information data, and set the measured distance as equal to the focal length. - Determine the user's pupil size. - The depth of field of the user's eyes is estimated by applying a user-calibrated eye model based on the focal length and the pupil size, and - Render the image to be displayed on the display based on the depth of field.

20. A method comprising any one of the technical features of claims 1-7 or any combination of the technical features.

21. A system comprising any one of the technical features or any combination of the technical features of claims 8-15.

22. A head-mounted display comprising any one of the technical features or any combination of the technical features of claims 16-18.

23. A non-transitory computer-readable medium comprising any one of the technical features of claim 19 or any combination of the technical features.

Images