Estimating prescription eyewear power for head mounted display user

Abstract

A method for estimating the degree of prescription glasses of a user of a head mounted display is disclosed. The method includes illuminating a user's eye and prescription glasses using an illuminator, capturing images of the eye, the prescription glasses, and reflections on the prescription glasses using at least one camera, and determining a power of a lens of the prescription glasses by analyzing the reflections. The method may be used to improve the user experience in virtual reality applications, augmented reality applications, and mixed reality applications by taking into account the prescription glasses of the user.

Description

Technical Field

[0001] This technology relates to the field of head-mounted displays (HMDs), specifically for virtual reality (VR), augmented reality (AR), and mixed reality (MR) applications. The focus is on eye-tracking systems and methods for estimating the prescription glasses worn by users of these head-mounted displays. Background Technology

[0002] Eye-tracking technology is becoming increasingly important across various fields, including virtual reality (VR), augmented reality (AR), and mixed reality (MR) applications. These applications typically involve the use of wearable devices that include a display (also known as a head-mounted display (HMD)) to provide users with immersive experiences. HMDs are well-known and are typically used in both VR and AR systems. In these systems, displays are used to provide users with experiences that simulate different realities in VR or augmented reality in AR. Eye-tracking systems integrated into HMDs enable various functionalities, such as gaze-based interaction, user attention analysis, and improved visual rendering. U.S. Patent Application Publication No. 2017 / 0090564 describes a wearable HMD that incorporates an eye-tracking system to detect the user's gaze direction.

[0003] A typical eye-tracking system consists of one or more cameras and illuminators that capture images of the user's eyes. The system then uses mathematical models and computer analysis to determine the gaze direction by identifying various eye features in the captured images, such as pupil and corneal reflections. The accuracy of eye-tracking systems is crucial for providing users with a seamless, immersive experience.

[0004] However, a significant portion of the population requires prescription glasses to correct their vision. The presence of prescription glasses between the user's eye and the eye-tracking system introduces distortion into the acquired images of the eye. This distortion affects the size, shape, and position of eye features such as the pupil and corneal reflex, thus impacting eye-tracking performance. The degree of distortion depends primarily on the prescription, such as the lens power. The shape and material of the lenses determine the prescription.

[0005] Existing technologies have attempted to address the distortion problem caused by prescription glasses in eye-tracking systems. For example, US10342425B1 describes using an illuminator based on flashes from the user's pupil to detect a user's gaze. This disclosure describes a technique for identifying and compensating for reflections caused by the user's prescription glasses to ensure accurate gaze direction can be determined.

[0006] However, existing technologies do not offer a comprehensive solution to the problem of eye-tracking performance being affected by prescription glasses. While the distortion introduced by prescription glasses is manageable when the glasses' parameters are known, it can affect the accuracy of eye-tracking systems in analyzing eye features and reflecting world features when the glasses' parameters are unknown. Summary of the Invention

[0007] According to a first aspect of this disclosure, a method is provided for estimating the power of prescription glasses for a user of a head-mounted display. The method includes illuminating the user's eye and the prescription glasses with an illuminator, capturing images of the eye, the prescription glasses, and reflections on the prescription glasses using at least one camera, and determining the lens power of the prescription glasses by analyzing the reflections. This method allows for the estimation of the user's prescription glasses power in a non-invasive and efficient manner, thereby providing a more accurate eye-tracking system. The mapping from image coordinates to world coordinates becomes more accurate. This enhanced accuracy, for example, helps to more accurately estimate corneal position based on distortion-free flash and pupil triangulation, enabling more accurate fixation mapping and entrance pupil position (EPP) estimation. Therefore, this improves eye-tracking performance and enhances the user experience.

[0008] Optionally, in some examples, the method further includes setting camera brightness control parameters for at least one camera to capture reflections on the prescription glasses. The camera brightness control parameters include at least one of exposure time or camera gain. This feature allows for better image quality and more accurate reflection capture, which can lead to a more precise estimate of the prescription glasses' diopter.

[0009] Optionally, in some examples, the method further involves determining the location of reflections in the image and identifying the illuminator corresponding to each reflection. This feature can help to accurately determine the lens power by providing more data points for analysis.

[0010] According to another aspect of this disclosure, a method for determining the lens power includes using the correlation between the lens power and the radius of curvature of the lens surface. The radius of curvature of the lens is estimated based on the location of the reflection in the image and the location of the corresponding illuminator. This method allows for a more direct and accurate estimation of the lens power.

[0011] Optionally, in some examples, the reflection used to estimate the radius of curvature of the lens is the reflection from the outer surface of the lens. This feature can simplify the estimation process and reduce potential errors because the outer surface is more accessible and easier to image.

[0012] Optionally, in some examples, the correlation used to determine the lens power is the base curve rule. This can provide a reliable and accurate estimate of the lens power.

[0013] Optionally, in some examples, the method further includes approximating the surface of the lens as a spherical shape. This approximation simplifies the calculation process and is generally accurate enough for most practical applications.

[0014] Optionally, in some examples, the method further includes determining the power of each of the right and left lenses in a prescription pair of glasses. This feature allows for a more personalized and accurate estimation of the prescription glasses' power because it takes into account the potential power difference between the two lenses.

[0015] According to another aspect of this disclosure, the method further includes utilizing image processing of the captured images to estimate the curvature of the outer and / or inner surfaces of the prescription eyeglasses for each lens. This feature can significantly improve the accuracy and efficiency of the estimation process.

[0016] Optionally, in some examples, the method includes using a pre-trained regression model to estimate the curvature of the outer and / or inner surfaces of the prescription eyeglasses. The pre-trained regression model can leverage machine learning techniques to make more accurate predictions based on the captured images. Therefore, the pre-trained regression model can estimate curvature through a machine learning process. This feature allows for continuous improvement in estimation accuracy over time, as the machine learning model can learn from new data and improve its predictions.

[0017] Optionally, in some examples, the machine learning process includes at least one of linear regression and / or nonlinear regression. These regression techniques can provide a robust and flexible framework for estimating lens power because they can model a wide range of relationships between lens curvature and power.

[0018] Optionally, in some examples, labeled features are used to train a pre-trained regression model. These labeled features include the curvature and reflection of the outer and / or inner surfaces of the lenses from images of users wearing prescription glasses. This improves the accuracy of the regression model because it learns from real-world data that accurately reflects changes in lens curvature and reflection.

[0019] According to another aspect of this disclosure, a head-mounted display is provided, comprising: an eye-tracking system configured to determine the gaze direction and entrance pupil position of a user's eyes; at least one camera configured to capture images of the user's eyes and prescription glasses; illuminators configured to illuminate the user's eyes and prescription glasses; and a computer analysis system configured to analyze the captured images and determine the lens power of the prescription glasses by analyzing reflections on the prescription glasses. This head-mounted display can provide a comprehensive solution for estimating the power of a user's prescription glasses, which can significantly enhance the user experience in virtual reality applications.

[0020] Optionally, in some examples, the eye-tracking system of the head-mounted display is configured to perform central pupillary reflex (PCCR) eye tracking. This feature can provide a more accurate and reliable estimate of the user's gaze direction, which can be used in many virtual reality applications.

[0021] Optionally, in some examples, the head-mounted display's illuminator is an LED illuminator with a wavelength range of 750 nm to 1400 nm. This feature can provide suitable illumination for capturing high-quality images of the user's eyes and prescription glasses, thereby enabling a more accurate estimation of the prescription glasses' power.

[0022] Optionally, in some examples, the computer analysis system of the head-mounted display is configured to optimize camera brightness parameters, including at least one of exposure time or camera gain, to capture reflections on the outer and / or inner surfaces of the prescription eyeglasses. This feature can enhance image quality and improve the accuracy of reflection capture, resulting in a more accurate estimate of the lens power.

[0023] Optionally, in some examples, the computer analysis system of the head-mounted display is configured to identify the illuminator corresponding to each reflection and determine the curvature of the lens by analyzing the reflection. This feature can provide more data points for analysis and improve the accuracy of lens power estimation.

[0024] Optionally, in some examples, the computer analysis system of the head-mounted display is configured to approximately calculate the radius of curvature and estimate the lens power based on the curvature. This feature simplifies the calculation process and provides a more direct and accurate lens power estimate.

[0025] Optionally, in some examples, the computer analysis system of the head-mounted display is configured to use a pre-trained regression model to estimate the curvature of the outer and / or inner surfaces of the prescription glasses for each lens. This feature can leverage machine learning techniques to improve the accuracy and efficiency of lens power estimation.

[0026] According to another aspect of this disclosure, a method for estimating the power of prescription glasses for a user of a head-mounted display is provided. The method includes: capturing images of each user's eye and the prescription glasses from different viewpoints; determining multiple candidate ellipses representing the same iris from the captured images; using triangulation of the candidate ellipses to determine the center and radius of each iris in 3D space; and analyzing the deviation of the determined radius of each iris to infer the lens power of the prescription glasses. This provides an alternative method for estimating the power of a user's prescription glasses, thereby allowing for accurate eye-tracking systems. Improved eye-tracking performance and an enhanced user experience are provided.

[0027] Optionally, in some examples, the method for determining each ellipse includes using edge detection to fit an ellipse for each iris. This feature can provide a robust and accurate representation of the iris, which can improve the accuracy of lens power estimation.

[0028] Optionally, in some examples, the method used to determine each ellipse includes utilizing an elliptic regression model. This feature can leverage machine learning techniques to provide a more accurate and efficient representation of the iris, thereby enabling more precise estimation of lens power.

[0029] Optionally, in some examples, the radius deviation for each iris is the difference in size between the radius and an estimated average human iris radius. This feature can provide a more direct and intuitive way to infer lens power because it directly links lens power to measurable physical properties of the user's eye.

[0030] Optionally, in some examples, the average radius of the human iris is 5.5 mm. This feature provides a suitable reference point for the radius deviation of each iris, which can improve the accuracy of lens power estimation.

[0031] Optionally, in some examples, the lens power is determined based on a pre-calculated relationship between the lens power of the prescription glasses and the deviation from a determined radius of the iris. This feature can provide a more direct and accurate estimate of the lens power because it utilizes the relationship between the lens power and the iris radius deviation.

[0032] Optionally, in some examples, the pre-computed relationship is determined through a machine learning process, where the estimated iris radius is known before refraction through the lens. This feature can improve the accuracy of lens power estimation because the machine learning process can learn from previous data and improve its predictions.

[0033] Optionally, in some examples, the machine learning process includes a regression model. This feature can provide a robust and flexible framework for estimating lens power because it can model a wide range of relationships between iris radius deviation and lens power.

[0034] According to another aspect of this disclosure, a head-mounted display is provided, comprising: an eye-tracking system configured to determine the gaze direction of a user's eyes; at least one camera configured to capture images of each user's eye and prescription glasses from different viewpoints; and a computer analysis system configured to determine multiple candidate ellipses representing the same iris from the captured images, use triangulation of the candidate ellipses to determine the center and radius of each iris in 3D space, and analyze the deviation of the determined radius of each iris to infer the lens power of the prescription glasses. This head-mounted display can provide a comprehensive solution for estimating the power of a user's prescription glasses, thereby significantly enhancing eye-tracking performance.

[0035] Optionally, in some examples, the computer analysis system of the head-mounted display is further configured to estimate the difference between the radius and the average radius of the human iris. This feature can provide a more direct and intuitive way to infer lens power because it directly links lens power to measurable physical properties of the user's eye.

[0036] Optionally, in some examples, the computer analysis system of the head-mounted display is further configured to determine the lens power of the prescription glasses based on a pre-calculated relationship between the lens power and the deviation of a defined radius of each iris. This feature can provide a more direct and accurate estimate of the lens power because it utilizes a pre-established relationship between the lens power and the iris radius deviation.

[0037] Optionally, in some examples, the pre-computed relationship is determined through a machine learning process, where the estimated iris radius is known before refraction through the lens. This feature can improve the accuracy of lens power estimation because the machine learning process can learn from previous data and improve its predictions.

[0038] Optionally, in some examples, the machine learning process includes a regression model. This feature can provide a robust and flexible framework for estimating lens power because it can model a wide range of relationships between iris radius deviation and lens power. Attached Figure Description

[0039] The examples are described in more detail below with reference to the accompanying drawings.

[0040] Figure 1This is a schematic diagram of a head-mounted display and an eye-tracking system, as exemplified by this disclosure.

[0041] Figure 2 This is a schematic diagram of an eye-tracking system that includes multiple illuminators, each of which is located at a corresponding fixed position relative to the user's eye.

[0042] Figure 3a An example eye image of a user who does not wear prescription glasses is shown.

[0043] Figure 3b An example eye image of a user wearing prescription glasses is shown.

[0044] Figure 4 Example reflections on the outer surface of the eyeglasses and detected reflections on the outer surface of the eyeglasses are shown.

[0045] Figure 5 This is a diagram of a user's eye at a certain distance from the lenses of prescription glasses.

[0046] Figure 6 This is a schematic diagram of a prescription eyeglass lens with curvature.

[0047] Figure 7a This is a diagram illustrating the geometry of an eye-tracking system including a camera and two illuminators, as exemplified by this disclosure.

[0048] Figure 7b The sufficiency of known geometry and the detection of reflections from two illuminators are demonstrated to reconstruct the base arc corresponding to the curvature of the outer surface of the lens.

[0049] Figure 8a The grouping of prescription lens power (Rx) on the same base curve is shown.

[0050] Figure 8b The Tscherning ellipse is displayed.

[0051] Figures 9a to 9d The illustration shows the inner and outer surface curvatures of prescription glasses worn by a user in an eye image, used to train a curvature regression model based on associated labeled metadata.

[0052] Figure 10 This is a diagram of the iris triangulation of the left eye, where r is the radius of the "virtual" iris estimated in 3D.

[0053] Figure 11 The iris radius is shown using triangulation with different spectacle diopters on an eye model with a known iris radius of approximately 2.5 mm. Detailed Implementation

[0054] The detailed description set forth below provides information and examples of the disclosed technology in sufficient detail to enable those skilled in the art to practice the contents of this disclosure.

[0055] Figure 1 A schematic diagram of a head-mounted display 100 and an eye-tracking system 110 according to an example of this disclosure is shown. The head-mounted display 100 may be a VR head-mounted device or other wearable device placed on a user's head to view VR, AR, or MX content using VR, AR, or MX systems and applications. The head-mounted display 100 includes an eye-tracking system 110, at least one camera 120, an illuminator 130, and a computer analysis system 140. The head-mounted display 100 is configured to provide a user with a virtual reality experience, an augmented reality experience, or a mixed reality experience. The eye-tracking system 110 is designed to determine the gaze direction and entrance pupil position of the user's eyes 150. The camera 120 is configured to capture an image 190 of the user's eyes 150 and prescription glasses 160. The illuminator 130 is configured to illuminate the user's eyes 150 and prescription glasses 160. The computer analysis system 140 is configured to analyze the captured image 190 and determine the power of the lens 165 of the prescription glasses 160 by analyzing reflections 180 on the prescription glasses 160. This provides a more accurate eye-tracking system 110. The mapping from image coordinates to world coordinates becomes more accurate. This enhanced accuracy, for example, helps to more accurately estimate the corneal position based on distortion-free flash and pupil triangulation, thereby enabling more accurate gaze mapping and accurate entrance pupil position (EPP) estimation. Therefore, this improves eye-tracking performance and enhances the user experience. As further described below, the power of lens 165 can be determined by using the correlation between the power and the radius of curvature 170 of the surface of lens 165, wherein the radius of curvature 170 of lens 165 can be estimated based on the position of reflection 180 in image 190 and the position of the corresponding illuminator 130.

[0056] Figure 2 Another schematic diagram of an eye-tracking system 110 according to an example is shown. The eye-tracking system 110 includes a plurality of illuminators 130. When a system user uses a head-mounted display 100 equipped with the eye-tracking system 110, each of the plurality of illuminators 130 is positioned at a corresponding fixed location relative to the user's eye (illustrated by the iris 156). Specifically, the plurality of illuminators 130 may be arranged along the periphery of a generally circular outline, which may be, for example, the periphery of one of the VR lenses in the head-mounted display 100. A reflection 180 from prescription glasses 160 is shown. A computer analysis system 140 communicates with one or more cameras 120 and illuminators 130 and is configured to analyze the reflection 180 and determine the power of the lens 165 of the prescription glasses 160, as per [reference to...]. Figure 1As described. In one example, computer analysis system 140 is configured to analyze reflection 180 and reflections on the cornea, i.e., flashes in the user's eye, to determine the power of lens 165.

[0057] Figure 3a An example eye image of a user who does not wear prescription glasses 160 is shown. A flash of light is shown in the user's eye. Figure 3b An example eye image of a user with prescription glasses 160 and a reflection 180 on the surface of the prescription glasses are shown. The prescription glasses 160 introduce distortion into the acquired eye image. This distortion affects the size, shape, and position of eye features such as the pupil and corneal reflection, which will affect eye-tracking performance in the prior art. Figure 4 The illustration shows example reflections 180 on the outer surface of eyeglasses 160 according to this disclosure, and the corresponding detection of these reflections 180 by the camera 120 of the head-mounted display 100. At least two reflections on the outer surface of the eyeglasses can be identified. Figure 5 This is a schematic diagram of a user's eye and iris 156 at a distance 172 from the lens 165 of the prescription glasses 160. The lens 165 has an outer surface 168 and an inner surface 178, wherein the inner surface faces the user's eye.

[0058] Figure 6 This is a schematic diagram of a lens 165 of a prescription eyeglass 160 having a curvature of 169. The curvature of the outer surface 168 is denoted as 169a. The curvature of the outer surface 168 of the lens 165 is commonly referred to as the base curve of the prescription eyeglass 160. Traditionally, the base curve is considered spherical, following general design rules. The curvature of the inner surface 178 is denoted as 169b. The inner surface 178 is commonly referred to in the art as the Rx surface and is the site where the prescription is implemented. The corresponding radius of curvature 170 is illustrated. The power of the lens 165 is estimated based on curvatures 169, 169a, 169b or the corresponding radius of curvature 170. The power of the lens 165 can be determined using methods such as... Figures 8a to 8b The correlation shown is used to determine this.

[0059] Figure 7a This is a diagram of the geometry of an eye-tracking system 110 according to an example of this disclosure, which includes a camera (square) and two illuminators (circles) positioned at the periphery of a VR lens (solid line), wherein the x-axis and y-axis indicate distances in mm. Figure 7b This is an example diagram illustrating how, for 100 random eyeglass positions and outer surface curvatures, the method according to the invention accurately reconstructs the base arc, i.e., the curvature of the outer surface 168 of the eyeglass. Here, an optimization method is employed to demonstrate the sufficiency of known geometry (camera position and illuminator position) and the detection of reflections from both illuminators for reconstructing the base arc.

[0060] Figure 8a The grouping of prescription lens power (Rx) on the same base curve is shown. Figure 8b The image shows the Tscherning ellipse, a graphical representation of the outer surface curvature (base curve) that minimizes astigmatism for different lens powers. The Tscherning ellipse illustrates its solution in two segments: the steeper Wollaston segment and the flatter Ostwalt segment. The Ostwalt segment has been used in modern optical design because the Wollaston segment produces an impractically steep lens surface. The Ostwalt segment of the Tscherning ellipse is cited as an example of the base curve rule. By employing the base curve rule, the radius of curvature of the prescription lens can be approximated, and the power assigned to the prescription lens can be determined. Optical design principles describe how the base curve should correspond to the desired lens power in order to minimize lens aberrations. This principle is known in the aforementioned Tscherning ellipse or some simplification thereof (e.g., "Vogel's rule"), and is known to those skilled in the art. While additional design considerations, aesthetic factors, or optometrist or subject preferences may influence the curvature, the deviation from the Ostwalt segment of the Tscherning ellipse must be small to avoid significant astigmatism in the lens. Therefore, by measuring the base curve of prescription glasses, a reasonable and accurate estimate of their lens power can be obtained.

[0061] Figures 9a to 9d Different examples of eye images 190 of a user wearing prescription glasses are illustrated. For each lens 165, the curvatures 169, 169a, 169b of the outer surface 168 and / or inner surface 178 of the prescription glasses 160 can be estimated using image processing of the captured image 190. In another method for estimating the power of the prescription glasses 160 of a user of the head-mounted display 100, a curvature regression model is trained based on labeled features, including the curvatures 169, 169a, 169b of the outer surface 168 and / or inner surface 178 of the lens from image 190, as well as reflections 180. Thus, labeled metadata with a specific reflection pattern 180 in image 190 can specify the corresponding diopter value and radius of curvature 170 for training the curvature regression model. The correlation between the estimated radius of curvature 170 of the outer surface 168 and / or inner surface 178 and the lens power (e.g., as shown in the image) can then be utilized. Figures 8a to 8b (As shown) to determine the lens power. Therefore, the computer analysis system 140 can be configured to use a pre-trained regression model to estimate the curvatures 169, 169a, 169b of the outer surface 168 and / or the inner surface 178 to estimate the lens power 165. Thus, improved eye-tracking performance can be provided.

[0062] Figure 10This is a diagram of the iris triangulation for the left eye, where r is the radius of the estimated "virtual" iris 195 in 3D. A similar method was performed for the right eye; however, Figure 10 An example for only one eye is shown. The figure illustrates the process of using triangulation of candidate ellipses to determine the center 157 and radius 158 of each iris 156 in 3D space. 3D space is typically represented by an x, y, z coordinate system of the space surrounding the user of the head-mounted display 100. For example, using a dual-camera 120 eye-tracking solution, two images 190 are captured for each eye 150 of the user wearing an XR head-mounted device, one for the left and one for the right. Computer analysis can be performed on the acquired eye images 190 to detect the presence of prescription glasses 160 on the user's eyes 150. If glasses 160 are detected in front of the user's eyes 150, the method is used to determine the position of the iris 156 for each eye image 190. This can be achieved by i) utilizing edge detection and fitting an ellipse using a RANSAC (Random Sample Consensus)-based method, or ii) using an iris-based elliptic regression model to determine the ellipse, or similar methods known in the art. For each eye (left or right), once two ellipses representing the same iris 156 are obtained from the camera, the center 157 and radius 158 of each iris in 3D (i.e., the determined virtual iris 195) are calculated. The resulting values ​​represent the “virtual” estimated iris position, taking into account refraction caused by the glasses 160. Typically, the average radius of a human iris is approximately 5.5 mm. However, when using triangulation to calculate the 3D iris radius, this value may vary due to the lenses of the glasses.

[0063] By analyzing the deviation of the triangulation iris radius (158) obtained from triangulation, the lens power can be determined. The more negative the refractive power, the smaller the radius obtained from triangulation. (See [link to relevant documentation]). Figure 11 To accurately approximate the deviation, data can be collected using an eye model representing the physical eye, in which the actual iris radius 158 before refraction through the glasses 160° is known, or data can be collected using a training model that labels the real human iris radius with and without prescription glasses of known power. By establishing a relationship between the lens power and the deviation from the calculated radius through a learning process (such as linear regression and reinforcement learning), the lens power 160° can be effectively inferred. Figure 11 The image shows an iris radius of 158 obtained by triangulation using an eye model with a known iris radius of approximately 2.5 mm and different spectacle diopters.

[0064] This disclosure includes several components that work together to estimate the prescription of glasses 160 for a user of the head-mounted display 100. These components include the head-mounted display 100, an eye-tracking system 110, a camera 120, an illuminator 130, and a computer analysis system 140.

[0065] In some embodiments, the head-mounted display 100 is designed to provide a user with a virtual reality experience, an augmented reality experience, or a mixed reality experience. The head-mounted display 100 includes an eye-tracking system 110, at least one camera 120, an illuminator 130, and a computer analysis system 140. The head-mounted display 100 is configured to be worn by a user and is designed to track the user's eye movements and gaze direction. The head-mounted display 100 can be used in a variety of applications, including but not limited to games, training simulations, virtual meetings, and other interactive experiences.

[0066] In one example, an eye-tracking system 110 is integrated into a head-mounted display 100. The eye-tracking system 110 is designed to determine the gaze direction and entrance pupil position of a user's eyes 150. The eye-tracking system 110 can use various technologies and techniques to track the user's eye movements and gaze direction. The eye-tracking system 110 may include one or more cameras 120 and illuminators 130. The eye-tracking system 110 may also include a computer analysis system 140 configured to analyze captured images 190 and determine the lens power 165 of prescription glasses 160 by analyzing reflections 180 on the glasses. The eye-tracking system 110 of the head-mounted display 100 can be configured to perform central pupillary reflex (PCCR) eye tracking. PCCR is a well-known and easily understood method for determining a user's gaze. Further information on this method can be found in several places, including Guestrin, ED. and Eizenman, E., “General theory of remote gaze estimation using the pupil center and corneal reflections”, Biomedical Engineering, IEEE Transactions, Vol. 53, No. 6, pp. 1124, 1133, June 2006.

[0067] Camera 120 is configured to capture an image 190 of the user's eyes 150, prescription glasses 160, and reflections 180 generated by illuminator 130. Camera 120 can be a digital camera, video camera, or any other type of image capture device. Camera 120 can be positioned at various locations on the head-mounted display 100 to capture images 190 of the user's eyes 150 and prescription glasses 160 from different angles or viewpoints. Camera 120 can be configured to capture images 190 at a high frame rate (e.g., 60-120 frames per second) to accurately track user eye movements and gaze direction.

[0068] Image 190 may include various features such as the user's eye 150, the prescription glasses 160, and reflections 180 on the prescription glasses 160. Image 190 may be captured at high resolution (e.g., 1920 × 1080 pixels) to provide detailed information about the user's eye 150 and the prescription glasses 160. Image 190 may be analyzed by computer analysis system 140 to determine the power of the lens 165 of the prescription glasses 160. Image 190 may be stored in the memory of the head-mounted display 100 or transferred to an external device for further analysis.

[0069] In some configurations, this process involves setting camera brightness control parameters to achieve optimal reflection capture. Camera brightness control parameters can include exposure time and camera gain. Exposure time refers to the length of time the camera shutter is open to allow light to reach the image sensor. Camera gain amplifies the signal from the image sensor. By adjusting these camera parameters, the process optimizes the capture of reflections 180 on the prescription glasses 160, thereby improving the accuracy of diopter estimation.

[0070] In one embodiment, the process begins by illuminating the user's eyes 150 and prescription glasses 160 with an illuminator 130. The illuminator 130 emits light, which is reflected by the lens 165 of the prescription glasses 160, producing a reflection 180 that can be captured by the camera 120. The illuminator 130 is thus configured to provide illumination to the user's eyes 150 and prescription glasses 160. The illuminator 130 can be positioned at various locations on the head-mounted display 100 to illuminate the user's eyes 150 and prescription glasses 160 from different angles. The illuminator 130 can be of various types, such as an LED illuminator, a VCSEL, or a fiber optic illuminator providing near-infrared light. The illuminator 130 can emit light of various intensities and wavelengths. For example, the illuminator 130 can emit light with an intensity of 500 to 1000 lumens and a wavelength range of 750 nm to 1400 nm. The illuminator 130 can be configured to project light through the lens 165 of the prescription glasses 160 onto the user's pupil 155. The light emitted by the illuminator 130 can be reflected by the lens 165 of the prescription glasses 160 and captured by the camera 120 as a reflection 180.

[0071] Computer analysis system 140 is configured to analyze image 190 captured by camera 120. Computer analysis system 140 may include one or more processors and memory for storing instructions and data. Computer analysis system 140 may be configured to perform various tasks, such as identifying a user's eye 150 and prescription glasses 160 in image 190, determining the position of reflections 180 in image 190, identifying the illuminator 130 corresponding to each reflection 180, determining the curvature 169 of lens 165 by analyzing reflections 180, approximating the radius of curvature 170 of lens 165, and estimating the diopter of lens 165 based on the radius of curvature 170. Computer analysis system 140 may use various algorithms and machine learning techniques to perform these tasks.

[0072] In one example, reflection 180 is light emitted by illuminator 130 reflected by lens 165 of prescription glasses 160. Reflection 180 can be captured by camera 120 and included in image 190. Reflection 180 can provide information about the curvature 169 and radius of curvature 170 of lens 165. Reflection 180 can be analyzed by computer analysis system 140 to determine the power of lens 165. The location of reflection 180 in image 190 can be used to estimate the radius of curvature 170 of lens 165. The location of reflection 180 can also be used to identify the illuminator 130 corresponding to each reflection 180. The analysis of reflection 180 can be based on various assumptions, such as assuming that the outer surface 168 of lens 165 follows a spherical shape, and assuming that the distance 172 between the surface of the cornea and lens 165 is constant.

[0073] In some configurations, the user of the head-mounted display 100 wears prescription glasses 160. The prescription glasses 160 include lenses 165 that correct the user's vision. Lenses 165 can have various powers, determined by the user's vision correction prescription and the wearing position.

[0074] This disclosure specifies a method for estimating the power of prescription glasses 160 for a user of a head-mounted display 100. The method includes illuminating the user's eye and the prescription glasses 160, capturing an image 190 of the eye 150, the prescription glasses 160, and reflections 180 on the prescription glasses, and determining the power of the lens 165 of the prescription glasses 160 by analyzing the reflections 180.

[0075] In one example, the method begins by illuminating the user's eyes 150 and prescription glasses 160. This illumination can be provided by one or more illuminators 130 as part of an eye-tracking system 110. Illuminators 130 can emit light of various intensities and wavelengths to illuminate the user's eyes 150 and prescription glasses 160. Illuminators 130 can be positioned at various locations on the head-mounted display 100 to illuminate the user's eyes 150 and prescription glasses 160 from different angles or viewpoints. The light emitted by illuminators 130 can be reflected by lenses 165 of the prescription glasses 160 and captured by a camera 120 as a reflection 180.

[0076] Illuminator 130 provides the necessary light to camera 120 to capture a clear image 190 of the user's eye 150 and prescription glasses 160. The light emitted by illuminator 130 is reflected by the lens 165 of prescription glasses 160, resulting in a reflection 180 that can be analyzed to determine the power of lens 165. Depending on the requirements of a specific application, illuminator 130 can be controlled to emit light of varying intensities.

[0077] In one configuration, the method includes capturing an image 190 of the user's eye 150, prescription glasses 160, and a reflection 180 on the prescription glasses 160. Image 190 is captured using a camera 120, which is part of an eye-tracking system 110. Camera 120 can be positioned at various locations on the head-mounted display 100 to capture images 190 of the user's eye 150 and prescription glasses 160 from different viewpoints. Camera 120 can be configured to capture images 190 at a high frame rate (e.g., 60-120 frames per second) to accurately track user eye movements and gaze direction.

[0078] In some examples, the method includes determining the location of reflection 180 in image 190. Reflection 180 is light emitted by illuminator 130 and reflected by lens 165 of prescription glasses 160. The location of reflection 180 in image 190 provides information about the curvature 169 and radius of curvature 170 of lens 165. The location of reflection 180 can be determined using various image processing techniques, such as edge detection and pattern recognition.

[0079] In one implementation, the method includes identifying an illuminator 130 corresponding to each reflection 180. The illuminator 130 is part of an eye-tracking system 110 and is configured to emit light that is reflected by a lens 165 of a prescription eyeglass 160. By identifying the illuminator 130 corresponding to each reflection 180, the method can determine the location of the reflection 180 in an image 190 and estimate the radius of curvature 170 of the lens 165.

[0080] In some configurations, the method includes determining the power of lens 165 of prescription glasses 160 by analyzing reflection 180. Reflection 180 is light emitted by illuminator 130 and reflected by lens 165 of prescription glasses 160. Reflection 180 provides information about the curvature 169 and radius of curvature 170 of lens 165. By analyzing reflection 180, the method can estimate the power of lens 165. The analysis of reflection 180 can be based on various assumptions, such as assuming that the outer surface 168 of lens 165 follows a spherical shape, and assuming that the distance 172 between the surface of the cornea and lens 165 is constant. The analysis of reflection 180 can be performed using various algorithms and machine learning techniques.

[0081] In some examples, the method includes using a base curve rule or curvature regression model to determine the power of lens 165 of prescription eyeglasses 160. Base curve rules and curvature regression models are techniques used to estimate the power of lens 165 based on its curvature 169 and radius of curvature 170. Base curve rules and curvature regression models can be combined with other techniques, such as image processing and machine learning, to accurately estimate the power of lens 165.

[0082] The base curve rule is a technique used to estimate the power of lens 165 based on its curvature 169 and radius of curvature 170. This method involves estimating the power of lens 165 using the correlation between its power and the radius of curvature 170. The correlation can be based on the base curve rule, such as... Figures 8a to 8b As shown. By utilizing this correlation, the method can estimate the power of lens 165.

[0083] In some implementations, the application of the base curve rule involves various assumptions and specifications. For example, it can be assumed that the outer surface 168 of the lens 165 follows a spherical shape. This assumption simplifies the analysis of the reflection 180 and the estimation of the power of the lens 165. It can also be assumed that the distance 172 between the surface of the cornea and the lens 165 is constant.

[0084] In one example, the method includes using a curvature regression model to estimate the diopter of the prescription glasses 160. The curvature regression model is a machine learning model trained to estimate the curvature 169 of the lens 165 based on reflections 180 in image 190. The curvature regression model can use a dataset including the user's eye 150 and the prescription glasses 160 (see, for example, image 190). Figures 9a to 9dThe image 190 is trained using real-world data (examples of this) and annotations specifying the curvature 169 and power of lens 165. Image 190 can be acquired by an eye-tracking camera (e.g., camera 120 imaging a user wearing prescription glasses 160). Therefore, known curvature and lens power values ​​are associated with image features, such as how reflection 180 is displayed in image 190, and relationships between known values ​​and image features can be established. Thus, a curvature regression model can be trained to identify these relationships. Therefore, the pre-trained regression model is able to estimate the curvature 169 of lens 165 based on reflection 180 in image 190. By using the curvature regression model, this method can accurately estimate the power of lens 165 based on reflection 180 in image 190.

[0085] In one example, the power of the lens 165 of the prescription glasses 160 is determined by analyzing the iris 156 of the user's eye 150. This process involves capturing multiple images 190 of each user's eye 150 and the prescription glasses 160 from different viewpoints, such as... Figure 10 As shown in the example, multiple candidate ellipses representing the same iris 156 are determined from these images 190. Triangulation of the candidate ellipses is then used to determine the center 157 and radius 158 of each iris 156 in 3D space. The deviation of the determined radius 158 of each iris 156 is analyzed to infer the power of the lens 165 of the prescription glasses 160.

[0086] In some implementations, the computer analysis system 140 uses a machine learning process to analyze image 190 and determine the power of lens 165. The machine learning process may include various steps, such as training a model, testing the model, and using the model to make predictions. The model can be trained using a dataset of image 190 including user eye 150 and prescription glasses 160, along with annotations specifying the power of lens 165. The model can be tested using a separate dataset of image 190 including user eye 150 and prescription glasses 160, along with annotations specifying the power of lens 165. The trained model is then used to estimate the power of lens 165 based on image 190 of user eye 150 and prescription glasses 160. The machine learning process can use various algorithms and techniques, such as linear regression, nonlinear regression, decision trees, neural networks, and deep learning. The machine learning process can be implemented using various software tools and libraries, such as TensorFlow, PyTorch, and scikit-learn. This disclosure has potential applications in various fields, including virtual reality, augmented reality, and mixed reality.

[0087] In one example, this disclosure is used for a virtual reality system. The virtual reality system includes a head-mounted display 100 that provides a virtual reality experience to a user. The head-mounted display 100 includes an eye-tracking system 110 that tracks the user's eye movements and gaze direction. The eye-tracking system 110 uses this disclosure to estimate the power of the user's prescription glasses 160, thereby improving the accuracy of eye tracking and enhancing the user's virtual reality experience.

[0088] In another example, this disclosure is used in an augmented reality system. The augmented reality system includes a head-mounted display 100 that overlays digital information onto a user's real-world view. The head-mounted display 100 includes an eye-tracking system 110 that tracks the user's eye movements and gaze direction. The eye-tracking system 110 uses this disclosure to estimate the power of the user's prescription glasses 160, thereby improving the accuracy of eye tracking and enhancing the user's augmented reality experience.

[0089] In summary, this disclosure provides a method and system for estimating the power of prescription glasses for a user of a head-mounted display. The disclosure includes illuminating the user's eye and the prescription glasses, capturing images of the eye, the prescription glasses, and reflections on the glasses, and determining the lens power of the prescription glasses by analyzing the reflections. This disclosure uses various techniques and methods, including base curve rules, curvature regression models, and machine learning, to accurately estimate the lens power. This disclosure has potential applications in various fields, including virtual reality, augmented reality, and mixed reality.

[0090] The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this disclosure. As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are also intended to include the plural forms. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. It will be further understood that when the terms “comprising” and / or “including” are used herein, they specify the presence of a feature, whole, action, step, operation, element, and / or component of the statement, but do not exclude the presence or inclusion of one or more other features, wholes, actions, steps, operations, elements, components, and / or groups thereof.

[0091] It should be understood that although the terms "first," "second," etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of this disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0092] In this document, relative terms such as "below," "above," "up," "down," "horizontal," or "vertical" may be used to describe the relationship between one element and another, as shown in the figures. It should be understood that these terms, and those discussed above, are also intended to cover different orientations of the device other than those depicted in the figures. It should be understood that when an element is referred to as "connected" or "linked" to another element, it may be directly connected or linked to the other element, or there may be intermediate elements present. In contrast, when an element is referred to as "directly connected" or "directly linked" to another element, there are no intermediate elements present.

[0093] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It should be further understood that, unless expressly defined herein, the terms used herein shall be interpreted as having the meaning consistent with their meaning in the context of this specification and the relevant field, and shall not be interpreted in an idealized or overly formal sense.

[0094] It should be understood that this disclosure is not limited to the aspects described above and shown in the accompanying drawings; in fact, those skilled in the art will recognize that many changes and modifications can be made within the scope of this disclosure and the appended examples and claims. Aspects have been disclosed in the drawings and specification for illustrative purposes only and not for limiting purposes, and the scope of this disclosure is set forth in the following claims.

[0095] Example

[0096] Example 1. A method for estimating the prescription glasses prescription of a user of a head-mounted display (100), the method comprising:

[0097] Illumination (130) is used to illuminate the user's eyes (150) and prescription glasses (160);

[0098] Images (190) of the eye (150), prescription glasses (160), and reflections (180) on the prescription glasses (160) are captured using at least one camera (120); and

[0099] The power of the lens (165) of the prescription glasses (160) is determined by analyzing these reflections.

[0100] Example 2. The method according to Example 1 further includes setting camera brightness control parameters of at least one camera (120) to capture reflections (180) on prescription glasses (160), the camera brightness control parameters including at least one of exposure time or camera gain.

[0101] Example 3. The method according to Example 1 or 2 further includes determining the position of reflections (180) in the image (190) and identifying the illuminator (130) corresponding to each reflection (180).

[0102] Example 4. The method according to any one of Examples 1 to 3, wherein determining the power of the lens (165) includes using the correlation between the power of the lens (165) and the radius of curvature (170) of the surface of the lens (165), wherein the radius of curvature (170) of the lens (165) is estimated based on the position of the reflection (180) in the image (190) and the position of the corresponding illuminator (130).

[0103] Example 5. According to the method of Example 4, wherein the reflection (180) used to estimate the radius of curvature (170) of these lenses (165) is a reflection (180) on the outer surface (168) of the lens (165).

[0104] Example 6. The method described in Example 4 or 5, wherein the correlation is a base-arc rule.

[0105] Example 7. The method according to Examples 4 to 6 further includes approximating the surface of the lens (165) as a spherical shape.

[0106] Example 8. The method according to any one of Examples 1 to 7 further includes determining the power of each of the right lens (167) and the left lens (166) in a pair of prescription eyeglasses (160).

[0107] Example 9. The method according to any one of Examples 1 to 3 further includes using image processing of the captured image (190) to estimate the curvature (169, 169a, 169b) of the outer surface (168) and / or inner surface (178) of the prescription glasses (160) for each lens (165).

[0108] Example 10. The method according to Example 9 further includes using a pre-trained regression model to estimate the curvature (169, 169a, 169b) through a machine learning process.

[0109] Example 11. The method according to Example 10, wherein the machine learning process includes at least one of linear regression or nonlinear regression.

[0110] Example 12. The method according to any one of Examples 10 or 11, wherein a pre-trained regression model is trained using labeled features, including the curvature and reflection (180) of the outer surface (168) and / or inner surface (178) of the lens from an image (190) of a user wearing prescription glasses.

[0111] Example 13. A head-mounted display (100), comprising:

[0112] An eye-tracking system (110) configured to determine the gaze direction and entrance pupil position of a user's eye (150);

[0113] At least one camera (120) is configured to capture an image (190) of a user's eye (150) and prescription glasses (160);

[0114] A illuminator (130) configured to illuminate a user's eyes (150) and prescription glasses (160); and

[0115] A computer analysis system (140) is configured to analyze the captured image (190) and determine the power of the lens (165) of the prescription glasses (160) by analyzing the reflection (180) on the prescription glasses (160).

[0116] Example 14. The head-mounted display (100) according to Example 13, wherein the eye-tracking system (110) is configured to perform pupil-central corneal reflex (PCCR) eye tracking.

[0117] Example 15. A head-mounted display (100) according to Example 13 or 14, wherein the illuminator (130) is an LED illuminator with a wavelength range of 750 nm to 1400 nm.

[0118] Example 16. The head-mounted display (100) according to any one of Examples 13 to 15, wherein the computer analysis system (140) is configured to optimize camera brightness parameters including at least one of exposure time or camera gain to capture reflections (180) on the outer surface (168) and / or inner surface (178) of the prescription glasses (160).

[0119] Example 17. The head-mounted display (100) according to any one of Examples 13 to 16, wherein a computer analysis system (140) is configured to identify the illuminator (130) corresponding to each reflection (180) and determine the curvature of the lens (169, 169a, 169b) by analyzing the reflection (180).

[0120] Example 18. The head-mounted display (100) according to Example 17, wherein a computer analysis system (140) is configured to approximately calculate the radius of curvature (170) and estimate the power of the lens (165) based on the curvature (169, 169a, 169b).

[0121] Example 19. The head-mounted display (100) according to any one of Examples 17 or 18, wherein the computer analysis system (140) is configured to use a pre-trained regression model to estimate the curvature (169, 169a, 169b) of the outer surface (168) and / or inner surface (178) of the prescription glasses (160) for each lens (165).

Claims

1. A method for estimating the diopter of prescription glasses (160) for a user of a head-mounted display (100), the method comprising: Images (190) of each of the user's eyes (150) and the prescription glasses (160) are captured from different viewpoints; Multiple candidate ellipses representing the same iris (156) are identified from the captured image (190); The center (157) and radius (158) of each iris (156) in 3D space are determined using triangulation of the candidate ellipse; and The deviation of the determined radius (158) of each iris (156) is analyzed to infer the power of the lens (165) of the prescription glasses (160).

2. The method according to claim 1, wherein, Determining each ellipse involves using edge detection to fit the ellipse to each iris (156).

3. The method according to claim 1, wherein, Determining each ellipse involves using an elliptic regression model.

4. The method according to any one of claims 1 to 3, wherein, The deviation of the radius (158) of each iris (156) is the difference in size between the radius (158) and the estimated radius of the human average iris.

5. The method according to claim 4, wherein, The average human iris radius is 5.5 mm.

6. The method according to any one of claims 1 to 5, wherein, The power of the lens (165) is determined based on a pre-calculated relationship between the power of the lens (165) of the prescription glasses (160) and the deviation of the determined radius (158) of the iris (156).

7. The method according to claim 6, wherein, The pre-calculated relationship is determined through a machine learning process, wherein the iris radius estimated before refraction through the lens (165) is known.

8. The method according to claim 7, wherein, The machine learning process includes a regression model.

9. A head-mounted display (100), comprising: An eye-tracking system (110) configured to determine the gaze direction of a user's eyes (150); At least one camera (120) configured to capture images (190) of each eye (150) of the user and the prescription glasses (160) from different viewpoints; and Computer analysis system (140), the computer analysis system being configured to Multiple candidate ellipses representing the same iris (156) are identified from the captured image (190); The center (157) and radius (158) of each iris (156) in 3D space are determined using triangulation of the candidate ellipse. The deviation of the determined radius (158) of each iris (156) is analyzed to infer the power (171) of the lens (165) of the prescription glasses (160).

10. The head-mounted display (100) according to claim 9, wherein, The computer analysis system (140) is further configured to estimate the difference between the radius (158) and the average human iris radius.

11. The head-mounted display (100) according to claim 9, wherein, The computer analysis system (140) is further configured to determine the power of the lens (165) of the prescription glasses (160) based on a pre-calculated relationship between the power of the lens (165) of the prescription glasses (160) and the deviation of the determined radius (158) of each iris (156).

12. The head-mounted display (100) according to claim 11, wherein, The pre-calculated relationship is determined through a machine learning process, wherein the iris radius (158) estimated before refraction through the lens (165) is known.

13. The head-mounted display (100) according to claim 12, wherein, The machine learning process includes a regression model.

Images