Reconstruction of entrance pupil using image data

Abstract

Embodiments of the present disclosure relate to entrance pupil reconstruction using image data. Embodiments of reconstructing human eye entrance pupil are provided. One aspect includes an apparatus including a camera system and processing circuitry coupled to a memory storing instructions that, when executed by the processing circuitry, cause the processing circuitry to receive image data from the camera system and extract a feature set from the image data. Extracting the feature set includes classifying the features into a plurality of feature types. The processing circuitry is further configured to calculate a set of feature measurements from the extracted set of features and perform an entrance pupil reconstruction process to determine entrance pupil position data using the set of feature measurements.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular, to the reconstruction of an entrance pupil using image data. Background Art

[0002] Extended reality (XR) applications (e.g., those in augmented reality (AR), virtual reality (VR), and mixed reality (MR)) typically use head-mounted devices to provide users with digital information that integrates with or replaces the physical world, creating an immersive experience. Accurately presenting digital information to the user's eyes (or binoculars) depends on various factors, including knowledge of projection optics and the characteristics and position of the user's eyes (or binoculars). Summary of the Invention

[0003] Embodiments of a method for reconstructing an entrance pupil of a human eye are provided. One aspect includes a device comprising a camera system and processing circuitry coupled to a memory storing instructions that, when executed by the processing circuitry, cause the processing circuitry to receive image data from the camera system and extract a feature set from the image data. Extracting the feature set includes classifying the features into a plurality of feature types. The processing circuitry is further configured to calculate a feature measurement set from the extracted feature set and perform an entrance pupil reconstruction process to determine entrance pupil position data using the feature measurement set.

[0004] This summary is provided to introduce some concepts in a simplified form that are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that address any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 A schematic diagram of an example display system for reconstructing the entrance pupil of a human eye is shown.

[0006] Figure 2 A block model showing an example process for converting image data into entrance pupil position data, which can be used Figure 1 The examples show the system to be implemented.

[0007] Figure 3 A block model of an example feature measurement process is shown, which can be used as Figure 2 is implemented as part of the example process depicted in .

[0008] Figure 4 A block model of an example stereo corneal entrance pupil reconstruction process is shown, which can be used as Figure 2is implemented as part of the example process depicted in .

[0009] Figure 5 A block model of an example stereo limbus-corneal entrance pupil reconstruction process is shown, which can be used as Figure 2 is implemented as part of the example process depicted in .

[0010] Figure 6 A block model of an example single corneal entrance pupil reconstruction process is shown, which can be used as Figure 2 is implemented as part of the example process depicted in .

[0011] 7A to 7F An example pupil shape derived from two-dimensional data of pupil features is shown, which can be used to Figure 6 The example process is depicted in .

[0012] Figure 8A and Figure 8B The graphs show the loss values ​​and entrance pupil position errors for different pupil shapes, which can be used to Figure 6 The example process is depicted in .

[0013] Figure 9 A block diagram illustrating an example decision process for determining which entrance pupil reconstruction process to perform, which may be used as Figure 2 is implemented as part of the example process depicted in .

[0014] Figure 10 A flowchart illustrating an example method for converting image data into entrance pupil position data, which may be used Figure 1 The examples show the system to be implemented.

[0015] Figure 11 A schematic diagram of an example computing environment in which the Figure 1 computer equipment. DETAILED DESCRIPTION

[0016] The performance of an XR system (e.g., a head-mounted eye display) in projecting digital information to a user's eyes (binoculars) may depend on various factors. Generally speaking, the projection of digital information in an XR system is limited in performance by the amount / efficiency of the projected light reaching the user's eyes (binoculars). For example, an XR system's projection system projects digital images into areas that the user cannot actually see, resulting in low light efficiency. Therefore, in such applications, knowledge of the entrance and exit pupils can provide more accurate and efficient configurations. In addition to image projection, various other applications can benefit from this knowledge, including but not limited to dynamic distortion compensation and dynamic color compensation.

[0017] Current methods for configuring XR systems to adapt to a user's entrance pupil(s) include simple implementations in which eye parameters are assumed and fixed to derive the entrance pupils used to configure various applications. Eye parameters can refer to eye characteristics that remain constant during use of the XR system. For example, pupil-corneal distance, pupil size, and / or corneal curvature are often assumed in XR systems. Typically, the population's mean or median parameter values ​​are used as the assumed parameters. However, because the assumed parameters do not accurately represent the population, this approach can perform differently across users, leading to reconstruction errors. Users whose eye characteristics deviate from the mean / median values ​​may perform poorly. Furthermore, current methods for reconstructing the entrance pupil are based on a single-point abstraction. However, the entrance pupil is a three-dimensional shape, so a single-point abstraction does not provide an accurate optical simulation. Other methods for providing entrance pupil information include performing an initial calibration of the user's eye parameters, which can be used to derive the entrance pupil. However, during use of an XR system (e.g., smart glasses), the user's eye(s), and therefore the entrance pupil, may move relative to the XR system. In such cases, the ability to gather real-time knowledge of the user's eye position can be advantageous in correcting runtime errors.

[0018] In view of the above observations, systems and methods are provided for reconstructing the entrance pupil of a human eye using image data. In short, a parameter-free method that does not rely on assumed user eye parameters can be implemented to reconstruct the entrance pupil using image data to provide a robust method that covers a larger population coverage than methods that utilize assumed eye parameters. Image data from a camera system configured to image the user's eyes (both eyes) can be provided in real time. For example, a head-mounted XR system can include an embedded camera configured to image one or both eyes of a user. Therefore, reconstruction of the entrance pupil can also be performed in real time, thereby achieving self-calibration and more accurate entrance pupil reconstruction over time.

[0019] Entrance pupil reconstruction can be performed in various ways. In some embodiments, the image data provided of the user's eye(s) is analyzed and features of the user's eye(s) can be extracted. The extracted features can be classified into various feature types corresponding to eye features, including but not limited to pupil, reflection, and limbus. Once classified, these features can be measured to provide feature measurements for deriving the entrance pupil position corresponding to the eye from which the image data was taken. In some embodiments, the feature measurements are quantitatively and / or qualitatively evaluated to determine their effective use in the entrance pupil reconstruction process. Missing features (or features of poor quality) can be replaced with assumed parameters or previously saved parameters. The derived entrance pupil position data can then be used for various XR applications, including the applications described in the above sections.

[0020] Turning now to the drawings, the entrance pupil reconstruction method is described in more detail. Figure 1A schematic diagram of an example display system 100 for reconstructing the entrance pupil of a human eye is shown. The example display system 100 includes a camera system 102 for providing image data 104 for reconstructing the entrance pupil. The example display system 100 can be any display system in which entrance pupil position data is utilized to provide various optical applications, including but not limited to dynamic distortion compensation and dynamic color compensation. For example, the example display system 100 can be a head-mounted display device, such as a heads-up display system, smart glasses, etc.

[0021] The camera system 102 can be implemented in various ways. Different types of cameras can be used, such as, but not limited to, visible light cameras and infrared cameras. The camera system 102 includes a camera configured to image the user's eyes. In some embodiments, the camera system 102 includes two cameras for imaging the user's eyes, thereby providing stereoscopic image data for the eyes. Similar embodiments can be provided for each of the user's eyes (e.g., two cameras each capture corresponding image data for each of the user's eyes, or two sets of two cameras can provide stereoscopic image data for each of the user's eyes).

[0022] exist Figure 1 In the depicted example, the entrance pupil reconstruction process is performed by a computing device 106 of the example display system 100. Any type of computing device can be implemented. In the example display system 100, the computing device 106 includes a processing circuit system 108 and a memory 110 storing instructions that, when executed by the processing circuit system 108, cause the processing circuit system 108 to perform the various processes described herein. The process includes receiving image data 104 provided by the camera system 102. The image data 104 can be in various formats. For example, the camera system 102 implemented using a visible light camera can provide image data 104 including RGB images. In some embodiments, the image data 104 includes more than one image of a given eye, which can provide a stereoscopic view of the given eye. The process described herein is described for one eye of a user, but can be similarly implemented for a second eye. For example, the image data 104 can include images of both eyes of the user.

[0023] The computing device 106 includes a feature extraction module 112 for performing a feature extraction process on the received image data 104. The feature extraction process can include extracting a feature set from the image data 104 and classifying the features into a plurality of feature types, thereby obtaining a classified feature set 114. The classification can be performed using different feature types corresponding to eye features, such as, but not limited to, pupil, reflective features, and limbus. A feature measurement module 116 can be implemented to perform a feature measurement process to calculate a feature measurement set 118 based on the classified feature set 114. The feature measurement set 118 can include feature measurements for each feature type. For example, the feature measurement set 118 can include one or more pupil feature measurements, one or more reflective feature measurements, and / or one or more limbus feature measurements. In some embodiments, the feature measurement module 116 is implemented to perform a quantitative assessment and a qualitative assessment on the classified feature set 114.

[0024] The entrance pupil reconstruction module 120 can be implemented to perform an entrance pupil reconstruction process to determine entrance pupil position data 122 using the feature measurement set 118. The entrance pupil position data 122 can include one or more entrance pupil positions that can be used by the example display system 100 for various applications. The entrance pupil reconstruction process can be performed on feature measurements derived from image data containing a single image or stereo images of a user's eyes. The process can also be performed on feature measurements derived from image data containing both eyes of the user. In some embodiments, the entrance pupil position data 122 is determined using a ray tracing process.

[0025] The entrance pupil reconstruction process can be performed in a variety of ways, which can depend on the type and number of feature measurements 118 provided by the feature measurement model 116. For example, the entrance pupil reconstruction process can include reconstructing a pupil model, reconstructing a corneal model, reconstructing a limbal model, and / or determining a corneal refractive index using the individual feature measurements provided. In some embodiments, locally stored eye parameters for a given user or assumed parameters (e.g., average or median parameters corresponding to average or median eye characteristics for a population) are used to replace certain feature measurements. For example, missing features or poor quality feature measurements can be replaced with parameter data retrieved from a database 124 storing parameters for user profiles. After the entrance pupil is reconstructed for a given user, the parameters associated with the reconstructed entrance pupil can be used to update parameters stored in a database 124 that stores parameters associated with one or more users.

[0026] In some embodiments, the entrance pupil reconstruction process includes estimating the pupil position based on two-dimensional pupil feature measurements derived from image data containing a monoscopic image of the user's eye. The estimated pupil position can be used to determine a set of candidate entrance pupil positions. Different techniques can be used to select the candidate entrance pupil positions from the candidate set. In some embodiments, a loss function is applied to the corresponding pupil shape of each candidate entrance pupil position. The resulting loss value can be used to select the candidate entrance pupil position or to iteratively optimize the process. For example, the candidate entrance pupil position corresponding to the pupil shape with the lowest loss value can be selected.

[0027] Figure 1 Illustrated is an overview of the data flow in a method for converting image data 104 into entrance pupil position data 122. The different processes described may be performed in various ways, as will be described in more detail below. Figure 2 A block model of an example process 200 for converting image data into entrance pupil position data is shown, which illustrates a logical overview of the example process 200. The example process 200 begins with image data 104, which can be provided by various sources in various formats. For example, the image data 104 can be received from a camera system, such as Figure 1 The example display system 100 depicts a camera system 102. Different camera systems may produce image data 104 in different formats (e.g., a visible light camera may provide an RGB image). The image data 104 may include one or more images of one or both eyes of a user. Depending on the content of the image data 104, different processes may be performed.

[0028] A feature extraction process 202 may be performed on the image data 104. For image data 104 containing more than one image, the feature extraction process 202 may be performed on each image in the image data 104. Various techniques may be implemented to perform the feature extraction process 202. In some embodiments, an image segmentation network is implemented. For example, a neural network for performing image segmentation may be implemented to extract features from an input image. Other techniques may also be used to perform feature extraction, such as, but not limited to, edge detection, shape detection, image thresholding, and the like.

[0029] Once features are extracted, classification can be performed to label the extracted features with a feature type. Any feature type can be implemented. In some embodiments, the extracted features are classified as a reflective feature type, a pupil feature type, or a limbus feature type. Feature types for other eye characteristics can also be utilized. In some examples, features that do not fall into any predefined categories may be discarded. Various techniques can be implemented to classify features. For example, a neural network can be implemented to classify the extracted features.

[0030] Following the feature extraction process 202, a feature measurement process 204 can be performed on the classified features to calculate a set of feature measurements. The feature measurement process 204 can be performed in a variety of ways. In some embodiments, the feature measurement process 204 includes both a quantitative assessment and a quality assessment of the classified features. For example, each feature type can be counted, and the quality of each feature can be determined. In some embodiments, the quality assessment provides information to determine a valid count for each feature type, which can be used to determine which entrance pupil reconstruction method to employ. Figure 3 An example feature extraction process is depicted and described in more detail.

[0031] continue Figure 2 In the example process 200, an entrance pupil position reconstruction process 206 can be performed using at least the feature measurement values ​​calculated in the feature measurement process 204 to generate entrance pupil position data 122. The entrance pupil position data 122 can be formatted in various ways and include various components. In some embodiments, the entrance pupil position data 122 includes data describing the entrance pupil position of a given eye of a user. In further embodiments, the entrance pupil position data 122 includes two entrance pupil positions, one for each of the user's two eyes.

[0032] Various techniques and methods may be implemented in the entrance pupil position reconstruction process 206 . Figures 4 to 6 An example of an entrance pupil location reconstruction process is depicted and described in more detail. In some embodiments, the valid count of a feature type can be used to determine the method to be implemented. For example, a valid count of more than two pupil features can indicate that the image data 104 includes stereoscopic image data, and a corresponding method for calculating the entrance pupil location using the stereoscopic image can be implemented. FIG7 depicts and describes in more detail an example of a decision tree for determining which method to implement based on the valid count of a feature type.

[0033] Figure 2The depicted entrance pupil position reconstruction process 206 can be performed by using parameters to compensate for missing features or features of poor quality in the image data 104. For example, parameters corresponding to the average or median eye characteristics of a population can be used to replace missing features or features of poor quality. In some embodiments, parameters associated with a user profile retrieved from a database 124 storing such information are used. Such parameters can be generated by a calibration process and / or provided by a previous entrance pupil position reconstruction process. For example, eye parameters 208 determined by the entrance pupil position reconstruction process 206 can be used to update the parameter database 124 for the user profile associated with the current user. The utilized eye parameters can include the corneal profile (e.g., curvature, lengths of the major and minor axes, refractive index, etc.), the pupil profile (e.g., shape, size at a specific display brightness, etc.), the limbus profile (e.g., shape, size, etc.), and various geometric relationships between the aforementioned eye features (e.g., distance between the corneal center and the pupil plane, distance and normal vector between the limbus and the pupil, distance between the corneal center and the limbus plane, etc.).

[0034] Figure 3 A block model of an example feature measurement process 300 is shown. The example feature measurement process 300 may be implemented, for example, as Figure 2 304. In the depicted example, process 300 begins with a classified feature set 302 that includes three predefined classes—reflective 304, pupil 306, and limbus 308. Other feature types may also be implemented. For each feature type, a quantity evaluation 310 and a quality evaluation 312 may be performed. The quantity evaluation 310 may be performed to provide a count of the number of each feature type detected. The quality evaluation 312 may be performed to determine features of poor quality, which may provide a valid count for each feature type that may be used to determine which entrance pupil position reconstruction method to employ. The collective information from the quantity evaluation 310 and the quality evaluation 312 provides an output of feature measurement data 314.

[0035] The quality assessment 312 may be performed differently for different feature types. For reflections, the quality assessment 312 may include determining whether adjacent reflections are too close. In some embodiments, the quality assessment 312 for reflections includes, for each reflection, i , calculate its size based on Gaussian distribution (s i ). Locate the nearest adjacent reflection (reflection j Located at, and calculate the distance between the two (d ij For pupils, the quality assessment 312 may include calculating, for each pupil feature, the percentage of its area in the entire image ( iIn some embodiments, the quality assessment 312 for pupil features includes providing a confidence score (pconf i ), the confidence score can be provided by the feature extraction process. For the limbus feature, the quality assessment 312 may include calculating the percentage of the area of ​​each limbus feature to the entire image (l i In some embodiments, the quality assessment 312 for the limbus feature includes providing a confidence score (lconf i ), the confidence score can be provided by the feature extraction process.

[0036] In some embodiments, the quality assessment 312 includes calculating an effective count for each feature type. Different methods can be used for different feature types. For example, for each reflection i that is too close to the nearest neighbor reflection j, the count of the reflection feature provided by the quantity assessment can be reduced to produce an effective count of the reflection feature. In some embodiments, if the distance between the reflections (d ij ) is greater than the combined size of the two reflections (s i +s j As a further example, the size of the pupil feature and the limbus feature and / or their confidence level may not reach a predetermined threshold (e.g., i >0.01 and pconf i >0.9, and l i >0.01 and lconf i >0.9) to reduce the counts of pupil features and limbal features, respectively.

[0037] The valid count can be used for various purposes, including determining which entrance pupil reconstruction process to utilize. As described above, different entrance pupil reconstruction processes can be implemented, the selection of which can depend on the type, quantity, and / or quality of the feature measurements. For example, in the case where the valid count for the pupil feature is two, a stereo process for analyzing image data comprising a stereo image is implemented. Figures 4 to 6 Illustrated are different entrance pupil reconstruction processes that can be performed based on different valid counts for certain feature types.

[0038] Figure 4 A block model of an example stereo cornea entrance pupil reconstruction process 400 is shown. The example process 400 may be implemented, for example, as Figure 22. In some embodiments, the example stereo corneal entrance pupil reconstruction process 400 is performed when the valid count of the pupil feature is two (or at least two) and the valid count of the limbus feature is less than two. For example, the example process 400 begins with a feature measurement set 402 provided by a feature measurement process. The feature measurements 402 include at least one or more reflectance measurements 404 and at least two pupil measurements 406.

[0039] One or more reflectance measurements 404 may be used to perform a corneal model reconstruction process 408. The corneal model may be modeled as a three-dimensional ellipsoid, as given by X′=R(XC), where R and C are rotation and translation matrices, respectively, and X′=(x′, y′, z′) and X=(x, y, z). Based on the number of reflectance measurements 404, a corresponding corneal model can be solved to estimate the corneal model. For example, when the number of reflectance measurements N=2 / 5 / 7, a corneal model reconstruction process 408 can be performed to reconstruct a sphere / oblate spheroid or a prolate spheroid / ellipsoid, respectively. Generally speaking, a greater number of reflectances can improve the stability of the corneal model reconstruction.

[0040] The pupil measurements 406 can be used to perform a pupil model reconstruction process 410. Using stereo image data (from two cameras imaging one eye), the pupil model can be reconstructed using the cornea model reconstructed according to the above-described cornea model reconstruction process 408. Parameters of the reconstructed pupil model can include, but are not limited to, pupil shape, pupil size (at the time the image data was acquired), and distance from the center of the cornea model.

[0041] The pupil model reconstruction process 410 may include determining two pupil position contours pp1 and appearance pp2 , which can be obtained by corneal refraction (using the corneal model provided by the corneal model reconstruction process 408 discussed above). The pupil position can be determined by finding the two pupil position contours pp1 and appearance pp2 The pupil model is reconstructed using the intersection of the pupil and the pupil. This results in two possible solutions. To find the correct solution, the following conditions need to be considered. First, the correct solution is a two-dimensional plane (also called the pupil plane). The second condition consists of verifying that the vector p*c* is perpendicular to the pupil plane, where the pupil center p* is defined as the geometric center of the intersection, and the cornea center c* is defined using the cornea model described above.

[0042] A ray tracing process 412 can be performed using the cornea model and the pupil model to determine entrance pupil position data 414. In some embodiments, the example stereo cornea entrance pupil reconstruction process 400 is performed on feature measurements 402 generated from image data of two eyes, and the entrance pupil position data 414 includes the entrance pupil positions of both eyes. The ray tracing process 412 can include setting an observation point OP on one of the cameras for a given cornea center and pupil center (derived from the cornea model and pupil model, respectively), thereby allowing the refraction point R1 to be calculated. The observations described herein can be similarly applied to two cameras configured to capture stereo images. The neighborhood of the observation point OP can be sampled in the camera plane (the normal vector is the camera pointing vector). Given a sampled neighborhood point OP i , i = 0, 1, 2, 3..., the entrance pupil position EPP that meets the following conditions can be calculated. From EPP to OP i Line i ) is refracted by the cornea to form a set of refractive lines (lines' i ). A point P can be defined as the distance from all lines ' i nearest point, and d* can be defined as all lines from P to ' i The root mean square distance of . P and d* can be solved using the pseudo-inverse method or the least squares method as follows:

[0043]

[0044] Available online EPP Find point Q on the line, where Q = R1 + t * line EPP .vector (where t is a one-dimensional parameter) that minimizes ||Pp*||. Therefore, Q is an EPP that satisfies the following conditions:

[0045]

[0046] For static viewpoints, the viewpoint OP can be changed to a specified viewpoint. For dynamic viewpoints, the optical axis of the eye can be solved using the corneal center c* (which can be provided by a reconstructed corneal model) and the pupil center p* (which can be provided by a reconstructed pupil model). By placing the viewpoint OP on the optical axis, the above formula can be used to obtain the entrance pupil position based on the dynamic viewpoint.

[0047] Using additional feature measurements and feature measurements for specific feature types, the entrance pupil location can be solved more accurately. Figure 5 A block model of an example stereo limbus-corneal entrance pupil reconstruction process 500 is shown. The example process 500 may be implemented, for example, as Figure 2In some embodiments, when the valid counts of pupil features and limbus features are two (or at least two), the example stereo limbus-corneal entrance pupil reconstruction process 500 is implemented. Regarding the reconstruction of the cornea model and pupil model using the glint feature and pupil feature, respectively, a portion of the example process 500 can be used in conjunction with Figure 4 In addition, using the provided limbus features 502, the process 500 further includes a limbus model reconstruction process 504.

[0048] The reconstruction of the limbus model can be achieved by finding the two limbus shapes (shapes 角膜球1 and appearance 角膜球2 ). In order to find the correct solution when there are multiple solutions, the following conditions need to be considered. First, the correct solution is a two-dimensional plane (also called the limbus plane). The second condition is that the dot product between the normal vector of the limbus plane and the normal vector of the pupil plane is minimal (min(N 角膜缘 , N 瞳孔 )). The limbus plane and pupil plane can be aligned (N 角膜缘 =N 瞳孔 In addition to the corneal model and pupil model, the corneal refractive index can also be used in the ray tracing process 506 to determine the entrance pupil position data 508.

[0049] Leveraging two stereo cameras imaging the user's eyes and knowledge of the camera positioning, the stereo information can be used to more accurately reconstruct a pupil model and / or limbus model compared to image data from a single camera. However, in some applications, such data may not be available. For example, limited space, power, or cost budgets may necessitate an XR device to be equipped with only one imaging camera for each eye of the user. Figure 6 A block model of an example single corneal entrance pupil reconstruction process 600 is shown. The example process 600 may be implemented, for example, as Figure 2 Entrance pupil position reconstruction process 206 described in

[0066] In some embodiments, when the valid count of pupil features is one and the limbus feature is missing, the example single corneal entrance pupil reconstruction process 600 is implemented. For example, the example process 600 starts with a feature measurement set 402 provided by a feature measurement process. The feature measurement 402 includes a pupil measurement 602 and at least one or more reflectance measurements 404. Because there is only one valid pupil measurement 602, the pupil information is provided as two-dimensional information. The center J of the pupil image corresponds to the two-dimensional projection of the entrance pupil position on the camera. The line J is obtained by back-projecting J. EPP , where the line EPP Based on the pupil measurement values ​​602 , a set of candidate entrance pupil positions is determined 604 .

[0050] Similar to the reference above Figure 4 A corneal model reconstruction process 408 is performed as described using the reflectance measurement(s) 404. A ray tracing process 606 can be performed using the reconstructed corneal model and the candidate entrance pupil position candidates to select corresponding pupil candidates 608 based on a loss function, which provides entrance pupil position data 610. The pupil candidates 608 can also be used to optimize the process for determining candidate entrance pupil positions.

[0051] The ray tracing process 606 may include setting an observation point OP0 at the camera pinhole position. The neighborhood of the observation point OP may be sampled in the camera plane (the normal vector is the camera pointing vector). Given a sampled neighborhood point OP i , i=0, 1, 2, 3..., the entrance pupil position EPP that satisfies the following conditions can be calculated. From EPP to ON i Line i ) is refracted by the cornea to form a set of refractive lines (lines' i ), the set of refracted lines intersects point P (the actual position of the pupil (PPP)). Point P should be on line '0 (where line '0 is line ' PPP , and after refraction through the corneal model is line ' EPP ). Point P is to all lines' i nearest point, and d* can be defined as all lines from P to ' i The root mean square distance of . P and d* can be solved using the pseudo-inverse method or the least squares method as follows:

[0052]

[0053] Available online EPP Find the point Q on the line that minimizes d* (which can be represented by a one-dimensional parameter t).

[0054]

[0055] Then point Q is EPP, and the corresponding P is PPP.

[0056] In some embodiments, pupil shape constraints are enforced. Pupil shape constraints may include, but are not limited to, pupil in-plane and geometric center invariance (focus = geometric center of pupil shape). An observation point OP may be set at the camera pinhole position. Thus, EPP = OP + t* line EPP Vectors and lines EPP Generate a refraction point A in the cornea model. The neighborhood of the observation point OP can be sampled to obtain OP i , i = 0, 1, 2, 3... From EPP to OP i Linei ) is refracted by the cornea to form a set of refractive lines (lines' i ), the set of refracted lines intersects with point P (the actual position of the pupil (PPP)). Using the above method, point P can be determined. Let N = Pc, where c is the center of the cornea model. The plane PL can be obtained: N*P=d, where the plane normal vector is N and a point on the plane is P. The detected pupil edge point e i The image of (i=1, 2, 3, ...) is back-projected to obtain (i=1, 2, 3, ...). pe in plane PL 线 The intersection point is (i=1, 2, 3, ...), where The geometric center of is pg. Then, the loss function can be defined as d * =||P-pg||, and d * Can be minimized relative to t (EPP = A + t * line EPP ·vector).

[0057] The single limbus-corneal entrance pupil reconstruction process can also be implemented as Figure 2 In some embodiments, when the valid counts of the pupil feature and the limbus feature are both one, a single limbus-corneal entrance pupil reconstruction process is performed. A portion of the process may be performed with Figure 5 The stereoscopic limbus-corneal entrance pupil reconstruction process 500 is similarly implemented. In addition, the limbus feature can be used in the entrance pupil reconstruction process. The center L of the limbus captured by the camera is projected back to obtain the line 角膜缘 .Wire EPP With line PPP Generate refraction points on the cornea model. You can find the line PPP and line 角膜缘 The nearest point P is taken as the intersection point. Point P can be set as the pupil center, and the pupil model can be reconstructed by the following steps. Let N = Pc, where c is the center of the cornea model. The plane PL can be obtained: N*P = d, where the plane normal vector is N and a point on the plane is P. The detected pupil edge point e i The image of (i=1, 2, 3, ...) is back-projected to obtain (i=1、2、3、…).pe 线 The intersection point in plane PL is (i=1、2、3、…).pe pts Provides the reconstructed pupil shape.

[0058] 7A to 7FExample pupil shapes derived from two-dimensional data of pupil features are shown. These example pupil shapes are reconstructed for different candidate entrance pupil positions. As shown, the closer the shape is to orthogonal symmetry, the smaller the loss function value. Other loss functions can also be implemented. For example, the loss function can take into account curvature, roundness, and other properties.

[0059] Figure 8A and Figure 8B Graphs showing loss values ​​and entrance pupil position errors for different pupil shapes are shown. The data results shown are from an example XR system and are provided for conceptual purposes. Different pupil shapes are represented by a one-dimensional parameter t. In the depicted example, when t is approximately 3.094 ( Figure 8A ), the minimum loss value is found, which indicates symmetry and corresponds to a minimum value of the entrance pupil position error of approximately five microns ( Figure 8B ).

[0060] The selection of the various entrance pupil reconstruction processes described above can be based on various factors, including Figures 4 to 6 For example, you can use different valid counts for each feature type to determine which process to implement. Figure 9 A block diagram of an example decision process 900 for determining which entrance pupil reconstruction process to perform is shown. The process 900 begins by first determining a valid count of a pupil feature 902. If the valid count of the pupil feature is at least two, then a stereo process is performed. In this case, the decision process continues by determining a valid count of a limbus feature 904. If the valid count of the limbus feature is at least two, then a stereo limbus-corneal entrance pupil reconstruction process 906 may be performed (which may be a Figure 5 If the valid count of the limbus feature is less than two, a stereo corneal entrance pupil reconstruction process 908 may be performed (which may be Figure 4 Stereo corneal entrance pupil reconstruction process 400).

[0061] Returning to step 902, if the valid count of the pupil feature is less than two, a single process of the type is implemented. The decision process continues by determining the valid count of the limbus feature 910. If the valid count of the limbus feature is at least one, a single limbus-corneal entrance pupil reconstruction process 912 may be performed. If the valid count of the limbus feature is zero, a single corneal entrance pupil reconstruction process 914 may be performed (which may be Figure 6 Single corneal entrance pupil reconstruction process 600).

[0062] Figure 10A flow chart of an example method 1000 for converting image data into entrance pupil position data is shown. Method 1000 includes receiving image data in step 1002. The image data can be received and provided in various ways. The image data can include images of one or both eyes of a user. In some embodiments, the image data is received from a camera system having two cameras for imaging each eye of the user (i.e., at least four cameras in total). In this case, the image data can include stereoscopic image data, which can include multiple images of the same eye captured from different angles. The image data can also be received from a camera system having only a single camera for imaging each eye of the user (i.e., two cameras in total). In this case, the image data can be a single-view image of any given eye.

[0063] At step 1004, method 1000 includes extracting a feature set from the image data. For image data comprising multiple images, a feature extraction process can be performed on each image in the image data. Example feature extraction processes include image segmentation neural networks, edge detection, shape detection, and image thresholding. The extracted feature set can be classified into multiple feature types. Example feature types can include, but are not limited to, a reflective feature type, a pupil feature type, and a limbus feature type. Feature types for other eye features can also be utilized. Various processes can be used to classify the features, including, but not limited to, using a neural network to classify the extracted features.

[0064] In step 1006, method 1000 includes calculating a set of feature measurements from the extracted feature set. Calculating the set of feature measurements may include performing a quantity assessment and / or a quality assessment. The quantity assessment may include counting each feature type. A quality assessment may be performed to provide confidence information for determining whether the feature measurements are reliable. In some embodiments, the quality assessment provides confidence information for determining a valid count for each feature type. For each feature measurement that does not meet a predefined confidence threshold level, the count for the corresponding feature type is deducted. The valid count for each feature type may be used to determine which entrance pupil reconstruction process to perform.

[0065] In step 1008, method 1000 includes performing an entrance pupil reconstruction process to determine entrance pupil position data using the set of feature measurements. The entrance pupil reconstruction process can be performed in various ways. Depending on the feature measurements and feature types available for processing, different methods can be implemented. For example, the process can include reconstructing a pupil model using one or more pupil feature measurements. A corneal model can be reconstructed using one or more reflective feature measurements, and a limbal model can be reconstructed using one or more limbal feature measurements. Different valid counts for feature types can affect which entrance pupil reconstruction process is performed.

[0066] In some embodiments, the provided feature measurements include only one pupil feature measurement for a given eye. In this case, a single entrance pupil reconstruction process can be performed. In some embodiments, the process includes estimating the pupil position using the provided one pupil feature measurement. The estimated pupil position can be used to generate a set of candidate entrance pupil positions. The candidates can be selected in various ways. For example, a loss function can be applied to the set of candidate entrance pupil positions and the resulting loss value can be evaluated. In some embodiments, the candidate with the lowest loss value is selected.

[0067] At step 1010, method 1000 optionally includes storing the data generated by the entrance pupil reconstruction process as one or more user profile parameters. For example, parameters associated with eye features calculated during the process described herein can be saved to a database storing these parameters for a given user. During future entrance pupil reconstruction processes, the stored parameters can be retrieved as needed. For example, during a future process, the limbus feature may be missing from the received image data. In this case, the parameters associated with the limbus feature for the given user can be used during the entrance pupil reconstruction process.

[0068] The process described herein provides a robust method for performing entrance pupil reconstruction without relying on assumed parameters for the user's eye (or eyes). This approach is able to cover a larger population because the assumed parameters are typically only applicable to users with average eye characteristics (such as average pupil size). In the process described herein, image data from one or both eyes of a user can be used to estimate the entrance pupil location. Compared to methods that use average parameter values, features extracted and analyzed from the image data provide a more accurate method. If the image data does not provide enough features, the remaining parameters can be assumed.

[0069] In some embodiments, the methods and processes described herein may be associated with a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as a computer application or service, an application programming interface (API), a library, and / or other computer program products.

[0070] Figure 11 A non-limiting embodiment of a computing system 1100 that can implement one or more of the above methods and processes is schematically shown. The computing system 1100 is shown in simplified form. The computing system 1100 can embody the computing systems described above and in Figure 1Components of computing system 1100 may be included in one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, video gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices, as well as wearable computing devices (such as smart watches and head-mounted augmented reality devices).

[0071] The computing system 1100 includes a processing circuit system 1102, a volatile memory 1104, and a non-volatile storage device 1106. The computing system 1100 may optionally include a display subsystem 1108, an input subsystem 1110, a communication subsystem 1112, and / or Figure 11 Other components not shown.

[0072] Processing circuitry typically includes one or more logical processors, which are physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise achieve a desired result.

[0073] The logical processor may include one or more physical processors configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the processing circuit system 1102 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The various components of the processing circuit system may optionally be distributed in two or more separate devices that can be remotely located and / or configured to coordinate processing. For example, various aspects of the computing system disclosed herein may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. It will be understood that in this case, these virtualized aspects run on different physical logical processors of various different machines. These different physical logical processors of different machines will be understood to be collectively contained by the processing circuit system 1102.

[0074] The non-volatile storage device 1106 includes one or more physical devices configured to store instructions that can be executed by the processing circuit system to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 1106 may change, for example, to store different data.

[0075] The non-volatile storage device 1106 may include a removable and / or built-in physical device. The non-volatile storage device 1106 may include optical memory, semiconductor memory, and / or magnetic memory, or may include other mass storage device technologies. The non-volatile storage device 1106 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content addressable device. It should be understood that the non-volatile storage device 1106 is configured to retain instructions even when power to the non-volatile storage device 1106 is removed.

[0076] Volatile memory 1104 may include physical devices that include random access memory. Volatile memory 1104 is typically used by processing circuitry 1102 to temporarily store information while processing software instructions. It should be understood that when power to volatile memory 1104 is removed, volatile memory 1104 typically does not continue to store instructions.

[0077] Aspects of the processing circuitry 1102, volatile memory 1104, and non-volatile storage 1106 may be integrated together into one or more hardware logic components. Such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASICs), program and application specific standard products (PSSP / ASSPs), systems on chips (SOCs), and complex programmable logic devices (CPLDs), among others.

[0078] The terms "module," "program," and "engine" may be used to describe an aspect of computing system 1100 that is typically implemented in software by a processor to perform a specific function using portions of volatile memory, the function involving transformations specifically configuring the processor to perform the function. Thus, a module, program, or engine may be instantiated by executing instructions stored by non-volatile storage 1106 via processing circuitry 1102 using portions of volatile memory 1104. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" may encompass single or multiple sets of executable files, data files, libraries, drivers, scripts, database records, etc.

[0079] The display subsystem 1108 (when included) can be used to present a visual representation of the data stored by the non-volatile storage device 1106. The visual representation can take the form of a GUI. Since the methods and processes described herein change the data stored by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 1108 can also be transformed to visually represent the changes in the underlying data. The display subsystem 1108 can include one or more display devices utilizing virtually any type of technology. Such a display device can be combined with the processing circuit system 1102, the volatile memory 1104 and / or the non-volatile storage device 1106 in a shared housing, or such a display device can be a peripheral display device.

[0080] Input subsystem 1110 (when included) may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, camera, or microphone.

[0081] The communication subsystem 1112 (when included) can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 1112 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured to communicate via a wired or wireless local area network or wide area network, a broadband cellular network, or the like. In some embodiments, the communication subsystem can allow the computing system 1100 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0082] The following paragraphs provide additional description of the subject matter of the present disclosure. One aspect provides an apparatus for reconstructing an entrance pupil, the apparatus comprising a camera system and a processing circuit system coupled to a memory storing instructions, the instructions, when executed by the processing circuit system, causing the processing circuit system to: receive image data from the camera system; extract a feature set from the image data, wherein extracting the feature set comprises classifying the features into a plurality of feature types; calculate a feature measurement set from the extracted feature set; and perform an entrance pupil reconstruction process to determine entrance pupil position data using the feature measurement set. In this regard, additionally or alternatively, the camera system comprises a first camera for capturing an image of a first eye of a user and a second camera for capturing an image of a second eye of the user, and wherein the entrance pupil position data comprises a first entrance pupil position for the first eye and a second entrance pupil position for the second eye. In this regard, additionally or alternatively, the camera system comprises a first camera and a second camera for capturing an image of the first eye of a user, and wherein the received image data comprises stereoscopic image data of the first eye. In this regard, additionally or alternatively, the feature measurement set includes one or more pupil feature measurements, performing the entrance pupil reconstruction process includes reconstructing a pupil model using the one or more pupil feature measurements, and the entrance pupil position data is determined based on a ray tracing process using the pupil model. In this regard, additionally or alternatively, the feature measurement set further includes one or more retroreflective feature measurements, performing the entrance pupil reconstruction process further includes reconstructing a corneal model using the one or more retroreflective feature measurements, and the pupil model is further reconstructed using the corneal model. In this regard, additionally or alternatively, the feature measurement set further includes one or more limbal feature measurements, and performing the entrance pupil reconstruction process further includes reconstructing a limbal model using the one or more limbal feature measurements, and reconstructing a corneal refractive index based on the limbal model and the pupil model, wherein the entrance pupil position data is further determined based on the corneal refractive index. In this regard, additionally or alternatively, the set of feature measurements includes pupil feature measurements, and wherein performing the entrance pupil reconstruction process includes estimating a pupil position based on the pupil feature measurements, determining a set of candidate entrance pupil positions based on the estimated pupil positions, and selecting a candidate entrance pupil position from the set of candidate entrance pupil positions based on a loss value resulting from applying a loss function to each candidate entrance pupil position. In this regard, additionally or alternatively, the set of feature measurements further includes a limbal feature measurement, and wherein the pupil position is estimated based on the limbal feature measurement. In this regard, additionally or alternatively, performing the entrance pupil reconstruction process includes using one or more user profile parameters to compensate for missing feature measurements.In this regard, additionally or alternatively, the instructions, when executed by the processing circuitry, further cause the processing circuitry to store data generated by the entrance pupil reconstruction process as one or more user profile parameters.

[0083] Another aspect includes a method for reconstructing an entrance pupil, the method comprising: receiving image data; extracting a feature set from the image data, wherein extracting the feature set comprises classifying the features into a plurality of feature types; calculating a feature measurement set from the extracted feature set; and performing an entrance pupil reconstruction process to determine entrance pupil position data using the feature measurement set. In this aspect, additionally or alternatively, the received image data comprises at least one image of a first eye of a user and at least one image of a second eye of the user, and wherein the entrance pupil position data comprises a first entrance pupil position for the first eye and a second entrance pupil position for the second eye. In this aspect, additionally or alternatively, the received image data comprises a stereoscopic image of the first eye of the user. In this aspect, additionally or alternatively, the feature measurement set comprises one or more pupil feature measurements and one or more retroreflective feature measurements, performing the entrance pupil reconstruction process comprises reconstructing a corneal model using the one or more retroreflective feature measurements and reconstructing a pupil model using the one or more pupil feature measurements and the corneal model, and the entrance pupil position data is determined based on a ray tracing process using the pupil model. In this regard, additionally or alternatively, the feature measurement set further includes one or more limbal feature measurements, and performing the entrance pupil reconstruction process further includes reconstructing a limbal model using the one or more limbal feature measurements, and reconstructing a corneal refractive index based on the limbal model and the pupil model, wherein the entrance pupil position data is further determined based on the corneal refractive index. In this regard, additionally or alternatively, the feature measurement set includes pupil feature measurements, and wherein performing the entrance pupil reconstruction process includes estimating a pupil position based on the pupil feature measurements, determining a set of candidate entrance pupil positions based on the estimated pupil position, and selecting a candidate entrance pupil position from the set of candidate entrance pupil positions based on a loss value generated by applying a loss function to each candidate entrance pupil position. In this regard, additionally or alternatively, the feature measurement set further includes limbal feature measurements, and wherein the pupil position is estimated based on the limbal feature measurements. In this regard, additionally or alternatively, the method further includes storing the data generated by the entrance pupil reconstruction process as one or more user profile parameters.

[0084] Another aspect includes an extended reality device for reconstructing an entrance pupil, the device comprising a camera system and a processing circuit system coupled to a memory storing instructions, the instructions, when executed by the processing circuit system, causing the processing circuit system to: receive image data from the camera system; extract a feature set from the image data, wherein extracting the feature set comprises classifying the features into a plurality of feature types; perform a feature measurement process to calculate a feature measurement value set, the feature measurement process comprising performing a quantity assessment on features of each of the feature types and performing a quality assessment on each feature; select a process to be performed from a set of entrance pupil reconstruction processes based on the feature measurement value set; and perform the selected process. In this regard, additionally or alternatively, the plurality of feature types include a reflective feature type, a pupil feature type, and a limbal feature type, performing the quality assessment on each feature produces a valid count for each of the feature types, and the selection of the selected process is based on the following items: when it is determined that the valid count of the pupil feature type is lower than a predetermined threshold number, the selected process is based on a single entrance pupil reconstruction process; and when it is determined that the valid count of the pupil feature type is higher than a predetermined threshold number, the selected process is based on a stereoscopic entrance pupil reconstruction process.

[0085] As used herein, "and / or" is defined as an inclusive OR, as specified by the following truth table:

[0086] A B A∨B real real real real Fake real Fake real real Fake Fake Fake

[0087] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions illustrated and / or described may be performed in the order illustrated and / or described, in other orders, in parallel, or omitted. Similarly, the order of the above-described processes may also be changed.

[0088] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A device for reconstructing an entrance pupil, the device comprising: Camera system; as well as processing circuitry coupled to a memory storing instructions that, when executed by the processing circuitry, cause the processing circuitry to: receiving image data from the camera system; extracting a feature set from the image data, wherein extracting the feature set includes classifying features into a plurality of feature types; Calculating a feature measurement value set from the extracted feature set; as well as An entrance pupil reconstruction process is performed to determine entrance pupil position data using the set of feature measurements.

2. The apparatus of claim 1 , wherein the camera system comprises a first camera for capturing an image of a first eye of a user and a second camera for capturing an image of a second eye of the user, and wherein the entrance pupil position data comprises a first entrance pupil position for the first eye and a second entrance pupil position for the second eye.

3. The device of claim 1 , wherein the camera system comprises a first camera and a second camera for capturing an image of a first eye of a user, and wherein the received image data comprises stereoscopic image data of the first eye.

4. The apparatus according to claim 1, wherein: The set of characteristic measurements includes one or more pupil characteristic measurements; Performing the entrance pupil reconstruction process includes reconstructing a pupil model using the one or more pupil characteristic measurements; and The entrance pupil position data is determined based on a ray tracing process using the pupil model.

5. The apparatus according to claim 4, wherein: The set of characteristic measurements further includes one or more reflective characteristic measurements; Performing the entrance pupil reconstruction process further includes reconstructing a corneal model using the one or more retroreflective feature measurements; and The pupil model is further reconstructed using the cornea model.

6. The apparatus of claim 4, wherein: The set of characteristic measurements further includes one or more limbal characteristic measurements; and Performing the entrance pupil reconstruction process further includes: reconstructing a limbus model using the one or more limbus feature measurements; as well as A corneal refractive index is reconstructed based on the limbus model and the pupil model, wherein the entrance pupil position data is further determined based on the corneal refractive index.

7. The apparatus of claim 1 , wherein the set of feature measurements includes pupil feature measurements, and wherein performing the entrance pupil reconstruction process comprises: estimating pupil position based on the pupil characteristic measurements; determining a set of candidate entrance pupil positions based on the estimated pupil position; as well as A candidate entrance pupil position is selected from the set of candidate entrance pupil positions based on a loss value generated by applying the loss function to each candidate entrance pupil position.

8. The apparatus of claim 7, wherein the set of feature measurements further comprises a limbus feature measurement, and wherein the pupil position is estimated based on the limbus feature measurement.

9. The apparatus of claim 1, wherein performing the entrance pupil reconstruction process comprises using one or more user profile parameters to compensate for missing feature measurements.

10. The apparatus of claim 1 , wherein the instructions, when executed by the processing circuitry, further cause the processing circuitry to: The data generated by the entrance pupil reconstruction process is stored as one or more user profile parameters.

11. A method for reconstructing an entrance pupil, the method comprising: receiving image data; extracting a feature set from the image data, wherein extracting the feature set includes classifying features into a plurality of feature types; Calculating a feature measurement value set from the extracted feature set; as well as An entrance pupil reconstruction process is performed to determine entrance pupil position data using the set of feature measurements.

12. A method according to claim 11, wherein the received image data includes at least one image of a first eye of a user and at least one image of a second eye of the user, and wherein the entrance pupil position data includes a first entrance pupil position for the first eye and a second entrance pupil position for the second eye.

13. The method of claim 11, wherein the received image data comprises a stereoscopic image of a first eye of a user.

14. The method according to claim 11, wherein: The set of feature measurements includes one or more pupil feature measurements and one or more reflective feature measurements; Performing the entrance pupil reconstruction process includes reconstructing a corneal model using the one or more retroreflective feature measurements, and reconstructing a pupil model using the one or more pupil feature measurements and the corneal model; and The entrance pupil position data is determined based on a ray tracing process using the pupil model.

15. The method according to claim 14, wherein: The set of characteristic measurements further includes one or more limbal characteristic measurements; and Performing the entrance pupil reconstruction process further includes: reconstructing a limbus model using the one or more limbus feature measurements; as well as A corneal refractive index is reconstructed based on the limbus model and the pupil model, wherein the entrance pupil position data is further determined based on the corneal refractive index.

16. The method of claim 11, wherein the set of feature measurements includes pupil feature measurements, and wherein performing the entrance pupil reconstruction process comprises: estimating pupil position based on the pupil characteristic measurements; determining a set of candidate entrance pupil positions based on the estimated pupil position; as well as A candidate entrance pupil position is selected from the set of candidate entrance pupil positions based on a loss value generated by applying the loss function to each candidate entrance pupil position.

17. The method of claim 16, wherein the set of feature measurements further comprises a limbus feature measurement, and wherein the pupil position is estimated based on the limbus feature measurement.

18. The method of claim 11, further comprising: The data generated by the entrance pupil reconstruction process is stored as one or more user profile parameters.

19. An extended reality device for reconstructing an entrance pupil, the device comprising: Camera system; as well as processing circuitry coupled to a memory storing instructions that, when executed by the processing circuitry, cause the processing circuitry to: receiving image data from the camera system; extracting a feature set from the image data, wherein extracting the feature set includes classifying features into a plurality of feature types; A feature measurement process is performed to calculate a feature measurement value set, the feature measurement process comprising: performing a quantitative evaluation on features of each of the feature types; and Perform quality assessment on each feature; selecting a process to be performed from a set of entrance pupil reconstruction processes based on the set of feature measurements; and The selected process is executed.

20. The extended reality device according to claim 19, wherein: The plurality of feature types include a reflective feature type, a pupil feature type, and a limbal feature type; performing said quality assessment on each feature produces a valid count for each of said feature types; and The selection of the selected process is based on the following: When it is determined that the valid count of the pupil feature type is below a predetermined threshold number, the selected process is based on a single entrance pupil reconstruction process; and When it is determined that the valid count of the pupil feature type is above a predetermined threshold number, the selected process is a stereo-based entrance pupil reconstruction process.

Images