Eye tracking in eyepiece displays

Abstract

A technique for tracking eye movements in an augmented reality system identifies multiple base images of an object or portion thereof. A search image may be generated, at least in part, based on at least some of the multiple base images. Deep learning results may be generated by performing a deep learning process on the base images using at least a neural network in a deep learning mode. A captured image may be located by performing an image registration process on the captured image and the search image using at least a Kalman filter model and the deep learning results.

Description

[Technical Field]

[0001] The present disclosure relates generally to methods and systems for tracking eye movements in eyepiece displays such as augmented reality systems, and more particularly to tracking eye movements by using retina-based tracking with a deep learning model having a deep convolutional neural network and an image registration process with a Kalman filter model, without using traditional pupil-phosphene-based techniques. [Background technology]

[0002] Spatial computing headsets, including virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR) headsets, have proven useful for many applications ranging from scientific visualization, medical and military training, engineering design and prototyping, remote operation and telepresence, and personal entertainment. In a spatial computing headset, a virtual or augmented scene is displayed to the user via an optical system assembly that is positionable and fixable to the user's head, in front of the user's eyes.

[0003] Head-mounted displays (HMDs) are being considered for a wide range of applications in the fields of 3D virtual and augmented environments. Accurate and fast eye tracking is crucial for enabling essential scenarios in HMDs, such as foveated display schemes for people and the tradeoff between field of view (FOV) and resolution through novel interactive interfaces. Eye trackers built into HMDs can be divided into invasive methods, such as scleral coils, and noninvasive video-based methods, the latter of which are more common. Current video-based methods primarily use different features of the eye, such as the iris, pupil, and phosphene, with the pupil-phosphene method being the most widely used. These methods have an average tracking error of 0.5°–1°, while the tracking resolution of such features is approximately 0.7°–1° per pixel. Further improving accuracy beyond the tracking resolution is challenging.

[0004] In addition to using ocular surface features, retinal imaging, such as eye-tracking scanning laser ophthalmoscopy (SLO), is also utilized for eye tracking in the medical field. They exploit scanning distortion for retinal movement estimation within small-FOV, high-resolution images. However, this technique is designed for small saccades, and SLO is not easily integrated into HMDs. Retina-based eye tracking in HMDs has its own advantages: higher tracking resolution without advanced sensors, a linear gaze estimation model, and direct localization of the fovea on the retina. Furthermore, retinal tracking offers a wide range of medical applications using HMDs.

[0005] Furthermore, modern approaches to object recognition make the use of machine learning methods essential. To improve its performance, some embodiments may collect larger datasets, learn stronger models, and use better techniques to prevent overfitting. Until recently, datasets of labeled images were relatively small, on the order of tens of thousands of images. Accordingly, simple recognition tasks, for which object tracking can be found, can be solved very well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the current best error rate (<0.3%) on the MNIST digit recognition task approximates human performance. However, objects in realistic settings exhibit enormous variability, and therefore, learning to recognize them may require the use of much larger training sets. Although the drawbacks of small image datasets are widely recognized, it has only recently become possible to collect labeled datasets with millions of images.

[0006] To learn about thousands of objects from millions of images, models with large learning capacities are desperately needed. Nevertheless, the significant complexity of object recognition tasks means that the problem cannot be defined even by datasets as large as ImageNet, and therefore models must, in some embodiments, also have a lot of prior knowledge to compensate for all the data, which modern techniques simply do not have. Convolutional neural networks (CNNs) constitute one such class of models. Their capacity can be controlled by varying their depth and categories, and they also make powerful and often correct assumptions about the nature of images (i.e., stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similar-sized layers, CNNs have far fewer connections and parameters, and therefore are easier to train, while their theoretical best performance is likely to be only slightly compromised.

[0007] Despite the compelling qualities of CNNs, and the relative efficiency of their local architecture, CNNs remain prohibitively expensive to apply to high-resolution images at large scales.

[0008] Furthermore, visual object tracking is a fundamental building block in various tasks in computer vision, such as autonomous driving and video surveillance. It is challenging given the large appearance variance caused by lighting, deformation, occlusion, and motion. In addition, speed is also important in practical applications.

[0009] Modern trackers can be roughly divided into two branches. The first branch is based on correlation filters, which exploit the properties of circular correlation and train regressors by performing operations in the Fourier domain. This allows for online tracking and efficient updating of filter weights simultaneously. The original version is performed in the Fourier domain and is then widely used within the tracking community. Recent correlation filter-based methods use deep features to improve accuracy, but significantly impair speed during model updating. Another branch of methods uses very robust deep features and does not update the model. However, because domain-specific information is not used, the performance of these methods is not always as good as correlation filter-based methods.

[0010] Various embodiments described herein address at least the aforementioned challenges and shortcomings of conventional approaches and present a real-time localization method for object tracking based on a video of the object's movement, where each frame can be localized on a mosaicked search image. Summary of the Invention [Means for solving the problem]

[0011] According to some embodiments, a method for tracking eye movements within an eyepiece display is described, in which a plurality of base images of an object or portion thereof may be identified, a search image may be generated based at least in part on at least some of the plurality of base images, deep learning results may be generated by performing a deep learning process on the base images using at least a neural network in a deep learning mode, and a captured image may be localized relative to the search image by performing an image registration process on the captured image and the search image using at least a Kalman filter model and the deep learning results.

[0012] In some of these embodiments, the object comprises the user's retina, and the captured image representing at least a portion of the retina is located relative to the search image without using a pupil-phosphene technique that captures phosphenes from the pupil in response to one or more input light patterns.

[0013] In some embodiments, generating deep learning results includes extracting features within a region within the captured image using a deep convolutional neural network (DCNN). Additionally, the features may be converted into multiple features comprising a first feature and a second feature. The region may be classified as a positive region or a negative region based at least in part on the first feature. A regression or correction may be determined for the region based at least in part on the second feature.

[0014] In some embodiments, to generate deep learning results, features in a region in the captured image may be extracted using a deep convolutional neural network (DCNN), and the features may be converted into multiple features comprising a first feature and a second feature.

[0015] In some of the preceding embodiments of the preceding section, to generate the deep learning result, a first response map may be generated by convolving at least the first feature using one or more convolutional layers in the DCNN, and regions may be classified as positive regions or negative regions based, at least in part, on the first response map.

[0016] In some embodiments, to generate a deep learning result, a second response map may be generated by convolving the second features using at least one or more convolutional layers or one or more different convolutional layers in the DCNN, and a regression or correction may be determined with respect to the predicted location of the region using at least the second response map.

[0017] In some embodiments, localizing the captured image includes embedding a deep learning process into a state transition model and receiving, at least, the deep learning results in a Kalman filter model as one or more measurements using the state transition model, wherein the state transition model is used to determine a next position state at a next time point based, at least in part, on a previous position state at a previous time point and at least one of process noise or control vectors of a state estimation model.

[0018] In some of the preceding embodiments in the preceding section, measurements at a subsequent time point may be determined by using at least a Kalman filter model in an image alignment process to locate the captured image, interference of similar backgrounds or one or more similar features may be reduced by performing at least an image alignment process, and the captured image may be located relative to the search image within the search image using the Kalman filter model and the image alignment process.

[0019] In some embodiments, to embed a deep learning process, one or more control vectors may be determined with respect to a state transition model, the process noise may be derived from a statistical distribution, and a time step may also be determined with respect to the state transition model.

[0020] In some of the preceding embodiments, locating a captured image within a search image using a Kalman filter model and an image alignment process includes performing a coarse alignment for the entire frame of the captured image and performing a fine alignment for one or more features or feature points within an outer region around the region referenced in the coarse alignment.

[0021] Additionally or alternatively, to perform a rough alignment for the entire frame of the captured image, one or more features or feature points within one or more regions within the entire frame of the captured image or search image may be detected, and the captured image may be aligned to one or more regions within the entire frame.

[0022] In some of the preceding embodiments of the preceding section, performing fine alignment for one or more features or feature points in an outer region around the region referenced in the coarse alignment includes selecting features or feature points in the enlarged region that correspond to the region referenced in the coarse alignment, and matching the features or feature points with corresponding features or feature points in the outer region of the search image.

[0023] In some embodiments, the step of performing a fine alignment with respect to one or more features or feature points in an outer region around the region referenced in the coarse alignment may further include locating the captured image within the search image based, at least in part, on the results of the step of matching the features or feature points with corresponding features or feature points in the outer region of the search image.

[0024] Some embodiments are directed to a system for tracking eye movement in an eyepiece display, comprising: a processor; a scanning fiber assembly; and a memory having stored thereon instructions that, when executed by the processor, cause the processor to implement any of the aforementioned methods for tracking eye movement in an eyepiece display.

[0025] Some embodiments are directed to a non-transitory computer-readable medium having stored thereon instructions that, when executed by a microprocessor, cause the microprocessor to perform any of the aforementioned methods for tracking eye movement in an eyepiece display.

[0026] Summary recitation of some embodiments of the present disclosure

[0027] (Claim 1) A method for tracking eye movements in an eyepiece display, the method comprising: identifying a plurality of base images of an object or a portion thereof; generating a search image based, at least in part, on at least some of the plurality of base images; generating deep learning results by performing a deep learning process on the base images using at least a neural network in a deep learning model; and locating the captured image relative to the search image by performing an image alignment process on the captured image and the search image using at least a Kalman filter model and the deep learning results.

[0028] (Claim 2) 10. The method of claim 1, wherein the object comprises a user's retina and represents at least a portion of the retina, and the captured image is located relative to the search image without using a pupil-phosphene technique that captures phosphenes from the pupil in response to one or more input light patterns.

[0029] (Claim 3) 10. The method of claim 1, wherein generating a deep learning result includes using a deep convolutional neural network (DCNN) to extract features in a region in the captured image and converting the features into a plurality of features comprising a first feature and a second feature.

[0030] (Claim 4) The method of claim 3 , wherein generating deep learning results includes classifying the regions into positive regions or negative regions based at least in part on the first feature.

[0031] (Claim 5) The method of claim 3 , wherein generating deep learning results includes determining a regression for the domain based at least in part on the second features.

[0032] (Claim 6) The method of claim 4 , wherein classifying the region is based at least in part on the first feature but not on the second feature.

[0033] (Claim 7) The method of claim 5 , wherein determining a regression for the region is based at least in part on the second feature and not on the first feature.

[0034] (Claim 8) 10. The method of claim 1, wherein generating a deep learning result includes using a deep convolutional neural network (DCNN) to extract features in a region in the captured image and converting the features into a plurality of features comprising a first feature and a second feature.

[0035] (Claim 9) 10. The method of claim 8, wherein generating a deep learning result further comprises: generating a first response map by convolving at least the first feature using one or more convolutional layers in the DCNN; and classifying regions into positive regions or negative regions based, at least in part, on the first response map.

[0036] (Claim 10) The method of claim 9 , wherein classifying the region is based at least in part on the first response map but not on the second response map.

[0037] (Claim 11) 10. The method of claim 9, wherein generating deep learning results further includes at least generating a second response map by convolving second features using one or more convolutional layers or one or more different convolutional layers in the DCNN, and determining a regression or correction for predicted locations of the regions using at least the second response map.

[0038] (Claim 12) The method of claim 11 , wherein determining the regression or correction is based at least in part on the second response map and not on the first response map.

[0039] (Claim 13) 10. The method of claim 8, wherein generating deep learning results further comprises at least compensating the DCNN and / or deep learning model using an image registration process.

[0040] (Claim 14) The method of claim 1 , wherein locating the captured image includes embedding a deep learning process into a state transition model.

[0041] (Claim 15) 15. The method of claim 14, wherein localizing the captured image includes at least receiving deep learning results in a Kalman filter model as one or more measurements using a state transition model.

[0042] (Claim 16) 15. The method of claim 14, wherein the state transition model is used to determine a next position state at a next time point based, at least in part, on a previous position state at a previous time point and at least one of process noise or control vectors of the state estimation model.

[0043] (Claim 17) 15. The method of claim 14, wherein locating the captured image includes determining measurements at subsequent times by using a Kalman filter model in at least an intra-image registration process.

[0044] (Claim 18) 20. The method of claim 17, wherein locating the captured image includes at least performing an image registration process to reduce interference of similar backgrounds or one or more similar features.

[0045] (Claim 19) The method of claim 17 , wherein locating the captured image comprises locating the captured image within the search image using a Kalman filter model and an image registration process.

[0046] (Claim 20) 15. The method of claim 14, wherein embedding the deep learning process includes determining one or more control vectors for a state transition model, deriving process noise from a statistical distribution, and determining a time step for the state transition model.

[0047] (Claim 21) 21. The method of claim 20, wherein the time step comprises a time duration between two most recent frames captured by the augmented reality system, and the state transition model is used to determine a next position state at a next point in time based at least in part on a previous position state at a previous point in time and at least one of process noise or control vectors of the state estimation model.

[0048] (Claim 22) 21. The method of claim 20, wherein the process noise is derived from a statistical distribution, the statistical distribution comprising a zero-mean multivariate normal distribution.

[0049] (Claim 23) 20. The method of claim 19, wherein locating the captured image within the search image using the Kalman filter model and the image registration process includes performing a coarse registration for the entire frame of the captured image and performing a fine registration for one or more features or feature points within an outer region around the region referenced in the coarse registration.

[0050] (Claim 24) 24. The method of claim 23, wherein performing a rough alignment for the entire frame of the captured image comprises detecting one or more features or feature points in one or more regions within the entire frame of the captured image or the search image.

[0051] (Claim 25) 25. The method of claim 24, wherein performing a coarse alignment for the entire frame of the captured image comprises aligning the captured image to one or more regions within the entire frame.

[0052] (Claim 26) 24. The method of claim 23, wherein performing fine alignment for one or more features or feature points in an outer region around the region referenced in the coarse alignment comprises selecting features or feature points in the enlarged region that correspond to the region referenced in the coarse alignment.

[0053] (Claim 27) 27. The method of claim 26, wherein performing fine alignment for one or more features or feature points in an outer region around the region referenced in the coarse alignment comprises matching the features or feature points with corresponding features or feature points in the outer region of the search image.

[0054] (Claim 28) 26. The method of claim 25, wherein performing fine alignment with respect to one or more features or feature points in an outer region around the region referenced in the coarse alignment comprises locating the captured image within the search image based at least in part on results of matching the features or feature points with corresponding features or feature points in the outer region of the search image.

[0055] (Claim 29) 10. The method of claim 1, wherein the deep learning model comprises a modified Siamese region proposal network, the deep learning model comprising a deep convolutional neural network and an image registration process, wherein a plurality of different captured images captured at different time periods are localized based on the search image rather than a plurality of search images.

[0056] (Claim 30) 10. The method of claim 1, wherein the deep learning model comprises a modified Siamese region proposal network, and the captured image is localized relative to the search image not using a bounding box but using one or more target locations in two orthogonal directions.

[0057] (Claim 31) The method of claim 1 , wherein the search image is generated by stitching, mosaicking, or combining at least some of the multiple base images into the search image.

[0058] (Claim 32) 2. The method of claim 1, wherein the deep learning model comprises a deep convolutional neural network and an image registration process, and wherein the deep learning model invokes only the deep convolutional neural network and not the image registration process when one or more measurements of the Kalman filter model drop out until the next successful registration of the first captured image.

[0059] (Claim 33) A system for tracking eye movement within an eyepiece display, comprising: a processor; a scanning fiber assembly; and a memory having stored thereon instructions that, when executed by the processor, cause the processor to perform any of methods 1-32 to track eye movement within the eyepiece display.

[0060] (Claim 34) A non-transitory computer-readable medium having stored thereon instructions that, when executed by a microprocessor, cause the microprocessor to perform any of methods 1-28 for tracking eye movement within an eyepiece display. The present invention provides, for example, the following. (Item 1) 1. A method for tracking eye movements in an eyepiece display, comprising: identifying a plurality of base images of an object or portion thereof; generating a search image based, at least in part, on at least some of the plurality of base images; generating deep learning results by performing a deep learning process on the base image using a neural network in at least the deep learning model; performing an image registration process on the captured image and the retrieved image using at least a Kalman filter model and the deep learning results to locate the captured image relative to the retrieved image; A method comprising: (Item 2) Item 10. The method of item 1, wherein the object comprises a user's retina, and the captured image representing at least a portion of the retina is located relative to the search image without using a pupil-phosphene technique that captures phosphenes from the pupil in response to one or more input light patterns. (Item 3) generating the deep learning result, extracting features within a region within the captured image using a deep convolutional neural network (DCNN); converting the feature into a plurality of features comprising a first feature and a second feature; classifying the region as a positive region or a negative region based at least in part on the first feature; determining a regression or correction for the region based at least in part on the second characteristic; and The method according to item 1, comprising: (Item 4) generating the deep learning result, extracting features within a region within the captured image using a deep convolutional neural network (DCNN); converting the feature into a plurality of features comprising a first feature and a second feature; The method according to item 1, comprising: (Item 5) Generating the deep learning result further comprises: generating a first response map by convolving at least the first feature using one or more convolutional layers in the DCNN; classifying the region as a positive region or a negative region based at least in part on the first response map; Item 5. The method according to item 4, comprising: (Item 6) Generating the deep learning result further comprises: generating a second response map by convolving the second features using at least the one or more convolutional layers or one or more different convolutional layers in the DCNN; determining a regression or correction for the predicted location of the region using at least the second response map; Item 6. The method according to item 5, comprising: (Item 7) Locating the captured image includes: Embedding the deep learning process in a state transition model; receiving the deep learning results in the Kalman filter model as one or more measurements using the state transition model, wherein the state transition model is used to determine a next position state at a next time point based at least in part on a previous position state at a previous time point and at least one of a process noise or a control vector of a state estimation model; The method according to item 1, comprising: (Item 8) Locating the captured image includes: determining measurements at the next time point by using the Kalman filter model in at least the registration process within the image; reducing interference of similar backgrounds or one or more similar features by performing at least an image registration process; locating the captured image within the retrieved image using the Kalman filter model and the image registration process; and Item 7. The method according to item 7, comprising: (Item 9) Embedding the deep learning process determining one or more control vectors for the state transition model; Deriving the process noise from a statistical distribution; determining a time step for the state transition model; Item 7. The method according to item 7, comprising: (Item 10) Locating the captured image within the search image using the Kalman filter model and the image registration process includes: performing a coarse alignment across the entire frame of the captured image; performing a fine alignment of one or more features or feature points within an outer region around the region referenced in said coarse alignment; Item 10. The method according to item 9, comprising: (Item 11) Performing the coarse alignment for an entire frame of the captured image includes: Detecting one or more features or feature points in one or more regions within an entire frame of the captured image or the retrieved image; registering the captured image to one or more regions within the entire frame; Item 11. The method according to item 10, comprising: (Item 12) performing a fine alignment with respect to one or more features or feature points within an outer region around the region referenced in said coarse alignment, selecting a feature or feature point within the enlarged region that corresponds to the region referenced in the rough registration; matching said features or said feature points with corresponding features or feature points in an outer region of said search image; Item 12. The method according to item 11, comprising: (Item 13) performing a fine alignment with respect to one or more features or feature points within an outer region around the region referenced in said coarse alignment, 13. The method of claim 12, comprising locating the captured image within the search image based at least in part on results of matching the features or feature points with corresponding features or feature points in an outer region of the search image. (Item 14) A system for tracking eye movements within an eyepiece display, comprising a processor, a scanning fiber assembly, and a memory having instructions stored thereon that, when executed by the processor, cause the processor to perform the method of any of items 1-13 to track eye movements within the eyepiece display. (Item 15) 14. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a microprocessor, cause the microprocessor to perform the method of any of items 1-13 for tracking eye movements within an eyepiece display. [Brief explanation of the drawings]

[0061] [Figure 1] FIG. 1 illustrates a high-level block diagram of a method or system for eye tracking in an eyepiece display in some embodiments.

[0062] [Figure 2]FIG. 2 illustrates some exemplary retinal, base, or template frames and mosaicked baseline or search images captured by a scanning fiber endoscope (SFE) in some embodiments.

[0063] [Figure 3] FIG. 3 illustrates an exemplary cumulative distribution function (CDF) of retinal tracking error in degrees over 400 frames in some embodiments.

[0064] [Figure 4] FIG. 4 illustrates some example benchmark results of an example retina-based tracking compared to a traditional pupil-phosphene method, according to some embodiments.

[0065] [Figure 5] FIG. 5 illustrates a simplified example of a system for capturing SFE images of the retina, according to some embodiments.

[0066] [Figure 6] FIG. 6 illustrates some example template or base images of the retina, according to some embodiments.

[0067] [Figure 7] FIG. 7 illustrates an example deep learning model, according to some embodiments.

[0068] [Figure 8] FIG. 8 illustrates a simplified working example of robot localization in some embodiments.

[0069] [Figure 9] FIG. 9 illustrates another simplified working example of microscope or endoscope localization in a surgical procedure, according to some embodiments.

[0070] [Figure 10A]FIG. 10A illustrates another simplified working example of face or eye localization for face or eye tracking in some embodiments.

[0071] [Figure 10B] FIG. 10B illustrates another simplified working example of object localization for object tracking in some embodiments.

[0072] [Figure 10C] FIG. 10C illustrates another simplified working example of object localization for object tracking in some embodiments.

[0073] [Figure 11A] FIG. 11A illustrates an exemplary configuration of an array of microprojectors and the coupling of the array of microprojectors with an optical system in one or more embodiments.

[0074] [Figure 11B] FIG. 11B illustrates a top view of some example components of a schematic representation of a wearable XR device in one or more embodiments.

[0075] [Figure 11C] FIG. 11C illustrates an exemplary embodiment of a schematic representation of a wearable XR device in one or more embodiments.

[0076] [Figure 12A] FIG. 12A illustrates a high-level block diagram for eye tracking in an eyepiece display in one or more embodiments.

[0077] [Figure 12B] FIG. 12B illustrates an exemplary region proposal network (RPN) in one or more embodiments.

[0078] [Figure 12C] FIG. 12C illustrates another example tracking as one-shot detection in one or more embodiments.

[0079] [Figure 12D] FIG. 12D illustrates exemplary center sizes in an RPN feature map in one or more embodiments.

[0080] [Figure 12E] FIG. 12E illustrates some example training error rates of a four-layer convolutional neural network with ReLU (rectified linear units), in one or more embodiments.

[0081] [Figure 13] FIG. 13 illustrates further details of a portion of the high-level block diagram shown in FIG. 12, in one or more embodiments.

[0082] [Figure 14A] FIG. 14A illustrates further details of another portion of the high-level block diagram shown in FIG. 12 in one or more embodiments.

[0083] [Figure 14B] FIG. 14B illustrates an exemplary deep convolutional neural network, according to one or more embodiments.

[0084] [Figure 15A] FIG. 15A illustrates further details of another portion of the high-level block diagram shown in FIG. 12 in one or more embodiments.

[0085] [Figure 15B] FIG. 15B illustrates further details of a portion of FIG. 15A in one or more embodiments.

[0086] [Figure 15C] FIG. 15C illustrates further details of another portion of FIG. 15A in one or more embodiments.

[0087] [Figure 16A] 16A-I illustrate example configurations of an array of microprojectors and the coupling of the array of microprojectors with an optical system in one or more embodiments. [Figure 16B] 16A-I illustrate example configurations of an array of microprojectors and the coupling of the array of microprojectors with an optical system in one or more embodiments. [Figure 16C] 16A-I illustrate example configurations of an array of microprojectors and the coupling of the array of microprojectors with an optical system in one or more embodiments. [Figure 16D] 16A-I illustrate example configurations of an array of microprojectors and the coupling of the array of microprojectors with an optical system in one or more embodiments. [Figure 16E] 16A-I illustrate example configurations of an array of microprojectors and the coupling of the array of microprojectors with an optical system in one or more embodiments. [Figure 16F] 16A-I illustrate example configurations of an array of microprojectors and the coupling of the array of microprojectors with an optical system in one or more embodiments. [Figure 16G] 16A-I illustrate example configurations of an array of microprojectors and the coupling of the array of microprojectors with an optical system in one or more embodiments. [Figure 16H] 16A-I illustrate example configurations of an array of microprojectors and the coupling of the array of microprojectors with an optical system in one or more embodiments. [Figure 16I]16A-I illustrate example configurations of an array of microprojectors and the coupling of the array of microprojectors with an optical system in one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0088] In the following description, certain specific details are set forth to provide a thorough understanding of various disclosed embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with eyewear, including spatial computing headsets and hinge systems for the temple arms of eyewear, have not been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0089] Unless otherwise required by context, throughout this specification and the following claims, the words "comprise" and variations thereof, such as "comprises" and "comprising," are to be interpreted in their open, inclusive sense, including, but not limited to, "including."

[0090] Throughout this specification, references to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0091] In the following description, certain specific details are set forth to provide a thorough understanding of various disclosed embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR) systems have not been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments. It should be noted that the terms “virtual reality (VR),” “augmented reality (AR),” “mixed reality (MR),” and “extended reality (XR)” are used interchangeably in this disclosure and may refer to a method or system for displaying at least virtual content to a user via at least the wearable optical assembly 12 described herein.

[0092] Unless otherwise required by context, throughout this specification and the following claims, the words "comprise" and variations thereof, such as "comprises" and "comprising," are to be interpreted in their open, inclusive sense, including, but not limited to, "including."

[0093] Exemplary devices, methods, and systems are described herein. It should be understood that the words "example," "exemplary," and "illustrative" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as being "example," "exemplary," or "illustrative" is not necessarily to be construed as preferred or advantageous over other embodiments or features. The exemplary embodiments described herein are not meant to be limiting. In general, it will be readily understood that aspects of the present disclosure, as described herein and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0094] Additionally, the particular arrangement shown in the figures is not to be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in a given figure. Additionally, some of the elements shown may be combined or omitted. Still further, exemplary embodiments may include elements not shown in the figures. As used herein with respect to measurements, "about" means + / - 5%.

[0095] Head-mounted displays (HMDs) are being considered for a wide range of applications in the fields of 3D virtual and augmented environments. Accurate and fast eye tracking is crucial for enabling essential scenarios in HMDs, such as foveated display schemes for people and the tradeoff between field of view (FOV) and resolution through novel interactive interfaces. Eye trackers built into HMDs can be divided into invasive methods, such as scleral coils, and noninvasive video-based methods, the latter of which are more common. Current video-based methods primarily use different features of the eye, such as the iris, pupil, and phosphene, with the pupil-phosphene method being the most widely used. These methods have an average tracking error of 0.5°–1°, while the tracking resolution of such features is approximately 0.7°–1° per pixel. Further improving accuracy beyond the tracking resolution is challenging.

[0096] In addition to using ocular surface features, retinal imaging, such as eye-tracking scanning laser ophthalmoscopy (SLO), is also utilized for eye tracking in the medical field. They exploit scanning distortion for retinal movement estimation within small-FOV, high-resolution images. However, this technique is designed for small saccades, and SLO is not easily integrated into HMDs. Retina-based eye tracking in HMDs has its own advantages: higher tracking resolution without advanced sensors, a linear gaze estimation model, and direct localization of the fovea on the retina. Furthermore, retinal tracking offers a wide range of medical applications using HMDs.

[0097] Various embodiments present a real-time localization method for object tracking based on a video of the object's movement, where each frame may be localized on a mosaicked search image. Figure 12A illustrates a high-level block diagram for eye tracking in an eyepiece display in one or more embodiments.

[0098] In these embodiments, multiple base or template images of an object or portion thereof (e.g., a retina or portion thereof) may be identified at 1202. Note that the terms “template image” and “base image” may be used interchangeably throughout this disclosure to distinguish from a search image, which may be constructed using, for example, some or all of the multiple base images. In some of these embodiments, the multiple base images may be captured, for example, by using a scanning fiber endoscope (SFE), while the base images may be captured by any image capture device in some other embodiments.

[0099] A search image may be generated at 1204 based, at least in part, on at least some of the multiple base images. For example, the search image may be mosaicked, stitched, or otherwise combined from multiple base images, which may be captured using the same or different image capture devices. The base images, in some embodiments, may have a lower resolution and / or a narrower field of view than the search image. The search image generated at 1204 may also be used to locate, and therefore track, moving objects. Note that while some embodiments are described herein with reference to a user's retina, the various techniques described herein may also be applied to other objects, as described below.

[0100] In some embodiments, once a search image is constructed, any subsequently captured image frames (e.g., by an eye tracking device in an XR system) may be aligned to the same search image, rather than to multiple images, as in some conventional approaches, although in some other embodiments, subsequently captured image frames (e.g., by an eye tracking device in an XR system) may be aligned to multiple search images as described herein.

[0101] Deep learning results may be generated at 1206 by performing a deep learning process on the base image or the captured image using at least a neural network in the deep learning model. Using the deep learning results generated at 1206, the captured image may be localized at 1208 by performing an image registration process on the captured image and / or the retrieved image using at least a Kalman filter model and the deep learning results. In these embodiments, the captured image, which represents at least a portion of the retina, is localized relative to the retrieved image without using a pupil-phosphene technique that captures phosphenes from the pupil in response to one or more input light patterns. Further details about some of the block diagrams illustrated in FIG. 12A are described below.

[0102] 12B illustrates an exemplary region proposal network (RPN) in one or more embodiments. More specifically, the exemplary RPN network includes a Siamese subnetwork 1202B and a region proposal network 1258B used together to perform feature extraction, which generates a first output having positive 1222B and negative 1224B regions, features, points, locations, etc. In the first output, 1226B, represents a group, and the first output includes a plurality of groups 1228B. The Siamese subnetwork 1202B and the region proposal network 1258B may further perform feature extraction, which further generates a second output having a plurality 1252B of individual groups 1250B.

[0103] Some embodiments may be trained offline on image pairs, which can take advantage of large-scale training data such as YouTube-BB. Ablation studies show that more data can help obtain even better performance. Additionally or alternatively, some embodiments may find that the region proposal subnetwork typically predicts accurate scale and ratio proposals, resulting in compact bounding boxes such as in Figure 12E.

[0104] Figure 12E illustrates that a four-layer convolutional neural network with ReLU (solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line). The learning rate for each network was chosen independently to make training as fast as possible. No regularization of any kind was employed. While the magnitude of the effect demonstrated here varies with network architecture, networks with ReLU consistently learn several times faster than their equivalents with saturating neurons.

[0105] Some embodiments, as illustrated in Figure 12B, may employ a Siamese region proposal network (Siamese-RPN), which is trained offline end-to-end on large-scale image pairs for tracking tasks. During online tracking, the proposed framework is formulated as a local one-shot detection task, which can refine proposals and discard expensive multi-scale testing. It achieves the best performance in the VOT2015, VOT2016, and VOT2017 real-time challenges with a speed of 160 FPS, demonstrating its advantages in both accuracy and efficiency.

[0106] Region proposal networks (RPNs) were first proposed in Faster R-CNN. Before RPNs, traditional proposal extraction methods were time-consuming. For example, selective search required two seconds to process one image. Additionally, these proposals were not good enough for detection. The enumeration of multiple anchors and shared convolutional features makes the proposal extraction method time-efficient while achieving high quality. RPNs are capable of extracting more precise proposals due to the supervision of both foreground-background classification and bounding box regression. Several variants of Faster R-CNN exist, which employ RPNs. R-FCNs consider component location information, while FPNs employ feature pyramid networks to improve the performance of small object detection. In contrast to two-stage detectors, improved versions of RPNs, such as SSD and YOLO9000, are efficient detectors. RPNs have been successful in many detection applications due to their speed and excellent performance; however, they have not been fully utilized in tracking.

[0107] In Figure 12B, the region proposal subnetwork 1258B is in the center and has two branches: one branch 1254B for classification and the other branch 1256B for regression. Pairwise correlation is employed to obtain the outputs of the two branches (1228B and 1252B). Details of these two output feature maps (1226B, 1228B, 1250B, and 1252B) are shown. In the classification branch 1254B, the output feature maps (1226B and 1228B) have 2k channels, which correspond to the foreground and background of the k anchors. In regression branch 1256B, the output feature maps (1250B and 1252B) have 4k channels, which correspond to two coordinates (dx in the x-direction and dy in the y-direction, instead of having a bounding box with dw in the width direction and dl in the length direction in addition to dx and dy) used for the refinement of the k anchor proposals. [ka] 1240B and 1214B show correlation operators.

[0108] Some embodiments describe the proposed Siamese-RPN framework in detail. As shown in Figure 2, the proposed framework consists of a Siamese subnetwork for feature extraction and a region proposal subnetwork for proposal generation. Specifically, there are two branches within the RPN subnetwork: one is responsible for foreground-background classification, and the other is used for proposal refinement. Image patches containing target objects are fed into the proposed framework, and the entire system is trained end-to-end.

[0109] For Siamese networks, some embodiments employ a fully convolutional network without padding. Let L denote the shift operator (Lτx)[u]=x[u−τ], and then all padding is removed to satisfy the definition of fully convolutional with stride k. [ka]

[0110] Some embodiments use a modified AlexNet, where groups from conv2 and conv4 are removed. The Siamese feature extraction sub-network consists of two branches. One, called the template branch, receives the target patch in the history frame as input (denoted as z). The other, called the detection branch, receives the target patch in the current frame as input (denoted as x). The two branches share parameters within the CNN so that the two patches are implicitly encoded by the same transformation suitable for subsequent tasks. For ease of illustration and explanation, some embodiments denote φ(z) and φ(x) as the output feature maps of the Siamese sub-network.

[0111] The region proposal subnetwork consists of a pairwise correlation section and a monitoring section. The monitoring section has two branches, one for foreground-background classification and the other for proposal regression. If there are k anchors, the network needs to output 2k channels for classification and 4k channels for regression. Therefore, the pairwise correlation section first divides the channels of φ(z) into two branches [φ(z)]. cls and [φ(z)] reg which have 2k and 4k times the channels due to the two convolutional layers. φ(x) also has two branches [φ(x)] due to the two convolutional layers. cls and [φ(x)] reg [φ(z)] is split into [φ(x)] and [φ(z)], but keeps the channels invariant. [φ(z)] serves as the correlation kernel for [φ(x)] in a "group" fashion, i.e., the number of channels in a group of [φ(z)] is the same as the overall number of channels in [φ(x)]. Correlations are calculated on both the classification and regression branches. [ka]

[0112] Template feature map [φ(z)] cls and [φ(z)] reg is used as the kernel, [ka] denotes the convolution operation. As shown in Figure 2, [ka] Denoted as, [ka] Each point in contains 2k channel vectors, which represent the negative and positive activations of each anchor at the corresponding location on the original map. A softmax loss is employed to supervise the classification branch. Similarly, [ka] Each point in Aregw×h×4k, denoted as Aregw×h×4k, contains 4k channel vectors, representing dx, dy, dw, and dh, which measure the distance between the anchor and the corresponding ground truth. When training a network with several anchors, some embodiments employ the loss function used in Faster R-CNN. The loss for classification is the cross-entropy loss, and some embodiments employ a smoothed L1 loss with normalized coordinates for regression. Aregw×h×4k contains 4k channel vectors, representing dx, dy, dw, and dh, which measure the distance between the anchor and the corresponding ground truth. When training a network with several anchors, some embodiments employ the loss function used in Faster R-CNN. The loss for classification is the cross-entropy loss, and some embodiments employ a smoothed L1 loss with normalized coordinates for regression. x , A y , A w , A h indicate the center point and shape of the anchor box, respectively, and T x , T y , T w , T h Let denote the ground truth boxes, respectively, then the normalized distance is: [ka]

[0113] They are then passed through a smooth L1 loss, which can be written as: [ka]

[0114] Finally, the loss function may be optimized. [ka]

[0115] where λ is a hyperparameter for balancing the two parts. L cls is the cross-entropy loss, and L reg is as follows.

Chemical formula

[0116] During the training phase, sample pairs are selected from ILSVRC and continuously from Youtube-BB at random intervals. Templates and detection patches are extracted from two frames of the same video. In some embodiments, after the Siamese subnetwork is pre-trained using ImageNet, Stochastic Gradient Descent (SGD) is used to train the Siamese-RPN end-to-end. Due to the need to train the regression branch, several data augmentations including affine transformation are adopted.

[0117] Some embodiments recognize that the same object in two adjacent frames will not change much, and thus select fewer anchors in the tracking task than in the detection task. Therefore, only one scale with different ratios of anchors is adopted, and the anchor ratios adopted by some embodiments may be [0.33, 0.5, 1, 2, 3].

[0118] The strategy for selecting positive and negative training samples is also important in this framework. The criteria used within the object detection task, where some embodiments use the IoU as a measurement together with two thresholds th_hi and th_lo, are adopted here. Positive samples are defined as anchors that have IoU > th_hi together with their corresponding ground truth. Negative ones are defined as anchors that satisfy IoU < th_lo. Some embodiments set th_lo to 0.3 and th_hi to 0.6. Some embodiments also limit the maximum number of positive samples to 16 and the total number of samples to 64 from one training pair.

[0119] In this subsection, some embodiments first formulate the tracking task as a local one-shot detection task. Then, the estimation phase under this interpretation is analyzed in detail and simplified to gain acceleration. Finally, some specific strategies are introduced to make the framework suitable for tracking tasks.

[0120] Some embodiments consider one-shot detection as a discrimination task, where the objective is to find parameters W that minimize the mean loss L of the prediction function ψ(x;W), computed over a data set of n samples x i and corresponding labels li. [ka]

[0121] One-shot learning aims to learn W from a single template z of the class of interest. The challenge in discriminative one-shot learning is to find a mechanism to incorporate categorical information into the learner, i.e., to learn to learn. To address this challenge, some embodiments use a meta-learning process, i.e., a method to learn the predictive parameters W from a single template z using a feedforward function ω that maps (z;W′) to W. z i Let x be the template samples in one batch, then the problem can be formulated as follows: [ka]

[0122] As above, let z denote the template patch, x denote the detection patch, function φ denote the Siamese feature extraction sub-network, and function ζ denote the region proposal sub-network. Then, the one-shot detection task can be formulated as follows: [ka]

[0123] Some embodiments may reinterpret the template branch in the Siamese subnetwork as a training parameter for predicting the kernel of the local detection task, which is typically a learning-to-learn process. In this interpretation, the template branch is used to embed category information into the kernel, and the detection branch uses the embedded information to perform detection. During the training phase, the meta-learner does not require any other supervision except for pairwise bounding box supervision. In the estimation phase, the Siamese framework is refined, excluding the initial frame and leaving only the detection branch, thus leading to high speed. The target patch from the first frame is sent into the template branch, and the detection kernel is pre-computed so that some embodiments can perform one-shot detection in other frames. Because the local detection task is based only on category information provided by the template on the initial frame, it can be considered one-shot detection, as shown in FIG. 12C. In FIG. 12C, 1214C represents weights for regression, 1216C represents weights for classification, conv represents a convolutional layer, and CNN represents a connected neural network. The various numbers (eg, 17x17x2K) represent the dimensionality of the data set (eg, feature map).

[0124] Some embodiments consider the output of the template branch as the kernel for local detection. Both kernels are pre-computed on the initial frame and fixed for the entire tracking period. Using the current feature map convolved with the pre-computed kernel, the detection branch performs online estimation as a one-shot detection, as shown in FIG. 3. A forward pass on the detection branch is performed to obtain the classification and regression outputs and thus the top M suggestions. Specifically, some embodiments define Equation 2. After notation, these embodiments represent the classification and regression feature maps as point sets. [ka]

[0125] Since odd channels on a classification feature map represent positive activations, some embodiments may [ka] Collect the top K points in , where l is an odd number, and define the point set as [ka] where I, J, and L are some example index sets. The variables i and j encode the location of the corresponding anchors, respectively, and l encodes the proportion of the corresponding anchors. Thus, some embodiments may denote the corresponding anchor set as [ka] Furthermore, some embodiments may be derived as [ka] ANC * We find the activation of and define the corresponding refinement coordinate as [ka] Then, the refined top-K proposal set [ka] can be obtained by following equation 12 below: [ka]

[0126] After the top-K proposals are generated, some embodiments use some proposal selection strategies to make them suitable for the tracking task, which will be discussed in the next section.

[0127] To make the one-shot detection framework suitable for tracking tasks, some embodiments may employ two strategies to select proposals. The first proposal selection strategy discards bounding boxes generated by anchors that are too far from the center. For example, some embodiments may select only the central g × g sub-regions. [ka] We keep on the classification feature map and get g×g×k anchors instead of m×n×k anchors. Because neighboring frames do not always have large motion, the discarding strategy can efficiently remove outliers. Figure 4 illustrates the steps of selecting target anchors whose distance is less than or equal to 7 from the center in the classification feature map.

[0128] In a second proposal selection strategy, some embodiments use a cosine window and a scale change penalty to re-rank the proposal scores and obtain the best ones. After outliers are discarded, a cosine window is added to suppress large displacements, and then a penalty is added to suppress large changes in size and proportion. [ka]

[0129] Here, k is a hyperparameter, r represents the ratio of the height and width of the proposal, and r' represents that of the last frame, and s and s' represent the overall scale of the proposal and the last frame, which are calculated as follows: [ka]

[0130] where w and h represent the width and height of the target, and p represents the padding, which is equal to (w+h) / 2. After these operations, the top K proposals are re-ranked after multiplying their classification scores by a time penalty. Non-maximum suppression (NMS) is then performed to obtain the final tracking bounding box. After the final bounding box is selected, the target size is updated by linear interpolation to keep shape changes smooth.

[0131] 12D illustrates an exemplary center size in an RPN feature map in one or more embodiments. More specifically, FIG. 12D illustrates a center size of 7 in an RPN feature map, where each grid represents the encoded features of k anchors at the corresponding location. For example, there are 2k channels representing foreground and background activations in the classification feature map. The anchor center size indicates the search region of the model.

[0132] With regard to Alexnet, described above, some embodiments trained one of the largest convolutional neural networks to date on the subset of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitors, achieving by far the best results reported to date on these datasets. Some embodiments invoke a highly optimized GPU implementation of 2D convolutions and all other operations inherent in training a convolutional neural network, which some embodiments make publicly available. The network includes several novel and unusual features that improve its performance and reduce its training time, as detailed below. The size of the network, even with 1.2 million labeled training examples, makes overfitting a significant problem; therefore, some embodiments used several effective techniques, described below, to prevent overfitting. In some embodiments, the network includes five convolutional layers and three fully connected layers; this depth is considered significant. That is, some embodiments find that removing any convolutional layers (each of which contains 1% or less of the model's parameters) results in lower performance. In some embodiments, the size of the network is limited primarily by the amount of memory available on current GPUs and the amount of training time some embodiments can tolerate. In some embodiments, the network takes 5-6 days to train on two GTX 580 3GB GPUs. All experiments suggest that results could be improved by simply waiting until faster GPUs and larger datasets become available.

[0133] ImageNet is a dataset of over 15 million labeled, high-resolution images belonging to approximately 22,000 categories. The images were collected from the web and labeled by human labelers using Amazon's Mechanical Turk crowdsourcing tool. Starting in 2010, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held as part of the Pascal Visual Object Challenge. ILSVRC uses a subset of ImageNet, with approximately 1,000 images in each of 1,000 categories. In total, there are approximately 1.2 million training images, 50,000 validation images, and 150,000 test images.

[0134] ILSVRC-2010 is the only version of ILSVRC for which test set indicators are available, and therefore this is the version on which some embodiments conducted the majority of their experiments. Because some embodiments also input their models in the ILSVRC-2012 competition, some embodiments also report results on this version of the dataset for which test set indicators are unavailable. For ImageNet, it is customary to report two error rates: first and fifth place. The fifth place error rate is the percentage of test images for which the correct indicator is not considered the most probable of the five indicators by the model. While ImageNet consists of variable-resolution images, the present system requires constant input dimensions. Therefore, some embodiments downsampled images to a fixed resolution of 256 × 256. Given a rectangular image, some embodiments first rescaled the image so that the shorter side was 256 in length, and then cropped a central 256 × 256 patch from the resulting image. Some embodiments did not pre-process the images in any other way except by subtracting the mean activity from each pixel across the training set. Thus, some embodiments trained the network based on the (center) raw RGB (red-green-blue) values ​​of the pixels.

[0135] The standard way to model the output of a neuron, f, as a function of its input, x, is to use f(x) = tanh(x) or f(x) = (1 + e -x ) -1In terms of training time with gradient descent, these saturating nonlinearities are much slower than nonsaturating nonlinearities f(x) = max(0,x). Some embodiments refer to neurons with this nonlinearity as rectified linear units (ReLUs). Deep convolutional neural networks with ReLUs train several times faster than their counterparts with tanh units. This is demonstrated in Figure 12E, which shows the number of iterations required to reach a 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional network. This plot indicates that if some embodiments used a traditional saturated neuron model, they would not have been able to experiment with such large neural networks for this study.

[0136] Some embodiments prevent overfitting, and therefore the effect they observe is different from the accelerated ability to adapt the training set that some embodiments report when using ReLU. Faster learning has a significant impact on the performance of large models trained on large data sets.

[0137] Some embodiments spread the network across two GPUs. Current GPUs are particularly well suited to inter-GPU parallelism because they can read from and write to each other's memory without directly going through the host machine memory. The parallelization scheme employed by some embodiments essentially places half of the kernels (or neurons) on each GPU, with one additional trick: GPUs communicate only within a layer. This means, for example, that a kernel in layer 3 takes inputs from all kernel maps in layer 2. However, a kernel in layer 4 takes inputs only from those kernel maps in layer 3, which reside on the same GPU. Choosing the connectivity pattern is a problem for cross-validation, but it allows us to fine-tune the amount of communication until it is an acceptable fraction of the amount of computation.

[0138] ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce positive inputs to a ReLU, learning will occur within that neuron. However, some embodiments still find that the following local normalization scheme aids generalization: i x,y Let us denote the activity of a neuron by applying kernel i to location (x,y) and then applying a ReLU nonlinearity, and the response-normalized activity b i x,y is given by the following formula: [ka]

[0139] where the summation is over n "adjacent" kernel maps at the same spatial location, where N is the total number of kernels in the layer. The ordering of the kernel maps is, of course, arbitrary and is determined before training begins. This type of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for large activity among neuron outputs computed using different kernels. The constants k, n, α, and β are hyperparameters whose values ​​are determined using a validation set; some embodiments use k=2, n=5, α=10. -4 , and β=0.75 were used. Some embodiments applied this regularization after applying a ReLU nonlinearity within a layer.

[0140] This scheme has some similarities with local contrast normalization schemes, but ours will more correctly be referred to as "brightness normalization" as some embodiments do not subtract the average activity. Response normalization reduces the top-1 and top-5 error rates by 1.4% and 1.2% respectively. Some embodiments also verified the effectiveness of this scheme on the CIFAR-10 dataset, where a 4-layer CNN achieved a test error rate of 13% without normalization and 11% with normalization.

[0141] Pooling layers within a CNN summarize the output of groups of neighboring neurons within the same kernel map. Conventionally, the neighborhoods summarized by adjacent pooling units do not overlap. More precisely, a pooling layer can be considered to consist of a grid of pooling units separated by s pixels, each summarizing a neighborhood of size z×z centered at the location of the pooling unit. When s = z is set, some embodiments obtain conventional local pooling as commonly employed within a CNN. When s < z is set, some embodiments obtain overlapping pooling. This is what some embodiments use throughout the network with s = 2 and z = 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3% respectively compared to a non-overlapping scheme s = 2, z = 2 that produces an output of equivalent dimensions. Some embodiments generally observe that models with overlapping pooling are somewhat more difficult to overfit during training.

[0142] The network contains eight layers together with weights, the first five being convolutional and the last three being fully connected. The output of the last fully connected layer is fed into a 1,000-class softmax which produces a distribution over 1,000 class labels. The network maximizes multinomial logistic regression objectives which is equivalent to maximizing the average across training cases of the log probability of the correct label under the predicted distribution.

[0143] The kernels of the second, fourth, and fifth convolutional layers are connected only to their kernel maps in the previous layer, which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fully connected layer are connected to all neurons in the previous layer. A response normalization layer follows the first and second convolutional layers. A max-pooling layer of the type described in Section 3.4 follows both the response normalization layer and the fifth convolutional layer. A ReLU nonlinearity is applied to the outputs of all convolutional and fully connected layers.

[0144] The first convolutional layer filters a 224 x 224 x 3 input image using 96 kernels of size 11 x 11 x 3 with a stride of 4 pixels (the distance between the receptive field centers of neighboring neurons in the kernel map). The second convolutional layer takes the (response-normalized and pooled) output of the first convolutional layer as input and filters it using 256 kernels of size 5 x 5 x 48. The third, fourth, and fifth convolutional layers are interconnected without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3 x 3 x 256 connected to the (normalized and pooled) output of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3 x 3 x 192, and the fifth convolutional layer has 256 kernels of size 3 x 3 x 192. Each fully connected layer has 4,096 neurons.

[0145] In some embodiments, the neural network architecture has 60 million parameters. The 1,000 classes of ILSVRC force each training example to impose a 10-bit constraint on the image-to-landmark mapping, which proves insufficient to learn a very large number of parameters without massive overfitting. Below we describe two main ways in which some embodiments address overfitting.

[0146] In some embodiments, reducing overfitting on image data involves artificially expanding the dataset using a sign-preserving transformation. Some embodiments employ two distinct forms of data augmentation, both of which allow transformed images to be produced from original images with very little computation, and therefore do not need to be stored on disk. In one exemplary implementation, the transformed images are generated in Python code on the CPU while the GPU trains on a previous batch of images. In these embodiments, these data augmentation schemes are effectively computation-free.

[0147] The first form of data augmentation consists of generating image translations and horizontal reflections. Some embodiments do this by extracting random 224 x 224 patches (and their horizontal reflections) from a 256 x 256 image and training the network on these extracted patches. This increases the size of the training set by a factor of 2,048, but the resulting training examples are, of course, highly interdependent. Without this scheme, the network would suffer from substantial overfitting, which would force the use of a much smaller network. During testing, the network extracts five 224 x 224 patches (four corner patches and a central patch) and their horizontal reflections (hence, 10 patches in total) and makes predictions by averaging the predictions made on the 10 patches by the network's softmax layer.

[0148] A second form of data augmentation consists of modifying the intensities of RGB channels in training images. Specifically, some embodiments perform PCA on the set of RGB pixel values ​​throughout the entire ImageNet training set. For each training image, some embodiments add a multiple of the principal components found, with magnitude proportional to the corresponding eigenvalue times a random variable derived from a Gaussian with mean zero and standard deviation 0.1. Thus, for each RGB image pixel [ka] to which some embodiments add the following quantities: [ka]

[0149] In the formula, p i and λ i are the i-th eigenvector and eigenvalue of the 3 × 3 covariance matrix of RGB pixel values, respectively, and αi is the aforementioned random variable. Each αi is derived only once for all pixels of a particular training image until that image is used for training again, at which point it is re-derived. This scheme approximately captures an important property of natural images: object identification is invariant to changes in illumination intensity and color. This scheme reduces the first-rank error rate by more than 1%.

[0150] Combining the predictions of many different models is a highly successful method for reducing test error, but it is considered prohibitively expensive for large neural networks that already require several days to train. However, a highly efficient version of model combination exists that costs only about twice as much during training. A recently introduced technique called "dropout" consists of setting the output of each hidden neuron to zero with a probability of 0.5. Neurons that are "dropped out" in this way do not contribute to the forward pass or participate in backward propagation. Thus, each time an input is presented, the neural network samples a different architecture, but all these architectures share weights. This technique reduces the complex coadaptation of neurons because neurons cannot rely on the presence of specific other neurons. This therefore forces them to learn more robust features that are useful in conjunction with many different random subsets of other neurons. During testing, some embodiments use all neurons but multiply their outputs by 0.5, which is a reasonable approximation to finding the geometric mean of the prediction distribution produced by exponentially larger dropout networks.

[0151] Some embodiments use dropout in the first two fully connected layers of Figure 2. Without dropout, the network exhibits substantial overfitting. Dropout approximately doubles the number of iterations required to converge.

[0152] Some embodiments trained the model using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. Some embodiments found that this small amount of weight decay was important for the model to learn. In other words, the weight decay here is not simply a regularizer, but reduces the training error of the model. The update rule for the weights w was: [ka]

[0153] where i is the iteration index, v is the momentum variable, and ε is the learning rate. [ka] Ha, w i is the average over the i-th batch Di of the derivative of the objective with respect to w evaluated at

[0154] Some embodiments initialized the weights in each layer from a zero-mean Gaussian distribution with a standard deviation of 0.01. Some embodiments initialized the neuron biases in the second, fourth, and fifth convolutional layers and the fully connected hidden layer with the constant 1. This initialization accelerates the early stages of learning by providing positive inputs to the ReLU (rectified linear unit). Some embodiments initialized the neuron biases in the remaining layers with the constant 0.

[0155] Some embodiments used equal learning rates for all layers, which some embodiments manually adjusted throughout training. The heuristic some embodiments followed was to divide the learning rate by 10 when the validation error rate stopped improving at the current learning rate. The learning rate was initialized at 0.01 and reduced by a factor of 3 prior to termination. Some embodiments trained the network for approximately 90 cycles through a training set of 1.2 million images.

[0156] FIG. 13 illustrates further details of a portion of the high-level block diagram shown in FIG. 12A in one or more embodiments. More specifically, FIG. 13 illustrates further details of generating deep learning results by performing a deep learning process on an image using a deep learning model at 1206 in FIG. 12A. In one or more of these embodiments, features (e.g., vertices, edges, surfaces, etc.) or feature points (e.g., feature-related point nodes) within regions of the captured image and / or search image may be extracted at 1302 by using at least a deep convolutional neural network (DCNN) within the deep learning model. In some of these embodiments, the deep learning model may further include an image registration process in addition to the DCNN. Note that unless explicitly distinguished otherwise from each other, the terms “feature” and “feature point” may be used synonymously throughout the disclosure.

[0157] The features or feature points extracted at 1302 may be converted to multiple features including at least a first feature and a second feature at 1304. The region may be classified at 1306 based at least in part on the first feature, and a regression or correction may be determined for the region at 1308 based at least in part on the second feature. In some embodiments, the region may be classified at 1306 based only on the first feature and not on the second feature. Additionally or alternatively, the step of determining a regression for the region is based at least in part on the second feature and not on the first feature.

[0158]

[0031] Figure 14A illustrates further details of another portion of the high-level block diagram shown in Figure 12A in one or more embodiments. More specifically, Figure 14A illustrates further details of generating deep learning results by performing a deep learning process on an image using a deep learning model at 1206 in Figure 12A. In one or more of these embodiments, features or feature points within regions of the captured image and / or search image may be extracted at 1402 using a DCNN.

[0159] The features or feature points extracted at 1402 may be converted into multiple features including at least a first feature and a second feature at 1404. A first response map may be generated at 1406 by convolving the first feature using one or more convolutional layers in the DCNN, such as providing at least the second feature to a set of convolutional layers in the DCNN. Regions may then be classified at 1408 into positive regions (e.g., target regions) or negative regions (e.g., non-target regions that may be ignored) using at least the first response map.

[0160] A second response map may also be generated at 1410 by convolving the second features with a DCNN, such as providing the second features to a set of convolutional layers in the DCNN. A regression or correction may be determined for the region using at least the second response map at 1412. In some embodiments, a deep learning model including a deep convolutional neural network and / or a DCNN may be compensated at 1414 using at least an image registration process. In some of these embodiments, the deep learning model may include an image registration process in addition to a deep convolutional neural network (DCNN).

[0161] In some embodiments, the deep convolutional neural network illustrated in Figures 13 and / or 14A may include, for example, several new and rare features, which improve its performance and reduce its training time, as described in more detail below. The size of the network, even with 1.2 million labeled training examples, makes overfitting a significant problem; therefore, some embodiments use several effective techniques to prevent overfitting, as described in more detail below. The final network contains five convolutional layers and three fully connected layers; this depth is considered important; namely, it was found that removing any convolutional layers (each of which contains 1% or less of the model's parameters) results in poorer performance.

[0162] In some embodiments, the size of the network is limited primarily by the amount of memory available on current GPUs and the amount of training time that some embodiments can tolerate. The network takes 5-6 days to train on two GTX 580 3GB GPUs. All of these experiments suggest that results could be improved simply by waiting until faster GPUs and larger datasets become available.

[0163] Dataset:

[0164] ImageNet contains a dataset of over 15 million labeled, high-resolution images belonging to approximately 22,000 categories. The images were collected from the web and labeled by human labelers using Amazon's Mechanical Turk crowdsourcing tool. Starting in 2010, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held as part of the Pascal Visual Object Challenge. The ILSVRC uses a subset of ImageNet, with approximately 1,000 images in each of 1,000 categories. In total, there are approximately 1.2 million training images, 50,000 validation images, and 150,000 test images.

[0165] ILSVRC-2010 is the version of ILSVRC for which test set labels are available, and thus this is the version on which some embodiments conducted the majority of their experiments. Some embodiments also entered models in the ILSVRC-2012 competition, so some embodiments also report results on this version of the dataset for which test set labels are not available. For ImageNet, it is customary to report two error rates: first place and fifth place; the fifth place error rate is the proportion of test images for which the correct label, out of the five labels, is not considered the most probable by the model.

[0166] While ImageNet comprises variable-resolution images, the exemplary system requires constant input dimensions. Therefore, some embodiments downsampled the images to a fixed resolution of 256x256. Given a rectangular image, some embodiments first rescaled the image so that the shorter side was 256 in length, and then cropped a central 256x256 patch from the resulting image. Some embodiments did not preprocess the images in any other way except for subtracting the mean activity from each pixel across the training set. Some embodiments trained the network based on the (central) raw RGB values ​​of the pixels.

[0167] architecture:

[0168] The network architecture is summarized in Figure 2. It contains eight trained layers: five convolutional layers and three fully connected layers. Below, some embodiments describe some of the novel or unusual features of the network architecture.

[0169] ReLU nonlinearity:

[0170] The standard way to model the output of a neuron, f, as a function of its input, x, is to use f(x) = tanh(x) or f(x) = (1 + e -x ) -1In terms of training time with gradient descent, these saturating nonlinearities are much slower than nonsaturating nonlinearities f(x) = max(0,x). Some embodiments refer to neurons with this nonlinearity as rectified linear units (ReLUs). Deep convolutional neural networks with ReLUs train several times faster than their counterparts with tanh units. This is demonstrated in Figure 12E, which shows the number of iterations required to reach a 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional network. This plot indicates that if some embodiments used a traditional saturated neuron model, they would not have been able to experiment with such large neural networks for this study.

[0171] Some embodiments consider replacing traditional neuron models in CNNs. In some embodiments, the nonlinearity f(x) = |tanh(x)| performs well with that type of contrast normalization, followed by local average pooling on the Caltech-101 dataset. However, with this dataset, the primary concern is preventing overfitting, and therefore the effect they observe is different from the accelerated ability to adapt the training set that some embodiments report when using ReLU. Faster learning significantly impacts the performance of large models trained on large datasets.

[0172] Local response normalization:

[0173] ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce positive inputs to a ReLU, learning will occur within that neuron. However, some embodiments still find that the following local normalization scheme aids generalization: i x,yLet us denote the activity of a neuron by applying kernel i to location (x,y) and then applying a ReLU nonlinearity, and the response-normalized activity b i x,y is given by the following formula: [ka]

[0174] where the summation is over n "adjacent" kernel maps at the same spatial location, where N is the total number of kernels in the layer. The ordering of the kernel maps is, of course, arbitrary and is determined before training begins. This type of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for large activity among neuron outputs computed using different kernels. The constants k, n, α, and β are hyperparameters whose values ​​are determined using a validation set; some embodiments use k=2, n=5, α=10. -4 , and β=0.75 were used. Some embodiments applied this regularization after applying a ReLU nonlinearity within a layer.

[0175] The embodiments described herein may more correctly be referred to as "brightness normalization" because some embodiments do not subtract the mean activity. Response normalization reduces the first- and fifth-place error rates by 1.4% and 1.2%, respectively. Some embodiments also validated the effectiveness of this scheme on the CIFAR-10 dataset, where a four-layer CNN achieved a test error rate of 13% without normalization and 11% with normalization.

[0176] Duplicate Pooling:

[0177] The pooling layer within the CNN summarizes the output of a neighborhood group of neurons within the same kernel map. Conventionally, the neighborhoods summarized by adjacent pooling units do not overlap. More precisely, the pooling layer can be considered to consist of a grid of pooling units separated by s pixels, each summarizing a neighborhood of size z×z centered at the location of the pooling unit. When s is set such that s = z, some embodiments obtain conventional local pooling as is generally employed within CNNs. When s is set such that s < z, some embodiments obtain overlapping pooling. This is what some embodiments use throughout the network with s = 2 and z = 3. This scheme reduces the first and fifth error rates by 0.4% and 0.3% respectively, compared to the non-overlapping scheme s = 2, z = 2 which produces outputs of equivalent dimensions. Some embodiments generally observe that models with overlapping pooling are somewhat more difficult to overfit during training.

[0178] Overall architecture:

[0179] Some embodiments are ready to describe the overall architecture of an exemplary CNN. As depicted in FIG. 2, the network includes eight layers, along with weights, with the first five being convolutional and the remaining three being fully connected. The output of the last fully connected layer is fed into a 1,000-class softmax, which produces a distribution over 1,000 class labels. The exemplary network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log probability of the correct label under the predictive distribution.

[0180] The kernels of the second, fourth, and fifth convolutional layers are connected only to their kernel maps in previous layers that reside on the same GPU (see FIG. 14B, which illustrates an exemplary deep convolutional neural network in one or more embodiments). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fully connected layer are connected to all neurons in the previous layer. A response normalization layer follows the first and second convolutional layers. A max-pooling layer of the type described below follows both the response normalization layer and the fifth convolutional layer. A ReLU nonlinearity is applied to the outputs of all convolutional and fully connected layers.

[0181] The first convolutional layer filters a 224 x 224 x 3 input image using 96 kernels of size 11 x 11 x 3 with a stride of 4 pixels (the distance between the receptive field centers of neighboring neurons in the kernel map). The second convolutional layer takes the (response-normalized and pooled) output of the first convolutional layer as input and filters it using 256 kernels of size 5 x 5 x 48. The third, fourth, and fifth convolutional layers are interconnected without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3 x 3 x 256 connected to the (normalized and pooled) output of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3 x 3 x 192, and the fifth convolutional layer has 256 kernels of size 3 x 3 x 192. Each fully connected layer has 4,096 neurons.

[0182] Reducing overfitting:

[0183] An exemplary neural network architecture has 60 million parameters. The 1,000 classes of ILSVRC force each training example to impose a 10-bit constraint on the image-to-landmark mapping, which proves insufficient to learn a very large number of parameters without massive overfitting. Below we describe two main ways some embodiments address overfitting.

[0184] Data Augmentation:

[0185] One method for reducing overfitting on image data is to artificially augment the dataset using a sign-preserving transformation. Some embodiments employ two distinct forms of data augmentation, both of which allow transformed images to be produced from original images with very little computation, and thus do not need to be stored on disk. In one exemplary implementation, the transformed images are generated in Python code on the CPU while the GPU trains on a previous batch of images. Thus, these data augmentation schemes are effectively computation-free.

[0186] The first form of data augmentation consists of generating image translations and horizontal reflections. Some embodiments do this by extracting random 224 x 224 patches (and their horizontal reflections) from a 256 x 256 image and training the network on these extracted patches. This increases the size of the training set by a factor of 2,048, but the resulting training examples are, of course, highly interdependent. Without this scheme, the network would suffer from substantial overfitting, which would force the use of a much smaller network. During testing, the network extracts five 224 x 224 patches (four corner patches and a central patch) and their horizontal reflections (hence, 10 patches in total) and makes predictions by averaging the predictions made on the 10 patches by the network's softmax layer.

[0187] A second form of data augmentation consists of modifying the intensities of RGB channels in training images. Specifically, some embodiments perform PCA on the set of RGB pixel values ​​throughout the entire ImageNet training set. For each training image, some embodiments add a multiple of the principal components found, with magnitude proportional to the corresponding eigenvalue times a random variable derived from a Gaussian with mean zero and standard deviation 0.1. Thus, for each RGB image pixel [ka] to which some embodiments add the following quantities: [ka]

[0188] In the above formula, p i and λ i are the i-th eigenvector and eigenvalue of the 3 × 3 covariance matrix of RGB pixel values, respectively, and αi is the aforementioned random variable. Each αi is derived only once for all pixels of a particular training image until that image is used for training again, at which point it is re-derived. This scheme approximately captures an important property of natural images: object identification is invariant to changes in illumination intensity and color. This scheme reduces the first-rank error rate by more than 1%.

[0189] Dropout:

[0190] Combining the predictions of many different models is a highly successful method for reducing test error, but it is considered prohibitively expensive for large neural networks that already take several days to train. However, a highly efficient version of model combination exists that costs only about twice as much during training. A recently introduced technique called "dropout" involves setting the output of each hidden neuron to zero with a probability of 0.5. Neurons that are "dropped out" in this way do not contribute to the forward pass or participate in backward propagation. Thus, each time an input is presented, the neural network samples a different architecture, but all these architectures share weights. This technique reduces the complex coadaptation of neurons because neurons cannot rely on the presence of specific other neurons. This therefore forces them to learn more robust features that are useful in conjunction with many different random subsets of other neurons. During testing, some embodiments use all neurons but multiply their outputs by 0.5, which is a reasonable approximation to finding the geometric mean of the prediction distribution produced by exponentially larger dropout networks.

[0191] Some embodiments use dropout in the first two fully connected layers of Figure 2. Without dropout, the network exhibits substantial overfitting. Dropout approximately doubles the number of iterations required to converge.

[0192] Learn more:

[0193] Some embodiments trained the model using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. Some embodiments found that this small amount of weight decay was important for the model to learn. In other words, the weight decay here is not simply a regularizer, but reduces the training error of the model. The update rule for the weights w was: [ka]

[0194] where i is the iteration index, v is the momentum variable, and ε is the learning rate. [ka] Ha, w i is the average over the i-th batch Di of the derivative of the objective with respect to w evaluated at

[0195] Some embodiments initialized the weights in each layer from a zero-mean Gaussian distribution with a standard deviation of 0.01. Some embodiments initialized the neuron biases in the second, fourth, and fifth convolutional layers and the fully connected hidden layer with the constant 1. This initialization accelerates the early stages of learning by providing positive inputs to the ReLU. Some embodiments initialized the neuron biases in the remaining layers with the constant 0.

[0196] Some embodiments used equal learning rates for all layers, which some embodiments manually adjusted throughout training. The heuristic some embodiments followed was to divide the learning rate by 10 when the validation error rate stopped improving at the current learning rate. The learning rate was initialized at 0.01 and reduced by a factor of 3 prior to termination. Some embodiments trained the network for approximately 90 cycles through a training set of 1.2 million images.

[0197] 15A illustrates further details of another portion of the high-level block diagram shown in FIG. 12A in one or more embodiments. More specifically, FIG. 15A illustrates further details of localizing an image by performing an image registration process using at least a Kalman filter model and the aforementioned deep learning results. In one or more of these embodiments, the deep learning process and / or the deep learning results generated by the deep learning process may be embedded in a state transition model at 1502A. In some embodiments, the state transition model determines a next position state at a next time point based, at least in part, on a previous position state at a previous time point and at least one of process noise or a control vector of a state estimation model. In some of these embodiments, the state transition model includes a first-order state estimation model.

[0198] An example state transition model may be as follows: [ka]

[0199] In the above equation (1), X k , Y k represents the position state at time k in the x and y directions. k is the process noise drawn from a zero-mean multivariate normal distribution in some embodiments. [ka] forms the control vector of the first-order state estimation model. It is the rate in time units calculated from the difference between the deep neural network results at time k and k-1. The proposed formulation allows embedding deep learning within a classical Kalman filter model, where one time step is the duration between successive frames.

[0200] The aforementioned deep learning model generated by the deep learning results may be received as measurements at 1504A in the Kalman filter model by using at least the state transition model described above with reference to 1502A. In some embodiments, the Kalman filter model is enabled to receive the deep learning results by using a state transition model in which the deep learning process and / or deep learning results are embedded.

[0201] The measurement value at the next time point may be determined at 1506A by at least using a Kalman filter model in the image registration process. In some embodiments, the time step may be determined as the duration between two immediately neighboring frames. For example, in a 30 frame / second video sequence, the time step may be determined as 1 / 30 seconds. Note that other consistent fixed time steps or even variable time steps may be used in different embodiments. In some embodiments, interference of similar backgrounds and / or features in the captured image and / or the search image may be reduced by at least performing an image registration process at 1508A. In some embodiments, the Kalman filter model may be configured to receive the image registration results. Additionally or alternatively, the Kalman filter model may comprise a linear model (e.g., a linear Markov model or any other suitable model) and additive Gaussian noise.

[0202] The captured image frames may be located relative to or within the search image at 1510A by using at least a Kalman filter model and an image registration process. Further details about some of the foregoing acts will be described below with reference to Figures 15B and 15C.

[0203] Figure 15B illustrates further details of a portion of Figure 15A in one or more embodiments. More specifically, Figure 15B illustrates further details of embedding a deep learning process into a state transition model at 1502A of Figure 15A. In one or more of these embodiments, one or more control vectors may be determined for the state transition model at 1502B. Process noise may be derived from a statistical distribution, such as, but not limited to, a zero-mean multivariate normal distribution, at 1504B.

[0204] A time step may be determined for the state transition model at 1506B. As explained above, the time step may in some embodiments be determined to be the duration between two immediately neighboring frames, but it should be noted that other consistent fixed time steps or even variable time steps may also be used in different embodiments.

[0205] The state transition model may be used at 1508B to determine one or more next position states at one or more corresponding points in time by using at least one or more previous position states at one or more corresponding previous points in time based, at least in part, on the process noise derived at 1504B and / or the one or more control vectors determined at 1502B. The example state transition model described above with reference to FIG. 15A allows for the embedding of a deep learning process into a Kalman filter model.

[0206] 15C illustrates further details of another portion of FIG. 15A in one or more embodiments. More specifically, FIG. 15B illustrates further details of locating an image within or relative to a search image by using a Kalman filter model and an image registration process at 1510A of FIG. 15A. In one or more of these embodiments, locating the captured image may include performing a rough registration for the entire frame of the captured image at 1502C. In some of these embodiments, performing the rough registration includes detecting one or more features or feature points within one or more regions within the entire frame of the captured image at 1506C.

[0207] In some of these embodiments, performing the rough alignment may further include detecting (if rough alignment has not been performed for the search image) or identifying (if rough alignment has already been performed for the search image) one or more corresponding features or feature points from one or more regions in the search image. Capturing the image may then, at 1508C, be aligned to the search image based, at least in part, on the one or more features or feature points detected from the captured image and one or more corresponding features or feature points from the search image.

[0208] Once the coarse registration is complete, a fine registration may be performed at 1504C with respect to at least one of the one or more features or feature points detected from the captured image and / or at least one of one or more corresponding features or feature points from the search image. In some embodiments, the coarse registration may be performed on the entire frame of the image, while the fine registration may be performed using only the outer ring. In these embodiments, the use of a Kalman filter model for measurements using the outer ring registration avoids repeated calculation of features or feature points. In some of these embodiments, the deep learning model includes a convolutional neural network and an image registration process.

[0209] In some embodiments, fine registration may be performed with respect to at least one feature or feature point within an outer annular region around the region by using a Kalman filter at 1504C. In some of these embodiments, performing the fine registration may include selecting a feature or feature point within the enlarged region at 1510C that corresponds to the region described above with reference to 1506C with respect to the coarse registration. Using an enlarged region may improve the robustness of the localization process, particularly in scenarios where the matched features or feature points in the coarse registration are more concentrated within an inner area or an area closer to the interior (as opposed to closer to the outer annular region or otherwise dispersed across the search image).

[0210] The features or feature points selected in 1510C may then be compared with corresponding features or feature points in the outer region on the search image. In some implementations of eye tracking where the search image includes multiple mosaicked base images of the user's retina, some embodiments take the imaging position on the retina as the ground truth once a frame is completed. As a result, the outer ring in each frame is closer to the ground truth.

[0211] Regarding the Kalman filter model, the Kalman filter comprises a set of mathematical equations that provide an efficient computational (recursive) means for estimating the state of a process so as to minimize the mean squared error. The filter is very powerful in several aspects: it supports estimation of past, present, and even future states, and can do so even when the precise nature of the modeled system is unknown.

[0212] Discrete Kalman Filter:

[0213] The Kalman filter is a method for estimating the state of a discrete-time control process governed by a linear stochastic difference equation. [ka] This addresses the general problem of trying to estimate [ka]

[0214] Measurements [ka] If , then this becomes: [ka]

[0215] Random variable w k and v k represent the process and measurement noise (respectively), which are assumed to be independent (from each other), white, and with normal probability distributions. [ka]

[0216] In practice, the process noise covariance Q and measurement noise covariance R matrices may change with each time step or measurement, however, some embodiments herein assume they are constant.

[0217] The n x n matrix A in difference equation (1.1) relates the state at the previous time step k-1 to the state at the current step k, in the absence of either a driving function or process noise. Note that in practice, A may vary with each time step, but some embodiments assume it is constant here. The n x Z matrix B represents the arbitrary control inputs [ka] to the state x. The m×n matrix H in the measurement equation (1.2) relates the state to the measurements zk. In practice, H may vary with each time step or measurement, but here some embodiments assume it is constant.

[0218] Some embodiments include: [ka] (note the "-" superscript) is the prior state estimate at step k, given knowledge of the process prior to step k, [ka] is given the measurement z k Define σ k to be the posterior state estimate at step k given σ k . Some embodiments then define the a priori and posterior estimation errors as follows: [ka]

[0219] The a priori estimated error covariance is then: [ka]

[0220] The posterior estimated error covariance is: [ka]

[0221] In deriving the equations for the Kalman filter, some embodiments use the a posteriori state estimate [ka] In (1.7), the prior estimate [ka] and the actual measured value z k and predicted values [ka] We start with the goal of finding an equation that is calculated as a linear combination of weighted differences between [ka]

[0222] Difference in (1.7) [ka] is called the measurement error, or residual. [ka] and the actual measured value z k A residual of zero means that the two are in perfect agreement.

[0223] The n × m matrix K in (1.7) is chosen to be the gain or mixing coefficient that minimizes the a posteriori error covariance (1.6). This minimization is done by first substituting (1.7) for e k This can be accomplished by substituting into the above definition of K, substituting it into (1.6), performing the expected value shown, taking the derivative of the resulting trace with respect to K, setting the result equal to zero, and then solving for K. One form of the resulting K that minimizes (1.6) is given by: [ka]

[0224] From equation (1.8), as the measurement error covariance R approaches zero, the gain K weights the residual more heavily. Specifically, [ka]

[0225] On the other hand, as the prior estimation error covariance Pk approaches zero, the gain K weights the residual more lightly. [ka]

[0226] Another way to think about weighting by K is that as the measurement error covariance R approaches zero, the actual measurement z k are increasingly "trusted", while predicted measurements [ka] On the other hand, as the prior estimation error covariance Pk approaches zero, the actual measurement z k is increasingly unreliable, while predicted measurements [ka] is becoming more and more trusted.

[0227] The justification for (1.7) is based on the probability of the prior estimate ick conditioned on all previous measurements zk (Bayes' theorem). For the moment, this is sufficient to show that the Kalman filter preserves the first two moments of the state distribution. [ka]

[0228] The posterior state estimate (1.7) reflects the mean (first moment) of the state distribution, i.e., it is normally dispersed if the conditions of (1.3) and (1.4) are satisfied. The posterior estimation error covariance (1.6) reflects the variance (second non-central moment) of the state distribution. In other words: [ka]

[0229] Discrete Kalman Filter Algorithm:

[0230] A Kalman filter estimates a process by using a form of feedback control. That is, the filter estimates the process state at a point in time and then receives feedback in the form of (noisy) measurements. The equations for a Kalman filter are therefore classified into two families: time update equations and measurement update equations. The time update equations involve projecting the current state and error covariance estimates forward (in time) to obtain a priori estimates for the next time step. The measurement update equations involve feedback, for example, incorporating new measurements into the prior estimates to obtain improved a posteriori estimates.

[0231] The time update equations can also be considered as prediction equations, while the measurement update equations can be considered as correction equations. In fact, the final estimation algorithm is similar to that of a prediction-correction algorithm for solving the numerical problem, as shown immediately below. [ka]

[0232] Specific equations for time and measurement updates are presented below in Table 1-1 and Table 1-2 immediately below. [Table 1]

[0233] The first task during the measurement update is to calculate the Kalman gain K k Note that the equation given here as (1.11) is identical to (1.8). The next step is to actually measure the process and calculate Z k and then generate the posterior state estimate by incorporating measurements as in (1.12). Again (1.12) is simply (1.7), repeated here for completeness. The final step is to obtain the posterior error covariance estimate via (1.13).

[0234] After each time and measurement update pair, the process is repeated, with the previous posterior estimate being used to project or predict a new prior estimate. This recursive nature is one of the very attractive features of the Kalman filter: it makes practical implementations much more feasible than (for example) Wiener filter implementations, which are designed to operate directly on all of the data for each estimate. The Kalman filter instead recursively conditions the current estimate on all of the past measurements.

[0235] Filter parameters and adjustments:

[0236] In practical implementations of filters, the measurement noise covariance R is typically measured prior to operation of the filter. Measuring the measurement error covariance R is generally practical (possible) because some embodiments need to be able to measure the process anyway (while operating the filter), and therefore these embodiments should generally be able to make some offline sample measurements to determine the variance of the measurement noise.

[0237] Determining the process noise covariance Q is generally more difficult because some embodiments typically do not have the ability to directly observe the process they are estimating. Sometimes a relatively simple (bad) process model can produce acceptable results if one "injects" enough uncertainty into the process through the selection of Q. Certainly, in this case, one would want to have confidence in the process measurements.

[0238] In any case, whether or not some embodiments have a rational basis for selecting the parameters, in many cases better filter performance (statistically speaking) can be obtained by tuning the filter parameters Q and R. The tuning step is usually frequently performed offline with the aid of another (distinct) Kalman filter, in a process generally referred to as system identification, as described below. [ka]

[0239] Under the condition that Q and R are virtually constant, the estimation error covariance Pk and the Kalman gain K k Both P and P will quickly stabilize and then remain constant (see the filter update equations immediately above). If this is the case, these parameters can be updated by running the filter offline or by updating, for example, the value P k can be pre-calculated either by determining the steady state of

[0240] However, measurement error often (especially at high frequencies) does not remain constant. For example, when targeting beacons in an optoelectronic tracker ceiling panel, the noise in measurements of nearby beacons will be smaller than that of distant beacons. Also, the process noise Q is sometimes dynamically changed during filter operation to adjust for different dynamics, i.e., Q k For example, in tracking the head of a user in a 3D virtual environment, some embodiments may reduce Q if the user is considered to be moving slowly. k If the dynamics begin to change rapidly, the magnitude of Q may decrease. k can be chosen to take into account both uncertainty about the user's intention and uncertainty in the model.

[0241] Extended Kalman Filter (EKF):

[0242] As explained above, the Kalman filter is used to calculate the state of a discrete-time control process governed by a linear stochastic difference equation. [ka] This paper addresses the general problem of attempting to estimate the process vector σ. But what happens when the process to be estimated and / or the measurement relationship to the process is nonlinear? Some of the most interesting and successful applications of Kalman filtering are in such situations. A Kalman filter that linearizes the current mean and covariance is called an extended Kalman filter, or EKF. In an analogy to a Taylor series, some embodiments may linearize the estimate around the current estimate using partial derivatives of the process and measurement functions to compute the estimate despite the nonlinear relationship. To do so, some embodiments start by modifying some of the material presented above. If the process is again [ka] where the process is assumed here to be governed by a nonlinear stochastic difference equation. [ka]

[0243] where the random variable w k and v k represents the process and measurement noise, again as in (1.3) and (1.4). In this case, the nonlinear function f in the difference equation (2.1) relates the state at the previous time step k-1 to the state at the current time step k. This is achieved by taking as parameters any driving function u k-1 and zero-mean process noise w k The nonlinear function h in the measurement equation (2.2) is k The measured value z k Relate to.

[0244] In practice, of course, the noise w k and v k The individual values ​​of , are not known. However, the state and measurement vectors can be approximated without them as follows: [ka]

[0245] Note that the fundamental flaw of EKFs is that the distributions (or densities in the persistent case) of the various random variables are no longer normal after undergoing their respective nonlinear transformations. EKFs are simply ad-hoc state estimators that only approximate the optimality of Bayes' theorem by linearization. Some interesting work has been done by Julier et al. in developing fluctuations for EKFs using methods that preserve normal distributions throughout nonlinear transformations.

[0246] To estimate processes with nonlinear difference and measurement relationships, some embodiments begin by writing new governing equations that linearize the estimates for (2.3) and (2.4). [ka]

[0247] Note that for notational convenience, some embodiments may not use the time step index k with the Jacobian matrices A, W, H, and V, even though in fact they are different at each time step.

[0248] Some embodiments provide a new notation for prediction error: [ka] Define

[0249] x in (2.7) k Recall that in practice, this is the actual state vector, e.g., the quantity we try to estimate. On the other hand, z in (2.8) k This means that if you have access to x k is the actual measurement that we are using to estimate . Using (2.7) and (2.8), some embodiments may write the governing equation for the error process as follows: [ka]

[0250] Note that equations (2.9) and (2.10) are linear and they are very similar to the difference and measurement equations (1.1) and (1.2) from the discrete Kalman filter. This is because the actual measurement residual in (2.8) [ka] and the prediction error given by (2.9) [ka] This motivates the use of a second (hypothetical) Kalman filter to estimate . [ka] ) can then be used with (2.7) to obtain the posterior state estimate for the original nonlinear process as follows: [ka]

[0251] The random variables in (2.9) and (2.10) have approximately the following probability distributions: [ka]

[0252] Given these approximations, [ka] If the predicted value of is set to zero, then [ka] The Kalman filter equation used to estimate σ is: [ka]

[0253] By substituting (2.12) back into (2.11) and utilizing (2.8), a second (hypothetical) Kalman filter may not be needed or used. [ka]

[0254] Equation (2.13) is now written as ick and Z resulting from (2.3) and (2.4) with appropriate substitutions for the measurement error covariances. k and the Kalman gain K resulting from (1.11) k may be used for measurement updates in the extended Kalman filter at

[0255] The complete set of EKF equations is shown below in Tables 2-1 and 2-2 below. Note that some embodiments may substitute ick and remain consistent with the previous "superscript-" prior notation, and some embodiments may append the subscript k to the Jacobian matrices A, W, H, and V to emphasize the idea that they are different (and therefore may be recalculated) at each time step. [Table 2-1]

[0256] Similar to the basic discrete Kalman filter, the time update equations in Table 2-1 project the state and covariance estimates from the previous time step k-1 onto the current time step k. Again, f in (2.14) arises from (2.3), and A k and W k is the process Jacobian matrix at step k, and Q k is the process noise covariance (1.3) at step k. [Table 2-2]

[0257] Similar to the basic discrete Kalman filter, the measurement update equations in Table 2-2 map the state and covariance estimates to the measurements z k Again, h in (2.17) arises from (2.4), and H k and V is the measurement Jacobian matrix at step k, and R kis the measurement noise covariance (1.4) at step k (note that some embodiments allow for the subscript R here to vary with each measurement).

[0258] The basic operation of the EKF is identical to that of a linear discrete Kalman filter as shown below, which combines the high-level diagram in Figure 1 with the equations from Tables 2-1 and 2-2 to give a complete picture of the operation of the EKF. [ka]

[0259] An important feature of the EKF is that the Kalman gain K k The Jacobian matrix H in the equation for k The role of the metric is to correctly propagate or "amplify" only the relevant components of the measurement information. For example, through h, the measurement Z k If there is no one-to-one mapping between and the state, then the Jacobian matrix H k is the residual that influences the state [ka] affects the Kalman gain so that it only expands a portion of the measurement Z through h. k If there is not a one-to-one mapping between the state and the filter, as might be expected, the filter will diverge quickly. In this case, the process may be unobservable.

[0260] Process model for estimating random constants:

[0261] The following attempts to estimate a scalar random constant, e.g., voltage. Some embodiments have the ability to make measurements of the constant, but the measurements are corrupted by 0.1-volt RMS (root-mean-square) white measurement noise (e.g., analog-to-digital converters are not very accurate). In this example, the process is governed by a linear difference equation. [ka]

[0262] The following measurements [ka] That is, [ka] accompanied by.

[0263] The state does not change from step to step, so A = 1. There is no control input, so u = 0. The noise measurement is directly in the state, so H = 1 (note that these embodiments omit the subscript k in some places because the individual parameters remain constant in this simple model).

[0264] Filter equations and parameters:

[0265] The time update equation is: [ka]

[0266] The measurement update equation is: [ka]

[0267] Assuming very little process variance, some embodiments may be implemented with Q=1e -5While some other embodiments may have Q=0, we assume that a slightly non-zero value gives more flexibility to "tune" the filter, as some embodiments will demonstrate below. In some embodiments, the true value of the random constant has a standard normal probability distribution, and thus some embodiments will "seed" the filter with a guess that the constant is 0. In other words, before starting, some embodiments [ka] Let's say.

[0268] Similarly, some embodiments may include P k-1 We may choose an initial value (called P0) for the initial state estimate [ka] is correct in some embodiments, these embodiments may let P0=0. However, the initial estimate [ka] Given the uncertainty in , choosing P0 = 0 means that the filter is initially and always [ka] In conclusion, the alternative is not important. Some embodiments may choose almost any P0≠0 and the filter may eventually converge. Some embodiments may start the filter with P0=1.

[0269] Various embodiments described herein may be implemented on other types of systems, such as a head-mounted display (HMD) or standalone computing node, such as an XR system, or a clustered environment involving multiple computing nodes. The system may include a display subsystem (e.g., a projector or micro-projector array) to project virtual content to a user. FIG. 11A illustrates an exemplary configuration of an array of micro-projectors and the coupling of the array of micro-projectors with an optical system, in one or more embodiments. More specifically, FIG. 11A shows an exemplary architecture 2500 for electronics operably coupled to an optical system or XR device, in one or more embodiments. The optical system or XR device itself or an external device (e.g., a beltpack) coupled to the XR device may include one or more printed circuit board components, such as left (2502) and right (2504) printed circuit board assemblies (PCBAs). As shown, the left PCBA 2502 contains most of the active electronics, while the right PCBA 604 supports primarily the display or projector elements.

[0270] The right PCBA 2504 may include several projector driver structures, which provide image information and control signals to the image generation components. For example, the right PCBA 2504 may carry a first, or left, projector driver structure 2506 and a second, or right, projector driver structure 2508. The first or left projector driver structure 2506 splices a set of signal lines (e.g., piezo driver wires) with a first, or left, projector fiber 2510. The second, or right, projector driver structure 2508 splices a set of signal lines (e.g., piezo driver wires) with a second, or right, projector fiber 2512. A first, i.e., left projector driving structure 2506 is communicatively coupled to the first, i.e., left image projector, while a second, i.e., right projector driving structure 2508 is communicatively coupled to the second, i.e., right image projector.

[0271] In operation, the image projector renders virtual content to the user's left and right eyes (e.g., retinas) via separate optical components, e.g., waveguides and / or compensatory lenses, and modifies the light associated with the virtual image.

[0272] The image projector may include, for example, left and right projector assemblies. The projector assemblies may use a variety of different image formation or production technologies, such as a fiber scanning projector, a liquid crystal display (LCD), an LCOS (liquid crystal on silicon) display, or a digital light processing (DLP) display. When a fiber scanning projector is employed, the image may be delivered along an optical fiber and projected therefrom via the tip of the optical fiber. The tip may be oriented so as to be routed within a waveguide. The tip of the optical fiber may project the image, which may be supported to flex or oscillate. Several piezoelectric actuators may control the oscillation (e.g., frequency, amplitude) of the tip. A projector driver structure provides the image to the individual optical fibers, controls the signals, controls the piezoelectric actuators, and projects the image to the user's eye.

[0273] Continuing with the right PCBA 2504, the button board connector 2514 may provide communication and physical coupling to a button board 2516, which carries various user-accessible buttons, keys, switches, or other input devices. The right PCBA 2504 may include a right earbud or speaker connector 2518 to communicatively couple audio signals to a right earbud 2520 or speaker of the head-worn component. The right PCBA 2504 may also include a right microphone connector 2522 to communicatively couple audio signals from a microphone of the head-worn component. The right PCBA 2504 further includes a right occlusion driver connector 2524 to communicatively couple occlusion information to a right occlusion display 2526 of the head-worn component. The right PCBA 2504 may also include a board-to-board connector to provide communication with the left PCBA 2502 via its board-to-board connector 2534.

[0274] The right PCBA 2504 may be communicatively coupled to one or more right outward-facing, or world-view, cameras 2528, either body- or head-mounted, and optionally a right camera visual indicator (e.g., LED) that illuminates to indicate to others that an image is being captured. The right PCBA 2504 may be communicatively coupled to one or more right-eye cameras 2532 carried by a head-mounted component that are positioned and oriented to capture right-eye images and enable tracking, detection, or monitoring of the orientation and / or movement of the right eye. The right PCBA 2504 may optionally be communicatively coupled to one or more right-eye illumination sources 2530 (e.g., LEDs), which illuminate the right eye with a pattern of illumination (e.g., temporal, spatial) to facilitate tracking, detection, or monitoring of the orientation and / or movement of the right eye, as described herein.

[0275] The left PCBA 2502 may include a control subsystem, which may include one or more controllers (e.g., a microcontroller, a microprocessor, a digital signal processor, a graphical processing unit, a central processing unit, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) 2540, and / or a programmable logic unit (PLU)). The control system may include one or more non-transitory computer- or processor-readable media that store executable logic or instructions and / or data or information. The non-transitory computer- or processor-readable media may take various forms, such as, for example, volatile and non-volatile forms, e.g., read-only memory (ROM), random access memory (RAM, DRAM, SD-RAM), flash memory, etc. The non-transitory computer- or processor-readable media may be formed, for example, as one or more registers of a microprocessor, FPGA, or ASIC.

[0276] The left PCBA 2502 may include a left earphone or speaker connector 2536 to communicatively couple an audio signal to a left earphone or speaker 2538 of the head-worn component. The left PCBA 2502 may include an audio signal amplifier (e.g., a stereo amplifier) ​​2542, which is communicatively coupled to the driven earphone or speaker. The left PCBA 2502 also includes a left microphone connector 2544 to communicatively couple an audio signal from a microphone of the head-worn component. The left PCBA 2502 further includes a left occlusion driver connector 2546 to communicatively couple occlusion information to a left occlusion display 2548 of the head-worn component.

[0277] The left PCBA 2502 may also include one or more sensors or transducers that detect, measure, capture, or otherwise sense information about the surrounding environment and / or about the user. For example, an acceleration transducer 2550 (e.g., a three-axis accelerometer) may detect acceleration in three axes, thereby detecting movement. A gyroscope sensor 2552 may detect orientation and / or magnetic or compass heading or orientation. Other sensors or transducers may be employed as well.

[0278] The left PCBA 2502 may be communicatively coupled to one or more left outward-facing or world-view cameras 2554, which may be body- or head-mounted, and optionally to a left camera visual indicator (e.g., LED) 2556 that illuminates to indicate to others that an image is being captured. The left PCBA may be communicatively coupled to one or more left-eye cameras 2558 carried by a head-mounted component that is positioned and oriented to capture images of the left eye and enable tracking, detection, or monitoring of the orientation and / or movement of the left eye. The left PCBA 2502 may optionally be communicatively coupled to one or more left-eye illumination sources (e.g., LEDs) 2556, which illuminate the left eye with patterns of illumination (e.g., temporal, spatial) to facilitate tracking, detection, or monitoring of the orientation and / or movement of the left eye, as described herein.

[0279] The PCBAs 2502 and 2504 are communicatively coupled to distinct computing components (e.g., belt packs) via one or more ports, connectors, and / or pathways. For example, the left PCBA 2502 may include one or more communication ports or connectors to provide communication (e.g., bidirectional communication) with the belt pack. The one or more communication ports or connectors may also provide power from the belt pack to the left PCBA 2502. The left PCBA 2502 may include power conditioning circuitry 2580 (e.g., DC / DC power converters, input filters) electrically coupled to the communication ports or connectors and operable to condition (e.g., step up voltage, step down voltage, smooth current, reduce transient current).

[0280] The communications port or connector may take the form of, for example, a data and power connector or transceiver 2582 (e.g., Thunderbolt® port, USB® port). The right PCBA 2504 may include a port or connector and receive power from the belt pack. The image generation element may receive power from a portable power source (e.g., chemical battery cells, primary or secondary battery cells, supercapacitor cells, fuel cells), which may be located within the belt pack, for example.

[0281] As shown, the left PCBA 2502 contains most of the active electronics, while the right PCBA 2504 supports primarily the display or projector and associated piezo drive signals. Electrical and / or fiber optic connections are employed across the front, back, or top of the optical system or body- or head-mounted components of the XR device. Both PCBA 2502 and 2504 are communicatively coupled (e.g., electrically and optically) to the belt pack. The left PCBA 2502 includes a power subsystem and a high-speed communication subsystem. The right PCBA 2504 handles the fiber display piezo drive signals. In the illustrated embodiment, only the right PCBA 2504 needs to be optically connected to the belt pack. In other embodiments, both the right and left PCBA may be connected to the belt pack.

[0282] Although illustrated as employing two PCBAs 2502 and 2504, the electronics of the body- or head-worn component may employ other architectures. For example, some implementations may use fewer or more PCBAs. As another example, various components or subsystems may be arranged differently than illustrated in FIG. 11A . For example, in some alternative embodiments, some of the components illustrated in FIG. 11A as residing on one PCBA may be located on the other PCBA without loss of generality.

[0283] For example, as described herein with reference to Figure 1, the optical systems or XR devices described herein may, in some embodiments, present virtual content to a user such that the virtual content can be perceived as three-dimensional content. In some other embodiments, the optical systems or XR devices may present virtual content to a user in a four- or five-dimensional light field (or light fields).

[0284] As illustrated in FIGS. 11B-C, the light field generation subsystems (e.g., 1100C and 1102C, respectively) are preferably operable to produce a light field. For example, an optical device 1160C or subsystem may generate or project light to simulate a four-dimensional (4D) or five-dimensional (5D) light field that would be produced by light reflecting from a real three-dimensional object or scene. For example, an optical device such as a Waveguide Reflector Array Projector (WRAP) device 1110C or a multiple depth plane three-dimensional (3D) display system may, in some embodiments, generate or project multiple virtual depth planes at distinct radial focal lengths to simulate a 4D or 5D light field. In these embodiments, the optical system or XR device functions as an eyepiece light field generator and display of a 4D or 5D light field by interpreting an input image as two-dimensional (2D) slices of a 4D function representing the light field. Note that Figures 11B-C may, in some embodiments, illustrate an optical system or XR device having a light field generation subsystem described herein, or in some other embodiments, a stereoscopic virtual content generation subsystem that projects light beams corresponding to multiple depth planes into the user's eyes.

[0285] In some embodiments, the optical system or XR device renders a stereoscopic representation of the virtual content to the user using image-based rendering, which generates different views of the virtual content from a collection of pre-acquired or pre-computed images. The virtual content may be blended into or placed within an environment in which the user viewing the virtual content is located, for example, by using one or more of an environment map, a world map, or a topology map (e.g., a map with point nodes representing individual locations and / or features, edges, etc. connecting the nodes and representing one or more relationships between the connected nodes). In these embodiments, the optical system or XR device uses one or more display or rendering algorithms for image-based rendering, which require relatively modest computational resources (e.g., compared to generating a light field for the same virtual content), particularly in real-time implementations of the virtual content. Furthermore, the cost of interacting with the generated virtual content may be independent of the complexity of the virtual content. Furthermore, the source of the images used in generating the virtual content may be real (e.g., photographs or video sequences of physical objects) or virtual (e.g., from one or more models).

[0286] These embodiments based on image-based rendering and one or more maps may be based on one or more fixed viewpoints (e.g., viewpoints from which a set of images for rendering image-based virtual content are obtained). Some of these embodiments mitigate the fixed viewpoint limitation through view interpolation using depth values ​​(e.g., depth information obtained by a depth sensor or calculated by localization techniques such as triangulation). In these embodiments, the optical system or XR device uses the depth information for view interpretation (e.g., depth data for a smaller subset of pixels in an image or all pixels in an image) to reproject points, for example, into an environment map (e.g., a geometric map with detailed geometric and / or geographic information of features, points, etc. in the map) to the user, e.g., based on the user's location, orientation, and / or gaze direction.

[0287] Some other embodiments using image-based rendering and one or more maps mitigate the fixed viewpoint limitation by determining corresponding points and / or correspondences in paired images used to render virtual content based, at least in part, on the positions of the image sensors capturing the pair of images. While both classes of embodiments involving image-based rendering effectively generate and present virtual content that can be perceived as stereoscopic by a viewing user, there may be situations where, for example, the step of determining correspondences between pairs of images is not necessarily performed deterministically.

[0288] Some other embodiments therefore generate 4D or 5D light fields using an optical system or XR device rather than employing the aforementioned image-based rendering. The light field may be generated using a 5D function (e.g., a 5D plenoptic function) and includes radiance at a point in a given direction in three-dimensional space. Thus, the light field may include a 5D function that defines a set of spatial-angle images. In these embodiments, the radiance R at a point A, having coordinates (x, y, z) in space propagating along a direction D(φ, θ), may have the form R(x, y, z, φ, θ), where φ has a range [0, π] inclusive and θ has a range [0, 2π] inclusive. In this form, φ denotes the angle from the horizontal plane defined by the x-axis and y-axis, and θ denotes the angle between a vector connecting the point in 3D space to the origin of the coordinate system and a reference unit vector (e.g., a unit vector along the x-axis).

[0289] In some embodiments, radiance is conserved within the medium (e.g., a transparent medium such as air). The above 5D function exhibits a certain amount of redundancy due to radiance conservation. In these embodiments, the aforementioned 5D function representing the light field is reduced to a 4D function R(x, y, φ, θ) when the optical system or XR device creates the 5D function within a surface (e.g., plane z=0), thus effectively reducing a 5D function with three spatial dimensions (x, y, z) and two angular dimensions (φ, θ) to a 4D function with two spatial dimensions (x, y) and two angular dimensions (φ, θ). Reducing the dimensionality of the light field function from a 5D function to a 4D function not only expedites the generation of light fields for virtual content, but also saves computational resources.

[0290] In these embodiments, the optical system or XR device described herein generates and presents a light field for the virtual content to a user by calculating the individual radiances of multiple points for the virtual content using the aforementioned 4D function (or, in more general applications of light field techniques, a 5D function). The calculated radiance (or radiance flux) for a point includes data about the light emitted, reflected, transmitted, or received by that point and may be calculated on a per-projected-area basis. The radiance for a point may also include frequency and / or wavelength information and is directional such that the radiance represents a point (e.g., a pixel or a collection of pixels) or portion of the virtual content that can be perceived by a user of the optical system or XR device. The radiance may be calculated using any technique, such as parameterizing a line (e.g., a line from the user's eye to a point in the virtual content) by point and direction using an orthogonal projection image or an image with a fixed field of view using homogeneous coordinates. For example, the radiance of a point may be determined by using a light slab technique to constrain points for the virtual content and points representing the user's eyes to lie within separate convex quadrilaterals, and using a linear projection map (e.g., a 3x3 matrix) to map between points of the virtual content (e.g., image pixels of the virtual content) and points representing the user's eyes.

[0291] For example, an optical system or XR device or electronics (e.g., the beltpack mentioned above) may generate a slab of light by rendering a 2D array of images, each image formed by representing a slice of the 4D slab of light in a fixed plane and placing the center of projection of a virtual camera at a sample location corresponding to a point in the virtual content by performing a shear perspective projection substantially similar to that used to generate the stereo pair of images. In some embodiments, the slab of light may be formed from a 2D array of orthographic views.

[0292] To generate and present a light field representation for virtual content to a user via an optical system or XR device, the lens of the optical system or XR device (e.g., 1180C in FIG. 11B) may include a stack of one or more planar or freeform waveguides, where the waveguides may each define one or more distinct focal planes corresponding to one or more distinct focal lengths. The stack of one or more planar or freeform waveguides may, in some embodiments, thus define multiple focal planes located at corresponding focal lengths. 2D slices of an image may be rendered onto focal planes at specific focal lengths, and a collection of 2D slices may thus be rendered onto multiple focal planes to represent the virtual content, which may then be perceived as stereoscopic by a user of the optical system or XR device.

[0293] In some embodiments, the waveguide comprises a first orthogonal pupil-expanding (OPE) element associated with a first surface of the planar optical waveguide to split the incombined light beam into a first set of orthogonal light beamlets, and a second orthogonal pupil-expanding (OPE) element associated with a second surface of the planar optical waveguide to split the incombined light beam into a second set of orthogonal light beamlets. In some embodiments, the first OPE element is disposed on the first surface of the planar optical waveguide, and the second OPE element is disposed on the second surface of the planar optical waveguide. The internal coupling element may be configured to optically couple the collimated light beam from the image projection assembly as an internally coupled light beam within the planar optical waveguide via total internal reflection (TIR) ​​along a first optical path that alternately intersects the first OPE element and the second OPE element for propagation, such that a portion of the internally coupled light beam is deflected via TIR as a set of distinct first and second orthogonal light beamlets that propagate within the planar optical waveguide along second parallel optical paths, in which case the second parallel optical path may be orthogonal to the first optical path.

[0294] In some embodiments, the semi-reflective interface is configured to split the in-coupled optical beam into at least two in-coupled optical beamlets. In this case, the DOEs each comprise an orthogonal pupil expansion (OPE) element configured to split the at least two in-coupled optical beamlets into a set of at least two orthogonal optical beamlets, the semi-reflective interface is further configured to split the set of at least two orthogonal optical beamlets into a set of at least four orthogonal optical beamlets, and the DOEs comprise an exit pupil expansion (EPE) element configured to split the set of at least four orthogonal optical beamlets into a set of out-coupled optical beamlets. The OPE element and the EPE element may be disposed on a face of the optical planar waveguide.

[0295] In some embodiments, the waveguide may comprise an exit pupil expansion (EPE) element associated with the planar optical waveguide to split the collection of orthogonal optical beamlets into an array of outcoupled optical beamlets (e.g., a two-dimensional outcoupled optical beamlet array) that exit the planar optical waveguide. The collimated optical beam may define an entrance pupil, and the outcoupled optical beamlet array may define an exit pupil that is larger than the entrance pupil, for example, at least 10 times larger than the entrance pupil, or even at least 100 times larger than the entrance pupil.

[0296] In some embodiments, the EPE element is disposed on one of the first and second surfaces of the planar optical waveguide. The first set of orthogonal optical beamlets and the second set of orthogonal optical beamlets may intersect with the EPE element such that a portion of the first set of orthogonal optical beamlets and the second set of orthogonal optical beamlets are deflected out of the planar optical waveguide as an outcoupled optical beamlet array. In some embodiments, the EPE element is configured to impart a convex wavefront profile onto the outcoupled optical beamlet array exiting the planar optical waveguide. In this case, the convex wavefront profile may have a center of radius at a focal point to produce an image at a given focal plane. In another embodiment, the IC element, the OPE element, and the EPE element are each diffractive.

[0297] The virtual image generation system further comprises one or more diffractive optical elements (DOEs) associated with the planar optical waveguide to further split the plurality of primary light beamlets into an array of outcoupled light beamlets (e.g., a two-dimensional outcoupled beamlet array) that exit a face of the planar optical waveguide. The collimated light beam may define an entrance pupil, and the outcoupled light beamlet array may define an exit pupil that is larger than the entrance pupil, for example, at least 10 times larger than the entrance pupil, or even at least 100 times larger than the entrance pupil. In some embodiments, the first thickness of the primary substrate and the second thickness of the secondary substrate are selected such that the center-to-center spacing of at least two adjacent ones of the outcoupled light beamlets is equal to or less than the width of the collimated light beam. In another embodiment, the first thickness and the second thickness are selected such that no gap more than half of adjacent ones of the outcoupled light beamlets reside between their edges.

[0298] In some embodiments, the semi-reflective interface is configured to split the incoupling light beam into at least two incoupling light beamlets. In this case, the DOEs each comprise an orthogonal pupil expansion (OPE) element configured to split the at least two incoupling light beamlets into a set of at least two orthogonal light beamlets, the semi-reflective interface is configured to further split the set of at least two orthogonal light beamlets into a set of at least four orthogonal light beamlets, and the DOEs comprise an exit pupil expansion (EPE) element configured to split the set of at least four orthogonal light beamlets into a set of outcoupling light beamlets. The OPE element and the EPE element may be disposed on a face of the optical planar waveguide.

[0299] The at least two in-coupled optical beamlets may propagate within the planar optical waveguide along a first optical path that intersects with the OPE element via total internal reflection (TIR), such that a portion of the at least two in-coupled optical beamlets are diffracted via TIR into a set of at least two orthogonal optical beamlets that propagate within the planar optical waveguide along a second parallel optical path. The second parallel optical path may be orthogonal to the first optical path. The set of at least two orthogonal optical beamlets may intersect with the EPE element such that a portion of the set of at least two orthogonal optical beamlets is diffracted out of the plane of the planar optical waveguide as a set of outcoupled optical beamlets. In some embodiments, the EPE element may be configured to impart a convex wavefront profile onto the outcoupled optical beamlet array exiting the planar optical waveguide. In this case, the convex wavefront profile may have a radius centered at a focal point to produce an image at a given focal plane.

[0300] According to a third aspect of the present disclosure, a virtual image generation system comprises a planar optical waveguide comprising a plurality of substrates including a primary substrate having a first thickness and at least one secondary substrate each having at least one second thickness, and at least one semi-reflective interface disposed between the substrates.

[0301] The first thickness is at least twice the thickness of each of at least one second thickness. In some embodiments, the first thickness is a non-multiple of each of the second thicknesses. In other embodiments, the secondary substrate comprises a plurality of secondary substrates. In this case, the second thicknesses may be equal to one another, or two or more of the secondary substrates may have second thicknesses that are unequal to one another. The first thickness may be a non-multiple of at least one of the second thicknesses. At least two of the unequal second thicknesses may be non-multiples of one another.

[0302] In some embodiments, each semi-reflective interface comprises a semi-reflective coating, which may be disposed between the substrates, for example, via one of physical vapor deposition (PVD), ion-assisted deposition (IAD), and ion beam sputtering (IBS), respectively. The coating may each be comprised of, for example, one or more of a metal (Au, Al, Ag, Ni—Cr, Cr, etc.), a dielectric (oxide, fluoride, and sulfide), and a semiconductor (Si, Ge). In yet another embodiment, adjacent ones of the substrates are comprised of materials having different refractive indices.

[0303] The virtual image generation system further includes an incoupling (IC) element configured to optically couple the collimated light beam from the image projection assembly as an incoupling light beam within the planar optical waveguide for propagation. The image projection assembly may include a scanning device configured for scanning the collimated light beam. The semi-reflective interface is configured to split the incoupling light beam into multiple primary light beamlets that propagate within the primary substrate.

[0304] The virtual image generation system further comprises one or more diffractive optical elements (DOEs) associated with the planar optical waveguide to further split the plurality of primary light beamlets into an array of outcoupled light beamlets (e.g., a two-dimensional outcoupled beamlet array) that exit a face of the planar optical waveguide. The collimated light beam may define an entrance pupil, and the outcoupled light beamlet array may define an exit pupil that is larger than the entrance pupil, for example, at least 10 times larger than the entrance pupil, or even at least 100 times larger than the entrance pupil. In some embodiments, the first thickness of the primary substrate and the second thickness of the secondary substrate are selected such that the center-to-center spacing of at least two adjacent ones of the outcoupled light beamlets is equal to or less than the width of the collimated light beam. In another embodiment, the first thickness and the second thickness are selected such that no gap more than half of adjacent ones of the outcoupled light beamlets reside between their edges.

[0305] In some embodiments, the semi-reflective interface is configured to split the incoupling light beam into at least two incoupling light beamlets. In this case, the DOEs each comprise an orthogonal pupil expansion (OPE) element configured to split the at least two incoupling light beamlets into a set of at least two orthogonal light beamlets, the semi-reflective interface is configured to further split the set of at least two orthogonal light beamlets into a set of at least four orthogonal light beamlets, and the DOEs comprise an exit pupil expansion (EPE) element configured to split the set of at least four orthogonal light beamlets into a set of outcoupling light beamlets. The OPE element and the EPE element may be disposed on a face of the optical planar waveguide.

[0306] The at least two in-coupled optical beamlets may propagate within the planar optical waveguide along a first optical path that intersects with the OPE element via total internal reflection (TIR), such that a portion of the at least two in-coupled optical beamlets are diffracted via TIR into a set of at least two orthogonal optical beamlets that propagate within the planar optical waveguide along a second parallel optical path. The second parallel optical path may be orthogonal to the first optical path. The set of at least two orthogonal optical beamlets may intersect with the EPE element such that a portion of the set of at least two orthogonal optical beamlets is diffracted out of the plane of the planar optical waveguide as a set of outcoupled optical beamlets. In some embodiments, the EPE element may be configured to impart a convex wavefront profile onto the outcoupled optical beamlet array exiting the planar optical waveguide. In this case, the convex wavefront profile may have a radius centered at a focal point to produce an image at a given focal plane.

[0307] According to a fourth aspect of the present disclosure, a virtual image generation system includes a pre-pupil expansion (PPE) element configured to receive a collimated light beam from an imaging element and split the collimated light beam into a set of initial outcoupled light beamlets. The virtual image generation system further includes a planar optical waveguide, an incoupling (IC) element configured to optically couple the set of initial outcoupled light beamlets into the planar optical waveguide as a set of incoupling light beamlets, and one or more diffractive elements associated with the planar optical waveguide to split the set of incoupling light beamlets into a set of final outcoupling light beamlets that exit a face of the planar optical waveguide. The diffractive elements may include an orthogonal pupil expansion (OPE) element associated with the planar optical waveguide to further split the set of incoupling light beamlets into a set of orthogonal light beamlets, and an exit pupil expansion (EPE) element associated with the planar optical waveguide to split the set of orthogonal light beamlets into a set of final outcoupling light beamlets.

[0308] In some embodiments, the collimated light beam defines an entrance pupil, the collection of initially outcoupled light beamlets defines a pre-dilated pupil that is larger than the entrance pupil, and the collection of final outcoupled light beamlets defines an exit pupil that is larger than the pre-dilated pupil. In one example, the pre-dilated pupil is at least 10 times larger than the entrance pupil, and the exit pupil is at least 10 times larger than the pre-dilated pupil. In some embodiments, the collection of initially outcoupled light beamlets is optically coupled into a planar optical waveguide as a two-dimensional array of light beamlets, and the final outcoupled light beamlet collection exits a face of the planar optical waveguide as a two-dimensional array of light beamlets. In another embodiment, the collection of initially outcoupled light beamlets is optically coupled into a planar optical waveguide as a one-dimensional array of light beamlets, and the final outcoupled light beamlet collection exits a face of the planar optical waveguide as a two-dimensional array of light beamlets.

[0309] In some embodiments, the PPE element comprises a compact planar optical waveguide, a compact OPE element associated with the compact planar optical waveguide for splitting the collimated light beam into a set of initial orthogonal light beamlets, and a compact EPE element associated with the compact planar optical waveguide for splitting the set of initial orthogonal light beamlets into a set of initial outcoupled light beamlets that exit a face of the compact planar optical waveguide. The PPE may further comprise a compact IC element configured to optically couple the collimated light beam into the planar optical waveguide.

[0310] In another embodiment, the PPE element comprises a diffractive beam splitter (e.g., a 1×N beam splitter or an M×N beam splitter) configured to split the collimated light beam into an initial set of diverging light beamlets, and a lens (e.g., a diffractive lens) configured to recollimate the initial set of diverging light beamlets into a set of initial outcoupled light beamlets.

[0311] In yet another embodiment, the PPE element comprises a prism (e.g., a solid prism or a hollow prism) configured to split the collimated light beam into a set of incoupling light beamlets. The prism may comprise a semi-reflective prism plane configured to split the collimated light beam into a set of incoupling light beamlets. The prism may comprise a plurality of parallel prism planes configured to split the collimated light beam into a set of incoupling light beamlets. In this case, the parallel prism plane may comprise a semi-reflective prism plane. The plurality of parallel prism planes may comprise fully reflective prism planes, in which case a portion of the collimated light beam may be reflected in a first direction by at least one semi-reflective prism, and a portion of the collimated light beam may be transmitted to the fully reflective prism plane for reflection in the first direction. The prism may comprise a first set of parallel prism planes configured to split the collimated light beam into a set of initial orthogonal light beamlets that are reflected in a first direction, and a second set of parallel prism planes configured to split the initial orthogonal light beamlets into a set of in-combined light beamlets that are reflected in a second direction different from the first direction, and the first and second directivities may be orthogonal to each other.

[0312] In yet another embodiment, the PPE element comprises a first planar optical waveguide assembly configured to split the collimated light beam into a two-dimensional array of outcoupled light beamlets (e.g., an N×N light beamlet array) that exit a face of the first planar optical waveguide assembly, and a second planar optical waveguide assembly configured to split the two-dimensional outcoupled light beamlet array into a plurality of two-dimensional arrays of outcoupled light beamlets that exit a face of the second planar optical waveguide assembly as a set of incoupled light beamlets. The first and second planar optical waveguide assemblies may each have unequal thicknesses.

[0313] The two-dimensional outcoupling light beamlet array has an inter-beamlet spacing, and the multiple two-dimensional outcoupling light beamlet arrays are spatially offset from one another by an inter-array spacing that is different from the inter-beamlet spacing of the two-dimensional outcoupling light beamlet array. In some embodiments, the inter-array spacing of the multiple two-dimensional outcoupling light beamlet arrays and the inter-beamlet spacing of the two-dimensional outcoupling light beamlet array are non-multiples of one another. The inter-array spacing of the multiple two-dimensional outcoupling light beamlet arrays may be greater than the inter-beamlet spacing of the two-dimensional outcoupling light beamlet array.

[0314] In some embodiments, the first planar optical waveguide assembly includes a first planar optical waveguide having opposing first and second faces; a first internal coupling (IC) element configured to optically couple a collimated light beam within the first planar optical waveguide along a first optical path via total internal reflection (TIR) ​​for propagation; and a first exit pupil expander (E) associated with the first planar optical waveguide for splitting the collimated light beam into a one-dimensional array of light beamlets that exit the second face of the first planar optical waveguide. a second IC element configured to optically couple a one-dimensional optical beamlet array into the second planar optical waveguide via TIR along a separate second optical path perpendicular to the first optical path for propagation therethrough; and a second exit pupil expander (EPE) element associated with the second planar optical waveguide for splitting the one-dimensional optical beamlet array into a two-dimensional optical beamlet array exiting the second face of the second planar optical waveguide. In this case, the first face of the second planar optical waveguide may be affixed to the second face of the first planar optical waveguide. The first and second planar optical waveguides may each have substantially equal thicknesses.

[0315] The second planar optical waveguide assembly includes a third planar optical waveguide having opposing first and second faces; a third IC element configured to optically couple the first two-dimensional optical beamlet array within the third planar optical waveguide via TIR along a separate third optical path for propagation; and a third EPE element associated with the third planar optical waveguide for splitting the two-dimensional optical beamlet array into a plurality of two-dimensional optical beamlet arrays exiting the second face of the third planar optical waveguide; The optical waveguide may include a fourth planar optical waveguide having a second surface; a fourth IC element configured to optically couple a plurality of two-dimensional optical beamlet arrays for propagation within the fourth planar optical waveguide via TIR along a separate fourth optical path perpendicular to the third optical path; and a fourth EPE element associated with the fourth planar optical waveguide to split the plurality of two-dimensional optical beamlet arrays into a plurality of two-dimensional optical beamlet arrays that exit the second surface of the fourth planar optical waveguide as an input set of optical beamlets. In this case, the first surface of the fourth planar optical waveguide may be affixed to the second surface of the third planar optical waveguide, and the first surface of the third planar optical waveguide may be affixed to the second surface of the second planar optical waveguide. The first and second planar optical waveguides may each have substantially equal thicknesses, and the third and fourth planar optical waveguides may each have substantially equal thicknesses. In this case, the substantially equal thickness of the first and second planar optical waveguides may be different from the substantially equal thickness of the third and fourth planar optical waveguides, and the equal thickness of the third and fourth planar optical waveguides may be greater than the equal thickness of the first and second planar optical waveguides.

[0316] An optical device 1160C in the form of a WRAP device 1110C or a multiple depth plane 3D display system, for example, may project images into each user's eye, either directly or indirectly. When the number and radial placement of virtual depth planes as a function of radial distance is comparable to the depth resolution of the human visual system, the discrete collection of projected depth planes mimics the psychophysical effects produced by actual, persistent three-dimensional objects or scenes. In one or more embodiments, the system 1100C may include a frame 1170C that can be customized for each AR user. Additional components of the system 1100C include electronics 1130C (e.g., some or all of the electronics illustrated in FIG. 11A ) that interconnect the various electrical and electronic subcomponents of the AR system.

[0317] System 1100C may further include a microdisplay 1120C that projects light associated with one or more virtual images into waveguide prism 1110C. As shown in FIG. 11B, light produced from microdisplay 1120C travels through waveguide 1110C, and a portion of the light reaches user's eye 1190C. In one or more embodiments, system 1100C may further include one or more compensatory lenses 1180C to modify the light associated with the virtual images. FIG. 11C illustrates the same components as FIG. 11B, but illustrates how light from microdisplay 1120C travels through waveguide 1110C and reaches user's eye 1190C.

[0318] It should be understood that optical device 1160C may include several linear waveguides, each with a separate series of resolved curved spherical reflectors or mirrors built into, located within, or formed within each of the linear waveguides. The series of resolved curved spherical reflectors or mirrors is designed to refocus infinitely focused light to a specific radial distance. A convex spherical mirror can be used to represent a virtual point source that produces an output spherical wave and appears to be located a defined distance behind the convex spherical mirror.

[0319] By concatenating a series of microreflectors of a certain shape (e.g., radius of curvature around two axes) and orientation together within a linear or rectangular waveguide, it is possible to project a 3D image corresponding to a spherical wavefront produced by a virtual point source at specific x, y, and z coordinates. Each 2D waveguide or layer provides an independent optical path to other waveguides, shaping the wavefront and focusing the incident light to project a virtual depth plane corresponding to a distinct radial distance. By using multiple 2D waveguides, each providing a focal plane at a different focal depth, a user viewing the projected virtual depth plane experiences a 3D effect.

[0320] Illustrative Embodiments and Working Examples

[0321] Head-mounted displays (HMDs) are being considered for a wide range of applications in the fields of 3D virtual and augmented environments. Accurate and fast eye tracking is crucial for enabling essential scenarios in HMDs, such as foveated display schemes for people and trade-offs between field of view (FOV) and resolution through novel interactive interfaces.

[0322] Eye trackers built into HMDs can be divided into invasive methods, such as scleral coils, and non-invasive video-based methods, the latter of which is more common. Current video-based methods mainly use different features of the eye, such as the iris, pupil, and phosphene, with the pupil-phosphene method being the most widely used. These methods have an average tracking error of 0.5° to 1°, while the tracking resolution of such features is approximately 0.7° to 1° per pixel. Further improving accuracy beyond the tracking resolution is not easy.

[0323] In addition to using ocular surface features, retinal imaging, such as eye-tracking scanning laser ophthalmoscopes (SLO), is also used for eye tracking in the medical field. They exploit scanning distortion for retinal movement estimation in small FOV high-resolution images; however, this technique is designed for small saccades, and SLO is not easily integrated into HMDs.

[0324] Retina-based eye tracking in HMDs has its unique advantages: higher tracking resolution without using an elevation sensor, a linear gaze estimation model, and direct localization of the fovea on the retina. Furthermore, retinal tracking offers a wide range of medical applications using HMDs.

[0325] Various embodiments present a real-time retinal localization method for eye tracking based on retinal movement video, where each frame may be localized on a mosaicked retrieved image. An overview of the proposed method is shown in FIG. 1. In some embodiments, the novelty of the method described herein is that it combines the performance of one or more machine learning or deep learning models 100 with a neural network 104 and an image registration method 102 using a Kalman filter 106, where the results of the deep learning are used to build a state transition model and the image registration provides measurements.

[0326] FIG. 6 illustrates several exemplary template or base images 602 of the retina that may be acquired by a scanning fiber endoscope described herein, for example, in some embodiments. For example, a scanning fiber endoscope (SFE) may capture multiple base or template images at separate instants in time. The base or template images may have a lower resolution and / or a narrower field of view (FOV) than a search image, which may be generated from multiple lower resolution and / or narrower FOV base or template images, for example. In some embodiments, once a search image is constructed, any subsequently captured image frames (e.g., by an eye tracking device in an XR system) may be registered to the same search image, rather than to multiple images, as in some conventional approaches. However, in some other embodiments, subsequently captured image frames (e.g., by an eye tracking device in an XR system) may be registered to multiple search images described herein.

[0327] In some embodiments, the method is validated on synthetic data and retinal movement video imaged using a scanning fiber endoscope (SFE). Details of the data set and its challenges are described herein. Using retinal video, the eye tracking resolution in the exemplary system is 0.05° / pixel. In some embodiments, the retinal localization method currently achieves a 0.68° mean error without considering annotation variation prior to optimization. Compared to classical pupil-phosphene methods, which have low tracking resolution, some embodiments significantly improve retinal-based eye tracking accuracy.

[0328] FIG. 5 illustrates a simplified example of a system for capturing SFE images of a retina, according to some embodiments. In these embodiments, a scanning fiber endoscope (SFE) 502 may emit a scanning light beam through one or more scanning fibers 504. The SFE 502 may scan an object 520 (e.g., a retina) by tracing a scan pattern 506 from a start point 506A to an end point 506B of the scan pattern 506. Reflected light 508 may be received through one or more return fibers within the SFE 502, and the reflected light 508 may be further processed to present a scanned image 510 (e.g., one or more base or template images 514 of the retina) on a display 512. Some techniques described herein, in some embodiments, generate a search image 516 by mosaic- ing multiple base or template images 514.

[0329] Data availability and characterization

[0330] Virtual retinal displays (retinal scanning displays) for AR / VR have long been proposed. To maintain the compactness of the HMD system, retinal imaging may share most of the optical path with the retinal scanning display. VRDs render the scanning display directly onto the retina, and therefore SFE imaging devices with a scanning pattern are used because they are low-cost and have small probe tips. SFE has a spiral scanning pattern from the center to the periphery, and a complete frame is imaged ring by ring. When the target is moving, the rings scanned at different times are from different regions, which creates motion distortion in the video frame. Some embodiments take the imaging position on the retina when the frame is completed as the ground truth, so the outer ring in each frame is closer to the ground truth.

[0331] For data collection, some embodiments mount a retinal phantom and a laser pointer on the tip of a robotic arm (Meca500) to simulate retinal movement, and use a position-sensing detector (PSD) to record the real-time position of the laser beam. After data preprocessing, the PSD data can be used for frame-by-frame annotation. The annotations in the current setup have an average error of 0.35°. Image 202 in FIG. 2 shows an example of a captured retinal frame, and image 204 in FIG. 2 illustrates a mosaicked image from a series of frames (e.g., 202). In some embodiments, due to the doughnut-shaped optical disk, it can be seen that the image has motion distortion. Note that the retinal image has many regions with similar backgrounds, and the imaging has low quality with local distortion on stationary frames, which increases the difficulty of localization.

[0332] Example process:

[0333] Given a large FOV mosaicked image as a reference, some embodiments perform real-time localization of captured SFE frames on the retrieved image, as shown in FIG. 2. Due to data challenges, some embodiments use deep learning methods to extract representative deep features for analysis. However, neural networks have uncertainties, and deep features are not always reliable. Therefore, image registration methods are used to compensate for the performance of deep learning. On the other hand, image registration results are also noisy due to data challenges. As described above, the two processes are combined with a Kalman filter, where the deep learning results are embedded in a transition model and the registration results are obtained as measurements within the Kalman filter. The Kalman filter requirements of a linear Markov model and additive Gaussian noise are satisfied in some embodiments. In this section, some embodiments introduce exemplary state transition models and measurement forms within the Kalman filter, respectively.

[0334] State transition model with deep learning

[0335] The state transition model assumes that the true state at time k is evolved from the state at k-1. In the proposed method, the transition model is formed as follows: [ka]

[0336] X k , Y k represents the position state at time k in the x and y directions. k is the process noise drawn from a zero-mean multivariate normal distribution. [ka] forms the control vector of the first-order state estimation model. It is the rate in time units calculated from the difference between the deep neural network results at time k and k-1. The proposed formulation allows embedding deep learning within a classical Kalman filter model, where one time step is the duration between successive frames.

[0337] The deep learning framework used in some embodiments is modified from the Siamese RPN. In some embodiments, Alexnet is first used to extract deep features of the frame and search image. Then, the frame features are converted into two different features using convolutional layers for classification and regression, respectively. Two corresponding response maps are created by convolution of the frame and search image features. One response map is used for target / non-target (positive / negative) region classification, and the other response map predicts the location refinement at each positive location. Unlike the Siamese RPN's learning robust representation of specific objects, some embodiments localize different templates on the same search image. The deep features of the search image are saved and used repeatedly after the training process. Because the imaging scale will not change much in an HMD, some embodiments focus on the target location in x and y instead of a bounding box with adjustable height and width.

[0338] 7 illustrates an exemplary deep learning model that may be utilized to implement at least some of the various techniques described herein in some embodiments. In these embodiments illustrated in FIG. 7, the deep learning model may be modified to receive an image frame 702 from a Siamese RPN (region proposal network) at Alexnet 705, which generates a feature map 706 for the received image frame 702 (e.g., a feature map for the frame having dimensions of 6×6×256). The feature map 706 may be provided to convolutions 710 and 712, respectively, which further generate upsampled data (e.g., a first output having dimensions of 6×6×(256×2k) for 710 and a second output having dimensions of 6×6×(256×4k) for 712, respectively). Both the first and second outputs may be individually processed by operator 710 to produce a first downsampled output (e.g., a first output having dimensions of 17x17x2K) and a second downsampled output (e.g., a second output having dimensions of 17x17x4K), respectively. Additionally or alternatively, the deep learning module extracts a feature map 708 of the search region in operator 710 (e.g., extracts a feature map having dimensions of 6x6x256).

[0339] Measurement with outer ring alignment:

[0340] In a Kalman filter, measurements are taken at the current time. [ka]

[0341] In the formula, z k is the measurement obtained by image registration, and v kis the measurement noise, similar to wk. In some embodiments, image registration is based on the SIFT method. While intensity-based registration methods (cross-correlation and mutual information) are not as robust as SIFT on the example data, these methods may be used when the captured retinal image, such as a near-infrared image, fails to provide detectable features.

[0342] As explained above, the outer ring in a frame indicates a more accurate retinal location, but globally matching the outer ring directly is difficult due to the very sparse features it contains. To reduce the interference of faint features of similar backgrounds and outer ring alignment, image alignment includes two steps: coarse alignment of the entire frame and fine alignment with only the outer ring. In coarse localization, some embodiments detect feature points from two images and search for corresponding regions on image f in frame f. ~ In outer ring alignment, the f ~ Feature points within an enlarged region around the target are selected and rematched with feature points falling within the outer ring region on the frame. Using an enlarged region improves the robustness of the algorithm when matched feature points in the rough registration are concentrated within the inner area. This method also avoids repeated calculation of feature points. Due to the challenges of retinal images, such measurements of the Kalman filter will occasionally drop out, and the tracking system then relies solely on the deep neural network until the next successful registration.

[0343] 4 illustrates some example benchmark results 402 of example retina-based tracking compared to conventional pupil-phosphene methods, according to some embodiments. More specifically, the tracking error of conventional pupil-phosphene methods typically falls within 0.5° to 1°, while retina-based tracking methods such as those described herein provide a tracking error of 0.68°. Furthermore, the tracking resolution of conventional pupil-phosphene methods typically falls within 0.7° to 1° per pixel, while retina-based tracking methods such as those described herein provide a tracking resolution of 0.05° per pixel.

[0344] Working example:

[0345] Experiments are conducted on two datasets: the synthetic retinal movement video introduced here and the SFE video. Some embodiments compare the performance of the proposed tracking method with a Kalman filter with the use of deep learning alone.

[0346] Synthetic data is generated from the public retinal dataset STARE (Structured Analysis of the Retina) from the STARE Project. Some embodiments generate a total of 36,000 frames of retinal movement video from 300 retinal images as a training set for deep learning, and 4,000 frames from 15 retinal images as a test set. Some embodiments add four different levels of image degradation to the test video frames to evaluate the robustness of the method: 1) Gaussian noise with mean 0 and variance selected from 0.001 to 0.005; 2) rotation and shear angles from -10° to 10° and -5° to 5°, respectively; and 3) scale changes from 0.8 to 1.2°. The degradation level increases uniformly within the parameter range. Table 1 shows that the method has an acceptable accuracy of 0.63° even under maximum degradation.

[0347] The SFE video experiments were implemented on a single retinal image. Some embodiments collected a total of 7,000 frames for training and 400 frames for testing. The test error 300 is summarized as a cumulative distribution function (CDF) in Figure 3. Some embodiments may recognize the presence of outliers greater than 5° when using deep learning alone. The accuracy of data annotation, as explained above, is approximately 0.35°, and the mean error of our method, excluding the effect of annotation, is 0.68°, while using neural networks alone has a mean error of 1.01°. Speed ​​can reach 72 fps using a Titan RTX GPU.

[0348] Some embodiments present the application of retina-based eye tracking for HMDs and a novel real-time localization method that combines the performance of deep learning and image registration using a Kalman filter. In some embodiments, this is the first systematic discussion of embedding retinal tracking within AR / VR headsets and providing an algorithmic solution.

[0349] With larger data sets from different users' retinas, increased accuracy is achievable. The details set forth herein are by way of example and for the sole purpose of illustrative discussion of preferred embodiments of the present invention, and are presented with the objective of providing what is believed to be the most useful and easily understood explanation of the principles and conceptual aspects of various embodiments of the present invention. In this regard, no attempt is made to show structural details of the present invention in more detail than is necessary for a basic understanding of the invention, and the description taken in conjunction with the figures and / or examples will make clear to those skilled in the art how several forms of the present invention can be embodied in practice.

[0350] Figure 8 illustrates a simplified working example of robot localization in some embodiments. More specifically, Figure 8 illustrates an example of a robot localization scenario, where box 804 illustrates the current FOV of the camera of the robot 802. With a known complete map of one or more rooms or buildings using sensor mapping, such as a panoramic optical camera scene, the robot may be localized, for example, by matching captured camera frames (e.g., 804) onto the scene map, as shown in Figure 8.

[0351] 9 illustrates another simplified working example of microscope or endoscope localization in a surgical procedure, according to some embodiments. More specifically, some embodiments may be applied to localize a microscope or endoscope 902 that views tissue in a surgical procedure. Video frames imaged with the microscope or endoscope may be localized on a larger scene map of human tissue (e.g., localized frame 904). These embodiments may be useful for path planning of surgical tools.

[0352] 10A illustrates another simplified working example of face or eye localization for face or eye tracking in some embodiments. More specifically, FIG. 10A illustrates identifying and tracking a target user's eyes 1004A and tracking a face 1002A for eye tracking and / or identification.

[0353] 10B illustrates another simplified working example of object localization for object tracking in some embodiments. More specifically, FIG. 10B illustrates the identification and tracking of a car 1002B (vehicle in FIG. 10B) for object tracking.

[0354] FIG. 10C illustrates another simplified working example of object localization for object tracking in some embodiments. More specifically, FIG. 10C illustrates a general tracking task when the object being tracked (e.g., vehicle 1002C) does not change much. In some embodiments, the object can be viewed as a template to be localized within each scene, and a Kalman filter is used to balance the performance of the neural network and image registration. In some embodiments, the tracking accuracy of the neural network can be improved down to the pixel level when image registration of the object works perfectly. When the object is distorted or occluded, this causes image registration to fail, and tracking will rely solely on the neural network in some of these embodiments.

[0355] 16A-I show example configurations of a microprojector array and the combination of the microprojector array with an optical system in one or more embodiments. Referring to FIG. 16G, multiple incident beamlets 11332 each pass through a small exit pupil 11330 to the eye 1158 in a discretized wavefront display configuration. Referring to FIG. 16H, subsets 11334 of the group of beamlets 11332 may be driven with matching colors and intensity levels to form thick subgroups (11334 may be considered "aggregated beams") that are perceived as if they were part of the same larger-sized light beam. In this case, the subsets of beamlets are parallel to each other, representing a collimated aggregate beam from optical infinity (such as light emanating from a distant mountain). The eye is accommodated to infinity, so that a subset of beamlets are deflected by the cornea and lens of the eye and all strike substantially the same location on the retina, and are perceived as comprising a single focused pixel.

[0356] FIG. 16I shows another subset of beamlets, representing a concentrated collimated beam 11336 emanating from the right side of the field of view of the user's eye 1158 when the eye 1158 is viewed in a coronal planar view from above. FIG. 16I depicts the same elements as FIG. 16H, and also includes an additional subset of beamlets representing the concentrated collimated beam 11336. Again, the eye is shown accommodated to infinity, so the beamlets strike the same spot on the retina, and the pixel is perceived as being focused. In contrast, if a different subset of beamlets is selected that arrives at the eye as a diverging fan of rays, those beamlets will not strike the same location on the retina (and will not be perceived as being focused) until the eye shifts accommodation to a nearby point that matches the geometric point of the origin of that fan of rays.

[0357] With respect to the pattern of intersection of the beamlets with the eye's anatomical pupil (e.g., the pattern of the exit pupil), the intersections may be organized into configurations such as a cross-section-efficient hexagonal or square lattice pattern or other two-dimensional arrays. Additionally, three-dimensional arrays of exit pupils may be created, as well as time-varying arrays of exit pupils.

[0358] The discretized aggregate wavefront may be created using several configurations, such as an array of microdisplays or microprojectors positioned optically conjugate with the exit pupil of a viewing optics, microdisplay, or microprojector array that is directly coupled to a viewing substrate (such as an eyeglass lens) so that they project light directly to the eye without additional intermediate viewing optics, continuous spatial light modulation array techniques, or waveguide techniques.

[0359] Referring to Figure 16A, in one embodiment, a stereoscopic (e.g., 3D) or 4- or 5-dimensional light field may be created by bundling a group of small projectors or display units (such as scanning fiber displays). Figure 16A depicts a hexagonal checkerboard projection bundle 11338, which may create, for example, a 7 mm diameter hexagonal array, with each fiber display outputting a sub-image 11340. If such an array has an optical system, such as a lens, placed in front of it so that the array is placed optically conjugate with the entrance pupil of the eye, this will create an image of the array at the eye's pupil, as shown in Figure 16B, which provides essentially the same optical arrangement as the embodiment of Figure 16G.

[0360] Each small exit pupil in this configuration is created by a dedicated small display within the bundle 11338, such as a scanning fiber display. Optically, this is as if the entire hexagonal array 11338 were positioned directly at the anatomical pupil 1145, in some embodiments. Such an embodiment may be used to drive different sub-images to different small exit pupils within the eye's larger anatomical entrance pupil 1145, comprising a superset of beamlets with multiple angles of incidence and intersections with the eye's pupil. Separate projectors or displays may each be driven with slightly different images, so that sub-images can be created that draw on different sets of light rays to be driven with different light intensities and colors.

[0361] In one embodiment, a strict image conjugate may be created as in the embodiment of FIG. 16B , in which case there is a direct one-to-one mapping between the array 11338 and the pupil 1145. In another variation, the spacing may be varied between the display and the optical system (lens 11342 in FIG. 16B ) within the array so that, instead of receiving a conjugate mapping of the array and the eye pupil, the eye pupil may capture light rays from the array at some other distance. Using such a configuration would still result in an angular diversity of beams through which a discretized aggregate wavefront representation could be created, but the mathematics regarding which light rays to drive with what power and intensity may be more complex (although, on the other hand, such a configuration may be considered simpler from the perspective of the viewing optics). The mathematics involved in light field image capture may be utilized for these calculations.

[0362] Referring to FIG. 16C , another light field creation embodiment is depicted in which an array of microdisplays or microprojectors 11346 may be coupled to a frame 11344, such as an eyeglass frame. This configuration may be positioned in front of the eye 1158. The depicted configuration is a non-conjugate arrangement, with no large-scale optical elements interposed between the display (e.g., a scanning fiber display) of the array 11346 and the eye 1158. One can imagine a pair of eyeglasses to which multiple displays, such as scanning fiber engines, are coupled, positioned orthogonal to the eyeglass surface and all angled inward, pointing toward the user's pupil. Each display may be configured to create a set of light rays representing a different member of the beamlet superset.

[0363] With such a configuration, at the anatomical pupil 1145, the user may receive results similar to those received in the embodiment discussed with reference to Figure 16G, in which every point in the user's pupil receives light rays with multiple angles of incidence and intersections contributed by different displays. Figure 16D illustrates a non-conjugate configuration similar to that of Figure 16C, but the embodiment of Figure 16D features a reflective surface 11348, facilitating moving the display array 11346 away from the field of view of the eye 1158 while also allowing a view of the real world 11144 through the reflective surface 11348.

[0364] Another configuration for creating angular diversity for discretized aggregate wavefront displays is also presented. To optimize such a configuration, the size of the display can be reduced to a maximum. While scanning fiber displays that can be utilized as displays can have baseline diameters in the 1 mm range, reductions in enclosure and projection lens hardware can reduce the diameter of such displays to approximately 0.5 mm or less, which is barely noticeable to the user. Another size-reducing geometric refinement can be achieved by directly coupling a collimating lens (which may comprise, for example, a gradient refractive index, or "GRIN," lens, a conventional curved lens, or a diffractive lens) to the tip of the scanning fiber itself, in the case of a fiber scanning display array. For example, referring to FIG. 16E, a GRIN (gradient refractive index) lens 11354 is shown fused to the end of a single-mode optical fiber. An actuator 11350, such as a piezoelectric actuator, can be coupled to the fiber 11352 and used for fiber tip scanning.

[0365] In another embodiment, the ends of the fibers may be shaped into a hemispherical shape using an optical fiber bend polishing process to create the lensing effect. In another embodiment, a standard refractive lens may be bonded to the end of each optical fiber using an adhesive. In another embodiment, the lenses may be constructed from a slightly transparent polymer material such as epoxy or glass. In another embodiment, the ends of the optical fibers may be fused to create a curved surface for the lensing effect.

[0366] Figure 16F shows an embodiment in which display configurations (e.g., a scanning fiber display with a GRIN lens, shown in the enlarged view of Figure 16E) can be coupled together through a single transparent substrate 11356, preferably with a refractive index that closely matches the cladding of the optical fiber 11352, so that the fiber itself is substantially invisible to outside viewing across the depicted assembly. It should be understood that if the index matching of the cladding is done precisely, the larger cladding / housing will be transparent, and only the small core, preferably about 3 microns in diameter, will obstruct the view. In one embodiment, the display rows and columns 11358 may all be angled inward so that they are directed toward the user's anatomical pupil (in another embodiment, they can remain parallel to each other, but such a configuration is less efficient).

[0367] As used herein, unless otherwise indicated, the terms "a" and "an" shall be taken to mean "one," "at least one," or "one or more." Unless otherwise required by context, singular terms used herein shall include pluralities and plural terms shall include the singular.

[0368] Unless otherwise required by context, throughout the description and claims, the words "comprises," "comprising," and the like, should be construed in an inclusive sense, i.e., in the sense of "including, but not limited to," as opposed to an exclusive or inclusive sense. Words using the singular or plural also include the plural and singular, respectively. In addition, the words "herein," "above," and "below," and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions thereof.

[0369] The description of the embodiments of the present disclosure is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. While specific embodiments and examples for the present disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the present disclosure, as those skilled in the art will recognize.

[0370] All references cited herein are incorporated by reference. Aspects of the present disclosure can be modified, where necessary, to adopt the systems, functions, and concepts of the above references and applications to provide still further embodiments of the present disclosure. These and other changes can be made to the present disclosure in light of the detailed description.

[0371] Specific elements of any of the foregoing embodiments can be combined with or substituted for elements in other embodiments. Moreover, the inclusion of specific elements in at least some of these embodiments may be optional, and further embodiments may include one or more embodiments that specifically exclude one or more of these specific elements. Furthermore, although advantages associated with certain embodiments of the present disclosure have been described in the context of these embodiments, other embodiments may also exhibit such advantages, and not all embodiments necessarily need to exhibit such advantages, to be within the scope of the present disclosure.

[0372] It will be understood that features and aspects of the various embodiments described above can be combined to provide further embodiments. These and other changes can be made to the embodiments in light of the above detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments, along with the full range of equivalents to which such claims are entitled.

[0373] Various exemplary embodiments of the present disclosure are described herein. These examples are referred to in a non-limiting sense. They are provided to illustrate broader applicable aspects of the present disclosure. Various changes may be made to the disclosure as described, and equivalents may be substituted without departing from the true spirit and scope of the disclosure. In addition, many modifications may be made to adapt a particular situation, material, composition, process, process act, or step to the objective, spirit, or scope of the disclosure. Moreover, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein has discrete components and features that can be readily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the disclosure. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[0374] The present disclosure includes methods that may be implemented using the subject devices. The methods may include the act of providing such a suitable device. Such provisioning may be performed by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, approach, locate, configure, activate, power on, or otherwise act to provide the device required in the subject methods. The methods recited herein may occur in any order of the recited events and the recited sequence of events that is logically possible.

[0375] Exemplary aspects of the present disclosure are described above, along with details regarding material selection and manufacturing. As for other details of the present disclosure, these may be understood in connection with the aforementioned referenced patents and publications and are generally known or may be understood by those skilled in the art. The same may be true with respect to the method-based aspects of the present disclosure in terms of additional operations as commonly or logically adopted.

[0376] Additionally, while the present disclosure has been described with reference to several embodiments incorporating various features, the present disclosure is not limited to that described or illustrated, as each variation of the disclosure is discussed. Various modifications may be made to the present disclosure as described, and equivalents (whether recited herein or not included for purposes of brevity to some extent) may be substituted without departing from the true spirit and scope of the present disclosure. Additionally, when a range of values ​​is provided, it is understood that all intervening values ​​between the upper and lower limits of that range, and any other stated value or intervening value within the stated range, are encompassed within the present disclosure.

[0377] It is also contemplated that any optional features of the described inventive variations may be set forth and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that plural identical items are present. More specifically, as used in this specification and the claims associated therewith, the singular forms "a," "an," "said," and "the" include plural references unless specifically stated otherwise. In other words, the use of articles allows for "at least one" of the items of the present subject matter in the above description and in the claims associated with this disclosure. Furthermore, it should be noted that such claims may be drafted to exclude any optional element. Accordingly, this language is intended to serve as a predicate for the use of exclusive terminology such as "solely," "only," and the like in connection with the recitation of claim elements, or the use of a "negative" limitation.

[0378] Without the use of such exclusive terminology, the term "comprising" in the claims associated with this disclosure shall be deemed to permit the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claim, or the addition of features may be deemed to change the nature of the elements recited in such claim. Except as specifically defined herein, all technical and scientific terms used herein should be given the broadest possible commonly understood meaning while maintaining claim legitimacy.

[0379] The scope of the present disclosure should not be limited to the examples provided and / or this specification, but rather should be limited only by the scope of the claim language associated with this disclosure.

[0380] The above description of the illustrated embodiments is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Specific embodiments and examples are described herein for illustrative purposes, but various equivalent modifications may be made, as will be recognized by those skilled in the art, without departing from the spirit and scope of the present disclosure. The teachings provided herein of various embodiments may be applied to other devices that implement VR, AR, MR, XR, or hybrid systems and / or employ user interfaces, not necessarily the exemplary optical system 12 generally described above.

[0381] For example, the foregoing detailed description describes various embodiments of devices and / or processes through the use of block diagrams, schematic diagrams, and examples. To the extent that the block diagrams, schematic diagrams, and examples contain one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation within such block diagrams, flowcharts, or examples may be individually and / or collectively implemented by a wide range of hardware, software, firmware, or virtually any combination thereof.

[0382] In one embodiment, the present subject matter may be implemented via an application-specific integrated circuit (ASIC). However, those skilled in the art will recognize that the embodiments disclosed herein may equivalently be implemented, in whole or in part, within a standard integrated circuit, as one or more computer programs executed by one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs executed on one or more controllers (e.g., microcontrollers), as one or more programs executed by one or more processors (e.g., microprocessors), as firmware, or virtually any combination thereof; and that designing the circuitry and / or writing the code for the software and / or firmware would be well within the skill of one of ordinary skill in the art in light of the teachings of the present disclosure.

[0383] When logic is implemented as software and stored in memory, the logic or information may be stored on any computer-readable medium for use by or in connection with any processor-related system or method. In the context of this disclosure, memory is a computer-readable medium that is an electronic, magnetic, optical, or other physical device or means for containing or storing a computer and / or processor program. The logic and / or information may be embodied in any computer-readable medium for use by or in connection with an instruction-execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that may fetch instructions from the instruction-execution system, apparatus, or device and execute the instructions associated with the logic and / or information.

[0384] In the context of this specification, a "computer-readable medium" may be any element that can store a program associated with logic and / or information for use by or in connection with an instruction execution system, apparatus, and / or device. A computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media would include the following: portable computer diskettes (magnetic, CompactFlash cards, Secure Digital, or equivalent), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, EEPROM, or flash memory), portable compact disc read-only memory (CD ROM), digital tape, and other non-transitory media.

[0385] Many of the methods described herein may be practiced with variations, for example, many of the methods may include additional acts, omit some acts, and / or perform acts in a different order than illustrated or described.

[0386] The various embodiments described above may be combined to provide further embodiments. To the extent they do not contradict the specific teachings and definitions of this specification, all U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications referenced herein and / or listed in the Application Data Sheets are incorporated herein by reference in their entirety. Aspects of the embodiments may be modified, if necessary, to employ systems, circuits, and concepts from the various patents, applications, and publications to provide still further embodiments.

[0387] These and other changes can be made to the embodiments in light of the above detailed description. In general, in the following claims, the terms used should not be construed to limit the scope of the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments, along with the full range of equivalents to which such claims are entitled. Therefore, the claims are not limited by this disclosure.

[0388] Furthermore, the various embodiments described above may be combined to provide further embodiments, and aspects of the embodiments may be modified, if necessary, to employ systems, circuits, and concepts from various patents, applications, and publications to provide still further embodiments.

[0389] These and other changes can be made to the embodiments in light of the above detailed description. In general, in the following claims, the terms used should not be construed to limit the scope of the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments, along with the full range of equivalents to which such claims are entitled. Therefore, the claims are not limited by this disclosure.

Claims

1. 1. A method for tracking eye movements in an eyepiece display, the method being executed by a processor, the method comprising: the processor identifying a plurality of base images of an object or a portion of the object; generating a search image based at least in part on at least some of the plurality of base images; The processor performs at least a deep learning process on a base image using a neural network in a deep learning model to generate a deep learning result; locating the captured image relative to the search image by at least performing an image registration process on the captured image and the search image using a Kalman filter model and the deep learning results; Including, Locating the captured image by the processor includes: the processor performing a coarse alignment across the captured image frames; performing a fine registration on one or more features or feature points within an outer region around the region referenced in the coarse registration; Including, The method, wherein the object comprises a user's retina, and the captured image representing at least a portion of the retina is located relative to the search image without the use of a pupil-flashing technique that captures flashes of light from a pupil in response to one or more input light patterns.

2. The processor generating the deep learning result comprises: the processor extracting features within a region within the captured image using a deep convolutional neural network (DCNN); the processor converting the feature into a plurality of features including a first feature and a second feature; classifying the region as a positive region or a negative region based at least in part on the first feature by the processor; the processor determining a regression or correction for the region based at least in part on the second characteristic; The method of claim 1 , comprising:

3. The processor generating the deep learning result comprises: the processor extracting features within a region within the captured image using a deep convolutional neural network (DCNN); the processor converting the feature into a plurality of features including a first feature and a second feature; The method of claim 1 , comprising:

4. The processor generating the deep learning result comprises: generating a first response map by at least convolving at least the first features using one or more convolutional layers in the DCNN; classifying the region into a positive region or a negative region based at least in part on the first response map; The method of claim 3 further comprising:

5. The processor generating the deep learning result comprises: generating a second response map by convolving at least the second features using the one or more convolutional layers or one or more different convolutional layers in the DCNN; the processor using at least the second response map to determine a regression or correction to the predicted location of the region; The method of claim 4 further comprising:

6. 1. A method for tracking eye movements in an eyepiece display, the method being executed by a processor, the method comprising: the processor identifying a plurality of base images of an object or a portion of the object; generating a search image based at least in part on at least some of the plurality of base images; The processor performs at least a deep learning process on a base image using a neural network in a deep learning model to generate a deep learning result; locating the captured image relative to the search image by at least performing an image registration process on the captured image and the search image using a Kalman filter model and the deep learning results; Including, Locating the captured image by the processor includes: embedding the deep learning process into a state transition model; The processor receives the deep learning results in the Kalman filter model as one or more measurements using at least the state transition model, the state transition model being used to determine a next position state at a next time point based at least in part on at least one of a previous position state at a previous time point and a process noise or a control vector of a state estimation model; the processor performing a coarse alignment across the captured image frames; performing a fine registration on one or more features or feature points within an outer region around the region referenced in the coarse registration; A method comprising:

7. The step of locating the captured image by the processor comprises: determining measurements at subsequent times by using at least the Kalman filter model in the image registration process; The processor performs at least the image registration process to reduce interference of similar backgrounds or one or more similar features; the processor locating the captured image within the search image using the Kalman filter model and the image registration process; The method of claim 6, comprising:

8. The processor embedding the deep learning process comprises: the processor determining one or more control vectors for the state transition model; the processor deriving process noise from a statistical distribution; the processor determining a time step for the state transition model; The method of claim 6, comprising:

9. The processor performing the coarse alignment for an entire frame of the captured image, the processor detecting the one or more features or feature points in one or more regions within an entire frame of the captured image or the retrieved image; the processor registering the captured image with respect to the one or more regions within the entire frame; The method of claim 8 , comprising:

10. The processor performing the fine alignment to the one or more features or feature points in the outer region around a region referenced in the coarse alignment comprises: the processor selecting a feature or feature point within the enlarged region that corresponds to the region referenced in the rough alignment; said processor matching said features or feature points with corresponding features or feature points in said outer region of said search image; 10. The method of claim 9, comprising:

11. The processor performing the fine alignment to the one or more features or feature points in the outer region around a region referenced in the coarse alignment, comprising: locating the captured image within the search image based at least in part on a result of the processor matching the features or feature points with corresponding features or feature points within the outer region of the search image. The method of claim 10, comprising:

12. A system for tracking eye movements within an eyepiece display, the system comprising a processor, a scanning fiber assembly, and a memory, the memory storing instructions which, when executed by the processor, cause the processor to perform any of the methods described in claims 1, 2 to 11 to track eye movements within the eyepiece display.

13. A non-transitory computer-readable medium having instructions stored thereon which, when executed by a microprocessor, cause the microprocessor to perform any of the methods described in claims 1, 2 to 11 to track eye movement within an eyepiece display.

Images