Visual defect determination and enhancement

Abstract

To provide visual defect determination and enhancement.SOLUTION: In some embodiments, visual defect information may be generated through a dynamic fixation point based on an eye feature. In the some embodiments, a first stimulation may be displayed at a first position on a user interface on the basis of the fixation point for visual inspection presentation. A fixation point for visual test presentation may be adjusted during the visual inspection presentation on the basis of eye feature information related to a user. As one example, the eye feature information may show the user's eye feature that is generated during the visual inspection presentation. A second stimulation may be shown during the visual inspection presentation at a second interface position on the user interface on the basis of the fixation point adjusted for the visual inspection presentation. Visual defect information associated with the user may be generated on the basis of feedback information showing a feedback related to the first stimulation and a feedback related to the second stimulation.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. patent application Ser. No. 16 / 687,512, entitled "Vision Defect Determination Via a Dynamic Eye-Characteristic-Based Fixation Point," filed November 18, 2019. U.S. patent application Ser. No. 16 / 687,512 is a continuation-in-part of U.S. patent application Ser. No. 16 / 654,590, entitled "Vision Defect Determination," filed October 16, 2019. U.S. Patent Application No. 16 / 654,590 claims the benefit of U.S. Provisional Application No. 62 / 895,402, entitled "Double and Binocular Vision Determination and Correction," filed September 3, 2019, and is a continuation-in-part of U.S. Patent Application No. 16 / 444,604, entitled "Vision Defect Determination via a Dynamic Eye-Characteristic-Based Fixation Point," filed June 18, 2019. U.S. Patent Application No. 16 / 444,604 is a continuation-in-part of U.S. Patent Application No. 16 / 428,932, entitled "Vision Defect Determination and Enhancement," filed May 31, 2019. U.S. Patent Application No. 16 / 428,932 is a continuation of U.S. Patent Application No. 16 / 367,633, entitled "Vision Defect Determination and Enhancement Using a Prediction Model," filed March 28, 2019. U.S. Patent Application No. 16 / 367,633 is a continuation-in-part of U.S. Patent Application No. 16 / 144,995, entitled "Digital Therapeutic Corrective Spectacles," filed September 27, 2018. U.S. Patent Application No. 16 / 144,995 claims the benefit of U.S. Provisional Application No. 62 / 563,770, entitled "Digital Therapeutic Corrective Spectacles," filed September 27, 2017.Each of these applications is incorporated herein by reference in its entirety, and this application claims priority to each of the other aforementioned applications filed within at least twelve months of the filing of this application.

[0002] This application is also related to i) U.S. patent application Ser. No. 16 / 662,113, filed October 24, 2019, entitled "Vision Defect Determination and Enhancement Using a Prediction Model," ii) U.S. patent application Ser. No. 16 / 538,057, filed August 12, 2019, entitled "Vision-Based Alerting Based on Physical Contact Prediction," iii) U.S. patent application Ser. No. 16 / 560,212, filed September 4, 2019, entitled "Field of View Enhancement Via Dynamic Display Portions," and iv) U.S. patent application Ser. No. 16 / 428,899, filed May 31, 2019, entitled "Visual Enhancement for Dynamic Vision Defects." iii) U.S. Patent Application No. 16 / 560,212 is a continuation of U.S. Patent Application No. 16 / 428,380, entitled "Field of View Enhancement Via Dynamic Display Portions for a Modified Video Stream," filed May 31, 2019. U.S. Patent Application No. 16 / 428,380 is a continuation of U.S. Patent Application No. 16 / 367,751, entitled "Field of View Enhancement via Dynamic Display Portions," filed March 28, 2019. iv) U.S. Patent Application No. 16 / 428,899 is a continuation of U.S. Patent Application No. 16 / 367,687, entitled "Visual Enhancement for Dynamic Vision Defects," filed March 28, 2019. Each of these applications is incorporated herein by reference in its entirety. This application also claims priority to each of the aforementioned applications filed within at least twelve months of the filing of this application.

[0003] The present invention relates to facilitating the determination and correction of visual defects associated with a user's vision. [Background technology]

[0004] While wearable technologies such as "smart glasses" exist to assist the visually impaired, typical wearable technologies do not adequately address many of the problems faced by traditional eyeglasses or contact lenses. For example, typical wearable technologies cannot address the challenges faced by individuals with higher-order visual aberrations (such as refractive errors that cannot be corrected by traditional eyeglasses or contact lenses) or dynamic aberrations that change with the state of accommodation or direction of gaze. These and other problems exist. Summary of the Invention

[0005] Aspects of the present invention relate to methods, apparatus, and / or systems for facilitating the determination and correction of visual defects associated with a user's vision. By way of example, such correction may include providing an enhancement of the user's visual field or vision (e.g., correcting the user's visual field or vision, enhancing the user's visual field or vision, etc.), providing a correction of the user's aberrations, or providing such an enhancement or correction via a wearable device.

[0006] In some embodiments, a first stimulus may be displayed at a first location on the user interface based on a fixation point for the visual test presentation. The fixation point for the visual test presentation may be adjusted during the visual test presentation based on eye characteristic information associated with the user. As an example, the eye characteristic information may indicate an eye characteristic of the user that occurred during the visual test presentation. A second stimulus may be displayed during the visual test presentation at a second interface location on the user interface based on the adjusted fixation point for the visual test presentation. Vision deficiency information associated with the user may be generated based on feedback information indicating feedback associated with the first stimulus and feedback associated with the second stimulus.

[0007] Various other aspects, features, and advantages of the present invention will become apparent by reference to the detailed description of the invention and the accompanying drawings. It should also be understood that both the foregoing summary and the following detailed description are exemplary only and are not intended to limit the scope of the present invention. As used in this specification and the claims, the singular forms "a," "an," and "the" indicate plural references unless the context clearly dictates otherwise. Furthermore, as used in this specification and the claims, the term "or" means "and / or" unless the context clearly dictates otherwise. [Brief explanation of the drawings]

[0008] [Figure 1A] 1 illustrates a system for facilitating corrections related to a user's vision, according to one or more embodiments.

[0009] [Figure 1B] 1 illustrates a system that implements a machine learning model to facilitate corrections related to a user's vision, according to one or more embodiments.

[0010] [Figure 1C] 1 illustrates an exemplary eyewear device according to one or more embodiments. [Figure 1D] 1 illustrates an exemplary eyewear device according to one or more embodiments. [Figure 1E] 1 illustrates an exemplary eyewear device according to one or more embodiments. [Figure 1F] 1 illustrates an exemplary eyewear device according to one or more embodiments.

[0011] [Figure 2] 1 illustrates an exemplary vision system according to one or more embodiments.

[0012] [Figure 3]FIG. 1 illustrates a device having a vision correction framework implemented in an image processing device and a wearable eyewear device, according to one or more embodiments.

[0013] [Figure 4] FIG. 1 illustrates an example process including an inspection mode and a vision mode, according to one or more embodiments.

[0014] [Figure 5] FIG. 1 illustrates an example process including an inspection mode and a vision mode, according to one or more embodiments.

[0015] [Figure 6A] FIG. 10 illustrates an example of an evaluation protocol for a test mode process including pupil tracking, according to one or more embodiments. [Figure 6B] FIG. 10 illustrates an example of an evaluation protocol for a test mode process including pupil tracking, according to one or more embodiments. [Figure 6C] FIG. 10 illustrates an example of an evaluation protocol for a test mode process including pupil tracking, according to one or more embodiments.

[0016] [Figure 7A] FIG. 10 illustrates an example of an evaluation protocol for a test mode process including pupil tracking, according to one or more embodiments. [Figure 7B] FIG. 10 illustrates an example of an evaluation protocol for a test mode process including pupil tracking, according to one or more embodiments. [Figure 7C] FIG. 10 illustrates an example of an evaluation protocol for a test mode process including pupil tracking, according to one or more embodiments.

[0017] [Figure 8] FIG. 1 illustrates a workflow including a testing module that generates and presents multiple visual stimuli to a user via a wearable eyewear device, according to one or more embodiments.

[0018] [Figure 9] 1 illustrates a test mode process according to one or more embodiments.

[0019] [Figure 10] FIG. 10 illustrates a process for an artificial intelligence correction algorithm mode that may be implemented as part of an inspection mode, according to one or more embodiments.

[0020] [Figure 11] FIG. 1 illustrates an inspection image according to one or more embodiments.

[0021] [Figure 12] FIG. 10 illustrates generating a simulated vision image, including overlaying a defective visual field onto a test image presented to a subject, according to one or more embodiments.

[0022] [Figure 13] 1A-1C illustrate examples of various correction transformations that may be applied to an image and presented to a subject, according to one or more embodiments.

[0023] [Figure 14] FIG. 1 illustrates an example of a translation method, according to one or more embodiments.

[0024] [Figure 15] FIG. 1 illustrates an example of a machine learning framework, according to one or more embodiments.

[0025] [Figure 16] FIG. 1 illustrates a process for an AI system in a machine learning framework, according to one or more embodiments.

[0026] [Figure 17] FIG. 1 illustrates an example of transforming an inspection image in accordance with one or more embodiments.

[0027] [Figure 18] 1A-1C illustrate examples of translation of an inspection image, according to one or more embodiments.

[0028] [Figure 19] 1 is a graphical user interface illustrating various implementations of an AI system according to one or more embodiments.

[0029] [Figure 20] 1 illustrates a framework for an AI system including a feedforward neural network, according to one or more embodiments.

[0030] [Figure 21] FIG. 1 illustrates an example of a testing mode process for an AI system including a neural network, according to one or more embodiments. [Figure 22] FIG. 1 illustrates an example of a testing mode process for an AI system including an AI algorithm optimization process, according to one or more embodiments.

[0031] [Figure 23] FIG. 10 illustrates an example process for performing inspection and vision modes, according to one or more embodiments.

[0032] [Figure 24A] 1 illustrates a wearable eyewear device including custom reality wearable glasses that allow images from the environment to pass through a transparent portion of a display of the wearable glasses, the transparent portion corresponding to a peripheral region of a user's field of vision, and other portions of the display of the wearable glasses being opaque, according to one or more embodiments.

[0033] [Figure 24B]1 illustrates a wearable eyewear device including custom reality wearable glasses that allow images from the environment to pass through a transparent portion of a display of the wearable glasses, the transparent portion corresponding to a central region of a user's field of view, and other portions of the display of the wearable glasses being opaque, according to one or more embodiments.

[0034] [Figure 24C] 1 illustrates the use of eye tracking to align between a viewing plane, a remapped image plane, and an optional transparent screen plane, in accordance with one or more embodiments.

[0035] [Figure 25A] 1 illustrates a use case in which a visual test presentation is displayed to a patient without strabismus, according to one or more embodiments.

[0036] [Figure 25B] 1 illustrates a use case in which a visual test presentation is displayed to a patient with strabismus, according to one or more embodiments.

[0037] [Figure 25C] 1 illustrates automated measurement and correction of diplopia, according to one or more embodiments. [Figure 25D] 1 illustrates automated measurement and correction of diplopia, according to one or more embodiments. [Figure 25E] 1 illustrates automated measurement and correction of diplopia, according to one or more embodiments. [Figure 25F] 1 illustrates automated measurement and correction of diplopia, according to one or more embodiments. [Figure 25G] 1 illustrates automated measurement and correction of diplopia, according to one or more embodiments. [Figure 25H] 1 illustrates automated measurement and correction of diplopia, according to one or more embodiments. [Figure 25I] 1 illustrates automated measurement and correction of diplopia, according to one or more embodiments.

[0038] [Figure 25J] FIG. 1 illustrates a binocular vision test and its results, according to one or more embodiments. [Figure 25K] FIG. 1 illustrates a binocular vision test and its results, according to one or more embodiments. [Figure 25L] FIG. 1 illustrates a binocular vision test and its results, according to one or more embodiments.

[0039] [Figure 25M] 1 illustrates a stereoscopic test according to one or more embodiments. [Figure 25N] 1 illustrates a stereoscopic test according to one or more embodiments.

[0040] [Figure 26] FIG. 1 illustrates a subject's normal binocular vision, showing how a monocular image from the left eye and a monocular image from the right eye are combined to produce a single perceived image that includes the central macular region and the peripheral visual field surrounding the central region.

[0041] [Figure 27] FIG. 1 illustrates tunnel vision, a condition in which peripheral areas are not visible to the subject.

[0042] [Figure 28] FIG. 1 illustrates an image shifting technique for correcting tunnel vision or improving vision, according to one or more embodiments.

[0043] [Figure 29] 1A-1C are diagrams illustrating image resizing and transformation techniques for improving vision or preserving central vision while expanding the field of view, according to one or more embodiments.

[0044] [Figure 30] FIG. 1 illustrates a binocular field expansion technique, according to one or more embodiments.

[0045] [Figure 31A] FIG. 1 illustrates a technique for assessing dry eye and corneal abnormalities that involves projecting a pattern onto the corneal surface and imaging the corneal surface reflecting the pattern, according to one or more embodiments.

[0046] [Figure 31B] 1A-1C are diagrams illustrating the presentation of a reference image including a grid pattern displayed to a subject or projected onto the subject's cornea or retina by wearable glasses, according to one or more embodiments.

[0047] [Figure 31C] 10 illustrates an example grid pattern for manipulation by a subject, according to one or more embodiments.

[0048] [Figure 31D] 31D illustrates an example of the manipulation of the grid pattern shown in FIG. 31C according to one or more embodiments.

[0049] [Figure 31E] FIG. 1 illustrates a scene to be perceived by a subject, according to one or more embodiments.

[0050] [Figure 31F] FIG. 31C illustrates an example of a corrected visual field that, when provided to a subject with visual distortions determined by grid pattern technology, would result in the subject perceiving the visual field illustrated in FIG. 31E, in accordance with one or more embodiments.

[0051] [Figure 31G] FIG. 1 illustrates a display including a steerable grid pattern for a subject to communicate distortions in their field of view, according to one or more embodiments.

[0052] [Figure 32] FIG. 1 illustrates an image of the corneal surface reflecting a pattern projected onto the corneal surface, according to one or more embodiments.

[0053] [Figure 33] FIG. 1 illustrates an example of a normal pattern reflection, according to one or more embodiments.

[0054] [Figure 34] 1A-1C illustrate examples of anomalous pattern reflections, according to one or more embodiments.

[0055] [Figure 35A] FIG. 1 illustrates the presentation of a vision test with a dynamic fixation point, according to one or more embodiments. [Figure 35B] FIG. 1 illustrates the presentation of a vision test with a dynamic fixation point, according to one or more embodiments. [Figure 35C] FIG. 1 illustrates the presentation of a vision test with a dynamic fixation point, according to one or more embodiments. [Figure 35D] FIG. 1 illustrates the presentation of a vision test with a dynamic fixation point, according to one or more embodiments. [Figure 35E] FIG. 1 illustrates the presentation of a vision test with a dynamic fixation point, according to one or more embodiments.

[0056] [Figure 35F] FIG. 1 illustrates a flowchart relating to a process for facilitating the presentation of a vision test using a dynamic fixation point, according to one or more embodiments.

[0057] [Figure 35G] 1 illustrates the presentation of a visual test including multiple contrast staircase stimuli and stimulus sequences at predetermined locations, according to one or more embodiments.

[0058] [Figure 36] FIG. 1 is a timing diagram illustrating the processing of a test sequence at one stimulus location, according to one or more embodiments.

[0059] [Figure 37]FIG. 1 illustrates the calculation of pixel width and height bounding the maximum bright field, according to one or more embodiments.

[0060] [Figure 38] 1A-1C illustrate inspection images used to inspect the four main quadrants of the field of view, according to one or more embodiments.

[0061] [Figure 39A] FIG. 1 illustrates an example of a field of view before remapping, according to one or more embodiments.

[0062] [Figure 39B] 10A-10C illustrate examples of fields of view after remapping, according to one or more embodiments.

[0063] [Figure 40A] 1 illustrates an example of a custom reality glasses device according to one or more embodiments. [Figure 40B] 1 illustrates an example of a custom reality glasses device according to one or more embodiments. [Figure 40C] 1 illustrates an example of a custom reality glasses device according to one or more embodiments.

[0064] [Figure 41] 1 illustrates a flowchart of a method for facilitating correction of a user's vision via a predictive model, according to one or more embodiments.

[0065] [Figure 42] 1 illustrates a flowchart of a method for facilitating an expansion of a user's field of view through a combination of portions of multiple images of a scene, according to one or more embodiments.

[0066] [Figure 43]1 illustrates a flowchart of a method for facilitating enhanced user vision via one or more dynamic display portions on one or more transparent displays, in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0067] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present invention. However, those skilled in the art will appreciate that embodiments of the present invention may be practiced without these specific details or with equivalent configurations. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring embodiments of the present invention.

[0068] FIG. 1A illustrates a system 100 for facilitating user vision modification in accordance with one or more embodiments. As illustrated in FIG. 1, the system 100 may include a server 102, a client device 104 (or client devices 104a-104n), or other components. The server 102 may include a configuration subsystem 112, a model manager subsystem 114, or other components. The client device 104 may include an inspection subsystem 122, a vision subsystem 124, or other components. Each of the client devices 104 may include any type of mobile, fixed, or other device. By way of example, the client devices 104 may include desktop computers, notebook computers, tablet computers, smartphones, wearable devices, or other client devices. Users may, for example, use one or more client devices 104 to interact with each other, one or more servers, or other components of the system 100.

[0069] It should be noted that while one or more processes are described herein as being performed by particular components of the client device 104, in some embodiments these processes may be performed by other components of the client device 104 or other components of the system 100. As an example, one or more processes described herein as being performed by components of the client device 104 may, in some embodiments, be performed by components of the server 102. It should also be noted that while one or more processes are described herein as being performed by particular components of the server 102, in some embodiments these processes may be performed by other components of the server 102 or other components of the system 100. As an example, one or more processes described herein as being performed by components of the server 102 may, in some embodiments, be performed by components of the client device 104. Additionally, while some embodiments are described herein with reference to machine learning models, it should be noted that other embodiments may use other predictive models (e.g., statistical models or other analytical models) instead of or in addition to machine learning models (e.g., one or more embodiments may use statistical models to replace machine learning models, and non-statistical models to replace non-machine learning models).

[0070] In some embodiments, system 100 may provide a user with a presentation of a visual test that includes a set of stimuli (e.g., light stimuli, text, or images displayed to the user). During (or after) the presentation, system 100 may obtain feedback related to the set of stimuli (e.g., feedback indicating whether or how the user is viewing one or more stimuli of the set). As one example, the feedback may include an indication of the user's response to one or more stimuli (of the set of stimuli) or an indication of the user's lack of response to such stimuli. The response (or lack thereof) may relate to eye movement, gaze direction, changes in pupil size, or user modifications to one or more stimuli or other user input (e.g., the user's response or other reaction to the stimuli). As another example, the feedback may include an image of the eye captured during the presentation of the visual test. The eye image may be an image of the retina of the eye (e.g., all or a portion of the retina), an image of the cornea of ​​the eye (e.g., all or a portion of the cornea), or other eye image.

[0071] In some embodiments, system 100 may determine one or more defective visual field portions of the user's visual field (e.g., automatic determination based on feedback related to a set of stimuli displayed to the user or other feedback). As an example, the defective visual field portion may be one of the user's visual field portions that fails to meet one or more vision criteria (e.g., whether or the degree to which the user perceives one or more stimuli, the degree of light sensitivity, distortion, or other aberrations, or other criteria). In some embodiments, system 100 may provide an enhanced image or adjust one or more configurations of the wearable device based on the determination of the defective visual field portion. As an example, an enhanced image may be generated or displayed to the user such that one or more predetermined portions thereof (e.g., a region of the enhanced image corresponding to the macular region of the visual field of the user's eye or a region within the macular region of the eye) are outside the defective visual field portion. As another example, the position, shape, or size of one or more display portions of the wearable device, the brightness, contrast, saturation, or sharpness level of said display portions, the transparency of said display portions, or other configurations of the wearable device may be adjusted based on the determined defective visual field portion.

[0072] In some embodiments, one or more predictive models may be used to facilitate determining visual defects (e.g., light sensitivity, distortion, or other aberrations), determining a correction profile (e.g., a correction / enhancement profile including correction parameters or functions) used to correct or enhance a user's vision, generating an enhanced image (e.g., derived from live image data), or other manipulations. In some embodiments, the predictive models may include one or more neural networks or other machine learning models. As an example, a neural network may be based on a large number of neural units (or artificial neurons). A neural network may roughly mimic the workings of a biological brain (e.g., with large clusters of biological neurons connected by axons). Each neural unit of a neural network may be connected to many other neural units of the neural network. Such connections may have a reinforcing or inhibitory effect on the activity state of the connected neural units. In some embodiments, each neural unit may have a summation function that combines the values ​​of all its inputs. In some embodiments, each connection (or the neural unit itself) may have a threshold function that a signal must exceed before it is propagated to other neural units. These neural network systems may learn and train automatically, rather than being explicitly programmed, and may significantly outperform traditional computer programs in solving problems in a given domain. In some embodiments, the neural network may have multiple layers (e.g., signal paths traverse from higher layers to lower layers). In some embodiments, the neural network may use backpropagation, where forward stimuli are used to reset weights to "higher layer" neural units. In some embodiments, stimuli and inhibitions to the neural network may become more fluid as the connection interactions become more chaotic and complex.

[0073] As an example, as shown in FIG. 1B , a machine learning model 162 may take in input 164 and provide output 166. In one use case, the output 166 may be fed back to the machine learning model 162 as an input to train the machine learning model 162 (e.g., alone or in combination with a user's instructions regarding the accuracy of the output 166, a label associated with the input, or other reference feedback information). In another use case, the machine learning model 162 may update its configuration (e.g., weights, biases, or other parameters) based on an evaluation of the prediction (e.g., the output 166) and the reference feedback information (e.g., a user's instructions regarding the accuracy, a reference label, or other information). In another use case, if the machine learning model 162 is a neural network, the connection weights may be adjusted to adjust for discrepancies between the neural network's predictions and the reference feedback. In a further use case, one or more neurons (or nodes) of the neural network may request that their respective errors be sent backward through the neural network to facilitate the update process (e.g., error backpropagation). The update to the connection weights may reflect, for example, the magnitude of the error to be propagated backward after forward propagation is complete. In this way, for example, a predictive model may be trained to produce better predictions.

[0074] In some embodiments, upon obtaining feedback related to a stimulus set (displayed to the user), feedback related to one or more of the user's eyes, feedback related to the user's environment, or other feedback, system 100 may provide this feedback to a predictive model, and the predictive model may be configured based on the feedback. As one example, the predictive model may be automatically configured for the user based on (i) an indication of the user's response to one or more stimuli (of the set of stimuli), (ii) an indication of the user's lack of response to such stimuli, (iii) eye images captured during presentation of a visual test, or other feedback (e.g., the predictive model may be personalized for the user based on feedback from the presentation of a visual test). As another example, the predictive model may be trained based on such feedback and other feedback from other users to improve the accuracy of the results provided by the predictive model. In some embodiments, once the predictive model is configured (e.g., for the user), system 100 may provide live image data or other data to the predictive model to obtain an enhanced image (derived from the live image data) and display the enhanced image. As an example, a wearable device of system 100 may obtain a live video stream from one or more cameras of the wearable device and display an enhanced image on one or more displays of the wearable device. In some embodiments, the wearable device may obtain an enhanced image (e.g., a file or other data structure representing the enhanced image) from the predictive model. In some embodiments, the wearable device may obtain a correction profile (e.g., correction parameters or functions) from the predictive model and generate the enhanced image based on the live video stream and the correction profile. In one use case, the correction profile may include correction parameters or functions used to generate the enhanced image from the live image data (e.g., parameters of a function used to transform or correct the live image data into the enhanced image).Additionally or alternatively, the modification profile may include modification parameters or functions for dynamically configuring one or more display portions (e.g., dynamically adjusting the transparent or opaque portions of a transparent display, dynamically adjusting the projection portion of a projector, etc.).

[0075] In some embodiments, system 100 can facilitate enhancement of a user's field of view via one or more dynamic display portions (e.g., a transparent display portion of a transparent display, a projection portion of a projector, etc.). As an example, with respect to a transparent display, the dynamic display portion may include one or more transparent display portions and one or more other display portions (e.g., a display portion of a wearable device or another device). System 100 may cause one or more images to be displayed on the other display portions. As an example, a user may be able to see through a transparent display portion of a transparent display but not the other display portions, and instead see an image presentation on the other display portions of the transparent display (e.g., a portion surrounding or adjacent to the transparent display portion). In one use example, live image data may be acquired via a wearable device, and an enhanced image may be generated based on the live image data and displayed on the other display portion of the wearable device. In some embodiments, system 100 may monitor one or more changes associated with one or more eyes of the user and cause adjustments to the transparent display portion of the transparent display based on the monitoring. As an example, the monitored changes may include eye movement, a change in gaze direction, a change in pupil size, or other changes. One or more positions, shapes, sizes, transparency, or other aspects of the transparent display portion of the wearable device may be automatically adjusted based on the monitored changes. In this way, for example, system 100 may improve mobility without restricting (or at least reducing) eye movement, gaze direction, pupil response, or other eye-related changes.

[0076] In some embodiments, system 100 may facilitate an expansion of a user's field of view through the combination of portions of multiple images of a scene (e.g., based on feedback related to a set of stimuli displayed to the user or other feedback), and system 100 may acquire multiple images of a scene. System 100 may determine areas common to the multiple images and, for each image of the multiple images, determine areas of the image that differ from corresponding areas in at least one other image of the multiple images. In some embodiments, system 100 may generate or display to a user an enhanced image based on the common areas and the difference areas. As an example, the common areas and difference areas may be combined to generate an enhanced image that includes a representation of the common area and a representation of the difference area. The common areas may correspond to portions of each of the multiple images that have the same or similar characteristics as each other, and each difference area may correspond to a portion of one image that differs from all corresponding portions of the other images. In some scenarios, the different portions of one image may include portions of the scene not represented in the other images. In this way, for example, combining the common areas and difference areas into an enhanced image expands the field of view otherwise provided by each image, and the enhanced image can be used to enhance a user's vision.

[0077] In some embodiments, system 100 may generate a prediction indicating that an object will physically contact the user and cause an alert to be displayed based on the physical contact prediction (e.g., an alert related to the object is displayed on the user's wearable device). In some embodiments, system 100 may detect an object in a defective portion of the user's visual field and cause an alert to be displayed based on (i) the object's presence in the defective portion of the visual field, (ii) the physical contact prediction, or (iii) other information. In some embodiments, system 100 may determine whether the object is outside (or not sufficiently within) any image portion of the enhanced image (displayed to the user) that corresponds to at least one visual field portion that meets one or more vision criteria. In one use case, if an object is determined to be within (or sufficiently within) an image portion of the enhanced image that corresponds to a healthy portion of the user's visual field, no alert (or a lower priority alert) may be displayed (e.g., even if the object is predicted to physically contact the user). On the other hand, if an object in the defective field of view is predicted to come into physical contact with the user and the object is determined to be outside (or not sufficiently within) the user's normal field of view, an alert may be displayed on the user's wearable device. In this way, for example, the user can rely on their own normal field of view to avoid approaching objects within the user's normal field of view, thereby reducing the risk of relying on the wearable device (e.g., through habit formation) to avoid such approaching objects. However, it should be noted that in other use cases, an alert related to an object may be displayed based on the physical contact prediction, regardless of whether the object is within the user's normal field of view.

[0078] 1C , client device 104 may include eyewear device 170 forming a wearable device for the subject. In some embodiments, eyewear device 170 may be part of a vision system described herein. Eyewear device 170 includes left and right eyepieces 172 and 174. Eyepieces 172 and 174 may each include or be associated with a digital monitor configured to display a reproduced image to the corresponding eye of the subject (e.g., projected onto a screen or onto the eye). In various embodiments, the digital monitor may include a display screen, a projector, and / or hardware that generates images displayed on the display screen or projects images onto the eye (e.g., the retina of the eye). It is contemplated that a digital monitor including a projector may be located elsewhere to project images onto the subject's eye or onto eyepieces including a screen, eyeglasses, or other image projection surface. In one embodiment, the left and right eye lenses 172, 174 may be positioned relative to the housing 176 to fit into the subject's orbital region so as to be able to collect data and display / project image data, which in another example includes displaying / projecting image data to different eyes.

[0079] Each of the eye lenses 172, 174 may further include one or more inward-facing sensors 178, 180, which may be inward-facing image sensors. In one example, the inward-facing sensors 178, 180 may include an infrared camera, photoreceiver, or other infrared sensor configured to track pupil movement and determine and track the subject's visual axis. The inward-facing sensors 178, 180 (e.g., including an infrared camera) may be positioned lower relative to the eye lenses 172, 174 so as not to obstruct the subject's field of view, either the actual field of view or the field of view displayed or projected to the subject. The inward-facing sensors 178, 180 may be oriented toward the expected pupil area for better pupil and / or eye tracking. In some examples, the inward-facing sensors 178, 180 may be embedded in the eye lenses 172, 174 so as to form a contiguous inner surface.

[0080] FIG. 1D is a front view of the eyewear device 170, showing the eyepieces 172, 174 as viewed from the front. The eyepieces 172, 174 are provided with corresponding outward-facing image sensors 182, 184 that form field-of-view cameras. In other embodiments, more or fewer outward-facing image sensors 182, 184 may be provided. The outward-facing image sensors 182, 184 may be configured to capture multiple sequential images. The eyewear device 170 or a corresponding vision system may then be configured to correct and / or enhance the images, which may be personalized based on the subject's visual impairment. The eyewear device 170 may also be configured to display the corrected and / or enhanced images to the subject using a monitor in the vision mode. For example, the eyewear device may generate the corrected and / or enhanced images on a display screen associated with the eyepieces or an adjacent region, project the images on a display screen associated with the eyepieces or an adjacent region, or project the images onto one or both of the subject's eyes.

[0081] 1E and 1F show another example of an eyewear device 170. With reference to FIGS. 1E and 1F, the eyewear device 170 includes a high-resolution camera(s) 192, a power supply unit 193, a processing unit 194, a glass screen 195, a see-through display 196 (e.g., a transparent display), an eye tracking system 197, and other components.

[0082] In some embodiments, eyewear device 170 may have an examination mode. In one example of an examination mode, inward-facing sensors 178, 180 track pupil movement to perform visual axis (e.g., line of sight) tracking according to an examination protocol. In this or another example, inward-facing sensors 178, 180 may be configured to capture images of patterns reflected by the cornea and / or retina to detect distortions and abnormalities in the cornea or ocular optics.

[0083] The examination mode may be used to perform vision assessments to identify ocular disorders, such as high- and / or low-order aberrations, optic nerve disorders such as glaucoma, optic neuritis and optic neuropathy, retinal disorders such as macular degeneration and retinitis pigmentosa, visual pathway disorders such as capillary defects and tumors, and other conditions such as presbyopia, strabismus, high- and low-order aberrations, monocular vision, anisometropia and anisometropia, light sensitivity, anisometropia, refractive error, and astigmatism. In the examination mode, data is collected for a particular subject and used to correct captured images before displaying those images. Displaying images may include projecting them onto the subject via a monitor, as described herein.

[0084] In some examples, external sensors may be used to provide additional data for assessing the subject's visual field. For example, data used to correct captured images may be obtained from external testing devices, such as a perimetry device, an aberrometer, an electro-oculogram or a visual evoked potential device. Data obtained from these devices, in combination with pupil or eye tracking to determine the visual axis, may be used to create one or more correction profiles (e.g., correction profiles, enhancement profiles, etc., used to correct or enhance such images) used to correct images projected or displayed to the user.

[0085] In addition to or instead of the inspection mode, the eyewear device 170 may have a vision mode. In the vision mode, one or more outward-facing image sensors 182, 184 capture images that are transmitted to an image processor for real-time image processing. The image processor may be embedded in the eyewear device 10, for example. Alternatively, the image processor may be external to the eyewear device 170, for example, associated with an external image processing device. The image processor may be a component of a vision module and / or may include a scene processing module, as described herein.

[0086] The eyewear device 170 may be communicatively coupled to one or more image processors via wired or wireless communication, for example, via a wireless transceiver embedded in the eyewear device 170. The external image processor may include a computer, such as a laptop computer, tablet, mobile phone, network server, or other centralized or distributed computing device, and may be characterized by one or more processors and one or more memories. In the illustrated example, the captured images are processed by the external image processing device. However, in other examples, the captured images may be processed by an image processor embedded in the digital glasses. The processed images (e.g., enhanced to improve functional visual field or other visual acuity and / or enhanced to correct a visual field disorder of the subject) are then transmitted to the eyewear device 170 and displayed on a monitor for viewing by the subject.

[0087] In one example of the operation of a vision system including an eyewear device, real-time image processing of captured images may be performed by an image processor (e.g., using customized MATLAB (Mathworks, Natick, Massachusetts, USA) code) running on a small computer embedded in the eyewear device. In another example, the code may be executed by an external image processing device or by another computer wirelessly networked to communicate with the eyewear device. In one embodiment, the vision system includes the eyewear device, the image processor, and corresponding instructions for performing a vision mode and / or an inspection mode, which may be embodied alone in the eyewear device or in combination with one or more external devices (e.g., a laptop computer). Such a vision system may operate in two modes: a vision mode and a separate inspection mode.

[0088] In some embodiments, as shown in FIG. 2 , the system 100 may include a vision system 200 that includes an eyewear device 202 communicatively coupled to a network 204 for communication with a server 206, a mobile phone 208, or a personal computer 210. The server 206, the mobile phone 208, or the personal computer 210 may each include a vision correction framework 212 that implements the processing techniques described herein, such as image processing techniques, which may include techniques related to an examination mode and / or a vision mode. In the illustrated example, the vision correction framework 212 includes a processor and memory that stores an operating system and applications for implementing the techniques described herein, and further includes a transceiver for communicating with the eyewear device 202 over the network 204. The framework 212 includes an examination module 214, which in this example includes a machine learning framework. The machine learning framework may be used in conjunction with an examination protocol performed by the examination module, either in a supervised or unsupervised manner, to adaptively adjust the examination mode to more accurately assess eye disease. The results of processing by the testing module may include the generation of a vision correction model 216 customized for the subject 218 .

[0089] The vision module 220, which in some embodiments may also include a machine learning framework with accessed and customized vision correction models, generates corrected visual images for display by the eyewear device 202. The vision correction framework 212 may further include a scene processing module that processes images for use in inspection mode and / or vision mode processing and includes the processing described herein with respect to the processing modules. As described herein, in some embodiments, the eyewear device 202 may include some or all of the vision correction framework 212.

[0090] In the examination mode, the eyewear device 170 or 202, and in particular one or more inward-facing image sensors including tracking cameras positioned along the inner surface of the eyewear device 170 or 202, may be used to acquire pupil and visual axis tracking data that is used to accurately align the processed image with the subject's pupil and visual axis.

[0091] 3, the system 100 may include a vision system 300 that includes a vision correction framework 302. The vision correction framework 302 may be implemented in an image processing device 304 and an eyewear device 306 that is placed on the subject. The image processing device 304 may be included entirely in an external image processing device or other computer, or in other examples, some or all of the image processing device 304 may be implemented within the eyewear device 306.

[0092] The image processing device 304 may have a memory 308 storing instructions 310 for implementing the test mode and / or vision mode described herein. Such instructions may include instructions for collecting high-resolution images of the subject from the eyewear device 306. In the vision mode, the eyewear device 306 may acquire real-time visual field image data as raw data, processed data, or preprocessed data. In the test mode, the eyewear device may project test images (such as "text" or images of vehicles or other objects) to test various aspects of the subject's visual field.

[0093] The eyewear device 306 may be communicatively connected to the image processing device 304 via a wired or wireless link. The link may be via Universal Serial Bus (USB), IEEE 1394 (Firewire), Ethernet or other wired communication protocol devices. The wireless connection may be via any suitable wireless communication protocol such as WiFi, NFC, iBeacon, Bluetooth, Bluetooth Low Energy, etc.

[0094] In various embodiments, the image processing device 304 may have a controller operatively connected to a database via a link connected to input / output (I / O) circuitry. Additional databases may be connected to the controller in a known manner. The controller may include a program memory, a processor (also called a microcontroller or microprocessor), a random access memory (RAM), and input / output (I / O) circuitry, all interconnected via an address / data bus. While only one microprocessor is illustrated, it is contemplated that the controller may include multiple microprocessors. Similarly, the controller's memory may include multiple RAMs and multiple program memories. The RAM(s) and program memory may be implemented as semiconductor memory, magnetically readable memory, and / or optically readable memory. The link may operatively connect the controller to the capture device via the I / O circuitry.

[0095] The program memory and / or RAM may store various applications (i.e., machine-readable instructions) executed by the microprocessor. For example, an operating system may provide overall control over the operation of the vision system 300, such as the eyewear device 306 and / or the image processing device 304, and in some embodiments may provide a user interface to the devices for implementing the processes described herein. The program memory and / or RAM may also store various subroutines for accessing specific functions of the image processing device 304 described herein. By way of example, and not limitation, the subroutines may include, among other things, obtaining high-resolution images of the field of view from the eyewear device, enhancing and / or correcting the images, and providing the enhanced and / or corrected images for display to the subject by the eyewear device 306.

[0096] In addition to the above, the image processing device 304 may include other hardware resources. The device may also include various types of input / output hardware, such as a visual display and one or more input device(s) (e.g., a keypad, keyboard, etc.). In one embodiment, the display is touch-sensitive and may accept user input in cooperation with a software keyboard routine as one of the software routines. It may be beneficial for the image processing device 304 to communicate with a wider network (not shown) using any of a number of known networking devices and networking technologies (e.g., via a computer network such as an intranet or the Internet). For example, the device may be connected to a database of aberration data.

[0097] In some embodiments, system 100 may store predictive models, correction profiles, visual deficiency information (e.g., indicative of a detected user's visual deficiency), feedback information (e.g., feedback related to stimuli displayed to the user, or other feedback), or other information in one or more remote databases (e.g., the cloud). In some embodiments, feedback information, visual deficiency information, correction profiles, or other information associated with multiple users (e.g., two or more users, ten or more users, one hundred or more users, one thousand or more users, one million or more users, or other number of users) may be used to train one or more predictive models. In some embodiments, one or more predictive models may be trained or configured for a user or a predetermined type of device (e.g., a particular brand of device, a particular brand and model of device, a device with a particular set of features, etc.) and may be stored in association with the user or device type. As an example, an instance of a predictive model associated with a user or device type may be stored locally (e.g., on the user's wearable device or other user device) and remotely (e.g., in the cloud), and such instances of the predictive model may be automatically or manually synchronized between one or more user devices and the cloud so that the user has access to the latest configuration of the predictive model on either the user device or the cloud. In some embodiments, multiple correction profiles may be associated with a user or device type. In some embodiments, each of the correction profiles may include a set of correction parameters or functions that are applied to live image data of a given context to generate an enhanced presentation of the live image data. As an example, a user may have a correction profile for each set of eye characteristics (e.g., a range of gaze directions, pupil size, limbus position, or other characteristics). As a further example, a user may additionally or alternatively have a correction profile for each set of environmental characteristics (e.g., a range of environmental light levels, environmental temperature, or other characteristics).Based on the currently detected eye characteristics or environmental characteristics, a corresponding set of correction parameters or functions may be obtained and used to generate an enhanced presentation of the live image data.

[0098] Subsystems 112-124

[0099] In some embodiments, with reference to FIG. 1A , the testing subsystem 122 may provide a visual test presentation to the user. As an example, the presentation may include a set of stimuli. During (or after) the presentation, the testing subsystem 122 may obtain feedback related to the set of stimuli (e.g., feedback indicating whether or how the user is viewing one or more stimuli of the set). As an example, the feedback may include an indication of the user's response to one or more stimuli (of the set of stimuli) or an indication of the user's lack of response to such stimuli. The response (or lack thereof) may relate to eye movement, gaze direction, changes in pupil size, or user modifications to one or more stimuli or other user input (e.g., the user's response or other reaction to the stimuli). As another example, the feedback may include an image of the eye captured during the presentation of the visual test. The eye image may be an image of the retina of the eye (e.g., all or a portion of the retina), an image of the cornea of ​​the eye (e.g., all or a portion of the cornea), or other eye image. In some embodiments, the inspection subsystem 122 may generate one or more results based on the feedback, such as an affected portion of the user's visual field, an extent of the affected portion, the user's visual pathology, a correction profile to correct the aforementioned problem, or other results.

[0100] In some embodiments, based on feedback associated with the set of stimuli (displayed to the user during the presentation of the visual test) or other feedback, the inspection subsystem 122 may determine light sensitivity, distortion, or other aberrations associated with one or more of the user's eyes. In some embodiments, the set of stimuli may include patterns, and the inspection subsystem 122 may project the patterns onto one or more of the user's eyes (e.g., using a projection-based wearable eyewear device). For example, the patterns may be projected onto the user's retina or cornea to determine defects affecting the retina or cornea. In one use example, the projected patterns may be used to correct and evaluate facial dyspmorphopsia in age-related macular degeneration and other retinal diseases. As shown in FIG. 31A , a digital projection of a pattern 3100 may be projected onto the subject's eye 3102. The pattern may be digitally generated by a projector located within the eyewear device. A digital camera 3104 (e.g., an inward-facing image sensor) may also be located within the eyewear device to capture an image of the pattern 3100 reflected by the eye 3102. For example, the image may be captured from the corneal surface of the eye, as shown in Figure 32. By capturing an image of the pattern 3100, the inspection subsystem 122 may determine whether the pattern appears normal (e.g., as shown in Figure 33) or whether there are any abnormalities (e.g., as shown in Figure 34 (3101)), which may be evaluated and corrected using one of the techniques described herein.

[0101] In some embodiments, inspection subsystem 122 may display a set of stimuli to a user, acquire images of one or more of the user's eyes (e.g., at least a portion of the user's retina or cornea) as feedback related to the set of stimuli, and determine one or more correction parameters or functions to address light sensitivity, distortion, or other aberrations (e.g., low- or high-order aberrations, static or dynamic aberrations, etc.) associated with the user's eyes. Such corrections may include transformations (rotations, reflections, translations / shifts, resizing, etc.), adjustments of image parameters (brightness, contrast, saturation, sharpness, etc.), or other modifications. As an example, when a pattern (e.g., an Amsler grid or other pattern) is projected onto the user's retina or cornea, the resulting image may include a reflection (e.g., reflected from the retina or cornea) of the projected pattern with aberrations. Inspection subsystem 122 may automatically determine correction parameters or functions to apply to the pattern so that when the modified pattern is projected onto the retina or cornea, the (subsequently obtained) image of the retina or cornea is a version of the unmodified pattern image without one or more aberrations. In one use example, with reference to FIG. 31C , when pattern 3100 is projected onto a user's retina, the resulting image may include pattern 3100 with distortion (e.g., an inverse version of the distortion depicted in modified pattern 3100′ of FIG. 31D ). A function (or parameters of such a function, e.g., one that inverts the distortion in the resulting image) may be determined and applied to pattern 3100 to generate modified pattern 3100′. When modified pattern 3100′ is projected onto the user's retina, a reflection of modified pattern 3100′ from the user's retina will include the unmodified, undistorted pattern 3100 of FIG. 31C . To the extent the reflection still includes distortion, inspection subsystem 122 may automatically update the correction parameters or function applied to the pattern to further mitigate the distortion (e.g., as shown in the retinal reflection).

[0102] In another use case, eye images (e.g., images of one or more of a user's eyes) capturing a projected stimulus (e.g., a pattern or other stimulus) reflected from the retina or cornea may be used to determine a function (or parameters of a function) for correcting one or more other aberrations. The determined function or parameters may be applied to the projected stimulus, and as long as the reflection of the modified stimulus still contains aberrations, the inspection subsystem 122 may automatically update the modified parameters or functions applied to the stimulus to further mitigate the aberrations (e.g., as shown in the reflection). In a further use case, the aforementioned automatic determination of parameters or functions may be performed for each eye of the user. In this manner, for example, appropriate parameters or functions for each eye may be used to provide correction for anisometropia or other conditions in which each eye has different aberrations. For example, with regard to anisometropia, typical corrective glasses cannot correct for the unequal refractive power of the two eyes. This is because corrective glasses create two unequal-sized images (e.g., one for each eye) (aniseiikonia), and the brain cannot fuse these two images for binocular monovision, resulting in visual confusion. The reason for this problem is simple: eyeglass lenses are either convex, magnifying the image, or concave, reducing the image. The degree of magnification or reduction varies depending on the amount of correction. Provided the appropriate parameters or functions can be determined for each eye, the above-described operations (or other techniques described herein) can correct anisometropia (along with other conditions in which each eye has different aberrations), thereby avoiding visual confusion or other problems associated with such conditions.

[0103] In some embodiments, with reference to FIG. 1A , the test subsystem 122 can display a set of stimuli to a user and, based on the user's modifications to the set of stimuli or other user input, determine one or more correction parameters or functions (addressing light sensitivity, distortion, or other aberrations associated with the user's eye). In some scenarios, with reference to FIG. 31C , the pattern 3100 can be a grid diagram (e.g., an Amsler grid) or any known reference shape designed to detect a transformation required to treat one or more ocular abnormalities. This transformation can then be used to inversely distort the image in real time to improve vision. In the implementation of FIG. 8 , the vision system 800 can include a test module 802. The test module 802 can be associated with wearable glasses or can be implemented in conjunction with an external device as described herein. The test module 802 can present test stimuli, including an Amsler grid, to the subject 806. The subject may manipulate the grid image via the user device 808 or other input device (e.g., by dragging or moving one or more portions of the grid lines) to improve distortion. The vision correction framework 810 may present the Amsler grid for further correction by the subject. Once the subject completes their manual corrections (e.g., creating a corrected pattern 3100′), the vision correction framework 810 may generate a correction profile for the subject to apply to the scene as seen when using the eyewear device. As an example, the vision correction framework 810 may generate an inverse function (or parameters of such a function) that outputs a corrected pattern 3100′ when the pattern 3100 is provided as an input to the function. The workflow described for the vision system 800 may be similarly applicable to processing the other testing modes described herein.

[0104] FIG. 31B is a schematic diagram illustrating an Amsler grid 3100 (e.g., an example of a reference image) being presented as an image on wearable glasses (e.g., a VR or AR headset). The Amsler grid 3100 may be displayed or projected onto the subject's cornea and / or retina. An example of a standard grid diagram 3100 is shown in FIG. 31C. The same grid pattern may be displayed on a user device. The subject may manipulate the lines of the grid pattern, particularly the lines that appear curved, using a keyboard, mouse, touchscreen, or other input on the user device, including a user interface. The subject may identify a reference point 3102 from which to begin manipulating the image. After identifying the reference point, the subject may adjust the identified lines using the user device (e.g., arrow keys) to correct for perceived distortion due to an abnormal macula. This procedure may be performed separately for each eye, resulting in two corrected grid diagrams.

[0105] Once the subject has completed modifying the line so that it appears straight, the vision correction framework uses the new grid diagram to generate a mesh of intersections corresponding to the applied distortion. This mesh, generated in test mode, can be applied to any image to compensate for the subject's anomalies. For example, as part of the test mode verification, the modified image corresponding to the appropriate mesh can be shown to each eye. The subject can then indicate on the user device whether or not there are any defects in the corrected image; if there are no defects, the correction is considered successful. For example, Figure 31E illustrates an actual scene as perceived by the user. Figure 31F illustrates a corrected visual field. When presented to a subject with visual distortions determined by the Amsler grid method, a subject viewing the visual field of Figure 31F will perceive the actual visual field of Figure 31E.

[0106] Such correction may be performed in real time on a live image, continuously presenting the corrected visual scene to the subject. Whether the eyewear device includes a display that generates the image field or not, or whether the eyewear device is custom reality and uses a correction layer to adjust for distortion or not, the correction may be achieved in real time, since in both cases the eyewear device may use a corrective mesh that is determined.

[0107] In some examples, a reference image, such as an Amsler pattern, may be displayed directly on a touchscreen or tablet PC, such as 3150 (e.g., a tablet PC) shown in FIG. 31G. The Amsler pattern is presented on the display of the device 3150, and the subject may use a stylus 3152 to manipulate the lines that appear curved to draw corrections to be applied to the lines so that they appear straight. During the test mode, after each change is made, the grid diagram may be redrawn to reflect the latest edits. This procedure may be performed separately for each eye, resulting in two corrected grid diagrams. After the subject completes changes in the test mode, the tablet PC runs an application to create and send mesh data to a companion application on the eyewear device to process the image applying the determined mesh.

[0108] Once the eyewear device receives the modifications in the test mode, it may apply them to any image to compensate for the subject's anomalies. The resulting image may then be displayed, possibly via a virtual reality or augmented reality headset. In one example, the display uses the headset to present images to the user in a holographic format. Each displayed image may correspond to a mesh generated for each eye. If the subject sees the corrected image as being free of anomalies, the correction may be deemed successful and the image may be retained for future image processing. In some test mode embodiments, instead of or in addition to presenting a single image modified according to the modified grid diagram, a video incorporating the modifications may be presented to the subject. In one example, the video includes a live video stream from a camera fed through the corrections and is displayed to the subject.

[0109] In some embodiments, with respect to FIG. 1A , the inspection subsystem 122 may determine one or more defective portions of the user's visual field (e.g., automatically determined based on feedback related to the set of stimuli displayed to the user or other feedback). As an example, the defective portion of the visual field may be one portion of the user's visual field that fails to meet one or more visual criteria (e.g., whether or the degree to which the user perceives one or more stimuli, the degree of light sensitivity, distortion, or other aberrations, or other criteria). In some cases, the set of stimuli displayed to the user includes at least one test image of text or an object. The defective portion of the visual field may include areas of reduced visual sensitivity, areas of high or low optical aberrations, areas of reduced brightness, or other defective portions of the visual field. In some cases, the sets of stimuli may differ in contrast level by at least 20 dB relative to each other and relative to a reference contrast level. In some cases, the sets of stimuli may differ in contrast level by at least 30 dB relative to each other and relative to a reference contrast level. In some cases, the test subsystem 122 can instruct the wearable eyewear device to display a set of test stimuli to the user in a test mode in descending or ascending contrast order.

[0110] In one use case, testing was conducted on four subjects. The testing protocol involved displaying letters of text at multiple different locations on one or more display monitors of the eyewear system. To assess the subject's visual field deficit, the word "text" was displayed on the eyewear monitor for each eye, and the subject was asked to identify "text." First, the operator intentionally placed the "xt" portion of the word "text" in the subject's blind spot. All four subjects reported that they could only see the "te" portion of the word. The display control software then shifted the letters. The word "text" was then moved away from the subject's blind spot, and the subject was asked to read the word again. The subject was able to read "text," and stated that they could now see the "xt" portion of the word.

[0111] An example of this assessment protocol in test mode is shown in Figures 6A-6C. As shown in Figures 6A and 6B, the code automatically detects blind spots in the Humphrey visual field. The word "text" 600 is projected so that the "xt" portion of the word is located in the blind spot 602 (Figure 6A). The subject is asked to read the word. The word "text" 600 is then moved away from the blind spot 602 (Figure 6B), and the subject is asked to read it again. The word "text" 600 can be displayed at multiple different coordinates in the subject's visual field, which is divided into four coordinates in the illustrated example. This protocol allows for the identification of multiple blind spots, including the peripheral blind spot 604. A string of letters may be moved across the subject's visual field, and the subject may be asked to identify when the string is partially or completely invisible, partially visible, or visible at reduced intensity.

[0112] Pupil tracking functions described herein may include the physical state of the pupil (e.g., visual axis, pupil size and / or limbus), alignment, dilation, and / or line of sight. Line of sight, also known as the visual axis, may be obtained by tracking one or more of the pupil, the limbus (the edge between the cornea and the sclera), or by tracking blood vessels on the surface of the eye or within the eye. As such, pupil tracking may include tracking of the limbus or blood vessels as well. Pupil tracking may be performed using one or more inward-facing image sensors, as described herein. In various embodiments, pupil tracking functions may be used to determine parameters for aligning a projected image with the subject's field of view (FIG. 6C).

[0113] With reference to FIG. 6C , a GUI 606 display may be displayed to the operator. The GUI 606 may provide information about the test. For example, the GUI 606 may indicate the measured visual field defect and indicate the relative position of the image to the defect. The GUI 606 may be configured to automatically distribute the image to functional portions of the visual field, but may include a button to allow the operator to override the automatic mode. The external image processor may be configured to determine where the evaluation string should be displayed and may wirelessly communicate instructions to the digital glasses to display the string at various locations in the test mode.

[0114] 7A-7C, instead of utilizing "text," subjects were tested to determine whether they could see a vehicle 700 positioned in multiple different parts of their visual field in order to track their pupils and determine the area of ​​the lesion. The pupil tracking feature allows the vision system to align the projected image with the subject's visual field.

[0115] In some embodiments, with reference to FIG. 1A , the testing subsystem 122 may determine one or more defective visual field portions of the user's visual field based on the response of the user's eyes to a set of stimuli displayed to the user, or the lack of response of the user's eyes to the set of stimuli (e.g., an eye movement response, a pupil size response, etc.). In some embodiments, one or more stimuli may be dynamically displayed to the user as part of a visual test presentation, and the response or lack of response to the stimuli may be recorded and used to determine which portions of the user's visual field are healthy. As an example, if the user's eyes respond to the displayed stimuli (e.g., by changing the direction of gaze toward the location of the displayed stimuli), the eye response may be used as an indication that the eyes can see the displayed stimuli (e.g., the corresponding portion of the user's visual field is part of the user's healthy visual field). On the other hand, if the user's eyes do not respond to the displayed stimuli (e.g., by not moving the direction of gaze toward the location of the displayed stimuli), the lack of eye response may be used as an indication that the eyes cannot see the displayed stimuli (e.g., the corresponding portion of the user's visual field is a defective visual field portion). Based on these instructions, the inspection subsystem 122 may automatically determine the defective portions of the user's visual field.

[0116] In some embodiments, the set of stimuli displayed to the user may include stimuli with different brightness, contrast, saturation, or sharpness levels, and a response or lack of response to a stimulus having a particular brightness, contrast, saturation, or sharpness level may be taken as an indication of whether a portion of the user's visual field (corresponding to the location of the displayed stimulus) has a brightness, contrast, saturation, or sharpness-related problem. As an example, if a user's eyes respond to a displayed stimulus having a particular brightness level, the eye's response may be used as an indication that the eyes can see the displayed stimulus (e.g., that the corresponding portion of the user's visual field is part of the user's healthy visual field). On the other hand, if the user's eyes do not respond to a stimulus at the same location but with a lower brightness level (e.g., one that a normal eye would respond to), the lack of eye response may be used as an indication that the brightness of the corresponding portion of the user's visual field is decreasing. In some cases, the brightness level of the stimuli may be increased in stages until the user's eyes respond to the stimuli or until a certain brightness level threshold is reached. If the user's eyes eventually respond to the stimuli, the current brightness level may be used to determine the level of light sensitivity of the corresponding portion of the visual field. If the user's eye does not respond to the stimulus even when a threshold brightness level is reached, the corresponding portion of the visual field may be determined to be a blind spot (e.g., if changing one or more of the corresponding stimuli in contrast, saturation, sharpness, etc. does not result in an eye response). Based on the aforementioned instructions, the inspection subsystem 122 may automatically determine the defective portion of the user's visual field.

[0117] In some embodiments, a fixation point for visual test presentation may be dynamically determined. In some embodiments, the location of the fixation point and the location of the stimuli displayed to the user may be dynamically determined based on the user's eye gaze direction or other aspects. For example, during the presentation of the visual test, both the fixation point and the location of the stimuli may be dynamically presented to the patient relative to the patient's eye movements. In one use case, the current fixation point may be set to the location of the visual test presentation where the patient is currently viewing a particular instance, and the test stimuli may be displayed relative to that fixation point. In this way, for example, the patient does not need to fixate their attention on a specific, predetermined fixation position. This can make the presentation of the visual test more objective and interactive and reduce the stress associated with long periods of fixating on a fixed point. Additionally, using a dynamic fixation point can eliminate patient errors related to the fixation point (e.g., when a patient forgets to focus on a static fixation point).

[0118] In some embodiments, a fixation point may be locked, and one or more test stimuli may be displayed relative to that fixation point until the fixation point is unlocked (e.g., FIG. 35F). Once unlocked, the current fixation point may be set to the location of the visual test presentation where the patient is currently viewing a particular instance. A new fixation point may then be locked, and one or more subsequent test stimuli may be displayed relative to the new fixation point. In some embodiments, multiple stimuli may be displayed at one or more different locations of the visual test presentation while the fixation point remains the same. As an example, one or more stimuli may be displayed, followed by one or more other stimuli, while the fixation point remains the same. In some embodiments, after each of the multiple stimuli is displayed, it may be de-highlighted or removed from the user interface through which the visual test presentation is performed. As an example, one or more stimuli may be displayed and de-highlighted / removed, followed by one or more other stimuli, while the fixation point remains the same. In one use case, de-emphasis of the stimulus may be performed by reducing the brightness or other intensity level of the stimulus (e.g., by a predetermined amount, to a default "low" threshold level, to a patient-specific threshold level determined to be invisible to the patient, etc.). In another use case, the stimulus may be removed from the user interface (e.g., the stimulus is no longer displayed by the user interface).

[0119] As described above, in some embodiments, the testing subsystem 122 may adjust a fixation point (e.g., for a visual test presentation) based on eye characteristic information related to the user (e.g., the patient's eye movements, gaze direction, or other eye-related characteristics as they occur during the visual test presentation). In one use example, the testing subsystem 122 may cause a first stimulus to be displayed at a first interface location on a user interface (e.g., of the user's wearable device or other device) based on the fixation point. The testing subsystem 122 may adjust the fixation point based on the eye characteristic information and cause a second stimulus to be displayed at a second interface location on the user interface during the visual test presentation based on the adjusted fixation point. As described above, in some embodiments, one or more stimuli may be displayed on the user interface (e.g., at different interface locations) between the display of the first stimulus and the display of the second stimulus. The testing subsystem 122 may obtain feedback information during the visual test presentation and generate vision deficiency information based on the feedback information. By way of example, the feedback information may indicate feedback related to a first stimulus, feedback related to a second stimulus, feedback related to a third stimulus displayed during the visual test presentation, or feedback related to one or more other stimuli. Such feedback may indicate (i) the user's response to the stimuli, (ii) the user's lack of response to the stimuli, (iii) whether or to what extent the user perceives one or more stimuli, the degree of light sensitivity, distortion, or other aberrations, or (iv) other feedback. The generated visual defect information may be used to (i) train one or more predictive models, (ii) determine one or more correction profiles for the user, (iii) facilitate live image processing to correct or modify images for the user, or (iv) perform other operations described herein.

[0120] In some embodiments, the use of a dynamic fixation point during presentation of a vision test can provide greater coverage of the user's visual field than the dimensions of the view provided via the user interface. As an example, as shown with respect to FIGS. 35A-35E , a user interface (e.g., of a user's wearable device or other device) may be configured to display a view having one or more dimensions, where each dimension corresponds to a degree (e.g., a width of 70 degrees, a height of 70 degrees, a width or height of another degree, etc.). However, the use of a dynamic fixation point may allow the testing subsystem 122 to generate vision defect information that has coverage that exceeds the degree for one or more dimensions (e.g., the horizontal dimension of the user's visual field compared to the width of the user interface view, the vertical dimension of the user's visual field compared to the height of the user interface view, etc.). In one scenario, based on such techniques, the visual defect information may have coverage of an area up to 2.85 times larger than the entire user interface view, and the coverage area may be expanded to a size approaching four times the entire user interface view (for example, if the distance between the wearable device and the user's eyes decreases or the distance between two monitors of the wearable device increases). Furthermore, the visual defect information may have coverage for a width up to twice the user's visual field area greater than the width of the user interface view, a height up to twice the user's visual field area greater than the height of the user interface view, or other expanded area of ​​the user's visual field. In another scenario, based on such techniques, the visual defect information may indicate whether or to what extent defects exist at two or more visual positions in the user's visual field, where the visual positions are separated from each other with respect to one or more dimensions of the user's visual field by more than the number of degrees of the dimension of the user interface view.

[0121] In one use example, with reference to FIG. 35A , the use of a dynamic fixation point and user interface 3502 (which is configured to provide, for example, a 70-degree view) can facilitate the generation of a visual field map 3504 having coverage greater than 70 degrees in both the horizontal and vertical dimensions. As an example, as shown in FIG. 35A , stimulus 3506 a may be displayed at the center of user interface 3502, causing the user to look at the center of user interface 3502, thereby initializing fixation point 3508 to the center of user interface 3502. Specifically, if a characteristic of the user's eye (e.g., a characteristic detected by eye tracking techniques described herein) indicates that the user is looking at stimulus 3506 a, fixation point 3508 for visual test presentation may be set to a location on user interface 3502 currently corresponding to stimulus 3506 a. In some use examples, the fixation point “floats” over user interface 3502 depending on where the user is currently looking.

[0122] In another use example, as shown in FIG. 35B , stimulus 3506b may be displayed in the lower left corner of user interface 3502 (e.g., 50 degrees away from the location on user interface 3502 where stimulus 3506a was displayed). If the user's eye characteristics indicate that the user is sensing stimulus 3506b (e.g., if the user's eye movement is detected as being toward stimulus 3506b), visual field map 3504 may be updated to indicate that the user can look at a corresponding location in the user's visual field (e.g., a location 50 degrees away in the same direction from the location of the fixation point in visual field map 3504). If the user's eye characteristics indicate that the user is currently looking at stimulus 3506b, fixation point 3508 for vision test presentation may then be set to the location on the user interface corresponding to stimulus 3506b.

[0123] In another use example, as shown in FIG. 35C, stimulus 3506c may be displayed in the upper right corner of user interface 3502 (e.g., 100 degrees away from the location in user interface 3502 where stimulus 3506b was displayed). If the user's eye characteristics indicate that the user is sensing stimulus 3506c, visual field map 3504 may be updated to indicate that the user can view a corresponding location in the user's visual field (e.g., a location 100 degrees away in the same direction from the location of the fixation point in visual field map 3504). If the user's eye characteristics indicate that the user is currently looking at stimulus 3506c, fixation point 3508 for visual test presentation may then be set to the location in user interface 3502 corresponding to stimulus 3506c. As shown in FIG. 35D, stimulus 3506d may be displayed in the lower left corner of user interface 3502 (e.g., 100 degrees away from the location in user interface 3502 where stimulus 3506c was displayed). If the user's eye characteristics indicate that the user is sensing stimulus 3506d, the visual field map 3504 may be updated to indicate that the user can view a corresponding location in the user's visual field (e.g., a location 100 degrees away in the same direction from the location of the fixation point in the visual field map 3504). If the user's eye characteristics indicate that the user is currently looking at stimulus 3506d, the fixation point 3508 for visual test presentation may then be set to a location in the user interface 3502 corresponding to stimulus 3506d. As shown in FIG. 35E , stimulus 3506e may be displayed to the left of the upper right corner of the user interface 3502 (e.g., 90 degrees away from the location in the user interface 3502 where stimulus 3506d was displayed). If the user's eye characteristics indicate that the user is sensing stimulus 3506e, the visual field map 3504 may be updated to indicate that the user can view a corresponding location in the user's visual field (e.g., a location 90 degrees away in the same direction from the location of the fixation point in the visual field map 3504).If the user's eye characteristics indicate that the user is currently looking at stimulus 3506b, then fixation point 3508 for the vision test presentation may be set to the location on user interface 3502 corresponding to stimulus 3506e. In this way, for example, even if the user interface view was only 70 degrees in both the horizontal and vertical dimensions, visual field map 3504 now has coverage corresponding to 200 degrees diagonal of the user's visual field, 140 degrees of the user's visual field in the horizontal dimension, and 140 degrees of the user's visual field in the vertical dimension.

[0124] 35B , if the user's eye characteristics indicate that the user is not looking at stimulus 3506b (e.g., there was no significant eye movement response to the display of stimulus 3506b, the user's gaze did not move to an area on user interface 3502 proximate to the location of stimulus 3506b, etc.), then visual field map 3504 may be updated to indicate that the user is unable to see a corresponding location in the user's visual field. Thus, in some scenarios, the visual field map may indicate visual defects and their corresponding locations in the user's visual field for an area larger than the size of the view of user interface 3502. As an example, even if the user interface view is only 70 degrees in the horizontal and vertical dimensions, the visual field map may indicate visual defects at visual field locations that are 70 degrees or more apart from each other in each of the horizontal and vertical dimensions (e.g., the distance between the visual defects thus indicated may be up to 140 degrees apart in the horizontal dimension and up to 140 degrees apart in the vertical dimension).

[0125] In some embodiments, to facilitate greater coverage of the user's field of view (e.g., despite limitations of hardware / software components associated with the user interface view), one or more locations on the user interface may be selected to display one or more stimuli based on interface locations that are farther from a current fixation point (e.g., for visual test presentation). In some embodiments, the test subsystem 122 may select a first interface location on the user interface and cause the first stimulus to be displayed at the first interface location based on the first interface location being farther from the fixation point than one or more other interface locations on the user interface. In some embodiments, after the fixation point is adjusted (e.g., based on characteristics of the user's eye), the test subsystem 122 may select a second interface location on the user interface and cause the second stimulus to be displayed at the second interface location based on the second interface location being farther from the adjusted fixation point than one or more other interface locations on the user interface.

[0126] As an example, a first stimulus may be selected to be added to a cue of stimuli (e.g., a cue of next-to-be-presented stimuli) to be displayed during a visual test presentation based on (i) the first stimulus being associated with a first visual field location in the user's visual field, and (ii) the first visual field location corresponding to a first interface location (e.g., determined by a fixation point and the location of the first visual field location relative to the fixation point). As a further example, a second stimulus may be selected to be added to a cue of stimuli to be displayed during a visual test presentation based on (i) the second stimulus being associated with a second visual field location in the user's visual field, and (ii) the second visual field location corresponding to a second interface location. By selecting a "more distant" stimulus / location to be displayed next, the test subsystem 122 adjusts the fixation point to a location farther away from the center of the user interface view, thereby increasing coverage of the user's visual field. In one use example, with reference to Figure 35B, stimulus 3506b and its corresponding location on user interface 3502 is selected as the next stimulus / location to be displayed during a visual test presentation as a result of a determination that the corresponding interface location is one of the furthest locations on the user interface from a fixation point (located at the center of user interface 3502). In doing so, the fixation point is adjusted to the lower left corner of user interface 3502 (e.g., by having the user look there), thereby allowing the next stimulus to be displayed as much as 100 degrees away from the fixation point (e.g., the distance between stimuli 3506b and stimuli 3506c in Figure 35C).

[0127] In some embodiments, one or more locations in a user's visual field may be included as part of a set of visual field locations tested during a visual test presentation. As an example, the test set of visual field locations may be represented by stimuli during the visual test presentation, and a determination of whether or to what extent the user has a visual defect at one or more visual field locations in the test set is made based on whether or to what extent the user perceives one or more of the corresponding stimuli. In some embodiments, a visual field location may be removed from the test set based on a determination that the visual field location has been sufficiently tested (e.g., by displaying stimuli at the corresponding locations on a user interface and detecting whether or to what extent the user perceives the displayed stimuli). As an example, removing a visual field location may include labeling the visual field location in the test set as no longer available for selection from the test set during a visual test presentation. Thus, in some scenarios, stimuli corresponding to the removed visual field location may not be displayed during subsequent visual test presentations, and stimuli corresponding to one or more other visual field locations in the test set may be displayed during subsequent visual test presentations. In a further scenario, the visual field location may subsequently be added to the test set (e.g., by labeling the visual field location in the test set as selectable during visual test presentation, removing any prior label specifying that the visual field location is not selectable during visual test presentation, etc.).

[0128] In some embodiments, when the fixation point is adjusted to a first user interface location on the user interface where a first stimulus is displayed during the visual test presentation, the inspection subsystem 122 may cause one or more stimuli to be displayed on the user interface based on the fixation point at the first interface location. The inspection subsystem 122 may also subsequently cause a second stimulus to be displayed at a second interface location on the user interface. As an example, the second stimulus may be displayed while the fixation point is still at the first interface location (e.g., the fixation point may be locked at the first interface location until just before the second stimulus is displayed, until the second stimulus is displayed, or until some other time). In some embodiments, the inspection subsystem 122 may detect that the user's eyes have fixated the second interface location based on eye characteristic information associated with the user, and the inspection subsystem 122 may adjust the fixation point to the second interface location based on the fixation detection.

[0129] In some embodiments, the inspection subsystem 122 may establish a fixation point lock for visual test presentation, thereby preventing adjustment (or readjustment) of the fixation point to a different interface location on the user interface while the lock is established. In this manner, for example, while the fixation point lock is established, one or more stimuli may be displayed on the user interface to test one or more positions of the user's visual field relative to the locked fixation point. When the fixation point lock is subsequently released, the fixation point may again be dynamically adjusted. As an example, the inspection subsystem 122 may present the stimuli at a new interface location on the user interface (different from the interface location where the fixation point was set). Based on detecting that the user's eyes have fixated on the new interface location, and after the fixation point lock is released, the inspection subsystem 122 may adjust the fixation point to the new interface location. In one use case, the fixation point lock may be released to allow the user to "capture" the stimulus (e.g., step 3544), as described above with respect to FIG. 35F, and then the fixation point lock may be restored at the new interface location based on the user's viewing the stimuli (e.g., step 3546). Specifically, when the user "acquires" a stimulus (and is still looking at it), the location of this stimulus becomes the new fixation point.

[0130] In some embodiments, while a fixation point is at a first interface position on the user interface, the inspection subsystem 122 may cause multiple stimuli to be displayed at an interface position different from the first interface position. As an example, one or more stimuli of the multiple stimuli may be displayed on the user interface, followed by one or more other stimuli of the multiple stimuli. As another example, a stimulus may be displayed on the user interface, then de-highlighted or removed from the user interface, and another stimulus may subsequently be displayed on the user interface, then de-highlighted or removed from the user interface. In one use example, with reference to FIG. 35F , the fixation point may be locked to the interface position (where the previous stimulus was displayed) (e.g., step 3546), and based on the fixation point, one or more stimuli may be displayed on the user interface at a new interface position (e.g., step 3528, step 3540a, etc.).

[0131] In another use case, multiple locations in the user's visual field may be tested by displaying multiple stimuli at different interface locations while the fixation point remains locked. As an example, with respect to Figure 35C, the fixation point may alternatively be locked to the interface location where stimulus 3506b is displayed in user interface 3502, and a portion of the user's visual field corresponding to the upper right corner of visual field map 3504 may be tested by displaying the stimuli at different locations in user interface 3502 while the fixation point remains locked to the interface location of stimulus 3506b.

[0132] In some embodiments, one or more interface locations of the user interface may be pre-designated to be relative fixation points at which the user's visual field is tested. As an example, if four corners of the user interface are pre-designated to be fixation points during visual test presentation, the test subsystem 122 may initially display stimuli in the center of the user interface so that the user first fixates on the central stimulus (e.g., initial fixation point). The test subsystem 122 may then display stimuli in the upper right corner of the user interface, and upon detecting (e.g., based on the user's eye characteristics) that the user is looking at the upper right stimulus, adjust and lock the fixation point to the upper right corner of the user interface. The test subsystem 122 may subsequently test portions of the user's visual field by displaying stimuli at different locations on the user interface while the fixation point remains locked. In one use example, if the user interface is represented by user interface 3502 of Figure 35A and the user's visual field is represented by visual field map 3504 of Figure 35A, then the portion of the user's visual field corresponding to the lower left quadrant of visual field map 3504 may be fully tested by displaying stimuli at different locations on the user interface while the fixation point remains locked in the upper right corner. The above process may then be repeated for other corners of the user interface to examine portions of the user's visual field corresponding to other portions of visual field map 3504.

[0133] In some embodiments, while the fixation point is at a first interface location on the user interface (the location where the first stimulus is displayed), the inspection subsystem 122 may display multiple stimuli and then de-highlight or remove them from the user interface while the first stimulus continues to be displayed at the first interface location on the user interface. As an example, if the first interface location is the upper right corner of the user interface, the first stimulus may continue to be displayed while a series of other stimuli are momentarily displayed on the user interface. In this way, a visual change occurring at another interface location (due to another stimulus appearing at another interface location) may cause the user to look at the source of the visual change if the other interface location does not correspond to a defective portion of the user's visual field (e.g., a blind spot in the user's visual field). However, when the other stimuli disappear, the first stimulus becomes the user's primary (or only) source of visual simulation, causing the user to return to fixate at the upper right corner.

[0134] In some embodiments, while the fixation point is at a first interface location on the user interface (where the first stimulus is displayed), the inspection subsystem 122 may de-highlight or remove the first stimulus from the user interface, and then highlight or re-display it at the first interface location on the user interface. In some embodiments, while the fixation point is at the first interface location, the inspection subsystem 122 may display multiple stimuli on the user interface, and following the display of at least one stimulus of the multiple stimuli, highlight or re-display the first stimulus at the first interface location on the user interface. In one use example, if the brightness of the first stimulus is decreased, increasing the brightness of the first stimulus may cause the user's eye to detect the visual change (and the increased visual stimulus) and return to fixate on the first interface location where the first stimulus is displayed on the user interface. In another use example, if the first stimulus is removed from the user interface, the re-display of the first stimulus similarly causes the user's eye to return to fixate on the first interface location on the user interface.

[0135] In some embodiments, one or more portions of the process shown in FIG. 35F may be used to facilitate visual test presentation using a dynamic fixation point. With reference to FIG. 35F, in step 3522, a matrix of possible stimuli (e.g., all possible stimuli) in the user's visual field is created or obtained. In step 3524, an eye tracker is used to lock a floating fixation point to the center of the visual field. As an example, the eye coordinates obtained from the eye tracker may be used to "float" the floating fixation point around the eye. In step 3526, the available stimuli in the matrix may be ranked (e.g., with points farthest from the floating fixation point first). As an example, a stimulus corresponding to a position in the user interface view that is at least as far from the fixation point as all other positions on the user interface (corresponding to available stimuli in the matrix) may be ranked higher than all other available stimuli (or may be ranked with the same priority as other stimuli of equal distance from the floating fixation point). As an example, the ranking may be performed in real time using an eye tracker (e.g., a pupil or eye tracker or other eye tracker).

[0136] In step 3528, following ranking, the first stimulus on the ranking list (e.g., the stimulus with the highest priority) may be the next stimulus displayed during the visual test presentation. As an example, the stimulus may be displayed in a color that contrasts highly with the background (e.g., the color of the stimulus may be black to contrast with a black background). In step 3530, eye movement vectors (or other representations of eye-related characteristics) may be consistently measured using an eye tracking device. If eye movement is detected not toward the stimulus (step 3532), the stimulus is counted as not viewed and removed from the matrix of available stimuli in step 3534. Steps 3528-3530 are repeated using the current highest-ranked stimulus on the ranking list (those within the matrix of available stimuli).

[0137] If eye movement is detected toward the stimulus (step 3536) (e.g., thereby indicating that the user is perceiving the stimulus), the stimulus is counted as seen (qualitatively) in step 3538, and the stimulus disappears from the user interface. In steps 3540a-3540d, a visual test presentation may test the degree to which the user can perceive the stimulus in a particular region of the visual field. As an example, in step 3540a, the stimulus is re-displayed, but in a darker shade of color (e.g., a shade of gray) each time this step is performed. In one use case, the stimulus may initially be re-displayed in a color similar to the background color (e.g., if the background color is white, the stimulus color may initially be light gray). In step 3540b, eye movement vectors (or other representations of eye-related characteristics) may be constantly measured using an eye tracking device. If eye movement is detected not toward the stimulus (step 3540c), steps 3540a and 3540b are repeated (e.g., using a darker shade of color to further contrast with the white background). If eye movement is detected toward the stimulus, visual sensitivity is indicated for a particular region of the visual field based on the shade of color (e.g., the shade of gray) of the displayed stimulus (step 3542).

[0138] In step 3544, the eye tracking / floating fixation point is unlocked (e.g., to allow the user to acquire the stimulus). In step 3546, the eye tracking / floating fixation point is restored (e.g., based on the user's current looking position). As an example, if the user has "acquired" the stimulus (and is still looking at it), the location of this stimulus becomes the new floating fixation point. In step 3548, this stimulus is removed from the matrix of available stimuli, and the process is repeated in step 3526 for the other available stimuli in the matrix.

[0139] In some embodiments, the location of the fixation point or the location of the stimulus displayed to the user may be static during the visual test presentation. As an example, the test subsystem 122 may display a stimulus at the center of the user interface (or at a location corresponding to the static fixation point) to encourage the user to look at the center of the user interface (or at another location corresponding to the static fixation point). When the test subsystem 122 detects that the user is looking at the static fixation point, it may display the next stimulus in a set of stimuli for testing one or more regions of the user's visual field. Each time the test subsystem 122 detects that the user is not looking at the static fixation point, it may repeatedly display a stimulus at the static fixation point.

[0140] As another example, with reference to Figure 35G, a visual test presentation applying a fast threshold strategy may utilize a stepped pattern of four contrasting stimuli covering a central 40-degree radius, using a 52-stimulus sequence at a given location. Other examples may use different numbers of contrast stimuli, coverage areas, and stimulus locations. In this example, stimuli were placed at the center of each cell shown in Figure 35G. The 12 corner cells were excluded from the test because the stimuli were not visible due to the circular lens of the display. The spacing between stimulus locations was approximately 10 degrees. Each stimulus sequence included four consecutive stimuli with different contrast levels against the background. The contrast of the stimuli ranged from 33 dB to 24 dB, with each contrast level varying in 3 dB descending increments. Thresholds were recorded for the last stimulus presented. If the patient did not see any stimulus contrast at a particular location, that location was considered invisible and assigned a value of 0 dB.

[0141] The background was brightly lit (100 lux) while the stimuli were dark dots, providing different contrast levels. Therefore, the test was photopic rather than mesopic. In some embodiments, the background may be dark and the stimuli may be brightly lit dots. Each stimulus was presented for a period of approximately 250 milliseconds, followed by a response wait of approximately 300 milliseconds. These periods could be adjusted by the control program according to the subject's response speed. For example, they could be adjusted before the test based on pre-test confirmation or dynamically during the test. Generally, a stimulus size of 0.44 degrees was used with a central 24-degree radius, corresponding to a standard Goldmann stimulus size III. The stimulus size in the periphery (24-40 degree radius) was doubled to 0.88 degrees. Doubling the stimulus size in peripheral vision was intended to compensate for the degradation of display lens performance in the periphery. This lens degradation was significant because normal human vision also deteriorates in the peripheral region. The test program also allowed for the stimulus size to be adjusted to suit different patient cases.

[0142] A fixation target (pattern) (Figure 35G) was placed at the center of the screen for each eye tested. This target was configured as a multicolored dot, rather than the single-color fixation point typically used in the traditional Humphrey test. The color change facilitated the subject's attention and helped them focus on the target. The frequency of the color change was not synchronized with the frequency of the stimulus appearance, so subjects would not associate the two and respond incorrectly. This test protocol also allowed for the fixation target size to be adjusted to suit the patient's condition. Additionally, an eye / pupil tracking system could be used to monitor the subject's eye fixation at various time intervals. The eye tracking system transmitted the gaze vector direction to the test program, informing the program whether the subject was properly focusing on the center.

[0143] Fixation was confirmed using pupil / gaze data for each eye separately. Pupil / gaze data was acquired at various times. If the gaze direction vector was approximately 0 degrees, the subject was focused on the central target; otherwise, the program paused and waited for the subject to fixate again. If the patient did not fixate, no stimulus was presented, and the test stopped until the participant refixated. A margin of error was allowed for deviation due to small eye movements at the fixation target. Fixation was confirmed at two main times for each stimulus position: before presenting each stimulus in the stimulus sequence (e.g., before each of the four stimulus contrast levels mentioned above) and before recording responses, whether positive (e.g., if the patient looked at the stimulus) or negative (e.g., if the patient did not look at the stimulus). Negative responses were recorded at the end of the stimulus sequence period, in addition to the allowed response time. Confirming fixation before presenting the stimulus sequence was intended to ensure that the patient was focused on the fixation target. If the participant did not fixate, the stimulus was not presented and the test stopped until the participant fixed again.

[0144] FIG. 36 is a timing diagram illustrating the processing of a test sequence at one stimulus location. In one example, a pupil tracker, which may be separate from or a component of the vision system or device, may include an inward-facing image sensor and be configured to provide data to direct the visual display. The visual display may include a projector to change the position of the projected stimulus according to eye movements. In this manner, even when the subject is looking around and not fixating, the stimulus may move with the subject's eyes, continuing to test the desired location in the visual field. Thus, rather than stopping the stimulus sequence when it is determined that the subject is focusing on something other than the fixation target, the stimulus sequence may continually change the stimuli to correspond to their intended location in the subject's visual field, as repositioned based on determining the subject's current fixation point.

[0145] For each subject, the visual field test began with an explanation to the subject of how the test would proceed. The eyeglass device was fitted to the patient, ensuring that the subject could clearly see the fixation target, and the target size was adjusted accordingly, if necessary. Calibration of the eye tracker was performed at a single point, the fixation target. After this, the subject was presented with a demonstration mode. This mode involved the same sequence as the main test, but in this case, a reduced number of positions (7 positions) and did not record any responses. The purpose of this mode was to train the subject on the test. This training mode also allowed the program administrator to check the accuracy of the eye tracking system, the patient's response speed, and the patient's eye position relative to the headset, ensuring that errors and deviations did not occur during the main test.

[0146] The scan then probed the normal blind spot by presenting suprathreshold stimuli positioned at four different locations, spaced 1 degree apart within a 15-degree radius, a useful step in avoiding rotational misalignment between the headset and the subject's eyes.

[0147] A sequence of 52 stimuli was then presented to the patient in random order at pre-specified locations. Subjects responded to the stimuli by activating an electronic clicker or making gestures. After recording the subject's responses at all locations, the locations of the "missing" points were temporarily stored. A search algorithm was then used to find the locations of all "seen" points on the boundary of the "missing" points. These two sets of points were then retested to eliminate random response errors by the participant and ensure a continuous visual field. False positive responses, false negative responses, and fixation losses (if present) were calculated and reported by the end of the test. All 52 responses were then interpolated using a cubic algorithm to generate a continuous visual field plot for the tested participant.

[0148] Visual field tests were conducted on 20 volunteers using simulated visual field defects created by covering a portion of the inner display lens of the eyeglass device. Results were assessed by point-by-point comparison using an image of the covered area on the display. As a measure of test accuracy, 52 responses were compared with approximately corresponding locations on the display image of the covered headset. The calculated errors are summarized in Table 1. [Table 1]

[0149] Meanwhile, visual field tests of 23 clinical patients were compared with the most recent results of the subjects' usual Humphrey Field Analyzer (HFA) examination during their clinical visit. The two visual field test devices were compared by matching a common central 24-degree field. Again, comparisons and relative error calculations were performed point-by-point within the common central 24-degree field; areas outside this field were determined by their continuity with the central field and the absence of isolated response points. The calculated errors are summarized in Table 2. [Table 2]

[0150] This was followed by an image remapping process, which involved identifying new dimensions and a center for the displayed image shown to the patient. The output image was fitted into the photopic field of the subject's eye by resizing and shifting the original input image.

[0151] The visual field was binarized by setting all "seen" patient responses to 1 and leaving all "not seen" responses at zero. This resulted in a small binary image of size 8x8. In other embodiments, larger or smaller binary images may be used. Subregions containing up to four connected pixels were removed from the binary visual field image. Four connected pixels represented a predetermined threshold for determining subregions, although in some embodiments the threshold may be larger or smaller. Such subregions were not considered in the image embedding process. The ignored subregions may represent either normal blind spots, insignificant defects, or any random error responses that may occur during the subject's visual field testing.

[0152] Based on this interpolated binary field image, properties of the bright field region were calculated. The properties calculated for the bright region included: 1) the bright region in pixels; 2) the region's bounding box; 3) the weighted region's centroid; and 4) a list of all pixels in the field that make up the bright region. The bounding box was the smallest rectangle that encompassed all pixels that make up the bright region. The region's centroid was the calculated center of mass of the region in terms of horizontal and vertical coordinates. The value of this property corresponds to the new center of the output image and the amount of image shift required for the mapping.

[0153] Using the list of pixels that make up the largest bright field, we calculated the width and height of all pixels that bound the bright field, as shown in Figure 37. For each row of the bright field, we identified two bordering pixels and subtracted their vertical coordinates to determine the width BF of the bright field for that row. widthsThis width calculation was carried out for all the rows that make up the bright field, using the BF widths The same repetition was repeated to calculate BF heights Then, one of two scaling equations can be used to calculate the size of the mapped output image, Width map and Height map may be newly determined as shown in FIG.

[0154] Width map may be calculated using the following resizing equation:

number

[0155] BF widths and B.F. heights are the pixel width and height, respectively, that bound the calculated bright field. In this scaling method, the median bright field size in each direction is calculated as the new output image size, as specified above, and the new image is centered there. This method of determining the median was used instead of the average value to avoid resizing distortions associated with bright field dimensions that are too large or too small. The mapping behavior of this method is to fit the image within the largest possible bright area, but this method does not maintain the aspect ratio, which may result in image stretching or shrinking.

[0156] Height map may be calculated using the following resizing equation:

number

[0157] I size is the size of the interpolated image (output image size), and BX widths , BX heightsare the width and height of the bounding box. The sums in the numerator of this equation are approximately equal to the calculated bright field areas in the horizontal and vertical directions, respectively. Therefore, dividing these sums by the square of the output image size gives an estimate of the image area proportional numbers to map in each direction. These proportional numbers are then multiplied by the corresponding bounding box dimensions calculated previously. The mapping behavior of this method is to fit the image into the maximum bright field while preserving the aspect ratio of the output image. Incorporating the bounding box dimensions into the calculation achieves this effect. However, the aspect ratio was not maintained for all field patterns with defects.

[0158] In one embodiment, the AI ​​system may use these two equations, as well as dozens if not hundreds of different equations, in an optimization process to determine which will fit more of the visible field of view into the image. Based on operator feedback, the system may learn to favor one equation over another based on the particular field of view to be corrected.

[0159] These remapping techniques were used in a hazard identification test. The remapping method was tested on 23 subjects using test images containing a safety hazard, in this case a vehicle. The test images were selected to test the four major quadrants of the visual field, as shown in Figure 38. An example of the visual field was used to remap the test image displayed to the subjects. The subjects were tested by viewing an image of an oncoming vehicle. The subjects were blind to the vehicle until presented with the remapped image. Figure 39A shows the image the subjects saw without remapping, and Figure 39B shows the image they saw after remapping. Our preliminary study showed that 78% of subjects (18 of 23) were able to identify safety hazards that they would not have been able to identify without the aid of this technology. Some subjects were tested separately with both eyes, resulting in a test of 33 eyes. The visual aid technology was found to be effective in helping subjects identify simulated oncoming hazards in 23 of the 33 eyes (P = 0.023).

[0160] As noted, in some embodiments, with respect to FIG. 1A , the testing subsystem 122 may determine one or more defective visual field portions of the user's visual field based on the user's eye response to a set of stimuli displayed to the user, or the lack of the user's eye response to the set of stimuli (e.g., an eye movement response, a pupil size response, etc.). In some embodiments, one or more moving stimuli may be dynamically displayed to the user as part of the visual test presentation, and the response or lack of response to the stimuli may be recorded and used to determine which portions of the user's visual field are healthy. As one example, during the dynamic portion of the visual test presentation, recording of the patient's eye response may begin after a stimulus is displayed in the visual test presentation and continue until the stimulus disappears (e.g., the stimulus may disappear after moving from a start point to a center point of the visual test presentation). As another example, during the visual test presentation, the stimulus may be removed (e.g., it may disappear from the patient's field of view) when it is determined that the patient recognizes the stimulus (e.g., when the patient's gaze direction changes to the current position of the stimulus). In this way, the visual test presentation can be made shorter and more interactive (e.g., the patient feels like they are playing a game rather than diagnosing a visual defect). Based on the aforementioned indications (indicating a response or lack thereof to a set of stimuli), the test subsystem 122 may automatically determine the portion of the user's visual field that is defective.

[0161] In some embodiments, inspection subsystem 122 may determine one or more defective visual field portions of the user's visual field, and vision subsystem 124 may provide an enhanced image or adjust one or more configurations of the wearable device based on the determination of the defective visual field portions. As one example, the enhanced image may be generated or displayed to the user such that one or more predetermined portions thereof (e.g., a region of the enhanced image corresponding to the macular region of the visual field of the user's eye or a region within the macular region of the eye) are outside the defective visual field portions. As another example, the position, shape, or size of one or more display portions of the wearable device, the brightness, contrast, saturation, or sharpness level of the display portions, the transparency of the display portions, or other configurations of the wearable device may be adjusted based on the determined defective visual field portions.

[0162] FIG. 4 illustrates a process 400 for illustrating an example implementation of both the test mode and the subsequent vision mode. In block 402, the test mode acquires data from a diagnostic device, such as an image sensor embedded within an eyewear device or other user input device, such as a mobile phone or tablet PC. In block 404, test mode diagnostics may be performed to detect and measure ocular abnormalities from the received data (e.g., visual field defects, eye misalignment, pupil movement and size, and images of patterns reflected from the corneal or retinal surfaces). In one example, the control program and algorithms were implemented using MATLAB R2017b (MathWorks, Inc., Natick, Massachusetts, USA). In various embodiments, the subject or examiner may be given the option of testing each eye individually or testing both eyes sequentially in one go. In some embodiments, the test mode may include an adaptive fast thresholding scheme using a stimulus sequence at predetermined locations, including contrast staircase stimuli covering a central radius of 20 degrees or more. As one example, the test mode may include an adaptive fast thresholding scheme utilizing a 52-stimulus sequence at predetermined locations covering a central 40-degree radius and including four contrasting stepped stimuli, as described herein with reference to Figures 35A-35G and 36. As another example, the test mode may include automatically determining a visual defect (e.g., defective visual field portion) based on one or more responses of the user's eye to a set of stimuli displayed to the user, or a lack of such responses of the user's eye to the set of stimuli (e.g., eye movement response, pupil size response, etc.), as described herein.

[0163] In block 406, the determined diagnostic data may be compared to a database or data set containing modified profiles to compensate for identifiable eye disorders (e.g., FIG. 16 and associated discussion).

[0164] The identified correction profile may then be tailored to the individual to compensate for, for example, visual axis differences, visual field defects, light sensitivity, diplopia, image size variations between the eyes, image distortion, and visual acuity loss.

[0165] The personalized profile may be used by block 408 along with real-time data to process the image (e.g., using an image processor, a scene processing module, and / or a vision module). The real-time data may include data detected by one or more inward-facing image sensors 410 providing pupil tracking data and / or data from one or more outward-facing image sensors, including one or more field-of-view cameras 412 positioned to capture a field-of-view screen. In block 414, real-time image correction may be performed, and the image may be displayed on the eyewear device (block 416) as a regenerated digital image, an augmented reality image passing through the eyewear device with the corrected portion superimposed, or an image projected onto the subject's retina. In some examples, the processing of block 414 is performed in combination with a calibration mode 418. In calibration mode 418, a user may adjust the image correction using a user interface. The user interface may be an input device or the like that allows the user to control the image and the correction profile. For example, a user may shift the image for one eye laterally, upward, and downward, or cyclotorted to alleviate double vision. In these or other examples, a user may tweak the degree of transformation (e.g., fisheye, polynomial, or conformal) or translation of the field of view, tweak brightness and contrast, or invert colors to increase the field of view without adversely affecting visual function or introducing unacceptable distortion.

[0166] FIG. 5 illustrates another example process 500. Process 500, like example process 400, implements an examination mode and a vision mode. Block 502 collects high-order and low-order aberration data as a function of pupil size, accommodation, and gaze. In some embodiments, all or part of this data may be collected from an aberrometer or by capturing images of patterns or grid patterns projected onto the cornea and / or retina and comparing them with reference images to detect aberrations in the cornea or the entire ocular optical system. The collected data may be sent to a vision correction framework. Block 504, like block 406 described above, may determine a personalized correction profile. Blocks 508 through 518 perform similar functions to corresponding blocks 408 through 418 of process 400.

[0167] FIG. 8 illustrates a workflow 800 depicting a test module 802 that generates and presents a plurality of visual stimuli 804 to a user 806 using an eyewear device. The user 804 has a user device 808 through which the user interacts to provide input responses to the test stimuli. In some examples, the user device 808 may include a joystick, an electronic tally, a keyboard, a mouse, a gesture detector / motion sensor, a computer, a phone such as a smartphone, a dedicated device, and / or a tablet PC through which the user interacts to provide input responses to the test stimuli. The user device 808 may further include a processor and memory storing instructions. The instructions, when executed by the processor, generate a GUI display for the user to interact with. The user device 808 may include memory, a transceiver (XVR) for transmitting and receiving signals, and an input / output interface for wired or wireless connection to a vision correction framework 810. The vision correction framework 810 may be stored in an image processing device. The vision correction framework 810 may be stored in the eyewear device, the user device, etc., but in the illustrated example, the framework 810 is stored in an external image processing device. The framework 810 receives test mode information from the test module 802 and user input data from the user device 808.

[0168] FIG. 9 illustrates a test mode process 900 performed by workflow 800. At block 902, a plurality of test stimuli are presented to the subject according to a test mode protocol. These stimuli may include images of text, images of objects, flashing lights, grid patterns, and other patterns. The stimuli may be displayed to the subject or projected onto the subject's retina and / or cornea. At block 904, the vision correction framework may receive detection data from one or more inward-facing image sensors, such as data corresponding to a physical state of the pupil (e.g., visual axis, pupil size, and / or limbus). Block 904 may further include receiving user response data collected from the user in response to the stimuli. At block 906, pupil position states may be determined for a plurality of different stimuli, such as by measuring differences in position and misalignment between one stimulus and another.

[0169] At block 908, astigmatism may be determined across the visual field. This may include analyzing pupil displacement data and / or the aberrations of the eye (e.g., projecting a reference image onto the retina and cornea and comparing a reflected image from the retinal or corneal surface to the reference image). At block 910, the total aberrations of the eye may be determined (e.g., by projecting a reference image onto the retina and / or cornea and then comparing a reflected image from the retinal or corneal surface to the reference image, as described in FIGS. 31A, 32-34, and the associated description). At block 912, visual distortions, such as optical distortions such as coma, astigmatism, or spherical aberration, or visual distortions due to retinal disease, may be measured across the visual field. At block 914, visual field sensitivity may be measured across the visual field. In various embodiments of the process of FIG. 9, one or more of blocks 904-914 may be optional.

[0170] In some examples, the vision systems described herein can evaluate data obtained in an examination mode to determine the type of ocular abnormality and the type of correction required. For example, FIG. 10 illustrates a process 1000 that may be implemented as part of the examination mode, including an artificial intelligence correction algorithm mode. A machine learning framework is loaded at block 1002. Example frameworks may include dimensionality reduction, ensemble learning, meta-learning, reinforcement learning, supervised learning, Bayesian, decision tree algorithms, linear classifiers, unsupervised learning, artificial neural networks, association rule learning, hierarchical cluster analysis, cluster analysis, deep learning, semi-supervised learning, etc.

[0171] In block 1004, the type of visual field defect is determined. Three example visual field defects are illustrated: an uncompensated blind spot field 1006, a low-sensitivity partial blind spot 1008, and a normal visual field 1010. After determining the visual field defect, block 1004 applies the appropriate correction protocol for the vision mode. For example, for the uncompensated blind spot field 1006, in block 1012, the vision correction framework tracks the vision using, for example, pupil tracking with an inward-facing image sensor and performs video tracking of moving objects in the visual field (e.g., using, for example, an outward-facing image sensor such as an external camera). In the illustrated example, in block 1014, a safety hazard located in or moving into the blind spot area is detected, for example, by comparing the location of the safety hazard with the defective mapped visual field measured in inspection mode. In block 1016, objects of interest may be monitored at various locations, including central and peripheral locations.

[0172] In the example of a partial blind spot 1008, an enhanced vision mode may be initiated in block 1018. From this block, objects in the field of view are monitored by tracking the central portion of the field of view. In block 1020, an image segmentation algorithm may be employed to separate the object from the field of view. Enhanced contours may then be applied to the object and displayed to the user, where the contours correspond to the identified outer edges of the segmented object. For a normal field of view 1010, customized correction algorithms may be applied in block 1022 to correct for aberrations, visual field defects, esotropia, and / or visual distortions.

[0173] In some embodiments, the inspection subsystem 122 may determine multiple correction profiles associated with a user (e.g., during a visual inspection presentation, while an enhanced presentation of live image data is displayed to the user, etc.). In some embodiments, each of the correction profiles may include a set of correction parameters or functions that are applied to the live image data in a given context. As an example, a user may have a correction profile for each set of eye characteristics (e.g., a range of gaze directions, pupil size, limbus position, or other characteristics). As a further example, a user may additionally or alternatively have a correction profile for each set of environmental characteristics (e.g., a range of environmental light levels, environmental temperature, or other characteristics).

[0174] Based on the currently detected eye-related characteristics or environmental-related characteristics, a corresponding set of modification parameters or functions may be obtained and used to generate an enhanced presentation of the live image data. As an example, the corresponding set of modification parameters or functions may be obtained (e.g., applied to an image to modify the image for a user) based on the currently detected eye-related characteristics matching a set of eye-related characteristics associated with the obtained set of modification parameters or functions (e.g., the currently detected eye-related characteristics falling within the set of associated eye-related characteristics). In some embodiments, the set of modification parameters or functions may be generated based on the currently detected eye characteristics or environmental characteristics (e.g., ad-hoc generation of modification parameters, adjustment of a set of modification parameters or functions of a currently stored modification profile associated with a user for a given context, etc.).

[0175] In one use case, a wearable device (implementing the aforementioned operations) may automatically adjust the brightness of the enhanced presentation of live image data for one or more eyes of a user based on the respective pupil sizes (e.g., such adjustment is independent of the brightness of the surrounding environment). As an example, subjects with anisocoria have unequal pupil sizes and monocular light sensitivity that prevents them from tolerating the brightness of light that a healthy eye can tolerate. In this manner, the wearable device allows for automatic adjustment of the brightness of each eye individually (e.g., based on the detected pupil size of each eye).

[0176] In another use example, the wearable device may detect pupil size, visual axis, optical axis, limbus position, line of sight, or other ocular accommodation conditions (e.g., including changes in said conditions) and modify the correction profile based on the detected conditions. As an example, for a subject with high-order aberrations (e.g., refractive errors that cannot be corrected by glasses or contact lenses), the subject's aberrations are dynamic and change depending on pupil size and the state of accommodation. The wearable device may detect accommodation by detecting the appearance of the pupillary near reflex (e.g., miosis (pupil constriction) and accommodative convergence (eyes moving inward)). Additionally or alternatively, the wearable device may include pupil and eye tracking devices to detect gaze direction. As another example, ocular aberrations change depending on the size and position of the visual system opening and can be measured in relation to different pupil sizes and pupil-visual axis positions. The wearable device may, for example, measure irregularities on the cornea to determine high-order aberrations (e.g., based on measurements) and then calculate a correction profile to address the high-order aberrations. Different correction profiles may be created for different sizes and positions of the pupil and visual axis (or other ocular accommodation states) and stored for future use to provide real-time enhancement. One or more of these detected inputs may cause the wearable device to use the appropriate correction profile (e.g., set of correction parameters or functions) to provide enhancement for the user.

[0177] As another example, a wearable device may be used to correct presbyopia. In this case, the wearable device provides near vision by automatically performing autofocus on an image displayed to the user. To further promote and enhance near vision, the wearable device may detect where the user is looking at a near object (e.g., by detecting near reflexes such as miosis (pupil constriction) and accommodative convergence (eyes moving inward)) and perform autofocus on an area of ​​the image corresponding to the object the user is looking at (e.g., the portion of the display the user is looking at, a near vision area surrounding the object the user is looking at, etc.). Additionally or alternatively, the wearable device may determine how far away a target (e.g., a target object or area) is by quantifying (e.g., via a sensor in the wearable device) the amount of near reflex the subject exhibits and the distance of the target from the eyes, and provide an appropriate correction based on the quantified amount and the distance of the target.

[0178] As another example, a wearable device may be used to correct double vision (e.g., associated with strabismus). The wearable device may monitor a user's eyes, track the user's pupils to measure deviation, and displace the image projected onto each eye (e.g., in conjunction with detecting a condition such as strabismus). Because double vision is typically dynamic (e.g., double vision increases or decreases toward one or more lines of sight), the wearable device can provide appropriate correction by monitoring the user's pupils and the user's line of sight. For example, if there is a problem moving a user's right pupil away from the user's nose (e.g., toward the edge of the user's face), the user's double vision may increase when the user looks to the right and decrease when the user looks to the left. In this manner, the wearable device can dynamically compensate for the user's condition (e.g., strabismus or other condition) by displaying an enhanced representation of the live image data to each eye such that a first version of the enhanced representation displayed to one of the user's eyes reflects a displacement from a second version of the enhanced representation displayed to the user's other eye (e.g., the amount of displacement is based on pupil position and gaze direction), thereby preventing double vision for all gaze directions.

[0179] While prisms can be applied to displace the image presented in front of an eye with strabismus (e.g., caused by strabismus or other conditions) to correct double vision, prisms cannot cause image distortion and are therefore not useful for correcting double vision resulting from conditions that cause images to appear tilted or cyclotorted (e.g., cyclotorsion, a form of strabismus that causes the images received from both eyes to appear tilted or cyclotorted). In some use cases, a wearable device can measure the degree of strabismus (e.g., including cyclotorsion) by monitoring the user's eyes and correlatively detecting the pupils, limbus, line of sight, or visual axis of both eyes. Additionally or alternatively, such measurements by the wearable device may be made by acquiring images of the retinas of both eyes and comparing the retinal and neural structures to each other. The wearable device may then detect and measure the relative positions of those eye structures and any torsional displacement. Such measurements may be provided to a predictive model to predict correction parameters for live image processing to correct the defect and mitigate diplopia. Continuous feedback from sensors in the wearable device (e.g., pupil trackers, eye-gaze trackers, trackers based on retinal images, etc.) may be used to modify the correction profile applied to the live image data in real time. A further use case may allow the user to fine-tune the correction. For example, an image may be displayed to the user on a user interface, and the user may move the image (or an object represented by the image) (e.g., using an input device such as a joystick) until it passes in front of one eye and rotate the object until it overlaps with the image seen by the other eye. In some embodiments, when an indication of diplopia is detected, the wearable device may make a measurement or correction related to diplopia by automatically moving (e.g., translating or rotating) the image that passes in front of one eye, without any user input explicitly indicating that the image should be moved or the amount or location of the movement.

[0180] As with other forms of strabismus, the resulting displacement due to rotational strabismus changes in real time based on the intended direction of action of the paralyzed (or partially paralyzed) muscles associated with the rotational strabismus, changing when such patients look to one side or the other. By tracking eye characteristics, the wearable device can dynamically compensate for the user's condition by displaying an enhanced representation of live image data to each eye such that a first version of the enhanced representation displayed to one eye of the user reflects a displacement from a second version of the enhanced representation displayed to the other eye of the user (e.g., the amount of displacement is based on pupil position and gaze direction).

[0181] In some embodiments, with respect to FIG. 1A , upon obtaining feedback related to a set of stimuli (displayed to the user during the visual test presentation), feedback related to one or more eyes of the user, feedback related to the user's environment, or other feedback, the inspection subsystem 122 may provide this feedback to a predictive model, and the predictive model may be configured based on the feedback. In some embodiments, the inspection subsystem 122 may obtain a second set of stimuli (e.g., during the visual test presentation). As an example, the second set of stimuli may be generated based on processing the set of stimuli by the predictive model and feedback related to the set of stimuli. The second set of stimuli may be additional stimuli derived from the feedback to further inspect one or more other aspects of the user's visual field (e.g., to facilitate more granular correction or other enhancements to the user's visual field). In one use example, the inspection subsystem 122 may display the second set of stimuli to the user (e.g., during the same visual presentation) and, in response, obtain further feedback related to the second set of stimuli (e.g., the further feedback indicating whether or how the user is viewing one or more stimuli in the second set). The inspection subsystem 122 may then provide further feedback related to the second set of stimuli to the predictive model, and the predictive model may be further configured based on the further feedback (e.g., during the visual test presentation). As an example, the predictive model may be automatically configured for the user based on (i) an indication of the user's response to one or more stimuli (e.g., of the set of stimuli, of the second set of stimuli, or of the other set of stimuli), (ii) an indication of the user's lack of response to such stimuli, (iii) eye images captured during the visual test presentation, or other feedback (e.g., the predictive model may be personalized for the user based on feedback from the visual test presentation). In one use example, for example, the feedback indicates one or more visual deficiencies of the user, and the predictive model may be automatically configured based on the feedback to address the visual deficiencies.As another example, the predictive model may be trained based on such feedback and other feedback from other users to improve the accuracy of the results provided by the predictive model (e.g., trained to provide a correction profile as described herein or trained to generate an enhanced presentation of the live image data).

[0182] In some embodiments, the vision subsystem 124 may provide live image data or other data (e.g., monitored eye-related characteristics) to the predictive model to obtain an enhanced image (derived from the live image data) and display the enhanced image. In some embodiments, the predictive model may continue to be configured during the display of the enhanced image (derived from the live image data) based on further feedback continuously provided to the predictive model (e.g., periodically, according to a schedule, or based on other automated triggers). As an example, the wearable device may obtain a live video stream from one or more cameras of the wearable device and display the enhanced image on one or more displays of the wearable device (e.g., within less than one thousandth of a second, less than one hundredth of a second, less than one tenth of a second, less than one second, etc. after the live video stream is captured by the camera of the wearable device). In some embodiments, the wearable device may obtain an enhanced image from the predictive model (e.g., in response to providing live image data, monitored eye-related characteristics, or other data to the predictive model). In some embodiments, the wearable device may obtain modification parameters or functions from the predictive model (e.g., in response to providing live image data, monitored eye-related characteristics, or other data to the predictive model). The wearable device may use the modification parameters or functions (e.g., parameters of a function used to transform or modify the live image data into the enhanced image) to generate the enhanced image from the live image data. As a further example, the modification parameters may include one or more transformation parameters, brightness parameters, contrast parameters, saturation parameters, sharpness parameters, or other parameters.

[0183] In one example, a vision correction framework having a machine learning framework including AI algorithms may be used to create an automated personalized correction profile by applying transformations, translations, and resizing of the visual field to better match the remaining functional visual field. The machine learning framework may include one or more of data collection, visual field classification, and / or regression models. A graphical user interface (GUI) and data collection program may be utilized to facilitate recording of participant responses, quantitative scores, and feedback.

[0184] With respect to transformations applied to images in visual mode, exemplary transformations of the machine learning framework may include one or more of the following: 1) conformal mapping, 2) fisheye, 3) a custom fourth-order polynomial transformation, 4) a polar polynomial transformation (using polar coordinates), or 5) a rectangular polynomial transformation (using rectangular coordinates) (e.g., FIG. 13).

[0185] The translations applied to the image in the visual mode may include, by way of example, one or more of the following: Center detection may utilize a weighted average of the best center and the closest point to the center. For example, the closest point may be determined by finding the point closest to the center location. The best center may be determined by one or more of the following: 1) the centroid of the largest component; 2) the center of the largest inscribed circle, largest inscribed square, largest inscribed rhombus, and / or largest inscribed rectangle; or 3) the center of the locally largest inscribed circle, largest inscribed square, largest inscribed rhombus, and / or largest inscribed rectangle (e.g., FIG. 14). For example, instead of searching for the largest shape, the framework may employ a weighted average of the closest points from multiple methods to avoid straying from the macular visual region.

[0186] In various embodiments, an AI algorithm may be initially trained using simulated visual field defects. For example, a dataset of visual field defects may be collected to train the AI ​​algorithm. For example, in one experimental protocol, a dataset of 400 visual field defects was obtained from glaucoma patients. This dataset may be used to simulate visual field defects on virtual reality glasses that are presented to normal subjects for grading. Feedback from this grading may then be used to train the algorithm.

[0187] For example, an AI algorithm may be used to automatically fit an input image into a region corresponding to a normal visual field pattern, specific to each patient. In various embodiments, the algorithm may have at least three degrees of freedom in remapping the image, although more or fewer degrees of freedom may be used. In one example, the degrees of freedom include translation, shifting, and resizing. Additional image transformations may be applied to maintain the quality of the central region of the image, corresponding to central vision, where visual acuity is highest, while concentrating peripheral regions of moderate quality. This may be applied so that the generated image is fully perceptible to the patient.

[0188] The image transformations included in the AI ​​algorithm may include one or more of a conformal mapping transformation, a polynomial transformation, or a fisheye transformation. In some embodiments, other transformations may be used. The machine learning technique may be trained on a landmark dataset before actually performing the task. In one example, the AI ​​algorithm may be trained on a visual field dataset containing various types of peripheral defects. For example, in one experiment, the dataset included 400 visual field defect patterns. After this training phase, normal participants assigned quantitative scores to the remapped images created by the AI ​​algorithm.

[0189] 11 illustrates an example test image (stimulus) image 1100. The test image 1100 may be designed to measure visual acuity, paracentral visual acuity, and / or peripheral visual acuity. The illustrated test image displays five letters in a central region, four inner diamonds 1102 in a paracentral region, and eight outer diamonds 1104 in a peripheral region, as shown in FIG. 11.

[0190] As mentioned above, a large amount of data is required to be able to train an AI system. As a first step, a patient's binocular vision may be simulated using a binocular field including defects, as shown in Figure 12. The simulated visual acuity may then be presented to the subject using an eyeglass device. In this way, the input image can be manipulated using various image manipulations and then presented again to the subject for rating. The correction process may continue, with the modified image being further modified and presented to the subject until an optimal modified image is determined. Figure 13 shows examples of different correction transformations applied to the image presented to the user. Figure 14 shows examples of different translation methods (shifting the image to fit into the normal field of view). The normal field is white and the blind spot field is black.

[0191] The AI ​​system may be designed using machine learning models such as artificial neural networks and support vector machines (SVMs). In some examples, the AI ​​system is designed to generate an output including an estimate of the best image manipulation method (e.g., geometric transformation and translation) by an optimization AI system. In a vision mode, the vision system may present images manipulated according to the output image manipulation method to the patient using a headset to provide the best possible vision for the patient based on their visual field, including any deficiencies. The machine learning framework of the vision correction framework (also referred to herein as the “AI system”) may be trained using the collected data (e.g., as described herein). A block diagram of an example AI system 1500 is shown in FIG. 15.

[0192] Process 1600 of AI system 1500 is shown in Figure 16. Inputs to system 150 include a test image and a visual field image. AI system 1500 estimates which geometric transformation would best be performed on the test image to allow it to present more detail across the visual field. AI system 1500 then estimates the best translation for the test image so that the displayed image covers most of the visual field. The test image is then transformed and translated, as shown in Figures 17 and 18, respectively. Finally, this image is combined again with the visual field for training purposes only, but is displayed as is to the patient during the testing phase. Figure 19 shows a screenshot of a graphical user interface presenting a summary of the visual field analysis, including an example of the final implementation of the visual field AI system, including the image transformation and translation parameters to be applied to the image.

[0193] In an example implementation, a machine learning framework ("AI system") is implemented in a vision correction framework using an artificial neural network model. The AI ​​system acquires a visual field image converted into a vector. As an output, the AI ​​system provides predicted image transformation and translation parameters to be applied to the scene image. These parameters are then used to manipulate the scene image. The AI ​​system has two hidden layers, each with three neurons (i.e., units) and an output layer. An example of such an AI system model is shown in FIG. 20. In other examples, this AI system may be extended to a convolutional neural network model to achieve even more accurate results. FIGS. 21 and 22 show, respectively, an example process 2100, which is the result of applying a neural network to an inspection mode, and an example process 2200, which is an AI algorithm optimization process using a neural network.

[0194] 1A , upon obtaining feedback related to the set of stimuli (displayed to the user during visual test presentation), feedback related to one or more of the user's eyes, feedback related to the user's environment, or other feedback, the test subsystem 122 may provide this feedback to the predictive model, and the predictive model may be configured based on the feedback. In some embodiments, further feedback may be obtained and provided to the predictive model continuously (e.g., periodically, on a schedule, or based on other automated triggers) to update the predictive model's configuration. As an example, the predictive model's configuration may be updated while one or more images augmented with live image data are displayed to the user.

[0195] In some embodiments, the vision subsystem 124 may monitor one or more eye-related characteristics of the user (e.g., gaze direction, pupil size or response, limbus position, visual axis, optical axis, eyelid position or movement, head movement, or other characteristics) and provide the eye characteristic information to the predictive model during the enhanced presentation of the live image data to the user. Additionally, or instead, the vision subsystem 124 may monitor characteristics related to the user's environment (e.g., environmental light level, environmental temperature, or other characteristics). As an example, based on the eye or environmental characteristic information (e.g., indicative of the monitored characteristics), the predictive model may provide one or more modification parameters or functions to be applied to the live image data to generate an enhanced presentation of the live image data (e.g., presentation to the user of one or more enhanced images derived from the live image data). In one use example, the predictive model may obtain the modification parameters or functions (e.g., stored in memory or one or more databases) based on currently detected eye characteristics or environmental characteristics. In another use example, the predictive model may generate the modification parameters or functions based on currently detected eye characteristics or environmental characteristics.

[0196] In some embodiments, with respect to FIG. 1A , the visualization subsystem 124 may facilitate an enhancement of the user's field of view via one or more dynamic display portions on one or more transparent displays (e.g., based on feedback related to the set of stimuli displayed to the user or other feedback). As an example, the dynamic display portions may include one or more transparent display portions and one or more other display portions (e.g., display portions of a wearable device or other device). In some embodiments, the vision subsystem 124 may cause one or more images to be displayed on the other display portions (e.g., no images may be displayed on the transparent display portions). As an example, a user may be able to see through a transparent display portion of a transparent display but not through other display portions, and instead see image presentations on other display portions of the transparent display (e.g., portions surrounding or adjacent to the transparent display portion). That is, in some embodiments, a dynamic hybrid see-through / opaque display may be used. In this way, for example, one or more embodiments may (i) avoid the bulky size and weight of typical virtual reality headsets, (ii) utilize the user's normal vision (e.g., utilize good central vision when the user has normal central vision but poor peripheral vision, and utilize good peripheral vision when the user has normal peripheral vision but poor central vision), and (iii) mitigate visual confusion that would be caused by typical augmented reality technologies where there is an overlap effect between the see-through scene and the internally displayed scene.

[0197] As one example, live image data may be acquired via the wearable device, and an enhanced image may be generated based on the live image data and displayed on another display portion of the wearable device (e.g., a portion of the wearable device's display that meets an opacity threshold or does not meet a transparency threshold). In some embodiments, the vision subsystem 124 may monitor one or more changes associated with one or more eyes of the user and cause adjustments to the transparent display portion of the transparent display based on the monitoring. As one example, the monitored changes may include eye movement, changes in gaze direction, changes in pupil size, or other changes. One or more positions, shapes, sizes, transparency, brightness levels, contrast levels, sharpness levels, saturation levels, or other aspects of the transparent display portion or other display portions of the wearable device may be automatically adjusted based on the monitored changes.

[0198] 24A , a wearable device 2400 may include a transparent display 2402 dynamically configured to have a transparent peripheral portion 2404 and an opaque central portion 2406, such that light from the user's environment can pass intact through the transparent peripheral portion 2404 but not through the opaque central portion 2406. For a patient with a diagnosed central visual field defect 2306, the dynamic configuration allows such a patient to see an actual, uncorrected view of the environment using their intact peripheral vision, while also being presented with a corrected depiction of the central region on the opaque central portion 2406.

[0199] In another use example, with reference to FIG. 24B , wearable device 2400 may include a transparent display 2402 dynamically configured to have an opaque peripheral portion 2414 and a transparent central portion 2416, such that light from the user's environment can pass intact through the transparent central portion 2416 but not through the opaque peripheral portion 2414. For patients with abnormal peripheral vision, the dynamic configuration described above allows such patients to use their intact central vision to see an actual, uncorrected view of the environment while being presented with a corrected depiction of the peripheral area on the opaque peripheral portion 2414. In each of the use examples described above, with reference to FIGS. 24A and 24B , the position, shape, size, transparency, or other aspect of one or more of transparent displaying portions 2404, 2416 or opaque displaying portions 2406, 2414 may be automatically adjusted based on changes related to one or more of the user's eyes monitored by wearable device 2400 (or other components of system 100). Additionally or alternatively, one or more brightness levels, contrast levels, sharpness levels, saturation levels, or other aspects of the opaque display portions 2406, 2414 may be automatically adjusted based on changes related to one or more eyes of the user monitored by the wearable device 2400. In some cases, for example, the user's pupils and gaze (or other eye characteristics) may be monitored and used to adjust the brightness levels of portions of the opaque display portions 2406, 2414 to dynamically respond to areas of reduced brightness in the user's field of vision (e.g., in addition to or instead of increasing the brightness levels of portions of the enhanced image that correspond to areas of reduced brightness in the user's field of vision).

[0200] 24C , based on a determination of the user's field of view (e.g., including defective field of view portions, healthy field of view portions, etc., as represented by field of view plane 2432), an enhanced image (e.g., as represented by remapped image plane 2434) may be generated as described herein. The enhanced image may be displayed to the user on one or more opaque display portions (e.g., as represented by selectively transparent screen plane 2416) within opaque region 2438 of the display, such that the displayed enhanced image enhances the user's view of the environment through transparent region 2440 of the display.

[0201] In one use example, with reference to FIG. 24C , the selectively transparent screen plane 2436 may be aligned with the other planes 2432 and 2434 via one or more eye-tracking technologies. As an example, an eye-tracking system (e.g., of the wearable device 2400 or another device) may be calibrated to the user to ensure proper image projection according to the user's individual visual field. The eye-tracking system may continuously acquire gaze coordinates (e.g., periodically, according to a schedule, or according to other automated triggers). It may also perform a coordinate transformation to convert the spherical coordinates (θ, φ) of the eye movement into the Cartesian coordinates (x, y) of the display. In this way, the device's controller may determine the center position of the displayed image. The camera image is truncated and shifted to align with the acquired gaze vector direction (e.g., FIG. 24C ). The same Cartesian coordinates may be sent to the selectively transparent screen controller to make transparent the area corresponding to the macular vision in the current gaze direction, enabling the use of central vision. In some cases, the gaze data may be low-pass filtered to remove micro-movements that cause shaking in the image displayed to the user (e.g., micro-movements due to constant movement or drafting that occurs even during fixation, since the eye is never completely still).

[0202] As noted above, in some embodiments, the wearable device may be configured to selectively control the transparency of a display area of ​​a monitor, such as a screen, glass, film, and / or multi-layer media. Figure 23 shows an example process 2300 for implementing testing and viewing modes and the use of a custom reality glasses device, which may use macular (central) versus peripheral vision manipulation.

[0203] In some examples, the custom reality glasses device (e.g., FIGS. 40A-40C) includes transparent glasses that overlay the modified image onto a viewed scene. The glasses may comprise a monitor that includes a screen, the screen having controllable transparency for projecting the image to be displayed. In one example, such a display includes a head-up display. In various embodiments, the custom reality glasses device includes glasses with multiple controllable layers that overlay the modified image onto a viewed scene through the glasses. The layers may be formed of glass, ceramic, polymer, film, and / or other transparent materials and may have a multi-layer configuration. The controllable layers may include one or more electrically controllable layers, for example, whose transparency can be adjusted in one or more portions of the field of view by addressing the pixels. In one embodiment, the glasses may include pixels or cells that are individually addressable (e.g., using current, electric field, or light). The controllable layer may be a layer controlled to adjust the contrast of one or more portions of the field of view, the color filter of each portion, the zoom in / out of each portion, the focus of each portion, or the transparency of the surface of the eyewear device displaying the image to block or allow light from the external environment to pass through at a particular location in the field of view. If there is a portion of the field of view that is manipulated to enhance the subject's vision (e.g., a peripheral vision portion or a macular vision portion or a portion that is part macular and part peripheral), the transparency of that portion of the glasses may be reduced to block the external environment from being seen through that portion of the glasses and allow the patient to more clearly see the manipulated image displayed in that portion of the glasses. In various embodiments, the vision system or custom reality eyewear device may dynamically control the transparency of each region to allow the subject to have a natural view of the external environment as eye movement changes eye direction in addition to head movement. For example, pupil tracking data (e.g., pupil and / or eye tracking) may be used to modify the less transparent portion of the eyeglasses such that the less transparent region translates relative to the subject's eye.

[0204] For example, the transparency of the glasses included in the eyewear device constituting the custom reality glasses may be adjusted, and the adjustment may be controllable to block light from the portion of the field of view where the image modification is performed (e.g., central or peripheral regions). Without such adjustment, the subject would see the manipulated image and perceive their actual field of view in this region superimposed on the image. To achieve this light blocking, the eyewear device may include a photochromic glass layer. Furthermore, the eyewear device may vary the location of the less transparent region of the glasses by measuring eye (pupil) movement using an inward-facing image sensor and compensating based on such movement through processing in a vision correction framework. In one example, the display screen of a monitor includes pixels or cells that include electric ink technology. These pixels or cells may be individually addressable to generate an electric field that changes the configuration of the ink within the cells to change transparency and / or generate display pixels. In an example implementation, FIG. 40A illustrates custom reality glasses 4000 comprised of a frame 4002 and two transparent eyeglass assemblies 4004. 40B and 40C, transparent eyeglass assembly 4004 has embedded therein an electronically controllable correction layer 4006. Correction layer 4006 may be controllable from completely transparent to completely opaque and may be a digital layer capable of generating a corrective image to overlay or replace a portion of the field of view of eyeglasses 4000. Correction layer 4006 may be connected via electrical connection 4008 to an image processor 4010 on frame 4002.

[0205] With specific reference to process 2300 of FIG. 23, in block 2302, test mode data may be received by a vision correction framework, and in block 2304, visual field distortions, defects, aberrations and / or other ocular abnormalities may be determined along with their location.

[0206] For a diagnosed central visual field abnormality 2306, in block 2308, the custom reality glasses device may pass images from the external environment through the glasses and into the user's peripheral vision (e.g., FIG. 24). As shown, the custom reality glasses device 2400 may include a multi-layered glasses viewfinder 2402. A peripheral region 2404 may be transparent to allow light to pass through, allowing the subject to see the actual, unmodified external environment. In block 2312, a central region 2406 of the external environment may be rendered opaque by the glasses device 2400, and a modified representation of the central region may be presented to the user via the display, using modifications such as those shown in FIGS. 13, 14, 17, and 18.

[0207] For a diagnosed peripheral vision abnormality 2308, in block 2314, a central region 2416 (e.g., FIG. 24B) of the external environment is allowed to pass through the transparent portion of the eyewear device 2400, and the transparency of the peripheral region 2414 is altered to block light, such that a modified peripheral region image is displayed within the peripheral region 2414, e.g., using a modifying transform described herein.

[0208] In some embodiments, with respect to FIG. 1A , the visual subsystem 124 can facilitate enhancement of the user's visual field via projection onto selected portions of the user's eyes (e.g., based on feedback related to the set of stimuli displayed to the user or other feedback). As described herein, an enhanced presentation of live image data may be displayed to the user by projecting the enhanced presentation (e.g., a modified image derived from the live image data) into the user's eyes. In addition to or instead of using dynamic display portions on the screen (e.g., so that the user can look through one or more portions of the screen and see the modified live image data displayed on one or more other portions of the screen), the modified image data can be projected onto one or more portions of the user's eyes (e.g., one or more portions of the user's retina) while avoiding the projection of the modified image data onto one or more other portions of the user's eyes (e.g., one or more other portions of the user's retina).

[0209] In some embodiments, the modified image data can be projected onto one or more healthy visual field portions of the user's eye while simultaneously avoiding the projection of the modified image data onto one or more other healthy visual field portions of the user's eye. As an example, for the other healthy visual field portions where the projection of the modified image data is avoided, light from the user's environment can pass through the user's retina (e.g., without significant interference from light emitted by a projector), thereby allowing the user to see the environment through such other healthy visual field portions. On the other hand, for the healthy visual field portions where the modified image data is projected, the projected light prevents the user from seeing the environment through the healthy, projected portions of the user's visual field. Nevertheless, by projecting the modified live image data onto the healthy visual field portions of the user's eye, the system allows the modified live image data to be used to enhance the user's visual field (e.g., in a manner similar to the use of dynamic display portions to enhance the user's visual field).

[0210] In some embodiments, the visual subsystem 124 may monitor one or more changes associated with one or more eyes of a user and, based on the monitoring, perform adjustments of one or more projection portions of the projector (e.g., portions including laser diodes, LED diodes, etc., emitting light at a threshold visible to the user's eyes). As an example, monitored changes may include eye movement, changes in gaze direction, changes in pupil size, etc., as well as adjustments of dynamic display portions on a screen. One or more positions, shapes, sizes, brightness levels, contrast levels, sharpness levels, saturation levels, or other aspects of the projection portions or other portions of the projector may be automatically adjusted based on the monitored changes.

[0211] In one use example, the wearable device may include a projector configured to selectively project an enhanced presentation (e.g., a modified image derived from live image data) onto one or more portions of the user's eye (e.g., one or more portions of each of the user's retinas corresponding to the user's normal field of vision), while simultaneously avoiding projecting the modified image data onto one or more other portions of the user's eye (e.g., one or more other portions of each of the user's retinas corresponding to the user's normal field of vision). In some cases, alignment of such selective projection planes may be aligned with other planes (e.g., a field of view plane, a remapped image plane, etc.) via one or more eye-tracking techniques (e.g., one or more techniques similar to those described in FIG. 24C with respect to the use of dynamic display portions on a screen).

[0212] 24A , a wearable device 2400 may include a transparent display 2402 dynamically configured to have a transparent peripheral portion 2404 and an opaque central portion 2406, such that light from the user's environment can pass intact through the transparent peripheral portion 2404 but not through the opaque central portion 2406. For patients with diagnosed central visual field defects 2306, the dynamic configuration allows such patients to see an actual, uncorrected view of their environment using their intact peripheral vision while being presented with a corrected representation of the central region on the opaque central portion 2406.

[0213] In another use example, with reference to FIG. 24B , wearable device 2400 may include a transparent display 2402 dynamically configured to have an opaque peripheral portion 2414 and a transparent central portion 2416, such that light from the user's environment can pass intact through the transparent central portion 2416 but not through the opaque peripheral portion 2414. For patients with abnormal peripheral vision, the dynamic configuration described above allows such patients to use their intact central vision to see an actual, uncorrected view of the environment while being presented with a corrected depiction of the peripheral area on the opaque peripheral portion 2414. In each of the use examples described above, with reference to FIGS. 24A and 24B , the position, shape, size, transparency, or other aspect of one or more of transparent displaying portions 2404, 2416 or opaque displaying portions 2406, 2414 may be automatically adjusted based on changes related to one or more of the user's eyes monitored by wearable device 2400 (or other components of system 100). Additionally or alternatively, one or more brightness levels, contrast levels, sharpness levels, saturation levels, or other aspects of the opaque display portions 2406, 2414 may be automatically adjusted based on changes related to one or more eyes of the user monitored by the wearable device 2400. In some cases, for example, the user's pupils and gaze (or other eye characteristics) may be monitored and used to adjust the brightness levels of portions of the opaque display portions 2406, 2414 to dynamically respond to areas of reduced brightness in the user's field of vision (e.g., in addition to or instead of increasing the brightness levels of portions of the enhanced image that correspond to areas of reduced brightness in the user's field of vision).

[0214] In some embodiments, the testing subsystem 122 may monitor one or more eye-related characteristics associated with the user's eyes during visual test presentation via two or more user interfaces (e.g., on two or more displays) and determine visual defect information for one or more of the user's eyes based on the eye-related characteristics occurring during the visual test presentation. As an example, the testing subsystem 122 may present one or more stimuli at one or more locations in at least one of the user interfaces and generate visual defect information for the user's eyes based on the one or more eye-related characteristics of the eyes occurring during the stimulus presentation. In some embodiments, an eye deviation measurement may be determined based on the eye-related characteristics (indicated by the monitoring as occurring during the stimulus presentation) and used to provide correction or other enhancement for the eye. As an example, the deviation measurement may indicate a deviation of the eye relative to the other eye, and the deviation measurement may be used to determine and correct diplopia or other visual defects. As an example, the amount of movement indicates the amount of strabismus (e.g., cross-eyes) of the eye, and the direction (or axis) of the movement indicates the type of strabismus. For example, if the eye movement is from "out" to "in," it means the strabismus is directed outward (e.g., exotropia). Thus, in some embodiments, diplopia can be autonomously determined and corrected via a wearable device.

[0215] In some embodiments, the inspection subsystem 122 may determine a deviation measurement or other visual defect information for a user's first eye by (i) presenting a stimulus at a location on a first user interface for the first eye while ensuring that a stimulus intensity on a second user interface for the user's second eye does not meet a stimulus intensity threshold, and (ii) determining the visual defect information based on one or more eye-related characteristics of the first eye that occur during the stimulus presentation. As an example, the stimulus presentation on the first user interface may occur while no stimulus is being presented on the second user interface. In one use example, if a first eye (e.g., the right eye) has exotropia immediately before the presentation of such a stimulus on the first user interface (e.g., FIG. 25D ), presenting the stimulus in front of only the first eye (e.g., the right eye) will cause the second eye (e.g., the left eye) to lose its advantage as a result of having nothing to look at, and the first eye will instinctively move toward and fixate the stimulus location (e.g., within one second). The testing subsystem 122 may measure the corrective movements of the first eye (and other changes in eye-related characteristics of the first eye) to determine a deviation measure for the first eye. As an example, the amount of first eye movement that occurs during such stimulus presentation may correspond to the amount of strabismus of the first eye.

[0216] In some embodiments, the inspection subsystem 122 may determine a deviation measurement or other visual defect information for a user's first eye by (i) presenting stimuli at a corresponding position on a first user interface for the first eye and a corresponding position on a second user interface for the second eye at a predetermined time, and (ii) determining the visual defect information based on one or more eye-related characteristics of the first eye that occur during the stimulus presentation. As an example, a target stimulus may be presented at a center position on both user interfaces or at another corresponding position on both user interfaces. In one use case, when stimuli are presented in front of both eyes (e.g., FIG. 25B ), the dominant eye (e.g., the left eye in FIG. 25B ) instinctively moves to the corresponding position and fixates the target stimulus (e.g., within one second). The other eye (e.g., the right eye in FIG. 25B ) also moves, but because this eye has exotropia, it does not instinctively fixate the target stimulus, resulting in double vision for the user. For example, the other eye instinctively moves, but that instinctive movement causes the gaze direction of the other eye to be directed to a different position. However, as the user focuses on viewing the target stimulus with the other eye, the other eye moves and fixates the target stimulus presented at a corresponding position on the user interface for the other eye. Because the target stimuli are presented at corresponding positions on both user interfaces, the dominant eye remains dominant and continues to fixate the target stimulus presented at the corresponding position on the user interface for the dominant eye. The testing subsystem 122 may measure the corrective movement of the other eye (and other changes in the eye-related characteristics of the other eye) to determine a deviation measurement for the other eye (e.g., the amount of movement of the other eye may correspond to the amount of strabismus of the other eye).

[0217] In some embodiments, after obtaining a deviation measurement or other visual deficiency information for a user's first eye by measuring a change in an eye-related characteristic of the first eye (e.g., a first eye movement occurring upon presentation of a stimulus at a corresponding position on the first user interface for the first eye), the inspection subsystem may cause the stimulus to be presented at the modified position on the first user interface for display to the first eye. As an example, the stimulus presentation at the modified position occurs while no stimulus is presented on the second user interface for the second eye (or while the stimulus intensity of at least the second user interface does not meet a stimulus intensity threshold such that the second eye does not respond to any stimulus on the second user interface). Based on one or more eye-related characteristics of the first eye or the second eye not changing beyond a change threshold upon presentation at the modified position, the inspection subsystem 122 may determine the deviation measurement or other visual deficiency information for the first eye. As an example, the deviation measurement for the first eye may be determined based on the first eye not moving beyond a movement threshold (e.g., no movement or other movement threshold) upon presentation of the stimulus at the modified position. Additionally or alternatively, the deviation measurement of the first eye may be determined based on the second eye not moving beyond its motor threshold.

[0218] In some embodiments, the inspection subsystem 122 may generate one or more correction profiles associated with the user based on one or more deviation measurements or other vision defect information (e.g., obtained via one or more visual test presentations) of one or more eyes of the user. As an example, each of the correction profiles may include correction parameters or functions used to generate an enhanced image from live image data (e.g., parameters of a function used to transform or correct live image data into an enhanced image). Thus, in some embodiments, the vision subsystem 124 may generate corrected video stream data to be displayed to the user based on (i) video stream data representative of the user's environment and (ii) the correction profile associated with the user.

[0219] As an example, a vision test may be performed to determine whether a deviation in the user's eyes exists, to measure the deviation in the user's eyes, or to generate one or more correction profiles for the user's eyes. In one use example, with reference to FIG. 25A , when a target stimulus 2502 is presented to a patient (e.g., a patient without strabismus) at a central position on the right and left displays 2503 a and 2503 b of a wearable device, both eyes (e.g., right and left eyes 2504 a and 2504 b) instinctively move to fixate the target stimulus 2502 at the central position on each wearable display, and thus the patient sees only one target stimulus 2502. In this manner, based on the above-described eye responses, the test subsystem 122 may determine that the user does not have double vision.

[0220] In another use example, with reference to FIG. 25B , when a target stimulus 2502 is presented to a patient with strabismus at a central position between the left and right displays of a wearable device, one eye (e.g., a dominant eye) instinctively moves to the central position and fixates on the target stimulus 2502 (e.g., left eye 2504b instinctively fixates on target stimulus 2502). The other eye (e.g., right eye 2504a) also moves, but because this eye has exotropia, it does not fixate on the target stimulus 2502, thereby causing the user to see double (e.g., the user sees two target stimuli instead of one). For example, the other eye instinctively moves, but the instinctive movement causes the other eye's gaze direction to be directed to a different position. Based on the above eye responses, the testing subsystem 122 may determine that the user has double vision. However, in a further use case, as the user focuses on viewing the target stimulus 2502 with the other eye (e.g., right eye 2504a, which has strabismus), the other eye moves to fixate the target stimulus 2502 presented in a central position on the other eye's user interface. Because the target stimulus 2502 is presented in a central position on both displays 2503a and 2503b, the dominant eye remains dominant and continues to fixate the target stimulus 2502 presented in a central position on the dominant eye's display. The corrective movement of the other eye (and other changes in the eye-related characteristics of the other eye) may be measured to determine a deviation measure for the other eye (e.g., the amount of movement of the other eye may correspond to the amount of strabismus of the other eye).

[0221] In another use example, with reference to FIG. 25C , at time t1, a stimulus (e.g., target stimulus 2502) may be presented to only left eye 2504b at a central position by presenting a stimulus to left display 2503b but not to right display 2503a. For example, if stimuli are presented to both eyes 2504a and 2504b at a central position as shown in FIG. 25B immediately before presenting a stimulus to only left eye 2504b (e.g., at time t0 immediately before presenting the stimulus at time t1), left eye 2504b will not move because it is already fixating the central position. However, if left eye 2504b has not yet fixated the central position, presenting a stimulus to only left eye 2504b will cause left eye 2504b to instinctively move to the central position and fixate target stimulus 2502.

[0222] As shown in FIG. 25D , a stimulus (e.g., target stimulus 2502) may be presented to the right eye 2504a only (e.g., at time t2) at a central position by presenting a stimulus to the right display 2503a and not to the left display 2503b. Because the left eye 2504b is not receiving the stimulus (e.g., there is nothing to look at), the left eye 2504b loses dominance and the right eye 2504a becomes dominance, causing the left eye 2504b to move outward. When the target stimulus 2502 is presented only to the right eye 2504a, the right eye 2504a instinctively becomes dominant and moves to fixate at a central position. The testing subsystem 122 may measure the movement of the right eye 2504a to determine a deviation measurement for the right eye 2504a (e.g., the amount of movement may correspond to the amount of strabismus of the right eye 2504a).

[0223] As shown in FIG. 25E, a stimulus (e.g., target stimulus 2502) may be presented to both eyes 2504a, 2504b at a central location (e.g., at time t3) by presenting the stimulus on left display 2503b and right display 2503a. In the case of alternating strabismus (neither eye is dominant), right eye 2504a remains fixated on the central location, while left eye 2504b remains strabismus. However, if left eye 2504b is dominant (as shown in FIG. 25E), left eye 2504b instinctively moves to fixate on the central location. This movement of left eye 2504b causes right eye 2504a to strabismus, which results in the gaze direction of right eye 2504a being directed to a different location. The inspection subsystem 122 may measure the movement of the left eye 2504b to determine or confirm the deviation measurement of the right eye 2504a (e.g., the amount of movement of the left eye 2504b may correspond to the amount of deviation of the right eye 2504a).

[0224] In a further use case, further testing may be performed to confirm the deviation measurement of the non-dominant eye. For example, as shown in FIG. 25F, following one or more of the aforementioned steps described with respect to FIGS. 25B-25E, a stimulus (e.g., target stimulus 2502) may be presented to only the left eye 2504b (e.g., at time t4) in a central position by presenting a stimulus on the left display 2503b and not on the right display 2503a. To the extent that the left eye 2504b loses fixation (e.g., due to the presentation of FIG. 25E), the presentation of FIG. 25F will cause the left eye 2504b to instinctively move to obtain fixation at the central position. This movement of the left eye 2504b will cause the right eye 2504a to squint, resulting in the gaze direction of the right eye 2504a being directed toward a different position. As shown in FIG. 25G, a corrective position for presenting the stimulus to the right eye 2504a may be determined based on the deviation measurement of the right eye 2504a. In this way, the target stimulus 2502 may be presented in a modified position on the right display 2503a (e.g., at time t5) while the target stimulus 2502 is presented in a central position on the left display 2503b.

[0225] Thereafter, with reference to FIG. 25H, the target stimulus 2502 may be presented only to the right eye 2504a (e.g., at time t6) by presenting the target stimulus 2502 at the corrected position on the right display 2503a but not on the left display 2503b. Specifically, for example, the target stimulus 2502 may be shifted to the right by an amount equal to the deviation measured in one or more of the preceding steps described with reference to FIGS. 25B-25E. If the deviation measurement is accurate, the right eye 2504a does not move. If the deviation measurement is inaccurate, the right eye 2504a moves slightly, and the amount of that movement may be measured by the wearable device (e.g., a pupil tracker in the wearable device), and this measurement of the slight movement may be used to fine-tune the deviation. By way of example, the measurement and the corrected position may be used to determine an updated corrected position for presenting the stimulus to the right eye 2504a, and one or more of the steps described with reference to FIGS. 25F-25H may be repeated using the updated corrected position. Additionally, or alternatively, one or more of the steps of Figures 25B-25E may be repeated to redetermine the deviation measurement for one or more of the user's eyes (e.g., redetermine the deviation measurement for right eye 2504a). With reference to Figure 25I, target stimulus 2502 may then be presented to both eyes 2504a and 2504b (e.g., at time t7) by presenting target stimulus 2502 in a corrected position on right display 2503a and in a central position on left display 2503b. Because target stimulus 2502 in front of right eye 2504a is deviated to the right according to the deviation measurement (e.g., determined or confirmed in one or more of the foregoing steps), the user will not see double, thereby providing an autonomous correction for the patient's diplopia.

[0226] In some embodiments, a vision test may be performed to determine which eye of the user is the deviated eye. Based on such determination, the deviation of the deviated eye may be measured, and the deviation measurement may be used to generate a correction profile to correct the deviation in the user's vision. As an example, the test subsystem 122 may present a stimulus at a first location on a first user interface for a first eye and at a first location on a second user interface for a second eye at a predetermined time. The test subsystem 122 may detect that the first eye is not fixating at the first location upon presentation of the stimulus on the first user interface. Based on detecting that the first eye is not fixating, the test subsystem 122 may determine that the first eye of the user is the deviated eye. 25B , when a target stimulus 2502 is presented to a patient with strabismus at a central position between the left and right displays of a wearable device, one eye (e.g., the dominant eye) instinctively moves to the central position and fixates on the target stimulus 2502 (e.g., left eye 2504b instinctively fixates on target stimulus 2502). The other eye (e.g., right eye 2504a) also moves, but because this eye has exotropia, it does not fixate on the target stimulus 2502, thereby causing double vision to the user (e.g., the user sees two target stimuli instead of one). Based on detecting this lack of fixation, it may be determined that the other eye is the deviated eye.

[0227] In some embodiments, a visual test may be performed while the eyes are looking in different gaze directions to detect the degree of diplopia in each gaze direction. In this way, diagnosis and correction of certain types of strabismus (e.g., non-interfering strabismus) can be performed. For example, a patient with paralyzed eye muscles will experience a greater deviation (angle of strabismus) between the eyes when looking in the direction of the affected eye muscle. For example, if the muscles that turn the left eye out are paralyzed, the left eye will look inward (commonly known as esotropia). When the left eye tries to look outward, the degree of strabismus increases. This is the case with paralytic strabismus. The wearable device (or other components connected to the wearable device) can accurately measure the degree of deviation by repeating the quantification test while presenting stimuli in different regions of the visual field. Furthermore, knowing the degree of deviation in different gaze directions allows for dynamic correction of diplopia. Such visual test presentation may be provided via a wearable device, and when the pupil tracking device of the wearable device detects that the eye is at a particular gaze point, the wearable device may displace the image to correspond to that gaze point.

[0228] In some embodiments, such testing can be performed while the patient is looking at a distant object and while looking at a nearby object. In some embodiments, the wearable device can automatically test the range of motion of the extraocular muscles by presenting a moving stimulus that the patient follows with their eyes, and the wearable device (or other component connected to the wearable device) measures the range of motion and determines information about the user's diplopia based on the range of motion measurements.

[0229] Thus, in some embodiments, multiple correction profiles can be generated for a user to correct for dynamic vision defects (e.g., diplopia or other vision defects). As an example, a first correction profile associated with a user may include one or more correction parameters applied to correct an image for a first eye of the user in response to the second eye's gaze direction being directed to a first position, the second eye having a first torsion (e.g., a first torsion angle), or other characteristics of the second eye. A second correction profile associated with a user may include one or more correction parameters applied to correct an image for a first eye in response to the second eye's gaze direction being directed to a second position, the second eye having a second torsion (e.g., a second torsion angle), or other characteristics of the second eye. The third correction profile associated with the user may include one or more correction parameters applied to correct the image for the first eye depending on, for example, the gaze direction of the second eye being directed to a third position, the second eye having a third torsion (e.g., a third torsion angle), or other characteristics of the second eye. And so on. In one use example, one or more of the steps described with respect to Figures 25B-25H may be repeated for one or more other positions (in addition to or instead of the center position) to generate multiple correction profiles for the user.

[0230] In some embodiments, the vision subsystem 124 may monitor one or more eye-related characteristics of one or more eyes of a user and may generate modified video stream data to be displayed to the user based on (i) video stream data representative of the user's environment, (ii) the monitored eye-related characteristics, and (iii) a modification profile associated with the user. As one example, if the monitoring indicates that the gaze direction of a second eye is directed toward a first location, the video stream data may be modified using a first modification profile (e.g., its modification parameters) to generate modified video stream data to be displayed to the first eye of the user. As another example, if the monitoring indicates that the gaze direction of a second eye is directed toward a second location, the video stream data may be modified using a second modification profile (e.g., its modification parameters) to generate modified video stream data for the first eye of the user. And so on. In this manner, for example, the foregoing description accounts for the typically dynamic nature of diplopia (e.g., diplopia increasing or decreasing toward one or more lines of sight). For example, if there is a problem with moving a user's right pupil away from the user's nose (e.g., toward the edge of the user's face), the user's diplopia may increase when the user looks to the right and decrease when the user looks to the left. Thus, by monitoring the user's pupil, the user's gaze, or other eye-related characteristics, appropriate corrections can be made by applying an appropriate correction profile specific to the user's real-time eye-related characteristics to the live video stream data.

[0231] In some embodiments, a vision test may be performed to assess a user's binocular vision. In some embodiments, a binocular vision test may be performed using a wearable device. As an example, one or more stimuli may be presented on a user interface of each wearable device display intended for the user's eyes, where the number or type of stimuli presented on one user interface differs from the number or type of stimuli presented on the other user interface (e.g., the number of stimuli on each user interface may be different, or at least one stimulus on one user interface may have a different color or pattern than the stimuli on the other user interface). Alternatively, in some scenarios, the number or type of stimuli presented on both user interfaces is the same. The test subsystem 122 may determine whether the user has double vision based on the user's indication of the number or type of stimuli the user is viewing.

[0232] In one use example, with reference to FIG. 25J, binocular vision testing may include a user wearing a wearable device having displays 2522a and 2522b (or viewing these displays 2522a and 2522b via another device), with each display 2522 configured to present one or more stimuli or other presentations to a respective eye of the user. As an example, stimuli 2524a and 2524b (e.g., green dots) may be presented to one eye of the user on display 2522a, and stimuli 2526a, 2526b, and 2526c (e.g., red dots) may be presented to the other eye of the user on display 2522b. With reference to FIG. 25K, testing subsystem 122 may determine that the user is seeing binocular monovision (and therefore does not have diplopia) based on the user's indication that they are seeing four dots. Additionally or alternatively, the testing subsystem 122 may determine or confirm that the user is seeing binocular monocular vision based on a user indication that the user is viewing one green dot (e.g., stimulus 2524a), two red dots (e.g., stimuli 2526a and 2526c), and one mixed-color dot (e.g., mixed stimulus 2528 combining stimuli 2524b and 2526b). Meanwhile, with respect to FIG. 25L, the testing subsystem 122 may determine that the user has double vision (e.g., diplopia) based on a user indication that the user is viewing five dots. Additionally or alternatively, the testing subsystem 122 may determine or confirm that the user has diplopia based on a user indication that the user is viewing two green dots (e.g., stimuli 2524a and 2524b) and three red dots (e.g., stimuli 2526a, 2526b, and 2526c).

[0233] In some embodiments, the testing subsystem 122 may monitor one or more eye-related characteristics associated with the user's eyes during visual test presentation via two or more user interfaces (e.g., on two or more displays) and autonomously determine whether the user has diplopia based on the eye-related characteristics occurring during the visual test presentation. In some embodiments, the testing subsystem 122 may determine the degree of the user's diplopia based on such eye-related characteristics (e.g., by measuring one or more eye deviations as described herein) and autonomously generate one or more correction profiles to correct the diplopia. As an example, the wearable device may include a pupil and eye-tracking device to detect the gaze direction of one or more of the user's eyes or other eye-related characteristics. Based on the gaze direction (or other eye-related characteristics), the testing subsystem 122 may determine the number of points the user fixated (e.g., by using the detected gaze direction to determine whether the user fixated on a location corresponding to the presented stimulus). In one use example, with respect to Figure 25J, if it is determined that the user has fixated four points (e.g., points corresponding to stimuli 2524a, 2526a, 2526c, and 2528 shown in Figure 25K), the inspection subsystem 122 may determine that the user does not have double vision. If it is determined that the user has fixated five points (e.g., points corresponding to stimuli 2524a, 2524b, 2526a, 2526b, and 2526c shown in Figure 25L), the inspection subsystem 122 may determine that the user has double vision.

[0234] As a further example, in response to determining that the user has fixated a particular point (e.g., a point corresponding to the presented stimuli or their respective display positions), the test subsystem 122 may mitigate the influence of the corresponding stimulus and increase the count of the number of stimuli viewed by the user. As one example, the corresponding stimulus may be removed from the visual test presentation (e.g., the corresponding stimulus may disappear while the remaining stimuli continue to be presented) or may be modified to reduce its influence (e.g., by reducing the brightness or other intensity level of the stimulus). As another example, other stimuli may be altered to increase their influence (e.g., by increasing the brightness or other intensity level of the other stimuli), thereby reducing the relative influence of the corresponding stimulus. In this manner, the user's eyes instinctively move to and fixate on one or more points corresponding to the remaining stimuli. With reference to FIG. 25K, for example, stimuli 2524b and 2526b (represented by mixed stimulus 2528) are removed when the user's eyes fixate on the positions corresponding to stimuli 2524b and 2526b. 25L (when the user has diplopia), stimuli 2524b and 2526b are removed at two different times because the user does not fixate on the same relative location when looking at stimulus 2524b or 2526b. The inspection subsystem 122 may continue to remove stimuli and increment the count (of the number of stimuli seen by the user) each time the user fixates on the corresponding point. When all stimuli have been removed or another threshold is met, the inspection subsystem 122 may provide the number of stimuli seen by the user.

[0235] In some embodiments, based on eye-related characteristics occurring during the visual test presentation, the test subsystem 122 may determine whether the user has stereopsis or the degree of the user's stereopsis. As an example, the test subsystem 122 may cause one or more stimuli to be presented at one or more locations on one or more user interfaces and autonomously perform such determination of stereopsis or other visual deficiency information based on the eye-related characteristics. In one use example, with reference to FIG. 25M, the visual test presentation may include the user wearing a wearable device having displays 2542a and 2542b (or viewing these displays 2542a and 2542b via another device), each display 2542 configured to present one or more stimuli or provide other presentations to a respective eye of the user.

[0236] As shown in FIG. 25M , one or more icons 2544 or other stimuli may be presented on each display 2542, where one or more pairs of icons 2544 are presented in corresponding positions on both displays 2542, and at least one pair of icons 2544 is presented in slightly different positions on displays 2542a and 2542b. Notably, in FIG. 25M , the arrangement of icons 2544 on both displays 2542 is the same, except that icon 2544 in the second row and third column on display 2542b is shifted slightly up and to the right (as indicated by indicator 2546). For a user without binocular diplopia or stereopsis, this slight difference causes the icon pair to appear as a three-dimensional icon to the user, and all other icons 2544 to appear as two-dimensional icons to the user. As such, the user will instinctively shift their gaze to and fixate on the three-dimensional icon. Based on a determination that the individual fixates the three-dimensional icon (e.g., within a predetermined threshold time), the testing subsystem 122 may determine that the user does not have stereopsis. As an example, the testing subsystem 122 may detect that the gaze direction of one or more of the user's eyes changed during stimulus presentation and is now directed toward an area where a corresponding icon 2544 is presented on the respective display 2542.

[0237] However, if the user has stereoscopic vision, slight differences may prevent the pair of icons from appearing as three-dimensional icons to the user, and the user will likely not fixate on the corresponding regions where the pair of icons are presented on their respective displays 2542. Based on this lack of fixation (e.g., within a predetermined threshold time), the testing subsystem 122 may determine that the user has stereoscopic vision.

[0238] In a further use example, with respect to FIG. 25M, the amount of difference between two icons 2544 in the second row and third column may be modified to determine the user's degree of stereopsis. As an example, icon 2544 (within the area indicated by indicator 2546) may first be shifted up or to the right so that the difference in position between icon 2544 on display 2542b and its corresponding icon 2544 on display 2542a is minimized. If the user does not fixate the corresponding area where the pair of icons is presented, icon 2544 on display 2542b may again be shifted up or to the right so that the difference in position between the two icons 2544 is slightly larger. The difference in position may be iteratively increased until the user fixates the corresponding area or until a threshold value for the difference in position is reached. The inspection subsystem 122 may use the amount of difference in position (or the number of times the shift operation is performed) to measure the user's degree of stereopsis.

[0239] In another use example, with reference to FIG. 25N, stimulus presentation during visual test presentation may be provided in the form of randomly generated noise. In FIG. 25N, the stimuli presented on display 2562a and the stimuli presented on display 2562b are identical, except that the set of blocks (e.g., pixels) within the area indicated by indicator 2564 is shifted to the right by 5 units (e.g., pixels) on display 2562b (compared to the same set of blocks on display 2562a). As with the previous use example with reference to FIG. 25M, this slight difference causes the collection of blocks to appear (or stand out) as a solid object to a user without binocular diplopia and stereopsis, resulting in the user quickly fixating the solid object. Based on a determination that the individual fixated the solid object, the test subsystem 122 may determine that the user does not have stereopsis. However, if the user has stereopsis, the slight difference may prevent the user from noticing the set of blocks, and the user will not fixate the corresponding area where the set of blocks is presented on each display 2562. Based on this lack of fixation, the testing subsystem 122 may determine that the user has stereopsis.

[0240] In some embodiments, with reference to FIG. 1A , the visual subsystem 124 may facilitate an expansion of the user's field of view through the combination of portions of multiple images of a scene (e.g., based on feedback associated with a set of stimuli displayed to the user). As an example, FIG. 26 illustrates a subject's normal binocular vision. In this case, a monocular image from the left eye 2602 and a monocular image from the right eye 2604 are combined to form a single perceived image 2606 that includes a central macular region 2608 and a peripheral vision region 2610 surrounding the central region 2608. However, in some cases, the subject may experience tunnel vision, in which case the peripheral region 2610 is invisible to the subject, as shown in FIG. 27 . As shown, in such cases, one or more objects do not appear in the field of view, a peripheral defect 2612 is observed in region 2610, and objects within region 2610 are invisible to the subject. Thus, for example, the visual subsystem 124 may combine portions of multiple images of a scene (e.g., common and different regions of such images) to expand the subject's field of view.

[0241] In some embodiments, the vision subsystem 124 may acquire multiple images of a scene (e.g., images obtained via one or more cameras at different positions or orientations). The vision subsystem 124 may determine areas common to the multiple images and, for each image of the multiple images, determine areas of the image that differ from corresponding areas in at least one other image of the multiple images. In some embodiments, the vision subsystem 124 may generate or display to a user an enhanced image based on the common areas and the difference areas. As an example, the common areas and difference areas may be combined to generate an enhanced image that includes a representation of the common areas and a representation of the difference areas. The common areas may correspond to portions of each of the multiple images that have the same or similar characteristics as one another, and each difference area may correspond to a portion of one image that differs from all corresponding portions of the other images. In some scenarios, the different portions of one image may include parts of the scene not represented in the other images. In this way, for example, the common areas and difference areas may be combined into an enhanced image to expand the field of view otherwise provided by each image, and the enhanced image may be used to enhance a user's field of view. In one use case, the common region may be any portion of at least one of the left-eye 2602 image or the right-eye 2604 image between any two of the four vertical dotted lines shown in each of these images in Figure 27. In another use case, with respect to Figure 27, one of the difference regions may be any portion of the left-eye 2602 image to the left of the left-most vertical dotted line in that image. Another of the difference regions may be any portion of the right-eye 2604 image to the right of the right-most vertical dotted line in that image.

[0242] In some embodiments, the common region is a region in at least one of the images corresponding to a macular region of the eye's visual field (or other central region of the eye's visual field) or a region within the macular region. In some embodiments, each of the difference regions is a region in at least one of the images corresponding to a peripheral region of the eye's visual field or a region within the peripheral region. As an example, with respect to FIG. 27 , the common region may be (i) a portion of the image corresponding to the macular region of the left eye 2602 or (ii) a portion of the image corresponding to the macular region of the right eye 2604 (e.g., assuming those portions are common to both images). As another example, the common region may be a portion of each of the images corresponding to a common region within the macular regions of the left eye 2602 and the right eye 2604. As a further example, based on the common region and the difference regions, image 2606 is generated to have a macular central region 2608 and a peripheral vision region 2610 surrounding central region 2608.

[0243] In some embodiments, the vision subsystem 124 may determine areas common to multiple images of a scene (e.g., captured via a user's wearable device) and, for each of the images, determine areas of the image that differ from corresponding areas in at least one other of the images. The vision subsystem 124 may perform a shift of each of the images and, following the shift, generate an enhanced image based on the common and difference areas. In some embodiments, the shift of each image may be performed such that (i) the size of the common area is modified (e.g., increased or decreased) or (ii) the size of at least one of the difference areas is modified (e.g., increased or decreased). In one scenario, the size of the common area may increase as a result of the shift. In another scenario, the size of at least one of the difference areas decreases as a result of the shift.

[0244] As an example, the defects shown in FIG. 27 may be corrected using a shift image correction technique. In one use case, with respect to FIG. 28, two field-of-view cameras (e.g., of a wearable device) may each capture monocular images 2802 and 2804 (e.g., each monocular image is different because it captures a scene viewed from a slightly different (shifted) position). The captured monocular images 2802, 2804 are then shifted closer to each other using a vision correction framework to obtain images 2802′ and 2804′. As shown in FIG. 28, the area (e.g., common area) between the leftmost and rightmost vertical dotted lines of each image 2802, 2804 is larger than the area (e.g., common area) between the leftmost and rightmost vertical dotted lines of each image 2802′, 2804′. Thus, the common area decreases in size after the shift, while the difference area increases in size after the shift (e.g., compare the area to the left of the leftmost vertical dotted line in image 2802 with the area to the left of the leftmost vertical dotted line in image 2802', and compare the area to the right of the rightmost vertical dotted line in image 2804 with the area to the right of the rightmost vertical dotted line in image 2804').

[0245] As a further example, these two shifted images are then combined to generate a binocular image 2806 that captures the entire periphery of the viewed scene. For eyewear devices with monitor displays, each display may display the corrected binocular image 2806 to the subject. In some use cases, for example, this shift transformation may increase the subject's field of view by 5%, 10%, 15%, 20%, or more without causing the subject to experience diplopia.

[0246] In some embodiments, the vision subsystem 124 may determine areas common to multiple images of a scene (e.g., captured via a user's wearable device) and, for each of the images, determine areas of the image that differ from corresponding areas in at least one other of the images. The vision subsystem 124 may resize one or more of the regions of the images and, following resizing, generate an enhanced image based on the common and difference regions. In some embodiments, the vision subsystem 124 may resize one or more regions of the images such that any range of resizing of the common region differs from any range of resizing of at least one of the difference regions. In some embodiments, the resizing may be performed such that a rate of change in size of the common region represented in a first region of the enhanced image is greater than or less than a rate of change in size of at least one of the difference regions represented in a second region of the enhanced image. As one example, the rate of change in size of at least one of the difference regions may be zero and the rate of change in size of the common region may be greater than zero. As another example, the rate of change in size of at least one of the difference regions may be greater than zero and the rate of change in size of the common region may be zero.

[0247] In one scenario, with reference to FIG. 29 , captured monocular images 2902 and 2904 are resized, for example, only in the peripheral region while leaving the central macular region (the central 20 degrees) intact, to generate modified images 2902′, 2904′. This resizing transformation maintains central vision while expanding the field of view. As shown in FIG. 29 , the combined binocular image 2906 captures previously invisible peripheral objects while maintaining the detail of the central macular region. The peripheral objects are still clearly perceived by the subject even after resizing. This is because peripheral vision is not as sensitive as central vision. In some use cases, for example, a reduction in image size of up to 20% can be achieved without causing the subject to experience diplopia. In various embodiments, resizing the peripheral region can be performed in addition to or instead of resizing the central region. For example, the size of the central macular region can be maintained while the peripheral region is resized to the size of the peripheral region (e.g., for glaucoma patients). In another scenario, for a patient with macular degeneration, the central region may be resized to reduce its size while leaving the peripheral vision intact (e.g., unresized), and an enhanced image (e.g., binocular image) can then be generated to include the resized central region.

[0248] In some embodiments, the vision subsystem 124 may determine areas common to multiple images of a scene (e.g., captured via a user's wearable device) and, for each of the images, determine areas of the image that differ from corresponding areas in at least one other of the images. The vision subsystem 124 may perform a fisheye transformation, a conformal mapping transformation, or other transformation on the common areas and, following performance of the transformation, generate an enhanced image based on the common and difference areas. In some embodiments, the vision subsystem 124 may perform a fisheye transformation, a conformal mapping transformation, or other transformation on areas (including the common areas) of the enhanced image.

[0249] As an example, a fisheye transformation can be performed on the region to modify the radical components of the image according to the following formula: r new =r+αr 3 Here, α is a constant.

[0250] As another example, a conformal mapping transformation can be performed on the region to modify the radial component of the image according to the following equation: r new =rβ where β is a constant exponent for the radial component and β>1.

[0251] In some embodiments, the visual subsystem 124 may modify at least one of the multiple images of a scene by moving one or more objects in the image (e.g., before generating the enhanced image based on the common and different regions of the images). As an example, with reference to FIG. 30, for a patient with a significantly outer peripheral defect in one eye, an object 3002 that is not visible in the affected eye's visual field 3004 can be digitally moved to a mid-peripheral region 3006 of the visual field 3004, while the other, normal eye's visual field 3008 covers this region. In other words, the previously invisible object 3002 appears within the normal visual field in the combined binocular image 3010. The subject will notice visual clutter in this region but can adapt to separating information in this region of the visual field in response to moving objects or a changing environment.

[0252] In some embodiments, the vision subsystem 124 may determine one or more defective field of view portions of the user's field of view (e.g., according to one or more techniques described herein). In some embodiments, the vision subsystem 124 may determine areas common to multiple images of a scene (e.g., captured via the user's wearable device) and, for each of the images, determine areas of the image that differ from corresponding areas in at least one other of the images. The vision subsystem may generate an enhanced image based on the common and difference areas of the images, such that at least one of the common and difference areas in the enhanced image does not overlap with one or more of the defective field of view portions.

[0253] In some embodiments, the vision subsystem 124 may detect objects in defective portions of the user's visual field and display an alert. As an example, after correcting defective portions of the user's visual field (e.g., via one or more techniques described herein), the vision subsystem 124 may monitor the remaining uncorrected areas to detect one or more objects (e.g., safety hazards or other objects) and generate an alert (e.g., a visual or audio alert) indicating the objects, their location, their size, or other information related to the objects. In some use cases, for patients with irregular or multi-regional defective visual fields, the generated correction profile may still not be optimal in matching the acquired visual field to the normal regions of the patient's visual field. Therefore, to maximize patient safety during transport, an automatic video tracking algorithm may be implemented to detect objects in one of the defective portions. Such objects may include moving objects (e.g., moving vehicles) or other objects in the defective portions of the patient's visual field.

[0254] In some embodiments, the vision subsystem 124 may generate a prediction indicating that an object will come into physical contact with the user and cause an alert to be displayed based on the physical contact prediction (e.g., an alert related to the object is displayed on the user's wearable device). In some embodiments, the vision subsystem 124 may detect an object (e.g., located in or predicted to be located in a defective portion of the user's visual field) and cause an alert to be displayed based on (i) the object being located in or predicted to be located in the defective portion of the visual field, (ii) the physical contact prediction, or (iii) other information. In some embodiments, the vision subsystem 124 may determine whether the object is located outside (or sufficiently within) any image portion of the enhanced image (displayed to the user) that corresponds to at least one portion of the visual field that meets one or more vision criteria. In one use case, if an object is determined to be located (or sufficiently within) an image portion of the enhanced image that corresponds to a healthy portion of the user's visual field, no alert (or a lower priority alert) may be displayed (e.g., even if the object is predicted to come into physical contact with the user). On the other hand, if an object in the defective field of view is predicted to come into physical contact with the user and the object is determined to be outside (or not sufficiently within) the user's normal field of view, an alert may be displayed on the user's wearable device. In this way, for example, the user can rely on their own normal field of view to avoid approaching objects within the user's normal field of view, thereby reducing the risk of relying on the wearable device (e.g., through habit formation) to avoid such approaching objects. However, it should be noted that in other use cases, an alert related to an object may be displayed based on the physical contact prediction, regardless of whether the object is within the user's normal field of view.

[0255] 10 , for uncompensated blind spot field 1006, pupil tracking or other visual tracking (e.g., using an inward-facing image sensor) or video tracking of moving objects within the field of view (e.g., using an outward-facing image sensor such as an external camera) may be used to detect safety hazards in or moving into the blind spot area in blocks 1012 and 1014. In one use case, vision subsystem 124 may compare the location of a safety hazard to a defective mapped field of view (e.g., measured in inspection mode) to detect when the safety hazard is in a blind spot area or when the safety hazard is moving into such an area.

[0256] As another example, after correcting defective visual field portions of a user's visual field (e.g., via one or more techniques described herein), the vision subsystem 124 may monitor the remaining uncorrected areas to detect (e.g., in real time) safety hazards approaching the user from such areas. If such detected safety hazards are predicted to come into physical contact with the user or to come within a threshold distance (e.g., one foot, two feet, or other threshold distance) of the user (as opposed to passing near the user at or above the user's threshold distance), the vision subsystem 124 may generate an alert (e.g., a visual alert displayed in an area visible to the user, an audible alert, etc.) related to the detected safety hazard.

[0257] In one use case, video signals (e.g., live video streams) acquired from one or more cameras of a user's wearable device are preprocessed and filtered to remove residual noise effects. In one use case, a search area may be limited to a user's blind spot or other defective portion of the visual field (e.g., failing to meet one or more visual criteria). Limiting the search area can, for example, reduce the amount of computational resources required to detect objects within the search area and generate associated alerts, or increase the speed of such detection and alert generation.

[0258] In some cases, two consecutive frames from a live video stream may be subtracted from each other to detect the movement of one or more objects. As an example, the occurrence of the movement may be stored in a first delta frame (e.g., delta frame 1), which may be used to visualize the moving object and cancel out the stationary background. Another two consecutive frames from the live video stream may be subtracted from each other to generate a second delta frame (e.g., delta frame 2). The second delta frame may also be used to visualize the moving object and cancel out the stationary background. Furthermore, a comparison between the first delta frame and the second delta frame may be performed. If an increase in size of the moving object is detected by subtracting the first delta frame from the second delta frame, the object may be determined to be approaching. If the increase in size exceeds a predetermined threshold size, an alert may be issued to the user (e.g., a visual alert displayed in an area visible to the user, an audible alert, etc.).

[0259] In some embodiments, the configuration subsystem 112 may store the predictive models, correction profiles, visual deficiency information (e.g., indicative of a detected user's visual deficiency), feedback information (e.g., feedback related to stimuli displayed to the user, or other feedback), or other information in one or more remote databases (e.g., the cloud). In some embodiments, the feedback information, visual deficiency information, correction profiles, or other information associated with multiple users (e.g., two or more users, ten or more users, one hundred or more users, one thousand or more users, one million or more users, or other number of users) may be used to train one or more predictive models. In one use case, if the predictive model being trained is a neural network or other machine learning model, the model manager subsystem 114 may provide the following (i) and (ii) as inputs to the machine learning model to predict the visual deficiency information, correction profiles, or other output: (i) stimulus information (e.g., indicative of a set of stimuli and their associated characteristics, such as intensity level, location where the stimuli are displayed, etc.), and (ii) feedback information (e.g., indicative of feedback associated with the set of stimuli). The model manager subsystem 114 may provide reference information (e.g., visual deficiency information or correction profiles determined to be accurate for the provided stimulus and feedback information) to the machine learning model. The machine learning model may evaluate its predicted output (e.g., predicted visual deficiency information, predicted correction profiles, etc.) against the reference information and update its configuration (e.g., weights, biases, or other parameters) based on the evaluation of the predicted output.The aforementioned operations may be performed using additional stimulus information (e.g., that displayed to other users), additional feedback information (e.g., feedback from other users related to the stimuli displayed to other users), and additional reference information to further train the machine learning model (e.g., providing such information as input and reference feedback to train the machine learning model so that the machine learning model can further update its configuration).

[0260] In another use case, if the machine learning model is a neural network, the connection weights may be adjusted to accommodate discrepancies between the neural network's predictions and reference information. In a further use case, one or more neurons (or nodes) of the neural network may request that their respective errors be sent backward through the neural network to facilitate the update process (e.g., backpropagation). The update to the connection weights may reflect, for example, the magnitude of the error to be propagated backward after forward propagation is complete.

[0261] In some embodiments, one or more predictive models may be trained or configured for a user or a predetermined type of device (e.g., a particular brand of device, a particular brand and model of device, a device with a particular set of features, etc.) and stored in association with the user or device type. As an example, instances of predictive models associated with a user or device type may be stored locally (e.g., on the user's wearable device or other user device) and remotely (e.g., in the cloud), and such instances of predictive models may be synchronized automatically or manually between one or more user devices and the cloud so that the user has access to the latest configuration of the predictive model on either the user device or the cloud. In one use case, upon detecting that a first user is using the wearable device (e.g., when the first user logs into their account or is identified via one or more other techniques), the configuration subsystem 112 may communicate with the wearable device to send the latest instance of the predictive model associated with the first user to the wearable device so that the wearable device can access a local copy of the predictive model associated with the first user. In another use case, if a second user is later detected using the same wearable device, the configuration subsystem 112 may communicate with the wearable device to send the latest instance of the predictive model associated with the second user to the wearable device so that the wearable device can access a local copy of the predictive model associated with the second user.

[0262] In some embodiments, multiple correction profiles may be associated with a user or a device type. In some embodiments, each correction profile may include a set of correction parameters or functions to be applied to live image data of a given context to generate an enhanced presentation of the live image data. As an example, a user may have a correction profile for each set of eye characteristics (e.g., a range of gaze directions, pupil size, limbus position, or other characteristics). As a further example, a user may additionally or alternatively have a correction profile for each set of environmental characteristics (e.g., a range of environmental light levels, environmental temperature, or other characteristics). Based on currently detected eye characteristics or environmental characteristics, a corresponding set of correction parameters or functions may be obtained and used to generate an enhanced presentation of the live image data. In one use example, upon detecting that a first user is using the wearable device (e.g., upon the first user logging in to their account or being identified via one or more other techniques), the configuration subsystem 112 may communicate with the wearable device to send a correction profile associated with the first user to the wearable device and enable the wearable device to access a local copy of the correction profile associated with the first user. In another use case, if a second user is later detected using the same wearable device, the configuration subsystem 112 may communicate with the wearable device to send the modified profile associated with the second user to the wearable device so that the wearable device can access a local copy of the modified profile associated with the second user.

[0263] 41 through 43 are example flowcharts illustrating process operations in methods for enabling various features and functions of the system described in detail above. The process operations of each method described below are for illustrative purposes only and are not intended to be limiting. In some embodiments, for example, the methods may be implemented with one or more additional processes not described and / or may omit one or more of the processes described. Additionally, the order in which the process operations of each method are illustrated (and described below) is not intended to be limiting.

[0264] In some embodiments, the methods may be implemented in one or more processing devices (e.g., digital processors, analog processors, digital circuits for processing information, analog circuits for processing information, state machines, and / or other mechanisms for electronically processing information). These processing devices may include one or more devices that perform some or all of the operations of the methods in response to instructions electronically stored on an electronic storage medium. These processing devices may include one or more devices configured with hardware, firmware, and / or software specifically designed to perform one or more of the operations of the methods.

[0265] FIG. 41 illustrates a flowchart of a method 4100 for facilitating correction of a user's vision via a predictive model, according to one or more embodiments.

[0266] In step 4102, a visual test presentation may be provided to the user. As an example, the visual test presentation may include a set of stimuli. The set of stimuli may include light stimuli, text, or images that are displayed to the user. According to one or more embodiments, step 4102 may be performed by a subsystem that is the same as or similar to test subsystem 122.

[0267] In step 4104, one or more characteristics of one or more eyes of the user may be monitored. As an example, the eye characteristics may be monitored during the visual test presentation. The eye characteristics may include gaze direction (e.g., during the visual test presentation), pupil size, limbus position, visual axis, optical axis, or other characteristics. According to one or more embodiments, step 4104 may be performed by a subsystem that is the same as or similar to test subsystem 122.

[0268] In step 4106, feedback related to the set of stimuli may be obtained. As an example, the feedback may be obtained during visual test presentation, and the feedback may indicate whether or how the user is viewing one or more stimuli of the set. Additionally or alternatively, the feedback may include one or more characteristics related to one or more eyes that occur when the one or more stimuli are displayed. According to one or more embodiments, step 4106 may be performed by a subsystem that is the same as or similar to test subsystem 122.

[0269] In step 4108, feedback related to the set of stimuli may be provided to the predictive model. As one example, the feedback may be provided to the predictive model during the visual test presentation, and the predictive model may be configured based on the feedback and the eye characteristic information. As another example, based on the feedback, the predictive model may provide modification parameters or functions to be applied to the image data (e.g., a live video stream) to generate an enhanced presentation related to the image data. According to one or more embodiments, step 4108 may be performed by a subsystem the same as or similar to the test subsystem 122.

[0270] At step 4110, the video stream data and the user's current eye characteristic information (e.g., indicative of the user's current eye characteristic) may be provided to the predictive model. As an example, the video stream data may be a live video stream obtained via one or more cameras of the user's wearable device, and the live video stream and the current eye characteristic information may be provided to the predictive model in real time. According to one or more embodiments, step 4110 may be performed by a subsystem that is the same as or similar to vision subsystem 124.

[0271] In step 4112, a set of correction parameters or functions may be obtained from the predictive model. As one example, the set of correction parameters or functions may be obtained from the predictive model based on the video stream and current eye characteristic information provided to the predictive model. As another example, the set of correction parameters or functions may be configured to be applied to the video stream to generate an enhanced image (e.g., an image that adapts to the user's dynamic aberrations). Additionally or alternatively, the set of correction parameters or functions may be configured to be applied to dynamically adjust one or more display portions of the display. According to one or more embodiments, step 4112 may be performed by a subsystem that is the same as or similar to visual subsystem 124.

[0272] In step 4114, an enhanced image may be displayed to a user based on the video stream data and the set of modification parameters or functions. According to one or more embodiments, step 4114 may be performed by a subsystem the same as or similar to vision subsystem 124.

[0273] FIG. 42 illustrates a flowchart of a method 4200 for facilitating an expansion of a user's field of view through a combination of portions of multiple images of a scene, according to one or more embodiments.

[0274] In step 4202, multiple images of a scene may be acquired. By way of example, the multiple images may be acquired via one or more cameras (e.g., cameras on a wearable device) at different positions or orientations. According to one or more embodiments, step 4202 may be performed by a subsystem that is the same as or similar to vision subsystem 124.

[0275] In step 4204, regions common to the multiple images may be determined. By way of example, the common regions may correspond to portions of the images that have the same or similar characteristics as one another. According to one or more embodiments, step 4204 may be performed by a subsystem that is the same as or similar to vision subsystem 124.

[0276] In step 4206, for each image of the plurality of images, regions of the image that differ from corresponding regions of at least one other image (of the plurality of images) may be determined. As an example, each difference region may correspond to a portion of one image of the plurality of images that is distinct from corresponding portions of all other images. According to one or more embodiments, step 4206 may be performed by a subsystem that is the same as or similar to vision subsystem 124.

[0277] In step 4208, an enhanced image may be generated based on the common region and the difference region. As one example, the enhanced image may be generated such that (i) a first region of the enhanced image includes a representation of the common region, and (ii) a second region of the enhanced image includes a representation of the difference region. As another example, the enhanced image may be generated such that the second region surrounds the first region in the enhanced image. According to one or more embodiments, step 4208 may be performed by a subsystem the same as or similar to vision subsystem 124.

[0278] In step 4210, the enhanced image may be displayed. By way of example, the enhanced image may be displayed via one or more displays of the user's wearable device. According to one or more embodiments, step 4210 may be performed by a subsystem that is the same as or similar to the vision subsystem 124.

[0279] FIG. 43 illustrates a flowchart of a method 4300 for facilitating an enhancement of a user's field of view via one or more dynamic display portions on one or more transparent displays, according to one or more embodiments.

[0280] Step 4302 may monitor one or more changes associated with one or more eyes of the user. By way of example, the eye changes may include eye movement, changes in gaze direction, changes in pupil size, or other changes. According to one or more embodiments, step 4302 may be performed by a subsystem the same as or similar to vision subsystem 124.

[0281] In step 4304, adjustments may be made to one or more transparent display portions of the wearable device based on the monitored changes. As an example, one or more positions, shapes, or sizes of one or more transparent display portions of the wearable device may be adjusted based on the monitored changes. According to one or more embodiments, step 4304 may be performed by a subsystem that is the same as or similar to vision subsystem 124.

[0282] In step 4306, the enhanced image (e.g., an image obtained from the live image data) may be displayed on one or more other display portions of the wearable device. As an example, at least one of the other display portions may be positioned around at least one of the transparent display portions of the wearable device such that the enhanced image is displayed around (e.g., not within) the transparent display portion. According to one or more embodiments, step 4306 may be performed by a subsystem that is the same as or similar to vision subsystem 124.

[0283] In some embodiments, the various computers and subsystems illustrated in FIG. 1A may include one or more computing devices programmed to perform the functions described herein. A computing device may include one or more electronic storage (e.g., one or more prediction databases 132, which may include one or more training data databases 134, one or more model databases 136, etc., or other electronic storage), one or more physical processors programmed with one or more computer program instructions, and / or other components. A computing device may include communications lines or ports that enable communication with a network (e.g., network 150) or other computing platforms via wired or wireless technologies (e.g., Ethernet, fiber optic, coaxial cable, WiFi, Bluetooth, near field communication, or other technologies). A computing device may include multiple hardware, software, and / or firmware components working together. For example, a computing device may be implemented by multiple computing platforms working together as a computing device.

[0284] Electronic storage may include non-transitory storage media that electronically store information. Electronic storage media may include one or both of: (i) system storage that is integral to (e.g., substantially non-removable from) a server or client device; or (ii) removable storage that is removably connected to the server or client device, for example, via a port (e.g., a USB port, a Firewire port, etc.) or a drive (e.g., a disk drive, etc.). Electronic storage may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drives, floppy drives, etc.), charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drives, etc.), and / or other electronically readable storage media. Electronic storage may also include one or more virtual storage resources (e.g., cloud storage, virtual private networks, and / or other virtual storage resources). The electronic storage may store software algorithms, processor-determined information, information obtained from a server, information obtained from a client device, or other information that enables the functionality described herein.

[0285] A processor may be programmed to implement information processing functions in a computing device. As such, a processor may include one or more of a digital processor, an analog processor, a digital circuit for processing information, an analog circuit for processing information, a state machine, and / or other mechanisms for electronically processing information. In some embodiments, a processor may include multiple processing units. These processing units may be located in the same physical device, or multiple processors may perform the processing functions of multiple devices working in coordination. A processor may be programmed to execute computer program instructions to implement the functionality of subsystems 112-124 or other subsystems described herein. A processor may be programmed to execute computer program instructions by software, hardware, firmware, any combination of software, hardware, or firmware, and / or other mechanisms for configuring processing functions in the processor.

[0286] It should be understood that the descriptions of the functionality provided by the different subsystems 112-124 described herein are for illustrative purposes and not intended to be limiting. Any of the subsystems 112-124 may provide more or less functionality than described. For example, one or more of the subsystems 112-124 may be omitted, and some or all of its functionality may be provided by other of the subsystems 112-124. As another example, additional subsystems may be programmed to perform some or all of the functionality attributed herein to one of the subsystems 112-124.

[0287] The technology disclosed herein may be used in any number of applications, including with healthy subjects, military personnel, and veterans, who frequently experience the rapid onset of optical-related disorders. Visual field defects impair military personnel, veterans, and other patients' essential abilities, as well as their ability to perform daily activities. Such visual impairments reduce independence, safety, productivity, and quality of life, leading to low self-esteem and depression. Despite recent scientific advances, treatment options to reverse damage to the retina, optic nerve, or visual cortex are limited. As a result, treatments often involve providing patients with visual aids to maximize function. Current visual aids are inadequate to achieve this goal. Therefore, there is a growing need for better visual aids to improve visual function, quality of life, and safety. The technology described herein, when incorporated into eyewear devices, can diagnose and mitigate common eye injuries, such as military-related eye injuries and disorders, that result in rapid onset visual field defects in both general as well as austere or isolated environments. The technology described herein is capable of diagnosing and quantifying visual field defects. The device uses this data to process the patient's visual field in real time and project a corrected image that matches the remaining functional visual field. This minimizes the negative impact of blind spots (or areas of reduced vision) in the visual field on the patient's visual function. Furthermore, the eyewear device does not rely on a separate medical device to diagnose visual field defects, making it particularly useful in harsh or isolated environments. Similarly, the technology described herein may be used to enhance the visual field of a normal subject, achieving better-than-normal vision or visual acuity.

[0288] While the present invention has been described in detail for purposes of illustration, based on what are presently considered to be the most practical and preferred embodiments, it should be understood that such detailed description is for illustrative purposes only. The present invention is not limited to the disclosed embodiments, but rather, it is intended to cover modifications and equivalent arrangements that fall within the spirit and scope of the appended claims. For example, it should be understood that the present invention may be combined, to the extent possible, with one or more features of any other embodiment.

[0289] The present techniques will be better understood with reference to the following enumerated embodiments. [A1] A method comprising: providing a presentation (e.g., a visual test presentation or other presentation) to a user that includes a set of stimuli; obtaining feedback regarding the set of stimuli (e.g., the feedback indicating whether or how the user perceives one or more stimuli of the set); and providing the feedback regarding the set of stimuli to a model (e.g., a machine learning model or other model), wherein the model is configured based on the feedback regarding the set of stimuli. [A2] The method of embodiment A1 further comprising the steps of: providing live image data, eye characteristic information, or environmental characteristic information to the model to obtain an enhanced image derived from the live image data; and displaying the enhanced image to the user, wherein the eye characteristic information indicates one or more characteristics of one or more eyes of the user that occurred during live capture of the live image data, and the environmental characteristic information indicates one or more characteristics of the environment that occurred during live capture of the live image data. [A3] The method of embodiment A2, further comprising obtaining the enhanced image from the model based on the live image data, eye characteristic information, or environmental characteristic information provided to the model. [A4] The method of embodiment A2, further comprising: obtaining one or more correction parameters from the model based on the live image data, eye characteristic information, or environmental characteristic information provided to the model; and generating the enhanced image based on the live image data or the one or more correction parameters to obtain the enhanced image. [A5] The method of embodiment A4, wherein the one or more modification parameters include one or more of a transformation parameter, a brightness parameter, a contrast parameter, a saturation parameter, or a sharpness parameter. [A6] The method of any of embodiments A1 to A5, wherein obtaining feedback regarding the set of stimuli includes obtaining images of eyes captured during the presentation, the images of the eyes being images of the user's eyes, and providing feedback regarding the set of stimuli includes providing the images of the eyes to the model. [A7] The method of embodiment A5, wherein the image of the eye is an ocular image, an image of the retina of the eye, or an image of the cornea of ​​the eye. [A8] The method of any of embodiments A1 to A7, wherein obtaining feedback regarding the set of stimuli includes obtaining an indication of the user's response to one or more stimuli in the set of stimuli, or an indication of the user's lack of response to one or more stimuli in the set of stimuli, and providing feedback regarding the set of stimuli includes providing the indication of the response or lack of response to the model. [A9] The method of embodiment A8, wherein the response includes eye movement, gaze direction, change in pupil size, or user modification to one or more stimuli by user input from the user. [A10] The method of embodiment A9, wherein the user modifications include movement of one or more stimuli by user input from the user, or supplemental data provided by user input from the user for one or more stimuli displayed to the user. [A11] The method of any of embodiments A1 to A10, further comprising: obtaining a second set of stimuli, the second set of stimuli being generated based on the model's processing of the set of stimuli and feedback regarding the set of stimuli; displaying the second set of stimuli to the user; obtaining feedback regarding the second set of stimuli (e.g., the feedback indicating whether or how the user views one or more stimuli in the second set); and providing the feedback regarding the second set of stimuli to the model, the model being further configured based on the feedback regarding the second set of stimuli. [A12] The method of any of embodiments A1 to A11, further comprising a step of determining a defective visual field portion of the user's visual field using the model based on feedback regarding the set of stimuli, wherein the user's visual field includes a plurality of visual field portions, and the defective visual field portion is one of the plurality of visual field portions that does not meet one or more visual criteria. [A13] The method of embodiment A12, wherein the enhanced image is based on one or more transformations corresponding to the defect field of view portion of the live image data, and the image portion of the live image data is represented in an image portion of the enhanced image that is outside the defect field of view portion. [A14] The method of any of embodiments A12 to A13, wherein the enhanced image is based on one or more modifications of brightness or contrast of the live image data, and (i) an increase in brightness, contrast, or sharpness level is applied to an image portion of the live image data corresponding to the defective field of view portion to generate the corresponding image portion of the enhanced image, and (ii) the increase in brightness, contrast, or sharpness level is not applied to other image portions of the live image data to generate the corresponding image portion of the enhanced image. [A15] The method of any of embodiments A12 to A14, further comprising the steps of: detecting an object (e.g., in the defective field of view portion or expected to be in the defective field of view portion); determining that the object is not sufficiently visible in any image portion of the enhanced image corresponding to at least one of the field of view portions that meets the one or more visual criteria; generating a prediction indicating that the object will come into physical contact with the user; and displaying a warning (e.g., on the enhanced image) based on (i) the prediction of physical contact and (ii) the determination that the object is not sufficiently visible in an image portion of the enhanced image corresponding to at least one field of view portion that meets the one or more visual criteria, the warning indicating a direction in which the object is approaching. [A16] The method of any of embodiments A1 to A15, wherein one or more of the preceding steps are performed by a wearable device. [A17] The method of embodiment A16, wherein the wearable device includes one or more cameras configured to capture the live image data and one or more display portions configured to display one or more enhanced images. [A18] The method of any of embodiments A16 to A17, wherein the one or more display portions include first and second display portions of the wearable device. [A19] The method of embodiment A18, wherein the wearable device includes a first monitor including the first display portion and a second monitor including the second display portion. [A20] The method of any of embodiments A16 to A19, wherein the one or more display portions include one or more dynamic display portions on one or more transparent displays of the wearable device, and the one or more display portions display one or more enhanced images. [A21] The method of any of embodiments A1 to A20, further comprising monitoring one or more changes related to one or more eyes of the user. [A22] The method of embodiment A21 further comprising the steps of providing the one or more changes to the model as further feedback, obtaining one or more correction parameters from the model based on the live image data, eye characteristic information, or environmental characteristic information provided to the model, and generating the enhanced image based on the live image data and the one or more correction parameters to obtain the enhanced image. [A23] The method of any of embodiments A21 to A22, further comprising adjusting one or more positions, shapes, sizes, or transparency of the first or second display portions on one or more transparent displays of the wearable device based on the monitoring, wherein displaying the enhanced image comprises displaying the enhanced image on the first or second display portions. [A24] The method of any of embodiments A1 to A23, wherein the model comprises a neural network or other machine learning model. [B1] A method comprising the steps of acquiring a plurality of images of a scene; determining areas common to the plurality of images; determining, for each image of the plurality of images, areas of the image that have differences from corresponding areas of at least one other image of the plurality of images; generating an enhanced image based on the common areas and the different areas; and displaying the enhanced image. [B2] The method of embodiment B1, wherein generating the enhanced image includes generating the enhanced image based on the common region and the difference region, wherein (i) a first region of the enhanced image includes a representation of the common region, (ii) a second region of the enhanced image includes a representation of the difference region, and (iii) the second region surrounds the first region in the enhanced image. [B3] The method of embodiment B2, wherein generating the enhanced image includes generating the enhanced image based on the common region, the difference region, and a second region common to the plurality of images, wherein (i) the first region of the enhanced image includes a representation of the common region and a representation of the second common region, and (ii) the second region of the enhanced image includes a representation of the difference region. [B4] The method of any of embodiments B1 to B3, wherein the common region is a region of at least one image of the plurality of images that corresponds to a macular region of the visual field of the eye or corresponds to a region within the macular region of the visual field. [B5] The method of any of embodiments B1 to B4, wherein each of the difference regions is a region of at least one image of the plurality of images that corresponds to a peripheral region of the eye's visual field or corresponds to a region within the peripheral region of the visual field. [B6] The method of any of embodiments B1 to B5, further comprising a step of shifting each of the plurality of images, wherein generating the enhanced image includes, following the shifting, generating the enhanced image based on the common regions and the difference regions. [B7] The method of embodiment B6, wherein the shifting step includes shifting each image of the plurality of images such that the size of the common area decreases and the size of at least one of the difference areas increases. [B8] The method of any of embodiments B1 to B7, further comprising a step of resizing one or more regions of the plurality of images, wherein generating the enhanced image includes, following the resizing step, generating the enhanced image based on the common regions and the difference regions. [B9] The method of embodiment B8, wherein the resizing step includes resizing one or more regions of the plurality of images such that an arbitrary resizing range of the common region is different from an arbitrary resizing range of at least one of the difference regions. [B10] A method of any of embodiments B8 to B9, wherein the resizing step includes resizing one or more regions of the multiple images so that a rate of change in size of the common region represented in a first region of the enhanced image is greater than or less than a rate of change in size of at least one of the difference regions represented in a second region of the enhanced image. [B11] The method of embodiment B10, wherein the rate of change in size of at least one of the difference regions is zero and the rate of change in size of the common region is greater than zero. [B12] The method of embodiment B10, wherein the rate of change in size of at least one of the difference regions is greater than 0, and the rate of change in size of the common region is 0. [B13] The method of any of embodiments B1 to B12, further comprising performing a fisheye transformation, a conformal mapping transformation, or other transformation on the common region, and generating the enhanced image includes generating the enhanced image based on the common region and the difference region subsequent to performing the transformation. [B14] The method of any of embodiments B1 to B13, further comprising determining a defective field of view portion of the user's field of view, wherein the user's field of view includes a field of view portion, the defective field of view portion being one of the field of view portions that does not satisfy one or more visual criteria, and wherein the enhanced image is generated based on the determined defective field of view portion, and at least one of the common area or the difference area in the enhanced image does not overlap with the defective field of view portion of the user's field of view. [B15] The method of any of embodiments B1 to B14, further comprising determining a field of view portion of the user's field of view that satisfies (i) one or more visual criteria, (ii) one or more position criteria, and (iii) one or more size criteria, and generating the enhanced image based on the field of view portion, wherein at least one of the common area or the difference area in the enhanced image is within the field of view portion. [B16] The method of embodiment B15, wherein the one or more size criteria include a requirement that the field of view portion be the largest field of view portion of the user that satisfies the one or more visual criteria and the one or more position criteria. [B17] The method of any of embodiments B15 to B16, wherein the one or more position criteria include a requirement that the center of the field of view portion corresponds to a point within the macular region of the user's eye. [B18] The method of any of embodiments B1 to B17, wherein one or more of the preceding steps are performed by a wearable device. [B19] The method of embodiment B18, further comprising the step of making one or more display portions of the wearable device transparent, and wherein the step of displaying the enhanced image includes the step of displaying the enhanced image on one or more other display portions of the wearable device other than the one or more transparent display portions. [B20] The method of embodiment B19, further comprising adjusting the one or more transparent display portions and the one or more other display portions of the wearable device. [B21] The method of embodiment B20, further comprising monitoring one or more changes related to one or more eyes of the user, wherein the adjusting step includes adjusting the one or more transparent display portions and the one or more other display portions of the wearable device based on the monitoring. [B21] The method of embodiment B20, further comprising monitoring one or more changes related to one or more eyes of the user, wherein the adjusting step includes adjusting the one or more transparent display portions and the one or more other display portions of the wearable device based on the monitoring. [B22] A method according to any one of B20 to B21, wherein the adjusting step includes adjusting one or more positions, shapes, sizes, or transparencies of one or more transparent display portions of the wearable device based on the monitoring. [B23] The method of any of embodiments B20 to B22, wherein the enhanced image or the adjustment is based on the one or more changes. [B24] The method of any of embodiments B18 to B23, wherein the step of displaying the enhanced image displays one or both of the common area or the difference area on the one or more other display portions of the wearable device, and at least one of the common area or the difference area is not displayed on the one or more transparent display portions of the wearable device. [B25] The method of any of embodiments B18 to B24, wherein the wearable device includes first and second cameras, and wherein obtaining the plurality of images includes obtaining at least one of the plurality of images with a first camera of the wearable device, and obtaining at least one other of the plurality of images with a second camera of the wearable device. [B26] The method of any of embodiments B18 to B25, wherein the one or more monitors of the wearable device include a first and a second monitor, and wherein displaying the enhanced image includes displaying the enhanced image by the first and second monitors. [B27] The method of any of embodiments B18 to B26, wherein the wearable device comprises a wearable eyewear apparatus. [B28] The method of any of embodiments B1 to B27, wherein the enhanced image or the adjustment is based on feedback regarding a set of stimuli (e.g., feedback indicating whether or how the user perceives one or more stimuli). [C1] A method comprising the steps of: monitoring one or more changes relating to one or more eyes of a user; adjusting one or more transparent display portions or one or more other display portions of a wearable device based on said monitoring; and displaying an enhanced image on one or more other display portions of the wearable device, wherein the enhanced image is based on live image data acquired via the wearable device. [C2] The method of embodiment C1, wherein the adjusting method includes adjusting one or more positions, shapes, sizes, brightness levels, contrast levels, sharpness levels, or saturation levels of one or more transparent display portions of the wearable device or one or more other display portions of the wearable device based on the monitoring. [C3] The method of any of embodiments C1 to C2, further comprising determining a defective visual field portion of the user's visual field, wherein the user's visual field includes a visual field portion, the defective visual field portion being one of the visual field portions that does not meet one or more visual criteria, and wherein the adjusting step includes adjusting one or more positions, shapes, or sizes of one or more transparent display portions of the wearable device so that they do not overlap the defective visual field portion. [C4] The method of embodiment C3, further comprising the steps of: detecting an object (e.g., in the defective field of view portion or expected to be in the defective field of view portion); determining that the object is not sufficiently visible in any image portion of the enhanced image corresponding to at least one field of view portion that satisfies one or more visual criteria; generating a prediction indicating that the object will come into physical contact with the user; and displaying a warning (e.g., on the enhanced image) based on (i) the prediction of physical contact and (ii) the determination that the object is not sufficiently visible in the image portion of the enhanced image corresponding to at least one field of view portion that satisfies the one or more visual criteria, the warning indicating a direction in which the object is approaching. [C5] The method of any of embodiments C1 to C4, further comprising: providing information about the one or more eyes to a model, wherein the model is configured based on information about the one or more eyes; and, subsequent to configuring the model, providing the one or more monitored changes about the one or more eyes to the model to obtain a set of correction parameters, wherein adjusting the one or more transparent display portions includes adjusting the one or more transparent display portions based on one or more correction parameters of the set of correction parameters. [C6] The method of embodiment C5, wherein the information about the one or more eyes includes one or more images of the one or more eyes. [C7] The method of any of embodiments C5 to C6, wherein the information about the one or more eyes includes feedback about a set of stimuli (e.g., feedback indicating whether or how the user is perceiving one or more stimuli). [C8] The method of any of embodiments C1 to C7, wherein the one or more changes include eye movement, a change in gaze direction, or a change in pupil size. [C9] The method of any of embodiments C1 to C8, wherein the enhanced image or the adjustment is based on feedback regarding a set of stimuli (e.g., feedback indicating whether or how the user perceives one or more stimuli). [C10] The method of any of embodiments C10 to C9, wherein the enhanced image or the adjustment is based on the one or more changes. [C11] The method of any of embodiments C1 to C10, wherein the adjustment occurs simultaneously with displaying the enhanced image. [C12] The method of any of embodiments C1 to C11, wherein one or more of the preceding steps are performed by the wearable device. [C13] The method of any of embodiments C1 to C12, wherein the wearable device comprises a wearable eyewear apparatus. [D1] A method comprising the steps of: monitoring one or more eyes of a user (e.g., during a first monitoring period during which a set of stimuli is displayed to the user); obtaining feedback regarding the set of stimuli (e.g., during the first monitoring period); and generating a set of correction profiles associated with the user based on the feedback regarding the set of stimuli, each correction profile in the set of correction profiles (i) being associated with a set of eye characteristics and (ii) including one or more correction parameters to be applied to an image for the user to correct the image when the user's eye characteristics match the associated set of eye characteristics. [D2] The method of embodiment D1, wherein the feedback regarding the set of stimuli indicates whether or how the user is viewing one or more stimuli of the set of stimuli. [D3] A method according to any of embodiments D1 to D2, wherein the feedback regarding the set of stimuli includes one or more characteristics regarding one or more eyes that occur when the one or more stimuli are displayed (e.g., during the first monitoring period). [D4] The method of any of embodiments D1 to D3, further comprising: monitoring one or more eyes of the user (e.g., during a second monitoring period); acquiring image data representative of the user's environment (e.g., during the second monitoring period); acquiring (e.g., from the second monitoring period) one or more correction profiles associated with the user based on (i) the image data or (ii) characteristics of the one or more eyes; and displaying corrected image data (e.g., during the second monitoring period) to the user based on (i) the image data and (ii) the one or more correction profiles. [D5] The method of embodiment D4, wherein the one or more eye characteristics include gaze direction, pupil size, limbus position, visual axis, optical axis, or eyelid position or movement. [D6] The method of any of embodiments D1 to D5, wherein obtaining feedback regarding the set of stimuli includes obtaining eye images captured during the first monitoring period, the eye images being images of the user's eyes, and generating the set of correction profiles includes generating the set of correction profiles based on the eye images. [D7] The method of embodiment D6, wherein the image of the eye is an image of the retina of the eye or an image of the cornea of ​​the eye. [D8] The method of any of embodiments D1 to D7, wherein obtaining feedback regarding the set of stimuli includes obtaining an indication of the user's response to the one or more stimuli or an indication of the user's lack of response to the one or more stimuli, and generating the set of modified profiles includes generating the set of modified profiles based on the indication of response or the indication of lack of response. [D9] The method of embodiment D8, wherein the response comprises an eye movement, a change in gaze direction, or a change in pupil size. [D10] The method of any of embodiments D1 to D9, wherein one or more of the preceding steps are performed by a wearable device. [D11] The method of embodiment D10, wherein the wearable device includes a wearable eyewear apparatus. [E1] A method comprising: displaying a first stimulus at a first interface location on a user interface for a user based on a fixation point for a visual test presentation; adjusting, during the visual test presentation, the fixation point for the visual test presentation based on eye characteristic information for the user, the eye characteristic information indicative of one or more characteristics of one or more eyes of the user that occurred during the visual test presentation; displaying a second stimulus at a second interface location on the user interface based on the adjusted fixation point for the visual test presentation; obtaining feedback information indicative of feedback regarding the first stimulus and feedback regarding the second stimulus, the feedback regarding the first or second stimulus indicative of the user's response or lack of response to the first or second stimulus; and generating visual deficiency information associated with the user based on the feedback information. [E2] The method of embodiment E1, wherein the user interface is configured to display a view having a horizontal dimension corresponding to a first power or a vertical dimension corresponding to the first power, and the visual defect information has a coverage greater than the first power with respect to the horizontal dimension of the user's field of view or with respect to the vertical dimension of the user's field of view. [E3] The method of any of embodiments E1 to E2, wherein the user interface is configured to display a view having a given dimension corresponding to a first power, and the visual defect information is generated such that (i) the visual defect information indicates at least two defects present at field of view locations in the user's field of view, and (ii) the field of view locations are spaced apart by more than the first power for the given dimension of the user's field of view. [E4] The method of any of embodiments E1 to E3, wherein the user interface is configured to display a view having a given dimension corresponding to a first power, and the feedback information further indicates feedback regarding a third stimulus displayed on the user interface during the visual test presentation, and the method comprises: determining, based on the feedback information, whether a visual defect exists at a visual field position in the user's visual field such that at least two of the visual field positions are separated from each other by more than the first power for the given dimension of the visual field; and generating the visual defect information based on determining whether a visual defect exists at the visual field position. [E5] The method of any of embodiments E1 to E4, further comprising: determining a first interface position of a first stimulus based on a fixation point for the visual test presentation and a first relative position associated with the first stimulus; and determining a second interface position of a second stimulus based on an adjusted fixation point for the visual test presentation and a second relative position associated with the second stimulus, wherein displaying the first stimulus comprises displaying the first stimulus at the first interface position on the user interface during the visual test presentation based on the determination of the first interface position; and displaying the second stimulus comprises displaying the second stimulus at the second interface position on the user interface during the visual test presentation based on the determination of the second interface position. [E6] The method of any of embodiments E1 to E5, further comprising the step of selecting the first interface location for the first stimulus based on the first interface location being farther from the fixation point than one or more other interface locations on the user interface during the visual test presentation, the one or more other interface locations corresponding to one or more other visual field locations of the test set, and wherein displaying the first stimulus comprises displaying the first stimulus at the first interface location on the user interface during the visual test presentation based on the selection of the first interface location. [E7] The method of embodiment E6, further comprising deleting the first field of view location from the test set. [E8] The method of embodiment E7, wherein the step of deleting the first visual field location includes a step of deleting the first visual field location from the test set such that the first visual field location is no longer selectable from the test set during the visual test presentation. [E9] The method of any of embodiments E7 to E8, further comprising, following the step of deleting the first visual field location from the test set, selecting the second interface location for the second stimulus based on the second interface location being farther from the adjusted fixation point than one or more other interface locations on the user interface, and wherein displaying the second stimulus includes displaying the second stimulus at the second interface location on the user interface during the visual test presentation based on the selection of the second interface location. [E10] The method of any of embodiments E6 to E9, wherein selecting the first interface position includes selecting the first interface position for the first stimulus based on the first interface position being at least the same distance away from the fixation point as all other interface positions on the user interface corresponding to visual field positions in the test set other than the first visual field position, for a given dimension. [E11] The method of any of embodiments E6 to E10, wherein selecting the second interface position includes selecting the second interface position for the second stimulus based on the second interface position being at least the same distance away from the adjusted fixation point as all other interface positions on the user interface corresponding to visual field positions in the test set other than the second visual field position, for a given dimension. [E12] The method of any of embodiments E1 to E11, further comprising: locking the adjusted fixation point, wherein readjustment of the fixation point is avoided while the adjusted fixation point is locked; displaying one or more stimuli on the user interface based on the adjusted fixation point while the adjusted fixation point is locked; and unlocking the adjusted fixation point before displaying the second stimulus. [E13] The method of any of embodiments E1 to E12, further comprising a step of de-emphasizing or removing a plurality of stimuli from the user interface after displaying the plurality of stimuli on the user interface while the adjusted fixation point is in the same position (e.g., the first interface position), wherein at least one stimulus of the plurality of stimuli is displayed on the user interface subsequent to at least one other stimulus of the plurality of stimuli being displayed on the user interface. [E14] The method of embodiment E13, wherein the plurality of stimuli are de-highlighted or removed after being displayed while the first stimulus continues to be displayed on the user interface at the first interface location. [E15] The method of any of embodiments E13 to E14, further comprising a step of de-highlighting or removing the first stimulus from the user interface following the display of at least one stimulus of the plurality of stimuli on the user interface, and then highlighting or re-displaying the first stimulus at the first interface location on the user interface. [E16] The method of any of embodiments E1 to E15, wherein the eye characteristic information indicates one or more of the user's gaze direction, changes in pupil size, eyelid movement, head movement, or other eye characteristics that occurred during the visual test presentation. [F1] A method comprising: monitoring eye characteristics of a user during visual test presentation via two or more user interfaces (e.g., on two or more displays) provided for each eye, the user's eyes including a first and a second eye; presenting one or more stimuli at one or more locations on at least one of the user interfaces; and determining visual deficiency information for the first eye (e.g., of the first eye) based on one or more eye characteristics occurring during the stimulus presentation. [F2] The method of embodiment F1, wherein determining the visual defect information includes determining a deviation measure for the first eye based on one or more eye-related characteristics of the first eye that occurred during the stimulus presentation. [F3] The method of embodiment F2, wherein the deviation measurement indicates a deviation of the first eye relative to the second eye. [F4] The method of any of embodiments F1 to F3, wherein the step of presenting the stimulus includes presenting the stimulus at a first time at a location on a first user interface used for the first eye, the presentation of the stimulus occurring while no stimulus is presented on a second user interface used for the second eye. [F5] The method of any of embodiments F1 to F4, wherein the step of presenting the stimulus includes presenting a stimulus at a location on the first user interface while the stimulus intensity of the second user interface does not meet a stimulus intensity threshold. [F6] The method of any of embodiments F4 to F5, further comprising causing a stimulus to be presented at the location on the second user interface at a prior time (before the first time) while no stimulus is presented on the first user interface. [F7] The method of any of embodiments F4 to F6, further comprising the steps of: presenting a stimulus at a first location on the first display and presenting a stimulus at a first location on the second display at a prior time before the first time; detecting that the first eye does not fixate at the first location when the stimulus is presented on the first display at the prior time; and determining that the user's first eye is the deviated eye based on detecting that the first eye is not fixating. [F8] The method of any of embodiments F4 to F7, further comprising: presenting a stimulus at a modified position on the first display at a subsequent time subsequent to the first time based on the visual deficiency information (e.g., the deviation measurement), wherein the presentation at the subsequent time occurs while no stimulus is presented on the second display and the modified position is different from the first position; and determining the visual deficiency information (e.g., the deviation measurement) based on characteristics of one or more of the first eye or the second eye that do not change by more than a change threshold upon presentation at the subsequent time. [F9] The method of embodiment F8, further comprising determining the corrected position as the position at which a stimulus will be presented on the first display at the subsequent time based on the visual defect information (e.g., the deviation measurement value). [F10] The method of any of embodiments F1 to F2, wherein presenting the stimulus includes presenting the stimulus at a given time at a location on a first user interface for the first eye and at a corresponding location on a second user interface for the second eye. [F11] The method of any of embodiments F1 to F10, further comprising generating a correction profile associated with the user based on the visual defect information (e.g., the deviation measurements), the correction profile including one or more correction parameters to be applied to correct an image for the user. [F12] The method of embodiment F11, further comprising: (i) video stream data representative of the user's environment; and (ii) displaying modified video stream data to the user based on a modified profile associated with the user. [F13] The method of embodiment F12, wherein the modification profile includes translation or rotation parameters that are applied to modify the image for the first eye when the gaze direction of the second eye is directed toward the first position, and wherein displaying the modified video stream data includes detecting that the gaze direction of the second eye is directed toward the first position, modifying the video stream data using the translation or rotation parameters based on the detection of the gaze direction of the second eye to generate the modified video stream data, and displaying the modified video stream data to the first eye of the user. [F14] The method of any of embodiments F1 to F13, further comprising: generating a first correction profile associated with the user based on the deviation measurement, the first correction profile including one or more correction parameters that are applied to correct the image for the first eye in response to the gaze direction of the second eye being directed to the first position; and generating a second correction profile based on a second deviation measurement for the first eye, the second correction profile including one or more correction parameters that are applied to correct the image for the first eye in response to the gaze direction of the second eye being directed to a second position different from the first position. [F15] The method of any of embodiments F1 to F14, wherein determining the visual defect information includes determining whether the user has diplopia or determining the degree of diplopia based on the number or type of stimuli the user views. [F16] The method of embodiment F15, further comprising determining the number or type of stimuli the user views based on user input indicating the number or type of stimuli the user views. [F17] The method of any of embodiments F15 to F16, further comprising determining the number or type of stimuli the user sees based on one or more eye-related characteristics that occur during presentation of the stimuli. [F18] The method of any of embodiments F1 to F17, wherein determining the visual deficiency information includes determining whether the user has stereoscopic vision or determining the degree of stereoscopic vision based on characteristics of one or more eyes that occur during presentation of the stimuli. [F19] The method of any of embodiments F1 to F18, wherein the eye characteristics include one or more of gaze direction, change in pupil size, or other eye characteristics. [G1] A tangible, non-transitory machine-readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform a process including the steps of any of embodiments A1 to A24, B1 to B28, C1 to C13, D1 to D11, E1 to E16, or F1 to F19. [G2] A system comprising one or more processors and a memory storing instructions, the instructions, when executed by the one or more processors, causing the one or more processors to perform a process including any of steps A1 to A24, B1 to B28, C1 to C13, D1 to D11, E1 to E16, or F1 to F19.

Claims

1. 1. A system for facilitating generation of visual deficiency information using a dynamic fixation point, comprising: a computer system having one or more processors programmed with computer program instructions, the computer system, when executed, displaying a first stimulus at a first interface location on a user interface of a user's wearable device during visual test presentation, the first interface location being at least as far away from a fixation point for the visual test presentation as one or more other interface locations on the user interface, the first interface location corresponding to a first visual field location of a test set, and the one or more other interface locations corresponding to one or more other visual field locations of the test set; adjusting a fixation point for the visual test presentation based on eye characteristic information associated with the user during the visual test presentation, the eye characteristic information indicating one or more characteristics of one or more eyes of the user that occurred during the visual test presentation after the first stimulus was displayed at the first interface location; displaying a second stimulus on the user interface at a second interface location during the visual test presentation subsequent to the adjustment of the fixation point; obtaining feedback information during the visual test presentation indicating feedback regarding the first stimulus and feedback regarding the second stimulus, the feedback regarding the first or second stimulus indicating the user's response or lack of response to the first or second stimulus; and generating vision deficiency information associated with the user based on the feedback information.

2. the computer system, locking the adjusted fixation point, wherein readjustment of the fixation point is avoided while the adjusted fixation point is locked; displaying one or more stimuli on the user interface based on the adjusted fixation point while the adjusted fixation point is locked; and unlocking the adjusted fixation point before displaying the second stimulus.

3. a user interface of the wearable device configured to display a view having a given dimension corresponding to a first degree; 2. The system of claim 1, wherein the visual defect information is generated such that (i) the visual defect information indicates at least two defects present at field locations in the user's field of vision, and (ii) the field locations are spaced apart by more than the first degree for the given dimension of the user's field of vision.

4. a user interface of the wearable device configured to display a view having a given dimension corresponding to a first degree; the feedback information further indicates feedback regarding a third stimulus displayed on a user interface of the wearable device during the visual test presentation; and The computer system includes: determining, based on the feedback information, whether a visual defect exists at field locations in the user's visual field, at least two of the field locations being separated from one another by more than the first number of degrees for the given dimension of the visual field; and generating the vision defect information based on a determination of whether a vision defect exists at the field of view location.

5. 2. The system of claim 1, wherein the first interface location is selected for the first stimulus based on the first interface location being at least as far away from the fixation point as all other interface locations on the user interface that correspond to visual field positions of the test set for a given dimension.

6. the second interface position corresponds to a second field of view position of the test set; 2. The system of claim 1, wherein the second interface location is selected for the second stimulus based on the second interface location being at least as far away from the adjusted fixation point as all other interface locations on the user interface that correspond to visual field positions of the test set for a given dimension.

7. The system of claim 1 , wherein the eye characteristic information indicates one or more of the user's gaze direction, pupil size change, eyelid movement, or head movement that occurred during the visual test presentation.

8. 1. A method implemented by a computer system having one or more processors executing computer program instructions, the computer program instructions performing the method when executed, the method comprising: displaying a first stimulus at a first interface location on a user interface of a wearable device of the user during the visual test presentation; adjusting a fixation point for the visual test presentation based on eye characteristic information associated with the user during the visual test presentation, wherein the fixation point is adjusted to the first interface location, and the eye characteristic information indicates one or more characteristics of one or more eyes of the user that occurred during the visual test presentation after the first stimulus is displayed at the first interface location; displaying one or more stimuli on the user interface based on the fixation point at the first interface location during the visual test presentation; during the visual test presentation, (i) while the fixation point is at the first interface location, and (ii) subsequent to displaying the one or more stimuli on the user interface, displaying a second stimulus at a second interface location on the user interface; obtaining feedback information during the visual test presentation, the feedback information indicating feedback regarding the first stimulus, feedback regarding the one or more stimuli, and feedback regarding the second stimulus; and generating vision deficiency information associated with the user based on the feedback information.

9. locking the fixation point at the first interface position, wherein readjustment of the fixation point is avoided while the fixation point is locked; displaying the one or more stimuli on the user interface based on the fixation point being at the first interface location while the fixation point is locked; 9. The method of claim 8, further comprising unlocking the fixation point before displaying the second stimulus.

10. displaying the one or more stimuli, displaying a plurality of stimuli at an interface location different from the first interface location while the fixation point is at the first interface location, and then de-emphasizing or removing the plurality of stimuli from the user interface; 9. The method of claim 8, wherein at least one stimulus of the plurality of stimuli is displayed on the user interface subsequent to at least one other stimulus of the plurality of stimuli being displayed on the user interface.

11. 11. The method of claim 10, further comprising: causing the first stimulus to be de-highlighted or removed from the user interface following display of at least one stimulus of the plurality of stimuli on the user interface, and then causing the first stimulus to be highlighted or re-displayed on the user interface at the first interface location.

12. 9. The method of claim 8, wherein the first interface location is selected for the first stimulus based on the first interface location being at least as far away from the fixation point as all other interface locations on the user interface that correspond to visual field locations of a test set for a given dimension.

13. the second interface position corresponds to a second field of view position of the test set; 13. The method of claim 12, wherein the second interface location is selected for the second stimulus based on being at least as far away from the adjusted fixation point as all other interface locations on the user interface that correspond to visual field locations in the test set for a given dimension.

14. The method of claim 8 , wherein the eye characteristic information indicates one or more of gaze direction, pupil size change, eyelid movement, or head movement of the user that occurred during the visual test presentation.

15. One or more non-transitory computer-readable storage media comprising instructions that, when executed by one or more processors, displaying a first stimulus at a first interface location on a user interface of a wearable device of the user during the visual test presentation; adjusting a fixation point for the visual test presentation based on eye characteristic information associated with the user during the visual test presentation, wherein the fixation point is adjusted to the first interface location, and the eye characteristic information indicates one or more characteristics of one or more eyes of the user that occurred during the visual test presentation after the first stimulus is displayed at the first interface location; displaying one or more stimuli on the user interface based on the fixation point at the first interface location during the visual test presentation; during the visual test presentation, (i) while the fixation point is at the first interface location, and (ii) subsequent to displaying the one or more stimuli on the user interface, displaying a second stimulus at a second interface location on the user interface; and generating visual deficiency information associated with the user based on feedback information indicative of feedback regarding the first stimulus, feedback regarding the one or more stimuli, and feedback regarding the second stimulus.

Images